Link Spider

How Googlebot webcrawler works - an example

2011-03-21T18:11:00.000-07:00

I've been wondering how does the Google spider work. The main webcrawler for Google is called Googlebot. So I created a website and checked my logs.

For this test I created a nine page website. It consists of the home page and three others, all linked to each other with symmetrical linking. The effect is that all four pages are linked equally. Then I added a fifth web page that was the top page in a four page section. I had one of the first four pages link to the top page of the new second, then had the top page link to the four new pages. Each of the four new pages links only back to the home page.

The first impression when Googlebot does visit will be to see the home page with only four internal links. Since my website is new and not linked from anything, I used a blog post to embed a link to the site. This pinged Google and got the process started.

I don't know how long it took for Googlebot to show up and request the frontpage and robots. It might have been a few minutes, it might have been a few hours. I forgot to clock it. But is was less than 6 hours I think.

Google got the links to the three other main pages and went silent. Then in about three and a half hours Googlebot came back and got the first of the three other pages. It got the remaining pages about one every nine minutes.

Then about two and a half hours later Googlebot came got the section header top page it found on one of the first three pages. Then about three hours later Googlebot came and got each of the four section pages, about one every nine minutes.

What I did next was to make some changes to all of the web pages, and I added several new web pages and links for the spider to pickup.

The webcrawler has not come back and re-indexed any of the pages it has already seen. I don't know if that will be days or weeks. I'll post here and tell you.

Good Advice on SEM and SEO

2011-03-20T01:24:00.000-07:00

Organic search is important, and will really pay off in the long run. But for instant results, many people turn to SEM programs based on PPC. (That's Search Engine Marketing based on Pay Per Click).

Basically you set an ad budget and let something like Google AdWords spend your money driving customers to your website. For a few pennies a day you can realistically start getting customers immediately. Even if you pay more, targeted online PPC advertising still gives you a better ROI than traditional advertising.

When you pay per click, you only absorb a cost when an interested person clicks on the ad. You can further refine this by only showing ads in selected geographic locations.

Facebook Profile Changes - "like" Spam

2011-02-24T07:49:00.000-08:00

Facebook recently changed the profile. Used to be you would expose portions of yourself by expressing your interests in text. Now you do the same by linking to pages of that name. I was reluctant to do this and lost all my profile settigns as a result.

Recently I put it back in, linking tot he pages as required. Now I get "like" spam many times a day form the commercial operators of these pages that everyone on Facebook "likes". On the one hand I'm jealous I don't own these pages myself, on the other hand I'm just annoyed by it.

So I created a community page expressing my disgust. Let's see if it goes viral.

Google $75 free trial for AdWords

2011-02-18T15:43:00.000-08:00

Google AdWords is giving away a $75 free trial coupon. You can apply for the offer here.

It's a one per person offer for new customer only. I honestly had not used AdWords for years, but wanted to try it again. It has given me good results in the past. I don't remember my AdWords account email address, so I just applied under my Blogger account email.

It took a couple of hours, but Google emailed me back an activation code. When I applied it I got $80 - the $75 plus the $5 AdWords activation fee. I'm happy. I have a campaign going now.

My activation code had an expiration date of the end of the month, about two weeks. I don't know if the credit expires then as well - my free facebook advertising did that.

I think this is a good way to introduce new people to Google Advertising with Adwords.

Strange Refferer URLs

2011-02-10T09:55:00.000-08:00

I had a website die on me. Oh, the domain name was fine. I just had the server crash without a backup. So, there has been no content for two or more years. But the web site still gets thousands of "hits" each month. Even after being down for years.

I brought my site back up recently and took a look at my web logs and found hundreds of pages being access, with 404 errors, every day. All of them were links referred from another site. Many of the site names made it obvious they were trying to sell something. This is what is called referrer spam.

It seems that many years ago web statistic programs like Webalizer created pages saying who your top referring sites were, and made these lists active html links back to the referring site. Someone then wrote and sold a program that allows you to fraudulently "drop links" into the logs giving you a backlink to your site.

Of course once this behavior was recognized, the Webalizer software to create the web stats was modified to prevent this - they just took out the link and made it plain text. But years later of the links not being on site - even years later of my site being down, the referrer spam people are still plugging away at my site.

It does nothing for them, but it destroys any chance I have of having meaningful statistics from my stats program. Thousands of page views and literally zero actual real people looking at a page.

Get Instant Approval for Adsense for Domains

2011-02-09T09:48:00.000-08:00

Google is now providing domain parking page hosting. They put sets of ad links on the page to help monetize your domain name.

This has been around for some time, but I only tried Adsense for Domains this week. It looks like everyone is automatically approved, though you have to go through an approval process that looks to me like is just Google verifying you have your DNS entries setup properly.

My first domain I tried took some time to get approved. My second domain was immediately approved. I edited my DNS settings before I submitted the domain name to Google. The Google automated verification process was able to verify me on the first try.

You can see an example of Adsense Parking Page.

There are several sets of ad blocks on the parking page. The domain name is at the top left. There is a search box under that, and also at the bottom of the page. (I'm assuming this is monetized through Adsense for Search.) There is a list of ten normal text links Adsense ads on the left. There is a graphic banner ad under this (we are at the bottom of the page now) and a 5 x 4 "popular categories" ad block under that. The top right also has a list of related topics.

I don't know if it will make money or not. The domain I purchased cost me a little over $2 for the first year. I don't have to make much money. But if I only get a nickle a click, that is about four clicks per month to break even - assuming my labor is free.

Google Penalty on Blog Links

2007-11-15T02:23:00.000-08:00

I've been looking at services such as PayPerPost. This company brokers between advertisers and bloggers who want to sell text links by blogging about the company's product.

The problem is that Google has decided to penalize blogs that sell textlinks unless they use the nofollow attribute or use javascript to mask the link to search engines.

Google's take on PR and paid text links can be found at http://www.google.com/support/webmasters/bin/answer.py?answer=66736

Getting a "Reaction"

2007-09-18T11:05:00.000-07:00

Chris Catto of http://blog.chriscatto.com/ was asking about how to get a "reaction" on Technorati.

A reaction on Technorati is a blog giving a hyperlink to another blog. To some, this is known as link love, or even a link exchange. To get a reaction, just get someone to link to your blog, or any article on your blog.

Reactions are only reactions if they come from Technorati indexed blogs. If the link is less than 180 days old, then this also adds to the blogs "Authority". A linking blog may provide as many Reactions as it wishes, but only the last Reaction counts as an Authority vote.

Once you have a reaction to your blog, then the next time you post an entry to your blog and Technorati gets pinged and indexes you, then the reaction is converted into Authority.

90 Days in One Day

2007-09-18T09:24:00.000-07:00

For some blog programs to make money, such as PayPerPost, there is a requirement to have your blog be 90 days old. Why wait? Age your blog in a day.

The PayPerPost Blog Requirements state:

Your blog must be at least 90 days old, verifiable by a third party index of the site.
If over 90 days old, your blog must have at least 20 entries in the past 90 days. If just around 90 days old, your blog must have 20 pre-existing entries.
No blogs with gaps over 30 days (no entries posted for 30+ days) will be accepted.

So to get around this just create a bunch of new blog entries and change the timestamps on your blog entries to spread them out over a three or four month period. (Hint: put more of your entries in the current month so they will still count in a few days for now. Your first attempt to get your blog accepted is likely not going to work. It may take you a couple of weeks to get everything right.)

Once you have done this you can then submit your blog to Google and get it indexed. Also submit your blog to the main Google index.

This probably won't really work for you. So what you need to do is get some "link love". Get several people to link to your site in exchange for you linking to them. While submitting your site will get Google to crawl your pages, the link love is what makes Google decide to put you in the public index so people see your site.

Freshness Variances on Technorati

2007-09-17T09:45:00.000-07:00

The term "freshness" as used on Technorati refers to when the last post was made on a blog.

For one blog I have the "everything known" page says the blog was last updated 1 day ago. Yet my favorites page indicates the blog was last updated 13 days ago. A freshness variance of 12 days. This same blog also seems to have questionable content and a few days ago Google ads stopped showing on it. So I do not know if this is a precursor to flagging, or if this is some strange hiccup in the Technorati data warehousing updates.

Technorati on again, off again flagging

2007-09-14T10:54:00.001-07:00

Technorati has a flagging policy I can't understand.

I created a blog sometime back called Poker Room Notes. My first post was in March of 2006, about 18 months ago. I'm a little embarrassed. The original purpose of the blog was to show how annoying online poker software was. So I would spend hours creating messages complete with custom graphics that detailed annoyances in the software. I develop software for a living and think that in general poker software has a lot of bad design issues. Initially, there was no advertising on the site. And later I expanded the site to talk about Atlantic City poker rooms.

Well, the Technorati spiders saw "poker" so they flagged the site for "spam." This sounds good now that I hear about it, but I don't email anyone about the site, so it is not a spam site. So what's a flagged person to do? Technorati will manually remove the flag when you ask them to. I did, and the site came into the index.

Well, it was indexed for five days or less, then reflagged. I've sent in a ticket to support and am waiting on a reply again. This re-flagging makes no sense. I'm waiting to hear the rationale behind it all.

Flagged for Review by Technorati

2007-09-06T10:42:00.000-07:00

Technorati will flag some blogs for review and not index them at all. In one case, the blog was about experiences in poker rooms. The blog was flagged as possible spam. Ironically, at the time there was no advertising of any sort on the blog. Just description of software problems with online poker, and talk of experiences in Atlantic City poker rooms.

The following email came from Technorati Support

I've taken a look at the issue regarding picking up your pings for
http://redwhiteandbluepoker.blogspot.com/. After making a small adjustment, I've sent our spiders to revisit your page and your blog should be indexed successfully from now on.

Please let us know if you experience any problems in the future. I apologize for any inconvenience. I'm afraid that your blogs were inadvertently flagged for review due to your blog subject. Unfortunately, poker is a subject that at high risk for spam. Do not hesitate to contact us if you have any other questions. Thank you for using Technorati!

Best Regards,
Janice Myint
Customer
Support Specialist
Technorati

Some Link Love

2007-09-05T10:48:00.001-07:00

Link Love is when you provide on your site a link to another site.

To help increase my Technorati Authority, I am linking to blogs of strangers. I hope to have some of them link back to me.

Authority works by Technorati receiving automatic (or manual) pings when a person updates the content to their blog. Technorati spiders the site looking for new posts (well, actually I think it checks the RSS feed) and extracts links to any page of other blogs. An Authority vote only counts if a link exists or not, not to which page or how many links. Authority also checks the "freshness" of the last link and only counts for 180 days.

In return for more Authority your blog becomes higher in the Technorati search engine.

Here are the blogs I have picked to link to.

http://www.mustardandcatsup.blogspot.com/

Everything and Anything ~ Abc's of fine design ~ Living well with diabetes ~ Mixing It Up Cash Buddy Cafe

My Single Mom Life//Twisted Psycho

My Thoughts, Ideas, and Ramblings Lisa Reviews

The Reality of a Brat - The Reality of a Brat

Grab a drink and head on over to The Den of Iniquity! Raspberry Patch Soapworks...reviews and recommendations on all things girly!

Fickle Food Shopping Blog Tiny Dog Blog All About Dogs Blog Mad Doll Doctor Doll & Teddy Repair Blog

magnusgrafex blogs:
http://www.secondlifeupdate.com/
http://www.lifediversity.com/
http://www.familyhelptree.com/blog
http://www.italypurses.com/blog

Technorati Profile

Authority vs. Page Rank

2007-09-04T22:54:00.000-07:00

Technorati's "Authority" number is similar in concept to Alexa's ranking and Google's Page Rank (Google PR). Alexa's is the easiest to manipulate, PR is the hardest to manipulate. Alexa's ranking is semi-worthless as a true ranking tool, though it is the standard many companies use to rate blogs. Google PR is the gold standard, but it has it's faults.

On the other hand, while not used by many for rating blogs, Technorati Authority is a potentially better way to rank blogs and is not as easy to manipulate as Alexa.

Page rank is a log scale number from 0 to 10 that tells how popular an individual web page is based on how many web pages link to it. Authority is a similar popularity contest but is a gauge for the entire blog site, not just a single web page. If nothing changes, PageRank lasts forever while Authority is only good for links made within the last 180 days. Also with Authority, only one link (vote) from each blog counts towards the receiving blog. This is unlike Page Rank, which allows many or even every page link to "vote" for a web page.

Authority is only for blogs, while PR is available for any web page. Authority is a non scaled number that is the count of the number of blogs that link to your blog. This must be a hard link in either an article or in the sidebar. A link in a javascript widget does not count. This link only counts for 180 days then the "voting" blog must post a new link or you lose that Authority point.

So a blog with an authority of "2" would have two blogs that link to it. A blog with an Authority of "500" would have 500 blogs linking to it.

Technorati uses Authority to determine search results. So even if it is not a gauge normally used to rate your blog by services such as PayPerPost, with a higher Authority you will get more traffic to your blog. Oh, and the links help increase your Page Rank as well.

Resources:
Technorati's Authority Checker widget (near the bottom of the page)
Technorati Manual Pinger

How much is your blog worth?

2007-09-04T12:48:00.000-07:00

There is a Technorati based widget for blogs that tells how much the blog is worth. This uses the Technorati API to test the blog for "goodness" then assign a monetary value to the blog. While a nifty widget, it is not what it appears on the surface.

First of all the value that is calculated has nothing at all to do with any actual value your blog has. The widget was developed by Dane Carlson and is based on the work of Tristan Louis's research into the numbers behind AOL's purchase of Weblogs, Inc.

Basically the original idea was that AOL paid maybe $25 million for the Weblogs company. Tristan Louis then threw away all assets of the company except for the blogs. The blogs were weighted based on Technorati ratings. Then the $25 million was divvied up among the blogs to give them a "worth".

Dane Carlson has done the same thing, upon request of a blogger asking for widget code, and divvied up the same money among all blogs in Technorati. So your blog "worth" includes the value of furniture and equipment, advertising contracts, etc that were held by Weblogs at the time of the AOL purchase.

The second thing that most people do not realize is that the widget does not calculate your "worth," but instead displays a static number embedded in the widget. So as your actual authority changes in value, the widget still displays it's initial worth.

Still, it is nifty if you like meaningless nifty things because everyone else is doing it. Here is my worth widget:

My blog is worth $3,456,387.24.
How much is your blog worth?

Get yours at http://www.business-opportunities.biz/projects/how-much-is-your-blog-worth/

Technorati Authority

2007-09-03T19:02:00.001-07:00

I've been trying to understand more about the authority system at Technorati. http://techorati.com/

With no authority, a blog has a ranking of 7,966,799th most popular blog in the universe. To me this does not suggest there are 7,966,799 blogs being indexed, but rather that there are 7,966,799 -1 = 7,966,798 blogs with an authority of 1. I guess it could go either way, but that is how I am guessing they designed it. This only means there are more than 8 million blogs, I just don't know how many more.

Some sample numbers at the low end include:

Authority	Rank
-	7,966,799
1	3,915,745
1	2,554,958
4	1,372,336
7	876,360

These numbers are interesting and show how many blogs have the same authority. And yes, there are two different numbers for an authority of one. I'm just recording what I see. I'm not sure of the reason for this yet.

First Post

2007-09-03T12:32:00.000-07:00

Even though many people may find this blog to be interesting reading, this blog embarasses me to own it. There are things I wish to do in the blogsphere but I just don't want them on my blogs, so I put them here instead.

I don't know what will show up here, I just know it is the stuff I don't want elsewhere.

Enjoy!