Tim Laughlin's Everything VB.NET Blog


Time to update your robots.txt file to include sitemap.

All the major search engines have announce that they are now supporting Sitemap Autodiscovery.  Which boils down to an additional line you will want to add to your robots.txt file. 

For Community Server 2.1 with the Google Sitemap Module installed. All I had to do was add this line to the end of my robots.txt file.

Sitemap: http://tlaughlin.pandorasystems.com/GoogleSiteMapIndex.aspx

You need to use a fully qualified domain name, unlike the Allow and Disallow lines in a Robots.txt file.

I add this today.  Time will tell.  Most of my traffic comes from Google. Maybe now I will see a little more from MSN, Yahoo and Ask.  Who have all adopted this standard.

Posted: Apr 13 2007, 09:06 AM by tlaughlin | with 3 comment(s)
Filed under: ,
Watch the latest videos on YouTube.com

Comments

Dave Burke said:

blog bits Quite a bit of feedback on CommunityServer.org's new look , mostly for the good. Other than

# April 14, 2007 12:46 PM

Dave Burke's Community Server Bits said:

I've not heard about the need or usefulness of adding a fully qualified url of your Google sitemap to

# April 14, 2007 1:02 PM

tlaughlin said:

Dave,

The reason for fully qualified domain, to the best of my knowledge, is help prevent the same URL from being indexed under multiple domain names.  There is some fancy SEO term for this, but it escapes me at the moment.  

Good to hear from you again, by the way.  

Thanks for reading.  I promise I will be back to the Community Server world soon.  Business has been crazy busy.  

# April 14, 2007 2:13 PM