Many people know that there is no better SEO friendly blogging platform than Wordpress but No matter how good a new Wordpress blog is, it won't likely get indexed if the sitemap is restricted by robots.txt! I meandered by Google Webmaster Tools to check on a client's indexing progress, then decided to check the status of the sitemap where I saw a dreaded red X! and the message "restricted by robots.txt" meaning Google wasn't allowed to look at my sitemap, and perhaps not the entire website. Robots.txt is a file that asks that some content such as contact forms or other pages intended to serve as modules on main pages not be indexed by Google. It is also a choice in Wordpress to block the entire site. It's important to use this feature when building a new site so that Google doesn't come along before you're ready and index "Hello World!" as the primary keywords of your site.
I'm not saying when or who but I once knew of a website that got indexed during the build phase and for a month, the phone number was displayed in Google search results as 000-000-0000! Not good, so keeping Google at bay is a good option under certain circumstances.
On the other hand, it doesn't pay to be asleep at the switch like I was, waiting for Google to finally do it's thing while I hadn't done mine.
For those who use Wordpress.
In the Dashboard:
1. Choose Privace under the Settings menu tab.

2. Choose your option. This sets up robots.txt to open or close the gateway to search engines.

ActiveRain Corp. is not responsible for the accuracy of the site's content (which is written by members of the ActiveRain Real Estate Network) and does not endorse the views of the real estate agents, mortgage brokers, and others listed here.
Powered by the ActiveRain Real Estate Network
© 2009 ActiveRain Corp. All Rights Reserved