There’s No Stopping Bad Behavior A problem you will have with both robots.txt and the robots tag is that these instructions cannot enforce their directives. While Google and Bing will certainly respect your instructions, someone using Screaming Frog, Xenu, or their own custom site crawler can simply ignore disallow and noindex directives. … [Read more...] about Have You Considered Privacy Issues When Using Robots.txt & The Robots Meta Tag?
Robots txt sitemap xml
Certainly the search engines need to get their act together more, however. It’s time to stop referring people to the REP site which is run by no one. It’s time to stop having a myriad of help pages scattered about within their respective sites. Yes, they should continue to have their own help pages (see Google’s webmaster help from here; Bing’s from here). But I’d like to see Google and Microsoft take the lead to also consolidate material into a common site, perhaps building off Sitemaps.org. … [Read more...] about ACAP Versus Robots.txt For Controlling Search Engines
Stephan Spencer is the creator of the 3-day immersive SEO seminar Traffic Control; an author of the O’Reilly books The Art of SEO, Google Power Search, and Social eCommerce; founder of the SEO agency Netconcepts (acquired in 2010); inventor of the SEO proxy technology GravityStream; and the host of two podcast shows The Optimized Geek and Marketing Speak. … [Read more...] about A Deeper Look At Robots.txt
One of the announcements that occurred during the week of SES was Ask.com joining Google, MSN and Yahoo in supporting the Sitemaps auto discovery. This feature allows webmasters to specify the location of their sitemaps within their robots.txt file. Keith Hogan of Ask.com mentioned this change in his presentation and its impact. This will eliminate the need to submit sitemaps to each engine separately. Essentially, sitemaps are a simple XML file that lists URLS and information about the URLS to help spiders do a better job of crawling a site. See www.sitemaps.org for more details. … [Read more...] about Up Close & Personal With Robots.txt