Google's John Mueller said on Twitter that having shared robots.txt across multiple domains is fine and should work for search. John wrote "It sounds like you have a shared robots.txt file across domains? That shouldn't be a problem, we might show those cross-domain URLs as errors in Search Console, but if they're on all domains, that should work regardless." … [Read more...] about Google: Shared Robots.txt Across Domains Work Okay
Example robots txt
Stephan Spencer is the creator of the 3-day immersive SEO seminar Traffic Control; an author of the O’Reilly books The Art of SEO, Google Power Search, and Social eCommerce; founder of the SEO agency Netconcepts (acquired in 2010); inventor of the SEO proxy technology GravityStream; and the host of two podcast shows The Optimized Geek and Marketing Speak. … [Read more...] about A Deeper Look At Robots.txt
Follow or nofollow instructs Web crawlers whether or not to follow the links on a page. It is like adding an rel=”nofollow” tag to every link on a page. Nofollow evaporates PageRank, the raw search engine ranking authority passed from page to age via links. Even if you noindex a page, it is probably a bad idea to nofollow it. Let PageRank flow through to its final conclusion. Otherwise, you could be pouring perfectly good link juice down the drain. … [Read more...] about Have You Considered Privacy Issues When Using Robots.txt & The Robots Meta Tag?
Urgent Removal: If you’re a site owner, a system to get pages out of an index in a guaranteed period of time would be very convenient. However, this is probably better handled through webmaster tools that the search engines offer, as they allow a site owner to proactively trigger a removal, rather than waiting for visit from a crawler, which could take days. Ironically, at Google, they had a system to remove pages quickly. I wrote about it two years ago (see Google Releases Improved Content Removal Tools). But the documentation today is terrible. Little is explained if you’re not logged in. If you are logged in, the link for the webmaster version doesn’t work. The entire feature Google described in 2007 is gone. … [Read more...] about ACAP Versus Robots.txt For Controlling Search Engines