At SMX East, I am putting together a brand new session on PPC & SEO: Can’t We All Just Get Along?, where Todd Friesen, Tim Mayer, and myself will look at how these two programs can be complimentary to each other and how to make them both work for you to increase your overall exposure. If you want to learn more about AdWords, I will be teaching an Advanced AdWords Course at the beginning of the conference. … [Read more...] about What PPC Practitioners Should Know About Robots.txt Files
Urls blocked by robots txt
Urgent Removal: If you’re a site owner, a system to get pages out of an index in a guaranteed period of time would be very convenient. However, this is probably better handled through webmaster tools that the search engines offer, as they allow a site owner to proactively trigger a removal, rather than waiting for visit from a crawler, which could take days. Ironically, at Google, they had a system to remove pages quickly. I wrote about it two years ago (see Google Releases Improved Content Removal Tools). But the documentation today is terrible. Little is explained if you’re not logged in. If you are logged in, the link for the webmaster version doesn’t work. The entire feature Google described in 2007 is gone. … [Read more...] about ACAP Versus Robots.txt For Controlling Search Engines
User-Agent: the robot the following rule applies to (e.g. “Googlebot,” etc.) Disallow: the pages you want to block the bots from accessing (as many disallow lines as needed) Noindex: the pages you want a search engine to block AND not index (or de-index if previously indexed). Unofficially supported by Google; unsupported by Yahoo and Live Search. Each User-Agent/Disallow group should be separated by a blank line; however no blank lines should exist within a group (between the User-agent line and the last Disallow). The hash symbol (#) may be used for comments within a robots.txt file, where everything after # on that line will be ignored. May be used either for whole lines or end of lines. Directories and filenames are case-sensitive: “private”, “Private”, and “PRIVATE” are all uniquely different to search engines. … [Read more...] about A Deeper Look At Robots.txt
to go at the root level of a web site. If you don’t put them there, then you … [Read more...] about Google Offers Robots.txt Generator
"Preserve," with similar time limits available for "index," would stipulate whether a copy may be stored in a search engine’s cache. … [Read more...] about ACAP Launches, Robots.txt 2.0 For Blocking Search Engines?