content=”index,follow” index HTML page, links follow content=”noindex,follow” do not index HTML page, links follow content=”index,nofollow” index HTML page, links donot follow content=”noindex,nofollow” do not index HTML page, links do not follow This tells the crawler whether it may take the HTML page into the index and whether it can follow the links in the HTML page. Links from “nofollow” HTML pages do not pass PageRank or other forms of link equity. The “nofollow” attribute can be specifically used to devalue links on an HTML page. … [Read more...] about SEO Basics – Indexing with / robots.txt, meta tags and canonicals –
Robots txt x robots tag
What robots.txt does not do is to keep files out of the search engine indexes. The only thing it does is instruct search engine spiders not to crawl pages. Keep in mind that discovery and crawling are separate. Discovery occurs as search engines find links in documents. When search engines discover pages, they may or may not add them to their indexes. … [Read more...] about Have You Considered Privacy Issues When Using Robots.txt & The Robots Meta Tag?
To entirely prevent a page from being added to a search engine’s index even if other sites link to it, use a “noindex” robots meta tag and ensure that the page is not disallowed in robots.txt. When spiders crawl the page, it will recognize the “noindex” meta tag and drop the URL from the index. … [Read more...] about A Deeper Look At Robots.txt
Google's John Mueller said on Twitter that having shared robots.txt across multiple domains is fine and should work for search. John wrote "It sounds like you have a shared robots.txt file across domains? That shouldn't be a problem, we might show those cross-domain URLs as errors in Search Console, but if they're on all domains, that should work regardless." … [Read more...] about Google: Shared Robots.txt Across Domains Work Okay
"However, if someone's linking to your robots.txt file and it would otherwise be indexed, we wouldn't be able to index its content & show it in search (for most sites, that's not interesting anyway)," he added. Meaning, Google might not show it in the Google index. Yes, Google does rank robotst.txt files if it has content people are searching for. … [Read more...] about Disallowing Robots.txt In Robots.txt Doesn’t Impact How Google Processes It