There is an interesting thread at Hacker News on DuckDuckGo being toyed by Google. That part honestly doesn't interest me as much as the core SEO topic in the thread.Google's Matt Cutts is very active on Hacker News and he questioned Gabriel Weinberg, the founder of DuckDuckGo, about the DuckDuckGo spiders and crawlers. There are some folks asking if DuckDuckGo's spider, aka DuckDuckBot, respectes the robots.txt directives. Some noticed DuckDuckGo crawling under the IP range they own but not declaring the useragent and thus not respecting the robots directives set by the webmaster.Matt Cutts asked Gabriel: Gabriel, does DuckDuckGo's crawler have a distinct user agent? Can you talk more about how DuckDuckGo observes/respects robots.txt?I emailed Gabriel and he explained that in this case, they are only checking for parked domains. He wrote, "what they're seeing there is not a crawler but a parked domain checker." He added, "it doesn't crawl through a site. In fact, it only checks … [Read more...] about DuckDuckGo Not Respecting Robots.txt Directives?
Duckduckgo robots txt
Here’s our daily recap of what happened in online marketing today, as reported on Marketing Land and other places across the web. From Marketing Land: Introducing Inbound.org: Community-Curated News For Marketers Late last week SEOmoz’s Rand Fishkin and Hubspot’s Dharmesh Shah collaborated to launch a “for fun” Hacker News-like site called Inbound.org. According to the site, “The goal of Inbound.org is to prevent great content from the world of inbound marketing from going unnoticed. There’s an amazing community of bloggers, marketers and enthusiasts passionate about non-paid channels like SEO, social […] Apple Cuts iAd Prices To Attract Brands, Compete With AdMob Apple is a company that rarely discounts its products. Not so with iAd, which is once again cutting prices to attract advertisers. According to AdAge, the new iAd minimum ad spend is being reduced from $500,000 to $100,000. It began at $1 million when the platform launched … [Read more...] about Marketing Day: February 15, 2012
After hours of coding, writing, designing and optimizing, finally, a new web page has gone live. But hey, why is it not appearing in Google Search? What have I done wrong? Why does Google hate me? Now, now, we have all been there before. And I have learned to give it at least a day or two before trying to seek out my new blog post on Google Search. Because I have long accepted that that’s how long Google needs to actually put my newborn, I mean my new blog post on Google Search. The process of getting a new web page or a new website on Google Search is a long and windy one. But it’s one worth learning about. Let’s start with the basics. What is Search? How does it works? Here’s a video from Google, where Matt Cutts tell you a little about how search works. Have you watched it? If yes, please bear with my little attempt at summarizing. For a content to appear on Google Search, it has to go through spiders. No, not real spiders, but a program called spider. The … [Read more...] about 5 Reasons Your Page Is Not Indexed On Google Search and How to Fix It
Here is a recap of what happened in the search forums today, through the eyes of the Search Engine Roundtable and other search forums on the web. Search Engine Roundtable Stories: Google Begins Penalizing Domain Leasing Google issued a warning about sites that lease out its own subdomains and subfolders so that other companies can rank their content better on leased domains. Well, now it seems Google is taking action by penalizing those sections of sites that have these leased out sections. Google Ads Tests Green "Advt." Label Google is possibly testing a new ad label, this one goes back from the black "Ad" text, to a green outlined "Advt." label. This was shared to me by JaiKishan Raturi on Twitter, here is the screen shot. Google Reacts To DuckDuckGo On Why Not To Use Google Search Danny Sullivan of Google responded to the claims and allegations from DuckDuckGo around Google's biased search results. Bing Webmaster Tools Lets You Verify Your Site Using Google Search Console This … [Read more...] about Daily Search Forum Recap: August 29, 2019