Link Fraudsters, Be Prepared to Experience Google’s “Death Penalty”

JCPenney is a big retailer, for sure, but not necessarily the first firm to come to mind when you think of most retail categories. So the New York Times suspected that something fishy was up when the retailer’s site came out tops for dozens of Google searches, including the phrases “skinny jeans,” “dresses,” “bedding,” “area rugs,” “home decor,” “comforter sets,” “furniture,” and “table cloths”. The phrase “Samsonite carry on luggage” even placed Penney ahead of Samsonite’s own site!

The Times reported that “someone paid to have thousands of links placed on hundreds of sites scattered around the Web, all of which lead directly to JCPenney.com.” And there was little question it was blatant link fraud. Phrases related to dresses and linking back to the retailer were coming from such nondress sites as nuclear.engineeringaddict.com, casino-focus.com, and bulgariapropertyportal.com. One SEO expert called the effort the most ambitious link farming attempt he’d ever seen.

Link fraud undercuts the credibility of Google’s core search product, so when the search giant discovers a firm engaged in link farming they drop the hammer. In this case Google both manually demoted Penney rankings and launched tweaks to its ranking algorithm. Within two hours JCPenney organic results plummeted, in some cases from first to seventy-first (the Times calls this the organic search equivalent of the “death penalty”). Getting a top spot in Google search results is a big deal. On average, 34 percent of clicks go to the top result, about twice the percentage that goes to number two. Google’s punishment was administered despite the fact that Penney was also a large online ad customer, at times paying Google some $2.5 million a month for ads.D. Segal, “The Dirty Little Secrets of Search,” New York Times, February 12, 2011.

Google is constantly playing defense against firms gaming organic search results. In another example a Brooklyn-based eyewear firm allegedly mistreated customers in order to get more ranking-influencing links (albeit from negative mentions) from service review sites. For a time these associated with bad ratings actually pushed the eyewear firm’s search results ahead of rivals, and since users typically focus on the top ranking, many customers went to the firm without seeing the bad reviews (Google has since changed search results to make it difficult to benefit from cultivating negative reviews). The owner of the busted eyewear retailer has also pled guilty to multiple counts, including sending threatening communications, one count of mail fraud, and one count of wire fraud.D. Segal, “Online Seller Who Bullied Customers Pleads Guilty,” New York Times, May 12, 2011.

JCPenney isn’t the first firm busted. When Google discovered so-called black hat SEO was being used to push BMW up in organic search rankings, Google made certain BMW sites virtually unfindable in its organic search results. JCPenney claims that they were the victim of rogue behavior by an SEO consultant (who was promptly fired) and that the retailer was otherwise unaware of the unethical behavior. But it is surprising that the retailer’s internal team didn’t see their unbelievably successful organic search results as a red flag that something was amiss, and this case highlights the types of things managers need to watch for in the digital age. Penney outsourced SEO, and the fraud uncovered in this story underscores the critical importance of vetting and regularly auditing the performance of partners throughout a firm’s supply chain.D. Segal, “The Dirty Little Secrets of Search,” New York Times, February 12, 2011.

Spiders and Bots and Crawlers—Oh My!

When performing a search via Google or another search engine, you’re not actually searching the Web. What really happens is that the major search engines make what amounts to a copy of the Web, storing and indexing the text of online documents on their own computers. Google’s index considers over one trillion URLs.A. Wright, “Exploring a ‘Deep Web’ That Google Can’t Grasp,” New York Times, February 23, 2009. Google starts to retrieve results as soon as you begin to type, and the upper right-hand corner of a Google query shows you just how fast a search can take place.

To create these massive indexes, search firms use software to crawl the Web and uncover as much information as they can find. This software is referred to by several different names—spiders, Web crawlers, software robotsSoftware that traverses available Web links in an attempt to perform a given task. Search engines use spiders to discover documents for indexing and retrieval.—but they all pretty much work the same way. The spiders ask each public computer network for a list of its public Web sites (for more on this see DNS in Chapter 12 "A Manager’s Guide to the Internet and Telecommunications"). Then the spiders go through this list (“crawling” a site), following every available link until all pages are uncovered.

Google will crawl frequently updated sites, like those run by news organizations, as often as several times an hour. Rarely updated, less popular sites might only be reindexed every few days. The method used to crawl the Web also means that if a Web site isn’t the first page on a public server, or isn’t linked to from another public page, then it’ll never be found.Most Web sites do have a link where you can submit a Web site for indexing, and doing so can help promote the discovery of your content. Also note that each search engine also offers a page where you can submit your Web site for indexing.

While search engines show you what they’ve found on their copy of the Web’s contents; clicking a search result will direct you to the actual Web site, not the copy. But sometimes you’ll click a result only to find that the Web site doesn’t match what the search engine found. This happens if a Web site was updated before your search engine had a chance to reindex the changes. In most cases you can still pull up the search engine’s copy of the page. Just click the “Cached” link below the result (the term cacheA temporary storage space used to speed computing tasks. refers to a temporary storage space used to speed computing tasks).

But what if you want the content on your Web site to remain off limits to search engine indexing and caching? Organizations have created a set of standards to stop the spider crawl, and all commercial search engines have agreed to respect these standards. One way is to put a line of HTML code invisibly embedded in a Web page that tells all software robots to stop indexing a page, stop following links on the page, or stop offering old page archives in a cache. Users don’t see this code, but commercial Web crawlers do. For those familiar with HTML code (the language used to describe a Web site), the command to stop Web crawlers from indexing a page, following links, and listing archives of cached pages looks like this:

〈META NAME=“ROBOTS” CONTENT=“NOINDEX, NOFOLLOW, NOARCHIVE”〉

There are other techniques to keep the spiders out, too. Web site administrators can add a special file (called robots.txt) that provides similar instructions on how indexing software should treat the Web site. And a lot of content lies inside the “dark WebInternet content that can’t be indexed by Google and other search engines.,” either behind corporate firewalls or inaccessible to those without a user account—think of private Facebook updates no one can see unless they’re your friend—all of that is out of Google’s reach.