Google – Providing the Worst Results… on Purpose?

The landscape of the search engine market has changed a lot over the years. Originally it was Yahoo! with their hand tinkered directory, providing extremely relevant results, but often years behind the times due to the manual effort. Then keyword based sites like Alta Vista or Webcrawler came around, which basically just did a full text search of the web. While they were up to date quickly, they were also filled with spam and useless sites. And then along came Google. Like a shining beacon in the dark world of the web, it changed the way sites would be found from that point on. Instead of using hand tinkering which is time consuming or keywords which are not useful, it used a unique PageRank system that looked at how many other sites link to you, and weighted the value of that ranking. Combined with some other magic, suddenly search results became very relevant. After all, most sites talking about Internet Explorer probably provided a link to the download site right? So that site then became the most relevant result for that search. The web however, was a different place in 1998. Websites were less dynamic, no one knew of the PageRank type formulation, and so Google’s method worked great. During this period, more people joined the web then in all the previous years combined, and many of them were unaware of a world before Google, and thus Google became synonymous with search. Yahoo! was that site that had the stocks and movie times and stuff. And ask someone about AltaVista, they will think you are talking about a telescope, or that person that cuts their lawn.

Unfortunately 8 years has passed since then, and a lot has changed on the web. But Google’s search does not seem to have changed with it. Perhaps they don’t want to “mess with a good thing”…. or maybe making they don’t want to make their search better….because a better search makes them less money? People still swear by Google, but let’s take me as an example of the relevance of the various search engines.

SearTech, Inc. makes a very very niche piece of software, ChimSoft, software for chimney sweeps. There are literally no other companies that currently make software exclusively for Chimney Sweeps. There are maybe 3-5 others that TARGET chimney sweeps in addition to other industries. Even including them, it’s a no-brainer that ChimSoft or SearTech should show up within the first 10-20 results for a search on ‘Chimney sweep software’. Let’s do a quick comparison of where www.ChimSoft.com shows up after a little over a year on the web.

On the search ‘Chimney Sweep Software’ (with no quotes, as most people search):

*Yahoo!: #3, behind the Chimney Safety Institute of America and another software company that targets Chimney Sweeps

*A9 Powered by Microsoft Live: #1

*MSN (which may be the same as A9, but I don’t think so): #1

*Clusty/Vivisimo: #2, behind a chimney sweep’s company’s site.

*Google.com: No where in the first 50 results. With quotes around the search, it comes up 7th, behind some forums…which I will get to in a second.


So all of the other popular search engines have quickly picked up and identified us as the most relevant for that search, and some of our competitors show up as well. On Google however, the only site that shows up is one that is using some good SEO techniques. SEO is now the name of the game. In theory, a good website should stand on its own. I should be able to make a site about beer-flavored nipple rings (patent pending) and if I’m the only one making them, am thus the most popular, I should be one of the first results.

Unfortunately, because of the way Google works, if I really want it to show up first, I have to play the SEO game. Get linked in by 500 sites, use the keywords a bunch of times on my site, have a domain that includes the keywords etc. I am intimately familiar with SEO stuff because my long term ISV project, Used Cars On-Line (http://www.AUsedCar.com) shows up on the first page of Google for the extremely popular terms “used cars”. I should point out that I never ever used any SEO techniques to get the site to the position it’s in. However, if you look who is linking to it, there are tons of link farms, and domain campers that litter the Internet these days. The reason they link to me is between 1998-2001 AUsedCar.com was the #1 result on Google and Yahoo (we are one of the oldest used cars classified on the internet, around since 1995). Around 1999 people realized how Google was doing things, and added us to their link farms and requested a reciprocal link. Some never took down their links to us, even though we never linked back, but because of our high rating (which probably wasn’t deserved, I mean AutoTrader.com had 100 times more listings) we got “linked around” as it were. I was even interviewed by an SEO magazine wanting to know “my secrets” but I had no clue what an SEO was at the time.

Ok, so back to my original dramatic headline. Why do I think Google has bad results on purpose? Am I just some sour-puss who thinks his site is the bestest and should be #1? No, of course not. Google has some of the smartest programmers in the world working for them; surely if I’m not at the top of their search then I’m not as popular as the other results. Well let’s take a look at some of the other top results for our chimney sweep software search…. Let’s see, this very blog…a blog about this blog, and a bunch of chatting in a forum…about my keywords not showing up on Google. Yes, a thread where I brought this very point up, which was linked to by NO ONE to my knowledge (it wasn’t that good) shows up as one of the most relevant pages for “chimney sweep software”. Presumably this is because the domain (joelonsoftware.com) is a very trusted domain. The problem is that Joel can do a lot of things with his domain, so a simple forum spammer can SEO his/her site right up.

The Washington Post had an article this weekend about the owners of sites like BistBuy.com and CircruitCity.com or whatever that camp at those domains and have nothing but Google ads. You’ve seen them in your Google search, a simple search these days involves wading through tons of forum discussions, link farms and other useless pages, most of which generate revenue from Google AdWords. I think Google has cursed the Internet in two ways… AdWords and “The more links the better”. The Internet is now filled with sites that do nothing but make money from “click farms” that burn real advertisers money, make spammers money, and also make Google money. Why does my site return at the top of all the other search engines besides Google? What do they know that Google doesn’t? Could it be that Google makes 98% of its revenue from paid ads? If you don’t want to go down the “Back alley” world of SEO, the only way to get at the top of Google is to buy ads (which you can see we have done).

So why should Google fix it’s ads? Why should it stop click farms from doing their thing? The counter argument is usually “oh I’m sure they are trying hard, Alta Vista can tell you that it only takes some upstart company and everyone will switch to another search engine like THAT”. Unfortunately the problem with that argument is that so few people (relatively) were on the Internet when Alta Vista was around, that it was easier for that base to leave. Now people don’t even know there ARE other options. It’s just Google or nothing. Better search engines have came along. Clusty.com is the best search engine no one knows about. If someone were to combine Clusty.com with Yahoo’s beta of search that has a slide bar to sort results as more commercial or more research, one would have a search engine 1000 times more useful then Google. The problem is no one would still switch. Google’s base is far too entrenched now. Apple may have a better system then Microsoft and be switching users by the dozen, but that doesn’t mean my Aunt or Uncle have even heard of anything besides Windows. Once you get a certain mass of people somewhere, you are guaranteed a large amount of eyes for a long time (think Amazon and eBay). If Google were to fix their search, I wouldn’t buy paid ads. You wouldn’t buy paid ads. If Google were to shut down the click farms (don’t tell me they can’t hire one person to just find them, I can find 10 this minute and I’d do the job for free), they would lose tons of money too. Even worse, the ad system itself is gamed in a hundred ways by spammers, click farmers, and competitors. While Google itself may do no evil, the unfortunate nature of humanity has wrought all sorts of evil to the Internet because Google is not working quick enough to fight it. Perhaps Google wants to fix their search and their ad system. Perhaps that’s why they are acquiring all these other companies. Once the revenue stream is from somewhere else they can “bite the hand that feeds them” as it were. Let’s hope that happens sooner rather then later.