First there was the World Wide Web. Then there was the problem of making sense of it. Just having a website was great but how would anyone find it? Since before the internet all business had to be listed elsewhere, like in phone directories or other forms of advertising, it was natural to have all of that direct prospective customers to the website. But that was only marginally successful.
The Age of Yahoo
In 1996, Yahoo started a web directory and as long as a business was listed in the directory, at least there was the hope of having it be found when customers were looking for a product or service. Web masters (usually website owners) and content providers began optimizing sites for search directories, at first this merely meant you built a website, checked it for spelling errors and then submit it to search engines to be cataloged. This was the early World Wide Web.
Once a web master had submitted a page, or URL, to the various directories, they would send a spider to "crawl" that page, extract links to other pages from it, and return information found on the page to be indexed.
The process has evolved so that now a Search Spider downloads a target page and stores it on the Search Directory server, where a second program, known as an indexer, breaks down the information, extracts various elements that make up the page, such as the words it contains, the coding information, META tags, description, title, keywords, link text, and where these are located, as well as any weight or added significance for specific words and all links the page contains, which are then placed into a scheduler for crawling at a later date. The Indexer then gives each element a value according to the search engine algorithm to determine the overall relevance of the page in regard to certain search terms.
Site owners started to recognize the value of having their sites highly ranked and visible in search engine results. As more users began to search for goods and services online, it was possible to track how the search results were used. It was easy to see that searchers would not often go through many pages of results to find what they wanted.
Early versions of search algorithms relied on web master-provided information such as the keyword meta tag, or index files in. Meta-tags provided a guide to each page's content. But using meta data to index pages was found to be less than reliable because the web master's account of keywords in the meta tag were not always relevant to the site's actual keywords.
In short using the information provided by the web master’s to determine the search result was like using the fox to guard the hen house. Inaccurate, incomplete, and inconsistent or manipulated data in meta tags caused pages to rank for irrelevant searches. Irrelevant search results began to frustrate searchers. They began to look to other ways to search so as to locate more relevant answers to queries. The search engines that used an algorithm based on information that involved elements actually in the content on the page and how it related to the search.
The Age of Google
While they were graduate students at Stanford University, Larry Page and Sergey Brin developed "backrub", a search engine that relied on a mathematical algorithm to rate the prominence of web pages. The number calculated by the algorithm, PageRank, is a function of the quantity and strength of inbound links. This means that some links are stronger than others, or give more weight to the PageRank because the contents of the page is more likely to be relevant to the searcher.
In 1998 the grad students founded Google. Google attracted a loyal following among Internet users, who liked the results their system produced. Off-page factors such as PageRank and hyper link analysis were considered, as well as on-page factors, to enable Google to avoid the kind of manipulation seen in search engines that only considered on-page factors for their rankings. Although PageRank was more complex, web masters went to work using link building schemes to manipulate search results to their favor. Web masters focused on exchanging, buying, and selling links, often on a massive scale. Some of these schemes, or link farms, involved the creation of thousands of sites for the sole purpose of link spamming.
To counter the adverse impact of link schemes, as of 2007, search engines again had to evolve to consider a wider range of undisclosed factors for their ranking algorithms. Google has since disclosed that it now uses more than 200 different elements to rank pages. The three leading search engines, Google, Yahoo and Microsoft's Live Search, closely guard the algorithms they use to rank pages.
What Does The Future Hold?
As Search Engines continue to experiment with self serving additions to search enquiries, for example Google has begun to place additional lines of text below Their Sponsored Links (see also PPC). These extra results are items available through Google's own "check out" serve mainly to put relevant search results farther down the page. Self serving in that these advertisers pay Google for these positions.
Why is this a bad thing? Well it isn't really if you like competition and are tired of Google being the only "Big Dog" in the show. However if you are like Most people that use the internet on a regular basis to find products and information, having the relevant results to those searches pushed father and farther down from the top of the search results page is annoying.
In effect they are just leaving the door open for the next grad students to take us in a whole new direction. As an Search Engine Optimizer, we have to optimize not just for Google, but also for the other players taking their share of the market now (i.e. MSN, Yahoo) and to be on the look out for the next up coming search engine that will forego the self serving promotion of paid advertisers and concentrate on giving the quality search results users are looking for.
As time goes on, net surfers are only going to become more particular. If the major search engines continue to drift further from the reason they became popular, the net will continue to change. One thing is for sure as time goes on, search engines will change and they will continue to struggle to give only accurate results to searchers. And as competition increases for the top positions for search results on those search engines, the online business presence will need a full time Search Engine Optimization Specialist just to compete.