The search engines have the indexing of thousands of millions of web pages as their main purpose. When you enter a word or a phrase in the search form, the search engine scans their entire database. It is in this database where the search engine has the indexed content of all these web pages. They also index other files and documents. The search engine then returns to you a result list. This list is ordered by the most relevant results for that specific search.
The criteria in determining the number of pages returned is the relevance of the content contained on them. Determining this relevance depends upon the factors and formulas (algorithms) used by the search engine in originally indexing the data found on the web pages.
The Search Engines Of The Past
The first search engines appeared somewhere in the early 90’s when Alan Emtage, a student at the McGill University in Montreal, Canada, created the first search-engine-like tool. It was called Archie. Its purpose was to search through the information available on the FTP servers. The files on these servers were available for anyone, but no one could find them unless they knew the exact Internet address (URL/URI) of the server, including the name of the file. Archie looked through this database and gathered lists of files for each server. It was used by people to match phrases and characters in order to take them to the server on which the file they were looking for resided.
Archie is now a generally abandoned method, but its creation was the first step in the search engine rally that was to follow. As the public grew more and more aware of the existence of the Internet, the need for powerful and effective search tools became obvious. A few companies and organizations stepped up to the challenge. A lot of detailed information about the history of search engines and their current status is contained in this article on Wikipedia, the online encyclopedia.
At first there were some software processes that became known as robots, or just bots. Using a concept that came to be called spidering from the notion of searching, or crawling, the World Wide Web, to index the content found. By following links from one website to another, and saving the text from all visited websites in a massive database, web data indexing was achieved.
Between 1994 and 1995 three important search engines appeared: Lycos, WebCrawler and AltaVista. At about the same time Yahoo! first appeared but it was not a search engine yet. Yes, it has a search engine function now, but Yahoo! was firstly a directory for data and articles, as well as providing different services as email and hosting. Recently Yahoo! has partnered with another search engines in order to more effectively provide search results.
The Search Engines Today
Today the search engines are in a continuous competition. Search is not just a service provided to those seeking information. The search habits of those looking for information, including the information they seek, tells an enormous amount about the searchers demographics, including interests, shopping and buying preferences, and more. There are thousands of search engines, though just a few big ones, all seeking to profit from the information they gather on those doing the searching. The top four search engines globally (Google, Baidu, Yahoo!, and Bing) are responsible for just less than 98% of all online search.
The search engines provide an amazing free service that can be used by everyone. What keeps them financially alive to do this is paid advertising and the information gleaned (and sold) from the analysis of search traffic using their services. The more people that use their services, the more they can make providing paid advertising and other promotion space, additional premium search related services, and the more analytical data they can collect and sell.
The search engine companies are competing to develop the best formulas and algorithms to evaluate the web pages accordingly to the keywords provided. If someone is looking for a top position in search engines, then he has to be sure that his site is optimized in such a way that search engines would find it easily, deem it relevant for the keywords and phrases likely searched for that content, and that it represents authority on the subject contained, among other factors. It is the search engine that sets the standards for search engine optimization.
If you are interested in the early days of the Internet, and if you can find a copy, Ed Krol’s The Whole Internet User’s Guide & Catalog (Nutshell Handbook)is a great read. The newer The Whole Internet: The Next Generation (Whole Internet User’s Guide and Catalog)by Kiersten Conner-Sax is much newer and more relevant to today.