How to Search Engines Work

Unlike humans, search engines are text-driven. Search engines crawl the Web, looking at particular site items (mainly text) to get an idea what a site is about. Search engines perform several activities

First, search engines crawl the Web to see what is there. This task is performed by e piece of software, called a crawler or a spider (or Googlebot, as is the case with Google). Spiders follow links from one page to another and index everything they find. It is impossible for a spider to visit a site daily just to see if a new page has appeared or if an existing page has been modified. Sometimes crawlers will not visit your site for a month or two, so during this time your SEO efforts will not be rewarded. But there is nothing you can do about it, so just keep quiet.Crawlers do not see images, Flash movies, JavaScript, frames, password-protected pages and directories. These will not be spidered, not indexed, not processed, etc. –  they will be non-existent for search engines.

After a page is crawled, the next step is to index its content. The indexed page is stored in a giant database. Essentially, the process of indexing is identifying the words and expressions that best describe the page and assigning the page to particular keywords. Sometimes they might not get the meaning of a page right but if you help them by optimizing it, it will be easier for them to classify your pages correctly and for you – to get higher rankings.When a search request comes, the search engine processes it – i.e. it compares the search string in the search request with the indexed pages in the database. Since it is likely that more than one page (practically it is millions of pages) contains the search string, the search engine starts calculating the relevancy of each of the pages in its index to the search string.

There are various algorithms to calculate relevancy. Each of these algorithms has different relative weights for common factors like keyword density, links, or metatags. That is why different search engines give different search results pages for the same search string.
Major search engines periodically change algorithms. If you want to keep at the top, adapt your pages to the latest changes. This is one reason (the other is your competitors) to devote permanent efforts to SEO. .

The last step in search engines’ activity is retrieving the results. Basically, it is nothing more than simply displaying them in the browser – i.e. the endless pages of search results that are sorted from the most relevant to the least relevant sites.