There is a very shocking fact about SEO is that many search engine optimizers are never trained rather they pick up the things on their job. So there is no surprise that most of the SEOs are quite confused about the how search engines works.
This article explores more about the Search Engines and how they work. Hope you will enjoy it.
What is a Web search engine?
A web search engine is a system that is designed to search for information in the World Wide Web. In the other words search engines are special websites that are developed to help people to find the information in the World Wide Web. The result is represented through search engine result page (SERPs). The information/data (images, videos and other text documents & files) are processed in real-time through running algorithms (mathematical procedures that sort and rank the website) and web crawlers.
Mathematically, a search engine is a subset of search as search engine can’t process all the information present in the World Wide Web due to its size and frequently changing nature.
Google, Yahoo and Bing are the top three search engines of the current time. Though the terminology and algorithm varies from search engine to search engine but most of them work on a similar protocol.
How web search engines work?
Mostof the modern day search engines are crawler-based, and what exactly they do is crawl the web all the time to fetch information where the priority is to find the most important sites.
All the search engines work on three main major parts:
Spider – a web crawler that finds and fetches web pages.
Index- everything that spiders collect is passed to the indexer that shorts and stores everything collected
The last part of the process is search engine algorithm/query processor, which returns the most important & relevant recommendation for the user query
Let us take a closer look of the each segment.
SPIDER:
Spiders or web crawlers are the special software used to collect the information of the publicly available websites in the World Wide Web. The main work of the spider is to find and crawl as many web websites possible. But the web is too vast to find each single website ifself. Again the continual changing nature of the web makes it much more difficult to gather the exact information. Still the spiders find millions of pages to fire back to their respective query. Crawlers basically collect text, metadata, HTML components, alternative file formats and URL’s for analysis and further crawling.
Search engine spiders find out web pages in three ways:
Extracting data form crawl pages: By extracting links from already crawled pages spiders add them to the queue for subsequent crawling.
From previous crawl: Crawlers uses the list of URL’s obtained by a previous crawl to visit them again. This helps the search engines to keep their index update.
Human Inputs: Many often the webmaster directly feed the website to the crawler. All the current search engines offer special interface for webmasters to submit their new websites directly to the search engines which enables the easy access to the new websites.
The search engines need to keep their index up-to-date which requires frequent crawling of WebPages to gather changed made to existing site or new information available online on the similar topic. There is no time limit on the crawling rate of the web site. But tomputer programs determine which sites to crawl, how often, and how many pages to fetch from each site.
Again the re-crawling of the web-pages depends upon the popularity of the website and how often the pages change. News websites and stock websites are crawled more frequently for their frequently changing nature.
INDEX:
The indexer is like the central filing system of a library which maps all the data as per it relevancy/presence.
Once the spider collects information, it passes the text and source to indexer. Then Indexer stores the pages and all the other information collected to create an index of the information based on its relevancy and importance
Sometimes you may find WebPages crawled but not indexed as it takes a while the websites to be added into the index.
QUERY PROCESSOR:
Query processor is actually the software/algorithm of the search engine that process and ranks the websites that are indexed now. In the basic level, when someone searches for a particular term, the query processor searches the index to find the most relevant pages and return them.
Bottom line:
All crawler based search engines work as the above described process but they differ in the way they handle the collected information. That is why they differ in result for similar search terms.
