So how does a search engine find the right answers and how does it deliver the most relevant information? There’s literally billions of pieces of content that it needs to scour through, so it is crucial for a website to be visible to the search engine, if not it will never show up in the SERP’s. If you don’t show up in a SERP, your website will most likely be overlooked or not seen at all.
Lets look at the three main functions of a search engine; crawl, index and rank. Crawling is pretty much what it sounds like and is often referenced to spiders. As a visual, if you can imagine digital spiders crawling around the internet searching for new and updated content, that would be close to what is actually happening. The spiders discover new content (a webpage, images, PDF’s, videos etc) by spinning its web between links. The main web becomes the database or index of good enough content that is delivered to the searcher. The more popular your website, the more it will get crawled.
The index is where a website will be analysed and stored. You do have the ability to tell a search engine how to index your site, or tell a crawler (spider) how you want your webpage to be treated or what pages you might not want indexed in search results. These instructions lie within the <head> of your HTML pages, via Robots Meta Tags.
Ranking is the order a search engine will place results in, according to the query. They use algorithms to determine the relevance of the search results that are presented. These algorithms are constantly changing and adapting to make sure the quality of the search results remains consistent. What you include in the content of your page will also contribute to how well it will be matched to the query and your ranking.