Three stages, in order
Google describes its search engine as working in three stages: crawling, indexing and serving search results. Bing describes a very similar process. The order matters, because a page can only move to the next stage if it passed the one before. A page that is never crawled cannot be indexed, and a page that is not in the index cannot appear as a result, however good it is.
Google is also explicit that it does not guarantee any of these stages for any page. Following its guidance makes a page eligible; it does not make inclusion certain. Keep that in mind whenever someone describes search as a machine with predictable outputs.
Crawling: finding pages
Search engines use automated programs, usually called crawlers or bots, to fetch pages. Google's is called Googlebot. There is no central list of every page on the web, so crawlers discover new addresses mainly in two ways:
- Links from pages they already know. When a crawler fetches a page, it notes the links on it and may visit them later. This is why a new page with no links pointing to it can stay undiscovered.
- Lists that site owners provide. A sitemap is a file that lists the pages you consider important. Google's documentation notes that small, well-linked sites may not need one, but it rarely hurts.
Crawlers also need to be allowed in. A file called robots.txt at the root of a site tells crawlers which paths they may fetch. Google's documentation is careful to say that robots.txt manages crawler traffic; it is not a reliable way to keep a page out of search results. A blocked page can still be listed, without a description, if other sites link to it.
Fetching a page is not the end of crawling. Many pages build their content with JavaScript, so Google renders the page, much as a browser would, to see what a visitor sees. If the important text only appears after an interaction, or depends on files the crawler is not allowed to fetch, the search engine may never see it.
Worked example
A small bakery publishes a new page for a seasonal menu but forgets to link to it from the home page or the menu page. It is not in the sitemap either. From a crawler's point of view, the page does not exist yet. Adding one ordinary link from the menu page is usually the simplest fix.
Indexing: understanding pages
After a page is fetched, the search engine analyses it: the text, the title, headings, images and their alternative text, and other signals about what the page is about and what language it is in. The results are stored in the index, a very large database of pages.
Part of indexing is dealing with duplicates. Many sites have the same content at several addresses, for example with and without tracking parameters, or a printer version. Google groups similar pages and picks one as the canonical version, the one most likely to be shown. You can suggest which address you prefer with a canonical link element, but Google treats that as a strong hint rather than a command.
Not everything that is crawled is indexed. Google's documentation lists common reasons a page is left out, including low-quality content, a noindex rule set by the site, or a site design that makes indexing difficult. A page that adds nothing beyond what other pages already say is a weak candidate.
Serving: choosing results
When someone searches, the engine looks for indexed pages that match and returns the ones it judges most relevant and useful. The ordering depends on many factors, and some of them belong to the searcher rather than the page: their language, location and device all affect which results appear. Two people typing the same words can see different results, which is one reason a single "rank check" from one computer tells you little.
Paid ads appear on the same results pages, but they are bought through a separate advertising system and are labelled as sponsored. Paying for ads does not change how the unpaid results are chosen. That separation is covered in more detail in the next guide.
Four common misreadings
- "Submitting a sitemap gets my pages indexed." A sitemap helps discovery. It does not oblige a search engine to crawl or index anything.
- "I blocked it in robots.txt, so it is hidden." Blocking crawling is not the same as blocking indexing. To keep a page out of results, the usual tool is a
noindexrule, and the crawler must be able to fetch the page to see it. - "Indexed means ranked." Being in the index only makes a page eligible to be shown. Where it appears, if at all, is decided query by query.
- "Search engines read pages exactly as I do." They come close, but text inside images, content behind logins and content loaded only after clicks are all harder or impossible for a crawler to use.
A short checklist
For any page you care about, you can check these points without any special software:
- Is there at least one ordinary link to it from another page on your site?
- Does robots.txt allow crawlers to fetch it, and the files it needs to render?
- Is there no
noindexrule on it by mistake? - Is its main content present as text on the page, not only inside images?
- If the same content exists at more than one address, does each copy point to one preferred address?
If you have verified your site in Google Search Console, its URL Inspection tool shows how Google last crawled and indexed a specific address. The measuring guide explains Search Console in more detail.