--- title: "How Search Engines Crawl and Index a Page" author: "Pradeep Kumar" published: "2026-11-21T06:04:53.020Z" updated: "2026-11-21T06:04:53.020Z" url: "https://zyvop.com/how-search-engines-crawl-and-index-a-page-1ss6z" canonical: "https://zyvop.com/how-search-engines-crawl-and-index-a-page-1ss6z" tags: ["seo","robots.txt","Technical SEO","Googlebot","Crawling and Indexing"] --- # How Search Engines Crawl and Index a Page Follow a URL from discovery to the search index, and learn what blocks pages at each step, from robots.txt and JavaScript to canonicals and noindex. ![Cover image]() Before a page can show up in search results, a search engine has to find it, fetch it, understand it, and store it. Google describes this as three stages: crawling, indexing, and serving. Not every page makes it through all three. This post covers the first two stages, plus the rendering step in between. Google is the example throughout, and other engines work in a broadly similar way. ```mermaid flowchart LR A["1. Discover
find the URL"] --> B["2. Crawl
fetch the page"] B --> C["3. Render
run JavaScript"] C --> D["4. Index
analyze and store"] D --> E["5. Serve
match queries and rank"] ``` > **No guarantees:** Google can't promise that any page will be crawled, indexed, or shown, and it doesn't accept payment to crawl a site more often or rank it higher. * * * ## 1\. Discovery: How a URL Gets Found A search engine can only crawl URLs it knows about. There are three main routes: | Source | How it works | | --- | --- | | **Links** | Crawlers follow links from pages they already know, on your site and on other sites | | **Sitemaps** | A file where you list the URLs you want crawled | | **Manual requests** | Search Console's URL Inspection tool lets you request indexing of a specific page | ```mermaid flowchart LR L["Links from other pages"] --> Q["Crawl queue
known URLs"] S["XML sitemaps"] --> Q M["Manual requests
URL Inspection tool"] --> Q Q --> G["Googlebot decides
what to crawl and when"] ``` **Tip:** Google can only discover links that are real `` elements with an `href` attribute. Click handlers on other elements, and URL fragments used to load different content, aren't reliable. A brand-new page or site can take days to weeks to be crawled and indexed. * * * ## 2\. Crawl Scheduling: What Gets Crawled, and How Often Finding a URL doesn't mean it's crawled right away. Googlebot uses an automated process to decide which sites to crawl, how often, and how many pages to fetch from each. The result is called your **crawl budget**: the set of URLs Google can and wants to crawl. ```mermaid flowchart TD A1["Fast, stable responses
limit goes up"] --> A["Crawl capacity limit
how much your server can handle"] A2["Slow responses, 5xx errors, 429
limit goes down"] --> A B1["Site size, update frequency,
quality, relevance"] --> B["Crawl demand
how much Google wants to crawl"] A --> C["Crawl budget
URLs Google can and wants to crawl"] B --> C ``` - **Capacity** protects your server. If it slows down or returns errors, Googlebot backs off. - **Demand** reflects how popular, fresh, and useful Google thinks your content is. A site move can temporarily raise it. **Do you need to worry about this?** Probably not. Google's own crawl budget guide targets sites with 1 million or more unique pages that change weekly, or 10,000 or more pages that change daily. For everyone else, an up-to-date sitemap and a regular look at the Page indexing report are enough. * * * ## 3\. Fetching: robots.txt, Status Codes, and Size Limits When Googlebot takes a URL from the queue, it first checks `robots.txt`. If crawling is allowed, it makes the HTTP request and acts on the response. ```mermaid flowchart TD A["URL taken from crawl queue"] --> B{"Allowed by robots.txt?"} B -- No --> X["Page not fetched
bare URL may still be indexed if linked elsewhere"] B -- Yes --> C["HTTP request"] C --> D{"Response status"} D -- "200" --> E["Queued for rendering"] D -- "301 / 302" --> F["Follow the redirect"] D -- "404 / 410" --> G["Page not indexed"] D -- "5xx / 429" --> H["Googlebot slows down
and retries later"] ``` A typical `robots.txt`: ```text User-agent: * Disallow: /cart/ Disallow: /internal-search/ Sitemap: https://example.com/sitemap.xml ``` ### robots.txt controls crawling, not indexing This is the most common mix-up. A URL blocked in `robots.txt` can still appear in search results, without a description, if other sites link to it. To keep a page out of results, use `noindex` and leave the page crawlable so Googlebot can see the tag. ```mermaid flowchart TD Q{"Goal: keep a page
out of search results"} --> A["Disallow in robots.txt"] Q --> B["noindex meta tag or header"] A --> A1["Googlebot never reads the page"] A1 --> A2["URL can still appear if others link to it
no description shown"] B --> B1["Page must stay crawlable
so the tag can be seen"] B1 --> B2["Page is dropped from the index"] ``` ```xml ``` For non-HTML files such as PDFs, send the same instruction as an HTTP header: `X-Robots-Tag: noindex`. | Tool | Stops crawling? | Stops indexing? | | --- | --- | --- | | `robots.txt` Disallow | Yes | Not reliably | | `noindex` tag or header | No (page must be crawlable) | Yes | | Password protection | Yes | Yes | **Avoid** combining `Disallow` and `noindex` on the same URL: the crawler is blocked, so it never sees the `noindex`. Also note that Google ignores `crawl-delay` and does not support `noindex` inside `robots.txt`. ### The 2 MB limit Googlebot currently fetches only the first 2 MB of an individual URL (64 MB for PDFs). Anything after the cutoff is not fetched, rendered, or indexed. CSS and JavaScript files are fetched separately, each with its own limit. Most pages are far smaller, but if yours is large: - Move heavy CSS and JavaScript into external files. - Avoid large inline base64 images. - Put the important parts (``, meta tags, canonical link, structured data) near the top of the HTML. Google notes this limit isn't permanent and may change. * * * ## 4\. Rendering: Running the JavaScript After fetching, Google renders the page with a recent version of Chromium so it can see content that JavaScript adds. Pages wait in a render queue, which can take a few seconds or longer. ```mermaid flowchart LR A["Crawl queue"] --> B["Fetch raw HTML"] B --> C["Extract links<br/>from the HTML"] C --> A B --> D{"noindex in<br/>raw HTML?"} D -- Yes --> E["Not rendered or indexed"] D -- No --> F["Render queue"] F --> G["Headless Chromium<br/>runs JavaScript"] G --> H["Rendered HTML<br/>processed and indexed"] H -.->|"new links found"| A ``` What this means in practice: - **Content must appear without interaction.** Googlebot won't click, swipe, or type, so don't load primary content only after those actions. - **Don't rely on JavaScript to remove** `noindex`**.** If the raw HTML says `noindex`, the page may never be rendered. - **Use real status codes.** Single-page apps should return a true 404 for missing pages, or redirect to a URL that does, to avoid soft 404s. - **Server-side rendering or pre-rendering is still a good idea.** It speeds up the page for users, and not every bot runs JavaScript. * * * ## 5\. Indexing: Understanding and Storing the Page Once Google has the rendered page, it analyzes the text, `<title>` elements, attributes like `alt`, images, and video. It then groups pages with similar content and picks one **canonical** version, the page that may be shown in results. The canonical page is the one stored and eligible to appear. ```xml <link rel="canonical" href="https://example.com/blog/how-search-works"> ``` Many pages that get crawled are never indexed. Common reasons are a `noindex` rule, a duplicate of another page, or content that Google judges to be low quality. ```mermaid flowchart TD A["Rendered page"] --> B{"noindex?"} B -- Yes --> N["Not indexed"] B -- No --> C{"Duplicate of<br/>another page?"} C -- Yes --> D["Grouped with duplicates<br/>one canonical chosen"] D --> E["Only the canonical<br/>may appear in results"] C -- No --> F{"Valuable enough<br/>to keep?"} F -- No --> G["Crawled - currently<br/>not indexed"] F -- Yes --> H["Added to the index"] ``` ### Mobile-first indexing Google uses the **mobile version** of your content, crawled with its smartphone crawler, for indexing and ranking. Make sure the mobile page has the same primary content as desktop, and that no `noindex` or `nofollow` tag appears only on the mobile version. * * * ## 6\. Helping Your Pages Get Indexed - **Link to every important page** from other pages on your site. - **Submit an XML sitemap** and list only canonical, indexable URLs. - **Keep servers fast and stable** so crawl capacity stays healthy. - **Request indexing** for important new or updated pages in Search Console's URL Inspection tool. This isn't a guarantee, and it can still take days to weeks. ```xml <?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>https://example.com/blog/how-search-works</loc> <lastmod>2026-11-01</lastmod> </url> </urlset> ``` A single sitemap can hold up to 50,000 URLs or 50 MB uncompressed. Beyond that, split it and use a sitemap index file. Keep `<lastmod>` accurate and only change it when the content really changes. Google ignores `<priority>` and `<changefreq>`. * * * ## 7\. Reading the Page Indexing Report Search Console's **Page indexing** report shows why URLs are or aren't indexed. A page not being indexed isn't automatically a problem; duplicates, redirects, and `noindex` pages shouldn't be. | Status | What it means | What to check | | --- | --- | --- | | Discovered - currently not indexed | Google knows the URL but hasn't crawled it yet | Internal links, server speed, crawl priority | | Crawled - currently not indexed | Google fetched the page but chose not to index it | Content quality, uniqueness, duplication | | Blocked by robots.txt | Crawling is disallowed | Remove the rule if the page should be indexed | | Excluded by 'noindex' tag | The page asks not to be indexed | Remove the tag if unintended | | Duplicate, Google chose different canonical than user | Google picked another URL as canonical | Canonical tags, redirects, internal links | | Soft 404 | The page looks like an error but returns 200 | Return a real 404, or fix the content | For one specific URL, use the **URL Inspection** tool instead. * * * ## 8\. Other Search Engines Bing and others follow the same crawl, render, index, and serve pattern. One difference is notification: the **IndexNow** protocol lets you tell participating engines such as Bing and Yandex the moment a URL is added, changed, or deleted. Google doesn't use it, so for Google rely on sitemaps and Search Console. Either way, a notification is a hint; the engine still decides whether to crawl and index the page. * * * ## 9\. Troubleshooting: Why Isn't My Page Indexed? ```mermaid flowchart TD S["Page missing from Google"] --> A{"URL Inspection:<br/>is the URL indexed?"} A -- Yes --> R["Indexed but not ranking:<br/>a serving and ranking issue"] A -- No --> B{"Blocked by robots.txt?"} B -- Yes --> B1["Remove the Disallow rule"] B -- No --> C{"noindex tag or header?"} C -- Yes --> C1["Remove noindex"] C -- No --> D{"Server errors or<br/>wrong status code?"} D -- Yes --> D1["Fix hosting, redirects, status codes"] D -- No --> E{"Content visible<br/>after rendering?"} E -- No --> E1["Fix JavaScript, use real links,<br/>consider server-side rendering"] E -- Yes --> F{"Duplicate or thin?"} F -- Yes --> F1["Consolidate or improve the content"] F -- No --> G["Add internal links and a sitemap,<br/>then request indexing and wait"] ``` * * * ## Quick Reference | Stage | What happens | Common blockers | | --- | --- | --- | | Discovery | Google finds the URL via links, sitemaps, or requests | No internal links, orphan pages, links that aren't `<a href>` | | Crawling | Googlebot checks `robots.txt` and fetches the page | Disallow rules, server errors, slow responses, 2 MB cutoff | | Rendering | Chromium runs JavaScript | Content needing interaction, `noindex` in raw HTML | | Indexing | Google analyzes, picks a canonical, and stores the page | `noindex`, duplicates, low-quality content | | Serving | Google matches indexed pages to queries and ranks them | Relevance and quality (not covered here) | * * * ## Final Thoughts Search engines work like a pipeline: a page has to be found, allowed, fetched, rendered, understood, and judged worth keeping. When a page is missing from results, work through the stages in order, as the troubleshooting diagram does, and you'll usually find the blocker quickly. **Remember the key distinction:** `robots.txt` controls what gets crawled, while `noindex` controls what gets indexed. Using the wrong one is behind a large share of indexing problems.