8 Ways to Find All Webpages on a Site (and Spot the Ones You Missed)

No single method lists every URL on a domain. A sitemap shows what the owner chose to declare. A crawler shows what internal links can reach. Search Console shows what Google has processed, and server logs show what bots actually requested. Each one has a blind spot, and the gaps between them are where forgotten, orphaned, and misconfigured pages live.

That matters for more than housekeeping. If search engines and AI crawlers cannot find a page, it is unlikely to be indexed or cited in AI-generated answers. The eight techniques below are ordered roughly by how fast they pay off, with what each one misses and how to combine them into a single URL inventory.

1. Read the XML Sitemap (and the Sitemap Index Behind It)

A sitemap is the owner's declared list of canonical URLs, which makes it the fastest starting point. Large sites usually split it into a sitemap index: a parent file that links to child sitemaps, often one per content type. Because the file is machine-generated, it reflects what the CMS believes is publishable, not necessarily what exists.

Suppose an ecommerce site's robots.txt points to sitemap_index.xml, which links to separate product, category, and blog sitemaps. Opening each child file and combining them gives you the declared URL set in minutes, already segmented by template.

1. Open /robots.txt and look for lines beginning with Sitemap:.
2. If none exist, try /sitemap.xml and /sitemap_index.xml.
3. Open every child sitemap listed in the index, including nested ones and compressed .xml.gz files.
4. Export the URLs to a spreadsheet, or load the sitemap into a crawler's list mode.

The common mistake is treating the sitemap as the full site. It omits orphan pages, and it often contains URLs that redirect or have been deleted, especially when the file is stale or hand-maintained. A sitemap that lists 12,000 URLs tells you what someone intended, not what is live.

To measure it, compare the number of sitemap URLs with the number of crawl-discovered URLs (covered next), and calculate the share of sitemap URLs that return a 200 status. A low 200 share means the sitemap is feeding crawlers bad signals.

2. Run a Full-Site Crawl with a Desktop or Cloud Crawler

A crawler starts at the homepage and follows internal links, mapping every page a visitor or bot could reach by clicking. Tools such as Screaming Frog SEO Spider and Sitebulb also record status codes, indexability, canonical tags, and crawl depth, the number of clicks from the homepage to a given URL. Depth matters because pages buried deep tend to be crawled less often and receive less internal link equity.

Imagine a crawl of a 4,000-page SaaS blog that reveals 300 pages at depth 6 or deeper, each with almost no internal links pointing to them. Those pages exist in the sitemap but are practically invisible to the site's own navigation.

1. Set the crawler's user-agent. Use a Googlebot-like string if you want to see what Google sees.
2. Turn on JavaScript rendering if the site builds its links client-side.
3. Connect the sitemap and the Search Console API if your crawler supports it, so the crawl can flag URLs found only in those sources.
4. Crawl, then export the Internal report with status code, indexability, and depth.

The classic failure is leaving JavaScript rendering off on a client-rendered site. The crawl then stops at the homepage or finds only a fraction of the links, and the resulting inventory looks complete when it is not. Check the first results: if a large site returns a few dozen URLs, suspect rendering. Also note that free versions have caps; Screaming Frog's free tier has historically been limited to 500 URLs, so confirm the current limit as of 2026.

A crawl cannot reach orphan pages, login-gated areas, or links that never render. Measure total crawlable URLs, the share returning 200, and the number of pages beyond depth 4.

3. Use Google Search Operators to See What Is Indexed

Search operators let you sample Google's index directly, including for sites you do not own. This is the only method here that shows what Google has actually chosen to store rather than what the site declares or links to. Its strength is surfacing surprises.

For example, running site:example.com inurl:? can expose parameter URLs generated by filters or sorting that were never meant to be indexed. A search like site:example.com intitle:"staging" might reveal a test environment leaking into results.

1. Start with site:domain.com to see the broad footprint.
2. Narrow by folder, for instance site:domain.com/blog/.
3. Filter by pattern with inurl: or intitle:.
4. Exclude known patterns with -inurl: to peel back layers and expose less obvious paths.
5. Record every unfamiliar URL or pattern for follow-up.

The common mistake is quoting the result count as exact. Google's number is a rough estimate, and results are capped, so you cannot page through an entire large site. Use operators for discovery and sampling, never for counting.

The useful measure is the number of unexpected URL patterns found: staging hosts, parameter strings, legacy folders, or duplicate paths. Each pattern is a lead that the crawl and logs can then size properly.

4. Pull Verified URLs from Google Search Console

For a property you own, Search Console is the authoritative record of how Google treats your URLs. The Pages report (under Indexing) groups URLs into indexed and non-indexed categories with a stated reason for each, while the Performance report lists pages that earned impressions and clicks.

Imagine the Pages report showing hundreds of URLs labelled Discovered - currently not indexed. That pattern often points to thin content, weak internal linking, or a site that is publishing faster than Google wants to crawl. It tells you which pages exist but are not earning a place in the index.

1. Open Indexing > Pages and export each status group.
2. Export the Performance report by page for a long date range.
3. For sites that exceed the interface's export row limits (confirm the current caps as of 2026), use the Search Console API to pull data in batches.

Do not read Excluded as an error. Many exclusions, such as canonicalized duplicates, redirects, and pages carrying a noindex tag, are intentional. The question is whether each exclusion matches your intent, not whether the count is zero.

Track the indexed-to-submitted ratio for your sitemap, and the count of URLs in each non-indexed reason category over time. A growing "crawled - currently not indexed" bucket is an early quality warning.

5. Mine Server Logs and Analytics for URLs Nobody Links To

An orphan page has no internal links pointing to it, so a crawler starting from the homepage will never find it. Bots and visitors can still arrive through old backlinks, bookmarks, or previous sitemaps, which means access logs and analytics record URLs the crawl cannot.

Suppose logs show Googlebot repeatedly requesting an old /promo-2023/ path that no page links to. The crawl never found it, yet it consumes crawl budget, the share of Google's crawling attention allocated to your site, and may still be returning content.

1. Export 30 to 90 days of access logs.
2. Filter to verified bot user-agents and 200 responses.
3. Extract the unique URLs and compare them to your crawl list.
4. Add analytics landing pages as a second source to catch human-visited orphans.

The pitfall is trusting user-agent strings. Anyone can claim to be Googlebot, so verify requests with a reverse DNS lookup before drawing conclusions. Otherwise scrapers inflate your bot data.

Measure the count of orphan URLs (present in logs, absent from the crawl) and the share of bot requests spent on non-indexable URLs. If a large slice of crawl activity goes to redirects, parameters, or noindexed pages, you have a crawl efficiency problem as well as an inventory gap.

6. Query the Wayback Machine and Common Crawl for Historic URLs

Public archives record URLs that existed in the past, including ones removed long ago. This is especially useful before a migration and when reviewing a competitor, where you have no logs or Search Console access. The Internet Archive's Wayback Machine exposes a CDX API for listing captures, and Common Crawl publishes an open index of pages its crawlers fetched.

Before a site migration, for instance, the CDX API might return old blog URLs that still have backlinks pointing to them. If those paths are not redirected, the link equity is lost the moment the new site launches.

1. Call the CDX API with the domain and a wildcard (for example, domain.com/*).
2. Collapse results by URL key so each URL appears once.
3. Filter to captures with a 200 status.
4. Status-check every URL against the live site.
5. For Common Crawl, query its URL index for the domain and merge the results.

Do not assume archived URLs are live. Most need a status check, and many should be redirected to a relevant current page, not restored. Archives also include junk: tracking parameters, feeds, and one-off assets.

Measure the number of historic URLs still returning 200, and the subset that has backlinks yet currently returns 404. That second group is your redirect priority list.

7. Pull Backlink Target URLs from SEO Tools

Pages that attract external links often disappear from navigation and sitemaps while the links keep pointing at them. Backlink and organic-ranking exports from tools like Ahrefs and Semrush list those targets, and they work on competitors as well as your own domain.

Imagine a backlink export showing a deleted guide with dozens of referring domains, now returning a 404. Redirecting it to the updated guide can recover link equity that was otherwise going nowhere.

1. Export the "Best by links" or "Top pages" report from your chosen tool.
2. Run the URLs through a status checker and filter for 404 and 3xx targets.
3. Map each broken or redirecting URL to the closest relevant live page.
4. Implement 301 redirects, and fix redirect chains while you are there.

Be careful about treating one tool's database as complete. Each provider crawls the web differently, so coverage varies, and a URL missing from one export may appear in another. Where it matters, combine exports from two sources.

Measure the number of linked URLs returning errors and the referring domains recovered after redirects. For competitor research, note which of their pages attract links, since those are the ones worth studying.

8. Merge, Deduplicate, and Re-Check Sources with Indexing Monitoring

The individual lists only become useful once combined. A master inventory with one column per source makes the blind spots visible: pages found in the crawl but missing from the sitemap, pages in logs but not in the crawl, pages in Search Console that nothing links to.

1. Normalize URLs: pick one protocol and host, settle trailing-slash handling, lowercase where appropriate, and strip tracking parameters.
2. Deduplicate the combined list.
3. Status-check every URL and record the response code and canonical target.
4. Add a flag column for each source (sitemap, crawl, Search Console, logs, archive, backlinks).
5. Schedule recurring crawls, and push new or updated URLs through IndexNow and automated sitemap updates so search engines learn about changes quickly.

The common mistake is building the list once and never refreshing it. On an actively published site it drifts out of date within weeks. Sight AI's website indexing tools, which pair IndexNow submission with automated sitemap updates, are built for that maintenance step, and its AI visibility tracking shows whether the pages you surface are actually being mentioned by AI models.

Track source-coverage gaps over time, the time from publish to indexing, and the count of valid URLs missing from the sitemap. If the gap list shrinks each cycle, the process is working.

Where to Start, and How the Sources Fit Together

Begin with the sitemap and a crawl for a fast baseline. On sites you own, add Search Console and server logs to catch orphans and see how Google treats them. For historic or competitor URLs, rely on web archives and backlink exports, then merge everything into one inventory and keep it maintained. A complete URL list is what lets you fix indexing problems and make sure crawlers, including those behind AI search, can reach the pages you want cited.

Stop guessing how AI models like ChatGPT and Claude talk about your brand. Get visibility into every mention, track content opportunities, and automate your path to organic traffic growth. Start tracking your AI visibility today and see exactly where your brand appears across top AI platforms.