7 Ways to View All Pages on a Website (and Find the Ones You Can't See)

A complete page inventory is the starting point for almost every serious site task: a content audit, a migration, competitive research, or a hunch that new pages are not being indexed. The trouble is that no single method produces one. A sitemap shows what the owner declared. A search operator shows a sample of what the engine indexed. A crawler shows what is linked. Archives show what used to exist. Each lens misses pages the others catch, and the differences between them are where the useful findings sit. The seven methods below run from a 30-second check to a full crawl, and they work best in combination.

1. Check the XML Sitemap

An XML sitemap is a file where the site owner lists the URLs they want search engines to discover. It is the fastest way to see the pages a site declares, and it often reveals structure the navigation hides. Suppose a SaaS site's sitemap_index.xml links to separate sitemaps for posts, pages, and products. Opening the page sitemap shows a /resources/ section that appears nowhere in the main menu. That one file has just told you the site has a content hub you would never have found by clicking around.

The mechanism is simple: most CMS platforms and SEO plugins generate sitemaps automatically, so the file reflects what the system believes is publishable. That is also its weakness. The sitemap protocol caps each file at 50,000 URLs or 50MB uncompressed (sitemaps.org), so larger sites split into an index file pointing to many child sitemaps. You have to open each child to get the full picture.

To work through it:

1. Try /sitemap.xml and /sitemap_index.xml on the domain.
2. Open /robots.txt and look for lines beginning with "Sitemap:", which often point to non-standard locations.
3. Open every child sitemap listed in an index file.
4. Copy the URLs into a spreadsheet, or use a sitemap extractor tool to export them in bulk.

The most common mistake is assuming the sitemap is complete or current. Plugins frequently omit pages set to noindex, custom post types, or anything added outside the CMS, and they sometimes keep URLs for pages that were deleted long ago. Treat the sitemap as a statement of intent, not an inventory.

What to measure: the number of sitemap URLs against the number of URLs you later crawl and find indexed. A large gap in either direction is a finding in itself.

2. Use the site: Search Operator

Typing site:domain.com into Google or Bing returns pages the engine has indexed from that domain. It is the quickest view of the indexed lens, and it works on any site, including competitors', with no access required.

The value is in the modifiers. Suppose you run site:example.com inurl:/blog/ and see results that look sparse, while the sitemap lists several hundred posts. That mismatch suggests a large share of the blog is not indexed. Adding intitle: or a quoted phrase lets you probe specific topics or templates the same way.

1. Run site:domain.com and scan the first few pages of results.
2. Narrow with inurl:, intitle:, or quoted terms to sample individual sections.
3. Repeat the searches on Bing, which indexes differently and sometimes surfaces other URLs.
4. Note anything unexpected: parameter URLs, staging or dev paths, old subdomains, or thin tag pages.

The classic error is treating the estimated result count as exact, or expecting the operator to list everything. Search engines show a sample, and the totals shift between queries. Use it for spot checks and for discovering strange indexed URLs, not for counting.

What to measure: how each indexed-section sample compares with the same section in the sitemap, and how many unexpected indexed URLs you find. Staging paths and parameter duplicates showing up in results are usually worth fixing immediately.

3. Crawl the Site with a Crawler

A crawler such as Screaming Frog starts at a URL, follows every internal link it finds, and returns each reachable page with status codes, titles, canonicals, and indexability data. This is the linked lens, and it is the closest thing to a true inventory of pages a visitor or bot can reach by clicking.

Imagine a crawl of a 400-page site. It returns 60 URLs with 404 responses, which are broken links the sitemap never showed, and several pages sitting more than five clicks from the homepage. Neither problem is visible from the sitemap or the search operator, yet both affect how well those pages get discovered.

Setup matters more than the tool:

1. Enter the start URL, usually the homepage.
2. Enable JavaScript rendering if the site builds its pages or navigation in the browser.
3. Add the sitemap to the crawl so pages that are declared but not linked are included.
4. Respect robots.txt, and throttle the crawl speed on sites you do not own.
5. Export the internal URL list with status and indexability columns.

The costly mistake is crawling a JavaScript-heavy site without rendering. The crawler sees an empty shell, finds a handful of links, and returns an almost empty list that looks like a small site. If the count seems implausibly low, check rendering first. Also check the tool's current free-tier URL limit before planning a large crawl, since those limits change.

What to measure: total crawlable URLs, the share that are indexable, and maximum click depth. Pages deeper than three or four clicks are candidates for better internal linking.

4. Pull URLs from Google Search Console

For a site you own, Google Search Console shows what Google itself knows: which URLs are indexed, which it found but excluded, and why. It is the only method here that reports Google's own view rather than inferring it. It does not work for competitor sites, since you must verify ownership.

Suppose the Pages report shows 120 URLs under "Discovered - currently not indexed", all in a recently launched section. That pattern points to weak internal linking: Google knows the URLs exist, perhaps from the sitemap, but nothing on the site gives it a reason to prioritize them.

1. Verify the property, preferably as a domain property so all subdomains and protocols are covered.
2. Open Indexing, then Pages, and review each exclusion reason in turn.
3. Export the lists for the reasons that matter.
4. Pull page-level data from the Performance report, or use the Search Console API for large sites, since the interface limits how many rows you can export.

The common mistake is looking only at the indexed count and ignoring why other pages were excluded. Reasons like "Duplicate, Google chose different canonical" or "Crawled - currently not indexed" each call for different fixes, and some exclusions are intentional and harmless.

What to measure: the ratio of indexed to not-indexed URLs, and the number of pages with zero impressions over a meaningful window. Indexed pages that earn no impressions are often as much of a problem as unindexed ones.

5. Query the Wayback Machine and Common Crawl

Archives record what a site looked like over time, so they surface pages that have dropped out of navigation, sitemaps, and sometimes the live site entirely. This is the historical lens, and it matters most before a redesign or migration, when forgotten URLs can quietly lose their backlinks.

For example, imagine an archive query before a redesign that turns up old resource URLs. Several still have inbound links from other sites. Without 301 redirects, that link equity disappears at launch.

1. Query the Internet Archive's CDX API with a domain wildcard (for example, domain.com/*) to list captured URLs. Common Crawl's index offers a similar lookup.
2. Filter to useful status codes and deduplicate the list, since archives store many captures per URL.
3. Run the cleaned list through a crawler in list mode to check each URL's current response.

The main pitfall is treating archived URLs as live pages. Most will return 404 or redirect, and a few will be live pages nothing links to. Always check current status codes before drawing conclusions.

What to measure: the count of legacy URLs that still return 200, plus any that carry backlinks or still receive traffic. Those are the ones worth redirecting, restoring, or deliberately retiring.

6. Find Orphan Pages by Combining Data Sources

An orphan page exists and may even be indexed, but no internal link points to it. A crawler cannot find these by following links, so you find them by comparing sources: anything that appears in the sitemap, analytics, server logs, or a backlink export but not in the crawl is a candidate.

Suppose an old webinar landing page shows up in both analytics and a backlink export, yet your crawl never reached it. People are arriving from external links, but the site gives them no path to or from it. That is an orphan worth a decision.

The procedure is a diff, and its success depends on cleaning the data first:

1. Export URL lists from the crawl, sitemap, analytics, logs, and a backlink tool.
2. Normalize them: pick one protocol, one host (www or not), a consistent trailing-slash rule, lowercase paths, and strip tracking parameters.
3. Deduplicate each list.
4. Subtract the crawl list from every other list to isolate URLs the crawl did not reach.
5. Check each remaining URL's status and decide: link it, redirect it, or remove it.

Skipping normalization is the mistake that wastes the most time. A trailing slash or an http versus https difference makes identical pages look different, producing dozens of false orphans.

What to measure: the number of true orphan URLs found, and how many you resolve through linking, redirecting, or removal over the following weeks.

7. Verify Discovery and Index Status, Then Fix the Gaps

A full inventory only pays off when you compare it with what search engines actually indexed. The useful list is the difference: pages that are live and worth ranking but absent from the index. Those pages cannot earn organic traffic, and they are also less likely to be surfaced by AI answer engines that draw on indexed web content.

Suppose a team merges its inventory with index data and finds 80 live product pages missing from the index. The cause is plain: the pages are not in the sitemap and are linked from nowhere but search results. They add the URLs to the sitemap, link them from relevant category pages, and submit the updates through IndexNow.

1. Merge the combined inventory with index data from Search Console and your site: samples.
2. Flag URLs that are live, indexable, and not indexed.
3. Add them to the sitemap and strengthen internal links from relevant, already-indexed pages.
4. Submit changes through IndexNow, which notifies participating engines such as Bing of new or updated URLs (check the current list of supporting engines, since Google has not been a participant), or use an automated indexing tool. Sight AI's website indexing handles IndexNow submission and sitemap updates automatically.
5. Track whether AI models mention your priority pages and brand, for instance with AI visibility tracking across ChatGPT, Claude, and Perplexity.

The frequent mistake is resubmitting the same URLs again and again without changing anything. If a page is thin, duplicated, or unlinked, notification will not fix it; the underlying quality or linking problem has to change first.

What to measure: time from publish to indexed, the percentage of live pages that are indexed, and AI mentions for priority pages. Falling time-to-index and a rising indexed share show the fixes are working.

Running the Methods in the Right Order

Start with the sitemap and a few site: searches. Together they give you a five-minute baseline of what the site declares and what the engine shows, and for a competitor that may be all you need. Next, run a crawl for completeness and to see how the pages connect. If the site is yours, add Search Console data and the orphan-page comparison, since those expose the pages the crawl cannot reach and the reasons others are excluded. Use the Wayback Machine whenever a migration or redesign is coming. Finish with the indexing fixes, because the inventory only matters once the gaps it reveals are closed. When auditing a site you do not own, keep crawls polite and respect robots.txt and the site's terms of service.

Stop guessing how AI models like ChatGPT and Claude talk about your brand. Get visibility into every mention, track content opportunities, and automate your path to organic traffic growth. Start tracking your AI visibility today and see exactly where your brand appears across top AI platforms.