7 Ways to See All Website Pages (and Find the Ones You Didn't Know Existed)

Ask most teams how many pages their site has and you will get a guess, often off by a wide margin. Old campaign pages, parameter URLs, staging leftovers, orphaned posts, and drafts that were never meant to be public all pile up out of sight. Every one of them affects crawl budget, index quality, and whether search engines and AI models find your best content.

No single source gives you the full list. A sitemap shows what you intended to publish, a crawler shows what is linked, Search Console shows what Google knows, and logs show what bots actually request. The seven methods below work best combined, and most of them work on a competitor's site too, within limits noted along the way.

1. Start With the XML Sitemap

A sitemap is a file that lists the URLs a site owner wants search engines to discover. Because the owner maintains it, it is the fastest approximation of the site's intended page set, and it is public on most sites, yours or a competitor's.

Suppose a 400-page SaaS blog exposes a sitemap index that points to separate child sitemaps for posts, categories, and static pages. Add up the URLs in each child file and you have a baseline in minutes, plus a first look at how the site is organized: if the category sitemap holds 150 URLs for a 400-post blog, that is already a question worth asking.

1. Try /sitemap.xml and /sitemap_index.xml on the domain.
2. Open /robots.txt and look for one or more Sitemap: lines, which are the most reliable pointer.
3. If you land on a sitemap index, open every child sitemap. Some are gzipped.
4. Paste the URLs into a spreadsheet, or use a sitemap extractor, and count them per child file.

The common mistake is treating the sitemap count as the total page count. A sitemap can contain stale URLs that now redirect or return 404, entries marked noindex, and it will omit any page the CMS or a plugin forgot to include. It lists intent, not reality.

Measure total sitemap URLs against indexed URLs in Search Console. The gap between those two numbers is your first audit target.

2. Use the site: Operator for a Quick Index Count

The site: operator restricts Google results to one domain. It needs no access to the property, so it is the quickest way to see what Google shows for any site, including a competitor's, and to notice sections you did not know were indexed.

Imagine site:example.com/blog returns far fewer results than the sitemap lists for that folder. That mismatch points to an indexing gap: pages submitted but not kept. The reverse is just as useful. If a search returns hundreds of results from a path that is not in your sitemap, something is generating indexable URLs you never planned.

Put it into practice by layering operators:

• site:domain.com for the whole domain.
• site:domain.com inurl:/blog/ to isolate a section.
• site:domain.com -inurl:/blog/ to see everything outside it.
• site:domain.com inurl:? to surface parameter URLs, and inurl:staging or inurl:dev for leaked test paths.
• site:domain.com intitle:"untitled" or other placeholder titles to catch thin or template pages.

The mistake to avoid is reporting the displayed result count as an exact figure. It is an estimate that can shift between searches and shrink as you page through results. Use it for direction, not for a number in a report.

Track a rough indexed count per section and compare it with the matching sitemap counts. Large, consistent gaps are the signal; small differences are noise.

3. Pull the Authoritative List From Google Search Console

For a site you own, Search Console is the closest thing to Google's own record. The Pages report (under Indexing) separates URLs into indexed and not indexed, and gives a reason for each non-indexed group, such as "Crawled - currently not indexed", "Discovered - currently not indexed", "Duplicate without user-selected canonical", or "Excluded by noindex tag". The distinction matters: discovered means Google knows the URL exists but has not crawled it, while crawled-not-indexed means it fetched the page and chose not to keep it.

Consider a site that finds several hundred URLs under "Crawled - currently not indexed", most of them thin tag archive pages. Seeing the pattern lets the team noindex or prune those pages and consolidate internal links toward the content that deserves to rank.

1. Verify the domain property so you see all protocols and subdomains.
2. Open Indexing, then Pages.
3. Use the sitemap filter to view only URLs from a submitted sitemap, or all known pages to include URLs Google found elsewhere.
4. Click into each reason group and export the example URLs.
5. For large sites, the report's examples and export are capped (check the current limits in Google's documentation), so use the Search Console API and URL Inspection API to pull more.

The classic pitfall is staring only at the indexed count. The non-indexed groups are where the diagnosis lives, and a healthy-looking indexed number can hide a large pile of pages Google rejected.

Track the indexed-to-submitted ratio and the number of URLs in each non-indexed reason over time. A reason group that keeps growing usually points to a template or plugin issue.

4. Crawl the Site With Screaming Frog or a Cloud Crawler

A crawler starts from one URL and follows internal links the way a search bot does, recording status codes, titles, canonicals, indexability, and depth for every page it reaches. Desktop tools like Screaming Frog work well for small and mid-size sites; cloud crawlers handle larger ones without tying up your machine. Because it needs only public access, crawling also works on competitor sites.

Imagine a crawl of a 2,000-URL site that shows key product pages sitting at click depth 6. Those pages are technically reachable but weakly linked, and they likely get less crawl attention. Compare that crawl against the sitemap and a second finding appears: pages in the sitemap that the crawler never reached because nothing links to them.

1. Enter the start URL and confirm the crawler respects robots.txt.
2. Enable JavaScript rendering if the site builds navigation or content client-side.
3. Connect the sitemap, plus Search Console and analytics data if the tool supports it.
4. Set a sensible crawl speed, especially on a site you do not own.
5. After the crawl, export Internal, All, then filter by status code and indexability.

Two mistakes dominate. Crawling without rendering on a JavaScript-heavy site returns a partial list, and ignoring robots.txt or rate limits can both disrupt a server and get you blocked. Also check the free-tier URL cap of your tool (Screaming Frog's free version has historically been limited to 500 URLs; confirm the current limit) so you do not mistake a capped crawl for the whole site.

Measure crawled URL count, the share of pages that are indexable, and the click depth distribution. A long tail beyond depth 4 or 5 is worth fixing through internal linking.

5. Find Orphan Pages by Cross-Referencing Sources

An orphan page is a live URL with no internal links pointing to it. A crawler cannot find orphans by design, so they only appear when you compare lists from different sources: the crawl, the sitemap, Search Console, analytics, and the CMS. Anything present in one of those but missing from the crawl is a candidate.

For example, imagine old campaign landing pages that still earn a few organic visits a month. They show up in analytics landing-page data but not in the crawl, because the navigation that once linked to them was removed years ago. Left alone, they may carry outdated pricing or offers.

1. Export URL lists from the crawl, sitemap, Search Console, and analytics.
2. Normalize them: pick one protocol, one host (www or not), a consistent trailing-slash rule, lowercase paths, and strip tracking parameters.
3. Combine them in a sheet and use VLOOKUP, a pivot table, or a short script to flag URLs absent from the crawl.
4. Check each candidate's status, since a 404 in analytics history is not an orphan, just a dead URL.
5. Tag every real orphan with one action: link it, redirect it, noindex it, or remove it.

The mistake that creates the most noise is comparing un-normalized URLs. Without normalization, http versus https and slash variants of the same page look like hundreds of false orphans, and the team wastes a week chasing them.

Track the number of orphan URLs and the share resolved each quarter. The count should fall steadily, and new orphans appearing after a site migration or redesign tell you where the process leaks.

6. Analyze Server Logs to See What Bots Actually Request

Logs record every request your server or CDN receives, which makes them the only source that shows what crawlers really do rather than what a tool simulates. They reveal URLs that no sitemap, crawl, or report lists, such as retired paths, parameter combinations, and files that bots keep requesting from old links elsewhere on the web. They also show which AI crawlers visit, which matters for AI visibility.

Suppose the logs show Googlebot repeatedly requesting retired parameter URLs that now redirect or return errors. Each of those hits is crawl activity that could have gone to new content. The fix might be a cleaner redirect rule, removing internal links that still point to those URLs, or blocking the pattern in robots.txt.

1. Export raw access logs from the server, load balancer, or CDN for at least a few weeks.
2. Filter to the bots you care about, using each vendor's published user-agent names (check the vendor documentation for current names of Googlebot, Bingbot, and AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot).
3. Verify those hits by IP, using reverse DNS or the vendors' published IP ranges.
4. Aggregate hits per URL and status code.
5. Compare the result to your page inventory to find requested URLs you did not know about and indexable pages that are never requested.

The common error is trusting the user-agent string alone. Anyone can spoof Googlebot, so unverified data inflates bot activity and hides real patterns. Verification takes more effort, but without it the analysis is unreliable.

Measure bot hits per URL, the share of hits landing on 404s or redirects, and the count of indexable pages that no verified bot requested in the period. That last number is a useful early warning for pages that are effectively invisible.

7. Check the CMS and Keep a Living Page Inventory

Everything above sees the site from the outside. The CMS sees it from the inside, including drafts, scheduled posts, private pages, and noindexed content that crawlers and sitemaps skip. Exporting from it, then merging every other source into one master list, turns a one-off audit into something you can maintain.

Imagine a WordPress export that reveals dozens of drafts and noindexed pages invisible to any crawler. Merged into a master sheet next to crawl status, Search Console reasons, traffic, and notes on whether the page is being cited in AI answers, it becomes a single place to decide what to keep, improve, or retire.

1. Export every post type from the CMS, not just posts and pages: landing pages, custom types, and attachments too.
2. Merge it with the crawl, Search Console, and analytics exports using normalized URLs as the key.
3. Add columns for status, owner, target topic, and next action.
4. Schedule a monthly refresh, and assign someone to run it.
5. Keep the sitemap updating automatically as pages change, and notify search engines of new and updated URLs with IndexNow so changes are discovered faster.

The typical failure is building the inventory once and never touching it again. Within a quarter it drifts from reality, and people stop trusting it. Tie the refresh to a calendar and an owner rather than good intentions.

Tools can shorten the loop. Sight AI's website indexing handles automated sitemap updates and IndexNow submissions, and its AI visibility tracking shows which brand mentions across models like ChatGPT, Claude, and Perplexity connect back to the pages in your inventory.

Measure inventory coverage against crawl and Search Console counts, the time from publish to indexed, and the number of pages cited in AI answers. If coverage falls below what the crawl finds, the inventory is stale.

Where to Begin and What Becomes Routine

Start with the sitemap and Search Console. Together they give you a baseline in under an hour: the pages you intended to publish and what Google actually kept. Follow with a full crawl and the orphan comparison, which is where the surprises tend to appear. Server log analysis and the maintained inventory are not one-time projects; they belong in a monthly or quarterly routine, since that is how a site stays understood as it grows.

On a competitor's site, you are limited to the sitemap, site: searches, and a polite crawl. That is still enough to map their content structure.

Stop guessing how AI models like ChatGPT and Claude talk about your brand. Get visibility into every mention, track content opportunities, and automate your path to organic traffic growth. Start tracking your AI visibility today and see exactly where your brand appears across top AI platforms.