8 Ways to List All Pages on a Website (and Find the Ones You Missed)

A complete list of URLs is the starting point for almost every serious site task: a content audit, a migration, a technical SEO review, or a check on what search engines and AI models can actually discover. The trouble is that no single method produces one. A crawler misses pages nobody links to, a sitemap lists only what the owner declared, and Search Console shows only what Google chose to process.

The eight methods below run roughly from the most accessible to the most thorough, and each one notes what it will miss. The real value is in comparing the lists against each other, because the differences between them are where the forgotten, broken, and invisible pages live.

1. Crawl the Site with a Desktop SEO Crawler

A crawler starts at a URL, fetches the page, extracts every link, and repeats until it runs out of new internal URLs. The result is the set of pages a bot can reach by following links, which is also roughly what a search engine finds without any other hints. It is the fastest route to a large, structured list, and it works on any site, yours or a competitor's.

Consider an illustration: a 600-page SaaS blog crawled with JavaScript rendering on returns 640 HTML URLs. Forty of them are tag archives nobody on the team knew existed, generated automatically by the CMS. That surplus is as useful as the pages you expected.

1. Enter the start URL (use the canonical version, such as the https www or non-www host).
2. Enable JavaScript rendering if the site builds navigation or content client-side. Without it, a crawler may see only a shell.
3. Run the crawl and let it finish; watch for blocked resources and timeouts.
4. Export the Internal tab filtered to HTML, keeping status code and indexability columns.
5. Filter to status 200 and indexable to get your working page list.

Free tiers of desktop crawlers are usually capped at a few hundred URLs, so check the current limit of your tool (as of 2026) before assuming a free run covers a large site.

The common mistake is treating the crawl as the full inventory. A crawler only follows links, so orphan pages, meaning pages with no internal links pointing to them, are invisible to it. Other frequent errors include skipping JavaScript rendering and ignoring parameter and pagination URLs, which either bloat or distort the list. If your crawl also turns up dead ends, it helps to scan your website for broken links while the data is fresh.

Measure the total count of 200-status indexable URLs, and the number of those missing from your sitemap. That second figure flags pages you link to but never declared.

2. Read the XML Sitemap (and Sitemap Index Files)

A sitemap is the list the site owner chooses to hand to search engines. Reading it tells you what the owner considers important, and it often includes pages a crawl cannot reach. Large sites use a sitemap index, a file that points to multiple child sitemaps, because the sitemaps.org protocol limits each file to 50,000 URLs or 50 MB uncompressed.

Illustration: a sitemap index points to separate child sitemaps for posts, pages, and products. Combining them gives 1,800 URLs, against 1,650 found in the crawl. The 150-URL gap is worth investigating, since it could be orphaned products or stale entries.

1. Open /robots.txt and look for lines starting with Sitemap:. Also try /sitemap.xml and /sitemap_index.xml.
2. If the file is an index, open it and note every child sitemap URL.
3. Fetch each child file, including compressed .xml.gz files.
4. Extract the loc values with a spreadsheet XML import, a short script, or a crawler's list mode.

The pitfall is trusting a stale sitemap. Many are generated by plugins that keep redirected, noindexed, or deleted URLs for months. Never equate "in the sitemap" with "a live, indexable page." Run the extracted URLs through a status check before counting them.

Measure the overlap percentage between sitemap URLs and crawled URLs. Low overlap in either direction signals a problem: URLs only in the sitemap are possible orphans or dead entries, and URLs only in the crawl are pages the owner forgot to declare. If you are also wondering whether those declared pages are really in Google, you can check if your website is indexed before going further.

3. Pull Indexed URLs from Google Search Console

Search Console is the only source that shows Google's own view of a site: which URLs it indexed, which it found but skipped, and the stated reason. That makes it less a page list than a diagnosis of which pages in your other lists actually count. It requires verified ownership, so it applies to your own site only. For a deeper walkthrough, see this guide on how to find indexed pages in Google.

Suppose the Pages report shows 300 URLs under "Discovered - currently not indexed." That pattern often points to thin content, weak internal linking, or a site producing more URLs than Google wants to crawl. It tells you where to look in your crawl data.

The biggest trap is the export limit. The interface export caps at 1,000 rows, so a site with 8,000 URLs will look far smaller than it is if you take the download at face value. Work around it in two ways:

• Export each status group separately (indexed, crawled but not indexed, excluded by noindex, and so on), which keeps each group under the cap more often.
• Use the Search Analytics API, which returns page-level performance data in much larger batches, and the URL Inspection API for per-URL index status.

Report names and layout change over time, so confirm where the Pages report sits under Indexing in the current interface (as of 2026). Also note that Search Analytics lists only pages that earned impressions, so indexed pages with zero impressions can be missing from performance exports.

Measure the ratio of indexed URLs to submitted URLs, and track it over time. A falling ratio after a content push is an early warning that quality or crawl demand is not keeping up with volume.

4. Use the site: Search Operator for a Quick Outside View

The site: operator asks a search engine to return pages from one domain. It is the only method here that needs no access, no tool, and no permission, which makes it the best first look at a competitor. Its strength is discovery of sections rather than a count.

Illustration: running site:competitor.com inurl:/alternatives reveals an entire comparison-page section you did not know the competitor had built, complete with titles and snippets showing how they position against rivals.

1. Run site:domain.com and scan the first pages of results.
2. Add inurl: filters for paths such as /blog, /docs, or /integrations to isolate sections.
3. Use -inurl: to strip out sections you have already seen and surface what remains.
4. Record visible URLs in a sheet, grouped by section.

The classic mistake is quoting the displayed result count as an exact page total. That number is an estimate, it fluctuates between queries, and results are truncated after a limited depth. Treat it as a rough sense of scale at most.

Because results are sampled, this method will never give you a full list. Use it to find structure, then confirm with a crawl or sitemap. Measure the number of distinct sections it discovered that your crawl or sitemap missed. Even one is a sign that your inventory has a gap.

5. Mine Server Logs and Analytics for Pages Bots and Users Hit

Logs record what was actually requested, which makes them the best source for pages that no list admits to: orphans, legacy paths, and URLs that exist only in old links. They also show how often Googlebot and other crawlers visit each URL, which bears directly on crawl budget, the amount of crawling a search engine is willing to spend on your site. If crawl activity looks sluggish, these notes on slow website crawling issues can help you interpret what the logs show.

Illustration: logs reveal Googlebot repeatedly requesting 200 old /2019/ URLs that return 404. Those requests waste crawl effort and hint at old backlinks that deserve redirects.

1. Export 30 to 90 days of access logs from your server or CDN, so seasonal and low-traffic pages appear.
2. Filter to 200 and 301 responses, and separately keep the 404s for redirect work.
3. Strip query strings where appropriate and extract unique paths.
4. Remove assets (images, CSS, JS, fonts) and obvious junk paths from vulnerability scanners.
5. Pair the result with the GA4 landing-page report to include pages real users entered on.

The common mistake is leaving asset files and bot-generated junk in the page list, which inflates the count with URLs that are not pages at all. Verify that crawler traffic is genuine by checking reverse DNS for claimed Googlebot hits.

Measure the count of requested URLs absent from your crawl. Those are your orphans, and each one needs a decision: link it, redirect it, or remove it.

6. Export Directly from the CMS or Database

The CMS knows what content exists whether or not anything links to it. An export is the closest thing to an authoritative record of intentional content, and it is the one method that reliably finds published pages with no internal links.

Illustration: a WP-CLI export lists 920 published posts and pages, 35 more than the crawl found. All 35 have no internal links, so they sit published but undiscoverable except through the sitemap.

1. Run the platform's export or query. On WordPress, wp post list with a post type and status filter works; on other systems, use the admin export or a database query.
2. Filter to published status, excluding drafts, private posts, and trashed items.
3. Include every content type: pages, posts, products, custom types.
4. Map slugs to full URLs using the site's permalink structure, including any category or date prefixes.

The pitfall is forgetting dynamically generated URLs. Filters, tag and author archives, search results, and paginated listings come from templates, not from content rows, so the CMS export will not contain them even though a crawler will. Decide deliberately which of these count as pages.

Measure published items in the CMS against URLs in your crawl. A positive difference means orphans; a negative one means the site is generating URLs the CMS does not know about, such as parameter or archive pages.

7. Recover Historical and Hidden URLs with the Wayback Machine and Common Crawl

Web archives remember URLs the current site has forgotten. The Internet Archive's Wayback Machine and the Common Crawl dataset both expose indexes of captured URLs, which is valuable before a migration, when old pages with backlinks can lose their equity if left without redirects. They also work on competitor domains.

Illustration: before a migration, a CDX API query reveals old resource pages that still have backlinks but no redirects. None appear in the current crawl, sitemap, or CMS, so only the archive would have exposed them.

1. Query the Wayback CDX endpoint for the domain with a wildcard, using collapse=urlkey to return each URL once.
2. Filter to statuscode:200 so you start with pages that once loaded successfully.
3. Optionally query Common Crawl's index for the same domain to widen coverage.
4. Test the current status code of each historical URL in bulk, with a crawler's list mode or a script.
5. Cross-reference survivors against backlink and analytics data.

Parameter names and limits on these APIs change, so check the current documentation before building a script (as of 2026).

The mistake to avoid is treating every archived URL as a live page. Most will be dead, redirected, or irrelevant. Measure the number of historical URLs that still have backlinks or traffic and need a redirect, since that is the figure that changes migration plans.

8. Merge, Deduplicate, and Validate the Lists, Then Submit the Gaps for Indexing

Each method above gives a partial view. The master inventory, one sheet with a column per source, is where completeness comes from, because the pattern of presence and absence tells you what each URL is. In the CMS but not in the crawl means orphan. In logs but not the CMS means legacy or generated. In the crawl but not the sitemap means undeclared.

Illustration: a master sheet shows 120 URLs in the CMS only. After internal links are added and the URLs are submitted through IndexNow, they appear in Bing's index, which you can verify in Bing Webmaster Tools. IndexNow is a protocol that notifies participating search engines of new or changed URLs; support varies by engine, so confirm the current list (as of 2026) and note that Google has not been a participant. For more on the topic, see this overview of how to speed up website indexing.

1. Normalize every URL: lowercase the host, force https, pick a trailing-slash convention, and strip tracking parameters and fragments.
2. Deduplicate, then add one column per source with a simple yes or no.
3. Filter by presence patterns to isolate orphans, sitemap-only entries, and log-only URLs.
4. Check each URL's status code and canonical, and drop redirected or non-canonical URLs from the page count.
5. Add internal links to valid orphans, update the sitemap, and submit the new URLs through IndexNow.

The most common mistake is skipping normalization, so http and https versions, or slash and no-slash variants, count as separate pages and inflate the total. Tools that automate sitemap updates and IndexNow submission, such as Sight AI's website indexing, reduce the manual work of the final step, and a roundup of automatic website indexing tools compares the options.

Measure the master URL count, the orphan count, and the share of valid URLs indexed after 30 days. If that last share is low, the problem is quality or linking, not discovery.

Where to Begin, and What to Do With the Gaps

Start with a crawl and the sitemap, since together they take an hour and expose the largest discrepancies. Add Search Console for the indexing view, then layer in logs, CMS exports, and archives to reach completeness. Finish by merging everything and submitting the gaps, because a list you never act on is only a spreadsheet.

Discovery is also the first condition for being cited in AI answers: pages that are not indexed or linked rarely get surfaced. Stop guessing how AI models like ChatGPT and Claude talk about your brand. Get visibility into every mention, track content opportunities, and automate your path to organic traffic growth. Start tracking your AI visibility today and see exactly where your brand appears across top AI platforms.