7 Ways to List All Web Pages on a Website (and Find the Ones You Missed)

A complete list of URLs is the starting point for almost every site job: a technical audit, a migration, a content prune, or a check on what search engines and AI models can actually reach. The trouble is that no single method produces one. A sitemap shows what the site declares, a crawl shows what is linked, Search Console shows what Google has seen, the CMS shows what exists, and logs show what was requested. Each answers a different question, and the gaps between those answers are often the most useful finding in the whole exercise. The seven methods below run from a two-minute sitemap check to server log analysis, with notes on what each one misses so you can combine them.

1. Read the XML Sitemap

A sitemap is the site's own declaration of the URLs it wants discovered, which makes it the fastest baseline you can get. It is also only a declaration. Whoever generates it decides what goes in, and plugins and templates often include things nobody checked.

Suppose a 300-page SaaS blog exposes /sitemap_index.xml with child sitemaps for posts, pages, and categories. Importing those three files into a sheet gives you a baseline list in minutes, and the per-file counts alone tell you how the site is structured. A sitemap index is simply a file that points to other sitemap files, so you need to open the index first and then each child.

1. Open /robots.txt and look for lines beginning with Sitemap:. If there are none, try /sitemap.xml and /sitemap_index.xml.
2. Open each index and note every child sitemap URL.
3. Fetch each child file and import the <loc> values into a spreadsheet, or paste the sitemap into a crawler's list mode.
4. Dedupe the combined list.

Keep the protocol limits in mind. As of 2026, sitemaps.org specifies a maximum of 50,000 URLs or 50 MB uncompressed per file, so a large site will always be split across several files. If you only pulled one, you have a partial list.

The common mistake is treating the sitemap as the full site. Stale entries are just as common: redirected URLs, noindexed pages, and deleted content that the generator never dropped. Check the status of every entry before you trust it, and if you suspect broken links are hiding in the list, run a dedicated check.

To measure it, calculate the share of sitemap URLs that return a 200 and are indexable, against the total listed. A site where that share is far below 100% has a sitemap that is hurting more than helping.

2. Crawl the Site with a Dedicated Crawler

A crawler starts at one URL and follows internal links, recording every page it can reach along with status codes, canonicals, titles, depth, and indexability. Where a sitemap shows intent, a crawl shows what a visitor or bot can actually navigate to. Crawl depth, the number of clicks from the start page, comes free with it and flags pages buried too deep.

Imagine a crawl of a 2,000-page ecommerce site that finds 2,600 URLs. The extra 600 turn out to be parameterized filter pages (color, size, sort order) that need canonical or noindex handling. No sitemap would have shown you that, because nobody declared those URLs. The crawl found a crawl-budget and duplication problem as a side effect of listing pages.

1. Set the start URL, usually the homepage.
2. Respect or ignore robots.txt depending on the situation. Ignoring it is reasonable on your own site when you want to see blocked sections, but not on a site you do not own.
3. Enable JavaScript rendering if the site builds its navigation client-side.
4. Connect the sitemap and Search Console data if your tool supports it, so the crawler can compare sources.
5. Export the Internal tab (or your tool's equivalent) with status, indexability, and canonical columns.

The classic error is running a default, non-rendered crawl on a JavaScript-heavy site and concluding that pages do not exist. If links only appear after scripts run, a raw HTML crawl sees almost nothing. Compare the rendered and non-rendered link counts on a few templates before you trust the result. If your crawl stalls or returns too little, the causes are covered in this guide to slow website crawling issues.

Also check licensing limits. As of 2026, Screaming Frog's free version caps crawls at 500 URLs, so a larger site will silently stop short of the full set unless you use a paid licence.

Measure crawled URLs against sitemap URLs, and count the orphans flagged once the sitemap is connected. An orphan page is one with no internal links pointing to it, so a crawl alone can never find it. It only appears when another source names it.

3. Pull URLs from Google Search Console

Search Console lists the URLs Google has discovered or shown in results, which is a different set again from what you declared or linked. Its value is in the pages Google knows about that you did not expect.

For example, imagine the Pages report shows 400 "Discovered, currently not indexed" URLs that the team did not know existed. Some may be old parameter variants, others thin pages generated by a plugin. Either way, they are in Google's queue and consuming attention. If you want to confirm the state of specific pages, you can check if your website is indexed before digging into the exports.

Exporting from the interface

Export the URL lists behind each status in the Pages report, and export the Performance report with the Pages dimension to see URLs that earned impressions. Both exports are useful, but they describe different things: the first is about indexing state, the second about visibility.

Going past the row caps

The interface export is limited. As of 2026, the Performance report UI gives you up to 1,000 rows, and the Pages report examples are capped too. For larger sites, use the Search Analytics API with the page dimension and slice by date range (a week or a day at a time), then merge the results. This gets well past the UI ceiling.

The common mistake is assuming the 1,000-row export is the whole dataset. A site with 8,000 pages that exports 1,000 rows has seen an eighth of the picture and may not realize it.

Measure the number of URLs per indexing status, and, more usefully, the count of URLs Google knows that your crawl did not find. That second figure is a direct list of candidates for redirect, removal, or relinking.

4. Use Search Operators and Web Archives

These are the only methods that work on a site you do not control, and they add something no live source can: history. Search operators approximate what is indexed now, and archives recover what used to exist.

Imagine you are preparing a rebrand. You query the Wayback Machine CDX API for the domain and find old /resources/ URLs that earned backlinks years ago and were never redirected. Those links are pointing at 404s, and no current crawl, sitemap, or CMS export would ever surface them.

1. Run site:example.com/blog/ style queries with path filters to sample sections of the index. Treat the output as a sample, not a count.
2. Call the CDX API with matchType=domain, collapse=urlkey, and a filter for status 200 to get one row per unique archived URL.
3. Request each archived URL live and record the current status code.
4. Cross-check the survivors against a backlink tool to see which ones still carry links.

Two misconceptions cause most of the trouble here. First, site: result counts are estimates and fluctuate between runs, so never report them as precise. Second, an archived URL is evidence that a page existed, not that it exists now.

Measure the number of archived URLs that still resolve, or that have backlinks and no redirect. For competitor research, stay within robots.txt and the site's terms, and keep the work to public data.

5. Export from the CMS or Database

The system that stores your content knows every page record, whether or not anything links to it. That makes it the best source for finding orphans, since a crawl by definition cannot see pages nobody links to.

Suppose a WP-CLI post list shows 120 published pages and 35 drafts. Comparing the 120 to the crawl shows that 18 published pages have no internal links at all. They exist and are live, but only a direct link or a sitemap entry could lead anyone to them. Pages like these are often the reason website pages are not getting indexed fast.

1. Use the CMS's native export, WP-CLI (for example, wp post list with the post type and status flags), or a SQL query against the posts table filtered to published.
2. Include every post type in scope, not just posts and pages.
3. Build the full URL from the slug and the permalink structure, including any category or date prefix.
4. Save the list with a status and last-modified column.

The common mistake is an incomplete filter. Custom post types (case studies, docs, landing pages), app-generated routes that live outside the CMS, and non-published statuses like private or scheduled all get forgotten. Decide deliberately which statuses count and write the rule down, because a draft that is secretly publicly accessible is a finding in itself.

This method only works on sites you can access. For a client, ask for a read-only export rather than admin credentials.

Measure published CMS entries that are absent from the crawl. That number is your orphan count, and it should fall after you add internal links or retire the pages.

6. Analyze Server Logs and Analytics Landing Pages

Logs record what was actually requested, by bots and people, regardless of what any list says should exist. They are the closest thing to ground truth for what the server served.

Imagine log analysis shows Googlebot repeatedly requesting 5,000 legacy parameter URLs that no crawl ever discovered, such as old session or tracking variants. Nothing links to them, yet Googlebot keeps returning, spending crawl activity on pages with no value. Understanding how to make Google crawl your website efficiently starts with seeing exactly where it spends its time.

1. Request three to twelve months of access logs from the host or CDN. Longer windows catch rarely requested URLs.
2. Filter to 200 responses, and separately keep 301s and 404s for redirect work.
3. Verify Googlebot with a reverse DNS lookup followed by a forward lookup, rather than trusting the user-agent string.
4. Extract unique paths, normalize query strings, and count requests per path.
5. Cross-check against the GA4 landing page report to see which URLs real visitors entered on.

Two mistakes dominate. Spoofed user agents inflate "Googlebot" activity, so unverified data can send you chasing fake bots. And query-string noise makes one page look like thousands, so decide which parameters to strip before you dedupe.

Logs can be hard to obtain on managed hosting, and they are not available for sites you do not own.

Measure the count of URLs requested in logs that are missing from both the crawl and the sitemap. Those are pages the server is serving that your own tooling cannot see.

7. Merge Sources into a Master URL Inventory

Each earlier method gives you a partial list. The inventory is where the differences become diagnostic: a URL in the CMS but not the crawl is an orphan, one in the crawl but not the sitemap is undeclared, one in logs only is a legacy leftover. Labeling each URL by which sources contain it turns a pile of lists into a decision table.

Take an agency auditing a client site. It builds one sheet with columns for sitemap, crawl, GSC, CMS, and logs, then assigns each URL an action: keep, redirect, consolidate, or remove. Once the keep list is settled, it submits the priority pages through IndexNow so search engines that support the protocol hear about them quickly. Sight AI's website indexing tools automate that step with IndexNow and sitemap updates, so the inventory feeds straight into discovery. For more on this approach, see these automatic website indexing tools.

1. Normalize every list: lowercase the host, force one protocol, settle on trailing-slash handling, and strip tracking parameters.
2. Dedupe on the normalized URL.
3. Use XLOOKUP or SQL joins to flag presence in each source.
4. Add status code, indexability, canonical, and traffic columns.
5. Segment by gap type and assign an action to each segment.

The common mistake is merging without normalization. http:// and https://, trailing slashes, and capitalization differences create duplicates and, worse, false orphans: a page appears missing from the crawl only because it was written differently.

Measure the percentage of URLs present in all relevant sources, the number of gaps you resolved, and how long priority pages take to get indexed after submission. If that time shrinks over successive rounds, the inventory is doing its job, and you can keep speeding up website indexing with each pass.

Which Methods to Run First, and When to Go Further

Start with the sitemap and a crawl for a quick baseline. Add Search Console and CMS exports for coverage, and bring in logs plus a merged inventory when you need completeness. On a competitor's site, stick to the sitemap, a polite crawl, and archives.

A full URL inventory also supports the work that comes after it. Pages that are discoverable, indexable, and well linked are the ones AI models can draw on, so it is worth knowing how those models describe your brand once the list is clean. Stop guessing how AI models like ChatGPT and Claude talk about your brand: get visibility into every mention, track content opportunities, and automate your path to organic traffic growth. Start tracking your AI visibility today and see exactly where your brand appears across top AI platforms.