A complete list of a website's pages is the starting point for content audits, migrations, indexing fixes, and competitor research. By the end of this guide you will have a de-duplicated inventory of URLs, whether the site is yours or a competitor's, and you will know which method fits which job. All you need is a browser. A free crawler and Google Search Console access (for your own site) are optional but helpful.
Step 1: Decide what kind of page list you need
Start with the goal, because it picks the method. A full inventory of your own site, a content audit of a competitor, and a check of which pages Google has indexed are three different jobs.
No method guarantees a complete list. Orphan pages (pages with no internal links pointing to them), noindex pages, pages behind logins, and dynamically generated URLs are often invisible to outside tools. Accept this now and plan to combine sources rather than hunting for one perfect export.
It also helps to separate three sets that people routinely confuse:
- Pages that exist: everything the server will return, including orphans and hidden pages.
- Pages that are crawlable: what a bot can reach by following links and sitemaps, within robots.txt rules.
- Pages that are indexed: what a search engine has chosen to store and show.
These three rarely match. A page can exist but be unlinked, be crawlable but marked noindex, or be in the sitemap yet never indexed. The gaps between the sets are usually the most useful finding of the whole exercise.
As a quick rule of thumb: for your own site, use a CMS export plus Search Console, then confirm with a crawl. For someone else's site, use the sitemap, search operators, and a crawler, and add archive and backlink data if you need depth. The steps below follow roughly that order.
Step 2: Check the XML sitemap and robots.txt
An XML sitemap is a file where a site owner lists the URLs they want search engines to find. It is the fastest way to get a large list, so check it first. Try these locations in your browser, replacing example.com with the domain:
- example.com/sitemap.xml
- example.com/sitemap_index.xml
- example.com/wp-sitemap.xml (the WordPress default since version 5.5)
- example.com/robots.txt
The robots.txt file is worth opening even if a sitemap loads straight away. It usually contains a Sitemap: line pointing to the correct location, and sometimes several. It also shows which paths the owner blocks from crawlers, which hints at admin areas, internal search, or staging sections.
Working through sitemap index files
Larger sites use a sitemap index, which lists child sitemaps split by type: posts, pages, products, categories. Open each child file and note the lastmod dates. Individual sitemap files are capped at 50,000 URLs or 50 MB uncompressed under the Sitemaps protocol, so big sites always split them.
To get the URLs into a spreadsheet, you can pull them into Google Sheets with the IMPORTXML function, pointing it at the sitemap URL and an XPath such as //*[local-name()="loc"]. Alternatively, use a free sitemap-to-CSV extractor. Either way, you end up with one column of URLs per sitemap.
Mistake to avoid: treating the sitemap as the complete truth. It lists only what the owner chose to submit, and it is often stale, auto-generated with gaps, or missing whole sections. Treat it as one source, not the answer.
For your own site, the fix is keeping the sitemap current automatically. Tools such as Sight AI's website indexing keep the sitemap updated and send IndexNow pings when pages change, which makes this list far more trustworthy over time.
Step 3: Use the site: search operator to see what Google has indexed
Type site:example.com into Google to see pages from that domain in the index. This reflects what Google actually shows, not what the owner submitted, which makes it a useful contrast to the sitemap.
Narrow the results to inspect sections:
- site:example.com/blog/ limits results to a folder.
- site:example.com inurl:product finds URLs containing a word.
- site:example.com intitle:pricing finds pages by title keyword.
- site:example.com -inurl:blog excludes a section to cut noise.
- site:example.com filetype:pdf surfaces PDFs and other hidden content types.
Be careful with the result count. It is an estimate, and Google does not display every indexed URL however far you page through. Use this method for sampling and spot checks, not as a full export.
The real value comes from comparing against Step 2. A URL in the sitemap that never appears under site: is likely an indexing problem: blocked, duplicate, thin, or simply not yet discovered. A URL that appears under site: but is absent from the sitemap is an unmanaged page, perhaps an old campaign page, a parameter variant, or a forgotten section. Note both lists in your spreadsheet as you go, since they feed the merge in Step 7.
Step 4: Crawl the site with a website crawler
A crawler starts at the homepage and follows links the way a search bot does, recording every URL it finds along with status codes, titles, and indexability. Screaming Frog SEO Spider, Sitebulb, and cloud-based crawlers all work. As of November 2026, check the vendor's page for current limits; Screaming Frog's free version has historically capped crawls at 500 URLs.
Enter the homepage URL and start the crawl, but adjust a few settings first:
- Respect robots.txt on competitor sites. Ignoring it is only appropriate on sites you own.
- Set a modest crawl speed. Lower the number of concurrent threads and URLs per second so you do not strain the server.
- Decide on subdomains. Enable crawling of all subdomains only if you want blog.example.com or shop.example.com included.
- Enable sitemap crawling. This lets the crawler find URLs that internal links do not reach, which partly closes the orphan-page gap.
If the site is built with a JavaScript framework, turn on JavaScript rendering. Without it, the crawler may see a nearly empty page and report only a handful of URLs, a clear sign something is wrong.
When the crawl finishes, export the Internal tab and filter it to HTML pages with a 200 status. Keep the 301 redirects, 404 errors, and noindex pages in separate exports rather than deleting them. Redirect chains and broken internal links are findings in themselves, and noindex pages tell you what the owner deliberately keeps out of search.
Logged-in areas will not appear unless you configure authentication, and for competitors you should not try.
Step 5: Pull the authoritative list from your own site's data sources
If you control the site, you can go beyond what any outside tool sees. This step does not apply to competitor research, so skip it in that case.
Google Search Console
The Pages report (under Indexing) separates indexed from not-indexed URLs and gives reasons such as "Crawled, currently not indexed" or "Duplicate, Google chose different canonical". The Performance report, filtered to the Pages view and exported, lists URLs that earned impressions, including ones your crawl missed. Export limits and menu paths change, so confirm the current options in the interface as of November 2026.
CMS exports
Your CMS holds the list of pages that exist:
- WordPress: Tools > Export, or an export plugin that outputs URLs to CSV.
- Shopify: product exports, plus page and collection lists from the admin.
- Webflow: CMS collection exports plus the static pages list.
Analytics and server logs
An analytics landing-page report shows URLs real visitors entered through. Server log files go further, recording every URL requested by users and bots. Both reveal URLs no crawler found, which is how you catch orphan pages that still receive traffic or bot hits. Logs also show whether Googlebot is spending time on pages you never intended to expose.
Export each of these as a single column of URLs, labelled by source, ready for the merge.
Step 6: Look for older and unlinked pages with the Wayback Machine and other sources
Pages that were once live often leave traces. The Internet Archive's Wayback Machine can list every URL it has captured for a domain. Visit web.archive.org/web/*/example.com/* and the tool returns a searchable list of captured paths. You can filter by status code and by extension, for instance to isolate PDFs or old HTML pages.
This is useful for finding removed pages, old redirects, and legacy content that may still attract backlinks. During a migration, these are the URLs that most often get forgotten, and losing them means losing link equity.
Backlink tools offer a second route. Most have a "top pages" or "best by links" report listing URLs that other sites link to. Compare it with your crawl: any linked URL the site no longer surfaces is either redirected, broken, or orphaned, and it deserves a decision.
Caution: archived and backlinked URLs may be dead. A capture from years ago proves the page existed, not that it exists now. Verify each one with a status-code check before it counts as a page. Keep these candidates in a separate tab until Step 7, where bulk checking will sort live URLs from dead ones.
Step 7: Merge, de-duplicate and verify your master list
Combine every export into one spreadsheet. Before removing duplicates, normalize the URLs so identical pages match:
- Trailing slashes: choose one form and apply it everywhere.
- Protocol: convert http to https.
- Host: collapse www and non-www to whichever version is canonical.
- Tracking parameters: strip utm_ and similar parameters, and review other query strings individually.
- Case and fragments: lowercase paths where the server is case-insensitive and drop #fragments.
Then remove duplicates. Add one column per source (sitemap, crawl, Search Console, site:, archive) and mark each URL with a 1 where it appeared. Pages that show up in only one source are your candidates: appearing only in the crawl suggests missing from the sitemap, only in the sitemap suggests orphaned or unindexed, only in the archive suggests stale or removed.
Next, check status codes in bulk, using your crawler's list mode or a bulk checker, and keep only 200-status canonical URLs for the final inventory. Pages that redirect or error go to a separate fix list.
The inventory then drives action. Update the sitemap, fix indexing gaps, redirect or restore valuable dead URLs, and for strategy work note which topics the site covers thinly. Sight AI users can pair this inventory with AI visibility tracking to see which pages AI models actually cite, and which are never mentioned.
Keeping the inventory current and putting it to work
A page list goes stale as soon as someone publishes, deletes, or restructures. Re-run the sitemap and crawl comparison monthly so new orphan or unindexed pages surface early, and keep the master spreadsheet as the base for content audits and AI visibility tracking.
Stop guessing how AI models like ChatGPT and Claude talk about your brand. Get visibility into every mention, track content opportunities, and automate your path to organic traffic growth. Start tracking your AI visibility today and see exactly where your brand appears across top AI platforms.
