Most guides on how to find all pages of a website stop at a sitemap check and a site: search. Neither gives you a full inventory. A sitemap lists only what the owner submitted, and Google shows only a sample of what it has indexed. This guide combines six sources and reconciles them into one verified, de-duplicated list of URLs, for your own site or a competitor's. You need a browser, a free Google account with Search Console access (own site only), and a crawler such as Screaming Frog's free tier. Steps 4 and parts of Step 5 require owner access, so for a competitor, rely on Steps 1 to 3 and the archive portion of Step 5.
Step 1: Check the XML sitemap and robots.txt
An XML sitemap is a file where a site owner lists the URLs they want search engines to find. Start with example.com/robots.txt and look for lines beginning with Sitemap:. The file can declare one or several sitemap locations, and these are the most reliable pointers because the owner put them there deliberately.
If robots.txt has nothing, try the common default paths:
- /sitemap.xml
- /sitemap_index.xml (common with Yoast and similar plugins)
- /wp-sitemap.xml (WordPress core)
Many sites use a sitemap index: a parent file that points to child sitemaps split by type, such as posts, pages, products, and categories. Open each child file and export its URLs. In a spreadsheet, you can paste the XML into a text editor and extract the <loc> values, or use a crawler's list mode to fetch the sitemap directly.
Check the lastmod dates as a quick freshness signal. A sitemap where every URL shows today's date, or the same date, is probably auto-stamped, so treat these values as a hint rather than evidence. Many sites populate them inaccurately.
Mistake to avoid: treating the sitemap as a complete inventory. It reflects what the owner chose to submit. Orphaned pages, noindexed pages, old landing pages, and anything the CMS excludes may be missing, and stale URLs may linger long after the page was removed. Save this export as your first source and label it "sitemap."
Step 2: Use Google search operators to see what is indexed
Search operators show you the other side of the picture: what Google has actually indexed, regardless of what the sitemap says. Run site:example.com to see indexed pages, then narrow it:
- site:example.com/blog/ limits results to a folder.
- site:example.com inurl:category finds URLs containing a word.
- site:example.com intitle:pricing finds pages by title keyword, useful for slicing by page type.
- site:example.com -inurl:blog excludes a section you have already covered, so you can work through the remainder.
Work through the results page by page and copy the URLs you find. A browser extension can speed up collection, but check that it complies with Google's terms before automating anything at volume.
Know the limit. The result count is an estimate, and Google does not display every indexed URL for a query. Use this method for sampling and discovery, such as spotting a forgotten subfolder or parameter-heavy duplicates, never as an exact total.
Tip: run the operator against each hostname variant separately: http:// and https://, with and without www. If more than one variant returns results, you have found duplicate versions of the site that should be redirected or canonicalized. A canonical is the tag that tells search engines which URL is the preferred version of a page, so note any variants that lack one. Record everything under the source label "Google index."
Step 3: Crawl the site with an SEO crawler
A crawler follows internal links the way a search engine does, so it finds pages that are linked but not in the sitemap. Screaming Frog SEO Spider is a common choice. Its free tier has been limited to 500 URLs as of writing, so verify the current limit on their site; Sitebulb or a cloud crawler works for larger sites.
- Enter the start URL, typically the homepage, and begin the crawl.
- Enable Crawl all subdomains only if you want blog.example.com or shop.example.com included. Otherwise leave it off to keep the list focused.
- Check the robots.txt setting. On your own site, you may want to ignore robots.txt so blocked sections still appear in the list. On a competitor's site, respect it.
- Export Internal > HTML when the crawl finishes.
JavaScript-rendered sites. By default a crawler reads the raw HTML. If the site builds its navigation or content links with JavaScript, the crawler will see few links and stop early. Switch the rendering mode to JavaScript in the crawler's configuration and recrawl. A sign you need this: a crawl that finishes with a handful of URLs on a site you know is much larger.
In the export, filter by status code to separate live pages from the rest:
- 200: live pages, your core inventory.
- 301/302: redirects. Keep the destination, note the source.
- 404/410: broken or removed URLs still being linked to.
Remember that a crawl only finds what internal links lead to. Pages with no inbound links, called orphan pages, are invisible to it, which is why the next two steps exist. Label this export "crawl."
Step 4: Pull URLs from Google Search Console and analytics (your own site)
These sources show what Google and real visitors have touched, including pages your crawl never reached. You need ownership access, so skip this step for competitor sites.
Search Console
Open Indexing > Pages. You can export both indexed and non-indexed URLs, and each non-indexed group comes with a reason, such as "Crawled, currently not indexed" or "Excluded by noindex tag." A noindex tag tells search engines to keep a page out of results, so these pages are deliberate or accidental gaps worth reviewing.
Then open Performance > Search results, switch to the Pages tab, set the longest date range available, and export. This captures every URL that earned impressions, even if no internal link points to it. The interface caps the number of exported rows, so verify the current limit; for larger sites, pull the data through the Search Console API instead.
GA4
Go to Reports > Engagement > Pages and screens, set a long date range, and export. Long ranges matter because old, low-traffic pages only appear when you look back far enough. Note that GA4 reports page paths, so you will need to prepend the domain before merging.
Why this matters: a URL that appears in Search Console or GA4 but not in your crawl is very likely an orphan page. Those are either forgotten assets worth relinking or leftovers worth removing. Label these exports "GSC" and "GA4."
Step 5: Recover deleted and hidden pages from archives and server logs
Some pages are gone from the live site but still matter, especially after a migration, because old URLs may hold backlinks or rankings. Other pages exist but nothing links to them.
Public archives
The Internet Archive lists captured URLs. Browse web.archive.org/web/*/example.com/* for a list, or query its CDX API for a machine-readable export you can filter by status code and collapse by URL. Common Crawl publishes an index of its crawls that you can search by domain, which is another third-party source. Coverage in both is partial, so a missing URL proves nothing.
These work for competitor sites too, and they are often the only way to see pages a competitor has since removed.
Your own server and CMS
- Review server access logs for URLs that bots and visitors requested successfully. These reveal pages that no link or sitemap exposes.
- Export published content directly from the CMS: posts and pages from WordPress, products and collections from Shopify. The database is the closest thing to ground truth for what exists.
Flag the 404s. Check each recovered URL's status. Any that return 404 but have history, such as backlinks or past traffic, should be redirected to the closest equivalent page or restored. Label these exports "archive," "logs," and "CMS."
Step 6: Merge, de-duplicate, and verify your master list
Put every export into one spreadsheet with two columns: URL and source. Before removing duplicates, normalize the URLs so the same page is not counted several ways:
- Pick one protocol and hostname (https, www or non-www).
- Remove or standardize trailing slashes.
- Lowercase the host, and the path where the site is case-insensitive.
- Strip tracking parameters such as utm_source, but keep parameters that change the content.
Then de-duplicate, keeping the source column by listing every source that found each URL. This is the most useful part of the exercise. A URL found only in the sitemap may be stale. One found only in Search Console, GA4, or the archive is a likely orphan or a removed page. One found only by the crawl but not in the sitemap is a gap in the sitemap.
Next, verify. Load the merged list into your crawler's list mode and crawl just those URLs. Check each one's status code, canonical, and noindex tag, so the final list reflects what the pages do today rather than what a year-old export said.
Finally, act on the clean list. Rebuild the sitemap from the verified, canonical, indexable URLs and submit it in Search Console. To help new and updated URLs get discovered faster, push them with IndexNow, a protocol that notifies participating search engines when content changes. Sight AI's website indexing tools handle this with IndexNow integration and automated sitemap updates, so the list stays current without manual resubmission.
The same inventory shows you which pages AI models could cite at all, and Sight AI's AI Visibility tracking can then show whether those pages are being mentioned.
Turning the Audit Into a Quarterly Content Plan
Re-run this process every quarter and after any migration or redesign. Compare the new master list against the previous one: pages that disappeared, new orphans, and URLs stuck in "crawled, not indexed" are all signals worth acting on. Gaps in the inventory, such as topics with no page at all, become your content to-do list.
Stop guessing how AI models like ChatGPT and Claude talk about your brand. Get visibility into every mention, track content opportunities, and automate your path to organic traffic growth. Start tracking your AI visibility today and see exactly where your brand appears across top AI platforms.
