7 Strategies to Check All Pages on a Website (and Fix What You Find)

Most sites contain more pages than anyone on the team can name. Old campaign pages, parameter URLs, orphaned posts, staging leftovers and template-generated duplicates pile up, and some of them are broken, blocked from indexing, or invisible to the AI answer engines that now shape how buyers research vendors. A one-time glance at the sitemap won't surface any of it.

To check all pages on a website, treat it as a pipeline: discover every URL, crawl it, fix technical faults, verify indexation, assess quality, repair internal linking, then notify engines and measure visibility. The seven strategies below follow that order, so the output of each step feeds the next.

1. Build a Master URL Inventory from Multiple Sources

The sitemap is a statement of intent, not a census. It lists what the CMS or a plugin thinks should be public, which excludes pages created outside the CMS, old campaign URLs, and anything the sitemap generator filters out. Every source of URLs has blind spots, but the blind spots differ, so combining sources exposes the gaps. If you want a deeper walkthrough of discovery methods, this guide on how to find all pages on a website covers them in detail.

Suppose, hypothetically, a sitemap lists 800 URLs, a crawl finds 950, and analytics shows 1,100 landing pages with sessions. That 300-URL spread is the interesting part. On investigation it might split into old campaign pages that still receive traffic, parameter URLs created by tracking tags, and blog posts that no internal link points to. None of these would have appeared in a sitemap-only review.

To build the inventory:

1. Export URLs from your XML sitemap (and any sitemap index children), your crawler, Google Search Console performance data, and your analytics landing page report.
2. Normalize every URL: force one protocol, one host (with or without www), consistent trailing slashes, lowercase paths where your server treats them as equal, and strip tracking parameters.
3. Merge the lists in a spreadsheet and dedupe.
4. Add one column per source with a yes/no flag, so each URL shows where it was found.

The common mistake is comparing un-normalized lists. If http://, https://, and trailing-slash variants count as separate pages, your "unique" total is inflated and the gap analysis is noise. The opposite error is trusting the sitemap alone and never seeing the gap at all.

Measure the number of URLs found per source and, most usefully, the count of URLs present in only one source. Those single-source URLs are your investigation queue, and that number should shrink as you clean up.

2. Crawl the Site with a Dedicated Crawler

A crawler follows links the way a search engine does, so it shows you the site as it is actually reachable, along with the technical attributes of each URL: status code, canonical, title, directives, depth, and inlinks. This is the backbone dataset for every later step.

Rendering is where crawls most often go wrong. Hypothetically, a product section built with client-side JavaScript might show only 20 pages in a basic HTML crawl, because the links to product pages don't exist until scripts run. With JavaScript rendering enabled, the same crawl finds 400. Without that check, you would audit a twentieth of the section and conclude it was healthy.

A workable setup:

1. Choose a crawler such as Screaming Frog or Sitebulb. Free tiers and URL limits change, so confirm the current cap before planning a crawl of a large site (as of 2026, check the vendor's pricing page).
2. Set the user agent, the crawl scope, and the subdomain rules. Decide deliberately whether blog, help, and app subdomains are in or out.
3. Enable JavaScript rendering if the site depends on it. Compare a rendered crawl against a raw one on a sample section if you're unsure.
4. Run the crawl, then export status codes, canonicals, crawl depth, and inlink counts.

The second classic mistake is letting the crawler wander into faceted navigation or endless parameter combinations. A filter system can generate thousands of near-identical URLs and bury the real pages. Use include/exclude patterns, parameter rules, or a crawl limit, and note which patterns you excluded so you can audit them separately. Sites that struggle with slow website crawling issues often trace the problem back to exactly this kind of URL bloat.

Measure the crawled URL count against the master inventory, and the share of crawled URLs that returned a successful response. Large shortfalls against the inventory point to orphans (strategy 6) or crawl configuration problems.

3. Audit Status Codes, Redirects, and Broken Links

Every request that ends in an error or a detour wastes crawl budget and degrades user experience. A 404 on a page with internal links is a dead end for visitors, and a long redirect chain adds latency and can dilute the signals passed to the destination. A dedicated process to check your website for broken links makes this step repeatable.

Consider a hypothetical post-migration scenario: your top blog URLs now resolve through three hops, from the old path to an intermediate path to the final one. The fix is not just collapsing the chain into single 301s. Internal links across the site also need updating to point straight at the final URLs, otherwise every internal click still triggers a redirect.

The working sequence:

1. Filter the crawl by response code: 3xx, 4xx, and 5xx separately.
2. Rank problem URLs by inlink count and organic traffic so you fix the most visible ones first.
3. Update internal links, including navigation, templates, and body content, to the final destination.
4. Apply one-hop 301 redirects from old URLs to the closest relevant replacement.
5. Re-crawl to confirm chains are gone and no new errors appeared.

Two mistakes recur. The first is redirecting every dead URL to the homepage, which search engines may treat as a soft 404 and which strands visitors who wanted something specific. Redirect to the closest equivalent page, or let a genuinely retired page return 404 or 410. The second is fixing the redirect rule but leaving internal links pointing to the old address.

Measure the count of 4xx and 5xx URLs and redirect chains before and after, along with the crawl error and server error trends in Search Console. Intermittent 5xx errors deserve a conversation with whoever runs your hosting, since a crawler can only report them.

4. Verify Indexation Page by Page with Search Console

A page that returns 200 is not necessarily in Google's index. Indexation is the stage where a search engine decides to store a page and make it eligible to appear in results, and the decision depends on directives, canonicals, content quality, and how the page is linked. Submitted does not mean indexed. If the concept is new to your team, start with what website indexing is before diving into reports.

Imagine, hypothetically, 150 service-area pages sitting under "Crawled - currently not indexed." The pages are technically fine, but each is the same template with only the city name swapped. Google has seen them and judged them near-duplicates. Requesting indexing won't help; consolidating them into fewer, richer pages with distinct local detail might.

To work through it:

1. Open the Page indexing report in Search Console and review each exclusion reason.
2. Export the affected URLs for the reasons that matter, such as "Crawled - currently not indexed," "Discovered - currently not indexed," and "Duplicate without user-selected canonical."
3. Match them against your master inventory to see which pages should have been indexed.
4. Spot-check one URL per template with the URL Inspection tool, comparing Google's chosen canonical with yours.
5. Fix the cause: stray noindex tags, conflicting canonicals, thin content, or weak internal linking.

Avoid using site: searches as an index count. The numbers are approximations and shift between queries. Also avoid hitting "Request indexing" repeatedly on pages with underlying problems; it doesn't override a quality or technical judgment. If you plan to automate inspection through the URL Inspection API, check the current per-property quotas first (as of 2026). When pages stall, the troubleshooting steps for website pages not getting indexed fast can help isolate the cause.

Measure indexed pages as a share of pages that should be indexed, and track the number of URLs in each exclusion category over time. A falling "crawled, not indexed" bucket is a better sign than a rising total.

5. Score Each Page for On-Page and Content Quality

Once you know a page exists, loads, and is indexed, the question becomes whether it deserves to. Scoring turns a list of thousands of URLs into a set of decisions: keep, improve, consolidate, or remove. Without it, audits produce spreadsheets that nobody acts on.

A hypothetical blog audit shows how this plays out. Of 120 posts with no clicks in 12 months, 50 overlap heavily with other posts and get merged into stronger guides with 301 redirects, 40 are worth refreshing because their topics still matter, and 30 are removed because they were thin and off-topic. The site ends up smaller and each remaining page carries more weight.

To build the scorecard:

1. Add on-page fields from the crawl: title and meta description presence and length, H1, word count, canonical, and structured data where relevant.
2. Join Search Console clicks and impressions for the last 12 months, plus backlink data from your link tool of choice.
3. Define thresholds, for example "no clicks, no backlinks, no conversions, thin content" for removal candidates.
4. Label every URL with an action and a named owner, so the work has someone attached.

The costly mistake is deleting low-traffic pages without checking backlinks, conversions, or internal linking roles. A page with few visits may hold the only strong external link to your domain, or may assist conversions further down the funnel. Redirect or improve those instead of removing them. A solid small business SEO checklist can serve as the baseline for what "meets the quality bar" means.

Measure the share of pages meeting your quality checklist and organic clicks per page group (refreshed, merged, untouched) in the months after changes. The untouched group serves as a rough control.

6. Find Orphan Pages and Fix Internal Link Depth

An orphan page is one with no internal links pointing to it. Crawlers that follow links can't reach it, so it is often discovered only through a sitemap or an old external link, and it receives little internal authority. Crawl depth, the number of clicks from the homepage needed to reach a page, matters for the same reason: deeper pages tend to be crawled less often and get fewer internal signals.

Hypothetically, you find a high-converting comparison page with zero inlinks. It earns revenue from direct and paid traffic but barely appears in organic results. After you link it from three relevant guides and a category hub, search engines can reach it through the site's structure, and you would expect crawl frequency and impressions to rise.

The process:

1. Diff your sitemap and analytics URLs against the crawl. URLs in those sources but missing from the crawl are orphan candidates.
2. List them and decide for each whether to link or retire it.
3. For pages worth keeping, add contextual links from topically related pages, using descriptive anchor text.
4. Check depth for priority pages and add hub links to pull important ones to within a few clicks of the homepage.
5. Re-crawl to confirm the pages are now reachable and shallower.

The tempting shortcut is dropping links into the footer or relying on the sitemap. Sitewide footer links carry little topical context, and a sitemap entry is not an internal link. In-content links from relevant pages tell both users and engines why the page matters.

Measure the orphan page count, the average click depth of your priority pages, and impressions for the pages you fixed, compared with the weeks before.

7. Push Fixed Pages to Search Engines and Check AI Visibility

Fixing pages only helps once engines notice the changes. IndexNow is an open protocol that lets you notify participating search engines when URLs are added, updated, or deleted, rather than waiting for the next crawl. Supported engines have changed over time, so confirm the current list on the IndexNow site (as of 2026). Google has not historically been a participant, so keep your sitemap accurate and use Search Console for it. For a broader look at the options, see these instant website indexing methods.

The second half of this step is checking whether any of it translates into visibility in AI answers. Indexed does not mean cited. Hypothetically, after fixing and re-indexing a set of comparison pages, a brand tracks its core prompts across ChatGPT, Claude, and Perplexity and finds it appears for some but is omitted for others. Those omissions become a content brief: the topics where competitors are named and you are not. It also helps to optimize your website for Claude AI specifically, since each model surfaces sources differently.

To run this:

1. Update the sitemap with accurate lastmod dates that change only when content meaningfully changes.
2. Send changed URLs through IndexNow. Sight AI's website indexing tools automate this alongside sitemap updates.
3. Set up prompt tracking in an AI visibility tool such as Sight AI, covering the questions your buyers actually ask.
4. Review mentions, sentiment, and which prompts omit you, and feed gaps into your content plan.
5. Schedule recurring audits, monthly for active sites and quarterly for stable ones.

Two mistakes to avoid: treating indexation as the finish line, and pinging unchanged URLs repeatedly, which adds noise and can erode trust in your lastmod signals.

Measure the time from update to indexation, and track AI mention rate, sentiment, and prompt coverage over time. Since AI answers vary between runs, look at trends across many prompts rather than single responses.

Where to Start, and How to Keep the Audit Running

Begin with the master inventory and the crawl. Until you know what exists, every other fix is partial. Next, clear status code errors and indexation blockers, since those stop pages from being found at all. Quality scoring, orphan repair, and AI visibility monitoring come after, and they work best as a recurring routine rather than a one-off project.

If you'd rather not stitch the pipeline together by hand, Sight AI combines AI visibility tracking, website indexing with IndexNow, and content generation in one platform. Stop guessing how AI models like ChatGPT and Claude talk about your brand: get visibility into every mention, track content opportunities, and automate your path to organic traffic growth. Start tracking your AI visibility today and see exactly where your brand appears across top AI platforms.