How to run a technical SEO audit step by step

Most sites that plateau in Google aren’t losing because the content is thin. They’re losing because Googlebot can’t crawl the pages efficiently, the canonical tags point at the wrong URL, or the Core Web Vitals numbers are bad enough to hold rankings back. Content teams keep writing while the technical layer quietly caps what any of that writing can do.

This is for anyone running a content site, agency account, or affiliate property past the point where “check Search Console once in a while” is enough, roughly 50+ pages, where you can’t eyeball every URL by hand. I run this exact process on theseodesk.com and on the other sites I operate. It’s the workflow I actually use, not a theoretical checklist copied from a certification course.

By the end you’ll have a prioritized list of technical fixes ranked by how many pages each one touches and how much it likely costs in crawl budget or rankings, not a forty-tab crawl export nobody opens again. If you want the surrounding context on crawl budget, indexing, and content strategy, the rest of the write-ups live on the blog index.

what you need

• Screaming Frog SEO Spider. The free version crawls up to 500 URLs, enough for a small site; the paid license (roughly £200/year, check current pricing at screamingfrog.co.uk) removes the cap and unlocks JavaScript rendering, custom extraction, and scheduled crawls
• a verified Google Search Console property for the domain, free
• PageSpeed Insights (web.dev) or the Chrome UX Report for real-world Core Web Vitals field data, free
• raw server access logs, or Cloudflare logs if the site sits behind Cloudflare, pulled for at least the last 30 days
• a terminal, curl is enough for manual header and robots.txt checks
• a spreadsheet (Google Sheets is fine) to turn the crawl export into a prioritized fix list
• a staging environment or at minimum a git branch, so fixes get tested before they hit production
• optional: Ahrefs or Semrush for backlink and historical ranking context, both run roughly $99-129/month on entry tiers

step by step

1. crawl the site with Screaming Frog

Point Screaming Frog at the root domain in spider mode. If the site renders content with JavaScript, switch rendering to “JavaScript” under Configuration > Spider > Rendering first, otherwise you’ll crawl an empty shell and think half your content is missing. Let the crawl finish before you touch any settings.

ScreamingFrogSEOSpiderCli.exe --crawl https://example.com --headless --save-crawl --output-folder "C:\audits\example"

Expected output: a full crawl summary with status codes, response times, word counts, and internal link counts per URL.

If it breaks: on a large site Screaming Frog defaults to in-memory storage and will run out of RAM mid-crawl. Switch to database storage mode under Configuration > System > Storage before you start, and raise the Java heap allocation if it’s still crashing.

2. sort by status code and indexability

Use the Response Codes and Indexability tabs to pull every 4xx, every 5xx, and every redirect chain longer than one hop.

Expected output: a clean list of broken internal links and redirect chains to flatten to a single hop.

If it breaks: some WAFs and bot-protection layers, Cloudflare included, block the Screaming Frog user agent by default and return a wall of 403s that aren’t real. Set a custom user agent and drop crawl speed to one or two threads before you conclude the whole site is broken.

3. audit robots.txt and the XML sitemap

Pull robots.txt directly and check it against Google’s robots.txt documentation, which follows the Robots Exclusion Protocol, RFC 9309. Then open the sitemap referenced in Search Console and confirm every URL in it actually returns 200.

curl -s https://example.com/robots.txt

Expected output: robots.txt doesn’t block the CSS or JS directories Google needs to render the page, and the sitemap contains only canonical, indexable URLs.

If it breaks: an old CMS plugin generated a sitemap that still references URLs that now 404 or redirect. Regenerate it from source, resubmit in Search Console, and confirm the resubmission was accepted.

4. check canonicals and duplicate content

Pull the Canonicals tab in Screaming Frog, look for pages missing a self-referencing canonical, and confirm http/https and www/non-www variants aren’t both live and indexable at once.

Expected output: every indexable page has exactly one canonical, pointing at itself unless it’s an intentional duplicate.

If it breaks: canonicals point at a staging domain or an old URL structure left over from a migration. That’s almost always a template bug, not a per-page one, so fixing the template clears it site-wide.

5. review Core Web Vitals

Run PageSpeed Insights against one URL per template (homepage, category or listing page, article page), then check the field data tab, not just the lab score. Google’s own Core Web Vitals documentation explains the difference and the current thresholds.

Expected output: LCP under 2.5 seconds, INP under 200 milliseconds, CLS under 0.1 on field data, per template.

If it breaks: the lab score is terrible but field data is fine, or the reverse. Trust the field (CrUX) data over the lab run, the lab score is one simulated pass and can be noisy.

6. validate structured data

Run Search Console’s Enhancements reports against each template and check the markup against Google’s structured data guidelines.

Expected output: no errors, ideally no warnings, on whatever schema type applies (Article, Product, FAQ, and so on).

If it breaks: schema validates clean but no rich result shows up in the SERP. That’s normal, valid markup makes a page eligible, it doesn’t guarantee Google decides to display it.

7. pull and read server logs

Filter the last 30 days of raw access logs for Googlebot’s user agent and check which URLs actually get crawled, how often, and which ones never get touched at all.

grep "Googlebot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Expected output: a picture of how crawl budget is actually being spent, plus a list of orphaned or rarely-crawled pages.

If it breaks: some log lines claim to be Googlebot but aren’t, spoofed user agents are common. A reverse DNS lookup confirms the real thing. Worth flagging too, logs increasingly mix in AI crawlers like GPTBot and ClaudeBot alongside search bots. I keep a running breakdown of who’s currently crawling for AI training data over at AI Tool Gazette if you need to separate those from your actual search traffic sources.

8. check mobile usability

Check Search Console’s Mobile Usability report, and manually verify the viewport meta tag and tap target spacing on each template.

Expected output: zero mobile usability errors reported.

If it breaks: the report looks empty or stale because Search Console hasn’t recrawled recently. Use the URL Inspection tool on individual pages to force a fresh check instead of waiting.

9. build the prioritized fix list

Score every issue found by pages affected multiplied by estimated impact, divided by engineering effort. Put it in a spreadsheet, sort descending.

Expected output: a ranked top 10 to 15 list, ready to hand to a developer or fix yourself in one sitting.

If it breaks: the list runs to 200 rows and nobody knows where to start. Cap it, ship the top five, re-crawl, repeat. An audit that never turns into a shipped fix wasn’t worth the time it took to run.

common pitfalls

• treating the crawl export itself as the deliverable. a spreadsheet nobody prioritizes doesn’t fix anything
• fixing the easy stuff first, like missing alt text, while an indexation blocker such as a stray noindex tag or a robots.txt disallow sits untouched
• never re-crawling after a fix ships, so a canonical change that quietly broke pagination goes unnoticed for a month
• trusting Search Console’s sampled data as the full picture instead of cross-checking with raw logs. Search Console will happily show a page as indexed when the logs show Googlebot hasn’t visited it in 60 days
• crawling a JavaScript-heavy site with rendering turned off and concluding content is “missing” when Googlebot would actually see it fine with rendering on

scaling this

At 10x, a few hundred to a couple thousand URLs, the free Screaming Frog tier stops being enough. Get the paid license, move to database storage mode, and start running the crawl on a monthly schedule instead of ad hoc.

At 100x, tens of thousands of URLs, manually reviewing individual URLs stops being feasible. Work in templates and URL patterns instead of one-off pages. Log file analysis stops being a spot check and becomes the primary signal, since Search Console’s sampling gets thinner as the site grows. Tools like Sitebulb or Botify start to earn their cost at this scale.

At 1000x and up, hundreds of thousands to millions of URLs, crawl budget management becomes its own job, not a step in an audit. This is where a dedicated engineering ticket queue for technical SEO makes sense, log analysis runs through something like BigQuery instead of grep on a text file, and the “audit” stops being a one-off event and turns into continuous monitoring with alerting on crawl errors and indexation drops.

where to go next

• if the audit turns up thin or duplicate content alongside the technical issues, how to build a topical authority content plan covers the content side of the same problem
• pair this with how to do keyword research in 2026 before deciding what new pages to build once the technical debt is cleared
• if you’re running this audit because rankings already dropped, how to recover from a google core update is the more targeted version of this same process

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-18.