We Audited Our Clients SEO With AI Agents — Heres What We Found

Last week I plugged Claude Code into a Screaming Frog MCP server, decrypted Google Search Console credentials from our own database, and ran a full technical SEO audit on two startup websites — without opening a browser.

Both sites had critical indexing failures. One site had zero pages indexable by Google. The other was getting 1,179 impressions but only 12 clicks over three months.

None of this was visible from the surface. The sites loaded fine. They looked professional. They were just invisible to search engines — and to every AI model pulling from search results.

Here's exactly what we found, how we fixed it, and why this matters more than ever now that AI engines are the new front page of the internet.

---

The Setup: AI Agents as SEO Auditors

At Griot, we install AI agents for marketing teams. One of the services we've been building is automated SEO/AEO monitoring — the idea that an AI agent should be continuously watching your search presence the same way you'd watch a deploy pipeline.

The stack for this audit:

• Claude Code as the orchestration layer
• Screaming Frog SEO Spider with its new MCP server (v24.0) — the crawler becomes a tool that Claude can control via JSON-RPC over HTTP
• Google Search Console API — accessed by decrypting OAuth tokens stored in our Supabase database
• The actual codebase — Claude reads the Next.js source, identifies the root cause, and pushes fixes directly

No dashboards. No manual exports. No copy-pasting CSV data into ChatGPT. The agent talks to the crawler, reads the code, diagnoses the issue, and writes the fix.

---

Site 1: A Poker Training App (4 Pages, Zero Indexable)

What Screaming Frog Found

The crawl returned 26 URLs. On the surface, the numbers looked okay — all pages returned 200 status codes. But dig one layer deeper:

• 14 out of 26 URLs (54%) were blocked by robots.txt
• 19 out of 24 internal URLs (79%) were non-indexable
• All 4 HTML pages were marked "Canonicalised" — pointing to a URL that 307-redirected

Every single page on the site was invisible to Google.

The Root Causes

Problem 1: robots.txt was blocking all JavaScript and CSS.

Disallow: /_next/

One line. This told Google "don't crawl any of my Next.js static assets." Google could see the HTML shell but couldn't render the actual page content. It's like handing someone a book with all the pages glued together.

Problem 2: Every page's canonical pointed to the wrong domain.

The site lived at www.preflopwizard.app, but the Next.js layout had:

metadataBase: new URL('https://preflopwizard.app'), // no www
alternates: { canonical: '/' },

Every page — homepage, privacy, terms, support — told Google: "the real version of me lives at https://preflopwizard.app/." That URL 307-redirected back to the www version. Google saw a canonical pointing to a redirect, marked every page as "Alternate page with proper canonical tag," and refused to index any of them.

Problem 3: Every page shared the same <title> tag.

"Preflop Wizard - Master Preflop Poker with AI Training" appeared on all four pages. Google had no way to differentiate them in search results.

The Fix

Claude Code read the source files, identified all three issues, wrote the fixes, and committed:

1. Removed Disallow: /_next/ from robots.txt
2. Changed metadataBase to https://www.preflopwizard.app
3. Added per-page canonicals (/privacy points to /privacy, not /)
4. Added unique titles and descriptions to each page
5. Removed duplicate <meta> tags that were declared twice
6. Removed a fake aggregateRating from structured data (4.8 stars, 2,500 reviews — numbers that weren't real)
7. Added security headers (X-Frame-Options, CSP, Referrer-Policy)

Re-crawl results:

• Blocked by robots.txt: 14 → 0
• Internal indexable: 5 (21%) → 23 (92%)
• Non-indexable pages: 19 → 0
• All canonicals self-referencing: 0% → 100%

The site went from completely invisible to fully indexable in one commit.

---

Site 2: A B2B SaaS With 13 Blog Posts and 12 Clicks

What Google Search Console Showed

This site was technically healthier — no robots.txt issues, pages were indexable. But the GSC data told a different story:

Metric	Value
Total impressions (3 months)	1,179
Total clicks	12
Average CTR	1.0%
Average position	3.5

Position 3.5 with a 1% CTR. That's catastrophically low — you'd expect 15%+ CTR at that position. And despite having 13 published blog posts targeting competitive keywords, the site was only ranking for branded queries. "Griot AI." "Is Griot good for ghostwriting agencies." Not a single non-branded keyword was driving traffic.

What Screaming Frog Found

The crawler revealed issues the GSC data couldn't:

• A 404 hitting 24 pages — Cloudflare's email obfuscation was rewriting mailto: links into /cdn-cgi/l/email-protection, which returned 404 on Vercel
• Dead LinkedIn company URL in structured data and footer (22 inlinks to a 404)
• Missing canonical on /waitlist — the only page without one
• Zero security headers across 98% of URLs
• 4 low-content pages under 200 words

The Fixes

Again, Claude Code read the source, diagnosed, and pushed:

1. Replaced all mailto: links with a booking link (eliminated the Cloudflare 404)
2. Fixed the LinkedIn company URL
3. Added security headers via next.config.ts
4. Added canonical + metadata to the waitlist page
5. Fixed structured data (removed blocked URL from contactPoint)

---

Why This Matters for AEO

Here's the thing most people miss: AEO is 90% dependent on SEO.

When ChatGPT, Perplexity, or Claude answers a question about your industry, they're pulling from indexed web content. If Google can't render your pages — because your robots.txt blocks your JavaScript — then AI models can't see your content either.

The poker app had zero pages indexable. That means:

• Google won't rank it
• AI models won't cite it
• It effectively doesn't exist on the internet

We've been building an automated monitoring system that catches these issues continuously — a VPS running cron jobs that pulls Google Search Console and PostHog data, spawns a Claude Code instance to analyze it, and pushes optimizations directly to the codebase. Style guides keep the changes consistent. The system runs at 6 AM and 6 PM, every day.

This is what we mean when we say "AI agents for marketing." Not a chatbot that writes your LinkedIn posts. An autonomous system that watches your search presence, diagnoses technical failures, and fixes them before you know they exist.

---

The Playbook: How to Run This Audit Yourself

Step 1: Connect Screaming Frog's MCP Server

Screaming Frog v24.0 ships with a built-in MCP server. When the app is open, it runs on localhost:11435. Any AI tool with shell access can control it:

# Initialize a session
curl -X POST http://localhost:11435/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize",
       "params":{"protocolVersion":"2024-11-05",
       "clientInfo":{"name":"your-tool","version":"1.0.0"}}}'

Step 2: Run the Crawl and Pull Issues

# Start a crawl
sf_call '{"method":"tools/call",
  "params":{"name":"sf_crawl",
  "arguments":{"crawl_url":"https://yoursite.com/"}}}'

# Get the issues report
sf_call '{"method":"tools/call",
  "params":{"name":"sf_generate_report",
  "arguments":{"category":"Issues Overview"}}}'

Step 3: Check What Google Actually Sees

Pull your GSC data. Look for the gap between impressions and clicks. If you're getting impressions at position 3-5 but your CTR is under 5%, your titles and descriptions are failing. If you're getting zero impressions on pages you've published, check indexability.

Step 4: Fix the Code, Not the Symptoms

The most common "SEO fixes" are band-aids: manually submitting URLs for re-indexing, tweaking meta descriptions in a CMS, adding keywords to headers. The real fixes are almost always in the code:

• A wrong metadataBase in your Next.js layout
• A Disallow rule that blocks your static assets
• A Cloudflare setting that rewrites your email links into 404s
• Client-side rendering that prevents Google from seeing your content

---

What We're Building

At Griot, we started as a context layer for ghostwriting agencies — solving the problem of transferring client voice across writers and AI tools. But the same infrastructure that captures and structures brand data also captures and structures search data.

We're now installing AI agents that do what we did manually in this audit — continuously, autonomously, across every client in an agency's portfolio. The agent watches Google Search Console, runs periodic crawls, cross-references with the codebase, and flags (or fixes) issues before they cost you rankings.

If you're an agency or a VC-backed startup that wants this running on your sites, book a call. We deploy in under a week.

---

Austin Kennedy is the founder of Griot, an AI agent platform for marketing teams. Previously, he built automated SEO systems for YC and Neo-backed startups including Origami and Northlight.