SEO update at The Reasonable Adjustment

17 October 2025 by Kieron JH

The Reasonable Adjustment SEO update is live. We made targeted changes to our Cloudflare Worker (which is now over 1300 lines of code long) and our robots.txt so responsible crawlers can find our work fast while junk traffic gets out of the way. The goal is simple. Grow visibility. Speed up indexing. Stop divvys filling the inbox with spray and pray SEO pitches.

Why we did this

• Increase organic reach by sending clear signals to search engines and reputable AI crawlers.
• Speed up indexing and remove duplicate URLs that waste crawl budget.
• Keep the site stable and fast for readers during traffic spikes.
• Cut bandwidth wasted on spammy bots and low value scrapers.

What changed in our Cloudflare Worker

• Good bot fast lane for Googlebot, GoogleOther, Bingbot, DuckDuckBot, Applebot, GPTBot, OAI SearchBot, CCBot and PerplexityBot. They receive clean HTML, index and follow signals, and sensible caching at the edge.
• Canonical redirects remove junk parameters like utm, gclid, fbclid, ref, mc_cid and mc_eid. This consolidates link equity and avoids duplicate indexing.
• Automatic SEO hygiene for crawlers. If a page is missing a canonical tag we add one. We also add JSON LD NewsArticle schema when needed.
• 404 hygiene returns noindex for bots with a short cache so dead URLs drop out of search quickly.
• Performance and stability are protected with edge caching, HTML streaming, and light throttles so humans always get a fast page.

What changed in our robots.txt

We now serve a simple static robots.txt with explicit rules.

• Allowed: Googlebot, GoogleOther, Bingbot, DuckDuckBot, GPTBot, OAI SearchBot, CCBot, Applebot. Full access plus a clear sitemap pointer.
• Blocked: AhrefsBot, SEMrushBot, MJ12bot, DataForSeoBot, DotBot, SEOkicks Robot, SiteAuditBot, BLEXBot, crawler4j. These are high noise crawlers with low value to readers.

The benefits we expect to see

• Faster indexing for new posts. Fresh articles should be picked up within hours rather than days.
• Cleaner Search Console with fewer duplicates and fewer crawled not indexed entries.
• Stable impressions and clicks as crawlers focus on the content that matters.
• Inclusion in responsible AI summaries where sources are cited. More paths for readers to reach our work.
• Lower noise and reduced bandwidth waste so pages stay quick for humans.

Timeline

Early crawl behaviour improvements should appear within a few weeks. Our aim is sustained gains in crawl efficiency, faster appearance of new posts in search, and steady growth in impressions by the end of the year.

What this means for the organisations we report on

• Faster discovery. Statements, responses, and linked documents will be found and refreshed more quickly.
• Greater accountability. Updates and corrections propagate faster, which keeps the public record clear.
• Clearer citations. Canonical URLs and structured data help preserve a clean trail of what was said and when.

What readers can expect

• Quicker page loads and more reliable previews on social platforms.
• Stable links that are easy to share and cite.
• No gimmicks. Fast pages, accurate reporting, and a tidy sitemap.

If you are here to sell miracle SEO, save the pitch.

Questions or corrections? Email [email protected].