BBA Index Coverage Audit

1. Do not trust the supplied coverage export

Consequence: 943 of the 999 rows are /broker/ URLs (clean + .html). Read naively it suggests "943 broker pages indexed", which is pure alphabetical-truncation artefact.

Chart.csv shows affected pages jumping 183 → 8,995 on 2026-09-19 and 8,995 → 12,373 on 2026-09-22 — that is the old bulk-sitemap crawl wave landing, and it created the current mess.

2. Declared vs actually indexed, by group

Index rate per sitemap

All 5 URLs in sitemap-brokers-verified.xml are unindexed — the cleanest single proof of the inversion.

2A. 🚨 The leak — how 10,409 files became 20,818 indexable URLs

Tier-2 bucket inventory — gs://bestbrokersaustralia-static/

All 10,409 broker/ objects are .html.

The local tree is blind to 98.7% of them

Empirical proof: 120 of 120 randomly sampled indexed broker URLs were absent from the local tree, yet 63% returned 200 + index, follow on the live site. They are served from GCS.

The mechanism — two lines in docs-core/_worker.js

// line ~259-262 — extensionless -> .html -> GCS
if (!pathname.includes('.') && pathname !== '/') {
  pathname += '.html';
}

// line ~181 — .html -> clean 301, with NO X-Robots-Tag
if (url.pathname.endsWith('.html') && ...) {
  return Response.redirect(url.origin + cleanPath + url.search, 301);
}

// line ~160-178 — noindex covers only 8 hardcoded routes
const NOINDEX_INTERNAL_ROUTES = new Set([...]);   // /broker/* NOT included
const applyNoindexHeader = (pathname, res) => {
  if (!NOINDEX_INTERNAL_ROUTES.has(pathname)) return res;   // falls straight through
};

So every GCS file is indexable two ways:

10,409 × 2 = 20,818 indexable broker URLs.

Leak progress

Chart.csv records affected pages climbing 183 → 8,995 → 12,373 between 2026-09-19 and 2026-09-22. That curve has not plateaued — it is this backlog draining.

3. What is indexed but should NOT be

Plus 4,097 /broker/ dossier URLs, where only 5 are declared. The legacy satellite dirs above are absent from the local tree and return 404, yet remain in the index.

Declared groups still not indexed (excluding /suburbs)

4. How much of /broker/ is dead?

Random sample, n=120 indexed broker URLs (curl -L, follow redirects):

So 37% (95% CI 29–46%) are dead 404 pages. Projected across 4,097 indexed broker URLs: ~1,502 dead 404s holding index slots and ~2,595 live dossier pages that should be noindexed.

Duplicate structure: 4,097 broker URLs → 3,323 distinct slugs → 774 slugs indexed twice (clean + .html), 961 .html-only stragglers, 1,735 .html URLs total.

5. URL Inspection sample (n=50)

6. Secondary defects

Canonical mismatches in sitemap

The mechanical cause of the duplicate bloat

$ curl -sI https://bestbrokersaustralia.org/broker/ryker-capital-ingleburn.html
HTTP/2 301
location: https://bestbrokersaustralia.org/broker/ryker-capital-ingleburn
# <- no x-robots-tag header

A bare 301 does not deindex the source URL. Google keeps the .html URL as a separate index entry indefinitely. This one omission accounts for the 1,735 duplicate broker URLs plus 5 root-level .html twins.

• Duplicate <title> sets (5): Kingston ×3, Brighton ×2, Manly ×2, Paddington ×2, one Brisbane broker ×2.
• /suburbs/index returns 308 — the only non-200 among the 305 declared URLs.
• 0 of 305 declared URLs contain noindex, and 304/305 return 200. The 272 unindexed pages are not blocked — Google is declining to index them. This rules out a robots/meta mistake and confirms a crawl-priority + site-quality problem.
• Param leak /?q=Hobart; www. host duplicates.

7. Root cause

Two competing indexes exist at once: an old index of ~3,300 broker dossiers plus 6 legacy satellite directories from the bulk-sitemap era, and the Phase-1 index where only 33 of 305 have landed. Google burns crawl budget on 404s and .html duplicates while 272 Phase-1 URLs sit in "Discovered – currently not indexed". On a domain where ~37% of URLs are dead and ~42% are duplicates, overall site-quality assessment is dragged down and the good pages are suppressed.

8. Remediation plan — REVISED (stop the leak first)

Verify step 0 landed: curl -sI https://bestbrokersaustralia.org/broker/<any-slug> and confirm x-robots-tag: noindex appears.

8b. Original table (superseded, kept for reference)

9. Independent verification checklist

1. Highest priority: confirm gs://bestbrokersaustralia-static/ holds 10,409 objects under broker/ while v3psycho/broker/ holds 132. Read-only devstorage.read_only scope; script in audit/gcs_evidence.md.
2. Confirm _worker.js line ~260 appends .html to extensionless /broker/* and serves it from GCS — each file indexable under two URLs.
3. Confirm applyNoindexHeader (line ~169) covers only 8 hardcoded routes, so /broker/* emits no x-robots-tag.
4. If you have GSC UI access, screenshot the Pages report breakdown — that is the one number this audit cannot verify independently, and the most valuable missing input. Treat ~9,000 as authoritative for "indexed"; 4,176 is a reproducible API floor.
5. Confirm sitemap-brokers-verified.xml holds exactly 5 URLs, 0 indexed.
6. Confirm /suburbs declares 230 but only ~28 appear in Google (6%).
7. Confirm /mortgage-brokers/barton/ returns 404 yet still draws GSC impressions — proves stale-index retention.
8. Confirm a 301 from any .html URL carries no x-robots-tag — the mechanical cause of the duplicate bloat.
9. Re-run the GSC Search Analytics query below and confirm ≥4,000 /broker/ URLs are returned.

from google.oauth2 import service_account
from googleapiclient.discovery import build
creds = service_account.Credentials.from_service_account_file(
    'service_account.json',
    scopes=['https://www.googleapis.com/auth/webmasters.readonly'])
sc = build('searchconsole', 'v1', credentials=creds)
sc.searchanalytics().query(
    siteUrl='sc-domain:bestbrokersaustralia.org',
    body={'startDate': '2026-06-01', 'endDate': '2026-10-07',
          'dimensions': ['page'], 'rowLimit': 25000, 'dataState': 'all'}
).execute()

10. Methodology caveats — read before disputing

• "Indexed" = the URL returned ≥1 impression in GSC Search Analytics. A page can be indexed with zero impressions and would be missed, so 4,176 is a lower bound and "33 of 305" is a floor. The error direction strengthens the conclusion.
• The 50-URL URL Inspection sample is stratified even-spacing, not random. The 120-URL broker probe is random (random.seed(11)).
• The GSC Index Coverage / Pages report is not exposed by any public API. Verbatim coverage labels per URL require the URL Inspection API, which is what §5 uses. Anyone claiming to have pulled the full coverage report via API is mistaken.
• Raw evidence for every table above: audit/evidence_bundle.json (319 KB).