Automation testing

The do-it runbook for the automated QA suite. It enforces the agency output standard — the design team’s UAT/QA checklist — automatically, across the three platforms. Pairs with Auto-input, the QA architecture reference, and feeds the Time savings & ROI.

Repo: supportsn/sn-automation-testing (the qa/ kit + .github/workflows/ + qa-dashboard/ + uptime-worker/). Dashboard: https://sn-automation-testing.pages.dev.

The standard → the engines

The checklist (qa/standards/design-checklist.json, ~60 items) is the single source of truth. Every item is tagged with the engine that automates it:

Engine	Tech / host	Cost	Covers
SEO (tier1)	cheerio on Playwright-rendered HTML (run-portfolio.ts)	free GitHub runner	title/meta/headings/alt, OG/Twitter, JSON-LD, canonical, mojibake, zero-CDN — every sitemap page
design	Playwright + axe on the real DOM (design-auditor.ts)	free GitHub runner	~23 deterministic rules: broken/distorted images, horizontal overflow / text cutoff, font + H2 + line-height consistency, contrast (WCAG AA), nav present, link/CTA HTTP status, button/table overflow, inline-link underline, duplicate content, FAQ accordion, H3-on-mobile, phone/zip present
visual	Playwright full-page screenshot + pixel-diff (run-design.ts)	free runner + R2 storage	responsive drift @ 375 / 768 / 1440 vs approved baselines, with per-frame provenance (baseline date + drift-first-seen, linked to runs)
agentic	Claude headless + MCP + vision (qa-verify)	Claude plan quota (~$1 equiv / site, ~5–7 min)	content-is-THIS-client, cross-service leakage, spelling, image↔service, correct logo, CTA wording, SOP order
uptime	edge Worker (15 min) + GitHub Actions (6 h)	free	Worker: HTTP + latency, live; GHA: TLS cert expiry + DNS + content keyword

One Playwright pass produces both design and visual. Sites are discovered from the sitemap (/sitemap.xml, *_index variants, robots.txt Sitemap:) — not just the homepage.

1. SEO sweep (whole fleet, scheduled daily)

Runs daily via the catch-up gate (see §7); dispatch manually to scope or to lift the page cap:

gh workflow run qa-portfolio.yml -f sites="<id>" -f max_routes=0   # 0 = all sitemap pages
# local:
cd qa && QA_SITES=<id> QA_MAX_ROUTES=0 npm run qa:portfolio

2. Design + visual (Playwright — manual / on-demand)

gh workflow run qa-design.yml -f sites="<id>" -f max_routes=6
# single page:
gh workflow run qa-design.yml -f sites="<id>" -f route="/services/"
# extra pages beyond the sitemap:
gh workflow run qa-design.yml -f sites="<id>" -f extra_routes="/landing-a,/landing-b"
# APPROVE current screenshots as the new baselines (review diffs first!):
gh workflow run qa-design.yml -f sites="<id>" -f update_baselines=true

Screenshots are full-page JPEG (sharp, ~1 MB each) at all three breakpoints, with per-breakpoint device emulation (mobile/tablet iOS UA + reload) so UA-sniffing builders like Duda serve the real device layout.

Where visual data lives (no rebuild)

Visual artifacts do not live in git. Each run uploads screenshots, the verdict (latest.json), and the canonical baselines to the qa-visual R2 bucket; the dashboard reads them live through the edge worker — new results appear without waiting for a Pages rebuild. Only latest + baseline are kept (no history).

Baselines: approve / reject

When a page drifts, the site detail page shows Baseline / New thumbnails → enlarge modal with Side-by-side / Slider / Diff and the provenance line (baseline captured date (run ↗) · drift since date (run ↗) — the page changed in that window). Two ways to accept a change:

• Update baseline (single frame) or Update all baselines (site) on the dashboard — the worker copies the already-captured “New” over the baseline in R2. Instant, no re-run.
• update_baselines=true on a fresh qa-design.yml run — re-captures and approves.

Leave the drift unapproved to keep it flagged.

3. Agentic pass (Tier 2 — qa-verify)

For the judgment no rule can make. Claude profiles the site over the SEO Navigator MCP, renders each route, reads design-checklist.json, judges the agentic items against the content oracle (MCP get_post/get_seo_meta, drive_get_baseline, the client Content Workbook via read_sheet), and writes a verdict.json. Report-only — never mutates the site.

gh workflow run qa-agentic.yml -f site_id=<id> -f platform=duda

Measured cost: ~$0.7–1.2 API-equivalent per site (22–48 turns, 4.6–6.6 min). Five parallel runs verified without rate-limiting. Run it selectively (pre-launch, content spot-checks) — not on a schedule.

4. Uptime — two complementary monitors

	Edge worker	GitHub Actions
Cadence	every 15 min	every 6 h
Checks	HTTP status + latency	HTTP + TLS expiry + DNS + keyword
Shows up	Sites grid Uptime column (“edge live” badge), /incidents	each site’s Uptime tab (+ run link, history)
Alerting	Slack on transitions (down / recovered) + 23:55 ICT daily summary	Slack for DOWN / expiring certs

The worker also keeps 30-day per-site aggregates that power the Uptime 30d column (SLO % + latency sparkline) and the noisy flag (≥3 slow/down events in 7 days). Manual worker endpoints: /run (check now), /daily-report (send the summary now), / (live status page).

5. Reusable workflow (project repos)

Astro/WordPress project repos don’t copy QA code — they call the reusable workflow, which runs qa/src/cli.ts against a built/preview/live URL:

tsx src/cli.ts --platform astro --site <id> --base <url> [--spec docs/<id>-pages.json]

--spec runs content-fidelity: every section’s verbatim text from the Figma / Content-Workbook spec must appear in the rendered page. This is why the Content tab on the dashboard says “runs from the project’s PR workflow” — it needs the project’s spec, so it can’t be launched fleet-wide.

6. Per-platform coverage

Platform	Source?	Coverage
Astro	full	SEO + design + visual + agentic + content-fidelity + unit tests + preview-deploy (MCP). Tested in-repo & on PR.
WordPress	theme/plugin	SEO + design + visual + agentic + MCP oracle (RankMath get_seo_meta, get_post) against the live/staging URL.
Duda	none	SEO + design + visual + agentic against the live URL — the primary safety net. No unit tests possible.

7. When does each suite run

Cadence is set on the dashboard Schedule page (off / hourly / daily / weekly, global + per-site overrides). The gate uses catch-up semantics: a site is due if it hasn’t been checked since the current cadence slot opened — so a delayed GitHub cron can’t silently skip the daily run.

Trigger	Suite	Notes
Worker cron, 15 min	edge uptime	live status, Slack transitions, daily summary
Cron every 6 h + gate (qa-uptime.yml)	GHA uptime	TLS/DNS; single merged job (~1 billed min)
Hourly cron + gate (qa-portfolio.yml)	SEO sweep	default preset: daily
Manual (qa-design.yml)	design + visual	scope by sites / route; update_baselines approves
Manual (qa-agentic.yml)	agentic	per site; uses Claude quota
PR / push (reusable)	static + content-fidelity	per-repo gate for Astro/WP source

Severity gates: blocker/critical = fix before ship; major = fix before client ship; minor = follow-up. The suite is report-only by default.

8. The dashboard

https://sn-automation-testing.pages.dev — free, static, no server. Dark mode on every page (moon toggle, remembers your choice).

• Sites — one row per site (AG Grid: sort/resize): aggregate status, per-dimension health, live Uptime pill + Uptime 30d SLO + latency sparkline + noisy flag, URL, findings, last run. Filters: status / check / platform (the check filter scopes the status filter — “Uptime + Failing” = sites whose uptime is down).
• Runs — one row per committed run, each linked to its GitHub Actions run.
• Incidents — live down/recovered event log + day-by-day fleet record + 30-day fleet uptime, straight from the worker’s KV.
• Schedule — per-check cadence, global + per-site overrides; Save applies on the scheduler’s next pass.
• Site detail — one tab per check (Run-now with a confirm modal that lists exactly what will run and which pages, plus extra-pages input for design/visual), run history, findings that open a detail modal naming the exact element/image/link at fault, and the visual comparison with baseline promotion.

9. Measure the real time

The ROI in Time savings & ROI starts as estimates — replace with the CI job duration (GitHub Actions run time) for 1 Astro + 1 Duda/WP pilot, then swap the estimates for measured averages after ≥ 3 templates.

What’s next

• QA architecture — how the system is built (hosts, data stores, data flow, cost budgets, secrets).
• Auto-input — the build-side counterpart.
• Time savings & ROI — what this saves per template.