SEO experiment preflight record

SEO experiment preflight record

An SEO experiment preflight record states what will change, why the change might affect a measured outcome, and what evidence will support the final decision. Write it before anyone sees the result. A record created afterward can quietly fit the hypothesis, date range, or success measure to the movement that already occurred.

Give the experiment a stable identifier and accountable owner. Record the canonical routes, page type, audience, query set, country, device, planned start, review dates, intended end condition, implementation revision, and every team allowed to change the same routes during the test.

Separate the experiment from routine maintenance. An urgent factual correction, broken link, accessibility repair, or security fix should not wait for a measurement window, but its timing and affected pages belong in the record because it may change how the result is interpreted.

Define one change and its expected mechanism

Describe the treatment in terms another editor can reproduce. Replace vague labels such as improve the page with the exact title, opening, internal link, template component, structured field, or content section that will differ. Save the control state and treatment state before release.

Write the expected mechanism as a testable chain. A more descriptive title may change how the result communicates the page's task, which may affect visits for a defined query group. That statement is narrower and easier to challenge than a promise that better copy will increase rankings.

List the evidence that would contradict the mechanism. If the title changes but the search appearance does not, or if visits move while impressions and position remain materially different across the same period, the original explanation may not fit.

Freeze the population and baseline

Name every route in the control and treatment sets, including exclusions. Record why those pages are comparable and note differences in age, template, topic, traffic level, backlink history, index state, and commercial intent that could make one group behave differently.

Preserve the baseline query and page data with the exact filters used to retrieve it. Include the date range, comparison window, country, device, search type, query rules, page rules, and export time. A screenshot without filters is difficult to reproduce and can hide a changed population.

Check the baseline for missing days, recent migrations, indexing problems, tracking changes, or unusual demand. Do not invent a clean history by removing inconvenient periods after the outcome is visible; document the problem and decide before launch whether the test can still answer its question.

Name outcomes, guardrails, and decision rules

Choose one primary outcome that matches the expected mechanism. Supporting measures can help explain the result, but they should not take over as the success measure when the primary outcome is flat. Record how each measure is calculated and which source owns it.

Add guardrails for harms the primary outcome could miss. Depending on the change, these may cover indexing, canonical ownership, page availability, conversions, accessibility, speed, engagement, or another reader consequence. State what requires rollback or investigation.

Define the decision threshold, minimum observation condition, and stop rule before launch. Include the actions for a positive, negative, mixed, and inconclusive result. Do not extend the window only because the latest reading is close to the preferred answer.

Control competing releases and search behavior

List planned releases that can touch the routes, template, navigation, internal links, metadata, rendering, or measurement source. Name the owner who can hold or document each change. Unexpected work should enter the experiment log with its revision and release time.

Record seasonality, campaigns, news, outages, and demand changes that may affect the selected queries or pages. These notes do not explain an outcome by themselves, but they give the reviewer concrete alternatives to a causal claim about the treatment.

Google Search Central advises against showing one set of URLs to Googlebot and another to people during website testing. Its guidance also describes canonical handling for alternate URLs and recommends ending a test once enough data has been collected. Add those search constraints to the implementation review when the experiment creates variants.

Validate the implementation before measurement

Build the exact candidate and inspect every treatment and control route. Confirm response codes, canonical tags, titles, H1s, visible treatment, internal links, schema, media, indexability, Blog index entries, and sitemap ownership. Check that excluded routes did not inherit the change through a shared component.

Save the immutable implementation revision and deployment proof. Request the live pages from a clean session and verify which experience an ordinary visitor receives. If assignment depends on a cookie or script, document the behavior when that mechanism is unavailable.

Start the measurement clock only after the live implementation matches the preflight record. A merge, deployment start, or partial rollout cannot establish a trustworthy start time. Record any exposure delay rather than moving the start date silently.

Review the outcome without rewriting the question

At the planned review point, retrieve data with the frozen filters and preserve the raw export. Compare the primary outcome first, then inspect guardrails and supporting measures. Note missing data, assignment failures, competing releases, and population changes before interpreting movement.

Report null and mixed results with the same care as a preferred result. An inconclusive test can still reveal that the expected mechanism did not create a measurable change under these conditions, or that the design could not separate the treatment from other events.

Close with the predefined action, reviewer, analysis revision, and affected production routes. Keep the preflight beside the outcome so later planning can see what was predicted before the result and avoid learning only from experiments that appeared to win.

Sources

1. Minimize A/B testing impact in Google Search, Google Search Central. Documents search-specific safeguards for website variants, including crawler parity, canonical handling, and ending tests after sufficient data is collected.

Related GTM guides

• Content brief search intent check
• Launch go or no go decision record
• Funnel stage conversion investigation

Frequently asked questions