Experimentation, Feature Flags, and Rollouts

Draft 0.1 — 2 August 2026
An experiment can support a claim only within its actual design, population, data quality, and limitations. Feature flags are delivery controls, not proof of causal impact or a substitute for authorization.

Objective

ANUKA treats product changes as learnable interventions rather than isolated releases.

The governing loop is:

question → hypothesis → design → authorization → exposure → observation → analysis → decision → evidence

A rollout may be operationally safe without being a valid experiment. An experiment may be statistically valid without being ethically or commercially acceptable. The protocol keeps these questions separate.

Change modes

Every material product change declares one mode.

Operational rollout

Purpose: deliver an already approved change safely.

Examples:

• staged deployment;
• canary release;
• region-by-region enablement;
• migration rollout;
• progressive permission enablement.

Product experiment

Purpose: estimate the effect of an intervention on predefined outcomes.

Examples:

• A/B test;
• randomized encouragement;
• phased introduction with comparison groups;
• controlled pricing or packaging experiment where lawful and fair;
• usability test.

Exploratory test

Purpose: discover problems or generate hypotheses.

Examples:

• moderated interview;
• prototype test;
• beta cohort;
• observational analysis;
• support-ticket review.

Exploratory findings must not be presented as confirmatory causal results.

Emergency control

Purpose: reduce harm or contain an incident.

Examples:

• disable a vulnerable feature;
• block a broken provider;
• reduce transaction limit;
• enter read-only mode.

Emergency changes prioritize safety and may not preserve experimental comparability.

Experiment registration

Before exposure, create an Experiment Record containing:

• experiment ID;
• product and owner;
• question;
• hypothesis;
• intervention;
• unit of assignment;
• eligible population;
• exclusions;
• variants;
• allocation;
• primary metric;
• secondary metrics;
• guardrails;
• minimum observation window;
• stopping rules;
• analysis plan;
• data fields;
• consent and notice requirements;
• risk tier;
• approval;
• feature-flag keys;
• expected decision.

Material changes after exposure create a new version and are disclosed in the analysis.

Hypothesis requirements

A useful hypothesis states:

• intended population;
• specific intervention;
• expected direction;
• primary outcome;
• time horizon;
• important guardrails;
• practical significance.

Bad:

The redesign will improve engagement.

Better:

For new self-serve users, replacing the three-step setup wizard with a guided checklist will increase the proportion completing first project creation within 24 hours by at least five percentage points, without increasing support contacts, abandonment by keyboard-only users, or consent errors.

Metric contracts

Every decision metric has a contract:

• canonical name;
• owner;
• purpose;
• numerator and denominator or calculation;
• inclusion and exclusion rules;
• source events;
• identity and deduplication rules;
• attribution window;
• timezone;
• freshness;
• missing-data behavior;
• bot and fraud handling;
• privacy classification;
• known biases;
• version.

Changing a metric definition after viewing results is a material analysis change.

Metric roles

Primary outcome

The main basis for the experiment decision.

Secondary outcome

Additional insight that does not replace the primary outcome unless predeclared.

Guardrail

A condition that prevents a superficially positive result from hiding harm.

Examples:

• error rate;
• accessibility failure;
• refund rate;
• support burden;
• fraud;
• retention;
• latency;
• privacy complaints;
• vulnerable-user impact.

Diagnostic

Used to understand mechanism or implementation quality.

Long-term follow-up

Measured after the immediate decision window where the effect may decay or reverse.

Feature flag model

ANUKA should use a vendor-neutral evaluation interface compatible with OpenFeature where practical.

A feature flag record includes:

• stable flag key;
• product and environment;
• type;
• owner;
• purpose;
• variants;
• default value;
• targeting rules;
• prerequisites;
• creation date;
• expiry or cleanup date;
• risk tier;
• audit history;
• emergency behavior;
• linked experiment or release.

Flag types

• release flag;
• experiment flag;
• operational kill switch;
• permission or policy gate;
• migration flag;
• temporary compatibility flag;
• commercial entitlement flag.

Policy authorization must remain authoritative. A feature flag cannot grant a user a capability they are not authorized to exercise.

Evaluation context

Feature evaluation may use only declared attributes.

Examples:

• anonymous assignment key;
• account or organization ID;
• product environment;
• region;
• plan;
• consent state;
• accessibility preference when explicitly provided for the feature purpose;
• risk or eligibility state produced by authorized policy.

Avoid targeting from sensitive traits or inferred categories unless a reviewed use case requires it.

Stable assignment

Experiments should provide stable assignment for the intended unit.

The design declares whether assignment is by:

• participant;
• account;
• organization;
• session;
• device;
• project;
• transaction;
• geographic cluster;
• time period.

Assignment mismatch can contaminate results and create inconsistent user experience.

Exposure logging

A participant counts as exposed only when the intervention was actually delivered or reached the defined exposure point.

Exposure events include:

• experiment and variant;
• assignment unit;
• flag version;
• evaluation reason;
• timestamp;
• product version;
• consent state where relevant;
• trace or request reference;
• data minimization rules.

Assignment and exposure are separate events.

Rollout stages

A default progression:

off → internal → invited pilot → 1% canary → 5% → 25% → 50% → 100% → cleanup

Stages are risk-adjusted rather than mechanically required.

Each stage declares:

• eligible cohort;
• maximum duration;
• health checks;
• success conditions;
• stop conditions;
• approver;
• rollback action;
• next-stage criteria.

Canary releases

A canary should be representative enough to reveal risk but bounded enough to limit harm.

Monitor:

• request success;
• latency;
• dependency failures;
• data integrity;
• user completion;
• support reports;
• financial reconciliation;
• security signals;
• accessibility reports;
• exposure correctness.

A canary is not successful merely because servers remain online.

Stop conditions

Examples:

• critical incident;
• guardrail breach;
• assignment corruption;
• consent or authorization failure;
• payment mismatch;
• accessibility regression in a critical journey;
• severe user confusion;
• unexpected sensitive-data collection;
• SLO burn threshold;
• provider degradation;
• invalid metric pipeline.

Automatic stop mechanisms must be testable and attributable.

Rollback and forward fix

The design states whether rollback is possible for:

• application code;
• configuration;
• database schema;
• data migration;
• external communication;
• credential issuance;
• payment movement;
• public claims.

Some actions cannot be undone. For them, the plan uses containment, correction, compensation, revocation, or supersession.

Experiment risk tiers

X0 — No participant exposure

Synthetic, local, or offline analysis.

X1 — Low-risk interface experiment

No sensitive data, money, high-impact decision, or material degradation.

X2 — Material product behavior

Can affect conversion, pricing presentation, support load, or meaningful user action.

X3 — Sensitive or financially meaningful

Can affect payments, identity, privacy, reputation, contributor opportunity, or business evidence.

X4 — High-impact

Can affect employment, eligibility, legal rights, significant financial outcomes, minors, health, or similarly consequential areas.

X4 experimentation is not enabled by default and requires specialist review.

Consent and notice

Not every ordinary product experiment requires a separate consent dialog, but every experiment must evaluate:

• whether data collection is already authorized;
• whether the intervention changes the purpose or audience;
• whether users are exposed to meaningful risk;
• whether law, contract, research ethics, or product promise requires notice or opt-in;
• whether a participant can reasonably avoid or exit;
• whether debriefing is appropriate.

Consent obtained for product use is not automatically consent for unrelated research or model training.

Accessibility in experiments

Experiments must not systematically exclude disabled users from assignment, analysis, or successful completion.

Required practices:

• test every variant for critical accessibility requirements;
• include accessibility guardrails;
• avoid using only pointer behavior as exposure or success;
• monitor assistive-technology reports;
• do not ship an inaccessible variant merely because it improves conversion;
• record population exclusions honestly.

Pricing and commercial experiments

Pricing tests require review of:

• truthful presentation;
• discrimination and protected-class risk;
• existing contracts;
• refund and renewal terms;
• geography;
• tax display;
• participant expectations;
• whether different prices are visible to connected users;
• retention of assignment evidence.

The protocol does not itself determine legal permissibility.

Sequential monitoring

Repeatedly checking results and stopping when a favorable number appears can inflate false-positive risk.

The analysis plan should state:

• fixed horizon or sequential method;
• review cadence;
• stopping boundary;
• minimum sample or information threshold;
• practical significance;
• uncertainty presentation.

ANUKA must avoid presenting statistical significance as product importance.

Analysis record

The final analysis contains:

• registered plan and versions;
• actual population and exposure;
• data-quality findings;
• exclusions;
• primary and guardrail results;
• uncertainty;
• effect size;
• deviations;
• subgroup analysis policy;
• known confounding;
• operational incidents;
• decision;
• follow-up;
• evidence strength.

Decision outcomes

• ship broadly;
• continue rollout;
• hold current stage;
• revise and retest;
• roll back;
• stop permanently;
• gather more evidence;
• retain as optional behavior;
• publish a negative or neutral learning record.

A neutral result is not a failed contribution when the experiment was well designed and executed.

Multiple products and network effects

Experiments affecting shared services must consider:

• cross-product contamination;
• inconsistent protocol behavior;
• shared identity or consent impact;
• event-schema compatibility;
• downstream analytics;
• other products’ SLOs;
• network marketplace effects;
• coordinated rollback.

Shared-core experiments require architecture and operations review.

Flag lifecycle

Temporary flags must have a cleanup owner and date.

Lifecycle:

proposed → configured → active → fully_rolled_out → cleanup_due → removed
                                   ↘ rolled_back
                                   ↘ expired

Long-lived flags require justification because stale flags increase complexity and hidden behavior.

Experiment registry

The registry should show, subject to confidentiality:

• experiment name and ID;
• product;
• owner;
• status;
• hypothesis;
• primary metric;
• exposure dates;
• current rollout;
• decision;
• evidence strength;
• publication status;
• related product change.

Sensitive details may remain controlled while the existence and decision are recorded.

Public learning records

A public learning record may include:

• problem;
• intervention;
• population summary;
• metric definitions;
• result direction and magnitude;
• guardrails;
• limitations;
• decision;
• contributors;
• evidence references.

Do not expose personal data, trade secrets, or unsupported causal claims.

Anti-patterns

• launching a change first and writing the hypothesis later;
• changing the primary metric after viewing data;
• counting assignment as exposure;
• testing multiple variants without handling multiplicity or uncertainty;
• ignoring guardrail harm;
• ending when results look favorable;
• using flags as authorization;
• never removing flags;
• excluding accessibility testing;
• hiding negative results;
• claiming causality from an observational dashboard;
• running high-impact experiments through ordinary product approval.

MVP acceptance criteria

• every material experiment has a registered record;
• metrics use versioned contracts;
• assignment and exposure are separate;
• flags have owners, types, audit history, and cleanup dates;
• rollout stages have stop and rollback conditions;
• consent and authorization are evaluated separately from flags;
• accessibility and reliability are guardrails;
• analysis records deviations and uncertainty;
• neutral and negative results remain visible;
• shared-core experiments receive cross-product review;
• high-impact experiments are disabled by default.

Sources

• OpenFeature Specification
• OpenFeature Introduction
• Google SRE Workbook — Canarying Releases
• 035 — Value Attribution and Outcome Measurement

Verification record

• Sources opened and checked: 2 August 2026
• OpenFeature role: Vendor-neutral evaluation interface
• Flags as authorization: Prohibited
• High-impact experimentation: Separate approval required