Observability, Reliability, and Operations

Draft 0.1 — 2 August 2026
Reliability targets are service commitments only when explicitly adopted and measured. Draft numbers and dashboards are not contractual guarantees.

Objective

A network built around verified contribution must make its own operations inspectable.

ANUKA needs to know:

whether services are working;
who and what is affected;
whether authorization and consent are enforced;
whether events and payments are delayed;
whether agent tasks produced side effects;
whether data can be recovered;
whether a change improved or harmed the system.

The rule is:

Operate from evidence, not from the absence of complaints.

Operational ownership

Every production component requires:

owning team or named maintainer;
service description;
dependency map;
on-call or escalation path;
runbook;
service-level indicators;
change and release process;
backup and recovery policy;
incident contact;
retirement plan.

An unowned service is not production-ready.

Service catalog

The catalog includes:

service ID;
operator;
product and legal-entity scope;
repositories;
deployment environments;
endpoints;
dependencies;
data classes;
capabilities;
criticality;
SLOs;
current version;
on-call route;
dashboards and runbooks;
status.

The catalog itself must not expose secrets or sensitive network topology publicly.

Criticality tiers

R0 — Informational

Documentation previews, non-authoritative analytics, and optional public visualizations.

R1 — Standard

General product features where short interruption is tolerable.

R2 — Important

Identity sessions, contribution workflows, notifications, and product integrations.

R3 — Critical

Authorization and consent enforcement, credential status, financial ledger, payment orchestration, incident controls, and production release authority.

Criticality affects SLOs, testing, redundancy, approvals, and recovery expectations.

Service-level indicators

Possible SLIs include:

successful request ratio;
latency distribution;
correct authorization-decision ratio;
event publication delay;
event processing delay;
webhook delivery success;
task completion and stuck-task rate;
ledger reconciliation difference;
payment orchestration failure;
consent revocation propagation delay;
credential-status freshness;
search-index freshness;
backup success;
restore verification;
deployment rollback rate.

An SLI must measure the user or protocol outcome, not only server process health.

Service-level objectives

An SLO declares:

service and capability;
indicator;
target;
measurement window;
excluded conditions;
source of truth;
owner;
consequence of breach.

Example draft:

99.9% of valid authorization checks complete successfully within 500 ms over 30 days.

Before adoption, the team must confirm that the metric excludes neither denied requests nor dependency failures in a misleading way.

Error budgets

Error budget is the allowed unreliability implied by an SLO.

When the budget is exhausted, actions may include:

slowing feature rollout;
requiring additional review;
prioritizing reliability work;
disabling risky experiments;
increasing incident readiness;
revising an unrealistic SLO through governance.

An error budget is not permission to intentionally harm users until the budget is spent.

OpenTelemetry baseline

ANUKA uses OpenTelemetry for vendor-neutral traces, metrics, and logs where practical.

Current source versions observed on 2 August 2026 include:

OpenTelemetry Specification 1.59.0;
OTLP Specification 1.11.0;
semantic conventions published independently.

Implementation versions must be pinned per repository and service.

Telemetry signals

Traces

Connect one workflow across API, event, tool, task, provider, and database boundaries.

Metrics

Measure aggregate system behavior and service objectives.

Logs

Record discrete operational and security-relevant information with structured fields.

Profiles

May be used when supported, with awareness that profile-signal stability can differ from traces, metrics, and logs.

Trace context

W3C trace context should propagate through:

HTTP APIs;
event envelopes;
background jobs;
webhooks;
MCP requests;
agent tasks;
provider calls where supported.

Trace IDs are diagnostic correlation identifiers, not authorization tokens or permanent user identifiers.

Common telemetry attributes

Recommended attributes include:

service name and version;
environment;
product ID;
organization scope where safe;
capability;
API or tool contract;
schema version;
task or workflow ID;
authorization result category;
error category;
dependency;
release ID;
data classification category.

Avoid raw personal data, secrets, full prompts, evidence contents, payment details, and unrestricted URLs.

Telemetry schemas

Telemetry contracts must be versioned.

OpenTelemetry schema URLs and transformation concepts should be reused where applicable.

ANUKA-specific semantic conventions may define attributes such as:

anuka.product.id
anuka.capability
anuka.task.id
anuka.consent.result
anuka.tool.contract
anuka.event.type

Attributes need classification, cardinality limits, and ownership.

Cardinality controls

High-cardinality labels can make systems expensive or unusable.

Do not use as general metric labels:

participant ID;
request ID;
full URL;
task prompt;
contribution ID;
raw error message;
email;
external transaction ID.

Use traces or controlled logs for per-request investigation.

Logging policy

Logs should be structured and purpose-specific.

Every log field must justify:

operational purpose;
privacy class;
retention;
access group;
redaction;
expected cardinality.

Debug logging must not be left enabled in production when it exposes sensitive payloads.

Audit versus operational logs

Operational logs help diagnose systems.

Audit records establish evidence of material actions.

Audit events require stronger controls, including:

stable event type;
actor and principal;
action and resource;
authorization reference;
result;
time;
integrity protection;
retention policy;
access review.

A general application log is not a sufficient financial, governance, consent, or agent audit record.

Dashboards

Dashboards should answer specific questions.

Examples:

Can participants authenticate and authorize actions?
Are consent revocations propagating?
Are opportunities and contributions moving through states?
Are agent tasks blocked at approval?
Are webhook subscribers receiving events?
Does the ledger reconcile with payment providers?
Are error budgets healthy?

A dashboard with many charts but no decision owner is decorative monitoring.

Alerts

Alerts must be actionable.

An alert includes:

condition;
severity;
affected service or capability;
owner;
runbook;
suppression and grouping behavior;
expected response;
review date.

Avoid paging humans for conditions that require no immediate action.

Synthetic checks

Synthetic workflows should verify user-critical paths such as:

authentication;
product discovery;
consent request and denial;
contribution submission in test scope;
webhook delivery;
MCP discovery and read-only tool invocation;
status-page accessibility.

Synthetic identities and data must be clearly isolated from real records.

Health endpoints

Health checks distinguish:

process alive;
ready to serve traffic;
dependency degraded;
safe for mutation;
safe for read-only behavior.

A service should not report healthy merely because the process is running while authorization or storage is unavailable.

Degraded modes

Services should define safe degradation.

Examples:

public docs remain available while account features fail;
read-only product profiles remain available while mutations pause;
opportunity search becomes stale but labeled;
agent tasks stop before side effects;
financial actions pause while ledger queries remain available;
consent-required data is withheld when policy service is unavailable.

Fail-open behavior is prohibited for authorization, consent, payment authority, restricted data, and high-impact agent actions.

Dependency management

Each service documents:

critical dependencies;
timeout;
retry policy;
circuit breaker;
fallback;
data consistency;
failure owner;
provider status and support path.

Retries use exponential backoff with jitter and idempotency protection.

Queues and backlogs

Queue monitoring includes:

age of oldest message;
depth;
processing rate;
retry count;
dead-letter count;
partition imbalance;
poison-message rate;
consumer version.

Queue depth alone is not enough; age and business criticality matter.

Capacity planning

Capacity reviews consider:

request and event volume;
data growth;
evidence artifact size;
agent task concurrency;
model and provider quotas;
webhook fanout;
payment volume;
telemetry cost;
tenant concentration;
failure recovery load.

Load tests must include authorization and realistic payload validation, not only empty endpoints.

Backups

Backup policy declares:

systems covered;
data excluded;
frequency;
encryption;
storage separation;
retention;
restore procedure;
access;
deletion obligations;
verification schedule.

A successful backup job is not proof of recoverability.

Restore testing

Restore exercises verify:

data integrity;
point-in-time objective;
time to recover;
key and secret availability;
event replay behavior;
ledger reconciliation;
authorization and consent state;
artifact references;
audit continuity.

Results become operational evidence and remediation tasks.

Recovery objectives

Services may define:

Recovery Time Objective — target time to restore capability;
Recovery Point Objective — acceptable data-loss window.

RTO and RPO are risk decisions, not vendor marketing defaults.

Financial ledger, authorization, consent, and governance records may require stronger objectives than search and analytics projections.

Disaster recovery

The plan addresses:

regional provider outage;
credential compromise;
database corruption;
queue loss;
object-storage failure;
DNS or certificate failure;
identity-provider outage;
payment-provider outage;
malicious deployment;
loss of maintainer access.

A recovery plan must identify who has legal and technical authority to activate it.

Change management

Every production change has:

owner;
release ID;
reviewed diff;
test evidence;
risk class;
rollout method;
observability plan;
rollback or forward-recovery plan;
approval according to risk;
post-deployment validation.

Emergency changes follow incident governance and receive retrospective review.

Progressive delivery

High-risk releases may use:

internal environment;
synthetic tenant;
canary traffic;
selected products;
percentage rollout;
feature flag;
shadow processing;
read-only validation.

Rollout groups must respect privacy and avoid discriminatory experimentation.

Release health

Release evaluation includes:

error and latency change;
authorization denials and unexpected allows;
event lag;
task failure;
provider errors;
cost change;
support reports;
business guardrails.

A rollout should stop automatically or by operator decision when declared thresholds fail.

Runbooks

A runbook includes:

symptoms;
likely causes;
safe diagnostics;
immediate mitigation;
authority requirements;
communication path;
rollback;
evidence preservation;
escalation;
recovery verification.

Runbooks must not contain plaintext secrets.

On-call health

Operational design should minimize unnecessary human interruption.

Track:

page volume;
false-positive rate;
time to acknowledge;
time to mitigate;
repeated incident categories;
after-hours load;
unresolved runbook gaps.

Reliability cannot depend on one founder never sleeping.

Status communication

Incident communication should distinguish:

investigating;
identified;
mitigation in progress;
monitoring;
resolved;
retrospective published.

Messages state affected capabilities and known impact without speculation or unnecessary disclosure.

Post-incident review

A review covers:

timeline;
impact;
detection;
contributing conditions;
decision and authority;
mitigation;
recovery;
evidence gaps;
customer and participant communication;
corrective actions;
owner and due date.

The purpose is system learning and accountability, not ritual blame.

Operational evidence

Evidence may support:

service verification;
resident-product trust status;
security review;
governance decisions;
provider diligence;
reliability claims.

Public reliability claims must define the measured service, time window, exclusions, and source.

Cost observability

Track cost by:

service;
environment;
product or tenant where appropriate;
model provider;
storage class;
event and webhook volume;
telemetry;
agent task;
external API.

Cost controls must not expose private customer information in shared dashboards.

MVP operational gates

Before production:

service catalog exists;
ownership and escalation are known;
critical paths have SLIs;
authorization and consent fail closed;
traces propagate across API, events, and tasks;
sensitive telemetry fields are reviewed;
backups run and a restore is tested;
deployment rollback is practiced;
incident severity and communication are defined;
external dependencies have timeouts and circuit breakers;
status page or equivalent communication path exists.

Acceptance criteria

every service has an owner and runbook;
SLIs measure user or protocol outcomes;
SLOs name sources and exclusions;
OpenTelemetry contracts are pinned;
trace context crosses API, event, webhook, and agent boundaries;
metrics avoid dangerous cardinality;
audit records remain separate from debug logs;
authorization and consent fail closed;
queues are monitored by age and business impact;
backups are restore-tested;
changes have rollout and recovery plans;
incidents produce accountable evidence and follow-up;
public reliability claims are substantiated.

Sources

Verification record

Sources opened and checked: 2 August 2026
OTel specification observed: 1.59.0
OTLP specification observed: 1.11.0
Production SLOs: Not yet adopted