Technical spec · all specs
Load-bearing observability invariant: No consequential TroveSnap action, provider attempt, retry, fallback, approval, policy decision, or paid operation may occur silently. Load-bearing cost invariant: No paid attempt may begin without an explicit, versioned budget authorization covering that attempt.
A unified observability + cost-governance system that makes TroveSnap operations traceable; bounded; explainable; replayable; comparable; quality-aware; tenant-aware; privacy-safe; financially predictable. For every scan or governed operation P8 answers: what was requested; which contract/policy versions applied; which provider/model + why; what was expected to cost; what spend was authorized; what actually happened; were there hidden/explicit retries; which validation failures; why retry/fallback/downgrade/stop; which result was accepted; what quality; what the user did with it; did it create a candidate/item/appraisal/publication/sale; was the spend economically useful; can it be replayed; should the policy be reconsidered. P8 is TroveSnap's operational nervous system, not API billing instrumentation.
2.1 No silent action — every meaningful operation emits telemetry. 2.2 No silent retry — every retry is a separate attempt span + durable record. 2.3 No silent fallback — records prior route; failure cause; new route; policy rule; cost impact; quality expectation. 2.4 No silent spend — every billable operation has preflight authorization + postflight reconciliation. 2.5 No silent degradation — a lower-cost model / reduced candidate limit / optional-output reduction occurs only through an explicit policy decision. 2.6 No silent quality regression — cost reduction below the scan-specific quality floor is not success. 2.7 No silent policy mutation — observed outcomes generate a policy-change proposal; they don't auto-rewrite production budgets/weights/routes. 2.8 No silent human override — manual corrections/approvals/rejects/pins/merges/promotions are observable events linked to the originating operation.
3.1 Product-execution: Tenant → Sale/Buyer Workflow → Workflow Run → Operation → Scan/Import/MCP Action → Attempt → Model Invocation/Tool Call/DB Action.
3.2 Engineering-work (optional): Organization → Project → Epic → Sprint → Story → Task → Agent Action → Tool Call → Model Invocation. Production seller scans don't require a sprint ID; benchmark/replay/harness/dev runs should include work_context when available.
operation:
id: op_881
type: vision_scan
tenant_id: tenant_123
actor: { type: seller, id: user_88 }
context: { estate_sale_id: sale_221, zone_id: garage_table_1 }
request: { id: req_8f29 }
policy: { observability_policy_version: 2.0.0, budget_policy_version: 3.1.0 }
status: completed
operation_type: [vision_scan, appraisal, ingestion_run, candidate_promotion, mcp_tool, export, buyer_notification, identity_binding, status_reconciliation, disposition_action, benchmark, replay]
P8 initially implements full cost governance for vision scans + paid appraisal, but the operation envelope is shared platform-wide.
Related but not interchangeable. 5.1 OTel trace — causal timeline; latency; errors; retries; routing; model/tool spans; correlation (Jaeger/Tempo/Grafana/Datadog/OTLP). 5.2 Structured logs — diagnostics; structured errors; policy-evaluation detail; redacted provider metadata (carry trace_id/span_id). 5.3 Metrics — rates; distributions; percentiles; alerts; dashboards; trends. 5.4 Durable audit events — business-significant actions in P5/P6 (plugin_events, audit_events, inventory_status_events, item_identity_binding_events). 5.5 Cost ledger — authorized/reserved/estimated/actual spend; tenant + plan accounting; credits inputs. 5.6 Durable operation record — stable business reference; final disposition; source/result refs; policy versions; trace IDs; replay info. OTel is not the financial ledger; plugin_events is not the span store; the cost ledger is not the full trace. Linked by stable operation/request/correlation/trace IDs.
Distributed context across Next.js routes; Cloudflare Workers; Supabase RPCs; P2 gateway; P3 validation/recovery; P4 ranking; MCP server; connectors; harness; background jobs.
correlation: { operation_id: op_881, request_id: req_8f29, trace_id: otel-trace-id, root_span_id: otel-span-id }
Asynchronous later workflows use trace links rather than holding one span open indefinitely (table_hunt → later item_scan links to candidate c1 + original trace → later promotion links to candidate lineage).
Each scan emits one durable operation record; one primary root trace; one+ provider-attempt spans; durable attempt + cost records; final quality + outcome observations.
trovesnap.operation
└─ trovesnap.scan
├─ trovesnap.profile.resolve
├─ trovesnap.contract.resolve
├─ trovesnap.schema.compile
├─ trovesnap.image.prepare
├─ trovesnap.budget.evaluate
├─ trovesnap.budget.reserve
├─ trovesnap.route.select
├─ trovesnap.provider.attempt
│ ├─ gen_ai.inference
│ ├─ trovesnap.adapter.decode
│ └─ trovesnap.adapter.normalize
├─ trovesnap.output.validate
│ ├─ trovesnap.validation.schema
│ ├─ trovesnap.validation.references
│ ├─ trovesnap.validation.geometry
│ ├─ trovesnap.validation.evidence
│ └─ trovesnap.validation.invariants
├─ trovesnap.recovery.decide
├─ trovesnap.provider.attempt # retry = sibling span, not hidden inside inference
├─ trovesnap.candidate.rank
├─ trovesnap.result.persist
├─ trovesnap.cost.reconcile
└─ trovesnap.scan.finalize
Use OTel semantic conventions (incl. gen_ai.*); TroveSnap-specific use the trovesnap.* namespace.
8.1 Resource — service.name, service.version, deployment.environment, cloud.region.
8.2 Operation — trovesnap.operation.{id,type,mode}, trovesnap.tenant.id (may be pseudonymized externally), trovesnap.sale.id, trovesnap.actor.{type,role}, trovesnap.correlation.id.
8.3 Scan — trovesnap.scan.{type,version}, trovesnap.result.{schema,version}, trovesnap.taxonomy.version, trovesnap.watchlist.versions_hash, trovesnap.image.{count,total_megapixels}, trovesnap.candidate.requested_max.
8.4 Routing — trovesnap.route.{policy_id,policy_version,provider,model,tier,reason,qualification_version,fallback_index,privacy_mode}.
8.5 Budget — trovesnap.budget.{policy_id,policy_version,reservation_id,scope,state,authorized_microusd,estimated_microusd,actual_microusd,remaining_microusd}.
8.6 Validation — trovesnap.validation.{disposition,issue_count,warning_count,repair_count,removed_candidate_count,retry_recommendation}.
8.7 Ranking — trovesnap.ranking.{policy_version,intent,candidate_count,partial_source}.
High-cardinality component detail belongs in durable records/events, not span attributes.
budget_authorized, budget_denied, route_selected, provider_started, provider_completed, provider_rate_limited, provider_timeout, provider_refusal, validation_failed, deterministic_repair_applied, same_provider_retry_selected, fallback_selected, candidate_removed, partial_result_selected, retake_required, result_accepted, cost_reconciled, circuit_breaker_opened. Each event includes typed fields, not free-form text alone.
Every P2 attempt produces a durable summary (never overwritten by later successful attempts).
provider_attempt:
id: attempt_901
operation_id: op_881
request_id: req_8f29
attempt_number: 1
fallback_index: 0
provider: { name: gemini, model: resolved-model-id, tier: economy }
route:
policy_id: scene-discovery-v3
reason_codes: [qualified_for_table_hunt, lowest_expected_cost, within_latency_target]
contract: { prompt_hash: sha256:..., schema_hash: sha256:... }
budget: { reservation_id: budget_res_41, authorized_microusd: 1200, estimated_microusd: 700 }
timing: { started_at: ..., completed_at: ..., latency_ms: 1050, time_to_first_token_ms: null }
usage: { input_tokens: 812, output_tokens: 294, cached_input_tokens: 0, images: 1, image_megapixels: 12.2, external_queries: 0 }
cost: { estimate_microusd: 700, provider_reported_microusd: null, reconciled_microusd: 710, pricing_catalog_version: pricing-2026-06-01 }
outcome: { provider_status: response_received, validation_disposition: rejected, reason_codes: [invalid_bbox] }
operation_outcome:
operation_id: op_881
disposition: [accepted, accepted_with_warnings, partial, retake_required, insufficient_evidence, rejected, failed, cancelled, budget_blocked]
attempts: { total: 2, providers_used: [local, gemini] }
result: { trusted_result_id: result_812, validation_receipt_id: validation_771, ranking_result_id: ranking_441 }
cost: { reserved_microusd: 2000, reconciled_microusd: 1460 }
quality: { immediate_quality_status: accepted }
A successful provider response is not necessarily a successful operation — P3's trusted disposition decides usability.
Budgets are hierarchical; a paid attempt must satisfy every applicable active budget; a lower-level allowance cannot override a higher-level denial.
Global emergency → Product/environment → Tenant billing-period → Tenant daily → Plan/credit entitlement
→ Sale/project → Actor/session → Workflow → Operation → Scan-type & attempt
budget_scope: [global, environment, product, tenant_billing_period, tenant_daily, plan_entitlement, sale, actor, session, workflow, operation, scan_type, provider, model, paid_feature]
Initial production may begin with global; tenant daily; tenant billing period; operation; scan type — the contract already supports later scopes.
budget_dimension: [monetary_microusd, input_tokens, output_tokens, total_tokens, image_count, image_megapixels, provider_attempts, external_queries, wall_time_ms, concurrent_operations, daily_operations]
Monetary values use integer micro-units, not floating-point dollars (cost: { currency: USD, amount_microusd: 1460 }).
budget_state:
green: { range: 0-70_percent }
yellow: { range: greater_than_70_to_90_percent }
red: { range: greater_than_90_to_100_percent }
critical: { range: greater_than_100_percent }
Green: normal routing. Yellow: warning; prefer cheaper qualified routes; increase visibility; suppress unnecessary escalation. Red: human-visible warning; no optional paid escalation; explicit authorization for discretionary spend; notify monitoring. Critical: block new paid attempts unless emergency override; preserve local/deterministic paths; critical alert; retrospective review. Thresholds are versioned configuration.
budget_policy:
id: vision-default
version: 3.1.0
applies_to: { product: trovesnap, operation_type: vision_scan }
accounting: { currency: USD, pricing_catalog: provider-pricing-2026-06-01 }
reservation: { required_before_paid_attempt: true, expires_after_ms: 120000, reserve_worst_case_output: true }
retry: { counts_against_same_operation_budget: true, requires_new_attempt_authorization: true }
over_budget:
actions: [use_cache, use_deterministic_path, use_local_qualified_model, select_lower_cost_qualified_route, remove_optional_outputs, reduce_configurable_candidate_limit, request_approval, reject]
quality: { never_cross_quality_floor: true }
unknown_price: { action: block_or_use_conservative_ceiling }
scan_cost_policy:
table_hunt: { provider_tier: { preferred: economy, allowed: [local, economy, standard] }, max_output_tokens: 500, max_candidates: 6, paid_escalation: policy_controlled }
room_scan: { provider_tier: { preferred: economy, allowed: [local, economy, standard] }, max_output_tokens: 1200, max_candidates: 8, paid_escalation: policy_controlled }
item_scan: { provider_tier: { preferred: standard, allowed: [local, economy, standard, strong] }, max_output_tokens: 700 }
mark_scan: { provider_tier: { preferred: local, allowed: [local, economy, standard] }, max_output_tokens: 300 }
condition_scan: { provider_tier: { preferred: standard, allowed: [local, economy, standard, strong] }, max_output_tokens: 700 }
appraisal_prepare: { provider_tier: { preferred: local, allowed: [local, economy, standard] }, max_output_tokens: 600, estimate_value: false }
appraisal_value: { provider_tier: { preferred: strong, allowed: [standard, strong] }, max_output_tokens: 1200, external_queries_max: 3, requires_supplied_comparables: true, paid_execution: explicit_or_policy_authorized }
Exact monetary limits live in versioned deployment/tenant policy, not hard-coded in adapters.
Default sequence: deterministic → cached → qualified local → economy hosted → standard hosted → strong hosted → external comparable research. Not an unconditional route order — P2 still considers capability, privacy, qualification, quality floor, availability, latency, budget, user/tenant policy. The principle: use the smallest qualified route that satisfies the current scan's quality + privacy requirements.
Local inference may have no provider invoice but is not operationally free.
local_usage:
provider_billed_microusd: 0
compute: { runtime: local_vllm, model: local-model-id, device_class: rtx_3060_laptop, gpu_time_ms: 1840, wall_time_ms: 2600, peak_vram_mb: 5420 }
allocated_cost: { amount_microusd: configured_internal_estimate }
Supports local-vs-hosted comparison; latency diagnosis; hardware-capacity planning; energy/compute estimation; qualification. A zero invoice must not erase local latency/failure/resource cost.
provider_pricing:
catalog_id: provider-pricing-2026-06-01
provider: configured-provider
model: resolved-model-id
effective_from: 2026-06-01T00:00:00Z
units: { input_tokens: configured-rate, cached_input_tokens: configured-rate, output_tokens: configured-rate, image_units: configured-rate, external_queries: configured-rate }
Pricing never embedded in adapters; effective dates retained; estimate + reconciled cost use a known catalog; catalog changes create new versions; stale/unknown pricing is visible; historical operations retain their original pricing version.
cost_estimate_request:
operation_id: op_881
scan: { type: table_hunt, version: 2.0.0 }
route: { provider: gemini, model: resolved-model-id }
estimated_usage: { input_tokens: 900, maximum_output_tokens: 500, images: 1, image_megapixels: 12.2, external_queries: 0 }
pricing_catalog_version: provider-pricing-2026-06-01
budget_authorization:
id: budget_res_41
decision: authorized
authorized_route: { provider: gemini, model: resolved-model-id }
maximum: { amount_microusd: 1200, output_tokens: 500, attempts_remaining: 2 }
expires_at: 2026-06-20T00:02:00Z
P2 must not begin the paid attempt without this authorization. This is the budget-authorization handshake P2/P3/P7 consume. Entitlement folds in here (D35): the reserve→authorize step also checks plan entitlement + reserves/decrements credits against the ledger before authorizing; G4 reads the ledger for commercial billing. One runtime gate — no paid work escapes the check.
22.1 Reserve — estimate max likely spend; verify applicable budgets; reserve; return authorization. 22.2 Execute — P2 performs the authorized route; may not silently switch to a more expensive one. 22.3 Reconcile — capture usage; calculate estimated actual cost; record provider-reported cost where available; release unused reservation; post final ledger entry; flag abnormal variance. 22.4 Later provider reconciliation — append a reconciliation adjustment; preserve prior estimate; do not overwrite history; link to operation + attempt.
cost_ledger_entry:
id: cost_991
tenant_id: tenant_123
operation_id: op_881
attempt_id: attempt_901
entry_type: [reservation, estimated_usage, reconciled_usage, adjustment, release, credit, refund]
amount_microusd: 710
currency: USD
provider: gemini
model: resolved-model-id
pricing_catalog_version: provider-pricing-2026-06-01
plan_id: seller_base
credit_bucket_id: null
occurred_at: 2026-06-20T00:00:02Z
Append-only. P8 owns measurement + enforcement; G4 owns commercial presentation + charging (see §51).
24.1 Shared operation budget — all attempts for one scan consume the same cumulative envelope; a failed first call does not reset it. 24.2 New authorization per attempt. 24.3 Remaining-budget check — P3 supplies failure cause + proposed next route + estimated next cost + cumulative spend; P8 returns authorized / lower-cost alternative / approval-required / blocked. 24.4 Hidden SDK retries — disabled, or instrumented as separate attempt events with usage + cost. 24.5 Retry caps — total attempts; same-route attempts; fallback attempts; cumulative tokens; cumulative cost; elapsed time.
Deterministic + ordered: would the route exceed budget? If yes → cache? deterministic sufficient? qualified local? lower-cost qualified hosted? optional output removable? configurable candidate cap reducible? tenant approval/credit? → else reject with typed reason.
25.1 Permitted degradation — only declared optional behavior (optional explanation detail; visual tags; max optional candidates; optional secondary comparison; optional external query).
25.2 Prohibited degradation — never silently remove required bboxes; change scan type; convert appraisal_value into unsupported guessing; omit comparable integrity; weaken tenant privacy; accept an unqualified model; lower P3 validation; treat an invalid result as complete; remove required evidence references.
25.3 Typed outcomes — [authorized, authorized_with_reduced_optional_scope, authorized_lower_cost_route, approval_required, plan_limit_reached, tenant_budget_reached, operation_budget_reached, pricing_unknown, qualified_route_unavailable, budget_blocked]. No failure appears as an unexplained generic provider error.
Once the call happened, spend can't be undone. P8 must: record actual usage; reconcile the overrun; mark budget state; block additional attempts where required; emit a cost-variance alert; update estimation metrics; consider opening a circuit breaker; include it in retrospective review. P8 must never falsify the ledger to make the operation appear within budget.
cost_approval:
id: approval_441
tenant_id: tenant_123
scope: { operation_id: op_881, feature: appraisal_value }
maximum_microusd: configured_limit
maximum_external_queries: 3
issued_by: { type: human, actor_id: user_88 }
expires_at: 2026-06-20T00:30:00Z
Approval sources: explicit seller action; tenant budget policy; P7 automation policy; plan entitlement; controlled operational override. The AI model cannot authorize its own spend.
The routing objective: minimize expected cost subject to quality floor, privacy policy, capability requirements, latency objective, budget limits. A cheaper model producing more invalid boxes / invented IDs / retries / false matches / unusable candidates / retakes / manual correction may be more expensive operationally.
trusted_result_rate, accepted_with_warning_rate, partial_result_rate, rejection_rate, retake_rate, validation_issue_rate, deterministic_repair_rate, retry_rate, fallback_rate, invalid_bbox_rate, invented_reference_rate, unsupported_claim_rate, candidate_count, valid_candidate_rate, provider_disagreement_rate.
candidate_acceptance_rate, candidate_rejection_rate, candidate_merge_rate, candidate_label_correction_rate, category_correction_rate, bbox_correction_rate, watchlist_match_acceptance_rate, watchlist_false_positive_rate, identity_correction_rate, condition_correction_rate, better_photo_request_rate, appraisal_recommendation_acceptance_rate. A human rejection is useful feedback, not automatically proof of model error without reason classification.
table_candidate_to_item_scan_rate, room_candidate_to_item_scan_rate, item_scan_to_candidate_rate, candidate_to_promoted_item_rate, promoted_item_to_published_rate, appraisal_prepare_to_appraisal_value_rate, appraisal_to_price_change_rate, candidate_to_sale_conversion, rank_position_to_selection_rate, unsold_recovery_conversion, recovered_revenue. Connect AI spend to product/economic value — must not imply direct causation without analysis.
cost_per_scan_attempt, cost_per_completed_scan, cost_per_trusted_result, cost_per_valid_candidate, cost_per_selected_candidate, cost_per_item_scan_conversion, cost_per_promoted_item, cost_per_completed_appraisal, retry_waste_cost, validation_rejection_cost, fallback_incremental_cost, provider_cost_per_trusted_result, local_vs_hosted_cost_delta, local_vs_hosted_quality_delta, manual_correction_cost_proxy. cost_per_accepted_candidate alone is insufficient (some scans produce no candidates; counts inflate; one strong candidate beats ten weak ones).
Require labeled truth — do not assume every returned candidate is correct, every unselected one false, or every unseen object should have been returned. Use reviewed gold sets; manually annotated scenes; benchmark harness; explicit accept/reject reasons; use-case definitions. Segment by scan type; version; provider; model; prompt hash; schema hash; category; visibility; scene density; local-vs-hosted; ranking intent.
P8 feeds P2 qualification + routing. Qualified per provider + model + scan type + scan version + schema version + prompt version. Considers trusted-result rate; validation failures; candidate quality; geometry quality; latency; cost; retry rate; availability; sample size; drift. A model isn't globally good/bad — it may be qualified for mark_scan and not room_scan.
Track drift in cost; latency; token usage; validation failures; candidate volume; confidence distribution; watchlist accuracy; coordinate quality; human correction; refusal rate; provider disagreement.
drift_observation:
provider: configured-provider
model: resolved-model-id
scan_type: table_hunt
baseline_window: benchmark-v4
current_window: trailing_7_days
drift_score: 0.81
affected_metrics: [invalid_bbox_rate, output_token_growth]
Default escalation around drift_score > 0.75 (versioned). A drift alert does not auto-replace the model — it may reduce route priority; require benchmark review; open a circuit breaker; generate a policy proposal.
May suspend a route on repeated provider failures; schema-violation spikes; cost variance; validation-rejection spikes; unknown pricing; unacceptable latency; material drift; incompatible privacy/retention. Scope: [provider, model, scan_type, tenant, environment]. Records reason; evidence window; opened by/at; test/recovery condition; closed at.
budget_yellow, budget_red, budget_critical, unexpected_cost_growth, reservation_variance, provider_failure_spike, retry_spike, fallback_spike, validation_failure_spike, invalid_bbox_spike, invented_reference_spike, quality_drift, latency_slo_breach, trace_missing, ledger_reconciliation_failure, pricing_catalog_stale, hidden_retry_detected, circuit_breaker_open. Human escalation when budget > 90%; critical spend attempted; drift over threshold; a security/privacy failure; validation failures cross a threshold; repeated paid attempts produce no trusted result.
Versioned + scan-specific: completion_rate, trusted_result_rate, p50_latency, p95_latency, maximum_retry_rate, maximum_budget_block_rate, maximum_validation_failure_rate, trace_completeness, cost_ledger_completeness. Do not use one target for every scan (room_scan ≠ mark_scan workloads).
single_provider_scan, local_first_scan, provider_fallback_scan, human_retake_scan, appraisal_readiness, paid_appraisal, MCP_paid_research, ingestion_import, candidate_promotion, human_approval, status_reconciliation. Each defines expected spans; events; metrics; failure modes; cost dimensions; required correlation fields; dashboard panels; replay requirements. A missing expected span can itself be an observability failure.
40.1 Deterministic validation replay — re-run P3 validation/repair/disposition against a stored P2 canonical draft; no provider call. 40.2 Ranking replay — re-run P4 with a stored trusted result + frozen context + selected policy; no provider call. 40.3 Provider replay — re-run the original scan against a provider when images remain available, privacy permits, budget is explicitly authorized, and it's needed for benchmark/regression; it's a new paid operation linked to the original, never overwriting it.
replay_bundle:
id: replay_bundle_71
operation_id: op_881
request_ref: request_8f29
image_refs: [img_table_01]
contract: { scan_type: table_hunt, scan_version: 2.0.0, result_schema_version: 2.0.0, taxonomy_version: 12, watchlist_versions_hash: sha256:... }
routing: { route_policy_version: 3.0.0, provider: gemini, model: resolved-model-id }
prompts: { prompt_hash: sha256:..., schema_hash: sha256:... }
attempts: [attempt_901]
outputs: { raw_response_refs: [raw_901], canonical_draft_refs: [draft_901], validation_receipt_id: validation_771, ranking_result_id: ranking_441 }
References + hashes, not unnecessary duplication of raw images / private payloads.
operation_failure, provider_weekly, scan_version_release, cost_variance, quality_drift, benchmark, tenant_budget, incident.
retrospective:
id: retro_41
scope: { scan_type: room_scan, scan_version: 3.0.0, provider: local, model: local-model-id }
window: { start: 2026-06-13, end: 2026-06-20 }
findings: [fallback rate increased, invalid primary sighting count increased, hosted fallback restored quality but doubled cost]
actions: [update local prompt candidate, add dense-room benchmark cases, retain current production route until replay passes]
policy_change: { proposed: true, automatically_applied: false }
No production policy is improved until it passes historical replay; gold-set evaluation; cost comparison; quality-floor checks; human review; explicit version publication.
candidate_accepted, candidate_rejected, candidate_not_relevant, bounding_box_corrected, identity_corrected, category_corrected, condition_corrected, watchlist_match_confirmed, watchlist_match_rejected, retake_followed, candidate_promoted, candidate_archived, appraisal_accepted, appraisal_disputed. Records operation; result; candidate/field; actor; reason; before/after values where safe; timestamp. Feedback must not silently alter model behavior — it's evidence for later analysis.
P6 provides durable operational structures equivalent to: operation_runs, provider_attempts, budget_reservations, cost_ledger_entries, quality_observations, policy_decisions, replay_bundles, operational_alerts, retrospectives, human_feedback_events. OTel traces stay in a trace backend. P6 stores operation IDs; trace IDs; durable summaries; audit-relevant facts; financial records; replay references. Do not copy an entire span store into PostgreSQL. (See §61 — this is the P8→P6 storage seam.)
Cost + audit facts are never sampled away. 45.1 Always retain durable records for every operation; provider attempt; paid usage entry; budget decision; final disposition; approval; canonical mutation; security failure. 45.2 Always retain full traces for failures; retries; fallbacks; partial results; budget blocks; red/critical budget events; high-cost operations; paid appraisal; drift cases; security failures; flagged operations; benchmark runs. 45.3 Success sampling — routine successful low-cost traces may be sampled after durable summaries + cost records are committed. Sampling policy is versioned + observable.
Telemetry must not contain raw images; image binary; full prompts; full responses; OCR with private info; private watchlist text; private addresses; buyer identity in seller-visible telemetry; OAuth tokens; service-role keys; local file paths; source credentials; unredacted email bodies; payment info. Use IDs; hashes; counts; classifications; redacted summaries; secure references. Raw prompt/response storage for replay uses protected storage governed by tenant access + retention policy.
Don't use as ordinary metric dimensions: request ID; candidate ID; image ID; arbitrary error text; uncontrolled raw category path; buyer ID; source URL; prompt text. High-cardinality values belong in traces; structured logs; durable records; exemplars. Metrics use bounded dimensions (scan type; provider; model family; route tier; outcome; validation stage; environment; policy version; plan tier).
48.1 Scan operations — volume by scan type; success + trusted-result rate; latency percentiles; retry/fallback; partial/retake; provider mix; budget state. 48.2 Cost — total spend; by tenant/plan/scan type/provider-model; reservation vs actual; retry waste; cost per trusted result; cost per promoted item; local-vs-hosted. 48.3 Quality — validation failures; invalid bboxes; invented references; human correction; candidate acceptance; watchlist accuracy; provider disagreement; drift. 48.4 Funnel — scene scan → candidate → selected → item scan → appraisal prepare → appraisal value → inventory candidate → promoted item → published → sold/recovered. 48.5 Provider (per scan/version) — qualification state; sample size; trusted-result rate; latency; cost; retries; corrections; drift; circuit state. 48.6 Retrospective — open findings; proposed policy changes; replay status; unresolved quality failures; cost anomalies; policy versions awaiting promotion.
Timeline/swimlane: request; contract resolution; budget decision; provider attempts; validation; retry/fallback; ranking; persistence; user review; later linked promotion. Detail drawer (per span/event): input/output references; provider; model; route reason; duration; token usage; cost; validation errors; warnings; retries; linked artifacts; approval state; policy version. Replay action: authorized operators may replay validation; replay ranking; create a provider-replay proposal. Raw private content remains permission-gated.
routing_observation:
scan_type: table_hunt
scan_version: 2.0.0
provider: gemini
model: resolved-model-id
sample_size: 480
trusted_result_rate: 0.94
p95_latency_ms: 2100
mean_cost_microusd: 710
fallback_rate: 0.03
bbox_failure_rate: 0.01
qualification: { state: qualified, version: qualification-17 }
P2 must not query raw observability tables ad hoc per route decision — it consumes approved versioned qualification snapshots.
P8 measures usage; cost; reservations; spend; entitlements; budget status; feature utilization. G4 defines subscription tiers; included scans; credit packages; overage policy; paid upgrades; user-facing billing; commercial margins. P8 must not embed pricing-page language into operational policy; G4 must not invent usage independent of the P8 ledger. (See §61.)
P7 paid tools use the same P8 budget service: run_appraisal, research_comparables, deep_listing_research, external_export_delivery, buyer_notification_batch. For every paid MCP action: create proposal; estimate cost; check plan + tenant budget; obtain approval or policy authorization; reserve spend; execute; reconcile; emit MCP audit + cost entries. An MCP model cannot increase its own budget. (Closes P7 §46 #2.)
Connector/harness operations use the same operation + trace envelope (trovesnap.ingestion with source.connect, import.run, source.discover/record/normalize, candidate.create, plugin.event). Track records discovered; unchanged; candidates created; duplicates; mapping failures; media-copy cost; retries; connector latency; plugin-event linkage. P8 doesn't replace P5 audit — it links operational traces to P5 records.
All core tests run without paid APIs.
54.1 Trace structure (mock providers) — expected trace tree for first-attempt success; same-provider retry; fallback; deterministic repair; partial; retake; budget block; cancellation. 54.2 No hidden retries — simulate adapter/SDK retries; every actual attempt has attempt record + span + usage + cost + reason. 54.3 Budget reservation — paid attempt without authorization fails; authorized starts; expired fails; route mismatch fails; output cap enforced; reservation released after reconciliation. 54.4 Hierarchical budgets — operation allows but tenant blocks; tenant allows but global circuit blocks; plan entitlement blocks; seller override authorizes bounded spend; critical prevents optional escalation. 54.5 Retry accumulation — retries consume one cumulative operation budget; second attempt can't behave as if the first cost nothing. 54.6 Downgrade policy — qualified lower-cost route selected; unqualified cheaper route rejected; required outputs preserved; optional reduced only when policy permits; decision observable. 54.7 Local accounting — local attempts record runtime; model; latency; compute; provider cost 0; allocated cost where configured; quality. 54.8 Pricing catalog — correct version; effective-date selection; stale catalog; missing model price; conservative ceiling; postflight adjustment. 54.9 Actual-cost variance — simulate actual above reservation; true cost recorded; alert emitted; additional retry blocked when required; estimate accuracy updates. 54.10 Trace/ledger consistency — every paid attempt has operation + trace + provider attempt + reservation + ledger entry + reconciliation; detect orphans. 54.11 Redaction — inject private address/OAuth token/watchlist text/buyer identity/local path/raw image; none appears in trace attributes, metrics, logs. 54.12 Sampling — failed/fallback/high-cost traces retained; routine success may be sampled; durable operation + cost records remain. 54.13 Replay — validation replay → same receipt; ranking replay → same hash; provider replay → new linked operation; replay doesn't overwrite originals. 54.14 Quality attribution — candidate review links to originating scan; bbox correction links to candidate + provider attempt; promotion links to candidate lineage; later outcomes don't mutate the original trace. 54.15 Policy immutability — published budget policy can't be edited; new values require a new version; historical operations retain old policy reference; a proposed policy doesn't affect production before publication.
For each live provider: execute one authorized scan; record route decision; record provider attempt; capture token/usage; calculate estimated cost; reconcile; validate through P3; record final disposition; confirm trace-to-ledger linkage; confirm no secrets/raw images in telemetry. Controlled fallback test: dev policy + fixture; force/simulate a retryable first-route failure; confirm P3 requests fallback; confirm P8 authorizes/blocks based on remaining budget; confirm attempts remain separate; confirm cumulative cost correct. A live over-budget test blocks before provider execution rather than overspending.
Before promoting a routing/budget change: select historical operations; use stored trusted results + attempts; replay validation + ranking; estimate route + cost under the proposed policy; compare quality + cost; identify changed provider choices; identify scans that would become blocked; identify quality-floor violations.
policy_comparison:
current_policy: vision-default@3.1.0
proposed_policy: vision-default@3.2.0
operations_replayed: 1000
projected: { spend_change_percent: -18, trusted_result_change_percent: -1, fallback_change_percent: 3 }
quality_floor_violations: 4
recommendation_status: review_required
No policy is auto-promoted solely because projected spend decreases.
57.1 Budgets per tenant + plan? Yes — hierarchical (tenant, plan, period, operation, scan-type). G4 owns commercial packaging; P8 owns enforcement + accounting.
57.2 Where do traces live? Full traces in an OTel backend; durable summaries, ledger, attempts, audit links, replay refs in P6. plugin_events link to traces but don't replace them.
57.3 One trace per scan? Each scan emits one primary root trace with child spans + explicit attempts; later async review/promotion use linked traces.
57.4 Is local inference free? Zero provider invoice possible, but compute/latency/failure/quality are measured.
57.5 Can P2 downgrade automatically? Only through an explicit versioned policy, only to a route that stays qualified + above the quality floor.
57.6 Who authorizes retries? P3 decides recovery is appropriate; P8 authorizes the spend; P2 executes.
57.7 Can actual cost exceed reservation? It can; it must be reconciled truthfully, alerted, and used to improve estimation or open a breaker.
57.8 Can observability auto-change routing? No — it creates qualification observations + policy proposals; production changes require replay + review + version publication.
57.9 All successful traces retained? Durable outcome + cost records always; full low-risk success traces may be sampled.
57.10 Is cost per accepted candidate sufficient? No — also cost per trusted result, selected candidate, promoted item, completed appraisal.
57.11 Does P8 expose billing to users? P8 supplies measured usage + budget state; G4 owns credits/pricing/billing UX.
57.12 Are provider token counts always available? No — usage distinguishes provider-reported, estimated, reconciled; missing usage is visible, not silently zero.
(1) Operation envelope. (2) OTel instrumentation package. (3) Trace propagation helpers. (4) Vision observability blueprint. (5) Provider-attempt instrumentation. (6) P3 validation instrumentation. (7) P4 ranking instrumentation. (8) P5 ingestion trace linkage. (9) P7 MCP trace linkage. (10) Standard telemetry attributes. (11) Span-event taxonomy. (12) Structured logging schema. (13) Metrics registry. (14) Budget-policy schema. (15) Scan-type budget policies. (16) Hierarchical budget evaluator. (17) Budget reservation service. (18) Attempt authorization contract. (19) Pricing catalog. (20) Cost estimation service. (21) Cost reconciliation service. (22) Append-only cost ledger. (23) Budget state calculator. (24) Over-budget decision engine. (25) Paid-action approval integration. (26) Local compute accounting. (27) Provider qualification snapshots. (28) Quality observation schema. (29) Human feedback events. (30) Drift detection. (31) Circuit-breaker contract. (32) Operational alerts. (33) Replay bundle. (34) Validation replay. (35) Ranking replay. (36) Provider replay proposal. (37) Retrospective record. (38) Historical policy replay tooling. (39) Policy comparison report. (40) Trace sampling policy. (41) Telemetry redaction rules. (42) Cardinality guardrails. (43) Scan operations dashboard. (44) Cost dashboard. (45) Quality dashboard. (46) Provider dashboard. (47) Funnel dashboard. (48) Retrospective dashboard. (49) Observatory timeline + detail-drawer requirements. (50) Deterministic observability tests. (51) Deterministic cost-governance tests. (52) Live provider trace test. (53) Budget-block test. (54) Ledger consistency test. (55) Privacy/redaction test. (56) Operator documentation.
P8 is complete when: (1) every scan creates a durable operation record; (2) one primary root trace; (3) provider attempts are distinct child spans; (4) same-provider retries individually visible; (5) fallback attempts individually visible; (6) hidden SDK retries disabled or separately observed; (7) trace records scan type+version; (8) result-schema version; (9) applicable routing/budget/observability policy versions; (10) every provider attempt records provider/model/tier/route reason/prompt hash/schema hash/latency/usage/cost estimate/validation outcome; (11) OTel standard attributes used where available; (12) TroveSnap attributes use a stable namespace; (13) logs carry trace+span IDs; (14) audit records link to trace IDs; (15) ledger records link to operation+attempt IDs; (16) no paid attempt begins without authorization; (17) every paid attempt gets a reservation; (18) every paid attempt is reconciled; (19) provider-reported/estimated/reconciled usage distinguishable; (20) monetary accounting uses integer micro-units; (21) pricing catalogs versioned + effective-dated; (22) pricing not hard-coded in adapters; (23) historical operations retain original pricing version; (24) retries consume the original operation's cumulative budget; (25) every retry requires new attempt authorization; (26) tenant/plan/period/operation/scan budgets evaluated together; (27) budget states support green/yellow/red/critical; (28) red prevents optional paid escalation by default; (29) critical blocks new paid attempts unless overridden; (30) over-budget decisions use a typed result; (31) budget handling never silently weakens required fields; (32) a cheaper route must remain qualified; (33) + satisfy the quality floor; (34) local inference records runtime/latency/resource/quality; (35) zero provider invoice doesn't make it unobservable; (36) actual spend above reservation recorded truthfully; (37) variance emits an alert; (38) variance affects estimation + breaker evaluation; (39) P3 determines whether retry is appropriate; (40) P8 determines whether retry spend is authorized; (41) P2 performs only the authorized route; (42) P4 ranking links to the source operation; (43) P5 candidates link to originating scans; (44) P6 promotion links to originating candidates + scans; (45) later review/promotion use linked traces; (46) immediate quality metrics include trusted/warning/partial/rejected/retake; (47) validation metrics include geometry/reference/unsupported-claim failures; (48) human-review metrics include candidate/category/identity/bbox corrections; (49) downstream metrics connect to selection/promotion/publication/sale; (50) cost per trusted result; (51) cost per valid candidate; (52) cost per selected candidate; (53) cost per promoted item; (54) retry-waste + validation-rejection spend measurable; (55) precision/recall use reviewed/annotated data; (56) provider quality segmented by scan+version; (57) local-vs-hosted quality + cost comparable; (58) qualification snapshots versioned; (59) routing consumes approved snapshots; (60) drift measured against an explicit baseline; (61) drift alerts don't silently change routes; (62) circuit breakers can suspend a provider/model/scan combo; (63) every breaker state change auditable; (64) failures/retries/fallbacks/high-cost/budget-blocks retain full traces; (65) routine success sampled only after durable facts stored; (66) raw images absent from normal telemetry; (67) credentials/tokens absent; (68) private addresses + buyer-private absent from general telemetry; (69) high-cardinality values excluded from metric labels; (70) replay bundles preserve hashes/versions/refs; (71) P3 validation replayable without a provider call; (72) P4 ranking replayable without a provider call; (73) provider replay creates a new linked operation; (74) replay never overwrites historical results; (75) policy changes require new immutable versions; (76) proposed changes comparable against historical operations; (77) lower projected cost alone can't auto-promote a policy; (78) retrospectives capture cost/quality/fallback/drift findings; (79) retrospective actions explicit + reviewable; (80) paid MCP tools use the same budget authorization + ledger; (81) MCP clients can't authorize their own spend; (82) ingestion/connector traces link to P5 records; (83) dashboards expose volume/cost/quality/latency/retries/provider mix; (84) the Observatory can display the complete scan timeline; (85) operators can inspect route/cost/validation/fallback in a detail view; (86) deterministic tests pass with mock providers; (87) live provider tests produce trace/usage/ledger records; (88) a deliberately over-budget dev scan is blocked before provider execution; (89) no scan may spend silently; (90) no scan may fail silently.
P8 is done when TroveSnap can observe, explain, bound, and learn from every scan and paid intelligent operation. For every operation TroveSnap can answer: what initiated it; which tenant/sale/actor/workflow; which scan/schema/prompt/taxonomy/policy versions; which routes considered; why the selected provider was qualified; estimated cost; reserved budget; which attempts; which failed; why retry/fallback; what each attempt cost; what result passed validation; what was removed/warned; how candidates were ranked; what the user did next; did it become inventory; did the item publish/sell; did the spend produce value; did quality/cost drift; was a breaker triggered; can it be replayed; was a retrospective created; did any production policy change as a result. The P8 result is a unified observability, cost-governance, replay, and continuous-learning control plane providing causal traceability; bounded spending; provider accountability; quality economics; human-feedback lineage; operational replay; policy retrospectives; tenant-aware monetization inputs; and confidence that no action, retry, fallback, or expense occurs invisibly.
Adopting P8 closes every remaining cross-gate open item:
Two boundaries to record:
operation_runs, provider_attempts, budget_reservations, cost_ledger_entries, quality_observations, policy_decisions, replay_bundles, operational_alerts, retrospectives, human_feedback_events) are realized in P6 (they subsume the P6 §61 #1 P3-attempt + #2 P4-ranking storage notes). OTel traces live in a separate trace backend.The most cross-cutting gate — sliced 3 ways (all gate P8, owner Paul); WI-066 holds the budget handshake P2/P3/P7 depend on: