Technical spec · all specs
Create the provider-independent execution boundary through which TroveSnap Vision invokes OpenAI, Gemini, Claude, local VLMs, and future providers. The boundary must:
The rest of the application should operate on:
P1 ScanRequest
→ P2 route and provider execution
→ ProviderAttempt containing a canonical-shaped draft
→ P3 validation, repair, retry, fallback, or acceptance
→ accepted P1 ScanResult
P2 is therefore more than a common run() function. It is the execution, compilation, normalization, provenance, and telemetry boundary for every vision provider.
Use the smallest qualified model that satisfies the scan's quality, privacy, latency, and budget requirements. Provider neutrality does not mean pretending every provider or model has identical capabilities. P2 must preserve:
The system must never silently weaken required semantics merely to make a provider appear compatible.
The adapter must not directly return an implicitly trusted ScanResult.
This interface is too broad:
interface ScanProvider {
run(req: ScanRequest): Promise<ScanResult>;
}
It hides routing; schema compilation; image preparation; provider-native request construction; raw responses; coordinate transformations; usage and cost; provider errors; output decoding; validation status; retry ownership.
Instead, P2 returns a ProviderAttempt.
interface ProviderAttempt {
attemptId: string;
requestId: string;
route: RouteDecision;
invocation: InvocationSummary;
status:
| "response_received"
| "provider_refusal"
| "provider_error"
| "cancelled";
rawResponseRef?: string;
canonicalDraft?: ScanResult;
transformations: AdapterTransformation[];
provenance: ProviderProvenance;
usage: ProviderUsage;
timing: ProviderTiming;
cost: CostObservation;
errors: ProviderError[];
warnings: AdapterWarning[];
}
canonicalDraft means decoded; mapped into the P1 field structure; deterministically normalized; not yet semantically trusted.
P3 decides whether the draft becomes an accepted canonical result.
P2 defines and implements:
scan_type and scan_version.0–1000.P2 does not own: final schema acceptance; semantic business-rule validation; hallucination detection; result repair; retries; fallback after failure; candidate ranking; provider benchmarking scores; local-model qualification; appraisal price calculation; comparable retrieval; durable item creation; inventory, listing, POS, sale, payment, or shipping state; UI business workflows. P2 may collect the inputs and metrics required by those systems, but it must not implement their decisions.
P1 ScanRequest
│
▼
Scan Definition Registry
│
▼
Capability Resolver ─────────── Provider/Model Registry
│
▼
Initial Route Selector ──────── P8 Budget Authorization
│
▼
Prompt & Schema Compiler
│
▼
Image Preparation
│
▼
Provider Adapter
├─ construct native request
├─ invoke provider
├─ capture raw response
├─ decode structured payload
└─ normalize transport representation
│
▼
ProviderAttempt
│
▼
P3 Validation / Retry / Fallback
Primary P2 components:
components:
scan_definition_registry:
purpose: resolve behavior and schema assets by scan type and version
provider_registry:
purpose: record configured providers and models
capability_registry:
purpose: determine whether a model is eligible for a scan version
provider_router:
purpose: choose the initial eligible provider and model
schema_compiler:
purpose: convert P1 schemas into provider-supported structured-output schemas
prompt_compiler:
purpose: compile provider-neutral behavioral instructions plus provider overlays
image_preparer:
purpose: securely materialize and transform image inputs
provider_gateway:
purpose: coordinate one provider attempt
provider_adapters:
purpose: invoke and decode provider-native APIs
attempt_recorder:
purpose: persist raw and normalized attempt evidence
provider_harness:
purpose: manually execute, inspect, compare, annotate, and replay attempts
Provider adapters must not contain the core behavioral meaning of a scan. Scan semantics are resolved from a versioned definition:
scan_definition:
scan:
type: table_hunt
version: 2.0.0
request_schema:
ref: scan-request/table-hunt/2.0.0
result_schema:
ref: scan-result/table-hunt/2.0.0
behavior:
objective: find a limited set of objects worth scanning separately
target: scene_candidates
required_outputs:
- visible_items_estimate
- candidates
- candidate.sightings
- candidate.sightings.region.bbox
- candidate.next_scan
prohibited_outputs:
- prices
- valuation_ranges
- complete_listing_descriptions
- invented_watchlist_ids
limits:
default_max_candidates: 6
default_max_output_tokens: 500
preferred_tier: economy
The scan definition is provider-neutral. Provider-specific tuning is layered onto this definition without changing its meaning.
type ProviderId = "openai" | "gemini" | "claude" | "local";
interface ProviderDescriptor {
id: ProviderId;
adapterVersion: string;
configured: boolean;
enabled: boolean;
endpointClass: "native_api" | "openai_compatible" | "local_runtime";
dataBoundary: "local" | "cloud";
credentialRef?: string;
models: ModelDescriptor[];
}
interface VisionProviderAdapter {
readonly descriptor: ProviderDescriptor;
evaluateSupport(input: ProviderSupportRequest): Promise<ProviderSupportDecision>;
compile(context: ProviderCompileContext): Promise<ProviderInvocation>;
invoke(invocation: ProviderInvocation, signal: AbortSignal): Promise<ProviderNativeResponse>;
decode(response: ProviderNativeResponse, context: ProviderDecodeContext): Promise<DecodedProviderPayload>;
normalize(payload: DecodedProviderPayload, context: ProviderNormalizationContext): Promise<CanonicalDraft>;
}
interface VisionProviderGateway {
execute(request: ScanRequest, routePolicy: RoutePolicy, signal: AbortSignal): Promise<ProviderAttempt>;
}
The gateway coordinates one attempt. It does not retry; invoke a fallback provider; repair semantic errors; rank candidates; or accept the result as trusted.
1. Resolve scan definition
2. Resolve taxonomy and profile inputs
3. Evaluate provider/model capabilities
4. Select initial route
5. Obtain budget authorization
6. Compile prompt package
7. Compile provider schema projection
8. Prepare image inputs
9. Construct native provider request
10. Start trace and timing
11. Invoke provider
12. Capture native response and usage
13. Extract structured payload
14. Normalize transport representation
15. Create canonical-shaped draft
16. Persist sanitized replay evidence
17. Return ProviderAttempt to P3
Every stage must be separately testable and observable.
P2 compiles a versioned prompt package rather than building prompts ad hoc inside each adapter.
prompt_package:
id: table_hunt-2.0.0
version: 5
scan:
type: table_hunt
version: 2.0.0
sections:
- system_role
- scan_objective
- targeting_rules
- evidence_rules
- prohibited_behaviors
- taxonomy_subset
- supplied_watchlists
- coordinate_rules
- result_field_semantics
limits:
max_candidates: 6
max_output_tokens: 500
hashes:
behavior_hash: sha256:...
prompt_hash: sha256:...
The core prompt package defines what the scan is attempting; what visual evidence may be reported; what must remain unknown; the targeted object/scene; required image roles; coordinate semantics; supplied taxonomy IDs; supplied watchlist IDs; prohibited outputs; output-field meaning; uncertainty expectations; scan-specific limits.
A provider overlay may tune presentation without altering business meaning:
provider_overlay:
provider: gemini
model_family: configured-vision-family
version: 3
formatting:
place_coordinate_rule_near_schema: true
repeat_unknown_rule: true
use_compact_field_descriptions: true
Provider overlays must be versioned; hashed; reviewable; separately testable; unable to add application-owned facts; unable to remove P1-required semantics.
Provider-specific prompts must not be embedded as untracked strings throughout adapter code. Every prompt-affecting asset must contribute to prompt_hash.
The P1 JSON Schemas are canonical, but providers and local runtimes may support different JSON Schema subsets. P2 therefore requires a schema compiler.
P1 canonical result schema
→ scan-specific schema resolution
→ provider capability analysis
→ provider-supported projection
→ native schema or grammar
→ compilation report
Deterministic transformations such as: resolve/flatten $ref; inline shared definitions; convert nullable representation; map supported enum syntax; remove documentation-only annotations; order properties consistently; add provider-required wrappers; generate a tool-parameter schema; generate a guided-JSON schema; generate a grammar for a local runtime; generate a canonical reconstruction map.
schema_compilation:
canonical_schema:
id: item_scan_result
version: 2.0.0
hash: sha256:canonical
provider_projection:
provider: openai
adapter_version: 1.0.0
hash: sha256:projection
compatibility: exact
transformations:
- refs_inlined
- descriptions_compacted
unsupported_constraints: []
reconstruction_required: false
Compatibility states:
schema_compatibility:
- exact
- projected
- degraded
- unsupported
If the provider cannot represent a required structural constraint, P2 must (1) record the unsupported constraint; (2) determine whether deterministic reconstruction is possible; (3) mark the provider/model degraded or unsupported for that scan version; (4) prevent production routing when required semantics cannot be preserved. The compiler must not silently remove a required field or restriction.
Production provider calls require one of: native JSON Schema output; strict schema-conforming tool output; guided JSON; grammar-constrained output; another explicitly qualified structured-output mechanism. Prompt-only "please return JSON" is not production-qualified. A permissive prompt-only mode may exist in the development harness but must be marked diagnostic and excluded from normal routing.
P2 owns preparation of provider-ready image inputs while preserving P1 coordinate meaning.
Resolve an expiring signed image reference; retrieve authorized image bytes; verify MIME type; verify dimensions; apply required orientation; strip unnecessary metadata; transcode unsupported formats; reduce dimensions or quality per route policy; calculate image hashes; assign stable image IDs; preserve image order and roles; package URLs, bytes, or provider file references; record all geometric transformations.
P1 coordinates are relative to the orientation-corrected canonical input image. If P2 resizes or transforms an image, it must retain a transformation map:
image_transform:
image_id: img_table_01
canonical:
width_px: 3024
height_px: 4032
provider_input:
width_px: 1512
height_px: 2016
operations:
- orientation_verified
- resized_proportionally
coordinate_transform:
scale_x: 2.0
scale_y: 2.0
offset_x: 0
offset_y: 0
Provider coordinates must be converted back to P1 normalized 0–1000 coordinates.
Cropping is prohibited unless the scan definition explicitly permits it; the crop region is recorded; the coordinate transform can be reversed; and the result remains correctly mapped to the canonical image.
Only the images required for the selected scan may be sent. item_scan should not auto-include every room image; mark_scan sends the mark close-up + only necessary parent context; appraisal_prepare may use normalized prior evidence; appraisal_value should not send unrelated sale images.
providers:
openai:
preferred_output_mode: native_json_schema
image_input: required
production_eligible: after_live_qualification
gemini:
preferred_output_mode: native_json_schema
image_input: required
production_eligible: after_live_qualification
claude:
preferred_output_mode:
- native_structured_output
- strict_tool_schema
image_input: required
production_eligible: after_live_qualification
local:
preferred_output_mode:
- guided_json
- json_schema
- grammar
image_input: required
production_eligible: per_model_and_scan_qualification
The exact model IDs must come from configuration and the provider/model registry. They must not be scattered through application code.
P2 defines the local-provider contract but does not claim local parity.
interface LocalVisionRuntime {
runtimeId: string;
runtimeType: "vllm" | "llama_cpp" | "ollama" | "openai_compatible" | "custom";
health(): Promise<RuntimeHealth>;
listModels(): Promise<LocalModelDescriptor[]>;
invoke(request: LocalVisionInvocation, signal: AbortSignal): Promise<LocalVisionResponse>;
}
The initial P2 deliverable may use a stub local provider that proves configuration loading; capability declaration; schema compilation; request construction; deterministic recorded-response decoding; telemetry shape; provider-attempt output.
E9 owns actual local VLM installation; image-input compatibility; VRAM/memory qualification; scan-specific benchmark results; local-vs-hosted comparisons; production eligibility.
A local model must be qualified independently for each scan type and version. A model may be eligible for mark_scan, appraisal_prepare, or simple item_scan while unqualified for crowded table_hunt, multi-image room_scan, or difficult condition analysis. Local support must never be one global true/false flag.
P2 distinguishes declared capability from qualified capability.
declared_capabilities:
image_input: true
multiple_images: true
structured_output: json_schema
tool_schema: true
grammar_output: false
max_images: configured
supported_mime_types:
- image/jpeg
- image/png
- image/webp
usage_reporting: true
qualified_capability:
provider: gemini
model: configured-model-id
scan:
type: table_hunt
version: 2.0.0
status: qualified
qualification:
fixture_set: table-hunt-gold-v1
benchmark_version: 3
qualified_at: 2026-06-19T00:00:00Z
quality:
schema_success_rate: 1.0
bbox_usable_rate: 0.92
candidate_precision: 0.84
operational:
median_latency_ms: 1650
observed_cost_usd: 0.0012
Qualification states:
qualification_status:
- unknown
- experimental
- qualified
- qualified_with_limits
- suspended
- failed
support_decision:
provider: local
model: local-vlm-1
scan:
type: room_scan
version: 3.0.0
status: unsupported
reasons:
- multi_image_not_qualified
- insufficient_vram_profile
Support states:
provider_support:
- supported
- supported_with_limits
- experimental
- unsupported
P2 selects the provider and model for the first attempt. P3 owns what happens after a failed, invalid, incomplete, or low-quality attempt.
(1) Tenant/user privacy policy. (2) Scan type and version. (3) Required image and schema capabilities. (4) Provider/model availability. (5) Scan-specific qualification. (6) Minimum quality threshold. (7) P8 budget authorization. (8) Estimated cost. (9) Latency preference. (10) Tenant/project overrides. (11) Experiment allocation, when explicitly enabled.
The default is not "cheapest first." The default is: select the least expensive currently available model that is qualified for the requested scan version and satisfies privacy, quality, latency, and budget constraints. This prevents a low-cost but unreliable model from repeatedly creating bad results and triggering more expensive retries.
scan_tiers:
table_hunt:
preferred_tier: economy
quality_requirement: usable_scene_detection
room_scan:
preferred_tier: economy_or_standard
quality_requirement: multi_image_scene_reasoning
item_scan:
preferred_tier: standard
quality_requirement: centered_object_identification
mark_scan:
preferred_tier: local_or_economy
quality_requirement: high_ocr_fidelity
condition_scan:
preferred_tier: standard
quality_requirement: evidence_linked_visible_condition
appraisal_prepare:
preferred_tier: local_or_economy
quality_requirement: evidence_normalization
appraisal_value:
preferred_tier: strong
quality_requirement: supplied_comp_synthesis
privacy_mode:
- local_only
- local_first
- cloud_allowed
- cloud_preferred
local_only may not invoke a cloud provider. local_first uses a qualified local model where available. cloud_allowed selects the smallest qualified model across eligible providers. cloud_preferred may prioritize a qualified hosted model for latency/quality. A route must fail clearly when privacy constraints leave no qualified provider. Privacy policy cannot be overridden by fallback.
route_decision:
route_id: route_551
selected:
provider: gemini
model: configured-model-id
scan:
type: table_hunt
version: 2.0.0
policy:
privacy_mode: local_first
preferred_tier: economy
max_cost_usd: 0.01
considered:
- provider: local
model: local-vlm-1
eligible: false
reasons: [table_hunt_not_qualified]
- provider: gemini
model: configured-model-id
eligible: true
estimated_cost_usd: 0.0012
- provider: openai
model: configured-model-id
eligible: true
estimated_cost_usd: 0.0021
reason:
smallest_qualified_model_within_policy
Every route decision must be explainable.
provider_invocation:
attempt_id: attempt_901
request_id: req_8f29
provider: openai
model: configured-model-id
adapter_version: 1.0.0
scan:
type: item_scan
version: 2.0.0
canonical:
request_schema_hash: sha256:...
result_schema_hash: sha256:...
prompt_hash: sha256:...
native:
endpoint: configured-endpoint
output_mode: json_schema
max_output_tokens: 700
temperature: configured
image_count: 1
controls:
timeout_ms: 30000
live_call_authorized: true
budget_authorization_id: budget_auth_31
Secrets, image bytes, and full signed URLs must not be included in normal logs.
Each adapter must explicitly decode the provider's native response shape: extracting a JSON Schema response object; strict tool arguments; structured text payload; guided JSON from a local runtime; detecting provider refusal/safety outcomes; detecting truncation; capturing provider request IDs and finish reasons.
Allowed: JSON parsing; native SDK response access; tool-argument extraction; deterministic wrapper removal; provider error decoding. Not allowed as a production strategy: regex extraction from a narrative answer; asking another model to guess the intended JSON; silently dropping unparseable content; creating missing business fields from context; interpreting provider prose as a valid canonical result.
P2 normalization converts provider representation into P1 representation. It does not repair semantic mistakes.
Provider keys → canonical keys via explicit maps; pixel → normalized 0–1000; provider coordinate ordering → P1 xyxy; resized/cropped image coordinates back to canonical space; integer/numeric normalization; explicit enum alias mapping; provider null → canonical null/unknown; tool wrapper removal; stable image-ID restoration; stable candidate-ID assignment when the provider supplies none; canonical result-envelope construction; usage unit mapping; finish-reason mapping.
P2 must not infer a missing brand; change a low-confidence identity into an exact identity; repair contradictory evidence; fabricate a missing bounding box; replace an invalid watchlist ID with a likely one; decide two room sightings are the same item; add an appraisal reason not returned; infer functionality; calculate candidate rank; calculate appraisal values; turn free text into unsupported structured claims. Those belong to P3 validation/repair, P4 ranking, or later domain logic.
transformations:
- type: coordinate_conversion
field: result.payload.candidates[0].sightings[0].region.bbox
from:
coordinate_space: provider_pixels
value: [196, 1250, 846, 2782]
to:
coordinate_space: normalized_1000
value: [65, 310, 280, 690]
- type: enum_alias
field: result.payload.item.identity.certainty
from: likely
to: probable
mapping_version: identity-enum-map-1
Every nontrivial normalization must be reproducible.
provider_attempt:
attempt_id: attempt_901
request_id: req_8f29
trace_id: trace_71
route:
route_id: route_551
provider: gemini
model: configured-model-id
scan:
type: table_hunt
version: 2.0.0
status: response_received
assets:
prompt_hash: sha256:...
canonical_schema_hash: sha256:...
provider_schema_hash: sha256:...
taxonomy_version: 12
watchlist_versions:
paul_personal: 17
response:
provider_request_id: provider-generated-id
finish_reason: completed
raw_response_ref: encrypted://provider-attempts/attempt_901/raw
canonical_draft_ref: vision-drafts/draft_801
usage:
input_tokens: 812
output_tokens: 294
image_count: 1
image_bytes: 884211
timing:
started_at: 2026-06-19T18:01:01Z
first_response_ms: 920
total_latency_ms: 1610
cost:
currency: USD
estimated: true
amount: 0.0004
price_catalog_version: provider-prices-2026-06-19
transformations:
- coordinate_conversion
warnings: []
errors: []
P2 stores both the unmodified provider response and the normalized canonical draft. The raw response remains untrusted evidence and must be tenant-scoped.
provider_error_code:
- provider_not_configured
- provider_disabled
- model_not_found
- model_unavailable
- scan_not_supported
- schema_not_supported
- image_format_not_supported
- image_too_large
- too_many_images
- authentication_failed
- authorization_failed
- budget_not_authorized
- rate_limited
- quota_exceeded
- timeout
- network_error
- provider_internal_error
- safety_refusal
- empty_response
- truncated_response
- malformed_structured_output
- decode_failed
- cancelled
- unknown_provider_error
errors:
- code: rate_limited
provider_code: provider_native_code
message: Provider rate limit reached.
retry_hint: retry_after_delay
retry_after_ms: 1500
retry_hint is advisory. P2 does not execute the retry.
SDK-level automatic retries must be disabled where practical, or explicitly configured; observable; counted in usage and timing; included in the provider-attempt record. TroveSnap must not pay for unobserved repeated requests.
P8 owns pricing policy, budgets, quotas, and spend enforcement. P2 integrates with P8 through explicit authorization and usage reporting.
budget_request:
tenant_id: tenant_123
request_id: req_8f29
provider: openai
model: configured-model-id
scan:
type: item_scan
version: 2.0.0
estimated_upper_bound_usd: 0.015
image_count: 1
max_output_tokens: 700
P2 must not call a paid provider without an authorization response when budget enforcement is enabled.
P2 records provider-reported input/output tokens; cached-token units where available; image count; image detail/resolution tier; runtime duration for local models; provider request count; estimated or actual charge; price-catalog version; whether usage information was incomplete.
Live provider calls require an explicitly enabled provider; configured credentials; a nonproduction development project during P2; a per-run or project-level cost ceiling; trace and cost recording; disabled unattended bulk execution by default. The provider harness must display the estimated maximum cost before execution where an estimate is available.
Stored in a secret manager; referenced by ID, not included in scan requests; never in logs, fixtures, replay bundles, or UI output; development and production credentials separate.
Short-lived signed URLs or authorized byte transfer; provider access limited to required images; signed URLs not stored in long-lived trace attributes; image bytes never written to general logs; cached provider files follow tenant retention policy.
Provider attempts are tenant-scoped; watchlists/profiles cannot cross tenant boundaries; raw responses cannot be shared across tenants; prompt/result caching must include tenant + permission scope; shared model prompts must not contain tenant-specific examples.
Only scan-relevant context is sent. Do not send unrelated buyer/seller information; payment data; complete estate records; private notes not required for the scan; unrelated watchlists; other tenants' taxonomy extensions; full inventory histories when normalized evidence is sufficient.
Text visible inside an image is untrusted evidence. The provider prompt must state that image text may be extracted as OCR; image text is never an instruction to the model; QR codes, labels, signs, screens, and handwritten notes cannot override the scan contract; no tools or external actions may be triggered from image content. Provider tools, web search, code execution, and external retrieval remain disabled for normal vision scans unless a future scan definition explicitly permits them.
data_policy:
provider_retention: configured
training_use: configured
region: configured
approved_for_private_sales: true
Routing must respect tenant data policy.
P2 emits a trace for every provider attempt.
trovesnap.scan.provider_attempt
├─ trovesnap.provider.capability_resolve
├─ trovesnap.provider.route
├─ trovesnap.prompt.compile
├─ trovesnap.schema.compile
├─ trovesnap.image.prepare
├─ gen_ai.inference
├─ trovesnap.provider.decode
├─ trovesnap.provider.normalize
└─ trovesnap.provider.attempt_store
Tenant ID or privacy-safe reference; request ID; attempt ID; trace ID; scan type; scan version; result-schema version; provider; resolved model; adapter version; prompt hash; provider-schema hash; taxonomy version; watchlist versions; privacy mode; route reason; qualification status; image count; image bytes; output-token limit; provider finish reason; provider error category; normalization count; live versus replay execution.
Provider attempts by scan type; success and transport-failure rates; native structured-output parse rate; canonical-draft production rate; latency by provider/model/scan; first-response latency; input/output tokens; image count and bytes; cost per attempt; cost per scan type; schema-projection failures; unsupported-capability decisions; coordinate transformations; provider refusals; rate limits; local runtime latency; local-vs-cloud utilization.
P3 later adds validation success, repair, retry, fallback, semantic failure reason. P4/E9 later add candidate quality, bbox quality, OCR quality, provider disagreement, local-vs-hosted quality delta.
P2 includes a thin developer-facing harness for firing shared photos at configured providers and inspecting the complete attempt. It may be a local web page; an internal development route; a CLI plus generated report; or a small combination. The preferred experience is a local/internal web view with exportable replay bundles.
Scan type; scan version; provider; model; image upload or fixture selection; image-role assignment; optional context; optional known facts; optional watchlist fixture; privacy mode; output-token limit; maximum authorized cost; prompt-package version; live or replay mode.
(1) Canonical P1 request. (2) Route decision. (3) Provider capability decision. (4) Compiled provider prompt, secrets removed. (5) Provider schema projection. (6) Schema compilation warnings. (7) Prepared image dimensions and transformations. (8) Provider-native structured payload. (9) Canonical-shaped draft. (10) Coordinate overlays on the source image. (11) Transformation log. (12) Provider request ID and finish reason. (13) Input/output usage. (14) Latency. (15) Estimated or actual cost. (16) Raw-response reference. (17) P1 structural schema check. (18) P3 validation result when P3 is available.
Send the same fixture to multiple provider/model routes. Output: side-by-side canonical drafts; overlaid bounding boxes; item-identity differences; OCR differences; missing-field differences; confidence differences; latency; cost; structured-output success; manual reviewer notes. Outputs are not expected to be textually identical; the purpose is to compare structural compliance, usable evidence, coordinate quality, identification quality, uncertainty discipline, cost, and latency.
manual_review:
reviewer: mark
verdict:
- acceptable
- acceptable_with_issues
- rejected
issue_tags:
- wrong_primary_object
- unusable_bbox
- unsupported_brand_guess
- missed_visible_mark
notes: Free-form review comments.
These annotations later feed E9 qualification and provider benchmarking.
replay_bundle:
canonical_request: included
route_decision: included
prompt_metadata: included
provider_schema: included
native_request_sanitized: included
native_response: included
canonical_draft: included
transformation_log: included
usage_and_timing: included
manual_review: optional
excluded:
- credentials
- reusable_signed_urls
- unrelated_tenant_data
Deterministic tests must not spend provider tokens.
For each provider: compile every supported scan type; snapshot the native request structure; snapshot the provider-schema projection; verify prompt and schema hashes; verify prohibited fields remain prohibited; verify required field semantics remain present; reject unsupported schema projections.
Recorded native responses must cover: successful structured result; provider refusal; truncated response; empty response; malformed JSON; valid JSON with missing wrappers; rate limit; timeout; model unavailable; invalid image; usage metadata absent; provider-generated request ID; SDK-level retry metadata where applicable.
Pixel→normalized; provider-normalized coords; resized-image reversal; cropped-image reversal where permitted; coordinate-order conversion; integer rounding; explicit enum aliases; null/unknown mapping; candidate-ID assignment; image-ID restoration; tool-wrapper extraction; usage normalization; finish-reason mapping.
Model supports scan; supports with limits; lacks multi-image; lacks required structured output; configured but disabled; qualified for one scan and not another; local-only policy with no local candidate; schema-version incompatibility.
Local-only; local-first; cloud-allowed; provider disabled; provider unavailable; budget too low; quality threshold not met; two equally qualified providers with different cost; tenant provider override; experimental model excluded from production; scan-specific model qualification.
Credentials never enter logs; signed URLs redacted; tenant IDs scoped; image text cannot modify the prompt contract; provider tools disabled; unrelated context not transmitted; replay bundles contain no secrets.
Live API tests are required but narrowly controlled. A shared, versioned photo set must be used so provider results can be compared.
For every implemented cloud adapter — item_scan: prove image input works; centered-object instructions received; native structured output returned; identity/evidence fields decode; canonical draft produced; usage/latency/cost recorded. mark_scan: prove image text extracted; raw OCR separate from normalized; structured output decodes; uncertain characters representable; usage/latency/cost recorded.
For at least two cloud providers, run table_hunt: multiple candidates; stable image IDs; bounding boxes normalize to 0–1000; overlays render in the harness; candidates within image bounds; provider cost recorded.
For at least one cloud provider, run room_scan: multiple input images; image roles distinguishable; candidate sightings map to correct images; multiple sightings decode; normalized coords render; visible and distinct counts separate.
P2 requires a functional local adapter stub; recorded-response decoding; schema compilation; local capability declarations; complete ProviderAttempt telemetry. A real local VLM result is desirable but not required for P2 completion (E9 owns local-model qualification).
The test suite must use a dedicated development project; require live-test enablement; enforce a maximum test-run budget; prevent accidental fixture fan-out; record every billable invocation; allow individual provider tests selectively; exclude live tests from normal unit-test execution.
ProviderAttempt.canonicalDraft. P3 determines acceptance.P2 is complete only when the repository contains: (1) VisionProviderAdapter interface; (2) VisionProviderGateway implementation; (3) ProviderAttempt schema + generated types; (4) provider/model registry; (5) capability registry + support-decision types; (6) initial-route policy; (7) scan-definition resolver; (8) prompt compiler; (9) provider-overlay format; (10) schema compiler; (11) schema compatibility report; (12) image-preparation service; (13) coordinate transformation utilities; (14) OpenAI adapter; (15) Gemini or Claude as the second cloud adapter; (16) stub local adapter + local-runtime interface; (17) raw-response + canonical-draft storage; (18) provider-neutral error taxonomy; (19) usage/timing/cost types; (20) OpenTelemetry spans + metrics; (21) deterministic adapter fixtures; (22) sanitized replay bundles; (23) live provider integration tests; (24) WI-056 provider test harness; (25) provider setup + budget-safety documentation.
Implementing all three cloud providers is preferred; two cloud providers are the P2 completion minimum.
P2 is complete when: (1) the rest of TroveSnap invokes vision through one gateway; (2) provider SDK types don't leak beyond P2; (3) the gateway returns a ProviderAttempt, not a trusted final result; (4) ≥2 cloud adapters implemented; (5) a local-provider contract + functional stub exist; (6) scan definitions resolved by type+version; (7) core scan semantics aren't embedded per-adapter; (8) P1 schemas compile into provider-compatible structured-output schemas; (9) every compilation produces a compatibility report; (10) required unsupported semantics make a model ineligible rather than silently removed; (11) production calls use native structured output / strict tool schemas / guided JSON / grammar; (12) prompt-only JSON excluded from production; (13) image prep preserves a reversible coordinate map; (14) provider bboxes normalize to 0–1000; (15) raw responses + canonical drafts stored separately; (16) prompt/canonical-schema/provider-schema/taxonomy/watchlist/adapter/provider/model versions recorded; (17) every attempt records latency + available usage; (18) every paid attempt records cost + price-catalog version; (19) provider errors map to the taxonomy; (20) adapters don't retry or silently fall back; (21) initial routing uses privacy/capability/qualification/budget/cost/latency, not a hard-coded order; (22) local-first routes only when a local model is qualified for that scan version; (23) tenant privacy mode enforced before invocation; (24) credentials/signed URLs absent from logs + replay bundles; (25) image text treated as untrusted OCR, not instructions; (26) provider tools/external retrieval disabled for normal scans; (27) deterministic recorded-response tests pass for every adapter; (28) live item_scan + mark_scan succeed against every implemented cloud provider; (29) table_hunt succeeds against ≥2 cloud providers with renderable overlays; (30) multi-image room_scan succeeds against ≥1 cloud provider; (31) WI-056 can run a photo through a selected provider and display route/schema/native output/canonical draft/overlays/transformations/latency/usage/cost; (32) WI-056 can compare providers + save manual review annotations; (33) live test execution is separately enabled + budget capped; (34) P3 can consume a ProviderAttempt without provider-specific details.
P2 is done when TroveSnap can send the same P1 request through multiple hosted providers, produce comparable canonical-shaped drafts, preserve raw evidence and full provenance, display the attempts in the provider harness, and record enough capability/quality/latency/privacy/cost information for later routing decisions. The P2 result is not merely a common provider interface. It is a provider-neutral vision execution gateway that makes every model invocation replaceable, constrained, replayable, explainable, costed, observable, privacy-aware, and safe to validate before the result enters the rest of TroveSnap.
Tracked while this draft is finalized (resolve before freeze):
ProviderAttempt and owns retry / simplified-prompt / provider-fallback / terminal-failure. P2 only emits retry_hint and never switches providers itself. P3 drives fallback by asking the gateway to execute a different route (with an exclusion list); see P3-output-validation.md §12–§14, §29.cost_estimate_request → budget_authorization) P2 calls before any paid invocation, plus post-call reconciliation. See P8-observability-cost.md §21–§22, §61.scan_trace (closes P1 §21 #5).This gate is the heaviest in the sprint, so WI-011 is split (both gate P2):
ProviderAttempt/provenance/telemetry shape + error taxonomy. Deterministic tests only.item_scan/mark_scan/table_hunt/room_scan tests + budget-capped live harness wiring.