# TroveSnap Vision Scanning & Item Discovery Specification **Status:** Draft implementation specification **Scope:** Image-driven discovery, tagging, watchlist matching, visual overlays, item identification, condition analysis, and appraisal routing. ## 1. Purpose TroveSnap Vision is the image-intelligence layer for estate-sale and garage-sale imagery. It progressively allocates inference cost: ```text Wide scene image(s) → low-cost discovery scan → candidate objects with bounding boxes → centered item scan → mark/condition evidence scans → appraisal readiness → valuation using TroveSnap sales data and external comps ``` The initial scan does not appraise every object. It identifies which objects deserve additional attention. ## 2. Architectural Boundary ### Vision-model output The image model may return: - candidate objects and bounding boxes - category and likely item label - observable attributes and visible condition - OCR text, labels, signatures, brands and models - confidence and certainty - matches against supplied watchlists - appraisal-candidate flags and reason codes - requested follow-up photos - room/table counts and approximate cross-image deduplication - appraisal evidence completeness - valuation synthesis only when supplied comparable data ### Added later by TroveSnap The image model must not own or invent: - inventory workflow state - listing publication state - platform synchronization status - POS status or sold price - markdown stage - payment or shipping status - buyer-demand counts not supplied in the request - post-sale disposition Those come from TroveSnap application logic, APIs, MCP tools, platform integrations, POS, and post-sale workflows. ## 3. Design Principles 1. **UI-triggered contracts:** UI actions resolve to `scan_type` and `scan_version`. 2. **One logical contract:** OpenAI, Gemini, Claude and local VLMs must normalize into the same result schema. 3. **Strict structure, probabilistic content:** Schemas enforce shape; evidence, confidence and `unknown` manage visual uncertainty. 4. **Progressive cost:** Cheap scene discovery, richer item identification, expensive appraisal only when justified. 5. **Separate configuration from output:** Human-authored YAML is compiled into provider-specific JSON Schema or grammar-constrained JSON. ## 4. Scan Types ```yaml scan_types: table_hunt: input: single_scene_image purpose: find objects worth scanning separately room_scan: input: multiple_images_of_one_zone purpose: categorize, count, deduplicate and identify candidates item_scan: input: single_centered_item_image purpose: identify and summarize the primary object mark_scan: input: single_mark_or_label_closeup purpose: extract maker, brand, model, serial, signature, date or mark condition_scan: input: one_or_more_images_of_same_item purpose: identify visible defects and missing parts appraisal_prepare: input: item_evidence_and_known_facts purpose: determine readiness and create a search fingerprint appraisal_value: input: normalized_item_evidence_plus_comparables purpose: estimate value ranges and explain adjustments ``` ## 5. Versioning Every request includes: ```yaml scan: type: table_hunt version: 2 ``` Every response records: ```yaml scan: type: table_hunt version: 2 result_schema: table_hunt_result result_version: 2 ``` `scan_version` controls behavior, ranking, routing, prompt rules and budgets. `result_version` controls field names, enums and serialization. Stored provenance should include: ```yaml provenance: taxonomy_version: 12 watchlist_versions: paul_personal: 17 regional_demand: 6 provider: gemini provider_model: resolved-model-id prompt_hash: sha256:... schema_hash: sha256:... ``` ## 6. Canonical Request Envelope ```yaml request: id: req_8f29 scan: type: table_hunt version: 2 input: images: - id: img_204 role: scene context: sale_id: sale_221 zone_id: garage_table_1 zone_type: garage location: 95682 profiles: watchlists: - id: paul_personal version: 17 - id: regional_demand version: 6 options: max_candidates: 6 ``` ## 7. Coordinate Contract Scene scans must return coordinates for every candidate. ```yaml coordinates: format: xyxy origin: top_left range: [0, 1000] relative_to: orientation_corrected_image ``` ```yaml bbox: [x_min, y_min, x_max, y_max] ``` Example: ```yaml image: id: img_table_01 width_px: 3024 height_px: 4032 orientation_applied: true coordinate_space: normalized_1000 region: bbox: [65, 310, 280, 690] center: [173, 500] ``` Bounding boxes are mandatory in the MVP. Polygons and masks are future optional extensions. ## 8. Watchlist Contract Watchlists are separate versioned inputs. ```yaml watchlist: id: paul_personal version: 17 entries: - id: vintage_pyrex query: vintage patterned Pyrex bowls aliases: [Pyrex nesting bowls, vintage milk-glass bowls] clues: [printed_pattern, nesting_set, milk_glass, bottom_maker_mark] priority: high - id: vintage_audio query: vintage stereo receivers makers: [Pioneer, Marantz, Sansui, McIntosh] clues: [analog_tuning_dial, silver_faceplate, wood_case] priority: high ``` Allowed match types: ```yaml watch_match_type: - exact - brand_model - maker - visual_pattern - attribute - semantic - category ``` The result schema must restrict `watch_id` to IDs supplied in the request. ## 9. Shared Enums ```yaml identity_certainty: [exact, probable, possible, unknown] visibility: [clear, partial, occluded, distant, blurred, unknown] condition: [sealed, like_new, excellent, good, fair, poor, parts_only, unknown] priority: [low, medium, high, critical] appraisal_state: [no, candidate, priority, blocked] appraisal_readiness: - not_recommended - candidate - needs_item_photo - needs_mark_photo - needs_condition_details - ready - ready_with_limitations ``` Appraisal reason codes: ```yaml appraisal_reason: - visible_maker_mark - visible_model - visible_signature - possible_antique - collectible_category - premium_material - scarce_item - unusual_design - complete_set - provenance_present - high_value_variance - possible_high_value - watchlist_interest - insufficient_identification ``` Next actions: ```yaml next_action: - none - isolate_item - centered_item - crop_item - photograph_front - photograph_back - photograph_top - photograph_sides - photograph_underside - photograph_maker_mark - photograph_brand_model - photograph_signature - photograph_damage - photograph_serial - enter_dimensions - verify_quantity - verify_working - request_appraisal ``` Overlay states: ```yaml overlay_state: - watchlist_match - appraisal_candidate - high_demand - unusual_item - uncertain_candidate - scan_recommended ``` The UI owns colors, icons, animation and callout placement. ## 10. Category Taxonomy Root categories: ```yaml category_root: - furniture - art - decor - kitchen - appliances - electronics - tools - garden - clothing - jewelry - collectibles - books_media - toys_games - sports_outdoor - musical - automotive - office - miscellaneous ``` Hierarchical examples: ```yaml category_examples: - furniture.table.dining - furniture.table.side - furniture.chair.arm - furniture.chair.lounge - kitchen.cookware.cast_iron - kitchen.glassware.mixing_bowl - electronics.audio.receiver - collectibles.ceramic - art.painting ``` The profile compiler should send only relevant category subsets where practical. # 11. `table_hunt` ## UI trigger **Find interesting items** ## Input ```yaml scan: type: table_hunt version: 2 input: images: - id: img_table_01 role: scene profiles: watchlists: - id: paul_personal version: 17 options: max_candidates: 6 include: [watchlist_match, potential_value, unusual_item] appraisal: false ``` ## Output ```yaml scan: type: table_hunt version: 2 result_schema: table_hunt_result result_version: 2 image: id: img_table_01 coordinate_space: normalized_1000 orientation_applied: true result: visible_items_estimate: 24 candidates: - id: c1 image_id: img_table_01 region: bbox: [65, 310, 280, 690] center: [173, 500] label: patterned nesting bowl set category: kitchen.glassware.mixing_bowl confidence: 82 identity: possible visual_tags: [patterned_milk_glass, nesting_set] watchlist: match: true watch_id: vintage_pyrex score: 86 match_type: visual_pattern appraisal: state: candidate reasons: [collectible_category, insufficient_identification] priority: high ui: overlay_state: watchlist_match overlay_label: "1" callout: Scan bottom mark selectable: true next_scan: type: item_scan version: 2 requested_photos: [centered_item, photograph_maker_mark] ``` Rules: - Do not appraise. - Do not generate full listing descriptions. - Do not return every object. - Every candidate requires a valid bounding box. - Prefer `unknown` to unsupported brand/model guesses. - Candidates may be returned for watchlist relevance, unusual design or potential value. # 12. `room_scan` ## UI trigger **Scan this room** ## Input ```yaml scan: type: room_scan version: 3 input: images: - {id: img_room_01, role: overview} - {id: img_room_02, role: overview} - {id: img_room_03, role: table} - {id: img_room_04, role: shelving} context: sale_id: sale_221 zone_id: living_room zone_type: living_room profiles: watchlists: - {id: organizer_priority, version: 4} - {id: local_demand, version: 8} options: deduplicate_across_images: true return_counts: true return_candidates: true max_candidates: 8 ``` ## Output ```yaml result: coverage: partial visible_instances: 44 estimated_distinct: 31 categories: - category: furniture count: 8 tags: [sofa, chair, table, cabinet] - category: decor count: 11 tags: [lamp, framed_art, ceramics] candidates: - id: c7 label: sculptural lounge chair category: furniture.chair.lounge confidence: 86 priority: high sightings: - image_id: img_room_02 bbox: [510, 165, 825, 770] center: [668, 468] visibility: clear primary: true - image_id: img_room_03 bbox: [80, 210, 370, 790] center: [225, 500] visibility: partial primary: false reasons: [unusual_design, possible_watchlist_match] ui: overlay_state: appraisal_candidate overlay_label: "A" callout: Photograph chair separately next_scan: type: item_scan version: 2 missing_coverage: [cabinet_interior, rear_shelf] ``` Rules: - Return both visible appearances and estimated distinct objects. - Deduplicate only when confidence is sufficient. - Every candidate sighting needs coordinates. - Counts are estimates. - Do not create rich descriptions for all room objects. # 13. `item_scan` ## UI trigger **Identify this item** ## Center targeting ```yaml center_targeting: center_region: [0.25, 0.20, 0.75, 0.80] selection_priority: - object_containing_image_center - largest_object_intersecting_center_region - most_complete_object_near_center reject_when: - no_clear_center_object - multiple_equally_prominent_objects - center_object_too_occluded - center_object_too_small ``` ## Input ```yaml scan: type: item_scan version: 2 input: images: - id: img_item_01 role: centered_item target: selection: center_primary_object ignore_background: true options: summary_max_words: 55 match_watchlists: true flag_appraisal: true ``` ## Success output ```yaml status: identified item: category: electronics.audio.receiver name: Pioneer SX-series stereo receiver summary: > Silver-face stereo receiver with an analog tuning display, wood cabinet, and visible wear along the upper cabinet edge. Pioneer branding is visible. The exact model should be confirmed from the rear label. visual_attributes: colors: [silver, brown] materials: [metal, wood_or_veneer] features: [analog_tuning_dial, wood_case, front_control_knobs] visible_condition: grade: fair issues: [cabinet_edge_wear, surface_dust] visible_text: brand: Pioneer model_candidate: possible_SX-780 identity: certainty: probable confidence: 91 tags: [stereo_receiver, vintage_audio, analog_tuner, silver_face, wood_case] appraisal: state: candidate reasons: [visible_maker_mark, high_value_variance] next_photos: [photograph_brand_model, photograph_back, photograph_top] ``` ## Rejection output ```yaml status: retake_required reason: multiple_center_objects guidance: Move closer and center one item. ``` Rules: - Analyze the primary centered object only. - Distinguish visible facts from possible attributes. - Never infer functionality. - Never claim authenticity, age, material, brand or model without supporting evidence. # 14. `mark_scan` ## UI trigger **Scan maker mark or label** ```yaml scan: type: mark_scan version: 1 input: images: - id: img_mark_01 role: mark_closeup context: parent_item_id: candidate_c2 options: expected: [brand, model, serial, maker, signature, date] preserve_raw_text: true ``` Output: ```yaml status: readable mark_type: manufacturer_label raw_text: - PIONEER - STEREO RECEIVER - MODEL SX-780 - SERIAL NO. ZF3602189 normalized: brand: Pioneer model: SX-780 serial: ZF3602189 confidence: 99 ``` Rules: - Preserve literal OCR text. - Return normalized values separately. - Represent uncertain characters explicitly. # 15. `condition_scan` ## UI trigger **Check condition** ```yaml scan: type: condition_scan version: 1 input: images: - {id: img_condition_front, role: front} - {id: img_condition_top, role: top} - {id: img_condition_back, role: back} - {id: img_condition_damage, role: damage_closeup} context: item_name: Pioneer SX-780 stereo receiver options: visible_only: true never_infer_functionality: true ``` Output: ```yaml grade: fair observations: - area: cabinet_top issue: finish_wear severity: moderate image_id: img_condition_top - area: cabinet_edge issue: veneer_chip severity: minor image_id: img_condition_damage apparently_missing_parts: [] functional_status: unknown confidence: 92 ``` Rules: - Report visible condition only. - Reference the supporting image for every observation. - Never infer working status. # 16. `appraisal_prepare` ## UI trigger **Check appraisal readiness** ```yaml scan: type: appraisal_prepare version: 2 input: images: - img_item_01 - img_mark_01 - img_condition_front - img_condition_top - img_condition_back known_facts: brand: Pioneer model: SX-780 powers_on: true audio_tested: false location: 95682 options: estimate_value: false produce_search_fingerprint: true ``` Output: ```yaml readiness: ready_with_limitations identification: brand: Pioneer model: SX-780 confidence: 98 sufficient: - model_identification - front_condition - cabinet_condition - rear_label - location missing: - audio_channel_test - service_history - interior_condition search_fingerprint: query: Pioneer SX-780 stereo receiver powers on cosmetic wear category: vintage_audio_receiver region: 95682 appraisal_confidence_ceiling: medium ``` Rules: - Do not estimate value. - Identify missing evidence. - Produce a normalized search fingerprint. # 17. `appraisal_value` ## UI trigger **Estimate value** ```yaml scan: type: appraisal_value version: 2 item: brand: Pioneer model: SX-780 visible_condition: fair powers_on: true audio_tested: false location: 95682 sale_context: estate_sale comparables: internal: - id: tc_101 sold_price: 390 channel: onsite_estate_sale condition: good tested: true region: northern_california sold_date: 2026-03-12 - id: tc_102 sold_price: 275 channel: onsite_estate_sale condition: fair tested: false region: northern_california sold_date: 2026-01-18 external: - id: ext_201 sold_price: 365 shipping_included: false condition: fair tested: true options: estimates: [quick_sale, normal_estate_sale, patient_local_sale] include_comp_adjustments: true ``` Output: ```yaml currency: USD estimates: quick_sale: [225, 285] normal_estate_sale: [285, 375] patient_local_sale: [350, 450] recommended_starting_price: 395 confidence: medium adjustments: - {factor: powers_on, effect: positive} - {factor: audio_untested, effect: negative} - {factor: cabinet_wear, effect: negative} - {factor: recognized_model, effect: positive} strongest_comps: [tc_102, tc_101, ext_201] missing_evidence: [audio_channel_test, service_history] ``` Rules: - TroveSnap’s pricing engine calculates the quantitative baseline. - The LLM assists with comp relevance, condition adjustments, anomalies, missing evidence and explanation. - Unsupported model memory must never be treated as a comparable sale. ## 18. Provider Adapters All providers normalize to the same canonical result. ```yaml providers: openai: strict_output: json_schema gemini: strict_output: json_schema claude: strict_output: structured_output_or_tool_schema local: runtime: vllm_or_llama_cpp strict_output: [guided_json, json_schema, grammar] ``` Local models should receive concise field semantics even when grammar constrains syntax: ```text Detect only objects worth scanning separately. Use only supplied category and watchlist IDs. Boxes use normalized 0–1000 xyxy coordinates. Confidence is 0–100. Appraisal means candidate screening only, never a price estimate. Use unknown rather than guessing. ``` ## 19. Normalization and Validation ```text Provider response → syntax validation → schema validation → enum normalization → coordinate validation → business-rule validation → canonical TroveSnap result ``` Retry policy: ```yaml retry: max_attempts: 2 first_retry: same_model: true simplified_prompt: true second_retry: fallback_provider: true ``` Validation examples: ```yaml validation: status: retryable reason: invalid_bbox ``` ```yaml validation: status: accepted_with_warnings warnings: [low_confidence_candidate] ``` ## 20. Candidate Ranking The model returns evidence and component scores. TroveSnap computes final ranking deterministically. ```yaml ranking_weights: personal_hunt: watchlist_match: 60 demand_signal: 10 value_potential: 15 confidence: 10 visibility: 5 estate_organizer: watchlist_match: 30 demand_signal: 25 value_potential: 25 confidence: 10 visibility: 10 ``` ## 21. UI/UX Requirements ### Table hunt - draw candidate bounding boxes - show ordinal labels - show semantic icons - allow tap selection - display why the item was flagged - launch requested next scan ### Room scan - show overlays per image - show category totals and estimated distinct count - show candidate queue - show missing-coverage checklist - support multiple sightings of one candidate ### Item scan - guide user to center one item - request retake for ambiguous scenes - guide requested follow-up angles ### Appraisal Clearly distinguish: - candidate - not ready - ready with limitations - ready - estimated value ## 22. Storage Model ```yaml scan_record: id: sc_881 scan: type: table_hunt version: 2 result_schema: table_hunt_result result_version: 2 input: image_ids: [img_table_01] profiles: watchlist_versions: paul_personal: 17 provider: name: gemini model: resolved-model-id result_ref: vision_result_981 metrics: input_tokens: 812 output_tokens: 294 latency_ms: 1610 estimated_cost_usd: 0.0004 ``` Store both raw and normalized results. Scene candidates remain provisional until promoted into an item scan or inventory record. ## 23. Observability Trace structure: ```text trovesnap.scan ├─ trovesnap.profile.resolve ├─ trovesnap.schema.compile ├─ gen_ai.inference ├─ trovesnap.output.validate ├─ trovesnap.result.normalize └─ trovesnap.candidate.rank ``` Track: - candidate precision and recall - watchlist-match acceptance rate - false-positive rate - retake rate - item-scan conversion rate - appraisal conversion rate - bounding-box correction rate - cross-image duplicate error rate - provider disagreement rate - local-versus-hosted quality delta - tokens, latency and cost per scan type ## 24. Security and Privacy - Scope image access to the sale, user and organization. - Use expiring signed URLs. - Send providers only required images and context. - Keep watchlists tenant-scoped. - Treat raw model output as untrusted until validated. - Prevent external appraisal searches from exposing private client information. - Make provider retention behavior configurable. ## 25. Cost Controls ```yaml cost_policy: table_hunt: max_output_tokens: 500 max_candidates: 6 provider_tier: economy room_scan: max_output_tokens: 1200 max_candidates: 8 provider_tier: economy_or_standard item_scan: max_output_tokens: 700 provider_tier: standard mark_scan: max_output_tokens: 300 provider_tier: economy_or_local condition_scan: max_output_tokens: 700 provider_tier: standard appraisal_prepare: max_output_tokens: 600 provider_tier: economy_or_local appraisal_value: max_output_tokens: 1200 provider_tier: strong external_queries_max: 3 ``` ## 26. Acceptance Criteria ### Table hunt - Every candidate has valid normalized coordinates. - Every candidate includes label, category, confidence, priority and next scan. - Supplied watchlist IDs are never invented. - No prices are estimated. - UI can render and tap every overlay. ### Room scan - Accepts multiple images. - Returns visible and estimated distinct counts. - Returns category totals. - Supports multiple sightings per candidate. - Every sighting includes a valid bounding box. - Identifies missing coverage when practical. ### Item scan - Analyzes the centered object only. - Returns `retake_required` when targeting is unreliable. - Produces concise image-derived summary and tags. - Never infers functionality. ### Mark scan - Preserves raw OCR text. - Separately normalizes brand, model and serial. - Represents uncertain characters. ### Condition scan - Links each observation to an image. - Never claims functionality. ### Appraisal preparation - Does not estimate value. - Returns readiness and missing evidence. - Produces search fingerprint. ### Appraisal value - Uses supplied internal and external comparables. - Returns multiple sale-context ranges. - Identifies strongest comps and adjustments. - Preserves uncertainty. ## 27. Implementation Phases 1. **Core discovery:** `table_hunt`, `item_scan`, strict schemas, overlays, one hosted provider. 2. **Room intelligence:** `room_scan`, multiple sightings, counts, approximate deduplication. 3. **Evidence capture:** `mark_scan`, `condition_scan`, follow-up-photo guidance. 4. **Appraisal routing:** `appraisal_prepare`, readiness UI, internal comparable retrieval. 5. **Pricing intelligence:** `appraisal_value`, API/MCP comparable ingestion, regional/channel adjustments. 6. **Local inference:** local VLM adapter, grammar-constrained output, provider routing and benchmarking. ## 28. MVP Decisions 1. Models return schema-constrained JSON; YAML remains the configuration and documentation format. 2. Use normalized `0–1000` `xyxy` bounding boxes. 3. Start with `table_hunt`, `room_scan`, and `item_scan`. 4. Keep watchlists separate and versioned. 5. Rank candidates deterministically after model inference. 6. Require `unknown` instead of unsupported guesses. 7. Store raw and normalized outputs. 8. Keep vision output separate from inventory, POS and lifecycle state. 9. Use economy models for discovery and stronger models only after item isolation. 10. Treat TroveSnap’s growing internal sales corpus as the long-term valuation advantage. ## 29. Product Summary TroveSnap Vision is a progressive discovery and evidence system that: - scans crowded sale environments - highlights the most relevant objects - personalizes discovery through saved watchlists - returns tappable visual overlays - guides users to better item photos - identifies centered objects - extracts marks and visible condition - determines appraisal readiness - combines evidence with TroveSnap’s proprietary sales corpus - controls inference cost through staged analysis > **Scan broadly, highlight selectively, inspect deeply, and appraise only when justified.**