Deep Enrichment¶

Design proposal — verify current implementation status before relying on this as shipped behavior

This page describes an architecture/design document for extending the enrichment pipeline. Its phasing and open-questions sections may not reflect what has actually been built. See Discovery & Enrichment for the confirmed-current enrichment behavior.

Goal¶

The proposed design merges the breadth-oriented discovery engine (buildings-scraper — a ZIP/city input that produces a building universe, URLs, owner/PM identity, and a commission verdict, stopping at search-card-level data) with a deeper extraction engine that fetches property detail pages directly. The deeper engine would extract floorplans, amenities, photos, prices, year-built, unit counts, contact info, HUD/Fair-Market-Rent enrichment, a rent-estimate ensemble, price history, and — for fields that cannot be scraped directly — a disclosed inquiry/chatbot-based recovery path.

Proposed output contract¶

Two deliverables per city:

<city>_properties.csv — building grain, a roughly 97-column client benchmark schema, with a provenance sidecar (X__source, X__confidence per column).
<city>_units.csv — unit/floorplan grain: unit_id, property_id/join_key, denormalized building identity, floorplan_name, beds/baths/sqft (validated 200–5000 sqft), rent (validated $400–15000), rent_per_sqft, availability, unit_amenities, pricing_status/is_placeholder, rent_source/confidence (scraped / hud_fmr / estimate / inquiry), and source_url/scraped_at.

Proposed pipeline stages 7 & 8¶

These would be appended after the existing six discovery layers — see Data Pipeline Architecture for those.

Stage 7 — Deep Enrichment¶

A tiered fetch (direct HTTP → unblocker → Playwright browser), a bounded follow-up crawl, and detail-page extraction (JSON-LD, listing-platform data attributes, proximity/text heuristics), feeding a unit grain plus amenity/policy/pet/parking normalization, HUD/FMR rent-ceiling mapping, an image pipeline, and school/geo connectors.

Stage 8 — Price/Commission Recovery¶

Contact-method detection, broker/referral-page scraping, and — where authorized — a disclosed inquiry (chatbot/form/email) flow feeding the outreach state machine.

Both proposed stages are designed to degrade gracefully:

No browser available → skip the Playwright tier.
No LLM key → deterministic extraction only.
No inquiry authorization → compose in dry-run only, never send.

Accuracy strategy¶

Five mechanisms are worth preserving as design principles regardless of implementation status:

Deterministic-first, LLM-last — an LLM never invents a value; it only reads source HTML or counterparty messages.
Numeric validation ranges per field.
Per-field provenance and confidence, with cross-source-agreement boosting and a needs_review flag.
A frozen evaluation set and coverage sweep against benchmark data, gating any release.
A human-in-the-loop review queue for low-confidence or inquiry-recovered values.

Reclassifying "unscrapable" fields¶

Commission and partnership fields, previously treated as unscrapable, are reclassified under this design as partly recoverable, via a broker/referral-page classifier plus inquiry confirmation.

Open questions¶

The following are currently unresolved design decisions, not settled facts:

The exact provenance-data format (sidecar file vs. inline vs. JSON blob).
The ID-minting strategy relative to source-provider IDs.
Which schools data provider to standardize on.
Which LLM provider to standardize on.
How to handle confidence for occupancy/effective-rent fields.
Most importantly: the legal sign-off gate required before any live (non-dry-run) inquiry/chatbot outreach can send — CAN-SPAM, state law, the Fair Housing Act, and vendor terms-of-service all apply.

Note

Before treating any specific claim on this page as current behavior, check it against Discovery & Enrichment and the actual enrich/ module source. This is a design document, and the two may have diverged.