Skip to content

Discovery & Enrichment

This page covers the internal layers of the leads command and the enrichment/quality pipeline in more detail than the architecture overview, which only summarizes them. See Data Pipeline Architecture for the six-trigger framing this page does not repeat.

Command usage

buildings-scraper leads --zip 60614
buildings-scraper leads --city "Chicago, IL"

A full-metro sweep is resumable and unions results per-municipality:

uv run python scripts/metro_sweep.py [--offline] [--limit N] [--resolve-max N] [--outlink-sites N]

Warning

metro_sweep.py is a paid, manual trigger — see Data Pipeline Architecture for why it is never wired to a schedule.

The six discovery/merge layers

leads runs six internal layers in order, each optional and each degrading cleanly — a missing API key or a failed source silently reduces coverage rather than failing the run. When data looks "missing," check which layers actually ran (config/keys present) before suspecting the code.

1. Government open data (Socrata)

Assessor parcel spine, affordable-housing registries, and building permits. Currently Cook County/Chicago-scoped as a source; south Florida is sourced separately — see miami-public-records.md.

Needs verification

Coverage of other counties/metros for this layer was not confirmed while writing this page.

2. Federal subsidized housing (HUD ArcGIS)

Section 8, LIHTC, and CHA public housing layers, pulled from HUD's public ArcGIS services.

3. Reference tables

Bundled CSVs under src/buildings_scraper/intel/data/: operator contacts, locator/ property-manager commission policies, and shell-LLC → real-principal unmasking.

4. Listing aggregator scrape

apartments.com, Zillow, RentCafe, Zumper, and HotPads, fetched through Bright Data's residential-proxy unblocker.

5. SERP discovery

Bright Data's SERP API finds candidate property-manager websites. Queries can be anchored to a specific building name via --anchor-max. Results are cached in data/intel/cache/serp_cache.json so a paid query is never paid for twice.

Warning

A failed fetch must never be cached as an empty success — this is a project-wide invariant, not specific to this cache. See Data Pipeline Architecture.

6. Site crawl

Sitemap discovery plus breadth-first crawl of discovered sites (bounded by --crawl-max-sites / --crawl-max-pages), extracting schema.org JSON-LD structured data and visible text. An optional one-hop outlink crawl (--outlink-sites) catches portfolio or directory pages that link out to individual buildings.

Merge and post-processing

All six layers merge by a canonical street key — see Amenities & Grains for the identity/grain model. After merge:

  1. Unmask LLC owners (reference tables layer).
  2. Resolve missing websites, bounded by --resolve-max.
  3. Classify locator-friendliness and commission.
  4. Attach a representative listing.
  5. Write outputs.

Two output grains

  • Building — the master deliverable, one owner-resolved property row.
  • Listing — one live rental card, or one own-site page.

Outputs land in data/intel/<scope-slug>/:

File Grain Contents
<scope>_rental_intel.csv building master lead list
<scope>_rental_intel_listings.csv listing live listings + crawled pages
<scope>_rental_intel.xlsx both spreadsheet view
discovered_sites.csv site candidate domains (audit trail)
needs_enrichment.csv / needs_enrichment_by_pm.csv building/PM flagged for manual follow-up

Weekly enrichment sweep

Free and scheduled — see Scheduled Jobs for cadence and trigger detail. scripts/enrich_sweep.py:

  1. Fetches each building's own site via curl_cffi (no proxy cost).
  2. Extracts/refreshes contacts, communication method, platform, and PM company.
  3. Re-classifies locator-friendliness.
  4. Re-tiers the building.
  5. Writes contact-quality verdicts.
  6. Upserts the union to the warehouse.
  7. Runs the HQL quality gate in-process, last.

The quality gate checks: is-a-building, distinct-from-existing, commission-actionable, has a usable contact method, and has enough info. Rows failing any check are tier-downgraded.

Note

The gate must run after the warehouse upsert, not before — an upsert can resurrect a tier that a prior quality check had downgraded. See Data Pipeline Architecture and Warehouse Schema for the invariant and the tables involved.

Deep enrichment

The enrich/ package fetches a building's own site plus roughly five sub-pages and maps extracted fields onto a roughly 99-column client benchmark schema. It produces <city>_properties.csv (building grain) and <city>_units.csv (unit/floorplan grain), each field tagged with a provenance label: scraped_live, derived, hud, or seed.

This is being actively extended. See deep-enrichment.md for the architecture of that in-progress work.

Design in progress

deep-enrichment.md documents work that is still under active development; treat it as a design reference rather than a finished contract.

uv run python scripts/run_enrich_sample.py 200       # offline sample
uv run python scripts/run_live_full_sample.py 60     # live sample

Recall/coverage philosophy

The goal is not matching 100% of any one client's existing file. It is, in priority order:

  1. Beating any single data provider on breadth.
  2. Maximizing recall of the web-discoverable population.
  3. Keeping records precise.
  4. Staying city-agnostic.

Coverage is measured, never asserted, by reproducing it against a benchmark file:

uv run python scripts/coverage_by_approach.py data/intel/<scope>

Note

Coverage percentages are a function of the sweep and the benchmark file used, and change every time either changes. Do not treat a coverage number quoted in a past run, meeting, or document as current — regenerate it with the command above.