Skip to content

Improvement Runbook

Purpose

Move a metro's data forward one deliverable at a time: buildings → floorplans → units, plus rents, photos, and amenities — in a single command that's safe to interrupt and safe to re-run. See Data Pipeline for how this fits into the broader six-trigger pipeline, and Warehouse Schema for the tables it writes.

The command

.\deploy\windows\run_improve.ps1              # FREE passes only — safe default
.\deploy\windows\run_improve.ps1 -Budget 5    # allow up to $5 of paid work
.\deploy\windows\run_improve.ps1 -Budget 5 -Hours 3
  • -Budget is a ceiling for the whole run, not per pass.
  • Free passes always run regardless of budget; each paid pass is capped by whatever budget remains.
  • Safe to interrupt (Ctrl-C) — every pass is idempotent and checkpointed, so a re-run picks up where the data actually is.

Passes

Pass Cost Moves
Breadth: own-site fetch Free¹ Floorplans, units, rents
Depth: depth engine Free Unit amenities (its only producer), photos
Depth: enrich Free¹ Building photos, building amenities, contacts
Depth: HQL quality gate Free Tier hygiene
Breadth: aggregator fetch Paid Floorplans/units the own-site pass couldn't reach
Report: metrics Free Dashboard rollups

¹ "Free" means no SERP/aggregator resolution is used; if an own-site page is blocked, the run escalates to the paid unblocker for that one fetch. A hard budget environment variable acts as an independent backstop regardless of the -Budget flag.

Stopping rule

Two consecutive passes that add no new floorplans, units, amenities, or photos end the run automatically — this avoids an all-night re-fetch loop that spends money without helping.

Reading the output

  • data\leads\<metro>\improve_report.json — baseline/final counts, per-pass timing and spend, and the stop reason.
  • logs\scheduled\improve_<metro>_<stamp>.log — full stdout.
  • Console output prints a before/after summary table.
  • Priority metrics to watch, in order: properties with floorplans, floorplan/unit counts, floorplans/units with rent, floorplan and unit amenity counts, building photo count.

Exporting results

uv run --env-file .env --extra db python -m buildings_scraper export --table floorplans --metro miami --out miami_floorplans.csv
uv run --env-file .env --extra db python -m buildings_scraper export --table units --metro miami --out miami_units.csv
uv run --env-file .env --extra db python -m buildings_scraper export --table buildings --metro miami --out miami_buildings.csv

Warning

Always pass --metro — these tables hold every metro's data together.

Known gotchas

  • Use uv run, never a bare python. A bare python resolves to the system interpreter, not the project's, and every pass will silently return instantly with a "consistent" result while doing nothing.
  • uv run pytest can be blocked by local Application Control policy on the Windows box — use uv run python -m pytest instead there.
  • The depth engine only retracts the rows it itself produced; running it against a building that already has own-site floorplans duplicates data. For this reason the improve runbook only targets buildings with zero existing floorplans — don't run the depth sweep by hand without an explicit target list.
  • Never re-run a sweep just to "fix" a bad upstream fetch. If the paid unblocker starts returning empty bodies repeatedly, a kill switch halts paid fetching after 5 consecutive failures — check proxy/zone health with a single manual request rather than retrying (retrying just spends more money against a broken backend).
  • The HQL quality gate must run last in any pass sequence — a union-wide upsert can resurrect a previously downgraded tier, so the gate has to follow it, not precede it.
  • Scope: this runbook does not perform discovery (that's the separate, paid, manual metro-sweep step) and does not deploy anything — it only moves data.
  • Known limitation: roughly a fifth of served rows share a website with another building (portfolio/directory pages), so floorplans scraped from those pages can attach to the wrong building. Running more sweeps doesn't fix this on its own — it needs dedicated "wrong building" suppression work.

See also