Cost Optimization¶

Discovery and enrichment mix free and paid data sources. This page describes how paid requests are tracked and the structural choices made to keep the paid surface as small as possible. See Data Pipeline Architecture for how paid steps are isolated from scheduled ones at the trigger level.

How paid requests are tracked¶

Paid requests — Bright Data SERP queries and Bright Data unblocker fetches — are billed at a single measured rate, currently approximately $1.50 per 1,000 requests for both SERP and unblocker traffic. That rate is tracked in one place in the code, core/cost.py, and recalibrated by scripts/calibrate_cost.py.

Needs verification

The $1.50/1,000 figure is the currently measured rate as described in project context at the time this page was written. Bright Data's actual billed rate can change; treat core/cost.py and scripts/calibrate_cost.py as the source of truth, not this number.

Weekly refresh is effectively free¶

The weekly building-level enrichment sweep (scripts/enrich_sweep.py) incurs close to zero marginal paid cost:

Own-site fetches go through curl_cffi, a Chrome-impersonating HTTP client — no proxy cost.
Zillow re-fetches are free.
SERP resolutions are cached once (data/intel/cache/serp_cache.json) and never re-paid for on subsequent runs.
Only apartments.com fetches consume paid unblocker requests, because it is the one aggregator that resists direct fetching.

Two structural cost reductions¶

1. DDG-first SERP¶

DuckDuckGo serves discovery queries for free first. Bright Data's paid SERP API only backstops a query when DuckDuckGo returns fewer than 3 results. Each sweep logs how many queries were served free versus how many fell back to paid, in the run summary.

2. Direct-first crawl¶

Own-site pages, Zillow, and HotPads are fetched directly via the Chrome-impersonating HTTP client — free. The paid unblocker is scoped only to apartments.com and whatever the free layers could not reach.

Discovery sweeps stay paid and manual¶

Full-metro discovery sweeps remain deliberately paid and manual — for example, on the order of $30–40 per metro — rather than scheduled, until cost/coverage tradeoffs are well understood for a given market. This is a standing decision, not an oversight: see Data Pipeline Architecture for why paid steps are never wired to a cron.

Getting current cost figures¶

uv run python scripts/cost_report.py

Note

The exact per-metro cost table changes every time a metro is swept. Always pull it fresh from scripts/cost_report.py rather than copying a figure from documentation — any dollar amount in this page is illustrative, not a current or guaranteed rate.