Operational Invariants¶

The rules below are the hard constraints the pipeline is designed around. They are not style preferences: violating one produces silent data loss, double-spending against a paid API, or a queue that looks drained but isn't. They are documented explicitly here, rather than left implicit in code, so that anyone changing a trigger, a writer, or the queue logic can check their change against the same list the original design relied on.

Every data-writing step must be reachable from one of the six triggers (see Data Pipeline Architecture). A step that isn't wired to a trigger will never run in production, no matter how correct its code is.
Every new quality signal is stamped directly on the row it describes — a CSV column plus a matching warehouse column — never written to a side file. The only exception is a standalone, clearly labeled read-only reporter that doesn't feed back into the pipeline.
The HQL quality gate must run immediately after every union-wide upsert, because an upsert can silently resurrect a tier that a previous run had downgraded. enrich_sweep runs the gate in-process for exactly this reason, and it must stay the last step in that pipeline.
Never cache a failed fetch as an empty success. A paid query must only be written to a cache (for example the SERP cache) once it actually returned something usable — caching a failure means the pipeline will treat that failure as a permanently-answered question.
Contact verdicts are stamped before tier CSVs or warehouse writes happen, not after — downstream tiering and export logic depend on the verdict already being present on the row.
Paid steps checkpoint every N rows, so an interrupted run doesn't have to re-pay for work it already completed.
The PHQL (probable-HQL) pass runs last in a paid weekly run, since it is the most expendable phase if the run's budget is exhausted partway through.
Only positive evidence licenses deleting existing floorplan/unit rows. A page that was successfully fetched, verified as the right building, and shows zero units triggers a retraction. A fetch that failed, was blocked, or matched the wrong building always leaves existing rows untouched — the system fails closed.
The warehouse (Postgres) is written first. If it can't be reached, the run fails outright rather than falling back to a CSV-only path.
The refresh drain worker stamps its refresh_queue row last, only after both the warehouse write and the refresh_policy stamp have succeeded. This makes a partial failure retry-safe instead of silently marking a queue item done that wasn't actually completed.
Concurrent writers of metro-scoped unit CSVs take the same named file lock (defined once in buildings_scraper.core.filelock) before writing, since more than one job can write those files and an uncoordinated write can corrupt or drop another writer's update.
A refresh serves exactly one source as the "official" reading — ranked by provenance trust, own_site > rentcafe > zillow > apartments_com — but records every source's reading into the append-only rent-observation ledger. See Rent Observation Ledger.
Nothing spends money on a cadence nobody explicitly asked for. The refresh worker is the one exception, and it is bounded by hard per-request, per-hour, and per-day guards rather than by operator intent, precisely because it is demand-driven rather than clock-driven.

Note

Several of these rules exist because of a specific historical incident or migration, not just as a precaution. See History & Migrations for that background if the "why did this happen" story matters for the change being made.