Skip to content

Data Pipeline Architecture

foundationaldata is, logically, a single pipeline: discover candidate buildings, enrich them with contact and commission detail, run outreach against qualified leads, and keep served prices fresh. In production that one logical DAG is executed by six independently scheduled triggers, each with its own cost profile, cadence, and failure boundary. Understanding those six triggers — what they run, where, how often, and at what cost — is the fastest way to understand the system.

One logical pipeline, six triggers

Splitting a single DAG across six triggers, rather than running it as one scheduled job, is a deliberate design choice. It buys three things:

  • Paid steps never run on a schedule nobody asked for. Discovery and website resolution cost real money per run; they are wired to manual invocation only, never to a cron.
  • Different data types have different natural cadences. Building inventory moves slowly enough to enrich weekly; outreach sends need daily/hourly granularity; refreshed prices are useful on a scale of minutes. Forcing all of these onto one cadence would either waste money (over-polling slow-moving data) or leave fast-moving data stale.
  • A failure in one trigger cannot cascade into another. The outreach daemon going down does not stop the refresh worker; a bad enrichment run does not block discovery. Each trigger owns its own retry and failure behavior.

The six triggers, in DAG order:

1. Discovery & resolution — paid, manual

scripts/metro_sweep.py (Bright Data SERP API + unblocker, roughly $30–40 per metro, budget-capped and checkpointed) finds building candidates for a metro. scripts/resolve_sweep.py finds official websites for buildings that discovery found but left URL-less.

Manual only

Both scripts are invoked by an operator, never by a scheduler. A paid step running because a cron fired is treated as a bug, not a feature — see Operational Invariants.

2. Weekly enrichment sweep — free, scheduled

Runs on the Windows scrape box via Task Scheduler, Sundays at 06:00. scripts/enrich_sweep.py fetches each building's own website (via curl_cffi, no paid unblocker needed), extracts contacts, commission terms, and property-manager identity, re-classifies and tiers the building, and upserts the result to the warehouse. It then runs scripts/hql_quality_gate.py in-process — this must run last, because an upsert can silently resurrect a tier that a previous quality check had downgraded. The gate re-checks is-a-building, distinct-from-existing-rows, commission-actionable, has-a-usable-contact, and has-info, downgrading any row that no longer passes.

3. Unit sweep — on demand, manual

scripts/unit_sweep.py --refresh-only, invoked via the Windows run_daily_units.ps1 task. Despite the script name, there is deliberately no scheduled cadence for this trigger — it runs only when an operator starts it, per the principle that nothing should spend on a cadence nobody asked for.

4. Outreach — VPS/Coolify, supercronic

Runs continuously on the VPS under supercronic: a daily web-form campaign, daily mailbox warmup, daily commission-confirmation email sends (A/B tested, gated to CDT/CST business hours), a daily dashboard-metrics snapshot, and a reply daemon that polls IMAP roughly every five minutes to ingest replies and draft responses. See Outreach Pipeline for full detail.

5. Refresh worker — VPS/Coolify, every 10 minutes

scripts/drain_refresh_queue.py drains refresh_queue (populated by the client API's refresh endpoint, or by trigger 6 below). For each claimed building it re-scrapes every source it can reach, serves the single highest-trust source as the "official" value (provenance rank: own_site > rentcafe > zillow > apartments_com), and records every source's reading into the append-only rent_observations ledger — see Rent Observation Ledger.

The worker is bounded by several guards: at most 100 ids per request, a 6-hour cooldown per property, at most 20 queue admissions per hour per API key, at most 50 paid resolutions per day, and an $8/day dollar rail that it degrades under rather than stopping on outright. A same-purpose task remains registered on the Windows box, running every 15 minutes, as a warm standby.

6. Refresh cadence planner — VPS/Coolify, every 30 minutes

scripts/plan_refresh_cycle.py enqueues buildings whose price is due for re-verification: standard-tier buildings at a weekly rate, priority-tier buildings at a daily rate. Both rates are configurable per property via refresh_policy, and priority decays back to standard after a configurable TTL.

This trigger only enqueues — it never fetches a page and never spends money directly. Its population is not a static list; it is the live, currently-served client_buildings set, recomputed on every run. That is what lets a newly served building join the refresh cadence automatically within 30 minutes, with no manual step.

Why this shape (design rationale)

  • Cost isolation. Paid steps (discovery, resolution, a capped share of refresh resolutions) are structurally separated from free steps, so a scheduling mistake cannot turn into an unbounded bill.
  • Demand-driven vs. clock-driven work. Some work should happen because someone asked for it (unit sweep, discovery); other work should happen because time passed (weekly enrichment, refresh cadence). Collapsing the two into one trigger would force one behavior to imitate the other.
  • Different data types age at different rates. Building inventory, unit availability, outreach sends, and rent prices all become stale at different speeds, so each gets its own cadence instead of a single lowest-common-denominator schedule.
  • Failure-domain isolation. Each trigger fails independently. A stuck IMAP poll cannot block the refresh worker; a discovery run's Bright Data outage cannot block enrichment.
  • Physical machine separation. Windows Task Scheduler jobs run on the scrape box (where browser automation and Windows-only tooling live); the VPS/Coolify jobs run the always-on daemons (outreach, refresh). Splitting triggers along that line keeps each machine's responsibilities narrow.

The leads discovery layers

The core buildings-scraper leads --zip <zip> / --city <city> command is itself a six-layer pipeline, run in order and merged by street key at the end. Every layer is optional and degrades gracefully — a missing API key or a failed source reduces coverage silently rather than failing the run:

  1. Government open data — Socrata assessor spine, affordable-housing records, permits.
  2. Federal subsidized housing — HUD ArcGIS layers (Section 8, LIHTC, CHA).
  3. Reference tables — operator contacts, locator/PM commission policies, shell-LLC → principal unmasking.
  4. Listing aggregator scrape — apartments.com, Zillow, RentCafe, Zumper, HotPads, via the Bright Data unblocker.
  5. SERP discovery — Bright Data SERP API finds candidate property-manager websites.
  6. Site crawl — sitemap-driven and breadth-first crawl of discovered sites, extracting schema.org JSON-LD and visible text.

Full detail on how each layer works lives in Discovery & Enrichment; it is not repeated here.

Delivery surface

The Postgres warehouse is the system of record for everything this pipeline produces. CSV files under data/leads/ are an optional mirror, off by default. See Warehouse Schema for the table layout and read paths.