System Overview¶

What this is¶

buildings-scraper discovers rental buildings by ZIP code or city, resolves their owners, and grades each lead through a funnel — candidate → PHQL ("probable high-quality lead") → HQL ("high-quality lead"). Everything is persisted to a single Postgres warehouse (DigitalOcean Managed Postgres, one DSN, DATABASE_URL, system of record since 2026-07-25). Supabase, the prior warehouse, is retired.

Two separate Vercel APIs, deliberately isolated¶

client-api and internal-api are two separate deployments, each with its own deploy pipeline, its own auth scheme, and its own database role — so that "a leaked credential on one can never reach the other."

API	Purpose	Auth	DB access
client-api	Public bulk-sync feed for the "Relive" partner (`GET /v1/properties`)	`X-API-Key`	Read-only `client_buildings` view via a separate `DATABASE_URL_RO`
internal-api	Backend for the internal ops dashboard, the outreach daemon's mail relay, and refresh-queue admin	`X-Internal-Key`	Full warehouse access via `DATABASE_URL`

See client-api reference and internal-api reference for full API documentation.

Reporting surface¶

Reporting is exposed two ways:

The CLI, via metrics, target, and report subcommands, reading precomputed warehouse rollup tables (funnel_daily, coverage_daily, outreach_daily).
A separate internal dashboard application (foundationaldata-dashboard, its own repo), fed by internal-api.

Note

An earlier Next.js dashboard was embedded directly in this repository but was removed. Current reporting is the CLI plus the separate dashboard application — not a resurrection of that earlier dashboard.

Apps built on this data¶

iOS app "Foundational Locator" — reads from a hosted "Resource API." See iOS App.
Chrome extension "relive-overlay" — overlaid Foundational Data intel onto a partner's site. See Chrome Extension.
Landing page (landing/) — the public marketing site, built with Next.js. See Landing Page.

Deprecated

The Chrome extension was retired in August 2026, along with its backing extension-api/ Vercel project. It is no longer deployed against anything and should be treated as legacy code, not an active integration.

Repo layout¶

Path	Purpose
`src/buildings_scraper/`	The core package: `cli/`, `db/`, `intel/` (including `intel/outreach/`), `discovery/`, `enrich/`, `units/`, `adapters/`, `core/`, `identity/`, `output/`, `benchmark/`, `depth/`, plus top-level modules `amenities.py`, `metros.py`, `models.py`
`scripts/`	Pipeline trigger entrypoints and read-only reporters
`client-api/`	Public bulk-sync API — own Vercel deploy
`internal-api/`	Internal ops/outreach/refresh backend — own Vercel deploy
`extension/relive-overlay/`	The Chrome extension (retired)
`landing/`	Next.js marketing site
`deploy/`	Coolify/VPS jobs and Windows scheduled-task scripts
`data/`	Per-run outputs (gitignored)

The pipeline¶

Discovery, enrichment, outreach, and refresh are one logical pipeline, run by six independently-scheduled triggers with deliberate cost and failure isolation. See Data Pipeline for the full breakdown of those triggers.