Skip to content

Rent Observation Ledger

Concept

rent_observations is an append-only, tidy fact table: each row is one price reading of one unit (or a building-level price, when unit-level detail isn't available) from one source on one date.

This single primitive is what lets the system do both:

  • Cross-source price comparison — many sources' current reading for one unit, and
  • Price-history tracking — one source's readings over time

without maintaining two separate structures.

Note

The ledger is purely additive. Nothing existing depends on it being absent, so it never conflicts with prior data — it sits alongside the "current price" fields on units / floorplans rather than replacing them.

Schema

Field Meaning
observation_id Stable hash of (join_key, source, unit_sig, observed_date)
join_key Building street-key
source One of apartments_com, zillow, hotpads, zumper, rentcafe, craigslist, own_site, outreach
listing_url, source_listing_id Provenance
unit_sig A (beds, baths, sqft) signature identifying the unit; empty for a building-level price
beds, baths, sqft Unit attributes
rent Numeric rent, parsed from the source's raw price text
rent_low, rent_high Populated when the source gives a range instead of a single number
concession_raw The verbatim concession text as published (e.g. "1 month free", "$500 off")
rent_effective Rent net of the amortized concession
observed_at Timestamp of this specific reading — the ledger's time axis
listed_at_raw, updated_at_raw The source's own claimed listing/update dates, where published
confidence exact / range / derived

Derived views

  • Per-provider time series — group by (join_key, source), ordered by observed_at, to see one source's price trajectory.
  • Cross-source snapshot — group by join_key at a given date, to see how sources currently disagree on price for the same building.

Concession handling

A parser amortizes concessions like "1 month free" over the lease term, so rent_effective can differ meaningfully from the nominal rent. rent_effective is the field to use for apples-to-apples price comparison across listings carrying different promotions.

Relationship to the refresh worker

The refresh worker (see Data Pipeline architecture, trigger 5 — the refresh worker) records every reachable source's reading into this ledger on each run, even though it only serves one source's value — the highest-trust one — as the property's "official" current price.

Note

This is why the ledger can show provider disagreement even when the served price looks stable: the served price is a single curated value, while the ledger retains every source's independent reading underneath it.

Why this exists

Before this ledger existed, a repeat rent sighting simply overwrote the previous stored price ("newest wins"), losing any history and making cross-source comparison impossible. The ledger's append-only design fixes both problems while requiring no changes to anything already relying on the "current price" field.

See also