Outreach Pipeline¶

The outreach pipeline contacts a building's leasing or owner contact to learn whether the building pays commission to outside locators (and at what rate) and to confirm current rents. It is designed to run at a low, natural-looking send volume, with provider failover so a single channel or infrastructure outage never breaks an entire campaign.

Channels¶

Channel	Module	Transport	Notes
Web form (primary)	`webform.py`	curl_cffi → Bright Data Web-Unlocker fallback	Discovers the form, fills fields by inferred semantic role; a hard CAPTCHA falls back to email; a heavier Playwright-based path (`browser.py`) is available via the `browser` extra
Email (fallback)	`delivery.py` / `imap_client.py`	SMTP + IMAP (Migadu hosting)	Rotating sender pool, threaded replies, a continuously-running reply daemon
LinkedIn / WhatsApp Web / Google Voice	`automation.py` + `browser.py`	Playwright over an authenticated operator session	No official API for these; gated behind `dry_run=False` plus a configured session; per-account daily caps (LinkedIn 20, WhatsApp 40, Google Voice 50)

Channel strategy: the web form is tried first, since it is free and avoids the email daily send cap. If a building has no usable form, email is used directly. If a form submission goes unanswered for roughly 10 days, the campaign escalates to email.

Web-form design (the semantic-schema approach)¶

Three properties make this channel work across a wide range of unrelated site markup:

Semantic schema, not raw HTML. Every discovered form's inputs are normalized to a role (name / email / phone / message / move_in / beds) rather than matched by field name, so wildly different site markup becomes fillable from one common "inquiry" shape. Hidden CSRF/framework tokens and a CAPTCHA-present flag are captured too. Records are stored append-only and deduplicated on (join_key, form_url).

{
  "join_key": "<street address|zip>",
  "building": "...",
  "form_url": "...",
  "method": "post",
  "form_type": "leasing",
  "fields": [{"name": "...", "type": "...", "required": true, "role": "email"}],
  "hidden": {"csrf_token": "...", "property_id": "..."},
  "has_captcha": false,
  "discovered_at": "ISO timestamp"
}

Zero extra requests. Forms are harvested from the same homepage fetch already used for the commission-page probe, so one crawl pass produces both commission evidence and the form registry.
Shared conversation store. Form submissions are logged into the same communications ledger as email/SMS/WhatsApp, tagged channel="web_form", so replies flow through the identical reply-extraction → building-writeback loop as every other channel. Submission is dry-run-gated by default and never attempts to solve a CAPTCHA.

Sending discipline (the "2-message rule")¶

Per contact: one initial message, at most one follow-up after roughly 4–5 business days, then stop permanently. Volume is never escalated in response to non-response.

A non-response is treated as data — a "measured non-responder" — not a failure. If every contact in a batch goes silent, that is read as a deliverability problem with the sending infrastructure, not as a genuine finding, and prompts rotating infrastructure rather than re-sending.

Sender infrastructure¶

The sender pool is defined in config/senders.csv (gitignored; config/senders.example.csv is the template), with columns: name, email, contact, smtp_host, smtp_port, smtp_user, smtp_pass, smtp_security, imap_host, imap_port, imap_user, imap_pass, area, company, style.
Each sender is health-scored on bounce/error rate; a degraded sender is quarantined and traffic shifts to a warm backup.
New mailboxes go through a warmup ramp (outreach/warmup.py) — sends to a private seed list on a linearly ramping schedule, roughly 2 → 20 sends/day over a few weeks, to build deliverability reputation before going live.
Every target's mail server is MX/SMTP-verified before sending, so reputation is never spent mailing a dead inbox. A do-not-contact suppression list is checked as well.

Drafting¶

Message copy is either static templates or LLM-drafted (via NVIDIA NIM, Google Gemini, or OpenRouter-compatible APIs), in two modes:

locator — generic outreach.
with_lead — references a specific prospective tenant.

Safety checks applied before every send: MX verification, a business-hours send window, randomized jitter, and sender health-score gating.

Reply handling¶

A reply daemon polls IMAP continuously (roughly every 5 minutes on the VPS). It:

Ingests new messages into the shared communications log and marks them read.
Matches an inbound reply to its outbound thread by (our address, their address).
Drafts a contextual LLM reply.
Extracts any commission terms mentioned.
Writes a verdict back to the building's master record when confident.

Every message logs a status (sent / failed) plus an error field on failure — a failed send is recorded as a failed row, not silently dropped, and rolls up into daily send-failure metrics.

Fair-housing / paired-testing note¶

If ever used for protected-class paired testing

If this infrastructure is ever run as a correspondence audit — matched personas differing only by a protected-class signal (e.g. race/ethnicity), used to test for discriminatory treatment — rather than for ordinary commission-discovery outreach, that is a recognized fair-housing testing methodology with real legal weight. It requires holding every variable constant except the tested signal, never making a fraudulent offer or signing anything, and running under or alongside a fair-housing organization or IRB. Testing tied to Fair Housing Act enforcement carries legal protections a lone actor does not have.

Ordinary B2B commission-discovery outreach — no protected-class signal — is not implicated by this and is normal use of the same infrastructure.