Skip to content

Infrastructure

Where everything runs, what talks to what, and why the few odd-looking pieces are shaped the way they are. Current as of 2026-08-06, after client-api and internal-api moved off Vercel onto our own droplet.

Runtime topology

flowchart TB
    subgraph clients[" "]
        relive["<b>Relive</b><br/>bulk sync client"]
        team["<b>Our team</b><br/>browser"]
    end

    subgraph edge["DigitalOcean droplet · 143.198.20.37 · 2 vCPU / 4 GB"]
        traefik["<b>Traefik</b><br/>TLS termination<br/>Let's Encrypt"]
        capi["<b>client-api</b><br/>api.foundationaldatalabs.com<br/>27 MiB"]
        capidev["<b>client-api-dev</b><br/>api-dev.…"]
        iapi["<b>internal-api</b><br/>internal.foundationaldatalabs.com"]
        daemon["<b>outreach daemon</b><br/>+ 7 cron jobs<br/>329 MiB"]
        pgb["<b>pgbouncer</b><br/>transaction pooling"]
        devpg[("<b>dev Postgres</b><br/>seeded sample<br/>vol: fd-dev-pgdata")]
        coolify["<b>Coolify</b><br/>coolify.foundationaldatalabs.com"]
    end

    subgraph external["Elsewhere"]
        frontend["<b>foundational-data-frontend</b><br/>Vercel"]
        dashboard["<b>dashboard</b><br/>Cloudflare Workers"]
        vmail["<b>internal-api /mail</b><br/>Vercel — SMTP egress only"]
        migadu["<b>Migadu</b><br/>SMTP 465 · IMAP 993"]
    end

    warehouse[("<b>DO Managed Postgres</b><br/>the system of record<br/>max_connections 25")]

    relive -->|"X-API-Key"| traefik
    team --> frontend
    team --> dashboard
    team --> coolify

    traefik --> capi
    traefik --> capidev
    traefik --> iapi

    frontend -->|"X-Internal-Key<br/>server-side proxy"| iapi
    dashboard -->|"X-Internal-Key"| iapi

    capi -->|"client_api_ro · SELECT only<br/>client_api_rw · 2 grants"| pgb
    pgb --> warehouse
    capidev --> devpg
    iapi -->|"DATABASE_URL · writable"| warehouse
    daemon -->|"DATABASE_URL · writable"| warehouse

    daemon -->|"HTTPS relay<br/>(SMTP blocked here)"| vmail
    vmail -->|"SMTP 465"| migadu
    daemon -->|"IMAP 993 — not blocked"| migadu

    classDef dropletBox fill:#0b3d5c,stroke:#1b7fb8,color:#fff
    classDef ext fill:#3d2b56,stroke:#8b5fbf,color:#fff
    classDef db fill:#14532d,stroke:#3f9e5a,color:#fff
    class traefik,capi,capidev,iapi,daemon,pgb,coolify dropletBox
    class frontend,dashboard,vmail,migadu ext
    class warehouse,devpg db

Why /mail is still on Vercel

DigitalOcean blocks outbound TCP 25, 465 and 587 droplet-wide and declined to lift it (ticket 2026-07-28). Migadu's submission port is 465. So the daemon cannot open an SMTP socket at all — but IMAP on 993 is unaffected, which is why reply polling works from the droplet while sending does not.

The workaround predates the migration: the daemon builds the full MIME message locally (Message-ID, In-Reply-To, List-Unsubscribe — everything reply threading needs) and POSTs it to internal-api's POST /mail?resource=relay, which submits it verbatim from Vercel, where there is no port block.

That is the one endpoint that cannot come home. Everything else in internal-api moved. Same codebase, two deployments, different consumers — the droplet copy simply never receives MAIL_ACCOUNTS.

flowchart LR
    daemon["outreach daemon<br/>(droplet)"]
    x1["✗ SMTP 465/587<br/>blocked by DigitalOcean"]
    vmail["internal-api /mail<br/>(Vercel)"]
    migadu["Migadu"]

    daemon -.->|blocked| x1
    daemon -->|"HTTPS POST<br/>finished MIME bytes"| vmail
    vmail -->|"SMTP 465"| migadu
    migadu -->|"IMAP 993 · replies"| daemon

    classDef bad fill:#5c1a1a,stroke:#c0392b,color:#fff
    class x1 bad

Deployment

Every merge deploys itself. Coolify holds a GitHub App on the PDGTeam organisation, so a push produces a webhook, a build on the droplet, and a container swap — no personal account involved, and any team member can trigger or roll back.

flowchart LR
    dev["developer"] -->|"feature branch"| pr["Pull Request"]
    pr -->|merge| develop["<b>develop</b>"]
    develop -->|merge| main["<b>main</b>"]

    develop -->|"GitHub App webhook"| cdev["client-api-dev<br/>api-dev.…<br/><i>dev database</i>"]
    main -->|"GitHub App webhook"| cprod["client-api<br/>api.…<br/><i>production database</i>"]
    main -->|"GitHub App webhook"| iprod["internal-api<br/>internal.…"]

    classDef prod fill:#0b3d5c,stroke:#1b7fb8,color:#fff
    classDef devc fill:#4a3b10,stroke:#c9a227,color:#fff
    class cprod,iprod prod
    class cdev devc

develop must never fall behind main — a feature branched from a stale develop starts from code production has already moved past. Merge main into develop after every release rather than rebasing: develop is shared, and rebasing rewrites history that teammates have already pulled.

The outreach daemon is the exception: it still deploys from an SSH deploy key with no webhook, so its deploys remain manual.

Data plane and credentials

Three different Postgres identities reach the warehouse, with deliberately different power.

flowchart LR
    capi["client-api"]
    iapi["internal-api"]
    daemon["outreach daemon"]
    pgb["pgbouncer<br/>fd-pgbouncer:5432"]
    wh[("DO Managed Postgres<br/>max_connections 25<br/>3 superuser-reserved")]

    capi -->|"DATABASE_URL_RO<br/><b>client_api_ro</b><br/>SELECT only<br/>read-only transactions"| pgb
    capi -->|"DATABASE_URL_RW<br/><b>client_api_rw</b><br/>2 grants on refresh_policy"| pgb
    pgb -->|"max_db_connections 12"| wh
    iapi -->|"DATABASE_URL<br/><b>writable</b>"| wh
    daemon -->|"DATABASE_URL<br/><b>writable</b>"| wh

    classDef db fill:#14532d,stroke:#3f9e5a,color:#fff
    class wh db

The client-facing surface never holds a credential that can write the store of record. client-api builds its read client from DATABASE_URL_RO and refuses to serve without it — there is no fallback to a writable DSN, by design.

The connection budget is small

The cluster allows 25 connections, 3 of them superuser-reserved. pgbouncer's default_pool_size is per (database, user), so two roles alone could claim 20. max_db_connections = 12 is what caps the total and makes max_client_conn = 100 safe — removing it re-opens the exhaustion path.

This is also why the development database is a container on the droplet rather than a database on the managed cluster: dev spends zero cluster connections, so a load test in dev cannot starve the client API.

Hostnames

hostname serves notes
api.foundationaldatalabs.com client-api Relive's bulk sync. TTL 600 for fast rollback
api-next.foundationaldatalabs.com client-api same container — the pre-cutover candidate hostname
api-dev.foundationaldatalabs.com client-api-dev dev database, develop branch
internal.foundationaldatalabs.com internal-api the frontends' only backend
coolify.foundationaldatalabs.com Coolify deploy panel — public, so 2FA is mandatory
internal-api-ten.vercel.app internal-api /mail SMTP egress only; nothing else points here

Known gaps

Recorded rather than glossed over.

  • Single point of failure. One droplet now serves the client API, the outreach daemon, both databases' proxies and the deploy panel. Leaving Vercel traded edge caching and DDoS absorption for control. Cloudflare in proxied mode in front of api.foundationaldatalabs.com would restore both and give an instant kill-switch independent of DNS TTL.
  • No alerting. If the droplet or a container dies, nothing tells anyone. The healthchecks restart containers but no one is paged.
  • The dev database is not backed up. Deliberate — it is rebuilt from deploy/dev-db/seed.sh in seconds — but it does mean uncommitted dev state is disposable.
  • The Coolify panel is on the public internet, protected only by its own login. 2FA is on; SSO in front of it (Cloudflare Access) would be better.
  • The outreach daemon still deploys manually, on a deploy key rather than the GitHub App.