I Thought I Had Three Weeks of Data. One GROUP BY Said Ten Minutes.

The collection pipeline had been "running since August 5th." Three weeks of one-minute market data, accumulating quietly on my laptop. Then I ran one sanity query before a migration — and watched three weeks shrink to ten minutes.

Start here: A pipeline that ran is not a pipeline that's running. My setup had no supervisor, it lived in a terminal session, and it wrote to a database inside Docker Desktop on a laptop. Yes, that is three layers of session software, each of which dies silently when you close a lid. The fix is three processes, in order of cheapness: (1) query the data, not the process — max(time) is a one-line monitoring system; (2) supervise anything that must survive your absence (launchd/systemd, not a terminal tab); (3) restore-test your backups before you need them.

The setup

A solo-built market data platform: a Rust binary subscribes to an FX broker's tick stream, aggregates 43 instruments into one-minute bars with microstructure stats (spreads, tradeable counts, mid-price variance), and hands them to a Python consumer that bulk-inserts into Postgres. The database runs in Docker Desktop on my laptop — an explicitly interim home while the permanent hardware was commissioned.

Historical prices you can backfill anytime. Intra-minute microstructure exists only if you were listening when it happened — every day the stream is down is permanently unrecoverable. So the day the pipeline first wrote bars, I noted it as an epoch event: the clock started August 5th.

Then I got busy on other parts of the build. The pipeline was "running."

The symptom

Three weeks later, preparing to migrate the database to its permanent machine, I opened Docker to check on things and got this:

CONTAINER ID   IMAGE                STATUS
a1b2c3d4e5f6   postgres:16-alpine   Up 18 seconds (healthy)

Created 3 months ago. Up 18 seconds — because Docker Desktop had just started, because I had just started it, because it hadn't been running before I looked. Meanwhile ps aux | grep stream_bars returned nothing. The collector wasn't running either.

At this point I still believed the optimistic version: the pipeline ran for a good while, then "Docker kept crashing" took it down somewhere along the way. Some data lost, but most safe. The logs even offered a villain — a wall of FATAL: database "db_name" does not exist errors repeating every five seconds, loud and red and completely irrelevant (a healthcheck probing a database name that never existed — config drift, harmless to data). The loud error was innocent. The real problem hadn't logged anything, because the real problem was silence. The cause is rarely standing where the symptom is — the same shape as the prose leak, where the broken component was innocent and the context was the culprit.

The diagnosis

The migration plan started with a row count:

SELECT count(*), min(time), max(time) FROM ohlcv_1min;
-- 252 bars · 2026-08-05 11:55 UTC → 2026-08-06 03:34 UTC

252 bars. Across 43 instruments. First red flag: that's ~6 bars per instrument over what the timestamps framed as 15½ hours — EUR/USD alone should produce hundreds. Second red flag: the window ended August 6th, not "sometime recently." Three weeks of assumed history was already down to fifteen hours. Then the query that ended the mystery:

SELECT date_trunc('hour', time), count(*) FROM ohlcv_1min GROUP BY 1;
--  2026-08-05 11:00 |  88
--  2026-08-05 12:00 |   1
--  2026-08-06 03:00 | 163

And with SELECT DISTINCT time — ten distinct minutes. Total. Aug 5, 11:55–12:00. Aug 6, 03:30–03:34. Ten minute-bars × ~43 instruments = 252 rows. The arithmetic closed perfectly, and the story it told was not "pipeline crashed after fifteen hours." It was: the pipeline ran exactly twice, for five minutes each, while I was watching it. Two test runs — I started it in a terminal, watched the bars land, felt great, and moved on. The process died when the terminal closed or the laptop slept, both times. There was never a third run.

Nothing crashed. Nothing failed. Nothing was ever deployed — and the difference between "I ran it and it worked" and "it is running" is the entire discipline of supervision, of which I had built zero.

The fix — three tiers, in order of cheapness

Tier 1 — monitor the data, not the process. Every check I could have pointed at the process (is the PID alive? is the container up?) answers the wrong question. The question is did data arrive? — and the table itself answers it in one line:

SELECT max(time) FROM ohlcv_1min;  -- if this isn't recent, nothing else matters

A bars-per-hour completeness query (the GROUP BY above) run on a schedule, with an alert when the current hour is short, would have caught this on day one instead of day twenty-four. It costs nothing. It's the first monitoring primitive the platform now has — born as an autopsy tool.

Tier 2 — supervise anything that must outlive your attention. Terminal sessions, Docker Desktop, anything that starts at login: session software. It dies with the session, silently, and a laptop is a machine designed to end sessions — lids close, updates reboot, batteries drain. The fix is boring and decades old: a real supervisor (launchd on macOS, systemd on Linux) with restart-on-failure, running on hardware whose job is to remain powered. "It's temporary" is not an exemption — my interim setup outlived its first failure by three weeks because nothing was watching.

Tier 3 — restore-test the backup, then trust it. Before touching anything, pg_dump the survivors and actually restore the dump somewhere else, counting rows on the far side. A backup that has never been restored is a hope, not a backup. Mine restored clean — 252 rows, exactly as sad as the original.

The rule of thumb

If a human started it, then a human's logout will kill it. If you haven't queried the data since you deployed it, you don't know it's running — you remember it running, which is a different thing. And when something breaks: the loudest error in the log is usually a bystander; interrogate the silence instead.

I'll confess the part that stings: the epoch note. I wrote down the day the clock started — August 5th, 11:46 UTC — like a foundational ceremony, because for this dataset, history only exists if you're documenting it. Then I spent three weeks not listening to the listener. The data I was so careful to timestamp stopped nine minutes after the ceremony, and I found out from a GROUP BY twenty-four days later. The clock I actually started was the one measuring how long a solo builder can believe in a pipeline nobody is watching. Answer: about three weeks. LOL!

The pipeline worked end-to-end, twice, flawlessly. That was the trap — "it works" and "it's running" are different claims, and only one of them shows up in a demo. The database knew the truth the whole time. All I had to do was ask it.

FAQ

Wasn't the real bug the Docker crashes? No — that's the misdirection for which this post exists. Docker Desktop stopping when the laptop reboots isn't a crash; it's the documented lifecycle of session software. The bug was architectural: no component anywhere owned the job of keeping the pipeline alive.

Why not just check the logs? The consumer's logs died with its terminal. The database logs were full of a loud, harmless red herring (a healthcheck probing a wrong database name). Logs tell you what happened; only the data tells you what didn't. max(time) interrogates the absence.

What actually catches this in production? Freshness checks (max(time) vs. now) and volume checks (rows per window vs. expectation), scheduled, with alerts. Process supervision keeps things running; data-level checks tell you when "running" stopped meaning "working." You want both, and if you can only have one, take the data check.

Did you lose anything irreplaceable? Three weeks of intra-minute microstructure that can never be backfilled — the one data class where absence is permanent. The consolation: it happened during an interim window before the data carried production load, which made it the cheapest possible version of this tuition.

Further reading

• launchd (macOS service management — KeepAlive is the property at the heart of this post): https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html
• Docker Desktop's lifecycle vs. server daemons (why containers "stop on their own" on laptops): https://docs.docker.com/desktop/
• PostgreSQL pg_dump custom format + pg_restore (the restore-test workflow): https://www.postgresql.org/docs/16/app-pgdump.html