Argus
Distributed multi-region service monitor: three checkers on three continents vote on whether your service is actually down before anything pages you.
- Replaced per-checker alerting with 2-of-3 majority consensus over a sliding 90-second window, so one checker on a bad network path cannot page anyone — a checker dropping 33% of its requests produced zero false alerts.
- Serialized per-monitor evaluation with Postgres advisory locks inside a single transaction, chosen non-blocking so contention skips rather than queues, and the lock releases on commit rather than needing cleanup.
- Inserted a four-state machine (up / degraded / down / recovering) between raw verdicts and notifications, collapsing 199 consensus edges into 0 alerts on sub-threshold flapping while still catching every sustained outage.
- Detected latency anomalies online with per-checker EWMA baselines and z-scores, distinguishing a service-wide slowdown (two of three paths) from one slow region, so a bad transit route is recorded rather than paged.
- Reconstructed uptime from an append-only transition log with half-open interval arithmetic, excluding scheduled maintenance and gaps where monitoring itself was down.
- Made alert delivery crash-safe with a transactional outbox and a pg-boss retry queue, keeping network I/O out of the locked transaction entirely.
- Sustained ~500 req/s ingestion on a single instance at p99 81 ms, behind ~210 tests including integration tests against real Postgres via Testcontainers.
TypeScriptNode 24FastifyPostgreSQLDockerGitHub Actions