Technical overviewData engineering and observability

Sentinel

A streaming telemetry pipeline from host collectors to a day-partitioned archive.

Source code

The repository is private. The architecture, engineering decisions, and verified evidence are documented below.

Role: Solo engineer across collection, streaming, storage, deployment, and the public interface
Evidence status: Live at sentinel.pesanth.com; collection, streaming, storage, and the dashboard are running, while baseline analytics and automated investigation are planned
Verified: 2026-08-15

Purpose and scope

Sentinel collects operational telemetry from two self-hosted machines and moves it through Kafka into a day-partitioned HDFS archive that a public dashboard reads back. Collectors publish system, container, and endpoint readings every ten seconds, buffer to local disk whenever the broker is unreachable, and replay in order once it returns. It is live at sentinel.pesanth.com.

What it is used for

Answer whether a server was actually down, using recorded history rather than a status page.
Track processor temperature, memory pressure, and disk headroom across hosts over time.
Confirm that the public sites respond, and how quickly, from more than one vantage point.
Retain a durable record that outlives stream retention, so later analysis has history to work with.

Architecture

Collection

Agent

Server collector

Runs under systemd as an unprivileged user with a strict filesystem policy

Agent

Workstation collector

Registered as a scheduled task that starts at logon

Durability

Disk spool

Bounded local buffer holding readings the broker could not accept

Publish · keyed by host

Transport

Message broker

Kafka broker

Single node in KRaft mode with separate listeners for local, container, and remote clients

Streams

Reading topics

System metrics, container inventory, and endpoint checks, each on its own topic

Consume · commit after write

Storage

Consumer

HDFS sink

Batches readings and commits stream offsets only once the files land

Coordinator

HDFS NameNode

Namespace and block metadata for the archive

Day partitions

Immutable newline-delimited JSON, partitioned by the day each reading was taken

Read · WebHDFS

Presentation

Web interface

Dashboard

Host cards, endpoint checks, and archive coverage, each row carrying its own age

Public ingress

Cloudflare Tunnel

Outbound-only ingress publishing the dashboard without opening a router port

Text equivalent: Collectors on each host publish readings to Kafka over a private network, buffering to local disk when the broker is unavailable. A sink consumes every topic and writes newline-delimited JSON into HDFS partitioned by the day each reading was taken. A dashboard reads that archive and is published through an outbound-only Cloudflare Tunnel.

Engineering decisions

Partition by observation time

Readings are filed under the day they were taken rather than the day they arrived, so a collector that buffered through an outage restores the correct history instead of concentrating it on the day the broker returned.

Commit offsets only after the data is durable

The sink writes its partition files before committing stream offsets, so an interruption repeats a batch rather than losing it. Duplicates remain identifiable by host, kind, and timestamp; a missing interval would not be recoverable.

Buffer at the edge

The broker runs on a machine that is not always available, so each collector keeps a bounded local buffer and replays it in order. The buffer file is claimed by rename before a replay begins, which prevents an asynchronous producer from discarding readings whose failure has not yet been reported.

Serve the dashboard from storage

The interface reads the HDFS archive rather than the live stream. A page fed by the stream would look the same whether or not anything had been stored, so reading from storage makes the archive demonstrable. Readings appear one batch behind live, and every row shows its own age.

Verification evidence

75 automated tests pass without Kafka, HDFS, a network, or a container runtime, covering an unavailable broker, a saturated producer queue, a replay that fails a second time, a batch spanning two days, and a failed write leaving stream offsets uncommitted.
Edge buffering was verified by interrupting the broker: five readings were retained on disk with the process exiting non-zero, and all five were replayed with an empty buffer once the broker returned.
Durability was verified by restarting the platform: the archive grew from nine files to twelve, earlier files were intact, and only the readings accumulated during the interruption were rewritten.
On first start the sink drained the retained stream into 345 readings across all three topics, and HDFS reported one live data node against 1,006 GB of configured capacity.
The published dashboard returned both hosts with current readings and six endpoint checks at HTTP 200, and every existing tunnel hostname was rechecked at HTTP 200 after the routing change.

verified-2026-08-15

$ python -m collector --once
5 readings · 0 delivered · 5 failed
$ docker compose start kafka
10 delivered · 5 replayed from disk
✓ /sentinel/raw/system/dt=2026-08-15

Limitations

This is a single-node deployment measured in gigabytes rather than terabytes, with one HDFS data node and one Kafka broker.
The broker, the archive, and the dashboard run on a workstation, so the public page is unavailable while that machine is off. Collectors continue buffering through those windows and replay afterwards.
Baseline analytics and automated investigation are planned rather than built. The current interface reports readings and does not yet interpret them.
Container inventory is empty on the server, because the collector runs unprivileged and cannot query the container runtime. Granting that access would make the collector effectively privileged, so the gap was accepted instead.
Each batch produces its own file, so the archive accumulates many small files. Compaction into a columnar format is planned alongside the analytics layer.