Upstream system
Sentinel archive
Day-partitioned HDFS from two hosts
Technical overviewMachine learning and observability
Anomaly Detection
An isolation forest that ranks machine telemetry better than a three-sigma rule, and still loses on F1 to flagging everything.
Source code
Inspect the implementation, commit history, and project documentation.
- Role
- Solo engineer: dataset, labels, baseline, model, evaluation, served API
- Evidence status
- Public repository with a held-out evaluation that reproduces under continuous integration; the containerised scoring service is deployed on a k3s cluster and live at anomaly.pesanth.com
- Verified
- 2026-08-30
01
Purpose and scope
An unsupervised detector over the telemetry Sentinel already archives, built to be scored honestly. On 211 held-out windows the isolation forest ranks better than a rolling three-sigma baseline (ROC AUC 0.622 vs 0.586), yet its F1 of 0.611 loses to flagging every window (0.762) on a 61.6% positive period. Both numbers stay published, with the trivial detectors kept in the table.
What it is used for
- Surface an unusual stretch of machine behaviour without a pre-written rule.
- Show which measurements drove a decision, so an operator can agree or not.
- Measure a model against named baselines, not a single score alone.
02
Architecture
Source
Extraction
Slice exporter
Merges inside the NameNode, copies one file out
Dataset
Pinned slice
17,813 readings over six days, checksummed
Label · from evidence outside the telemetry
Labelling
Evidence
Run ledger
When each scan started; knows nothing of the CPU
Derivation
Label seeder
Bounds runs by artefact times, leaves ambiguous unlabelled
Ground truth
Label file
565 of 591 windows labelled, each with a reason
Window · 5 minutes, no overlap
Modelling
Transformation
Feature builder
26 features, handling resets and reboots
Comparator
Three-sigma baseline
Rolling mean and stdev, reset across gaps
Detector
Isolation forest
Unsupervised, fitted before the test period
Result
Evaluation report
Metrics for every detector, plus a shift table
Score · same feature code as training
Serving
Interface
Scoring API
Returns score, threshold, decision, top features
Packaging
Container image
Builds the model, runs unprivileged
Enforcement
Pipeline gates
Secret, dependency, image, config scans
03
Engineering decisions
Take the labels from outside the signal
A window is positive when a known job was running, established from a run ledger and file timestamps, so the score is not measured against a spike detector reading the same telemetry.
Split on a gap, not on a percentage
The train and test cut snaps to a hole in the archive rather than a fixed percentage, so no single event straddles both sides.
Publish the trivial detectors beside the model
The period is 61.6% positive, so flagging everything scores F1 0.762; keeping that row forces the honest reading, the ranking metrics a constant detector cannot game.
04
Verification evidence
- The held-out evaluation reproduces on demand and is committed in machine-readable form, with a pipeline job that fails if the numbers stop matching.
- 33 automated tests cover window construction, counter resets, the baseline's gap handling, and the API's rejections.
- A sensitivity run dropping the worst-shifted feature raises ROC AUC to 0.645, so the result does not rest on an artefact.
$ python tools/report.py
held-out · 211 windows · 130 positive · 61.6%
3-sigma F1 0.320 ROC 0.586 AP 0.666
forest F1 0.611 ROC 0.622 AP 0.725
flag-all F1 0.762 ROC 0.500 AP 0.61605
Limitations
- The model ranks better than the baseline yet its F1 loses to flagging every window; both halves have to travel together.
- The deployed service serves a model frozen on the pinned six-day slice and does not retrain on new telemetry.
- One machine, six days, about 49 hours of readings; a second host was deliberately excluded.