The watchman doesn't know who you are — when integrity monitoring points at the administrator
The alarm that pointed at me
On Saturday night I reworked a maintenance script that is part of my pipeline. Common task: improve the routine, save it, move on. Since this script lives in a directory watched by my file integrity monitor, I followed the very process I had defined: I updated the hash baseline in the same work block, with a backup, and checked the result.
Six hours later, at four in the morning, the integrity alarm fired. The cause: the same script, at the same path, with exactly the change I had just made and documented.
Nobody broke in. No file was corrupted. The watchman was pointing at the owner of the house.
What a baseline really is
An integrity monitor works on a brutally simple premise: there is a known good state, and anything different from it is suspicious.
# The baseline is just a list of fingerprints
sha256sum /home/me/.hermes/scripts/*.sh > baseline.sha256
# The monitor recalculates and compares
sha256sum -c baseline.sha256
# watched.sh: FAILEDThe problem is that this premise has no notion of author. The monitor can’t tell the difference between:
- an attacker who replaced the script with a backdoored version;
- the administrator himself who made a legitimate improvement at eleven at night.
To the hash, both cases are identical: today’s file differs from the baseline file. And that’s the point. A monitor that “knew” the admin edits things now and then would be a monitor that can be talked out of alarming — exactly what an attacker would try to do.
The rule I had, and where it failed
The process I wrote myself says: every edit to a watched file updates the baseline in the same block. On Saturday I followed the rule to the letter. And the alarm still fired six hours later.
When I dug in, I found the trap: there were two repair routines running in the pipeline — one that re-runs housekeeping tasks when a service restarts, and another that watches clocks and memory to keep everything up. Both ran roughly in the same window. The one that re-runs tasks also keeps a copy of the baseline. And when it ran after my edit, it re-applied the old state on top of my re-baseline — rewriting the script with the old content, or restoring the old baseline over the new one, depending on which copy won the race.
The practical result: my re-baseline lasted minutes. At four in the morning, the monitor looked, saw the divergent hash, and went off. Correctly.
The diagnosis that works for any pipeline
Three lessons came out of this incident, all transferable:
1. Re-baselining in the same block is necessary, but not sufficient. If another routine in your pipeline restores state (and almost every mature pipeline has some self-repair routine), your re-baseline can be undone minutes later. The rule needs to be: re-baseline and verify, ten minutes later, that the hash still matches. If it doesn’t, the problem is no longer the edit — it’s the race between routines.
2. Every automatic restore needs to know where it restores from. A self-repair routine that restores from an old copy is a machine for undoing legitimate work. I now treat any restore source as part of the baseline’s scope: if it can rewrite watched files, it must be baseline-aware, or at least never restore something the baseline just absorbed.
3. The false alarm wasn’t the mistake — the missing process was. The temptation here is “add an admin exception”. That would be the same mistake as the silenced alarm: quieting the watchman instead of defining scope. I did the opposite — I let the alarm keep screaming (it screamed right!), and I fixed the process that produced the divergent state.
The current protocol
After the incident, the editing flow looks like this:
# 1. Edit the watched script
nano ~/.hermes/scripts/routine.sh
# 2. Re-baseline in the SAME block, backing up the previous one
cp baseline.sha256 baseline.sha256.bak
sha256sum /home/me/.hermes/scripts/*.sh > baseline.sha256
# 3. Late confirmation — the step that was missing
sleep 600
sha256sum -c baseline.sha256 > /dev/null 2>&1 \
&& echo "re-baseline stable" \
|| echo "CONFLICT: another routine restored old state"Step 3 is what turns a re-baseline from an action into a verified process. Without it, I had a window of a few hours where the real state (restored by the other routine) and the expected state (my baseline) diverged in silence — and the only one who noticed was the watchman, at four in the morning.
Lessons learned
- An integrity monitor has no notion of authority — and it shouldn’t. The separation between admin and attacker lives in the process, never in the watchman.
- Self-repair without baseline awareness is a machine for undoing legitimate work. Map every pipeline routine that restores state.
- Re-baselining without late confirmation is just hope. Ten minutes later, confirm the hash still matches.
- The alarm that wakes you at 4 AM pointing at your own edit from yesterday isn’t being petty: it’s telling you there’s a race between routines you didn’t know about.
The next evolution I want to try: making the late confirmation itself part of the re-baseline routine, with the alarm silenced only by the verification window — never by the identity of whoever edited.