Why I Track What I Got Wrong More Carefully Than What I Got Right
Nightwatch has been running overnight tasks for about six months.
In that time, it has completed several thousand work units, surfaced dozens of useful findings, and caught at least four issues that would have cost real time to diagnose in the morning. I track all of this in a log.
I also have a separate log for what went wrong.
The second log is the one I actually read.
What the Failure Log Catches That the Success Log Doesn't
When something works, the log entry looks like this: task completed, output verified, duration within expected range. Those entries are useful for confirming the system is running. They tell me nothing about where it is fragile.
When something fails, the entry tells me something I did not already know. A task that was expected to complete in 45 seconds ran for 7 minutes before hitting a timeout. A web scrape returned empty content on a URL that has been stable for months. An LLM call produced an output format that the parsing layer did not expect, and the whole downstream chain stalled.
Each failure is a probe into the system's assumptions. The successful runs confirm that the assumptions held. The failures reveal where the assumptions were wrong.
This distinction matters more as the system becomes more complex. When Nightwatch was doing simple tasks — check this URL, fetch this data, write this file — failures were obvious and their causes were usually obvious too. As the task graph has expanded, the failure modes have become subtler. The log is often the only way to distinguish between "this broke in a way I need to fix" and "this broke because the external environment changed and I need to adapt."
The Operational Discipline That Actually Scales
There is a specific practice that I have found separates automation that compounds value over time from automation that degrades.
The practice: every failure mode, once identified, gets classified and categorized before it gets fixed.
The instinct is to fix immediately. The bug is obvious, the fix is obvious, and the system will be back to normal in fifteen minutes. But skipping the classification step means the system never gets smarter about why things fail — only that they do.
The categories I use:
External environment failures — the system was correct, but the world changed. A third-party API changed its response format. A website restructured. A rate limit tightened. Fix: update the integration, add a monitoring check, accept that this will happen again.
Assumption failures — the system was built on a premise that turned out to be wrong more often than expected. I assumed a particular data field would always be populated. I assumed a parsing step would be fast enough to complete within a timeout. I assumed an LLM would return structured output reliably. Fix: find the assumption, test it explicitly, redesign the step to handle the failure case.
Cascade failures — one component failed cleanly, but the downstream handling was not designed for the failure case. The error propagated in unexpected ways. Fix: tighten the error boundaries, add explicit handling at each dependency point.
Most failures I encounter are assumption failures. Most of the interesting work in building reliable automated systems is finding and testing your own assumptions.
What I Actually Changed After Reading the Failure Log
Six months of failure data has changed three things about how I build.
I add explicit failure paths before I add features. Before a new capability ships, I now ask: what does this component do when its dependency fails? What does the downstream handler receive? If I cannot answer those questions specifically, the feature is not ready. This was not how I built the first version of anything.
I instrument anomalies, not just errors. An error is something that breaks explicitly. An anomaly is something that completes successfully but produces output outside the expected range. A task that normally takes 30 seconds and ran in 4 minutes did not fail — but it is telling me something. Tracking anomalies surfaces fragility before it becomes failure.
I separate monitoring from alerting. Everything gets logged. Almost nothing triggers an immediate alert. I review the log in the morning with fresh context rather than reacting to alerts at 2am. This keeps the overnight automation from training me to be reactive in the middle of the night, which defeats the point of having overnight automation in the first place.
The failure log has become more useful than the success log because failures are where the system's edges live. Successes tell me the system is working. Failures tell me what the system is.
Every stable automated system I have built has gotten to stability the same way: not by preventing all failures, but by building a faster and more accurate loop for understanding what each failure means.