Integration Monitoring for Engineering Teams
The hard part is not detection. It is that the alert reaches somebody who knows what the thing does and can change it.
Once more than three people touch a codebase, monitoring stops being a detection problem and becomes a routing problem. The alert fires correctly and reaches someone who does not know what the job does, has no permission to fix it, and forwards it into a channel where it sits.
Ownership belongs next to the code
An alert that names a service without naming an owner requires somebody to work out who to ask, and at 2am that step is where the time goes.
The pattern that works is the one you already use for code review: a declared owner per component, kept in the repository, updated in the same pull request that changes the component. If your monitor definitions live in a UI that nobody diffs, they drift from the code within a quarter.
That severity field does more for alert quality than any threshold. Most alerts should create a ticket rather than wake somebody, and forcing the author to choose at the time of writing prevents everything defaulting to page.
Runbooks written for the person who has never seen it
A runbook that says restart the worker is a note to yourself. The reader is someone on call for a service they did not build, at an hour when they are not at their best.
Four things make one useful: what this job does in one sentence, what breaks downstream when it stops, the three most common causes with their fixes, and who to escalate to if none of those apply. Anything longer will not be read under pressure.
Link the runbook from the alert itself. A runbook nobody can find during an incident is documentation, not operations.
Coverage reviews, because coverage rots
Monitoring degrades quietly as a codebase changes. A job gets renamed and its monitor keeps checking an endpoint that no longer exists, reporting healthy forever. A new integration ships with no monitor because the person who knew the convention was on leave.
| Drift | Symptom | How to catch it |
|---|---|---|
| Job renamed, monitor not | Monitor green, job gone | Reconcile monitor list against scheduler |
| New integration, no monitor | Silent gap | Checklist item in the PR template |
| Owner left the company | Alerts to nobody | Validate owners against the directory |
| Threshold never revisited | Noise, then muting | Review anything that fired falsely twice |
Run that in CI. An unwatched job should be a visible warning on the pull request that introduced it, not a discovery made during an incident six months later.
Post-incident, ask the detection question
Most retrospectives focus on the cause. The more useful question for monitoring is how long the problem existed before anyone knew, and whether any existing alert could have caught it sooner.
If the answer is that a customer told you, that is a coverage gap regardless of how quickly you fixed the underlying bug. Track time-to-detection as a number alongside time-to-resolution and it will improve; leave it unmeasured and it will not.
Alert on absence, not only on errors
Teams with mature error tracking still miss the job that stopped running, because error trackers catch thrown exceptions and a job that never started throws nothing.
That gap is what heartbeat monitoring closes, and it is covered in heartbeat monitoring. SensaCat handles the absence side alongside credential and certificate expiry. Start free.