Back to sensacat

Home  /  Troubleshooting

· SensaCat Team

Credential Rotation Without Downtime

The overlap window is the whole technique. Everything else is making sure you found every consumer before you close it.

Rotate in four steps with an overlap window in the middle: create the new credential, distribute it everywhere while the old one still works, verify nothing is using the old one, then revoke. Skipping the verification step is what turns a routine rotation into an incident.

The order that matters

Step Action Old credential Risk if skipped
1 Create the new credential Still valid None
2 Deploy the new value to every consumer Still valid None; this is the safe window
3 Verify the old one has no traffic Still valid Revoking blindly breaks what you missed
4 Revoke the old credential Dead Leaving it live is the security hole

Step 3 is the one that gets dropped under time pressure, and it is the only step that tells you whether steps 1 and 2 actually worked.

Not every provider gives you an overlap

Check before you plan the rotation, because the answer changes the whole approach.

Provider type Overlap support Approach
AWS IAM access keys Two active keys per user Classic four-step rotation
Stripe restricted keys Multiple keys can coexist Create, migrate, delete
Webhook signing secrets Often verifiable against several Accept both during cutover
Single-key SaaS APIs None: one key at a time Hard cutover, plan a maintenance window
OAuth client secrets Often two, check the provider Add secondary, migrate, remove primary

For webhook signing secrets, verifying against a list of secrets rather than one is the pattern. Accept either during the window, then drop the old one.

Finding every consumer, which is the actual work

A rotation fails because of the place nobody remembered, not because of the mechanics. Search deliberately rather than relying on memory.

The consumers that get missed are consistent everywhere: background workers deployed separately from the web application, scheduled jobs on a different host, CI secrets used by nightly pipelines, serverless environment variables, a second region, a staging environment pointed at production, and a partner or contractor holding the key outside your infrastructure.

That last one has no technical fix. If an external party holds the credential, rotation requires coordinating with them, and that has to happen before step 4 rather than after.

Verifying the old credential is idle

Where the provider exposes a last-used timestamp, this is straightforward and definitive.

Where there is no last-used field, you have to infer it. Watch your own outbound request logs for the old value, or if the provider supports scoped keys, create the new key with a distinguishable identity and watch the provider's audit log for which one is calling.

Wait at least one full cycle of your least frequent job before revoking. If something runs monthly, a rotation verified over three days has not tested it. This is the single most common cause of a rotation that breaks something four weeks later.

Rotating in an emergency

A leaked credential does not get an overlap window, because the whole point is to invalidate it now. Accept the outage, revoke first, then work through consumers with the incident channel open.

Decide in advance which credentials fall into this category and write down where each one is used. Compiling that list while the key is public on GitHub is the worst possible time to start.

Making the next one boring

Keep a record per credential of where it is used, who owns it, when it expires and what breaks without it. That document is what turns rotation from an investigation into a checklist.

Tracking those dates and owners is what SensaCat's credentials module is for, and the wider discipline is covered in credential expiry monitoring.