API Key Expired and Broke Production
Before you issue a replacement, confirm it is expiry and not revocation, a scope change, or a quota. The fix differs.
Read the full error body before assuming expiry. A 401 and a 403 mean different things and lead to different fixes, and providers are inconsistent about which they return.
| Status | Typical meaning | Fix |
|---|---|---|
| 401 | Key invalid, expired, or revoked | Issue a new key |
| 403 | Key valid, permission missing | Fix scopes; a new key will not help |
| 429 | Rate or quota limit | Back off; nothing to do with the key |
| 400 invalid_grant | OAuth refresh token dead | User must re-authorize |
| Works in one region, fails in another | Key is region-scoped | Check where it was created |
The 403 case is worth pausing on. If somebody tightened a permission or a role rather than the key expiring, replacing the key produces a new key with the same missing permission and you will have burned an hour.
Rule out a rotation you did not do
Before issuing anything, check the provider's audit log or key list for a key created or deleted recently. A colleague rotating a shared key without telling anyone presents identically to expiry, and if you now issue a third key you will have two people deploying different values.
Find every consumer before you deploy
This is where outages get extended. The key is in more places than the one you are looking at, and each one you miss is a second incident an hour later.
The consumers that get missed are consistent across organisations: background workers in a separate deployment, scheduled jobs on a different host, a CI secret used by a nightly pipeline, a serverless function's environment variable, a Terraform variable, and a second region or staging environment that quietly talks to production.
Write the list down as you find it. That list is the inventory you should have had before today, and it is far easier to compile while you are already looking.
Deploy in an order that does not extend the outage
If the provider supports having two keys valid at once, create the new one and leave the old in place until every consumer is confirmed on the new value. Then revoke. Doing it in the other order turns a partial outage into a total one.
If the provider only allows one key, you have a hard cutover and the order is: update the configuration everywhere you can reach, then rotate, then verify each consumer in turn. Expect a short window where everything fails.
The full pattern is in credential rotation without downtime.
Verify you actually fixed it
Do not stop at the API responding in your terminal. Check the specific code path that broke, then check the ones you did not think about.
Why you had no warning
Because there is nothing to observe. A key that expires tomorrow behaves exactly like one that expires in three years, and no metric, log line or health check distinguishes them until the moment it stops working.
That makes this an inventory problem rather than a monitoring problem. Every credential needs a recorded expiry date, a named owner, and a note of what breaks when it lapses. SensaCat's credentials module exists for that list; a spreadsheet with reminders also works, provided someone reads it.
Background in credential expiry monitoring.