The goal
The previous four lessons covered measuring risk, pinning versions, generating SBOMs and hardening CI. This one tests using all four under time pressure with incomplete information — that is, in the conditions they are actually used.
The situation — 09:15. A message in the team channel:
"Dependabot just opened a PR bumpingCorp.Telemetryfrom 4.2.1 to 4.2.2. 4.2.2 was published at 02:40 this morning, and the changelog says onlyfix: minor issues. Four months ago 4.2.1 shipped with a three-paragraph changelog. Feels off to me."
Corp.Telemetry is an internal package maintained by another team in the company, used in 12 services, and it ships metrics so it has outbound network access.
What to produce.
- Decide in the first 10 minutes: is this an incident or an ordinary patch? On what evidence?
- If undecided, what do you do to stop it spreading while investigating?
- If it is an incident: establish the blast radius. Which services, which secrets?
- Afterwards: what change prevents a recurrence?
Question 2 is the most important and the one people skip, because the instinct is to reach a conclusion before acting. In a supply-chain incident that order is backwards.
Comments
Commenting needs an account with at least one completed lesson. That condition is what keeps this thread worth reading: every point belongs to someone who can be asked back, and reputation accrues over time.
You can still read every comment below without an account. Signing in brings you back to this exact spot, not to the top of the page.
Loading comments…