Arc Light Energy · Grid operations Postmortem · Incident 17 · SEV-1 · Tue 22 Sep 2026

17 minutes.

A configuration bundle reached our command router without a routing key for the north-fjord region. The router resolved the missing key to an empty route instead of failing closed, and 31 sites stopped receiving commands while every health signal stayed green. The incident was short; the ambiguity that enabled it was not.

Duration
17 min · 09:14–09:31
Sites affected
31 of 214 · north-fjord
Detected by
Desk operator, 09:19 · no alert
Durable fixes
1 of 3 done · 2 open
From 09:14 to 09:31 the console showed all 31 north-fjord sites healthy while none of them acknowledged a command. Commands waiting in the queue rose from 6 to 121. Undetected for 5 minutes, diagnosed in 8, recovered in 4.
01

What it cost

31 of 214 sites could not receive commands for 17 minutes. No site lost power and no safety function was affected: protection runs on each site’s own controller, and every site held its last setpoint.

North-fjord31
North-inland28
West coast44
Central39
South41
Islands31
Each mark is one of the 214 sites, by region. The 31 hollow marks accepted commands for 17 minutes and acknowledged none.
Sites
31 of 214 (14%), every site in the north-fjord region. The other five regions were unaffected.
Commands
121 sent into the gap. 112 waited in the queue and were delivered late, by 09:33. 9 dispatch setpoints expired after their 10-minute limit and were re-issued by hand by 09:40. None were lost silently.
Dispatch
NF-12 and NF-19 missed the 09:20 discharge window. 1.4 MWh of scheduled discharge was not delivered.
Safety
None. Site protection does not depend on the command path.
Detection
5 minutes, by a person reading the acknowledgement column. No alert fired.
Resolution
12 minutes after detection, 17 minutes in total.
02

Minute by minute

Every minute of the window, quiet ones included. Left of the spine is what operators were shown; right of it is what the command path was doing.

  1. Console showed31 / 31 healthyCommand API answers 202 Accepted.
    What was trueFallback activatesBundle 0922-1 reaches the router without a north-fjord key. The lookup returns an empty route instead of failing closed; commands to 31 sites are accepted and go nowhere.
    Waiting6
  2. Console showed31 / 31 healthy
    What was trueA setpoint to NF-07 passes its 30-second acknowledgement window. 0 of 31 north-fjord sites are acknowledging.
    Waiting13
  3. Console showed31 / 31 healthy
    What was trueThe fleet acknowledgement rate falls from 99.6% to 85.5%. It is on a chart. It is not on an alert.
    Waiting20
  4. Console showed31 / 31 healthy
    What was trueNothing new. Commands keep queuing.
    Waiting26
  5. Console showed31 / 31 healthy09:20 dispatch schedule marked “sent”.
    What was trueThe 09:20 schedule for north-fjord sits in the queue with everything else.
    Waiting33
  6. Console showed31 / 31 healthyCommand log: 31 sites “sent”, none “acknowledged”.
    What was trueOperators see driftThe north-fjord desk operator notices the empty acknowledgement column and calls the shift lead. Five minutes have passed without an alert.
    Waiting40
  7. Console showed31 / 31 healthy
    What was trueThe discharge window opens. NF-12 and NF-19 miss it; 1.4 MWh of scheduled discharge is not delivered.
    Waiting55
  8. Console showed31 / 31 healthy
    What was trueThe shift lead pages platform on-call.
    Waiting61
  9. Console showed31 / 31 healthySite links: 31 / 31 up.
    What was trueOn-call acknowledges and checks site connectivity. Every site answers, over the telemetry uplink, which is not the path that broke.
    Waiting68
  10. Console showed31 / 31 healthy
    What was trueFalse lead closed: the sites are reachable, so the fault is on our side of the command path.
    Waiting75
  11. Console showed31 / 31 healthy
    What was trueSEV-1 declaredThe duty manager declares SEV-1 and takes incident command. Status goes to the operations channel.
    Waiting81
  12. Console showed31 / 31 healthy
    What was trueThe routing lead joins the call.
    Waiting88
  13. Console showed31 / 31 healthy
    What was trueA traced command shows route=[] for every north-fjord site.
    Waiting94
  14. Console showed31 / 31 healthy
    What was trueMissing key foundA diff of bundles 0922-0 and 0922-1 shows the region split added north-inland and dropped north-fjord.
    Waiting101
  15. Console showed31 / 31 healthy
    What was trueDecision: promote the previous bundle rather than hand-add the key under pressure.
    Waiting108
  16. Console showed31 / 31 healthy
    What was truePromotion of 0922-0 starts, with a two-person check.
    Waiting114
  17. Console showed31 / 31 healthy
    What was trueBundle 0922-0 is live on 3 of 5 router replicas.
    Waiting121
  18. Console showed31 / 31 healthyThe acknowledgement column fills. Health never moved.
    What was trueRoute restoredThe previous configuration is live on all replicas. NF-07 acknowledges at 09:31:12; all 31 sites by 09:31:48. The queue starts draining.
    Waiting121
  1. After the windowQueue empty. 112 commands delivered late, in order.
  2. The desk re-issues the 9 setpoints that expired in the queue.
  3. Incident closed after 30 minutes of clean acknowledgements across all regions.
03

Why the console stayed green

Health and commands travel in opposite directions. We only measured one of them.

Control room console and command API 31 sites north-fjord region NF-01 … NF-31 Heartbeat every 10 s over the telemetry uplink, site to us shown as 31 / 31 healthy Commands, us to site API answers 202 Accepted Command router route_for("north-fjord") = [] nothing leaves the router Control room console and command API 31 sites north-fjord region Heartbeat, site to us 31 / 31 healthy Commands, us to site 202 Accepted Command router route_for(…) = [] nothing leaves the router
FIG. 2Device health is measured on the path from the sites to us. The fault was on the path from us to the sites, and that path had no health signal of its own.

Root cause. The router’s lookup for a region key fell back to an empty route when the key was missing. An empty route is a valid answer, so the router accepted every command for north-fjord, queued it and reported nothing wrong.

  1. The fallback was meant to be temporary. CHG-2291, merged on Wed 16 Sep, let the router tolerate missing region keys so the staged region split could roll out without crashing replicas. It had no removal date, no alert and no owner after the split.
  2. Health checks measure the other direction. Device health comes from heartbeats the sites send us. It tells us a site can reach us, not that we can reach it.
  3. Acknowledgements were charted, not alerted. The fleet rate dropped from 99.6% to 85.5% in two minutes. Nobody had set a threshold, so nothing paged.
  4. Validation checked shape, not completeness. Bundle 0922-1 was valid against the schema. Nothing compared its region keys with the site registry.
  5. “Accepted” read as “sent”. The API returns 202 when a command is queued. The console labelled those commands “sent”.
router/resolve.py · CHG-2291 · merged Wed 16 Sep41def route_for(key):42−    return ROUTES[key]42+    # tolerate keys missing mid-split43+    return ROUTES.get(key, [])
INTRODUCEDSix days before the incident. The comment described a window; the code had none.
router/resolve.py · CHG-2317 · shipped Mon 28 Sep41def route_for(key):42−    return ROUTES.get(key, [])42+    route = ROUTES[key]  # raises43+    if not route: raise EmptyRoute(key)44    return route
REMOVEDA missing or empty route now stops the bundle at publish and returns 503 to the caller.
04

Durable fixes

Three changes, each with an owner, a date and a test that tells us it worked. Status as of Wed 30 Sep.

Durable fixes for incident 17 with owner role, due date, status and the check that proves each one.
FixOwnerDueStatusHow we will know it worked
RemoveThe empty-route fallbackThe router fails closed. A bundle with an unresolved region key is rejected at publish, and the command API returns 503 for anything it cannot route. Platform routing lead 29 Sep DoneCHG-2317 · 28 Sep A test bundle without the north-fjord key is rejected in staging, and the publish job names the missing key. Passing since 28 Sep.
AddAn acknowledgement SLO and alert99% of commands acknowledged within 60 seconds, per region. Page the desk and platform on-call when any region stays below 90% for 2 minutes. Control-room operations manager 9 Oct In progressshadow since 25 Sep Replaying the 22 Sep traffic through the rule pages at 09:16, three minutes before a person noticed.
ProveA regional failover drillRemove one region key on purpose, first in staging and then in a production maintenance window, and run the recovery from the runbook. Reliability programme lead 23 Oct Scheduled21 Oct staging · 23 Oct prod The page arrives within 2 minutes, and a responder who was not on incident 17 restores routes in under 5.
05

What held, and what was luck

We keep what worked on purpose. The luck is a list of things to stop relying on.

Held up on purpose

  • A person read the right column. The desk operator saw the empty acknowledgement column at 09:19 and escalated at once.
  • Rolling back beat patching. Promoting bundle 0922-0 took three minutes and needed no code change.
  • The queue kept its order. 112 of 121 commands were delivered late rather than dropped.
  • Site protection stayed local. No site needed the command path to stay safe.

Luck not a control

  • The hour. At 09:14 on a weekday the north-fjord desk is staffed. Overnight the region is watched from the central console, which hides the acknowledgement column by default.
  • A small diff. Bundle 0922-1 changed three keys. A larger bundle would have taken longer to read by eye.
  • Two sites, not twelve. Only NF-12 and NF-19 had a discharge due inside the window.
  • Low stakes in the queue. The nine expired commands were routine setpoints, not curtailment orders.
Open the prevention checkWhat keeps this fixed after the fixes ship

The durable check is ownership: the recovery path now has a named owner, a tested alert and a scheduled review. If any of the three lapses, this postmortem reopens.

Owner
The platform routing lead owns route resolution and the fail-closed rule, recorded in the service catalogue on 28 Sep.
Alert
The acknowledgement SLO pages below 90% for 2 minutes. Tested by replaying 22 Sep traffic; it fired at 09:16.
Review
First on Fri 6 Nov 2026 with the drill results, then every quarter. The reliability programme lead chairs it.