Incident: SLO breach on app-00063-suh (e0574f5a)
T3live, issue closedissue #11 · 12 Oct 17:23:07 to 12 Oct 17:55:07 (32 min 00 s) · 17 ledger entries · raw JSON · related: commit e0574f5a
Run timeline
Issue
17:23:07
Triage
skipped
Delivery
17:23:14
1 min 32 s
1 min 32 s
Pipeline
17:24:50
7 min 26 s
7 min 26 s
Canary
17:33:06
20 min 01 s
20 min 01 s
Promoted
17:53:08
Close
17:55:07
GitLab DevSecOps stages this run touched
Plan
issue #11Verify
MR pipeline 2939644020Release
production/app environmentConfigure
feature flagsMonitor
SLO probe results (ledger)Govern
ledger entriesTier and evidence
Tier T3, 5 points. A signal's floor sets a minimum tier; points add to it. No LLM is in this decision.
| Signal | Detail | Points | Floor |
|---|---|---|---|
| auth_or_secrets | app/tests/test_login_dark_launch.py, app/users.py | 0 | T3 |
| touched_line_coverage | unknown, +1 (no coverage data yet, so the scorer assumes worse) | 1 | |
| track_record | penalty 4 from rollbacks/reverts on these paths | 4 |
Human actions
None. No person acted on this run after it started.
Ledger entries
| Time | Entry | Kind | What happened |
|---|---|---|---|
| Issue | |||
| 17:23:07 | L126-05bcc33330f2 | incident_opened | incident #11 opened for e0574f5a, fix tier T3 |
| Delivery | |||
| 17:23:14 | L128-e44d6da117b0 | flow_requested | delivery flow requested, tier T3 |
| 17:23:15 | L129-803b56b0072d | flow_started | session 9339123 started; session privileges [1,2,3,4,5,6,7] |
| 17:24:45 | L130-b2289f970a12 | mr_opened | MR !5 opened on t3/issue-11-1 |
| Pipeline | |||
| 17:24:50 | L131-b0e567cb0691 | tier_computed | tier T3 (5 points, scorer v1) |
| 17:25:10 | L132-0073f7754343 | pipeline_started | pipeline 2939644020 started: RISK_TIER=T3 |
| 17:26:59 | L133-d8c494b3f81b | pipeline_finished | pipeline 2939644020 finished: success |
| 17:27:09 | L134-ac0db5cf4d59 | merged | MR !5 merged as 699cca68 |
| 17:32:16 | L135-ab553ee9ffd5 | deployed | revision app-00066-buq deployed with no traffic (pipeline 2939647960) |
| Canary | |||
| 17:33:06 | L136-38867a39c47e | flag_not_used | flag t3_dark_launch kept off: the change removes flag-gated code; the flag stays off so the stable revision's gated path stays dark, and the canary does not read the flag |
| 17:33:07 | L137-eaba6ff1dc3a | traffic_shift | 1% of traffic to app-00066-buq, soak 20 min |
| 17:53:08 | L138-5b76b509c22e | slo_check | SLO ok: 200 samples, 0 errors, p95 77.5 ms at 1% |
| Promote / roll back | |||
| 17:53:08 | L139-22124b961d5f | traffic_shift | 100% of traffic to app-00066-buq |
| 17:53:08 | L140-1d5e583ba0dc | slo_check | SLO insufficient: 10 samples, 0 errors, p95 72.0 ms at 100% |
| 17:53:09 | L141-19075c1f7397 | promoted | app-00066-buq promoted to 100% (flag off) |
| 17:53:09 | L142-6e59ad0f4817 | release_outcome | release outcome: clean_release |
| Close | |||
| 17:55:07 | L143-9a14fec880bb | issue_closed_by_governor | issue #11 closed by the governor after app-00066-buq went live |