题记与说明
- 精读日期:2026-11-03
- 主题形式:主题研究(先检索并逐源精读 8 个来源,非单篇论文)
- 检索状态:网络可用。注意工具链情况:
WebFetch直连 sre.google 全部失败(5 次并行抓取 4 次ECONNRESET、1 次 connect timeout;curl -sL返回 exit 35),改用 web-reader 全文抓取成功——8 个来源全部在线抓取原文并精读,无凭记忆补写。 - 来源清单:
- Google SRE Book 第 3 章 *Embracing Risk*(错误预算的动机与风险数学)— https://sre.google/sre-book/embracing-risk/
- Google SRE Book 第 4 章 *Service Level Objectives*(SLI/SLO/SLA 定义与选择纪律)— https://sre.google/sre-book/service-level-objectives/
- Google SRE Workbook 第 2 章 *Implementing SLOs*(SLO 落地配方:SLI 菜单、首版取数、错误预算政策)— https://sre.google/workbook/implementing-slos/
- Google SRE Workbook 第 5 章 *Alerting on SLOs*(多窗口多烧率告警 + 低流量服务特例)— https://sre.google/workbook/alerting-on-slos/
- Google SRE Book 第 14 章 *Managing Incidents*(故障响应角色与活文档)— https://sre.google/sre-book/managing-incidents/
- Google SRE Book 第 15 章 *Postmortem Culture*(免责复盘文化)— https://sre.google/sre-book/postmortem-culture/
- Google SRE Book 附录 *Example Postmortem*(复盘模板实样,可抄字段)— https://sre.google/sre-book/example-postmortem/
- cloudandsre.com *SLI, SLO, SLA, and error budgets — the reliability contract explained*(2026-05,第三方压缩版框架 + AI Agent 时代延伸)— https://cloudandsre.com/blog/sli-slo-sla-error-budgets/
- 与既有笔记的分工:[[2026-11-01-knowledge-qyxuo8eu-3-Prometheus-Grafana监控实践]] 覆盖了 Workbook 第 5 章的告警语法细节与 Prometheus 指标命名;本篇覆盖它没展开的另一半——SLO 框架本身(定义→取值→政策→复盘闭环)+ 故障响应流程,并补齐该笔记未做的"个人服务适配"。两篇在多窗口烧率告警上有少量重叠,本篇只取结论不重复推导。
一、主题概览
SRE 基础常被讲成名词解释(SLI 是指标、SLO 是目标、SLA 是合同),这次精读 8 个来源后,我认为真正的骨架是一条把"可靠"变成可运营决策的闭环:
用户在乎什么 → SLI(把体验翻译成 good/total 比率)→ SLO(目标 + 窗口 + 测量点)→ 错误预算(SLO 的另一半,决定发布节奏)→ 按烧率告警(预算花太快才响)→ 故障响应(角色分离 + 活文档 + 先止血)→ 免责复盘(行动项闭环)→ 反哺 SLO 与架构。
8 个来源指向同一组判断:
- 100% 可靠是错误目标,不是做不到,而是不该要——它会杀死迭代速度、吞噬预算、把团队变成纯救火队(Book Ch.3/4、Workbook Ch.2 三处独立论证)。
- 错误预算是 SRE 里唯一的"货币":它把"该不该为了稳定暂停发版"从吵架变成算术(Book Ch.3 "Hope is not a strategy";cloudandsre "turn arguments about *should we ship this* into arithmetic")。
- 没有配套政策的 SLO 只是又一个 KPI(Workbook Ch.2:"SLO compliance will simply be another KPI... rather than a decision-making tool";cloudandsre:"A team that has an SLO but no error-budget policy has half a contract.")。
- 告警不该对原因、也不该对静态阈值,而该对错误预算的消耗速度(Workbook Ch.5 烧率体系)。
- 故障响应的核心不是技术而是分工与记录:指挥、操作、沟通、规划四个角色分离;活文档贯穿始终;"先止血、再查因、留证据"。
- 复盘的生死线是"免责":blameless 不是客气,而是让信息浮出水面的前提——对单人项目同样成立(对自己的失误免责,才不会把复盘写成自我批斗然后再也不写)。
二、逐源精读
来源 1:Google SRE Book 第 3 章《Embracing Risk》
出处:https://sre.google/sre-book/embracing-risk/ (Google 官方,2016;错误预算一节署名 Mark Roth)
核心论证——为什么 100% 是错的(这是全 SRE 文献被引用最多的段落之一):
"Extreme reliability comes at a cost: maximizing stability limits how fast new features can be developed and how quickly products can be delivered to users... a user on a 99% reliable smartphone cannot tell the difference between 99.99% and 99.999% service reliability!"
成本不是线性增长的:
"cost does not increase linearly as reliability increments—an incremental improvement in reliability may cost 100x more than the previous increment."
因此 Google 把可用性目标同时当作下限和上限:
"We strive to make a service reliable enough, but no *more* reliable than it needs to be... we view the availability target as both a minimum and a maximum. The key advantage of this framing is that it unlocks explicit, thoughtful risktaking."
风险数学(可直接抄的数字):
- 时间口径:99.99% 全年允许宕机 52.56 分钟;
- 请求口径(Google 实际用这个):日 250 万请求、99.99% 目标 → 每天可容忍 250 个失败请求;
- 成本/收益算例:收入 $1M 的服务从 99.9% 提到 99.99%,可用性只提升 0.09%,收益 = $1M × 0.0009 = $900——加一个九的成本超过 $900 就不该做;
- 天花板参照:实测 ISP 背景错误率在 0.01%–1% 之间,把错误率压到这个噪声以下,用户根本感知不到。
错误预算的运行机制("Motivation for Error Budgets"一节):
"Nor should such decisions be driven by politics, fear, or hope. (Indeed, Google SRE's unofficial motto is 'Hope is not a strategy.')"
四条实践原文:
"Product Management defines an SLO... The actual uptime is measured by a neutral third party: our monitoring system. The difference between these two numbers is the 'budget' of how much 'unreliability' is remaining... As long as the uptime measured is above the SLO—in other words, as long as there is error budget remaining—new releases can be pushed."
算例:SLO 99.999%(季度错误预算 0.001%),一次事故烧掉 0.0002%,即消耗 20% 预算。
最妙的副作用是激励相容:
"When the budget is nearly drained, the product developers themselves will push for more testing or slower push velocity... In effect, the product development team becomes self-policing."
对个人服务的适配点:单人开发者同时是"产品开发"和"SRE",错误预算恰好替你把这两个内心的自己分开——预算充足时放心折腾新东西,预算见底时连你自己都会想先补测试。这个"自我执法"机制不需要团队也成立。
来源 2:Google SRE Book 第 4 章《Service Level Objectives》
出处:https://sre.google/sre-book/service-level-objectives/
三个术语的官方定义(原文):
"An SLI is a service level indicator—a carefully defined quantitative measure of some aspect of the level of service that is provided."
"An SLO is a service level objective: a target value or range of values for a service level that is measured by an SLI."
"SLAs are service level agreements: an explicit or implicit contract with your users that includes consequences of meeting (or missing) the SLOs they contain."
区分 SLO 和 SLA 的土办法(很实用):
"An easy way to tell the difference between an SLO and an SLA is to ask 'what happens if the SLOs aren't met?': if there is no explicit consequence, then you are almost certainly looking at an SLO."
选哪些 SLI——按服务类型的经典分类:
| 服务类型 | 关心的 SLI |
|---|---|
| 用户侧服务系统 | availability、latency、throughput |
| 存储系统 | latency、availability、durability(耐久性) |
| 大数据管道 | throughput、端到端 latency |
| 所有系统 | correctness(答案对不对,虽然通常不算 SRE 的责任) |
两条选择纪律:
"You shouldn't use every metric you can track in your monitoring system as an SLI... We typically find that a handful of representative indicators are enough to evaluate and reason about a system's health."
"Start by thinking about (or finding out!) what your users care about, not what you can measure."——从目标倒推指标,好过从"容易测的"顺推。
平均数会撒谎:本章 Figure 4-1 的例子——典型请求 50ms,但 5% 的请求慢 20 倍;只看平均延迟,监控全天显示"一切正常"。结论是 SLI 要用分位数(99th/99.9th),并且用户研究显示"人们宁可整体略慢也不要高方差"。
SLO 的写法(规格化示例,可直接抄格式):
- 99% (averaged over 1 minute) ofGetRPC calls will complete in less than 100 ms (measured across all the backend servers).
- 90% / 99% / 99.9% ofGetRPC calls will complete in less than 1 ms / 10 ms / 100 ms.(多条阈值刻画整条性能曲线)
本章最有个人服务价值的两段:
其一,错误预算的精辟定义:
"(An error budget is just an SLO for meeting other SLOs!)"
其二,松紧策略——对首版 SLO 尤其重要:
"It's better to start with a loose target that you tighten than to choose an overly strict target that has to be relaxed when you discover it's unattainable."
以及 SLO 的"辩护测试":
"Defend the SLOs you pick: if you can't ever win a conversation about priorities by quoting a particular SLO, it's probably not worth having that SLO."
控制回路(SLO 不是报表而是控制系统的设定值):
"1. Monitor and measure the system's SLIs. 2. Compare the SLIs to the SLOs, and decide whether or not action is needed. 3. If action is needed, figure out *what* needs to happen in order to meet the target. 4. Take that action."
两个反直觉提醒(Google 用血泪换的):
- 内部 SLO 要比对用户承诺的更紧:"Using a tighter internal SLO than the SLO advertised to users gives you room to respond to chronic problems before they become visible externally."
- 用户会依赖你的"实际表现"而不是"承诺表现":Chubby 因为过度可用反而被迫引入计划内宕机——"Google's Chubby service introduced planned outages in response to being overly available"。给个人服务的启示:SLO 别定得太好看,否则用户(包括未来的自己)会把它当成硬承诺。
来源 3:Google SRE Workbook 第 2 章《Implementing SLOs》(落地配方,全篇最实用)
出处:https://sre.google/workbook/implementing-slos/
开篇自评很高:
"Because SLOs are key to making data-driven decisions about reliability, they're at the core of SRE practices. In many ways, this is the most important chapter in this book."
"One could even claim that without SLOs, there is no need for SREs."
"100% 为什么错"的四条完整论证(比 Book 版更细,值得整段记住):
"- ...there is a nonzero probability that one or more components will fail simultaneously...
- Even if you could achieve 100% reliability within your system, your customers would not experience 100% reliability. The chain of systems between you and your customers is often long and complex...
- If you do manage to create an experience that is 100% reliable... you can never update or improve your service. The number one source of outages is change...
- An SLO of 100% means you only have time to be reactive. ... Reliability of 100% is not an engineering culture SLO—it's an operations team SLO."
SLI 的推荐形态——一律写成比率:
"we generally recommend treating the SLI as the ratio of two numbers: the number of good events divided by the total number of events."
比率形态的好处:天然落在 0–100%、直接推出错误预算(100% − SLO)、工具链只需认 numerator/denominator/threshold 三元组。算例:99.9% SLO、4 周 300 万请求 → 预算 3,000 个错误;一次 1,500 错误的故障 = 烧掉 50% 预算。
SLI 规格 vs SLI 实现(重要区分,决定你从哪采集数据):
- SLI 规格(SLI specification):与测量方式无关的、用户在乎的结果——"Ratio of home page requests that loaded in < 100 ms";
- SLI 实现(SLI implementation):规格 + 具体测量方法。同一规格可有三种实现:服务器日志(漏掉没到后端的请求)、浏览器探针(覆盖到网络但可能漏部分用户)、前端 JS 上报(最真实但要建采集设施)——各自在 quality(保真)、coverage(覆盖)、cost(成本) 上取舍。
起步配方(给"从零到第一版 SLO"的完整步骤):
- 选一个应用(多应用以后再加);
- 明确"用户"是谁;
- 列出用户与系统的常见交互(关键任务);
- 画高层架构图,把组件归入三类:Request-driven / Pipeline / Storage;
- 从 SLI 菜单(Table 2-1)选 SLI——"We recommend choosing a small number (five or fewer) of SLI types that represent the most critical functionality to your customers."
Table 2-1 摘录(SLI 菜单核心行):
| 类型 | SLI | 说明 |
|---|---|---|
| Request-driven | Availability | 成功响应的请求占比 |
| Request-driven | Latency | 快于某阈值的请求占比 |
| Request-driven | Quality | 降级服务时"未降级响应"的占比 |
| Pipeline | Freshness | 新于某时间阈值的数据占比 |
| Pipeline | Correctness | 产出正确结果的记录占比 |
| Storage | Durability | 写入后可成功读出的记录占比 |
worked example 的数字(这是"从现状取整出首版 SLO"的实演):4 周窗口内 API 总请求 3,663,253、成功 3,557,865(97.123%)、p90 延迟 432ms、p99 891ms。向下取整得到首版 SLO:
| SLO | 目标 | 4 周错误预算 |
|---|---|---|
| Availability | 97% | 109,897 个失败 |
| Latency | 90% < 450 ms | 366,325 个慢请求 |
| Latency | 99% < 900 ms | 36,632 个慢请求 |
"Your first attempt at an SLI and SLO doesn't have to be correct; the most important goal is to get something in place and measured, and to set up a feedback loop so you can improve."
窗口怎么选:
"Rolling windows are more closely aligned with user experience: if you have a large outage on the final day of a month, your user doesn't suddenly forget about it on the first day of the following month."
窗口用整周数(保证窗口内周末数量一致,否则 SLI 会因流量结构波动);日历窗口只在对齐业务规划时用。推荐组合:
"We have found a four-week rolling window to be a good general-purpose interval. We complement this time frame with weekly summaries for task prioritization and quarterly summarized reports for project planning."
利益相关方三方协议(个人版就是"和未来的自己签"):产品侧认这个阈值够用户用;开发侧认预算耗尽时愿意降风险;运维侧认这个 SLO 不靠英雄主义就能守住——"defensible without Herculean effort, excessive toil, and burnout"。三方有任何一方不认,就迭代 SLI/SLO 直到都认。
错误预算政策(没有它 SLO 就没牙)——政策要写明预算耗尽时谁做什么,常见动作:
"- The development team gives top priority to bugs relating to reliability issues over the past four weeks.
- The development team focuses exclusively on reliability issues until the system is within SLO...
- To reduce the risk of more outages, a production freeze halts certain changes to the system until there is sufficient error budget to resume changes."
SLO 文档必含要素(清单原文,直接当模板):作者/评审者/批准者;批准日期与下次复审日期;服务简介;SLO 目标与 SLI 实现;错误预算如何计算与消耗;数字的依据("Even if the SLOs are totally ad hoc, this fact should be documented")。复审节奏:起步期每月,成熟后季度或更久。
用预算给故障定量分级(把"哪个故障更值得查"变成算术):
例 1:新 API 版本引发 100% NullPointerException,4 小时后回滚,14,066 个错误 = 13% 预算;例 2:单点数据库宕 20 小时,72,000 个错误 = 65% 预算。五年才一次的硬件故障 vs 每年两三次的坏发布 → "bad pushes cost twice as much error budget as database failures",所以优先修发布流程。SLO 决策矩阵(Table 2-5,按 SLO 达成 × toil 高低 × 用户满意度三维给行动,例如"SLO 达成 + toil 低 + 满意度低 → 收紧 SLO";"SLO 未达成 + toil 低 + 满意度高 → 放松 SLO"——最后这条最反直觉:经常差点 miss 但用户毫无怨言,说明 SLO 定太紧了)。
一个防坑警告(对加机器求高可用的人):
"It can be tempting to try to math your way out of these problems. If you have a service that offers 99.9% availability in a single zone... simply deploying the service in two zones should solve that requirement... However, this reasoning assumes that both services are wholly independent, which is almost never the case... any such calculations will be deceptive."
结论的三连:
"- SLOs are the tool by which you measure your service's reliability.
- Error budgets are a tool for balancing reliability with other engineering work...
- You should start using SLOs and error budgets today."
来源 4:Google SRE Workbook 第 5 章《Alerting on SLOs》(告警=预算烧率)
出处:https://sre.google/workbook/alerting-on-slos/
告警策略的四个评价指标:Precision(响的都是大事)、Recall(大事都响)、Detection time(多久发现)、Reset time(恢复后多久消警)。本章用 6 次迭代推导出推荐方案,关键中间结论:
- 对"最近 10 分钟错误率 ≥ SLO"告警:检测快但精度极差——"you could receive up to 144 alerts per day every day, not act upon any alerts, and still meet the SLO";
- 拉长窗口到 36h:精度好了,但 100% 宕机后告警要响 36 小时(reset time 灾难);
- 用
for:持续时间参数:明确不推荐——"we do not recommend using durations as part of your SLO-based alerting criteria"(抖动型故障可能永远凑不满 for 时长,但预算已经烧光)。
烧率(burn rate)定义:
"Burn rate is how fast, relative to the SLO, the service consumes the error budget."
烧率与耗尽时间(99.9% SLO):1× → 30 天;2× → 15 天;10× → 3 天;1000× → 43 分钟。
最终推荐——多窗口多烧率(Table 5-8,即上一条笔记 qyxuo8eu-3 引过的同款):
| 严重度 | 长窗口 | 短窗口 | 烧率 | 预算消耗 | 通知方式 |
|---|---|---|---|---|---|
| Page | 1 h | 5 min | 14.4 | 2% | 电话/即时 |
| Page | 6 h | 30 min | 6 | 5% | 电话/即时 |
| Ticket | 3 d | 6 h | 1 | 10% | 工单 |
短窗口取长窗口的 1/12,用于确认"还在持续烧",消警更快、误报更少。
低流量服务一节——个人服务的命门(本章对单人项目最重要的内容):
"if a system receives 10 requests per hour, then a single failed request results in an hourly error rate of 10%. For a 99.9% SLO, this request constitutes a 1,000x burn rate and would page immediately... This scenario allows for only seven failed requests in 30 days."
低流量的对策四选(可组合):
- 造合成流量:"as a low-fidelity solution... set up a remote monitoring service that performs some kind of periodic health check on the service (a ping or an HTTP GET)"(UptimeRobot 类外部拨测就是这招);
- 把小服务合并成一组监控(共享故障域才合并);
- 改产品降低单次失败的影响:客户端指数退避重试 + 抖动、失败落 fallback 队列延后执行;
- 降低 SLO:"users would be equally happy with a lower SLO. With a lower SLO, an engineer is notified only of a larger sustained outage."(这是产品决策,但个人服务的"产品决策人"就是你自己。)
来源 5:Google SRE Book 第 14 章《Managing Incidents》(故障响应)
出处:https://sre.google/sre-book/managing-incidents/
本章用一个失控事故的故事开场:值班工程师 Mary 一边修一边被 CEO 和 VP 轰炸,同事"就看看"各自乱碰,凌晨 3:30 被叫醒的开发者又拉来别人,有人在没人协调的情况下直接改生产——"Within seconds, the servers restarted, picking up the change. And then died." 本章的定论很温和也很致命:
"Note that everybody in the preceding scenario was doing their job, as they saw it. How could things go so wrong?"
"Incident management skills and practices exist to channel the energies of enthusiastic individuals."
基于 Incident Command System 的四个角色:
| 角色 | 职责(原文要点) |
|---|---|
| Incident Commander | "holds the high-level state about the incident... De facto, the commander holds all positions that they have not delegated." 指挥不亲手修 |
| Operations Lead | 动手修的人——"The operations team should be the only group modifying the system during an incident." |
| Communications Lead | 对外脸面,定期向利益相关方发进展 |
| Planning Lead | 记 bug、安排交接、记录"系统偏离常态了什么"以便事后还原 |
活文档——指挥官最重要的职责:
"The incident commander's most important responsibility is to keep a living incident document. ... This living doc can be messy, but must be functional."
工具选择上的神来之笔:
"...depending on the software you are trying to fix as part of your incident management system is unlikely to end well."(别用你正在修的那个系统来管理这次事故。)
显式交接:下班前指挥权必须明确移交,交接口令都要写死——"You're now the incident commander, okay?",对方确认前不许挂电话。单人项目里"交接对象"是几小时后的自己:所以活文档更重要。
什么时候宣布进入事故流程(早宣布比晚宣布好):
"It is better to declare an incident early and then find a simple fix and close out the incident than to have to spin up the incident management framework hours into a burgeoning problem."
三条判据(任一为真即是事故):需要拉第二个人/组进来?用户可见?集中分析一小时还没解决?
技能会退化:
"Incident management proficiency atrophies quickly when it's not in constant use."——所以要例行使用(发布、演练也走这套流程)。
总结七原则(原文):Prioritize("Stop the bleeding, restore service, and preserve the evidence for root-causing")、Prepare、Trust、Introspect、Consider alternatives、Practice、Change it around(上次当指挥这次换角色)。
来源 6:Google SRE Book 第 15 章《Postmortem Culture: Learning from Failure》(复盘)
出处:https://sre.google/sre-book/postmortem-culture/
章首引言只有一句:
"The cost of failure is education." — Devin Carraway
复盘的正式定义:
"A postmortem is a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring."
什么情况必须写复盘(触发条件要事先定义,这点最容易被忽视):
"- User-visible downtime or degradation beyond a certain threshold
- Data loss of any kind
- On-call engineer intervention (release rollback, rerouting of traffic, etc.)
- A resolution time above some threshold
- A monitoring failure (which usually implies manual incident discovery)"
"It is important to define postmortem criteria before an incident occurs so that everyone knows when a postmortem is necessary."
免责(blameless)的精确含义——不是和稀泥,是换问题:
"For a postmortem to be truly blameless, it must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior."
"A blamelessly written postmortem assumes that everyone involved in an incident had good intentions and did the right thing with the information they had."
"You can't 'fix' people, but you can fix systems and processes to better support people making the right choices..."
免责的理由是功利性的:
"If a culture of finger pointing and shaming individuals or teams for doing the 'wrong' thing prevails, people will not bring issues to light for fear of punishment."
"An atmosphere of blame risks creating a culture in which incidents and issues are swept under the rug, leading to greater risk for the organization."
免责不等于不批评系统——措辞换掉的是"指向人"而不是"指向问题",书中给了改写示范:把"We need to rewrite the entire complicated backend system! ...if I get paged one more time I'll rewrite it myself…"(发泄式)改写成"An action item to rewrite the entire backend system might actually prevent these annoying pages... I'm sure our future on-callers will thank us!"(行动项式)。
评审标准(草稿要过的五问):关键数据留档了吗?影响评估完整吗?根因挖得够深吗?行动计划和修复优先级合适吗?结论同步给相关方了吗?以及那句狠话:
"An unreviewed postmortem might as well never have existed."
文化运营(对单人项目的启发是"仪式感"):月度最佳复盘(postmortem of the month)、复盘读书会、灾难题角色扮演(Wheel of Misfortune,重演旧复盘)。还有一个著名故事:一位 SRE 的发布引发 4 分钟宕机,因立刻回滚未酿大祸——在全员 TGIF 上收获两个 peer bonus 和全场掌声。写复盘、快速止血是被奖励的行为,不是污点。
来源 7:Google SRE Book 附录《Example Postmortem》(复盘模板实样)
出处:https://sre.google/sre-book/example-postmortem/ (Shakespeare 服务 66 分钟宕机虚构示例)
模板字段实录(可直接抄成个人复盘模板):
- 头部:Date / Authors / Status("Complete, action items in progress")/ Summary(一句话:宕机 66 分钟 + 背景)/ Impact(估计 1.21B 查询丢失)/ Root Causes(级联失效:高峰负载 + 搜索词不在语料时触发资源泄漏)/ Trigger(潜伏 bug 被流量激增触发)/ Resolution(引流到牺牲集群 + 10x 扩容 + 更新索引)/ Detection(Borgmon 检测到 HTTP 500 风暴并 page 值班);
- Action Items 表:列 =
Action Item | Type | Owner | Bug,Type 分四类:mitigate / prevent / process / other。样例行包括"Update playbook"、"Plug file descriptor leak"、"Schedule cascading failure test during next DiRT",还有一条把错误预算和复盘连起来的关键行:
"Freeze production until 2015-11-20 due to error budget exhaustion, or seek exception..."
- What went well / What went wrong / Where we got lucky(三栏,"我们运气好在哪"是 Google 模板的招牌字段——把侥幸显式化,才能变成机制);
- 时间线:精确到分钟,标注
OUTAGE BEGINS/INCIDENT BEGINS/OUTAGE MITIGATED/OUTAGE ENDS/INCIDENT ENDS,且事故结束有明确退出判据——"reached exit criterion of 30 minutes' nominal performance"。
来源 8:cloudandsre.com《SLI, SLO, SLA, and error budgets — the reliability contract explained》(2026-05-22,作者 Ajin Baby)
出处:https://cloudandsre.com/blog/sli-slo-sla-error-budgets/
第三方压缩版,价值在于两句金句和一张九位表。开场:
"SLIs, SLOs, SLAs, and error budgets are the four numbers that make the answer explicit — and turn arguments about *should we ship this* into arithmetic."
SLI 两条保命规则:
"Measure where the user is, not at the server — a 200 OK that the client never saw is not a success."
"define the denominator carefully — excluding 4xx errors from availability is a defensible choice, but it has to be a *choice*, not an accident."
合格 SLO 一句话四要素(示例原文):
"*99.9% of HTTP requests will return a 2xx or 3xx response in under 300 ms, measured at the edge load balancer, over a rolling 28-day window.* That sentence has all four required parts: the SLI, the target, the measurement point, and the window. Drop any one and the SLO becomes unenforceable."
九位成本表(30 天窗口允许宕机时长):99% → 7h12m;99.5% → 3h36m;99.9% → 43m12s;99.95% → 21m36s;99.99% → 4m19s;99.999% → 26s。"Each additional nine costs roughly 10× in engineering effort."
SLA 与 SLO 的缓冲关系:
"SLA < SLO. If your SLA promises 99.9%, your internal SLO is 99.95% (or tighter)... Teams that set SLO = SLA are running their entire engineering organization on the wrong side of the tripwire."
"Not every service has an SLA. Every production service should have an SLO."
预算=货币 + 政策三档:healthy → 自由发布;burn rate 升高 → 放慢、加评审;exhausted → "no non-essential releases until it regenerates"。
2026 延伸——AI Agent 也烧预算(与用户的工作流直接相关):
"An agent that auto-restarts a stuck pod is a contributor to your availability SLI, exactly like a human SRE running kubectl rollout restart is. If it gets it wrong twice in a week, it should hit the same budget gate the humans do.""...keep the SLI measured at the user — *did the request succeed?* — so the budget remains the honest broker regardless of who or what is driving the change."
三、跨源综合:六个环节如何咬合
- SLI 是翻译层:把"感觉不稳定"翻译成
good/total比率(Workbook 的统一形态),并且尽量贴近用户测(Book Ch.4 的客户端延迟论证 + cloudandsre 的 "a 200 OK that the client never saw is not a success")。 - SLO = 比率 + 目标 + 测量点 + 窗口,四要素缺一即不可执行(cloudandsre);首版从现状向下取整(Workbook worked example:97.123% → 97%),宁松勿紧(Book Ch.4)。
- 错误预算 = 100% − SLO,是"另一个 SLO 的 SLO"(Book Ch.4);它的存在让"可靠性 vs 迭代"从立场之争变成预算会计(Book Ch.3),但必须配套书面政策才有牙(Workbook Ch.2)。
- 告警的对象是烧率不是错误率本身:多窗口多烧率(1h×14.4 / 6h×6 → page;3d×1 → ticket),短窗口=长窗口/12(Workbook Ch.5);低流量场景必须做适应性改造,否则一个失败请求就能打爆 99.9% 的月度预算。
- 故障响应 = 角色分离 + 活文档 + 先止血:单人项目里角色分离退化为"分阶段"——先当 Operator 止血并留证据,再当 Commander 记时间线,恢复后才当 Planner 写复盘(Book Ch.14)。
- 复盘 = 事先定义的触发条件 + 免责措辞 + 行动项闭环:行动项表(mitigate/prevent/process/other + owner + ticket)直接对接错误预算政策(生产冻结行就写在官方模板里);"Where we got lucky"把侥幸转成机制(Book Ch.15 + 附录)。
来源间的小分歧(记录以便选型):
- 可用性口径:Book Ch.3 说 Google 内部用请求成功率而非时间口径(分布式下总有分区活着);个人服务两者都可以算,拨测产出的时间口径反而更容易实现。
- 窗口口径:rolling 4 周(贴近用户体验)vs 日历季度(对齐规划)——Workbook 的方案是两者并用:4 周 rolling 做日常决策,季度汇总做项目规划。
四、个人服务可执行版(本篇的落地产出)
以下是我基于上述来源为"一个人维护的服务"(如跑在 VPS/NAS 上的 API、博客、n8n、家庭自动化)整理的最小可行方案。每一步都标注了来源依据;未经实测的部分明确标注"推断"。
4.1 第一周就能做完的最小闭环
| 步骤 | 动作 | 依据 |
|---|---|---|
| 1 | 写下 ≤3 条用户旅程(例:打开首页 / 调 API / Webhook 回调) | Workbook Ch.2 起步配方 |
| 2 | 选 1 个现成数据源当 SLI 实现:服务器日志 / 云监控面板 / 外部拨测(UptimeRobot、healthchecks.io) | Workbook Ch.2 SLI 实现 + Ch.5 低流量方案 1 |
| 3 | 用最近 4 周数据算现状,向下取整定首版 SLO(拨测间隔 5min × 4 周 ≈ 8064 次,现状 99.6% → SLO 定 99%) | Workbook Ch.2 worked example + Book Ch.4 "宁松勿紧" |
| 4 | 写一页 SLO 文档(要素见 4.3),同时写一句话错误预算政策 | Workbook Ch.2 documenting 清单 |
| 5 | 配两层通知(见 4.4),别配 page | Workbook Ch.5 Table 5-8 + 低流量节 |
| 6 | 建一个空的事故模板 + 复盘模板文件(字段见 4.5/4.6) | Book Ch.14 活文档 + 附录模板 |
4.2 首版数字怎么定(示例演算,非实测)
以拨测可用性 SLI(5 分钟一次 HTTP GET,4 周 rolling ≈ 8,064 次探测)为例:
- 现状 99.6%(4 周内约 32 次失败探测)→ 首版 SLO 99% → 错误预算 = 1% × 8,064 ≈ 80 次失败探测/4 周;
- 一次 30 分钟没发现的宕机 ≈ 6 次失败 = 7.5% 预算——够写进复盘但远不至于冻结;
- 政策:预算耗尽 → 未来一周冻结非必要部署,只修可靠性问题(对应 Workbook Ch.2 政策动作 3 + 官方复盘模板的 freeze 行);
- 稳定两个窗口后再收紧到 99.5% 或 99.9%(Book Ch.4:"先松后紧")。
- ⚠️ 注意低流量陷阱(Workbook Ch.5):如果改用真实请求计数,10 req/h 的服务一次失败就是 1000× 烧率——低流量个人服务优先用时间/拨测口径,或明确降低 SLO。
4.3 一页 SLO 文档骨架(对应 Workbook Ch.2 必含清单)
# SLO 文档:<服务名>
- 作者/批准:<你的名字>;批准日期:<date>;下次复审:<date+1个月>
- 服务简介:一句话(谁用、干什么、跑在哪)
- 用户旅程(≤3 条):…
- SLO:
- Availability: 每次探测返回 2xx/3xx 的比例 ≥ 99%,拨测 5min/次,4 周 rolling
- Latency:(可选,二期再加)
- SLI 实现:UptimeRobot 外部拨测(规格 vs 实现的差异:测不到"服务活着但数据库坏了")
- 错误预算:100% − 99% = 1% ≈ 80 次失败探测 / 4 周;数字依据:2026-10 现状 99.6% 向下取整
- 错误预算政策:预算耗尽 → 冻结非必要发布 1 周,优先修可靠性 bug;争议时回到本档复审4.4 告警配置(个人版两层,不打扰睡眠)
| 层 | 条件 | 通知 | 对应 Workbook 参数 |
|---|---|---|---|
| 看一眼 | 6h 窗口烧率 > 6(5% 预算) | 邮件/IM,白天处理 | Page 6h/30m 6× |
| 必须处理 | 3d 窗口烧率 > 1(10% 预算) | 邮件 + 每日摘要 | Ticket 3d/6h 1× |
| 完全宕机 | 连续 2 次拨测失败 | 即时推送(这是唯一值得吵醒你的) | 症状式告警(qyxuo8eu-3 笔记原则) |
(低流量下 1h×14.4 的 page 层不适用——Workbook Ch.5 低流量节;用"连续 N 次拨测失败"替代,属我的适配推断。)
4.5 事故响应单人 SOP(半页纸)
- 宣布:集中排查 15 分钟没解决 / 用户可见 / 需要动生产配置 → 算事故,开活文档(Book Ch.14 判据的个人化阈值);
- 止血优先:"Stop the bleeding, restore service, and preserve the evidence"——先回滚/重启/切 fallback,保留日志和现场再清理;
- 单一生产入口:事故期间所有变更只从一条路径进(Ops 唯一原则的单身版:别让昨晚的你和今早的你并行改生产);
- 活文档:一个
incident-<date>.md,只记时间戳 + 动作 + 观察三列,乱就乱点,"messy, but must be functional"; - 退出判据:恢复后 30 分钟指标正常才算结束(官方模板的 exit criterion)。
4.6 复盘模板(个人版,字段抄官方附录)
# 复盘:<一句话标题>
- 日期 / 状态(行动项是否闭环)
- 摘要:一次线
- 影响:烧掉 X% 错误预算 / N 个用户请求 / M 分钟
- 根因(系统性原因,不是"我手滑") / 触发器
- 处置与检测:怎么发现的(监控 or 手动发现=监控失明,单独记)
- 行动项表:| 行动 | 类型(mitigate/prevent/process/other) | Owner | Ticket |
- What went well / What went wrong / **Where we got lucky**
- 时间线(尽量记,事后能补多少补多少)触发条件事先写死(贴在 SLO 文档里):数据丢失任何量级 / 手动介入(回滚、重启) / 恢复超 1 小时 / 监控失明才发现的事故——全盘照抄 Book Ch.15 清单并按个人服务降阈值。对自己免责:复盘假设"当时的我基于当时的信息做了合理的事",问题永远指向系统(缺监控、缺 runbook、缺测试),这是防止个人复盘沦为自责并最终弃写的唯一办法(Book Ch.15 blameless 论证的单人推广,属推断)。
4.7 与我既有笔记体系的接驳
- 监控与告警语法细节 →
2026-11-01-knowledge-qyxuo8eu-3-Prometheus-Grafana监控实践…精读.md(本篇 4.4 的烧率规则可直接翻它找 PromQL 实现); - 埋点与 trace →
2026-11-01-knowledge-qyxuo8eu-2-OpenTelemetry入门…(SLI 实现升级路径:拨测 → OTel 指标); - 部署与回滚 →
2026-11-01-knowledge-x8tjinj_-1-DockerCompose个人服务部署…(错误预算政策的"冻结发布"落在 compose 工作流上); - 发布节奏 →
2026-11-01-knowledge-x8tjinj_-3-CICD最小闭环…(预算充足=可以激进发布,预算见底=CI 里只合并 fix); - Durability SLI →
2026-10-31-knowledge-fexxiou8-4-数据备份与恢复演练-321原则…(备份的可恢复性就是一个 storage durability SLI)。
五、七维提取(本主题)
| 维度 | 内容 |
|---|---|
| 核心问题 | 如何把"服务够不够可靠"从主观感受变成可测量、可决策、可复盘的运营机制——并且让单人维护的个人服务也能跑得起这套机制 |
| 主要主张 | 100% 可靠是错误目标;SLI 统一写成 good/total 比率、贴近用户测;SLO 四要素(比率+目标+测量点+窗口)缺一不可;错误预算必须配书面政策;告警对烧率不对静态阈值;故障响应靠角色分离与活文档;复盘必须免责且行动项闭环 |
| 方法 | 官方书/Workbook 给配方(起步步骤、SLI 菜单、决策矩阵、告警参数表、模板);第三方文章做压缩与 2026 语境延伸(AI Agent 烧预算);本笔记再加一层"个人服务适配"推导并逐条标注来源 |
| 数据与证据 | 关键数字均可溯源:99.99%=52.56min/年;$1M 收入加一个九只值 $900;ISP 背景错误率 0.01%–1%;3M 请求@99.9%=3000 错误预算;worked example 97.123%→97%(预算 109,897);烧率表 1×/2×/10×/1000× = 30d/15d/3d/43min;推荐告警 2%@1h(14.4×)/5%@6h(6×)/10%@3d(1×);九位表 99.9%=43m12s/30d。证据强度:一手官方文档 + 自洽算例,但多为 Google 规模经验,无个人服务规模的实测数据 |
| 结论 | SLO 是测量工具,错误预算是资源分配工具,烧率告警是防守工具,事故 SOP 是止血工具,复盘是改进工具——五者构成闭环;"You should start using SLOs and error budgets today" |
| 局限 | ① 7/8 来源来自 Google 生态,组织假设是"有产品经理+SRE 双方",个人服务需要自行退化角色(本笔记的适配部分属推断,未实测);② Google 的请求口径/双区可用性数学在个人服务上用不上;③ Workbook 的 Example SLO Document 附录未抓到(URL 404),SLO 文档骨架按正文章节清单重构;④ 低流量告警的个人化参数(连续 N 次拨测失败)无官方出处;⑤ 官方数字为 2016/2018 出版内容,2026 年工具生态已变(如 Sloth/Grafana SLO 等现成工具)但方法论未过时 |
| 对我的启发 | 个人服务最小闭环一周可建成:拨测 SLI + 4 周 rolling SLO(现状取整)+ 一句话预算政策 + 两层通知 + 半页事故 SOP + 抄官方字段的复盘模板;错误预算的"自我执法"特性天然适合单人开发者;AI Agent 自动化运维(我的工作流场景)应被纳入同一预算约束,SLI 永远测在用户侧 |
六、这次学到什么
- 错误预算的本质是把价值观写进算术。组织里"稳定派 vs 迭代派"的争吵,数学上等价于"预算还剩多少";对单人开发者,它是"今天折腾新东西还是修可靠性"的客观裁判——而且它让守规矩变成自发行为(self-policing),因为烧完预算的代价是你自己承受。
- 先测量、后定标、宁松勿紧。首版 SLO 从现状向下取整(97.123% → 97%)看起来不体面,但它保证 SLO 一开始就可辩护、可执行;先建立反馈回路,再收紧目标。"An SLO of 100% means you only have time to be reactive."
- 低流量是个人服务的第一性约束,不是细节。官方的多烧率告警体系默认"请求多到能平滑统计";10 req/h 的服务里一个失败就是 1000× 烧率。个人服务应优先用拨测(时间口径)或主动降 SLO,这是把官方方法论迁移到自己场景时最容易被忽略的一步。
- 文档即基础设施:活文档(事故中)、SLO 文档(决策时)、复盘(事故后)三种文档各管一段;"An unreviewed postmortem might as well never have existed"——写了不复盘等于没写,闭环在行动项表和错误预算政策里。
- 免责是一种工程手段,不是情商。blameless 假设"当事人基于当时信息做了合理决策",从而把分析推向系统缺陷;单人场景里它保护的是"未来的自己还愿意诚实记录"。
*检索与阅读方式备注:8 个来源均于 2026-11-03 通过 web-reader 抓取全文精读(WebFetch 直连 sre.google 因网络原因失败,curl exit 35);引文均为抓取原文中的英文原句,翻译为笔记作者所加。本文档仅写入 notes 目录,未改动 matrix.csv、backlog、nightly-log.md 及其他条目。*