SRE Interview Questions
10 questions for junior, mid-level and senior SRE interviews, with model answers, code examples and common mistakes. Answer each one out loud first, then reveal the answer.
SLO vs SLI vs SLA Differences
// interview question
Your team just launched a new API service. Your manager asks you to set up SLOs for it. Can you walk me through what SLOs, SLIs, and SLAs are, and how they relate to each other?
Sample answer
An SLI (Service Level Indicator) is a measurement of your service's behavior -- something you can actually observe and put a number on. For example, the percentage of HTTP requests that return in under 300ms, or the ratio of successful responses to total responses. An SLO (Service Level Objective) is a target you set for an SLI. You're saying "we want this SLI to meet this threshold over this time window." For example, "99.9% of requests should succeed over a rolling 30-day window." An SLA (Service Level Agreement) is a contract with your customers that says what happens if you miss your targets. It has business consequences -- refunds, credits, or penalties. Not every service needs an SLA, but every production service should have SLOs. The relationship flows like this: you pick SLIs (what to measure), set SLOs (what target to hit), and if you make a promise to customers, that becomes an SLA. Your SLOs should always be tighter than your SLAs. If your SLA promises 99.9% availability, your internal SLO should be something like 99.95%. That gap gives you a buffer to catch problems before they become contract violations.
Why this matters
This is a foundational SRE question. You're testing whether the candidate understands the relationship between these three concepts or just memorized definitions. Strong candidates will explain the hierarchy (SLI feeds SLO, SLO is stricter than SLA) and give concrete examples. Weak candidates mix up SLOs and SLAs or can't give real numbers.
Code examples
SLO definition in a config file (OpenSLO format)
apiVersion: openslo/v1
kind: SLO
metadata:
name: api-availability
spec:
service: payment-api
description: Payment API availability SLO
budgetingMethod: Occurrences
objectives:
- displayName: Availability
target: 0.999
op: gte
indicator:
metadata:
name: availability-ratio
spec:
ratioMetric:
good:
source: prometheus
queryType: promql
query: sum(rate(http_requests_total{status!~"5.."}[5m]))
total:
source: prometheus
queryType: promql
query: sum(rate(http_requests_total[5m]))Quick availability calculation from logs
# Count total requests and errors in the last hour
total=$(grep "$(date -d '1 hour ago' '+%Y-%m-%d %H')" access.log | wc -l)
errors=$(grep "$(date -d '1 hour ago' '+%Y-%m-%d %H')" access.log | grep ' 5[0-9][0-9] ' | wc -l)
# Calculate availability
echo "scale=6; ($total - $errors) / $total * 100" | bc
# Output: 99.972000Common mistakes to avoid
- Treating SLOs and SLAs as the same thing. SLAs are contracts with business consequences; SLOs are internal engineering targets.
- Setting SLOs at 100%. No service should target 100% reliability -- it leaves zero room for deployments, maintenance, or any changes at all.
- Picking SLIs that don't reflect user experience, like CPU usage instead of request latency or error rate.
Likely follow-ups
- Why should your SLO always be stricter than your SLA?
- Can you have SLOs without SLAs? When would you do that?
- What happens if you set your SLO target too high, like 99.99% for an internal tool?
Answer out loud first, then check yourself against the model answer.
Application Performance Optimization
// interview question
How do you identify and resolve performance bottlenecks in a production application?
Sample answer
Performance optimization process: 1) Measure first - use APM tools (Datadog, New Relic) to identify slow endpoints. 2) Profile - CPU profiling, memory analysis, database query analysis. 3) Common bottlenecks: N+1 queries, missing indexes, synchronous operations that should be async, memory leaks, inefficient algorithms. 4) Load test to find breaking points. 5) Optimize incrementally - cache frequently accessed data, optimize database queries, implement connection pooling, use CDNs for static assets. Always measure impact of changes.
Why this matters
Premature optimization is the root of all evil, but measured optimization is essential. Start with observability - you can't improve what you can't measure. APM tools show where time is spent. Database queries are often the bottleneck. Caching (Redis, CDN) can dramatically improve performance but adds complexity. Load testing reveals how the system behaves under stress.
Code examples
Database query analysis
# PostgreSQL slow query log
# In postgresql.conf:
# log_min_duration_statement = 1000 # Log queries > 1s
# Analyze query plan
EXPLAIN ANALYZE SELECT * FROM orders
WHERE user_id = 123
ORDER BY created_at DESC
LIMIT 10;
# Find missing indexes
SELECT schemaname, tablename,
seq_scan, seq_tup_read,
idx_scan, idx_tup_fetch
FROM pg_stat_user_tables
WHERE seq_scan > idx_scan
ORDER BY seq_tup_read DESC;Redis caching pattern
# Kubernetes Redis deployment for caching
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis-cache
spec:
replicas: 1
selector:
matchLabels:
app: redis
template:
spec:
containers:
- name: redis
image: redis:7-alpine
command: ["redis-server"]
args:
- "--maxmemory"
- "256mb"
- "--maxmemory-policy"
- "allkeys-lru" # Evict least recently used
resources:
limits:
memory: 300MiCommon mistakes to avoid
- Optimizing without measuring - guessing where bottlenecks are
- Caching everything without considering cache invalidation complexity
- Fixing symptoms (adding more servers) instead of root causes
Likely follow-ups
- When should you use caching vs. optimizing the underlying operation?
- How do you prevent cache stampede problems?
- What tools do you use for load testing?
Answer out loud first, then check yourself against the model answer.
SLI, SLO, and SLA Definitions
// interview question
Explain the difference between SLI, SLO, and SLA with examples.
Sample answer
SLI (Service Level Indicator) is a metric measuring service behavior (e.g., latency, error rate). SLO (Service Level Objective) is an internal target for that metric (e.g., 99.9% availability). SLA (Service Level Agreement) is an external contract with customers specifying consequences for missing targets. Example: SLI = request latency, SLO = 95% of requests < 200ms, SLA = credits if monthly availability drops below 99.5%.
Why this matters
These concepts form the foundation of reliability engineering. SLIs help measure user experience, SLOs balance reliability with feature velocity, and SLAs formalize business commitments. Without clear SLOs, teams either over-engineer for unnecessary reliability or ship too fast and burn out on incidents.
Code examples
SLO definition example
# Service Level Objectives for API service
slos:
availability:
description: Service should be available
indicator: (successful_requests / total_requests) * 100
target: 99.9%
window: 30d
latency:
description: API should respond quickly
indicator: p95_response_time_ms
target: 95% of requests < 200ms
window: 30d
error_budget: 0.1% # 43.2 minutes/month of allowed downtimeSLI measurement in Prometheus
# Availability SLI
sum(rate(http_requests_total{status!~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
# Latency SLI (P95)
histogram_quantile(0.95,
sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
)
# Error budget remaining
1 - (
(1 - availability_sli) / (1 - availability_target)
)Common mistakes to avoid
- Setting SLOs at 100% (impossible and prevents any changes)
- Confusing SLOs (internal targets) with SLAs (customer contracts)
- Not tracking error budget, making it meaningless
Likely follow-ups
- How do you handle situations when you're close to exhausting your error budget?
- What's the difference between availability and reliability?
- How do you choose appropriate SLO targets?
Answer out loud first, then check yourself against the model answer.
Choosing the Right SLIs
// interview question
You're joining a team that runs a checkout service for an e-commerce platform. There are no SLOs yet. How would you decide which SLIs to track?
Sample answer
Start with what your users care about. For a checkout service, users care about three things: can I complete my purchase (availability), how long does it take (latency), and does it charge me the right amount (correctness). I'd pick SLIs from the four golden signals: latency, error rate, throughput, and saturation. But not all of them become SLOs. For a checkout service, I'd focus on: 1. Availability -- the ratio of successful checkout requests (non-5xx) to total requests. This is your most important SLI because a broken checkout directly costs money. 2. Latency -- the 95th percentile response time for checkout completion. Not the average, because averages hide tail latency. A p95 of 2 seconds means 5% of your users are waiting longer than that, and those are often paying customers mid-purchase. 3. Correctness -- the percentage of checkout transactions where the charged amount matches the cart total. This is easy to overlook but a wrong charge is worse than a slow page. I'd skip throughput as an SLO target because it varies naturally with traffic patterns. It's good to monitor but bad as an objective. I'd also look at the SLIs from the user's perspective, not from the server's. Measure at the load balancer or edge, not inside your application, because that's closer to what the user actually experiences.
Why this matters
This question tests practical thinking. You want to see if the candidate can connect SLIs to business impact rather than just listing textbook metrics. Strong candidates will reason from user experience backward to metrics. They'll mention why certain metrics matter more for this specific service type. Watch for candidates who just list generic metrics without tying them to the checkout use case.
Code examples
Prometheus queries for checkout SLIs
# Availability SLI: ratio of successful requests
sum(rate(http_requests_total{service="checkout", status!~"5.."}[30d]))
/
sum(rate(http_requests_total{service="checkout"}[30d]))
# Latency SLI: proportion of requests under 2s (good events / total events)
sum(rate(http_request_duration_seconds_bucket{service="checkout", le="2.0"}[30d]))
/
sum(rate(http_request_duration_seconds_count{service="checkout"}[30d]))
# Correctness SLI: transactions with matching amounts
sum(rate(checkout_transactions_correct_total[30d]))
/
sum(rate(checkout_transactions_total[30d]))Grafana dashboard panel for SLI tracking
panels:
- title: "Checkout Availability (30d rolling)"
type: stat
targets:
- expr: |
sum(rate(http_requests_total{service="checkout", status!~"5.."}[30d]))
/
sum(rate(http_requests_total{service="checkout"}[30d]))
* 100
fieldConfig:
defaults:
unit: percent
thresholds:
steps:
- color: red
value: 99.0
- color: orange
value: 99.5
- color: green
value: 99.9Common mistakes to avoid
- Using averages instead of percentiles for latency. An average of 200ms can hide a p99 of 10 seconds, which means some users are having a terrible experience.
- Picking infrastructure metrics (CPU, memory, disk) as SLIs instead of user-facing metrics. High CPU doesn't necessarily mean users are affected.
- Setting the same SLIs for every service regardless of what the service does. A batch job and a real-time API need very different indicators.
Likely follow-ups
- Would you pick different SLIs for an internal batch processing service vs. this user-facing checkout service?
- How do you handle SLIs when the service depends on third-party payment providers?
- Where would you instrument the measurement -- at the load balancer, in the application code, or using synthetic probes?
- How many SLOs should a single service have? What's the risk of having too many?
Answer out loud first, then check yourself against the model answer.
Error Budget Management
// interview question
Your service has a 99.9% availability SLO over a 30-day window. How much downtime does that give you, and what do you actually do with that error budget day-to-day?
Sample answer
A 99.9% SLO over 30 days gives you an error budget of 0.1%, which is about 43.2 minutes of total downtime (or roughly 4,320 failed requests out of every 4.32 million). The error budget is not just a number to watch -- it's a decision-making tool. Here's how I'd use it: When the budget is healthy (say, 70%+ remaining), the team moves fast. Ship features, run experiments, do risky deployments. You have room to absorb failures. This is the whole point -- error budgets turn the reliability vs. velocity argument into a data-driven decision. When the budget is getting low (under 30% remaining with time left in the window), slow down. Require extra review on deployments, hold off on risky changes, and focus on reliability work like adding retry logic or improving rollback speed. When the budget is burned (0% remaining), stop feature deployments entirely. The team shifts to reliability work only: fixing the issues that burned the budget, improving monitoring, adding automated rollbacks. This is the error budget policy, and it needs to be agreed on before you need it, not during an incident. The key insight is that error budgets align incentives. Product teams want to ship fast, SRE teams want reliability. The error budget gives both sides a shared framework: "We can ship this risky feature because we have budget" or "We need to pause and fix things because we're out of budget."
Why this matters
This question separates candidates who understand error budgets as a cultural and operational tool from those who just know the math. The calculation is table stakes -- what matters is whether they can explain the error budget policy and how it changes team behavior. Strong candidates will talk about the policy being pre-agreed and about using budgets to balance velocity and reliability.
Code examples
Error budget calculation and burn rate
from datetime import datetime, timedelta
def calculate_error_budget(slo_target: float, window_days: int, total_requests: int, failed_requests: int):
"""Calculate remaining error budget."""
budget_fraction = 1 - slo_target # e.g., 0.001 for 99.9%
total_allowed_failures = total_requests * budget_fraction
remaining_failures = total_allowed_failures - failed_requests
budget_remaining_pct = (remaining_failures / total_allowed_failures) * 100
return {
"slo_target": f"{slo_target * 100}%",
"allowed_failures": int(total_allowed_failures),
"actual_failures": failed_requests,
"remaining_failures": int(remaining_failures),
"budget_remaining": f"{budget_remaining_pct:.1f}%",
"budget_exhausted": budget_remaining_pct <= 0,
}
# Example: 99.9% SLO, 30-day window
result = calculate_error_budget(
slo_target=0.999,
window_days=30,
total_requests=4_320_000,
failed_requests=2_800
)
print(result)
# {'slo_target': '99.9%', 'allowed_failures': 4320,
# 'actual_failures': 2800, 'remaining_failures': 1520,
# 'budget_remaining': '35.2%', 'budget_exhausted': False}Error budget policy document (team agreement)
error_budget_policy:
service: payment-api
slo_target: "99.9%"
window: 30d
thresholds:
healthy:
budget_remaining: ">50%"
actions:
- Normal deployment cadence
- Feature work proceeds as planned
- Risky experiments allowed with rollback plan
warning:
budget_remaining: "20-50%"
actions:
- Reduce deployment frequency
- Require extra review on risky changes
- Start investigating top error sources
- No experimental features
critical:
budget_remaining: "<20%"
actions:
- Feature freeze -- reliability work only
- All hands on fixing error sources
- Post-incident reviews for budget-burning events
- Escalate to engineering leadership
exhausted:
budget_remaining: "0%"
actions:
- Full deployment freeze
- Mandatory blameless postmortem
- Reliability improvements before any new features
- Weekly check-ins until budget recoversCommon mistakes to avoid
- Treating the error budget as just a monitoring metric instead of a decision-making tool with real consequences (like deployment freezes).
- Not having a pre-agreed error budget policy. If you decide what to do after the budget is burned, the conversation becomes political instead of data-driven.
- Confusing burn rate with budget remaining. A high burn rate with 90% budget left is more urgent than a low burn rate with 20% left -- you need to react to the rate of consumption, not just the current level.
Likely follow-ups
- What's the difference between a time-based and request-based error budget, and when would you pick one over the other?
- How do you handle a situation where a single incident burns 80% of the error budget? Is that different from many small incidents adding up to 80%?
- What do you do when the product team pushes back on a feature freeze because of a burned error budget?
- How do rolling windows vs. calendar windows affect error budget behavior?
Answer out loud first, then check yourself against the model answer.
Capacity Planning and Scaling
// interview question
How do you approach capacity planning for a growing production system? What metrics and strategies do you use?
Sample answer
Capacity planning ensures systems can handle current and future load. Process: 1) Establish baselines - current CPU, memory, disk, network utilization and request rates. 2) Understand growth patterns - historical trends, seasonality, planned campaigns. 3) Define headroom - typically 30-40% buffer for unexpected spikes. 4) Model scenarios - what happens at 2x, 5x, 10x traffic? 5) Identify bottlenecks - database connections, API rate limits, stateful components. 6) Plan scaling strategy - vertical vs horizontal, auto-scaling policies. 7) Load test regularly. Review capacity quarterly.
Why this matters
Capacity planning is both art and science. Too much capacity wastes money; too little causes outages. Cloud auto-scaling helps but doesn't solve everything - databases, third-party APIs, and stateful services often can't scale horizontally. Senior engineers must think about bottlenecks that aren't obvious and plan for Black Friday scenarios before they happen.
Code examples
Horizontal Pod Autoscaler
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: api-server
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: api-server
minReplicas: 3
maxReplicas: 50
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Pods
pods:
metric:
name: requests_per_second
target:
type: AverageValue
averageValue: 1000
behavior:
scaleUp:
stabilizationWindowSeconds: 60
scaleDown:
stabilizationWindowSeconds: 300Capacity analysis queries
# PromQL: CPU headroom percentage
100 - (
avg(rate(container_cpu_usage_seconds_total{pod=~"api-.*"}[5m]))
/
avg(kube_pod_container_resource_limits{resource="cpu"})
) * 100
# Current vs limit memory usage
sum(container_memory_working_set_bytes{pod=~"api-.*"})
/
sum(kube_pod_container_resource_limits{resource="memory"})
# Request rate trend (7-day growth)
predict_linear(rate(http_requests_total[1h])[7d:1h], 30 * 24 * 3600)Common mistakes to avoid
- Only planning for average load, not peak load
- Forgetting about dependent services that may become bottlenecks
- Not accounting for the time it takes to scale (cold start, provisioning)
Likely follow-ups
- How do you handle capacity planning for stateful services like databases?
- What is the difference between scaling up and scaling out?
- How do you account for third-party API rate limits in capacity planning?
Answer out loud first, then check yourself against the model answer.
Chaos Engineering Practices
// interview question
What is chaos engineering and how would you implement it safely in a production environment?
Sample answer
Chaos engineering is the practice of intentionally injecting failures to test system resilience. Implementation: start with hypothesis about system behavior, define minimal blast radius, begin in staging, use tools like Chaos Monkey or Litmus, inject failures (pod terminations, network latency, resource exhaustion), monitor golden signals, automate rollback on unexpected impact, gradually expand to production during low-traffic periods with team monitoring.
Why this matters
Chaos engineering's goal is finding weaknesses before they find you. It builds confidence in system resilience and uncovers hidden dependencies. The key is controlled, observable experiments with safety measures - not random destruction. Netflix pioneered this with Chaos Monkey, and it's now standard practice at large-scale organizations.
Code examples
Litmus ChaosEngine example
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: nginx-chaos
spec:
appinfo:
appns: default
applabel: 'app=nginx'
appkind: deployment
chaosServiceAccount: litmus-admin
experiments:
- name: pod-delete
spec:
components:
env:
- name: TOTAL_CHAOS_DURATION
value: '30'
- name: CHAOS_INTERVAL
value: '10'
- name: FORCE
value: 'false'Manual chaos injection
# Network latency injection
tc qdisc add dev eth0 root netem delay 100ms 20ms
# CPU stress
stress-ng --cpu 4 --timeout 60s
# Kill random pod in deployment
kubectl delete pod $(kubectl get pods -l app=myapp -o name | shuf -n 1)
# Network partition simulation
iptables -A INPUT -s 10.0.0.0/8 -j DROPCommon mistakes to avoid
- Running chaos experiments without proper monitoring in place
- Starting with production before validating in staging
- No automated rollback mechanism when experiments go wrong
Likely follow-ups
- How do you define and control the blast radius of chaos experiments?
- What metrics should you monitor during chaos experiments?
- How do you convince leadership that intentionally breaking production is valuable?
Answer out loud first, then check yourself against the model answer.
Incident Postmortems
// interview question
Describe a production incident you handled and how you structured the postmortem. What makes a good blameless postmortem?
Sample answer
A blameless postmortem focuses on systems improvement, not individual blame. Structure: incident timeline (what happened), impact assessment (users affected, duration), root cause analysis (5 whys technique), contributing factors, action items with owners and deadlines, and lessons learned. Share widely to prevent similar issues across teams.
Why this matters
Blameless postmortems create psychological safety where engineers report issues honestly. If people fear punishment, they hide problems. Focus questions on 'how did our systems allow this?' not 'who made a mistake?'. This cultural shift is essential for building reliable systems and learning organizations.
Code examples
Postmortem template
# Incident Postmortem: API Outage 2024-01-15
## Summary
API service was unavailable for 47 minutes due to database connection pool exhaustion.
## Impact
- Duration: 10:15 - 11:02 UTC (47 minutes)
- Users affected: ~12,000 (30% of active users)
- Revenue impact: ~$8,500 in failed transactions
## Timeline
- 10:15 - PagerDuty alert: API error rate > 50%
- 10:18 - On-call engineer begins investigation
- 10:25 - Root cause identified: DB connection pool
- 10:45 - Fix deployed: increased pool size
- 11:02 - All systems normal
## Root Cause
A slow query caused connections to be held longer, exhausting the pool.
## Action Items
1. [ ] Add connection pool monitoring (@alice, due: Jan 20)
2. [ ] Implement query timeout (@bob, due: Jan 22)
3. [ ] Add runbook for connection issues (@charlie, due: Jan 25)Common mistakes to avoid
- Blaming individuals instead of focusing on systemic improvements
- Not following up on action items from previous postmortems
- Writing postmortems only for major incidents (minor ones teach too)
Likely follow-ups
- How do you ensure action items from postmortems actually get completed?
- What's the difference between root cause and contributing factors?
- How do you balance incident response speed with thoroughness?
Answer out loud first, then check yourself against the model answer.
Error Budget Burn Investigation
// interview question
It's Monday morning. You check the dashboard and see that your service burned 80% of its monthly error budget over the weekend. Walk me through how you'd investigate this and what you'd do next.
Sample answer
First, I need to understand the shape of the burn. Was it one big incident or a slow leak? I'd start by pulling up the error budget burn-down chart for the weekend. If I see a cliff -- a steep drop at a specific time -- that's an incident. I'd look at the deployment history: did someone ship something Friday afternoon? Check the change log, git history, and any config changes. Most budget burns trace back to a change. If the chart shows a steady downward slope, that's a slow degradation. This is harder to track down. I'd look at the error breakdown by endpoint, by error code, and by dependency. Often it's a single endpoint or a downstream service that started flaking. Maybe a database connection pool is saturating under weekend batch job load, or a third-party API started rate-limiting you. Once I know the cause, here's what I'd do: 1. Fix the immediate problem. Roll back the bad deploy, scale up the bottleneck, or disable the flaky feature behind a feature flag. 2. Trigger the error budget policy. With only 20% budget left, we're in the critical zone. That means feature freeze -- no new deployments except reliability fixes. I'd communicate this to the product team immediately, not as a request but as the policy kicking in. 3. Run a blameless postmortem. Focus on why the burn wasn't caught sooner. Were the burn rate alerts misconfigured? Was nobody watching the on-call channel? Did the slow-burn alert fire but get ignored? 4. Build in prevention. If a Friday deploy caused this, maybe we need a policy of no deploys after Thursday, or better canary analysis. If a dependency caused it, we need circuit breakers or fallbacks. The bigger question is why 80% of the budget burned before anyone noticed. That's a monitoring and alerting gap, and it's usually more important to fix than the original incident.
Why this matters
This is a scenario-based senior question that tests incident response thinking, not just SLO knowledge. You want to see structured investigation (not random guessing), awareness of the error budget policy, and a focus on systemic improvements. The best candidates will immediately ask about the burn shape (cliff vs. slope) because the investigation path is completely different. Watch for whether they mention the process gap (why wasn't this caught?) -- that's what separates senior from mid-level thinking.
Code examples
Investigation queries to diagnose budget burn
# 1. Error budget remaining over time (burn-down chart)
1 - (
sum(increase(http_requests_total{service="checkout", status=~"5.."}[30d]))
/
(sum(increase(http_requests_total{service="checkout"}[30d])) * 0.001)
)
# 2. Error rate by endpoint -- find the hot spot
sum by (endpoint) (
rate(http_requests_total{service="checkout", status=~"5.."}[1h])
)
/ sum by (endpoint) (
rate(http_requests_total{service="checkout"}[1h])
)
# 3. Error rate by status code -- understand the failure mode
sum by (status) (
increase(http_requests_total{service="checkout", status=~"5.."}[2d])
)
# 4. Dependency error rates -- is it us or a downstream?
sum by (dependency) (
rate(dependency_requests_total{service="checkout", status="error"}[2d])
)Quick weekend incident timeline from deploy history
# Check what was deployed over the weekend
kubectl rollout history deployment/checkout-api -n production
# REVISION CHANGE-CAUSE
# 14 deploy: v2.3.1 (Thu Apr 9 17:42:00)
# 15 deploy: v2.3.2 (Fri Apr 10 16:58:00) <-- Friday afternoon deploy
# 16 rollback to v2.3.1 (Mon Apr 13 09:15:00)
# Check the diff between the Friday deploy and the previous version
git log --oneline v2.3.1..v2.3.2
# a1b2c3d Update payment retry logic
# d4e5f6g Remove connection pool size limit <-- suspicious
# Correlate with error spike timing
kubectl logs deployment/checkout-api -n production --since=48h \
| grep -c 'connection refused'
# 14,832Postmortem template for budget burn events
postmortem:
title: "80% error budget burn - checkout API (Apr 10-12)"
date: 2025-04-13
authors: ["oncall-engineer"]
status: action-items-pending
summary: |
Friday deploy v2.3.2 removed connection pool limits,
causing DB connection exhaustion under weekend batch load.
80% of monthly error budget consumed over 48 hours.
timeline:
- time: "Fri 16:58"
event: "v2.3.2 deployed to production"
- time: "Fri 23:00"
event: "Weekend batch jobs begin, connection pool grows unbounded"
- time: "Sat 02:15"
event: "Slow-burn alert fires (10x burn rate)"
- time: "Sat 02:15"
event: "Alert acknowledged but not investigated (oncall assumed transient)"
- time: "Mon 09:00"
event: "Budget burn discovered during morning review"
- time: "Mon 09:15"
event: "Rollback to v2.3.1"
root_cause: |
Connection pool size limit was removed in v2.3.2.
Under normal weekday traffic this was fine, but weekend
batch jobs created enough concurrent connections to
exhaust the database connection limit.
action_items:
- owner: sre-team
action: "Add connection pool limits to deploy-time config validation"
priority: P1
- owner: oncall-lead
action: "Update runbook: slow-burn alerts require investigation, not just ack"
priority: P1
- owner: platform-team
action: "Add canary analysis gate for Friday deploys"
priority: P2Common mistakes to avoid
- Jumping straight to 'fix the bug' without first understanding the full scope. Is it one endpoint or the whole service? Is it still happening or did it stop? You need the full picture before acting.
- Skipping the process failure analysis. The budget burned over a weekend and nobody noticed until Monday -- that's a bigger problem than whatever caused the errors. Always ask why the alerting and response process failed.
- Treating the error budget policy as optional. If the policy says feature freeze at 20% budget remaining, enforce it. Making exceptions undermines the entire system and trains teams to ignore SLOs.
Likely follow-ups
- How would your response differ if the burn was caused by a third-party dependency you don't control?
- The product team has a major launch next week and is pushing back on the feature freeze. How do you handle that conversation?
- Should you adjust the SLO target after repeated budget burns, or is the problem always in the system?
Answer out loud first, then check yourself against the model answer.
SLO-Based Alerting and Burn Rates
// interview question
Traditional alerting fires when error rate crosses a static threshold, like 'alert if errors > 1%'. What's wrong with that approach, and how would you set up SLO-based alerting instead?
Sample answer
Static threshold alerts have two problems. First, they fire too often on things that don't matter. A brief 2% error spike for 30 seconds won't affect your monthly SLO at all, but a static alert pages someone at 3am for it. Second, they miss slow burns. If your error rate sits at 0.5% for days, that might look fine against a 1% threshold, but it's quietly eating your 99.9% error budget. SLO-based alerting fixes both problems by alerting on burn rate -- how fast you're consuming your error budget relative to your window. The approach uses multi-window, multi-burn-rate alerts. You set up two kinds: 1. A fast-burn alert for acute incidents: "At this rate, you'll exhaust your entire 30-day error budget in 2 hours." This is a 36x burn rate (burning 36 times faster than sustainable). You check this over a short window (5 minutes) and a longer confirmation window (1 hour). This pages someone immediately. 2. A slow-burn alert for gradual degradation: "At this rate, you'll exhaust your budget in 3 days." This is a 10x burn rate, checked over a 30-minute short window and 6-hour long window. This creates a ticket, not a page. The two-window check prevents false positives. The short window catches the current condition, while the long window confirms it's not just a blip. Both must be true before the alert fires. This directly maps to Google's SRE book approach and it works well in practice because you only get paged for things that actually threaten your SLO.
Why this matters
This is a strong mid-to-senior question because it tests whether the candidate has actually operated SLO-based systems or just read about them. The multi-window burn rate concept trips up a lot of people. Listen for whether they understand why two windows are needed (short window alone gives false positives, long window alone alerts too late). Candidates who have done this in practice will mention specific burn rate numbers and the tradeoff between alert sensitivity and noise.
Code examples
Prometheus alerting rules for multi-window burn rate
groups:
- name: slo-burn-rate-alerts
rules:
# Fast burn: 36x burn rate -- pages immediately
# Will exhaust 30-day budget in ~2 hours
- alert: HighErrorBudgetBurn_Critical
expr: |
(
1 - (sum(rate(http_requests_total{service="checkout", status!~"5.."}[5m]))
/ sum(rate(http_requests_total{service="checkout"}[5m])))
) > (36 * 0.001)
and
(
1 - (sum(rate(http_requests_total{service="checkout", status!~"5.."}[1h]))
/ sum(rate(http_requests_total{service="checkout"}[1h])))
) > (36 * 0.001)
for: 2m
labels:
severity: critical
annotations:
summary: "Checkout error budget burning fast (36x rate)"
description: "At current rate, 30-day error budget exhausts in ~2 hours."
runbook: "https://runbooks.internal/slo-budget-burn"
# Slow burn: 10x burn rate -- creates a ticket
# Will exhaust 30-day budget in ~3 days
- alert: HighErrorBudgetBurn_Warning
expr: |
(
1 - (sum(rate(http_requests_total{service="checkout", status!~"5.."}[30m]))
/ sum(rate(http_requests_total{service="checkout"}[30m])))
) > (10 * 0.001)
and
(
1 - (sum(rate(http_requests_total{service="checkout", status!~"5.."}[6h]))
/ sum(rate(http_requests_total{service="checkout"}[6h])))
) > (10 * 0.001)
for: 5m
labels:
severity: warning
annotations:
summary: "Checkout error budget burn elevated (10x rate)"
description: "At current rate, 30-day error budget exhausts in ~3 days."Calculate current burn rate from Prometheus
# Query current 1-hour error rate
ERROR_RATE=$(curl -s 'http://prometheus:9090/api/v1/query' \
--data-urlencode 'query=1 - (sum(rate(http_requests_total{service="checkout", status!~"5.."}[1h])) / sum(rate(http_requests_total{service="checkout"}[1h])))' \
| jq -r '.data.result[0].value[1]')
# Calculate burn rate relative to SLO budget
SLO_BUDGET=0.001 # 1 - 0.999
BURN_RATE=$(echo "scale=2; $ERROR_RATE / $SLO_BUDGET" | bc)
echo "Current error rate: $ERROR_RATE"
echo "Burn rate: ${BURN_RATE}x"
echo "Time to budget exhaustion: $(echo "scale=1; 720 / $BURN_RATE" | bc) hours"
# Output:
# Current error rate: 0.0035
# Burn rate: 3.50x
# Time to budget exhaustion: 205.7 hoursCommon mistakes to avoid
- Setting up SLO alerts but still keeping the old static threshold alerts running alongside them. This leads to alert fatigue because you get paged twice for the same incident from different systems.
- Using only a single window for burn rate alerts. A short window alone gives false positives on brief spikes. A long window alone means you don't catch fast-moving incidents until significant budget is already gone.
- Forgetting that burn rate alerts need enough request volume to be meaningful. For low-traffic services, a single failed request can show a 100x burn rate over a 5-minute window.
Likely follow-ups
- Why do you need both a short window and a long window? What goes wrong if you only use one?
- How would you tune these burn rates for a service that has natural traffic spikes, like an e-commerce site during sales events?
- What's the relationship between burn rate multiplier and the time to budget exhaustion? How do you pick the right multipliers?
- How do you handle SLO alerting for services with very low traffic where statistical significance is a problem?
Answer out loud first, then check yourself against the model answer.
More interview topics
Also worth your time on this topic
SLO, SLI and Error Budget Checklist
A step-by-step checklist for defining service level objectives, picking the right service level indicators, and using error budgets to make better decisions about reliability vs. feature velocity.
45-90 minutes
SLOs, SLIs, and Error Budgets: A Practical Implementation Guide
Your service went down at 2 AM and nobody could agree on whether it was "bad enough" to page someone. SLOs, SLIs, and error budgets fix that. Here is how to define, measure, and act on them with real Prometheus queries and alerting rules.
Kubernetes Horizontal Pod Autoscaler
Configure and test Horizontal Pod Autoscaler to automatically scale applications based on CPU and memory usage.
90 minutes