Performance and load testing
Goals and regressions
"Fast enough" needs a number. How to set performance goals (SLOs) people can agree on, turn them into pass/fail targets, and catch an API getting slower from one release to the next.
A load test without a goal ends with a shrug: "p95 was 340 ms… is that good?" A goal turns the result into an answer.
SLIs, SLOs and SLAs
Three related words you'll hear:
- An SLI (service level indicator) is what you measure: p95 response time, error rate, availability.
- An SLO (service level objective) is the target for it: "p95 under 300 ms; fewer than 0.1% errors". It's the team's own promise.
- An SLA (service level agreement) is a contract with a customer, usually looser than the SLO, with consequences (refunds) if it's broken.
For load testing, what you need is SLOs — goals specific enough to pass or fail.
Writing a good goal
A useful performance goal says what, how fast, for how many, and under what load:
- Use percentiles, not averages (reading results) — p95 for "almost everyone", p99 for the tail.
- Include errors. A fast API that fails 3% of the time hasn't met any reasonable goal.
- Tie it to a load. "Under 300 ms" at 5 users and at 500 users are different promises.
- Different endpoints, different goals. A search can take longer than a product page; a report export longer than both. One number for everything fits nothing.
- Start from the person. How long will the user wait on that screen? How many API calls are behind it? Work backwards to each call's share.
- Start from reality, then improve. If today's p95 is 900 ms, a 100 ms goal will just fail forever. Write down today's numbers, agree on a target, move towards it.
From goals to pass/fail
Load-test tools turn goals into thresholds: limits checked during and after the run. Breaking one fails the test — and in a pipeline, fails the build (API tests in a pipeline). Some tools can stop early when a goal is clearly missed, which saves time and stops hammering a system that's already struggling.
Remember the checks too: a goal like "at least 99.9% of answers pass their checks" makes sure speed isn't bought with wrong answers.
Regressions: getting slower without anyone noticing
A performance regression is when a change makes the API slower — a new feature adds a query, a library update is less efficient, a config change shrinks a pool. Each one is small; nobody notices; six months later the API is twice as slow.
The defence is comparing against a baseline:
- Run a load test you trust — same request, same load, same environment — and keep it as the baseline (a reference run).
- After each change, run the same test again and compare: p95, p99, errors, throughput.
- Decide in advance what counts as worse — "more than 10% slower at p95" — so the comparison is a decision, not a debate.
A few cautions:
- Compare like with like — same load, same environment, same data, similar time of day. Otherwise you're comparing the conditions, not the code.
- Expect noise. Two identical runs differ by a few percent. Small differences mean nothing; look for clear changes, or repeat the run.
- Look at the trend, not just the last pair: a slow creep over ten releases is invisible between any two of them.
Key takeaways
- SLI = what you measure, SLO = your target, SLA = the contract. Load tests need SLOs.
- A good goal: percentiles + errors + a load, per endpoint, starting from today's reality.
- Catch regressions by running the same test against a baseline after every change, with a limit decided in advance.
In the app: Goals (pass or fail) · Reference runs · Command line and CI