Performance and load testing
Finding the bottleneck
When an API slows down under load, something specific ran out first — CPU, a connection pool, a database lock, a slow dependency. The usual suspects, the signs each one leaves, and how to narrow it down.
A load test tells you that an API slows down at 200 users. The next question is why — and the answer is almost always that one thing ran out first. That thing is the bottleneck: the narrowest part of the bottle, which decides how fast everything flows. Make anything else faster and nothing changes; widen the bottleneck and the whole system speeds up — until the next-narrowest part becomes the bottleneck.
Queues: why "a bit slower" becomes "suddenly unusable"
Everything that can only do so much at once — a CPU, a pool of database connections, a single-threaded worker — has a queue in front of it. While requests arrive slower than it can handle them, the queue stays empty and response times stay flat. As arrivals get close to its limit, the queue grows faster and faster: at 80% busy, a request might wait as long as it takes to process; at 95%, many times longer. That's why response times look fine, fine, fine — and then shoot up over a small increase in load. The point where they bend is the capacity of the bottleneck.
The usual suspects
| Bottleneck | Signs under load | Common causes |
|---|---|---|
| CPU | Server CPU near 100%; times rise steadily with load | Heavy computation, inefficient code, JSON of huge answers, encryption, too few servers |
| Memory | Memory climbs over time; pauses; eventually crashes or restarts | Leaks, caches with no limit, loading whole tables into memory |
| Database | DB CPU or I/O high; specific requests slow; locks and timeouts | Missing indexes, slow queries, too many queries per request (N+1), lock contention on hot rows |
| Connection pool | Times jump at a fixed concurrency; "timeout waiting for connection" errors; DB itself not busy | Pool too small for the load; connections held too long (slow queries, forgotten closes) |
| Threads / workers | A fixed number of requests in progress, the rest wait; CPU not maxed | Too few worker threads; blocking calls (waiting on I/O) in a small pool |
| A slow dependency | One external call dominates the time; your servers idle | A downstream API, a payment provider, an auth server — slow, rate limited, or failing |
| Locks | Times grow with concurrency even at low CPU; deadlocks | Everyone updating the same row, counter or file |
| Network / bandwidth | Big answers slow; time in transfer, not in the server (low TTFB, high download) | Huge payloads, no compression, the test machine's own network |
| Rate limits / firewalls | Sudden 429s or 403s at a round number | A gateway, CDN or WAF protecting the API — by design |
| The load generator | Every request slows at once; the test machine's CPU is maxed; requests not sent | The machine running the test can't keep up |
The N+1 query: the most common one
An API returns a list of 50 orders. The code runs 1 query for the orders, then 1 more per order for its customer — 51 queries. Each is fast; together, they're slow, and they multiply under load. The fix (one query with a join, or a batch fetch) often makes an endpoint 10× faster. Signs: response time grows with the size of the answer, and the database sees many tiny identical queries.
How to narrow it down
- Find where the time goes in one request. Is it the network (connection, TLS, download) or the server (time to first byte)? A high TTFB means inside the API.
- Test endpoints separately. If
/productsstays fast and/ordersslows, the problem is in what/ordersdoes — not the servers in general. - Change one thing at a time and compare: more users, a bigger page size, a different endpoint. Two changes at once tell you nothing.
- Watch the server while you test. A load test only sees the outside. Look at the API's own monitoring at the same time: CPU, memory, the database's busiest queries, the connection pool, garbage-collection pauses, errors in the logs. The load test makes the problem happen; monitoring shows where.
- Find the knee. Run a step test (rising load in steps) and note where response times bend upward or errors start. That number is your current capacity.
- Rule out the test itself. If the load machine's CPU is maxed or requests weren't sent, the bottleneck may be the test. Run from a bigger machine, or closer to the API.
- Fix the bottleneck, then test again — the next one is now in charge.
Things that fool you
- Warm-up. The first minute is slower: caches fill, connections open, code gets optimised (JIT). Ramp up, and judge the steady part.
- Caching. The same request a thousand times may be served from a cache — fast and meaningless. Use test data so requests differ like real ones.
- Autoscaling. A cloud system may add servers mid-test: times rise, then fall. Know whether it's on, and how long it takes to react.
- Shared environments. Someone else's test on the same system makes your numbers worse. Book the environment.
- A different environment. A test server with a quarter of production's resources finds a quarter of its capacity — useful for comparing, risky for promising.
Key takeaways
- One resource runs out first — the bottleneck — and queues in front of it make response times bend sharply near its limit.
- The usual suspects: CPU, memory, database (and N+1 queries), connection pools, threads, slow dependencies, locks, rate limits — and the load machine itself.
- Narrow it down: time in the server or the network, one endpoint and one change at a time, and watch the server's monitoring during the test.
In the app: Reading the report · Compare
Next: Goals and regressions