Testing APIs

Performance tests

Send the same request many times at once to see how fast it stays and whether it holds up — with six test types (Smoke, Load, Stress, Spike, Soak, Breakpoint), simulated users or a fixed request rate, and goals.

A performance test sends your request again and again, from many simulated users at the same time, for a while. It runs on your computer with k6, the open-source load-testing engine bundled with the app.

Press Run performance test (Ctrl+Shift+Enter) to start. Press Send first to check the answer.

Before it starts

Signing in. With OAuth 2.0, the token is fetched once before the test, and each simulated user renews it if the test outlasts it. With Run first, the sign-in request runs once and its values are shared.

One check request. The app sends your request once before the performance test. If the answer is not 2xx, the test does not start: the Response tab says Performance test not started and shows the answer — so a wrong token never turns into thousands of failed requests. If that status is what you expect (for example, an endpoint that answers 404 on purpose), press Run anyway. The test then starts without the check.

A safety question for heavy load. The app asks Start a … performance test against …? when the API is not on your own computer and:

  • the type is Stress, Spike, Soak or Breakpoint; or
  • the run reaches 25 or more simulated users, or 50 or more requests per second.

Press Start performance test to go ahead, or Cancel. Tests against localhost, 127.0.0.1, or ::1 never ask.

Test types

Open the Performance tab and pick a test type — then change anything you like. Each type asks one question, and its report answers it.

TypeWhat it doesThe questionWhat the report adds
Smoke2 users for 30 secondsDoes it work under a little load?— (the verdict says it)
Load10 users for 5 minutesDoes it hold up on a normal day?—
StressUp to 20 users over 30 s, hold 1 min, back downHow does it do above normal?At the peak: p95 and errors with the most users
Spike5 users, a sudden jump to 50, then back to 5Does it survive a burst — and recover?Recovery: p95 before, during and after the spike, and whether it came back
Soak10 users for 30 minutesDoes it slow down over time?Over time: p95 at the start against the end of the steady part
BreakpointAdds users steadily, up to 200 over 10 minutes, and stops when a target breaksHow much can it take?Breaking point: the most users (or requests per second) it handled within your targets, and where it broke
CustomYour own numbersAnything else—

Settings → Performance tests chooses the type a new test starts as. On the command line: atc run … --preset smoke|load|stress|spike|soak|breakpoint.

Breakpoint

A breakpoint test is meant to fail: it keeps adding users until your Goals are missed — p95 or the error rate — and then k6 stops it (10 seconds after the start at the earliest). With no goals set it uses p95 under 2 s and errors under 5%. A target counts as broken when it's missed for 3 seconds in a row, so one slow second isn't the limit. The report says, for example, It handled up to 60 users within p95 under 500 ms and errors under 1% (your targets). It broke at 70 users — p95 2.4 s. If it never broke, raise the top (Step by step on the Performance tab) and run again.

Run it against a test system, not production, and tell whoever runs the API first — it's built to overload it. On the command line it exits 1 unless it never broke, so use it to measure, not as a CI gate.

Simulated users or requests per second

Drive load by chooses how the load is measured:

  • Simulated users — each user sends a request, waits for the answer, optionally pauses (think time), and sends the next. A slow API therefore receives fewer requests. Good for “what do 50 people feel?”
  • Requests per second — the app keeps a fixed rate, whatever the API does, adding users as needed up to Max simulated users. This is the honest way to find how much an API can take. If the API is too slow to keep up, the report shows Requests not sent.

Shape over time

  • Ramp · hold · down — go up to the target over Ramp up, stay for Hold for, come down over Ramp down.
  • Step by step — a list of stages; each moves steadily to its target over its duration. Repeat a target to hold it. Add stage adds one (up to 20); ✕ removes one.

Write durations like 30s, 5m, or 1h.

Planned load, under the settings, draws the run before you start: users (or requests per second) over time, with the peak and the total length — for example 10 VUs · 5m.

Pacing options

Click Pacing options to show:

  • Think time (ms) and Random extra (ms) — a pause between a user's requests, like a person reading the page. They do not apply when holding a request rate.
  • Graceful stop — how long requests still in flight get to finish when the test ends. The default is 10s.

Limits

SettingAllowed
Simulated users (and stage targets in users)1–500
Requests / second (and stage targets in requests per second)1–100,000
Max simulated users1–2,000 (default 100)
StagesUp to 20
Think time and Random extra0–600,000 ms

While a setting is not valid, Run performance test is disabled and its tooltip says why.

Goals (pass or fail)

The Goals tab decides when a run passes. Without your own goals, the app uses built-in limits:

ResultWhen
Warningp95 is 500 ms or more, or 1% or more of requests fail
Failingp95 is 2 s or more, or 5% or more of requests fail — or, when the request has checks, fewer than 99% of them pass

Tick Use my own performance targets to set your own:

  • p95 must stay under (ms) — 95% of requests must be faster than this;
  • p99 must stay under (ms) — optional; 0 means no p99 target;
  • Error rate must stay under (%);
  • Checks must pass at least (%) — only used when the request has checks;
  • Stop the test early if a target is clearly missed — after a 10-second warm-up, the run ends as soon as a target fails. It saves time and stops hammering a struggling API.

Your own goals are sent to k6 as thresholds, and they decide the result and the report's colours. The built-in limits send nothing to k6.

Goals belong to the app, not to a request:

  • They are remembered on this computer and apply to every performance test and compare, until you change them or untick the box.
  • Runs you open from History are judged again with the goals you have now.
  • They are not saved with your requests, not included in the CI zip, and not used by the command line.

On the command line, set the same limits with --p95, --p99, --errors, --checks, and --abort. See Command line and CI.

While it runs

The Performance report fills in live: requests per second, response times, failures, and users, second by second. A countdown shows the time left, and the window title shows it too (0:42 left · …). Stop ends the run early; what ran so far is kept.

Notify me when done asks for a system notification when the run ends, so you can carry on with other work. The first time, you are asked to allow notifications. After that the button switches between Will notify and Notify me. A notification only appears when the app is not in front; while you are looking at it, a short message in the app is enough. If notifications are turned off in your system or browser settings, the button says Notifications blocked. See Working in the app.

Every finished run is saved in History. Reading the result: Reading the report.

Earlier runs and more room

  • Recent runs, above the Performance report, lists your 30 most recent runs with their result (the number next to it says how many). Pick one to open its report here. ◆ marks a reference run.
  • The expand button next to it widens the results to the whole window and hides the request editor. Press it again to bring the editor back. It is there on wide windows only.

Good habits

  1. Smoke first. A tiny run catches a wrong URL or token before a big one.
  2. Change one thing at a time — users, or the API version — and compare with a reference run.
  3. Test from close to the API when you can. Your network and computer are part of every number.
  4. Only test what you're allowed to. Heavy load on someone else's system can look like an attack.

Something unclear or missing? Write to support@apitestcenter.com — the guide improves with every question.