Testing APIs
API tests in a pipeline
Tests that only run when someone remembers don't protect much. How API tests run by themselves on every change — CI, exit codes, reports, secrets, and keeping the pipeline trusted.
A test you run by hand protects the API on the days you remember to run it. A test that runs by itself on every change protects it every day. That's CI — continuous integration: every time someone pushes code, a pipeline builds it and runs the tests, and a failure stops the change before it reaches users.
How a pipeline runs API tests
A pipeline is a list of steps a CI service (GitHub Actions, GitLab CI, Azure DevOps, Jenkins…) runs on a fresh machine:
- get the code;
- start the API (or deploy it to a test environment);
- run the API tests against it;
- pass or fail the whole run based on the result;
- keep the reports.
The tests live in the repository, next to the code, so a change to the API and the change to its tests arrive together and are reviewed together.
Exit codes: how a pipeline knows
Command-line tools tell the pipeline how it went with an exit code: 0 means passed, anything else means failed. The pipeline stops on non-zero. So the test runner must exit with 1 when a test fails — and only then. (A runner that exits 0 on failure makes a pipeline that is always green and protects nothing.)
APITestCenter's command line uses: 0 passed · 1 a test or target failed · 2 couldn't run (a typo in a name, no such environment).
Reports people can read
When a pipeline fails, someone has to find out why, quickly:
- JUnit XML — the standard test-report format; CI services show it as a list of passed and failed tests, with the reason.
- HTML — a readable report to download.
- JSON — for tools and dashboards.
A good failure message says what and why: "Get order: 1 of 3 checks failed: $.total should be number (it is string)", not "assertion failed".
Secrets in pipelines
Your test files are committed, so they must hold no secrets. Write {{token}} in the test, and give the value to the pipeline as a secret variable in the CI service's settings. The pipeline passes it in at run time (--var token=$TOKEN), and CI services hide it in logs.
Use a dedicated test account for the pipeline, with only the permissions the tests need.
What to run where
Not every test belongs on every push:
| When | What | Why |
|---|---|---|
| Every push / pull request | Fast functional and contract tests (seconds to a minute) | Fast feedback before merging |
| After deploying to a test environment | The full collection, end-to-end flows | Real environment, real config |
| Nightly / before a release | Load tests, long flows, soak tests | Too slow or heavy for every push |
A smoke load test (a few users for 30 seconds) can run on every deploy to catch a big slowdown early; serious load tests run on a schedule, against an environment built for them.
Keeping the pipeline trusted
A pipeline people don't trust gets ignored. The enemy is the flaky test — one that fails sometimes for no real reason: a shared test record someone else changed, a timing check with no headroom, a test that depends on another test's leftovers.
- Make tests independent — each creates what it needs and cleans up.
- Don't depend on order between tests unless it's an explicit chain.
- Give time checks headroom; leave tight performance limits to load tests.
- When a test is flaky, fix it or remove it — never "just re-run until green".
Key takeaways
- CI runs your API tests on every change, so regressions stop before they reach users.
- The runner's exit code decides pass or fail; JUnit reports show why. Secrets come from the CI service, never from committed files.
- Fast tests on every push, heavy ones on a schedule — and fix flaky tests, or nobody will trust the pipeline.
In the app: Command line and CI · From the app to your pipeline
Next: Why test performance