Testing APIs

API tests in a pipeline

Tests that only run when someone remembers don't protect much. How API tests run by themselves on every change — CI, exit codes, reports, secrets, and keeping the pipeline trusted.

A test you run by hand protects the API on the days you remember to run it. A test that runs by itself on every change protects it every day. That's CI — continuous integration: every time someone pushes code, a pipeline builds it and runs the tests, and a failure stops the change before it reaches users.

How a pipeline runs API tests

A pipeline is a list of steps a CI service (GitHub Actions, GitLab CI, Azure DevOps, Jenkins…) runs on a fresh machine:

  1. get the code;
  2. start the API (or deploy it to a test environment);
  3. run the API tests against it;
  4. pass or fail the whole run based on the result;
  5. keep the reports.

The tests live in the repository, next to the code, so a change to the API and the change to its tests arrive together and are reviewed together.

Exit codes: how a pipeline knows

Command-line tools tell the pipeline how it went with an exit code: 0 means passed, anything else means failed. The pipeline stops on non-zero. So the test runner must exit with 1 when a test fails — and only then. (A runner that exits 0 on failure makes a pipeline that is always green and protects nothing.)

APITestCenter's command line uses: 0 passed · 1 a test or target failed · 2 couldn't run (a typo in a name, no such environment).

Reports people can read

When a pipeline fails, someone has to find out why, quickly:

  • JUnit XML — the standard test-report format; CI services show it as a list of passed and failed tests, with the reason.
  • HTML — a readable report to download.
  • JSON — for tools and dashboards.

A good failure message says what and why: "Get order: 1 of 3 checks failed: $.total should be number (it is string)", not "assertion failed".

Secrets in pipelines

Your test files are committed, so they must hold no secrets. Write {{token}} in the test, and give the value to the pipeline as a secret variable in the CI service's settings. The pipeline passes it in at run time (--var token=$TOKEN), and CI services hide it in logs.

Use a dedicated test account for the pipeline, with only the permissions the tests need.

What to run where

Not every test belongs on every push:

WhenWhatWhy
Every push / pull requestFast functional and contract tests (seconds to a minute)Fast feedback before merging
After deploying to a test environmentThe full collection, end-to-end flowsReal environment, real config
Nightly / before a releaseLoad tests, long flows, soak testsToo slow or heavy for every push

A smoke load test (a few users for 30 seconds) can run on every deploy to catch a big slowdown early; serious load tests run on a schedule, against an environment built for them.

Keeping the pipeline trusted

A pipeline people don't trust gets ignored. The enemy is the flaky test — one that fails sometimes for no real reason: a shared test record someone else changed, a timing check with no headroom, a test that depends on another test's leftovers.

  • Make tests independent — each creates what it needs and cleans up.
  • Don't depend on order between tests unless it's an explicit chain.
  • Give time checks headroom; leave tight performance limits to load tests.
  • When a test is flaky, fix it or remove it — never "just re-run until green".

Key takeaways

  • CI runs your API tests on every change, so regressions stop before they reach users.
  • The runner's exit code decides pass or fail; JUnit reports show why. Secrets come from the CI service, never from committed files.
  • Fast tests on every push, heavy ones on a schedule — and fix flaky tests, or nobody will trust the pipeline.

In the app: Command line and CI · From the app to your pipeline

Next: Why test performance

A question these lessons don’t answer? Write to support@apitestcenter.com — we add lessons for the questions people ask.