Hi! I’m Oleg, a QA Engineering Manager focused on test strategy, automation, and AI-assisted QA workflows.

This is a continuation of a series on where AI actually earns its place in QA — and this time the answer starts somewhere less glamorous than AI: with the OpenAPI spec.

The short version: we generate a Java API client from Swagger, write tests against that generated client, measure coverage against the spec automatically in GitLab CI, and hand the resulting gaps to an LLM as a structured JSON report so it can write the missing tests.

The problem: tests that drift away from the contract

A very common failure mode in API testing is that tests live separately from the API contract.

A QA engineer assembles URLs by hand, writes JSON by hand, parses responses by hand. Swagger exists somewhere nearby — analysts and developers keep it updated, the tests nominally check the same API — but there is no actual link between them. Nothing breaks when they diverge. You find out later, usually in production.

We decided to make Swagger the source of truth for API tests rather than documentation nobody reads.

Using our Cohorts service as the running example, the pipeline looks like this:

cohorts.swagger.yaml ↓ OpenAPI Generator + Mustache templates ↓ cohorts.api.CohortsApi cohorts.model.* ↓ JUnit + RestAssured tests ↓ Swagger Coverage ↓ HTML / JSON coverage report ↓ LLM skills that generate the missing tests
Tests don't call given().get("/api/..."). They call a generated client with generated models, both produced from the OpenAPI spec.

How Swagger becomes a Java client

Take the endpoint that creates a cohort: POST /api/v1/cohort.

The OpenAPI file describes it at two levels. The first is paths — which endpoint, which HTTP method, the operationId, the request schema to send and the response schema to expect:

/api/v1/cohort: post: tags: - cohorts summary: Create an empty cohort operationId: createCohort requestBody: content: application/json: schema: $ref: '#/components/schemas/CreateCohortsRq' responses: 201: description: Success content: application/json: schema: $ref: '#/components/schemas/CreateCohortsRs'
The second is components.schemas — the fields that make up the DTOs referenced above via $ref:

CreateCohortsRq: type: object required: - name - description properties: name: type: string description: Cohort name description: type: string description: Cohort description case_type: type: integer description: Cohort use case type CreateCohortsRs: type: object required: - id properties: id: type: integer description: Cohort ID
After generation, that becomes Java:

operationId: createCohort → cohortsDefaultApi().createCohort() CreateCohortsRq → cohorts.model.CreateCohortsRq CreateCohortsRs → cohorts.model.CreateCohortsRs
If Swagger has a name field, the Java model has a builder method .name(...). If Swagger has id, the response model has getId().

This is the part that matters: QA doesn't invent the DTO. The test can only work with what the contract describes.

Why we use Mustache templates

OpenAPI Generator works fine out of the box — the standard templates will produce API classes, models, methods for path/query/body parameters, and a basic client wrapper.

But when the generated code doesn't quite match your project's style, you can customise it with Mustache templates. This is a standard OpenAPI Generator mechanism: take the default template and change how the generated API classes or models look.

We don't use Mustache because generation is impossible without it. We use it because we want the generated code to sit comfortably inside our existing test framework.

src/test/resources/openapi-templates/api.mustache src/test/resources/openapi-templates/pojo.mustache
They're wired into the Gradle generation task:

templateDir.set("$rootDir/src/test/resources/openapi-templates")
api.mustache controls the generation of API classes — for example cohorts.api.CohortsApi. It's what defines the shape of the generated methods:

public CreateCohortOper createCohort() { return new CreateCohortOper(createReqSpec()); }
The request execution methods:

public <T> T execute(Function<Response, T> handler) { return handler.apply( RestAssured.given() .spec(reqSpec.build()) .expect() .spec(respSpec.build()) .when() .request(REQ_METHOD, REQ_URI) ); }
And typed response deserialization:

public CreateCohortsRs executeAs(Function<Response, Response> handler) { TypeRef<CreateCohortsRs> type = new TypeRef<CreateCohortsRs>() {}; return execute(handler).as(type); }
So execute(...), executeAs(...), .idPath(...), .pageQuery(...), .body(...) aren't RestAssured magic on their own. They're the shape of the generated client, which we get from OpenAPI Generator plus our templates.

pojo.mustache handles the models — CreateCohortsRq, CreateCohortsRs, ListUsersWithPagination. It describes what the generated model classes look like: fields from the Swagger schema, Jackson annotations, getters and setters, builder-style methods, equals/hashCode/toString.

In practice this lets us keep the convenience of generation while controlling the shape of what comes out: Allure steps, a fluent API, execute/executeAs, how RequestSpecification is handled, and the style of the models.

Wiring it together with Gradle

OpenAPI Generator is added in build.gradle:

plugins { id "org.openapi.generator" version "7.18.0" }
The generated code is registered as a test source set:

sourceSets { test { java { srcDir('build/generate-resources/src/main/java') } } }
Which means the Java classes OpenAPI Generator drops into build/generate-resources/src/main/java are available in tests like any other test code.

Each service gets two Gradle tasks: one to download and prepare the Swagger file, one to generate the Java RestAssured client. For Cohorts:

tasks.register('downloadCohortsSwaggerFile', Exec) { group = "openapi tools" def url = "https://example.com/cohort/openapi_cohort.yaml" def outputFile = file("${projectDir}/src/test/resources/cohorts.swagger.yaml") if (outputFile.exists()) { commandLine("curl", "-z", outputFile, "-o", outputFile, url) } else { commandLine("curl", "-o", outputFile, url) } }
A note from experience: sometimes the spec needs a little preparation before generation, to get it into a shape that both OpenAPI Generator and the coverage tool handle predictably. This happens when a service's Swagger has grown organically — inconsistent in places, overly generic descriptions in others, not specific enough for the generator somewhere else. I won't go into our internal cleanup details here. The point is that what you feed the generator should reliably produce a client and models.

Then generation runs:

tasks.register("openApiCohorts", GenerateTask) { dependsOn 'downloadCohortsSwaggerFile' apiPackage.set("cohorts.api") modelPackage.set("cohorts.model") generatorName = "java" library = "rest-assured" inputSpec.set("$rootDir/src/test/resources/cohorts.swagger.yaml") outputDir.set("$buildDir/generate-resources") templateDir.set("$rootDir/src/test/resources/openapi-templates") }
Producing:

cohorts.api.CohortsApi cohorts.ApiClient cohorts.model.CreateCohortsRq cohorts.model.CreateCohortsRs cohorts.model.ListUsersWithPagination ...
And to make all of it happen automatically before the tests compile:

compileTestJava.dependsOn tasks.openApiBanners, tasks.openApiGifts, tasks.openApiCohorts

So when CI runs the Gradle test task, this chain fires on its own:

downloadCohortsSwaggerFile ↓ openApiCohorts ↓ compileTestJava ↓ test
Using the generated client in tests

Building cohortsDefaultApi

We don't construct a RestAssured request by hand every time. There's a base method that returns the generated API class:

public class BaseCohortTests { protected CohortsApi cohortsDefaultApi() { return ApiClient.api(ApiClient.Config.apiConfig().reqSpecSupplier( () -> new RequestSpecBuilder() .addRequestSpecification(CohortsSpec.defaultReqSpec()) )).cohorts(); } }
A RequestSpecification is passed into the generated client, built in CohortsSpec.defaultReqSpec():

public static RequestSpecification defaultReqSpec() { return new RequestSpecBuilder() .setBaseUri(config.COHORTS_URL()) .setConfig(Specifications.hiddenTokensConfig()) .addFilter(new AllureRestAssured()) .addFilter(new SwaggerCoverageV3RestAssured( new FileSystemOutputWriter(Paths.get("build/swagger-coverage-output/cohorts")))) .addHeader("x-token", valuesConfig.cohortsToken()) .addHeader("x-device", "ekb") .log(LogDetail.ALL) .build(); }
There are two distinct layers here, and keeping them separate is what makes the whole thing maintainable.

The generated client owns everything that came from Swagger: endpoint, HTTP method, path parameters, query parameters, request body, response model.

The RequestSpecification owns the infrastructure of a particular test run: base URL, tokens, technical headers, logging, the Allure filter, the Swagger Coverage filter.

The Allure filter adds HTTP requests and responses to the Allure report — handy for triaging failures, unrelated to client generation. The Swagger Coverage filter does something different: it records the actual HTTP calls into build/swagger-coverage-output so a coverage report can be built after the run.

The test itself

@Test @DisplayName("Create an empty cohort") void createCohort() { CreateCohortsRq request = CreateCohortsRq.builder() .name("Autotest cohort " + UUID.randomUUID()) .description("Cohort created by an autotest") .build(); CreateCohortsRs result = cohortsDefaultApi() .createCohort() .body(request) .executeAs(response -> response .then() .statusCode(201) .extract().response()); cohortId = result.getId(); step("And I verify the cohort was created", () -> { assertNotNull(result, "Response must not be null"); assertNotNull(cohortId, "Created cohort ID must not be null"); assertNotEquals(0, cohortId, "Created cohort ID must not be 0"); }); }
Almost every line traces back to Swagger: CreateCohortsRq is the request model, createCohort() is the operationId, body(request) is the request body, CreateCohortsRs is the response schema.

Path and query parameters are generated too

Another example — fetching the list of cards in a cohort. In Swagger:

/api/v1/cohort/{id}/users: get: operationId: getUsers parameters: - in: path name: id - in: query name: page - in: query name: size - in: query name: search_pattern responses: 200: content: application/json: schema: $ref: '#/components/schemas/ListUsersWithPagination' Which generates a fluent API: operationId: getUsers → .getUsers() path parameter id → .idPath(...) query parameter page → .pageQuery(...) query parameter size → .sizeQuery(...) query parameter search_pattern → .searchPatternQuery(...) response schema → ListUsersWithPagination result In the test: @Test @DisplayName("Get the list of cards in a cohort") void getUsers() { step("Given cohort ID: %s".formatted(cohortId)); ListUsersWithPagination result = cohortsDefaultApi() .getUsers() .idPath(cohortId) .pageQuery(PAGE) .sizeQuery(PAGE_SIZE) .searchPatternQuery(number) .executeAs(response -> response .then() .statusCode(200) .extract().response()); step("And I verify the response body", () -> { assertNotNull(result, "Response must not be null"); assertNotNull(result.getNumbers(), "Card list must not be null"); assertTrue(result.getNumbers().contains(number), "Card list must contain the test card"); assertNotNull(result.getPageNumber(), "pageNumber must not be null"); assertNotNull(result.getPageSize(), "pageSize must not be null"); assertNotNull(result.getTotalPages(), "totalPages must not be null"); assertNotNull(result.getTotalCount(), "totalCount must not be null"); }); }
QA doesn't need to remember what a query parameter is called in the URL. If it's described in Swagger, the generator makes a method for it.

execute vs executeAs

execute(...) makes a plain RestAssured call and hands control back out. Useful when the response has no body:

cohortsDefaultApi() .addUsers() .idPath(cohortId) .body(request) .execute(Validatable::then) .statusCode(204);
executeAs(...) is for when Swagger describes a response schema. Under the hood it executes the request and deserializes the response into the generated model.

For us this isn't just a convenience wrapper — it's an extra contract check. If Swagger says the response should be CreateCohortsRs, the test tries to read the actual JSON as exactly that. When the backend returns something that doesn't match the contract, the test fails on deserialization or on the assertions. That catches:

Swagger calls the field id, the API returns cohort_id

Swagger types id as integer, the API returns a string

Swagger describes an object, the API returns an array

with strict deserialization, the backend added a field and nobody updated Swagger

Which is how you gradually train a team to keep the contract current — the contract breaking becomes your problem, immediately, instead of a documentation debt nobody feels.

The next problem: we couldn't see what was actually covered

At this point we had a large pile of API tests using generated clients and models, much closer to the contract than the old hand-rolled ones.

But a different problem appeared: we had no idea how much of the API those tests really covered. Lots of tests, all green — but which Swagger operations do they actually call? Which endpoints are untouched entirely? Which are only partially covered? Checking that by reading code got old fast.

So we took a different approach: record the actual HTTP calls during the test run, then compare that log against the OpenAPI spec afterwards. Tests run as usual; after the run you get a report showing what part of the contract was exercised.

Swagger Coverage in GitLab CI

For this we use a modified version of swagger-coverage, originally created by Victor Orlovsky. The core idea fit us well: split the process in two stages. During the test run, a RestAssured filter collects the actual HTTP calls. After the run, a separate CI job compares those calls against the OpenAPI spec and builds a coverage report.

The original project hasn't been maintained for a while, and we hit bugs and limitations while integrating it. So Andrey Poletaev (@fenixnow) forked it, fixed the problems, and adapted the tool to our needs. The fork lives at github.com/fenixnow/swagger-coverage, and its README covers configuration and running it via Docker.

During tests: collecting the calls

The RestAssured request specification includes the coverage filter:

.addFilter(new SwaggerCoverageV3RestAssured( new FileSystemOutputWriter(Paths.get("build/swagger-coverage-output/cohorts")))) It hooks into the HTTP execution chain and sees every request that goes through our RequestSpecification: test calls a generated API method ↓ generated client executes a RestAssured request ↓ RestAssured passes request/response through filters ↓ SwaggerCoverageV3RestAssured records the call ↓ data is written to build/swagger-coverage-output/cohorts
Worth being clear about: the filter does not build an HTML report during the test. It only collects raw data — which endpoint was called, with which HTTP method, and what status came back.

The CI jobs

After the tests, separate coverage jobs run in a Docker image containing the modified swagger-coverage:

swagger-coverage-report--banners

swagger-coverage-report--gifts

swagger-coverage-report--cohorts

Because the image ships swagger-coverage-commandline, each job stays simple — it only needs the spec, the input directory, and a config:

swagger-coverage-commandline \ -s src/test/resources/cohorts.swagger.yaml \ -i build/swagger-coverage-output/cohorts \ -c swagger-coverage-cohorts-config.json
Where -s is the OpenAPI spec, -i is the data collected during the tests, and -c is the report rules config.

The job then copies out two artifacts:

cohorts-coverage-report.html → a readable report for humans

cohorts-coverage-results.json → structured JSON for an LLM

The HTML report looks like this:

Getting the links into the Merge Request

The coverage job stores the HTML and JSON URLs in a dotenv artifact, and a separate publish-coverage-report job posts a comment into the Merge Request with links to both report types for every service.

Why two formats: the HTML is for a person — open it in a browser, look at the summary, methods, groups, coverage variants, and quickly see where the gaps are. The LLM JSON exists because HTML is genuinely bad input for a model: awkward to feed in, hard to parse reliably. So the modified swagger-coverage gained a separate JSON export aimed specifically at LLM consumption.

The whole chain

GitLab CI ↓ test job runs the Gradle test task ↓ Gradle downloads Swagger ↓ OpenAPI Generator generates Java clients/models ↓ JUnit/RestAssured tests run through the generated client ↓ SwaggerCoverageV3RestAssured writes build/swagger-coverage-output ↓ GitLab stores swagger files and coverage output as artifacts ↓ coverage job runs swagger-coverage-commandline in Docker ↓ HTML and JSON coverage reports are generated ↓ publish-coverage-report posts the links into the Merge Request ↓ an LLM reads the JSON report and knows which tests are still missing
That connection is what makes it automatic. Nobody downloads Swagger by hand, runs the generator, hunts for coverage output, or assembles a report. The pipeline does it, and the team gets a readable HTML report plus a JSON file suitable for model analysis.

Connecting an LLM to API coverage

Once we had coverage reports, the obvious next step was to use them as structured input for an LLM rather than just a web page for humans.

The JSON report contains a coverage summary, a list of paths, the state of each operation, and the requirements not yet closed by tests. Simplified:

{ "summary": { "total_operations": 89, "fully_covered": 5, "partially_covered": 20, "not_covered": 128, "coverage_percent": 13.0 }, "paths": { "/api/v1/cohort/{id}/users": { "GET": { "state": "PARTY", "coverage": "3/6", "deprecated": false, "requirements": { "status_codes": [], "parameters": [], "body": [], "properties": [] } } } } }
Some people on the team work in Cursor, others in Claude Code, so we described the test generation rules as skills for both tools.

The important design decision was splitting the skills by responsibility. The first analyses the coverage report and acts as an orchestrator. The second writes the actual tests.

First skill: analysis and orchestration

It takes two files as input — cohorts.swagger.yaml and cohorts-coverage-results.json. The spec tells it the contract: endpoints, operationIds, parameters, request bodies, response models, statuses. The coverage JSON tells it what's covered and what isn't.

It reads the operation states:

EMPTY → endpoint not covered by tests at all

PARTY → endpoint partially covered

FULL → endpoint fully covered

Then looks at the requirements to work out what specifically is missing:

status_codes → a required HTTP status was never asserted

parameters → a path/query/header parameter was never used

body → the request body isn't covered

properties → response fields were never checked

And decides what to do:

EMPTY endpoint → propose a basic happy-path test

PARTY endpoint → add the missing assertions or scenarios

FULL endpoint → skip

deprecated endpoint → skip

Then it calls the second skill and passes not the whole report, but a specific task: which endpoint to cover, which operationId to use, which request/response schemas to look at, which requirements to close.

Second skill: generating tests to the project's rules

The second skill doesn't analyse anything globally. It receives a concrete task, reads the OpenAPI spec, finds the right generated API method and models, checks the existing tests, and only then writes the new test.

It follows our constraints:

use the generated Swagger client

use the generated models

don't hand-write raw JSON

don't invent your own DTOs

don't assemble URLs manually

put tests in the correct package

pick executeAs or execute according to the contract

assert through the generated models' getters

wrap steps in Allure

So for an uncovered Cohorts endpoint, it goes to Swagger, finds the operation, reads the operationId and schemas, and works out which generated code to use:

API method → cohortsDefaultApi().getUsers()

path parameter id → .idPath(...)

query parameter page → .pageQuery(...)

query parameter size → .sizeQuery(...)

response model → ListUsersWithPagination

execution method → executeAs(...)

This is the crux of it: the LLM must not invent its own way of talking to the API. It has to use the same contract-bound client that every other test uses. Otherwise you get tests that pass while quietly bypassing the very mechanism that makes the suite trustworthy.

When Swagger alone isn't enough

Being honest about the limits: not every test can be generated from Swagger and a coverage report.

For simple CRUD it's often plenty — there's an endpoint, a clear request body, a clear response body, and you can build a happy path with basic assertions.

But complex microservices exist where calling a single method requires a lot of preconditions:

create an entity in another system

prepare a user in a particular state

obtain a valid token

enable a feature flag beforehand

pass parameters you cannot guess from Swagger

respect a business-level sequence of calls

Swagger describes the contract of one HTTP method well. It doesn't explain business context: where the data comes from, what state the system must be in, which steps precede the call.

For those cases we need test cases written by QA, with preconditions, steps, test data, and expected results. So we have separate skills that work from test cases in Allure TestOps instead of from a coverage report. We built our own MCP server for TestOps so an LLM can pull a test case, parse it, and turn it into an autotest:

Allure TestOps test case ↓ our MCP server for Allure TestOps ↓ an LLM skill reads preconditions, steps, and expected result ↓ the skill maps the steps onto the OpenAPI spec ↓ finds the generated API methods and models ↓ writes the test to the project's rules
So we're not trying to replace QA and test design with Swagger alone. Swagger owns the contract, coverage owns visibility into the gaps, and test cases own the complex business scenarios where a human description of preconditions and expected behaviour is unavoidable.

What about gRPC

Every example here is deliberately REST, where OpenAPI is the contract.

For gRPC the idea is the same — tests should lean on the contract — but instead of Swagger you have .proto files generating service stubs, RPC methods, and request/response messages. The technical implementation differs enough (different transport, different contract format, different generated classes, a different approach to coverage) that mixing both into one article would help nobody. REST here; gRPC deserves its own write-up.

Keeping generated code and test code apart

There are two kinds of code in the project.

Generated from Swagger, never written by hand:

cohorts.api.CohortsApi

cohorts.ApiClient

cohorts.model.CreateCohortsRq

cohorts.model.CreateCohortsRs

cohorts.model.ListUsers

cohorts.model.ListUsersWithPagination

And our test code — BaseCohortTests, CohortsSpec, CohortV1Tests — responsible for scenarios, steps, assertions, tokens, base URLs, and wiring up filters.

Simplified, the division of responsibility runs like this:

Swagger describes the contract ↓ OpenAPI Generator creates API classes and models ↓ Mustache templates define the shape of the generated code ↓ BaseCohortTests connects the generated client to the test infrastructure ↓ CohortV1Tests describes a specific test scenario
Generated code owns the technical API contract; our test code owns scenarios, data, steps, and assertions.

What this buys us

The main benefit is that the tests sit close to the contract.

Change an operationId in Swagger and the generated method name changes. Change the request body and the generated model changes. Remove a field from a response and the getter disappears. A lot of divergence becomes visible at compile time or during deserialization — not a week after release.

And the test stays readable:

CreateCohortsRs result = cohortsDefaultApi() .createCohort() .body(request) .executeAs(response -> response .then() .statusCode(201) .extract().response());
From that chain you can immediately see which API is called, what body is sent, what status is expected, and which model the response is deserialized into.

Swagger Coverage closes the second part of the problem: which operations from the spec were actually exercised, and which are still untouched. The LLM skills close the third: turning that report into new tests — not randomly, but to the project's rules. And test cases from TestOps cover the complex scenarios where Swagger isn't enough on its own.

The end result is that Swagger stops being a documentation page. It becomes a working contract between analysts, developers, and QA — and the coverage report stops being a pretty HTML artifact and becomes a backlog for the test suite.

What's next

This covered one slice of a bigger process: contract-driven API tests and a coverage report an LLM can read.

Next in the series: how we triage failed autotests through defects in Allure TestOps, tie that to an issue tracker, and use AI not only to generate tests but to analyse results and file tasks. And after that, a system where AI connects to our services, prepares test plans itself, verifies a task through the database, REST, and gRPC, and posts the final report straight into the tracker issue.

If you've built something similar on top of OpenAPI, I'd be curious what your coverage numbers looked like the first time you measured them. Ours were humbling.

Let’s Work Together

I’m open to QA leadership opportunities and collaborations with teams looking to improve their testing strategy, build scalable test automation, introduce AI-assisted QA workflows, and establish meaningful quality metrics.

If this sounds relevant to your team, feel free to connect with me on LinkedIn.