A client came to us with a straightforward question: could we review their test suite and confirm whether it provided adequate coverage?

Their development team had built a Playwright end-to-end suite for a production SaaS platform. The application included a public website, a workspace for professional firms, an administration panel and a client portal. AI had been used to accelerate the work, and the result appeared impressive: more than 170 tests, complete route coverage, clean TypeScript and a well-organized project structure. The CI pipeline was green, and the team had every reason to feel confident.

Then we audited the suite.

We extracted every validation message from the source code—302 in total—and checked which of those rules were exercised by the tests. Only 17 were covered. That is approximately 6%. None of the 129 rules in the Service and Controller layers, where the platform’s financial, authorization and workflow logic resides, were tested.

This article explains what we found, why AI-assisted test generation can produce this kind of gap, and what it takes to turn a green CI pipeline into meaningful quality assurance.

Why the Suite looked good

The client’s team had not produced poor work. In fact, the AI-generated output was well structured and delivered real value. Understanding that value is important, because it also clarifies where AI needs expert direction.

1. Route extraction and project generation

AI analyzed the Symfony application’s source archive, extracted the route annotations from the controllers, parsed the templates and security configuration, and generated a complete Playwright project. The project included separate configurations for each sub-application, authentication setup files and specification files organized by module. Work that might have taken a QA engineer two or three days was completed in roughly 15 minutes.

2. Removing repetitive setup work

The suite handled common tasks such as login flows with TOTP two-factor authentication, storageState reuse, environment templates, TypeScript configuration and basic smoke tests. These included checking that pages loaded, forms were present and responses returned a successful status code.

3. Broad route coverage

AI does not tend to skip routes. If 20 controllers expose 180 endpoints, it can create a smoke test for each of them. A developer writing tests manually is more likely to prioritize the most important areas and leave apparently obvious pages untouched.

4. Consistency

The tests followed the same naming conventions, assertion patterns and error-handling approach throughout the project. That consistency made the suite look professional and dependable.

What the Audit revealed

Our standard audit begins by extracting validation messages, domain exceptions and access checks from the source code. We then compare them with the assertions in the test suite. The goal is not to chase a code coverage percentage. It is to determine whether the tests verify the rules that protect the business.

This is an approximate measure. A test may verify a rule without quoting the exact validation message. Even so, the fact that none of the 129 Service and Controller messages appeared in the tested scenarios was clear. These layers contain the rules governing payments, authorization, document generation and core business workflows.

Three Categories of Risk

We grouped the gaps into three broad categories. Each called for a different kind of testing expertise—expertise that, despite the development team’s technical ability, had not been applied to the test strategy.

1. The tests checked the interface, not the business rules

The invoicing module contained 21 tests, but none checked the values that mattered: invoice amounts, VAT calculations, rounding or the remaining balance. The application stores amounts in integer cents and applies custom rounding logic. None of that logic can fail if a test only checks whether an invoice number appears in a list.

Payments exceeding the remaining balance, zero-value payments, credit notes greater than the original line value and duplicate invoice lines were all untested. The suite confirmed that the interface worked. It did not confirm that the business rules held.

2. False negatives that appeared green

Some tests looked like negative tests but did not actually verify that the operation was rejected. For example, negative invoice scenarios checked whether the Cancel button was hidden, rather than sending a direct HTTP request and confirming that the server denied the action. As the application’s own architecture documentation stated, hiding a button is not authorization.

Other examples compared element positions using boundingBox rather than comparing the underlying data. Some absence checks used not.toBeVisible(), which can pass because a dropdown has not been opened—or because the test is looking at the wrong page altogether.

A test that passes when it should fail is more dangerous than an obvious missing test. It creates confidence without providing protection.

3. Entire modules had no meaningful coverage

The timer module had no tests at all, despite containing important behavior such as pause, resume, stop, a ten-timer limit and automatic stopping after 12 hours.

Case archival was also untested, even though it is blocked when unfinished tasks, open events or unpaid invoices remain. Email notifications for sharing, assignment and task completion were not covered. The legally significant audit log had one test for the history of a single case. File upload validation—permitted formats, the 20 MB size limit and MIME type checks—had only one test, which uploaded a safe PNG.

How we addressed the Problems

After we delivered the audit, the client asked us to improve the suite. We followed the same process we use across our testing engagements.

1. Source-code analysis

We extracted every validation message, DomainException and access check from the codebase, then mapped each one against the test assertions. The resulting cross-reference table became the central record of what was covered, what was missing and where the highest risks lay.

2. Risk-based prioritization

Not every gap has the same consequences. Financial calculations—including amounts, VAT, rounding and credit notes—and server-side authorization were treated as critical because they protect revenue and client data.

File upload validation was treated as an attack surface. Email notifications and audit logs carried legal and operational significance. Timer limits and archival restrictions were essential product behavior.

3. AI-assisted refactoring

Once the priorities and target structure had been defined, we used AI for the mechanical work. It helped restructure 40 specification files into seven specifications and four shared modules, extract common utilities, add a pageGuard fixture for JavaScript errors and 5xx responses, and configure a reproducible Faker seed and timezone.

This is where AI performs particularly well: carrying out structural changes after a human has decided what the structure needs to be.

4. Business-rule test design

We designed the tests for the critical paths ourselves, including payment validation, credit-note edge cases, archival blockers and server-side access denial through direct HTTP requests. AI was then used to reproduce those patterns across similar modules.

Every test was reviewed against the source code to confirm that it exercised the actual validation logic rather than an assumption about how the application was supposed to behave.

5. Validation in staging

We ran the complete suite against the staging environment. Failed tests were investigated to determine whether they exposed real defects. Passing tests were reviewed for false positives. The client received a coverage report mapped to business rules, rather than a report limited to routes and page loads.

6. The Lesson

The client’s developers were skilled, and the application itself was solid. But building production software and building tests that protect that software are different disciplines. AI increases the speed of both—and can amplify the blind spots as well as the benefits.

The most dangerous situation is not a project with no tests. It is a project with a green CI pipeline that covers only 6% of its business rules. The team ships with confidence, but the evidence does not justify it.

AI can reduce the time needed to create a test suite from weeks to days. Those days still need to be guided by someone who understands what must be tested, not only how to write a test. That means reading the Service layer, mapping the validation rules and building scenarios around business logic rather than the interface alone.

If your team has built its test suite in-house and wants to know whether it genuinely protects the product, that is the question our test suite audits are designed to answer.

About SQA Service

SQA Service is a software testing consultancy with more than 15 years of experience in QA processes, test automation and AI-assisted quality engineering. We audit test suites, build test automation frameworks and help teams integrate AI into their QA workflows without losing the reliability that comes from professional QA expertise.