AI-Powered QA Features in Evaficy Smart Test
From intelligent regression selection to self-learning risk prediction — a practical guide to how AI features work, what they produce, and how user feedback makes them sharper over time.
For QA Engineers, Tech Leads, and Product Owners who want AI that explains its reasoning and improves with use.
Five AI capabilities built into your QA workflow
Most QA tools bolt AI onto test case generation and stop there. Evaficy Smart Test goes further: AI is built into the decision points that slow teams down the most — choosing what to test before a release, keeping test suites accurate as the application evolves, and understanding which areas carry the most defect risk at any given time.
Every AI feature produces an explainable output — you see not just what the AI decided, but why. And every decision can be corrected with a single click, feeding a feedback loop that improves future results for the same project.
AI Regression Selection
AI Test Maintenance
AI Defect Prediction
Explainable AI
Feedback Loop
AI Regression Selection
Before every release, your team faces the same question: which test scenarios should we run? Running everything takes too long. Running too little risks shipping regressions. AI Regression Selection solves this by analysing your change description and ranking every scenario in the project by its likelihood of being affected.
How it works
You describe what changed in the release — in plain language, as a few sentences or bullet points. The AI reads each scenario's name, description, and affected page or module, then scores every scenario from 0 to 100 for relevance. The highest-scoring scenarios are surfaced at the top with a one-sentence reason explaining the connection to your change.
Clicking a result populates the scenario selector in the Create Test Run dialog — so you go from change description to test run in under a minute.
What the score means
A score of 80+ indicates a direct, high-confidence relationship between the change and the scenario. A score below 30 means the AI found little to no connection. The score is not a guarantee — it is a prioritisation signal based on the text you provide and the scenario metadata in your project.
Where to find it
Open Create Test Run and expand the AI Regression Assist panel at the top of the dialog. Describe what changed and click Analyse.
AI Test Maintenance
Test suites drift. Features change, flows get redesigned, and test cases written six months ago silently stop matching the application they describe. AI Test Maintenance surfaces this drift automatically and gives you a clear action for each test case: keep it, update it, or retire it.
The trigger: automatic staleness detection
Any scenario not reviewed in more than 30 days displays an amber banner when you open it in the Scenario editor. The banner prompts you to run an AI maintenance review — a process that takes under 30 seconds and covers every test case in the scenario at once.
The review: three verdicts per test case
Valid — no action needed
The test case accurately describes a behaviour that still exists and has not changed. The AI found no reason to flag it.
Needs update — revised title suggested
The test case is still relevant but needs editing to reflect the change. The AI suggests a revised title — you can apply it with a single click, replacing the original title in your scenario.
Obsolete — reason provided
The test case describes a flow that no longer exists or has been replaced by the changes. A reason is shown so you can decide whether to delete or repurpose it manually.
Triggered by: change attribution
For every flagged test case, the AI surfaces the specific phrase from your change description that caused the flag — shown as Triggered by: two-factor authentication requirement. This makes it immediately clear why the AI reached its verdict, without having to cross-reference the change description manually.
Where to find it
Open any scenario in the editor. If it has not been reviewed in over 30 days, an amber banner appears at the top. Click Review with AI to open the maintenance modal.
AI Defect Prediction (Risk Insights)
Risk Insights analyses your completed test run history and produces a risk score for each scenario — telling you where defects are most likely to appear in your next release, based on how each scenario has actually performed in the past.
The panel requires a minimum of five completed test runs across your project. Once that threshold is met, the analysis runs on demand and returns results in seconds.
The risk score formula
Each scenario's score is calculated from two inputs: its historical fail rate (how often test cases fail across all runs) and a trend adjustment based on whether recent runs are improving or worsening compared to earlier ones. A scenario with a 40% fail rate that is trending down carries a different score than one with the same fail rate that is getting worse.
AI narrative summary
Alongside the per-scenario scores, the AI generates a 2–3 sentence plain-English summary of your project's current risk profile — highlighting the highest-risk areas and any notable trend patterns. This is the section most useful to share with a Product Owner or Tech Lead before a release sign-off.
Where to find it
Go to the Dashboard and scroll down to the Risk Insights panel. Select the project you want to analyse and click Analyse Risk.
Explainable AI — Every Result Shows Its Reasoning
A core design principle in Evaficy Smart Test is that AI outputs must be transparent. You should never be in the position of accepting or rejecting an AI decision without understanding what drove it. Each AI feature surfaces its reasoning directly in the UI.
Risk score breakdown
Every Risk Insights scenario card has a Why this score? chevron. Expanding it shows a breakdown table: how many points came from the fail rate, what the trend adjustment added or subtracted, how many runs were analysed, and when the last failure occurred. The total always adds up to the displayed score — no black box.
Regression selection reason
Each regression selection result shows a highlighted reason callout — colour-coded to match the risk level — that explains in one sentence why the AI connected that scenario to your change description. A team reviewing results can immediately distinguish a strong AI match from a speculative one.
Maintenance trigger attribution
When the AI flags a test case as needing an update or as obsolete, it also returns the specific phrase from your change description that triggered the flag. Seeing Triggered by: password reset flow redesign below a test case makes the AI's reasoning immediately verifiable — and gives you confidence to apply or dismiss the suggestion.
Explainability is not optional
Explainable outputs reduce the mental overhead of acting on AI suggestions. Teams that understand why a score is high are more likely to act on it — and more likely to correct it when the AI is wrong, which feeds the improvement loop described below.
Feedback Loop — AI That Learns From Your Team
Every AI output in Evaficy Smart Test can receive feedback. When a score is wrong, when a scenario is not affected by a change, or when a maintenance verdict is incorrect, team members can record that directly in the UI. That feedback is injected as context into future AI calls for the same project — so the AI makes better decisions the next time.
This is not fine-tuning and does not modify the underlying model. It is prompt augmentation: a concise record of prior corrections is prepended to future requests, steering the AI toward the preferences and patterns specific to your project.
Feedback actions by feature
Risk Insights — Score is wrong
If the risk score for a scenario does not reflect its actual criticality, click the thumbs-down icon on the card and select Score is wrong. Future narrative summaries will note the discrepancy and adjust their framing.
Risk Insights — Always treat as critical
Some scenarios are business-critical regardless of their historical pass rate — for example, the core payment flow. Click the star icon on a card to mark it as always critical. The card is highlighted amber, and future AI analyses will treat that scenario as high-priority regardless of the score.
Regression Selection — Not affected
If the AI ranks a scenario as highly relevant but your team knows it is not affected by this type of change, click the thumbs-down on the result. Future regression analyses that include similar changes will factor in this correction.
Test Maintenance — AI was wrong
If the AI flags a test case as obsolete or needing an update but the verdict is incorrect, click the thumbs-down on that result. Feedback is recorded against the specific test case and feature, and improves the precision of future reviews.
How feedback flows into future AI calls
Before each regression selection or risk insights analysis, Evaficy fetches all feedback recorded for your project and builds a brief context block — something like: "User feedback for this project: 'Checkout Flow' is always considered critical. 'User Profile' was previously marked as not affected by login-related changes." This block is prepended to the AI prompt so the model can apply prior corrections without overriding its core judgement on unrelated scenarios.
Feedback accumulates over time. The more your team records, the more precisely the AI aligns with your project's real risk profile.
No overhead — feedback takes one click
There is no form to fill in, no ticket to open. Every feedback action is a single click. The optional note field allows teams to add context, but the bare click alone is enough to influence future outputs.
Plan availability
| Feature | Advanced | Enterprise |
|---|---|---|
| AI Test Maintenance — review, trigger attribution, feedback | ✓ | ✓ |
| AI Regression Selection — ranking, reason callout, feedback | — | ✓ |
| AI Defect Prediction / Risk Insights — score, breakdown, narrative | — | ✓ |
| Always Critical star toggle + feedback popover (Risk Insights) | — | ✓ |
| Aggregate feedback count on Risk Insights cards | — | ✓ |
AI Test Maintenance is available on Advanced because it lives in the scenario editor, which Advanced users can access. AI Regression Selection and Risk Insights require Test Runs — an Enterprise-only feature — so they are only available on Enterprise. None of these features consume your monthly AI generation quota.
Common questions
Does the AI learn globally across all projects and customers?
No. Feedback is scoped to the project it was recorded in. The AI never uses feedback from one team's project to influence another team's results. Each project's feedback context is private and isolated.
What model powers the AI features?
All AI features use GPT-4.1 with structured JSON output, temperature 0.3 for analytical tasks. The model is called directly via the OpenAI API on each request — there is no cached or pre-computed output.
How many test runs does Risk Insights need to be useful?
A minimum of five completed test runs are required before the panel will produce results. The more runs in your history, the more accurate the trend detection — particularly the improving/worsening classification, which compares the last three runs against earlier ones.
Can I use AI Regression Selection without having all scenarios filled out?
Yes, but quality scales with the information available. Scenarios with a name only will receive a lower-confidence score than those with a description and an affected page or module. Investing time in scenario metadata improves both regression selection and maintenance review accuracy.
Does the AI consume my monthly test case generation quota?
No. Regression selection, maintenance review, and risk insights are separate AI calls that do not count toward your monthly test case generation allowance.
Free Resource
This topic is covered in the QA Leader’s Handbook
A free 10-chapter PDF guide for Tech Leads and Product Owners.
See the AI features in action
Regression selection, risk prediction, and a feedback loop that improve with every use — on Enterprise. AI Test Maintenance included on Advanced and above.
Start your trial