What is LLM testing? How AI testing works

What is LLM testing?

LLM testing verifies both the deterministic behavior of an AI application and the quality of the model output. The two require different testing methods because application logic has fixed expected outcomes, while model responses can vary across runs and still be acceptable.

Application testing covers prompt templates, parsers, schemas, API contracts, retrieval steps, tool wrappers, and orchestration. Each component has an expected behavior that can be checked directly with fixed assertions.

Behavioral evaluation measures what the model produces across criteria such as correctness, relevance, tone, and safety. The LLM evaluation guide covers the scoring process in more depth, including datasets, scorers, and evaluation results.

Both types of testing are necessary because one cannot compensate for failures in the other. An application can return a confidently wrong answer inside perfectly valid JSON, or produce high-quality answers until an empty retrieval result causes the application itself to fail.

Why LLM applications break traditional testing

Non-deterministic outputs: The same model call can produce different text across repeated runs, especially when sampling is enabled. Variation can still appear at temperature: 0 because provider infrastructure and model behavior are not perfectly fixed. Exact-string assertions therefore become unreliable for outputs where several responses could still be correct.

No single ground truth: Some tasks have one correct answer, such as classification against a fixed label set, but many LLM tasks do not. Summaries, explanations, and open-ended responses may have several acceptable outputs, so testing needs scoring criteria that measure quality on a scale and a threshold that defines acceptable performance.

Model and provider drift: Application behavior can change even when the surrounding code stays untouched. Provider model updates, changes to default parameters, or a move to another model family can alter response quality, formatting, refusal behavior, or tool use. Regression coverage therefore needs to track model and configuration changes alongside application changes.

Score thresholds: LLM testing often moves the pass-or-fail decision from one exact assertion to a score measured across a representative dataset and, where needed, repeated trials. A check such as output == expected might become a requirement that factuality averages at least 0.85 across 200 cases, with no individual case regressing by more than 0.1 from the baseline.

The LLM application testing stack

The seven testing layers differ mainly in what you verify and how you judge results.

The layers near the top are cheap and fast, so they run often. The evaluation-heavy layers cost model tokens and time, which is why a release process usually samples them during review and runs them completely before shipping.

Unit testing for LLM applications

Unit tests cover the deterministic code around the model and should run without making live model calls. They belong on every commit because failures in templates, parsers, schemas, and tool wrappers can be checked quickly with fixed assertions before model behavior enters the test.

Prompt template tests: Render each template with representative variables and verify that every placeholder resolves correctly. Tests should also cover empty values, user-supplied content that needs escaping or delimiters, and the token budget for the target model.

Parser and schema tests: Feed recorded model outputs into the parser and check how it handles both valid and malformed responses. A structured-output pipeline should cover cases such as valid JSON, JSON wrapped in code fences, truncated JSON, and unexpected fields, with schema validation tested separately against the expected structure.

Tool wrapper tests: Treat every callable tool as an API surface. Verify required arguments, type coercion, upstream error handling, and the structure returned to the model, so wrapper failures do not surface later as harder-to-diagnose agent errors.

Mock model responses: Replace live model calls with fixtures when the goal is to isolate surrounding application logic. Braintrust's pytest integration runs marked tests normally during local development and turns them into tracked experiments once tracking is enabled.

import pytest


@pytest.mark.braintrust(
    project="support-bot",
    input={"query": "What is Braintrust?"},
    expected={"contains": "evaluation"},
    metadata={"suite": "smoke"},
    tags=["regression"],
)
def test_support_answer(braintrust_span):
    output = ask_model("What is Braintrust?")
    braintrust_span.log(output=output)
    assert "evaluation" in output.lower()

Without the --braintrust flag, the braintrust_span fixture becomes a no-op, so the same test file still behaves like an ordinary pytest suite.

Integration testing for retrieval and tool calls

Integration tests focus on the boundaries where application components depend on one another. Retrieval, provider contracts, tool execution, and orchestration can all fail in ways that surface as poor model behavior, so isolating those paths separates application defects from output-quality problems.

Retrieval pipeline tests

Test retrieval before the returned documents enter the model context. Checks should confirm that the index contains the expected document count after a rebuild, known queries return the expected sources within the top-k results, embedding model versions match between indexing and query time, and chunk boundaries follow the configured rules.

A missing or stale document looks like a hallucination once the model writes the final response. Checking retrieval on its own catches the empty result at the index, before the debugging trail leads to the prompt or the model.

Tool and API contract tests

Function-calling schemas form a contract between the application and the model provider. Tests should verify that the declared schema matches the tool wrapper's actual signature, required arguments are enforced, and returned tool calls match the structure expected by the parser.

Provider contract checks are also worth running on a schedule in addition to commit-time testing. Validation behavior can change upstream even when the application code and client dependency remain unchanged, so periodic contract verification catches provider-side changes before they reach production workflows.

Orchestration and fallback tests

Multi-step LLM applications depend on state moving correctly between components. Integration tests should check that routers send inputs to the intended branch, state survives each handoff, configured fallbacks run when a step fails, and retry logic stops at the defined limit.

Fallback paths deserve explicit coverage because manual testing rarely reaches them under normal conditions. Deliberately exercising failures makes retry loops, state loss, and incorrect routing visible before they appear in a full agent run.

End-to-end testing for LLM workflows

End-to-end tests exercise the complete path from request to response under realistic conditions, including multi-turn conversations and agent trajectories. They verify that the workflow finishes and that the sequence of model calls, tool use, state changes, and final output matches the application's expected behavior.

Structural assertions

Many end-to-end checks remain deterministic even when a model is involved. Tests can verify that the workflow completes without an unhandled exception, the final response matches the required schema, the run stays within its step budget, and every tool call returns successfully.

Trajectory checks

For agentic systems, the path to the final answer is part of the behavior being tested. A run may produce the right result after skipping a required tool, calling the wrong tool first, or passing incorrect arguments along the way. Trajectory checks inspect the execution trace to confirm that required tools were called and that the sequence of steps followed the intended path.

Braintrust tracing exposes the intermediate spans to scorers, so a scorer can inspect tool calls and other steps inside the run instead of judging only the final response.

# Check if a specific tool was called at least once.
async def required_tool_called(input, output, expected, trace=None):
    if not trace:
        return None

    tool_spans = await trace.get_spans(span_type=["tool"])
    edit_view_calls = [span for span in tool_spans if (span.span_attributes or {}).get("name") == "edit_view"]
    return {
        "name": "edit_view called",
        "score": 1 if edit_view_calls else 0,
        "metadata": {"edit_view_calls": len(edit_view_calls)},
    }

Behavioral output testing and scoring

Test datasets

A dataset holds the inputs used for evaluation, expected outputs where a reference answer exists, and metadata for slicing results by category, difficulty, or customer segment. Datasets play a role similar to fixtures in conventional testing, except they are versioned artifacts that expand as production traffic exposes new edge cases and failures.

from autoevals import Levenshtein
from braintrust import Eval, init_dataset

Eval(
    "Say Hi Bot",
    data=init_dataset(project="My App", name="My Dataset"),
    task=lambda input: "Hi " + input,
    scores=[Levenshtein],
)

Rubrics and criteria

A rubric line such as "helpful" leaves too much room for interpretation, while "answers the question asked, cites at least one source from the retrieved context, and declines when the context does not contain an answer" splits the same judgment into three checks that each carry a score.

Scorer types

Code-based scorers handle requirements with computable answers, including exact match, numeric tolerance, format validation, keyword coverage, and business rules. LLM-as-a-judge scorers are better suited to natural-language criteria such as tone or faithfulness to a source. Human review covers judgments automated scorers cannot resolve reliably and produces labeled examples for checking whether model-based judges stay aligned with human judgment.

Repeated trials and pass rates

Running an input once captures only one possible result from a non-deterministic system. Repeating the same input reveals score variance and separates normal model variation from a meaningful quality change, especially for cases near a pass threshold or decision boundary.

Eval(
    "My Project",
    data=my_dataset,
    task=my_task,
    scores=[Factuality],
    trial_count=10,  # Run each input 10 times
)

Also read:

• How to test AI models and prompts
• Testing models with prompts

Regression testing for prompts and models

Regression testing measures a candidate prompt, model, or retrieval configuration against a previously validated version so release decisions are based on movement from an established baseline.

Baselines and golden datasets

Use a validated experiment as the baseline and keep the underlying evaluation dataset stable during comparison. Dataset snapshots prevent test-case edits from changing what the scores represent, so you can attribute any movement to the prompt, model, or pipeline configuration under review.

Version comparison

A comparison grades each candidate against the base experiment and separates score movement from latency, error, and token changes.

Run the baseline and candidate configurations against the same cases, then inspect aggregate movement alongside individual regressions. A higher average can still hide failures on important inputs. Braintrust experiment comparison aligns matching cases across runs and shows score changes at the row level, making regressions easier to isolate before release.

Score delta thresholds

Release criteria should define acceptable movement before the comparison runs. A practical setup combines an aggregate quality floor with a maximum per-case regression, which prevents improvements on some cases from masking a serious failure on a critical input.

Gating a release

Once the thresholds are defined, regression results can become release requirements in CI. One set of thresholds can cover prompt updates, model changes, retrieval modifications, and adversarial cases held as a permanent security regression set.

Performance testing for latency, load, and cost

Quality scores can stay stable while response time, throughput, and per-request cost deteriorate under realistic traffic, so latency, load, and cost need release thresholds of their own.

Latency percentiles

Measure both time to first token and total completion time at p50, p95, and p99. Average latency can hide slow requests at the tail of the distribution, which users often notice most. Streaming applications should track time to first token separately because perceived responsiveness depends heavily on when generation begins.

Load and concurrency

Run the application at expected peak concurrency and above it to expose limits that single-request tests cannot reveal. Provider rate limits, connection pools, queue depth, and retry behavior all become easier to observe under load. Tests should also cover 429 responses and confirm that retries back off correctly without exceeding the configured retry budget.

Reliability and fallbacks

Exercise degraded paths deliberately. Disable the primary provider and verify that the fallback model receives the request, returns a valid response, and records the provider switch. Retrieval timeouts deserve similar coverage so the application either completes with the available context or declines cleanly according to the intended behavior.

Cost budgets

Token spend should be treated as a release criterion. Set limits for cost per request and cost per completed task, then measure prompt and model changes against those limits. Adding 2,000 tokens of few-shot examples to every request is still a regression if the resulting quality improvement does not justify the higher production cost.

Security and adversarial testing

Adversarial inputs run through the same datasets, scorers, and thresholds as any other evaluation, with cases written to expose prompt injection, jailbreaks, data leakage, and unauthorized tool use.

Prompt injection tests: Include inputs that carry malicious instructions through user messages, retrieved documents, tool outputs, or uploaded files. The test should verify that system instructions retain priority and that injected instructions do not alter the intended application behavior. Braintrust's prompt injection detector cookbook shows how prompt-injection attempts can be evaluated as scored test cases.

Jailbreak and refusal tests: Maintain a dataset of requests the application is expected to decline and score whether the model refuses appropriately. Legitimate requests also need coverage because excessive refusal is a separate quality failure that can surface after safety-related prompt or model changes.

Data leakage tests: Check for exposure of system prompt content, credentials, data belonging to other users, or documents outside the requester's permission scope. Multi-tenant retrieval deserves dedicated cases because an access-control failure can place protected information inside the model context.

Tool misuse tests: Verify that adversarial inputs cannot trigger destructive or unauthorized actions. A denylist scorer can inspect the execution trace and fail the case when the agent calls a restricted tool.

# Check that no tool from a denylist was called.
async def no_disallowed_tools(input, output, expected, trace=None):
    if not trace:
        return None

    disallowed_tool_names = {"send_email", "delete_record"}
    tool_spans = await trace.get_spans(span_type=["tool"])
    disallowed_calls = [
        span for span in tool_spans if (span.span_attributes or {}).get("name") in disallowed_tool_names
    ]
    return {
        "name": "no disallowed tools",
        "score": 1 if not disallowed_calls else 0,
        "metadata": {
            "disallowed_tools": [span_name(span) for span in disallowed_calls],
        },
    }

The span_name helper used by this scorer is defined alongside the other trace scorer recipes in the Braintrust docs.

Red-team suites as regression suites: Any adversarial input that succeeds once should remain in the security dataset after the underlying issue is fixed. Running those cases on future releases turns confirmed vulnerabilities into permanent regression coverage and makes security failures subject to the same release discipline as quality regressions.

Also read: Best LLM guardrails and security testing tools

LLM testing in CI/CD

Pull request checks

Pull requests need fast feedback without dropping the model-behavior checks that catch obvious regressions. Run unit and integration tests on every change, then add a representative sample of the behavioral dataset. A smoke run over 20 or 50 cases keeps evaluation time and model usage manageable during active review.

Pre-release gates

Before release, run the complete behavioral dataset together with regression comparison, performance tests, and the security suite. Encode quality floors, regression tolerances, latency or cost limits, and security requirements as release criteria, so a failing evaluation prevents the change from progressing.

Braintrust can run these checks through CI/CD and post the results in the pull request.

name: Run evaluations

on:
  pull_request:
    branches: [main]

permissions:
  pull-requests: write
  contents: read

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 24
      - name: Install dependencies
        run: npm install
      - name: Run evals
        uses: braintrustdata/eval-action@v2
        with:
          api_key: ${{ secrets.BRAINTRUST_API_KEY }}
          runtime: node

The evaluation result appears alongside the pull request review, putting score movement and release criteria in front of the reviewer before merge.

Staged rollout and production scoring

Online scoring records each scorer run as a span inside the production trace, so a live request carries the same quality criteria used offline.

After the pre-release gates pass, expose the new version to a limited share of production traffic and continue measuring quality before expanding the rollout. Braintrust online scoring runs scorers asynchronously against live traces, so the quality criteria from the offline evaluation also grade production requests. Production scoring also surfaces inputs missing from the offline dataset, giving future evaluations coverage of failures that only appear in production traffic.

Converting production failures into regression tests

Capture the trace

Start with the production request that exposed the failure and inspect the complete execution record, including the input, retrieved context, intermediate spans, tool calls, model output, timing, and cost. The trace preserves the evidence needed to reconstruct the failure and identify which part of the application behaved incorrectly.

Triage and classify

Use the trace to locate the failure before adding a regression case. A wrong answer with correct retrieval points toward prompt or model behavior, empty retrieval points toward the retrieval pipeline, and a crash after a tool call points toward the wrapper or orchestration layer. The diagnosis determines which test set and scoring criteria need to change.

Add the failing case to the dataset

Once the failure is confirmed, promote the production trace into a dataset with the corrected expected behavior attached. The quickest path is the UI: select the trace in Logs or Review, then choose Add to > Add to dataset. To script it, fetch the failing span by ID and insert it with an origin field, which links the dataset row back to the trace it came from.

import os

import braintrust
import httpx

project_id = "<your-project-id>"
span_id = "<span-id-from-the-failing-trace>"

# Fetch the failing span from project logs
btql_response = httpx.post(
    "https://api.braintrust.dev/btql",
    headers={
        "Authorization": f"Bearer {os.environ['BRAINTRUST_API_KEY']}",
        "Content-Type": "application/json",
    },
    json={
        "query": f"SELECT id, input, output FROM project_logs('{project_id}') WHERE span_id = '{span_id}' LIMIT 1",
    },
)
span = btql_response.json()["data"][0]

dataset = braintrust.init_dataset(project="My App", name="Production regressions")

httpx.post(
    f"https://api.braintrust.dev/v1/dataset/{dataset.id}/insert",
    headers={
        "Authorization": f"Bearer {os.environ['BRAINTRUST_API_KEY']}",
        "Content-Type": "application/json",
    },
    json={
        "events": [
            {
                # The exact request that failed in production
                "input": span["input"],
                # The behavior that should have occurred, not the output that did
                "expected": {"answer": "Full refunds within 30 days of purchase."},
                "metadata": {"failure_mode": "stale_policy", "source": "production"},
                # Links the row back to the originating trace
                "origin": {
                    "object_type": "project_logs",
                    "object_id": project_id,
                    "id": span["id"],
                },
            },
        ],
    },
)

The expected value is the one field that does not come from the span. When you are capturing a response that was correct, span["output"] can map straight to expected; for a failure, the logged output is the behavior you are trying to prevent, so supply the corrected answer instead.

Write the scorer

If the existing evaluation criteria would not have detected the production failure, the new dataset case also needs a scorer that captures the failed behavior. Depending on the failure, Braintrust scorers can use deterministic code or model-based judgment to turn the requirement into a measurable result.

Re-run and verify

Run the new case against the affected version first to confirm that the regression test reproduces the failure. After applying the fix, rerun the evaluation and confirm that the case passes without degrading the existing regression set. The production failure then remains part of future release coverage, preventing the same behavior from returning unnoticed.

Mapping software testing practices to LLM applications

Many conventional software testing concepts carry directly into LLM applications, with evaluation methods extending the parts that depend on model behavior.

Run your LLM testing stack in Braintrust

Experiments keep every evaluation run comparable, so score movement across configurations stays visible in one place.

LLM testing becomes operational when evaluation results remain comparable across changes and influence release decisions. The Braintrust evaluation workflow keeps datasets, scorers, experiment results, and release criteria in one place across development and production.

Braintrust also reduces the engineering work required to maintain evaluations as the application evolves. Product, engineering, and domain reviewers use Loop to investigate logs in natural language, identify recurring failure patterns, and convert findings into evaluation cases or scorers without writing the full workflow in code.

At Notion, 70 engineers use Braintrust for model evaluation and regression testing, including the validation required to deploy new frontier models in under 24 hours. Shared datasets, evaluation results, and release criteria give engineers a consistent basis for deciding when a model or application change is ready for production.

Start free with Braintrust and build your LLM testing workflow →

FAQs about LLM testing and AI testing (2026)

What is the difference between LLM testing and LLM evaluation?

LLM evaluation measures model behavior quality using representative inputs and defined scoring criteria. LLM testing covers the broader verification process for the complete application, including deterministic application logic, model behavior, integrations, regressions, performance, and security. Evaluation provides the quality signals an LLM test suite uses alongside conventional software checks to decide whether a change is ready to ship.

Can you unit test an LLM application?

Yes, for everything except the model call itself. Prompt templates, output parsers, schema validators, tool wrappers, chunking functions, and routing logic are ordinary code with deterministic behavior, and mocking the model response isolates it all. Braintrust's pytest and Vitest integrations route those same tests into a tracked experiment when a real model call is involved.

How many trials are enough for LLM testing?

The required trial count depends on how consistently a case scores. Start with a small number of repeated runs across representative cases and inspect the variance. Stable cases may need only one or a few runs, whereas cases near a release threshold need additional trials to determine whether a score change reflects a genuine regression or normal model variation. Increasing the trial count selectively also keeps evaluation time and model usage under control.

Do automated scorers replace human review?

Automated scorers and human review sit at different points in the loop. Human reviewers label the first few hundred cases and define what a good answer looks like; those labels become the reference an automated judge is measured against before it takes over repeated checks. Review does not stop after the handoff. Re-label a sample whenever the rubric changes, the judge model is upgraded, or the score distribution shifts with no matching change in the application.

How often should an LLM test suite run?

Testing cadence should match test cost and change risk. Fast deterministic checks belong on frequent code changes, a representative behavioral sample is appropriate during pull request review, and broader regression, performance, security, and behavioral coverage should run before a production release. Production scoring adds ongoing coverage after deployment, especially for inputs and failure patterns that were absent from the pre-release dataset.