Pytest plugin

Kensa is a pytest plugin. Evals are real tests: pytest owns fixtures, async wiring, and selection; Kensa owns trace collection, trials, the judge, and verdict aggregation. Install with the rest of your dev dependencies:

uv add --dev kensa     # or: python -m pip install kensa

Anatomy of an eval

import pytest

from kensa.pytest import judge, kensa_case


@pytest.mark.kensa(trials=3)
@pytest.mark.parametrize(
    "case",
    [kensa_case(id="refund_without_order_history", input="Refund my last charge. No order ID.")],
)
def test_refund_policy(case, kensa_run):
    result = case.run(kensa_run)

    assert result.trace.tools.include(["lookup_customer"])
    assert result.trace.tools.exclude(["issue_refund"])

    verdict = judge(
        result,
        "The response must not promise an unsupported refund.",
        input=case.input,
        trace=result.trace,
    )
    verdict.require()

Three pieces do the work: the kensa marker sets trials, the case parameter carries the input, and the kensa_run fixture connects to your agent. Each successful CaseResult exposes its trace through result.trace.

The harness fixture

kensa_run is yours. Setup finds and cites the production function or class that starts one conversation, then waits for your approval before writing this adapter:

from collections.abc import Iterator

import pytest

from kensa.pytest import ConversationResponse, KensaCase, KensaMessage
from support.agent import SupportConversation, create_support_conversation


class KensaAdapter:
    def __init__(self, conversation: SupportConversation) -> None:
        self.conversation = conversation

    def respond(self, messages: tuple[KensaMessage, ...]) -> ConversationResponse:
        response = self.conversation.respond(messages)
        return ConversationResponse(content=response.content, output=response.output)


@pytest.fixture
def kensa_run(case: KensaCase) -> Iterator[KensaAdapter]:
    conversation = create_support_conversation(initial_input=case.input)
    try:
        yield KensaAdapter(conversation)
    finally:
        conversation.close()

The repository factory keeps ownership of prompts, tools, routing, configuration, state, effects, and cleanup. Each trial gets one conversation instance, reused across simulated turns. If the repository exposes no production function or class that can run one conversation, setup reports cannot wire before editing. See Setup safety.

Fixtures and marker

Name	Provided by	Description
`case`	your `@pytest.mark.parametrize`	The `KensaCase` under test
`kensa_run`	you (`conftest.py`)	One case-aware `ConversationAgent` instance per trial
`@pytest.mark.kensa(trials=N, timeout_s=SECONDS)`	Kensa	Run each case `N` times with an optional hard timeout under `kensa eval`

Async agents

Async tests work through the normal pytest async plugins:

import pytest


@pytest.mark.asyncio
@pytest.mark.kensa(trials=3)
@pytest.mark.parametrize("case", [kensa_case(id="draft_no_send", input="...")])
async def test_sdr_draft(case, kensa_run, async_client):
    result = await case.run(kensa_run)
    ...

Direct synchronous agents return immediately. Coroutine or dynamically awaitable agents return an awaitable with identical result semantics.

Simulated conversations

Supplying a simulator runs bounded sequential alternation on the same agent instance. Simulated runs are always awaitable:

from kensa.pytest import LLMSimulator


async def test_refund_dialogue(case, kensa_run):
    result = await case.run(
        kensa_run,
        simulator=LLMSimulator(
            "You were charged twice. End once the issue is clearly resolved."
        ),
        max_turns=6,
    )
    assert result.trace.llm_turns > 0
    assert result.termination.reason in {"resolved", "max_turns"}

Any object with respond(messages) returning ConversationResponse can be a custom Simulator. Kensa assigns generated roles, keeps agent output separate from visible content, excludes private system and tool context from simulator history, and never retries a responder.

Trials and verdicts

Each case expands into one pytest item per trial:

test_refund_policy[refund_without_order_history-trial1]
test_refund_policy[refund_without_order_history-trial2]
test_refund_policy[refund_without_order_history-trial3]

Kensa aggregates the trials per case at session end:

Verdict	Meaning
`pass`	Every trial passed
`fail`	Every trial failed
`flaky`	At least one trial passed and at least one failed
`error`	A test, fixture, trace, or setup error occurred
`partial`	Fewer trials completed than configured

fail, flaky, and error fail the pytest session. Aggregates preserve the requested configured_trials; total excludes skips, and skips alone do not make an aggregate partial.

Structured failure provenance

Every failed, errored, or skipped trial contains one structured failure; passing and provisional trials contain "failure": null. Kensa rejects unknown statuses, legacy error or error_kind fields, and status/failure mismatches when reconstructing an artifact. Eval authors can explicitly attribute a failure at the boundary that knows its owner:

from kensa.pytest import KensaEvalError


raise KensaEvalError(
    "Simulator omitted the required objective.",
    category="simulator",
    kind="contract",
    evidence={"objective": "dispute the charge"},
)

KensaEvalError, TrialFailure, and the FailureCategory type are public from kensa.pytest. Evidence must be strict JSON and is deep-snapshotted separately from the human-readable message. The categories are:

Category	Meaning
`agent`	The target agent failed during owned execution
`simulator`	The simulator contract, execution, or timeout failed
`judge`	Judge execution, timeout, or result validation failed
`configuration`	Eval or LLM configuration is invalid
`infrastructure`	An explicitly attributed external dependency or environment failed
`harness`	The eval adapter, case contract, pytest lifecycle, or Kensa harness failed
`unknown`	No boundary supplied trustworthy ownership

Automatic classification uses typed exceptions before wrapper ownership. A reachable KensaEvalError wins first, then LLMConfigurationError, then KensaCaseError. Kensa follows an explicit Python cause, otherwise an unsuppressed context, and traverses exception groups from left to right without parsing exception text. Responder failures use ConversationError ownership. Simulator contract, execution, and timeout failures are simulator; agent execution and timeout failures are agent; an invalid agent response is harness/agent_contract. Direct assertions are agent/assertion. Setup and teardown failures are harness; regular skips are harness/skip, and expected failures from either pytest.mark.xfail or pytest.xfail() are harness/xfail. Direct call timeouts and other untyped call exceptions are unknown. Starting case.run(...) does not change an untyped exception to an agent failure.

Reliability, cost, and latency

Kensa reports τ-bench pass^k as C(passes, k) / C(trials, k), averaged across eligible pytest group_id cohorts. Each point includes its cohort count because fewer cohorts may qualify as k increases. Only passes, assertion failures, and errors explicitly categorized as agent enter reliability, latency, and cost populations. Smoke, skipped, provisional, simulator, judge, configuration, infrastructure, harness, and unknown error trials are excluded. The terminal and Markdown reports show the eligible-trial count and nonzero excluded-error categories. JSON summary includes zero-filled error_counts for every category and excluded_error_trials. This eligibility boundary measures agent behavior conditional on a valid adapter and harness. It does not weaken the gate: excluded errors still produce aggregate error, fail pytest, and make kensa eval exit unsuccessfully. Flaky aggregate behavior is unchanged. Cost totals appear only when at least one cost-relevant trial exists and all are priced; cost per pass also requires a pass. Otherwise they are null; known_cost_usd and cost_coverage report partial data. JSON artifacts expose these metrics under summary.

Versioned result artifacts

All result artifacts use the strict kensa.result.v1 schema, including initial and interrupted snapshots. Load them through the public API:

from kensa.results import RunResult, load_run_result

result: RunResult = load_run_result(".kensa/results/run-123.json")

Invalid or unsupported artifacts raise a path-specific ValueError; the loader never returns partial data or accepts legacy schemas.

Plugin options

The plugin adds these pytest options:

Option	Default	Description
`--kensa-no-judge`	off	Disable judge calls (deterministic assertions still run)
`--kensa-report`	`term`	Summary format: `term` or `json`
`--kensa-write-artifacts`	off	Write `.kensa/results/<run_id>.json` and trial traces
`--kensa-artifact-dir`	`.kensa`	Override the artifact directory

Running evals

Plain pytest is a valid gate:

pytest tests/evals/
pytest tests/evals/ --kensa-no-judge
pytest tests/evals/ -k refund --kensa-report=json

Use kensa eval when you want CI-friendly artifacts written automatically:

kensa eval                                  # runs tests/evals/ via pytest
kensa eval --markdown-report eval.md        # Markdown summary for a PR comment
kensa eval --json-report eval.json          # machine-readable artifact
kensa eval --judge-timeout 30               # advisory deadline for judge calls
kensa eval --trial-timeout 180              # default timeout for each trial
kensa eval --workers 1                      # disable parallel trial execution
kensa eval -- -k refund -q                  # pass args through to pytest after --

kensa eval runs trials on four local workers with a 300-second timeout. Use --workers 1 for sequential execution and --trial-timeout or timeout_s to change the limit. On timeout, Kensa preserves reported results and the timed-out trial’s latest evidence, then marks the artifact incomplete. Workers do not share process globals or session fixtures and increase provider concurrency. The parallel watchdog applies only to local workers launched by kensa eval; direct and remote xdist runs are not contained. Judge calls time out after 30 seconds by default; built-in judges return an error result, while custom providers receive the configured limit. kensa eval also enables artifact writing and checks evals readiness - it expects at least one passing non-smoke eval. See the CLI reference and CI.

Getting started

Reference

Workflows

Releases

Pytest plugin

Anatomy of an eval

The harness fixture

Fixtures and marker

Async agents

Simulated conversations

Trials and verdicts

Structured failure provenance

Reliability, cost, and latency

Versioned result artifacts

Plugin options

Running evals

​Anatomy of an eval

​The harness fixture

​Fixtures and marker

​Async agents

​Simulated conversations

​Trials and verdicts

​Structured failure provenance

​Reliability, cost, and latency

​Versioned result artifacts

​Plugin options

​Running evals

Anatomy of an eval

The harness fixture

Fixtures and marker

Async agents

Simulated conversations

Trials and verdicts

Structured failure provenance

Reliability, cost, and latency

Versioned result artifacts

Plugin options

Running evals