Run audit — EvalJev

Runs, newest first

Point this at your own workflow. Three steps, about five minutes. Nothing leaves your machine and there is no service to sign up for.

Wrap the client you already have. Your call sites do not change; every decision is appended to a file as it happens.
Ship as usual. Each configuration you run on becomes a run this page can judge — you never tell it that you shipped.
Audit. evaljev console traces.jsonl.

pip install "git+https://github.com/xxlya/evaljev"

from evaljev import watch
client = watch(client, workflow_id="my-app")

with client.request(request_id):
    with client.step("classify"):
        response, _ = client.decide(state=state,
                                    questions=questions)
client.record_outcome(what_happened, correct=True)

$ evaljev console traces.jsonl --open