Why I built it
Teams ship an AI feature, and then it becomes a black box. When a customer gets a wrong answer, nobody can see which step went wrong. A small prompt change can quietly make answers worse, and the bill grows with no clear reason. I built this to learn how tracing and evaluation actually work, and how to measure whether an AI feature is getting better or worse.
What I built
Trace every request
See each step an AI feature took, from retrieval to tool calls to the final answer, with timings and token counts.
Score answers automatically
Every answer is checked for relevance, faithfulness to sources and tone, using test sets and an LLM judge.
Catch regressions early
Each prompt or model change is tested against the same questions, with an alert if quality drops.
Spot hallucinations
Answers that aren’t supported by the retrieved sources are flagged for review.
Control cost and speed
It shows cost and latency per feature, model and step, to show where to optimise.
Works with common stacks
Built on open tracing standards, with LangSmith or Langfuse underneath, so it isn’t tied to one tool.
How it works
- InstrumentA few lines of code send a trace for every request the AI feature handles.
- EvaluateA test set of real questions is scored every day and on every prompt change.
- AlertIf quality, cost or speed crosses a threshold, an alert goes out straight away.
- ImproveFix the prompt, the retrieval or the model, and see the effect in the next run.
Built with
- Python
- OpenTelemetry
- Langfuse
- LangSmith
- LLM-as-judge evals
- PostgreSQL