Know whether your LLM feature actually works
You change a prompt, swap a model or adjust retrieval. The output looks better. Is it? Without a test set and a measure you trust, you are guessing.
The problem
Most teams check LLM output by reading a handful of examples. That misses rare failures, hides regressions, and makes every release a judgment call.
Teams that try automatic scoring often use an LLM as a judge without checking whether it agrees with people. A judge that was never checked can be confidently wrong.
Who this is for
- Teams with a chatbot, summarizer, extractor or other LLM feature in production or close to it.
- Teams comparing models or prompts and needing a fair way to choose.
- Teams that were asked “how good is it?” and could not answer with numbers.
What I do
- Understand the task. What the system is for, who uses it, and what a bad answer costs you.
- Build a test set. From your real inputs or logs, plus hard cases that tend to break things.
- Define what “good” means. Clear criteria and a rating guide, so different reviewers score the same way.
- Set up automatic scoring. Including an LLM judge where it fits, checked against human ratings so you know how far to trust it.
- Report and hand over. Results, known weak spots, and a simple process to rerun the tests on every change.
What you get
- A test set you own, with documented criteria.
- A scoring setup and a measured agreement rate between the automatic scores and human ratings.
- A written report: what works, what fails, and what to fix first.
- Optionally, the evaluation code as a Python service or pipeline your team can run.
Illustrative example
Invented to show how the work goes. Not a real client. More on the examples page.
A startup has a support chatbot and changes prompts every week. I build a test set from past tickets, define what a correct answer looks like, and set up a scorer checked against human ratings. From then on, each prompt change comes with a before-and-after comparison, and the team stops arguing about whether it helped.
Why me
I spent years designing evaluation and benchmarking frameworks for machine translation engines and LLMs, covering more than 35 languages. My PhD is in measuring the quality of machine output. I know where automatic metrics break down and how to check them.
What this does not cover
I do not build your product or promise a score. I tell you what the numbers say and how much to trust them.
Common questions
Do we need labeled data first? No. We start from real inputs and label a sample together.
Can you work with confidential data? Yes, under an NDA, and I can work inside your environment if needed.
How long does it take? It depends on the task and the data. I will give an estimate after a first call.
Next step
Describe your system and what you are trying to decide. I will reply with how I would approach it.