Rasoul Kaljahi

Know whether your LLM feature actually works

You change a prompt, swap a model or adjust retrieval. The output looks better. Is it? Without a test set and a measure you trust, you are guessing.

Discuss your system

The problem

Most teams check LLM output by reading a handful of examples. That misses rare failures, hides regressions, and makes every release a judgment call.

Teams that try automatic scoring often use an LLM as a judge without checking whether it agrees with people. A judge that was never checked can be confidently wrong.

Who this is for

What I do

  1. Understand the task. What the system is for, who uses it, and what a bad answer costs you.
  2. Build a test set. From your real inputs or logs, plus hard cases that tend to break things.
  3. Define what “good” means. Clear criteria and a rating guide, so different reviewers score the same way.
  4. Set up automatic scoring. Including an LLM judge where it fits, checked against human ratings so you know how far to trust it.
  5. Report and hand over. Results, known weak spots, and a simple process to rerun the tests on every change.

What you get

Illustrative example

Invented to show how the work goes. Not a real client. More on the examples page.

A startup has a support chatbot and changes prompts every week. I build a test set from past tickets, define what a correct answer looks like, and set up a scorer checked against human ratings. From then on, each prompt change comes with a before-and-after comparison, and the team stops arguing about whether it helped.

Why me

I spent years designing evaluation and benchmarking frameworks for machine translation engines and LLMs, covering more than 35 languages. My PhD is in measuring the quality of machine output. I know where automatic metrics break down and how to check them.

What this does not cover

I do not build your product or promise a score. I tell you what the numbers say and how much to trust them.

Common questions

Do we need labeled data first? No. We start from real inputs and label a sample together.

Can you work with confidential data? Yes, under an NDA, and I can work inside your environment if needed.

How long does it take? It depends on the task and the data. I will give an estimate after a first call.

Next step

Describe your system and what you are trying to decide. I will reply with how I would approach it.

Email me