Improve your prompts by hill-climbing with Evaluations

Highlight

Optimizing AI features starts by using Cohen’s kappa to calibrate agreement between the model judge and human experts. Once the evaluation ruler is accurate, you can iterate on the business prompt with confidence.

Core Content

When integrating AI features into an app, one common problem is that the generated result looks acceptable, but something feels wrong and it is hard to say exactly why.

Apple’s Book Tracker app faced this issue. The app lets readers record book reviews and automatically generates tags for books so they can be searched and browsed later. But generated tags often had two problems: either they turned a reader’s personal reaction, such as “poignant”, into a book tag, or they extracted phrases directly from the review, such as “quiet-steadiness”, which are useless for retrieval.

A developer’s first instinct may be to modify the tag-generation prompt directly. That creates two problems. First, after changing the prompt, how do you objectively know whether the result improved? Second, even if the tag-generation prompt improves, the “model judge” evaluating those tags may itself be a crooked ruler: its scoring criteria may not match human experts.

Apple’s solution is hill-climbing: build an evaluation pipeline, change one variable at a time, and let evaluation results guide the next improvement. But effective hill-climbing depends on a reliable evaluation tool.

This introduces the problem of judge alignment. When the dataset is small, scoring differences between human experts and the model judge may not be obvious. As the dataset grows, that drift can become larger and eventually make evaluation results untrustworthy.

The tricky part is that the traditional accuracy metric fails here. If 80% of samples in the dataset are high-quality outputs, both human experts and the model judge tend to give high scores. Even if the model is guessing, accuracy can look high. That hides disagreement between their scoring standards.

Apple introduces Cohen’s kappa to solve this. This statistic subtracts the probability of agreement by random chance from raw accuracy, giving a truer view of agreement between two raters. Only when kappa reaches 0.6 or above can the model judge be considered meaningfully aligned with human experts.

Judge calibration is itself a hill-climbing process: modify the judge prompt, add detailed descriptions of scoring dimensions, provide few-shot examples, and change one variable at a time while using kappa to judge the result. Once the judge is calibrated, it can evaluate improvements to the product feature, such as adding tool calling so the model can retrieve more context.

This methodology turns AI feature optimization from “prompt alchemy by feel” into a measurable scientific experiment.

Details

Basic components of the Evaluations framework

03:54

The core of the Evaluations framework is the Evaluation protocol. A complete evaluation consists of four components:

Dataset: provides test samples. Each sample includes input and expected output.

Subject: calls the feature being evaluated through subject(from:) and obtains model output.

Evaluators: define how output quality is judged, including heuristic evaluators and model-judge evaluators.

Aggregation: computes metrics such as means and standard deviations through aggregateMetrics.

struct BookTaggingEvaluation: Evaluation {
    func subject(from sample: ModelSample<BookTags>) async throws -> ModelSubject<BookTags> {
        let result = try await BookTaggingService.generateTags(for: sample.promptDescription)
        return ModelSubject(value: result)
    }

    var dataset = ArrayLoader(samples:
        Book.sampleBooks.map { book in
            ModelSample(prompt: book.review, expected: BookTags(tags: book.tags))
        }
    )

    var evaluators: Evaluators {
        // Heuristic evaluator
        Evaluator { _, subject in
            let count = subject.value.tags.count
            if (count >= 3 && count <= 8) {
                return tagCount.passing(rationale: "\(count) tags")
            }
            return tagCount.failing(rationale: "Got \(count) tags, expected 3-8")
        }
        // ... More evaluators
    }

    func aggregateMetrics(using aggregator: inout MetricsAggregator) {
        aggregator.group("Heuristics") { group in
            group.computeMean(of: tagCount)
        }
    }
}

Key points: