L3VLUP
Sources

Video

How to Systematically Set Up LLM Evals

Dave Ebbelaar

A lesson from Dave Ebbelaar, part of the Primer topic AI Systems: RAG, Agents and Evals.

1 idea from this video

  • TechniqueArtificial intelligence located

    Without evaluation you have a demo, not a system

    Build the test set before the feature, measure every change against it, and keep a person in the loop where the metric cannot be trusted.

    Any AI product discussion, any system design that includes a model, and the question of how you know an AI feature works.