Evaluating AI systems

How to measure whether an AI system's outputs are actually good, not just plausible-looking: the four required parts of an eval, building a representative dataset, writing a scoring rubric, when deterministic scoring applies versus human and LLM graders, pairwise versus pointwise comparison, per-slice reliability with confidence intervals, evaluating an agent's full trajectory rather than one response, avoiding contamination and overfitting, gating a release on a written decision rule, closing the loop between offline evals and online production signal, and building a minimal eval harness before adopting a framework.