LLM Evaluation · QA Management

LLM Evaluation
at Scale AI

Final-line quality decisions on 5,000+ AI conversations. Grew from freelance contributor to QA Manager in under a year by reducing reviewer variance across a rotating team to under 3%.

5,000+
Conversations reviewed
<3%
Reviewer variance
50+
Reviewers trained
100%
Client delivery rate
01 — The Role

Final-line approver for human feedback data going to model training

Scale AI builds the human feedback datasets used to train large language models. As a final-line approver, every accept or reject decision on AI outputs ran through me — checked against safety, accuracy, and compliance guidelines before the data shipped to clients.

The decisions were high-stakes. A conversation flagged incorrectly could corrupt a training signal. A conversation passed incorrectly could introduce a defect into a production model. The job was to hold the line at both ends, consistently, across thousands of edge cases.

02 — How It Progressed

Three roles. Under a year.

Start
Freelance Contributor
Producing individual LLM conversation evaluations — rating quality, safety, and instruction-following on a per-output basis
›
Promoted
Contractor Reviewer
Reviewing and approving the evaluations submitted by other contributors — catching calibration drift and enforcing consistency across outputs
›
Promoted
QA Manager
Owning the full quality standard for the reviewer pool — calibration sessions, rubric development, variance tracking, and final-line decisions
03 — What I Actually Did

Building a shared quality standard across a rotating team

The hardest problem in human feedback at scale is not finding people who can evaluate AI output — it is getting a group of independent reviewers to agree on the same judgment call when the cases are ambiguous. Edge cases do not have obvious answers. The quality standard only holds if the team calibrates to it consistently.

Calibration Sessions
Regular review passes on shared samples where everyone scores independently, then compares. Disagreements get discussed explicitly until the team reaches a shared interpretation of the rubric.
Rubric Development
Translating vague quality criteria into specific, scoreable decision rules. If a reviewer cannot apply a rule consistently across two similar cases, the rule gets rewritten until it can be.
Variance Tracking
Measuring inter-reviewer agreement on shared test sets after each calibration round. Tracking who is drifting high or low and running targeted correction sessions before variance compounds.
Edge Case Documentation
Every contested call that required discussion got written up as a reference case. The library of documented edge cases became the institutional memory of the team's quality standard.
04 — Results

By the numbers

5,000+
Conversations reviewed
Accept / reject decisions
<3%
Post-calibration variance
Across the reviewer pool
50+
Reviewers trained
Rotating contractor pool
100%
Client delivery rate
No batch rejections

Under 3% variance means a team of independent reviewers agreed on the same quality call more than 97% of the time when evaluating ambiguous AI outputs. Getting a rotating team to that level of consistency is the core challenge of human feedback at scale.

05 — Why It Carries Forward

The practice built at Scale AI is the same practice used now

The work done at Scale AI was not specialized LLM research. It was the operational discipline of defining a quality standard, enforcing it across a group of people who each have their own interpretations, and measuring whether the enforcement is working.

That discipline maps directly to AI governance work: writing standards that hold across different users, measuring consistency, catching drift early. The context changed. The approach did not.

What transfers directly

Rubric development, calibration design, variance measurement, and edge case documentation are all reusable skills. The Ecom Way governance system at Shoe Carnival was built on the same framework: define the rule precisely, test it against edge cases, document the gray areas, measure whether the team applies it consistently.

06 — Stack

What this work required

LLM Evaluation Human Feedback (RLHF) Rubric Design Inter-Rater Reliability Calibration Sessions Safety Review Edge Case Documentation Quality Management
LLM Evaluation·QA Management·RLHF·Scale AI·Calibration·Reviewer Training·Human Feedback·Safety Review·Variance Tracking·Quality Standards· LLM Evaluation·QA Management·RLHF·Scale AI·Calibration·Reviewer Training·Human Feedback·Safety Review·Variance Tracking·Quality Standards·