AI Model
Evaluation
Engineer

The Problem

AI teams and businesses adopting LLMs often struggle to choose the right model for their specific use case. Generic benchmark leaderboards don’t tell you which model performs best on your task type, or where each model silently fails. The goal of this project was to run a structured, repeatable evaluation of the three leading LLMs across multiple real-world task categories.

My Approach

I designed a QA evaluation suite covering five distinct task categories, each containing a set of standardised test prompts. Every prompt was issued identically to all three models. Responses were scored independently against a shared rubric before being compared.

Task Categories Tested

• Logical reasoning & multi-step problem solving
• Factual accuracy and knowledge retrieval
• Instruction-following and output format compliance
• Edge-case handling and ambiguous input behaviour
• Consistency across repeated similar prompts

QA Testing Protocol

• Identical prompt sets issued to all three models — no paraphrasing
• Blind evaluation: outputs labelled Model A/B/C during scoring
• Each response scored across 4 dimensions: Accuracy, Completeness, Format Compliance, Reasoning Quality
• Failure modes documented individually with root-cause notes
• Regression runs for edge cases that showed inconsistency

Model Comparison Results

Key Findings

• Claude outperformed across ambiguous and edge-case prompts — showing the strongest ability to flag uncertainty rather than confabulate.
• GPT-4 led on structured knowledge retrieval but showed higher hallucination rates on low-frequency facts.
• Gemini showed the most output variance — identical prompts on different runs yielded structurally different responses.
• All three models degraded in quality when prompts were long, compound, and multi-constraint.
• Instruction format compliance: Claude followed structured output instructions with 91% accuracy vs GPT-4’s 78%.

What I Delivered

• Structured benchmark report with per-model, per-category scores
• Failure mode documentation with root-cause notes and example outputs
• Model selection decision framework for 4 common use-case types
• Regression test set for ongoing consistency monitoring
• Annotated prompt library (60+ prompts) reusable for future evaluations

The Problem

Evaluating LLM output quality is notoriously subjective. Without a structured framework, different evaluators produce inconsistent scores — and no one can clearly articulate why a model response is good or bad. The result is vague feedback that engineering teams can’t act on.

Design Process

The Rubric

Scoring Scale

Outcomes & Impact

The Problem

Content is the lifeblood of modern digital presence — but producing it consistently at quality is exhausting. The business needed a reliable way to source relevant content, evaluate its quality, and publish across multiple social platforms without any manual review, rewriting, or posting.

The existing process was entirely manual: find content, read it, judge quality, rewrite for the platform, schedule, post. This took hours daily, produced inconsistent results, and didn’t scale. When the person responsible was unavailable, the pipeline simply stopped.

The Pipeline

Skills Applied

Outcomes & Impact

What I Delivered