What it does. This skill should be used for advanced LLM evaluation: LLM-as-judge systems, direct scoring, pairwise comparison, rubric calibration, evaluator bias mitigation, confidence scoring, and automated quality assessment. Part of muratcankoylan/Agent-Skills-for-Context-Engineering (muratcankoylan/Agent-Skills-for-Context-Engineering).

Upstream	muratcankoylan/Agent-Skills-for-Context-Engineering
Skill file	skills/advanced-evaluation/SKILL.md
License	MIT
Author	Muratcan Koylan
Fetched	2026-09-10

Install

• npx skills add muratcankoylan/Agent-Skills-for-Context-Engineering --skill advanced-evaluation, or copy the skill folder into ~/.claude/skills/advanced-evaluation/.
• Raw file: curl -sL https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/advanced-evaluation/SKILL.md

SKILL.md (verbatim)

name: advanced-evaluation
description: "This skill should be used for advanced LLM evaluation: LLM-as-judge systems, direct scoring, pairwise comparison, rubric calibration, evaluator bias mitigation, confidence scoring, and automated quality assessment."

Advanced Evaluation

This skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It synthesizes research from academic papers, industry practices, and practical implementation experience into actionable patterns for building reliable evaluation systems.

Key insight: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops.

When to Activate

Activate this skill when:

• Building LLM-as-judge systems for LLM outputs
• Comparing multiple model responses to select the best one
• Establishing consistent quality standards across evaluation teams
• Debugging evaluation systems that show inconsistent results
• Designing A/B tests for prompt or model changes
• Creating rubrics specifically for LLM or human/LLM hybrid judges
• Analyzing correlation between automated and human judgments

Do not activate this skill for adjacent work owned by other skills:

• General deterministic checks, regression suites, production quality gates, or outcome metrics: evaluation.
• Autonomous loop governance, locked rubrics, rollback, or PR approval boundaries: harness-engineering.
• Tool API contracts for evaluation tools: tool-design.

Core Concepts

The Evaluation Taxonomy

Select between two primary approaches based on whether ground truth exists:

Direct Scoring — Use when objective criteria exist (factual accuracy, instruction following, toxicity). A single LLM rates one response on a defined scale. Achieves moderate-to-high reliability for well-defined criteria. Watch for score calibration drift and inconsistent scale interpretation.

Pairwise Comparison — Use for subjective preferences (tone, style, persuasiveness). An LLM compares two responses and selects the better one. Pairwise methods often correlate better with human preference than open-ended direct scoring for subjective tasks (claim-advanced-evaluation-position-swap). Watch for position bias and length bias.

The Bias Landscape

Mitigate these systematic biases in every evaluation system:

Position Bias: First-position responses get preferential treatment. Mitigate by evaluating twice with swapped positions, then apply majority vote or consistency check.

Length Bias: Longer responses score higher regardless of quality. Mitigate by explicitly prompting to ignore length and applying length-normalized scoring.

Self-Enhancement Bias: Models rate their own outputs higher. Mitigate by using different models for generation and evaluation.

Verbosity Bias: Excessive detail scores higher even when unnecessary. Mitigate with criteria-specific rubrics that penalize irrelevant detail.

Authority Bias: Confident tone scores higher regardless of accuracy. Mitigate by requiring evidence citation and adding a fact-checking layer.

Metric Selection Framework

Match metrics to the evaluation task structure:

Task Type	Primary Metrics	Secondary Metrics
Binary classification (pass/fail)	Recall, Precision, F1	Cohen's kappa
Ordinal scale (1-5 rating)	Spearman's rho, Kendall's tau	Cohen's kappa (weighted)
Pairwise preference	Agreement rate, Position consistency	Confidence calibration
Multi-label	Macro-F1, Micro-F1	Per-label precision/recall

Prioritize systematic disagreement patterns over absolute agreement rates because a judge that consistently disagrees with humans on specific criteria is more problematic than one with random noise.

Evaluation Approaches

Direct Scoring Implementation

Build direct scoring with three components: clear criteria, a calibrated scale, and structured output format.

Criteria Definition Pattern:

Criterion: [Name]
Description: [What this criterion measures]
Weight: [Relative importance, 0-1]

Scale Calibration — Choose scale granularity based on rubric detail:

• 1-3: Binary with neutral option, lowest cognitive load
• 1-5: Standard Likert, best balance of granularity and reliability
• 1-10: Use only with detailed per-level rubrics because calibration is harder

Prompt Structure for Direct Scoring:

You are an expert evaluator assessing response quality.

## Task
Evaluate the following response against each criterion.

## Original Prompt
{prompt}

## Response to Evaluate
{response}

## Criteria
{for each criterion: name, description, weight}

## Instructions
For each criterion:
1. Find specific evidence in the response
2. Score according to the rubric (1-{max} scale)
3. Justify your score with evidence
4. Suggest one specific improvement

## Output Format
Respond with structured JSON containing scores, justifications, and summary.

Require evidence before the score in scoring prompts so the judge must anchor its decision in observable output features before emitting a number.

Pairwise Comparison Implementation

Apply position bias mitigation in every pairwise evaluation:

1. Run deterministic pre-checks first: both candidates must satisfy the same schema, source-evidence requirements, and scope constraints.
2. First judge pass: Response A in first position, Response B in second.
3. Second judge pass: Response B in first position, Response A in second.
4. Consistency check: If passes disagree, return TIE with reduced confidence.
5. Final verdict: Consistent winner with averaged confidence and explicit tie-breaker rationale.

Prompt Structure for Pairwise Comparison:

You are an expert evaluator comparing two AI responses.

## Critical Instructions
- Do NOT prefer responses because they are longer
- Do NOT prefer responses based on position (first vs second)
- Focus ONLY on quality according to the specified criteria
- Ties are acceptable when responses are genuinely equivalent

## Original Prompt
{prompt}

## Response A
{response_a}

## Response B
{response_b}

## Comparison Criteria
{criteria list}

## Instructions
1. Analyze each response independently first
2. Compare them on each criterion
3. Determine overall winner with confidence level

## Output Format
JSON with per-criterion comparison, overall winner, confidence (0-1), and reasoning.

Confidence Calibration — Map confidence to position consistency:

• Both passes agree: confidence = average of individual confidences
• Passes disagree: confidence = 0.5, verdict = TIE

Rubric Generation

Generate rubrics to reduce evaluation variance compared to open-ended scoring. Treat exact variance reduction as workload-specific unless measured on the target eval set.

Include these rubric components:

1. Level descriptions: Clear boundaries for each score level
2. Characteristics: Observable features that define each level
3. Examples: Representative text for each level (optional but valuable)
4. Edge cases: Guidance for ambiguous situations
5. Scoring guidelines: General principles for consistent application

Set strictness calibration for the use case:

• Lenient: Lower passing bar, appropriate for encouraging iteration
• Balanced: Typical production expectations
• Strict: High standards for safety-critical or high-stakes evaluation

Adapt rubrics to the domain — use domain-specific terminology. A code readability rubric mentions variables, functions, and comments. A medical accuracy rubric references clinical terminology and evidence standards.

Practical Guidance

Evaluation Pipeline Design

Build production evaluation systems with these layers: Criteria Loader (rubrics + weights) -> Primary Scorer (direct or pairwise) -> Bias Mitigation (position swap, etc.) -> Confidence Scoring (calibration) -> Output (scores + justifications + confidence). See Evaluation Pipeline Diagram for the full visual layout.

Decision Framework: Direct vs. Pairwise

Apply this decision tree:

Is there an objective ground truth?
+-- Yes -> Direct Scoring
|   Examples: factual accuracy, instruction following, format compliance
|
+-- No -> Is it a preference or quality judgment?
    +-- Yes -> Pairwise Comparison
    |   Examples: tone, style, persuasiveness, creativity
    |
    +-- No -> Consider reference-based evaluation
        Examples: summarization (compare to source), translation (compare to reference)

Scaling Evaluation

For high-volume evaluation, apply one of these strategies:

1. Panel of LLMs (PoLL): Use multiple models as judges and aggregate votes to reduce individual model bias. More expensive but more reliable for high-stakes decisions.

2. Hierarchical evaluation: Use a fast cheap model for screening and an expensive model for edge cases. Requires calibration of the screening threshold.

3. Human-in-the-loop: Automate clear cases and route low-confidence decisions to human review. Design feedback loops to improve automated evaluation over time.

Examples

Example 1: Direct Scoring for Accuracy

Input:

Prompt: "What causes seasons on Earth?"
Response: "Seasons are caused by Earth's tilted axis. As Earth orbits the Sun,
different hemispheres receive more direct sunlight at different times of year."
Criterion: Factual Accuracy (weight: 1.0)
Scale: 1-5

Output:

{
  "criterion": "Factual Accuracy",
  "score": 5,
  "evidence": [
    "Correctly identifies axial tilt as primary cause",
    "Correctly explains differential sunlight by hemisphere",
    "No factual errors present"
  ],
  "justification": "Response accurately explains the cause of seasons with correct
scientific reasoning. Both the axial tilt and its effect on sunlight distribution
are correctly described.",
  "improvement": "Could add the specific tilt angle (23.5 degrees) for completeness."
}

Example 2: Pairwise Comparison with Position Swap

Input:

Prompt: "Explain machine learning to a beginner"
Response A: [Technical explanation with jargon]
Response B: [Simple analogy-based explanation]
Criteria: ["clarity", "accessibility"]

First Pass (A first):

{ "winner": "B", "confidence": 0.8 }

Second Pass (B first):

{ "winner": "A", "confidence": 0.6 }

(Note: Winner is A because B was in first position)

Mapped Second Pass:

{ "winner": "B", "confidence": 0.6 }

Final Result:

{
  "winner": "B",
  "confidence": 0.7,
  "positionConsistency": {
    "consistent": true,
    "firstPassWinner": "B",
    "secondPassWinner": "B"
  }
}

Example 3: Rubric Generation

Input:

criterionName: "Code Readability"
criterionDescription: "How easy the code is to understand and maintain"
domain: "software engineering"
scale: "1-5"
strictness: "balanced"

Output (abbreviated):

{
  "levels": [
    {
      "score": 1,
      "label": "Poor",
      "description": "Code is difficult to understand without significant effort",
      "characteristics": [
        "No meaningful variable or function names",
        "No comments or documentation",
        "Deeply nested or convoluted logic"
      ]
    },
    {
      "score": 3,
      "label": "Adequate",
      "description": "Code is understandable with some effort",
      "characteristics": [
        "Most variables have meaningful names",
        "Basic comments present for complex sections",
        "Logic is followable but could be cleaner"
      ]
    },
    {
      "score": 5,
      "label": "Excellent",
      "description": "Code is immediately clear and maintainable",
      "characteristics": [
        "All names are descriptive and consistent",
        "Comprehensive documentation",
        "Clean, modular structure"
      ]
    }
  ],
  "edgeCases": [
    {
      "situation": "Code is well-structured but uses domain-specific abbreviations",
      "guidance": "Score based on readability for domain experts, not general audience"
    }
  ]
}

Guidelines

1. Always require evidence before scores - Evidence-first prompts make judgments easier to audit and reduce ungrounded numeric scoring

2. Always swap positions in pairwise comparison - Single-pass comparison is corrupted by position bias

3. Match scale granularity to rubric specificity - Don't use 1-10 without detailed level descriptions

4. Separate objective and subjective criteria - Use direct scoring for objective, pairwise for subjective

5. Include confidence scores - Calibrate to position consistency and evidence strength

6. Define edge cases explicitly - Ambiguous situations cause the most evaluation variance

7. Use domain-specific rubrics - Generic rubrics produce generic (less useful) evaluations

8. Validate against human judgments - Automated evaluation is only valuable if it correlates with human assessment

9. Monitor for systematic bias - Track disagreement patterns by criterion, response type, model

10. Design for iteration - Evaluation systems improve with feedback loops

Gotchas

1. Scoring without justification: Scores lack grounding and are difficult to debug. Always require evidence-based justification before the score.

2. Single-pass pairwise comparison: Position bias corrupts results when positions are not swapped. Always evaluate twice with swapped positions and check consistency.

3. Overloaded criteria: Criteria that measure multiple things at once produce unreliable scores. Enforce one criterion = one measurable aspect.

4. Missing edge case guidance: Evaluators handle ambiguous cases inconsistently without explicit instructions. Include edge cases in rubrics with clear resolution rules.

5. Ignoring confidence calibration: High-confidence wrong judgments are worse than low-confidence ones. Calibrate confidence to position consistency and evidence strength.

6. Rubric drift: Rubrics become miscalibrated as quality standards evolve or model capabilities improve. Schedule periodic rubric reviews and re-anchor score levels against fresh human-annotated examples.

7. Evaluation prompt sensitivity: Minor wording changes in evaluation prompts can cause material score swings. Version-control evaluation prompts and run regression tests before deploying prompt changes.

8. Uncontrolled length bias: Longer responses systematically score higher even when conciseness is preferred. Add explicit length-neutrality instructions to evaluation prompts and validate with length-controlled test pairs.

Integration

This skill owns judge design and bias mitigation. Adjacent skills own broader quality gates and infrastructure:

• evaluation: general deterministic checks, regression suites, quality gates, and production monitoring.
• context-fundamentals: context structure for judge prompts.
• tool-design: schemas and error handling for evaluation tools.
• context-optimization: token and latency efficiency for high-volume evals.
• harness-engineering: locked evaluator surfaces and governance for autonomous loops.

References

Internal reference:

• LLM-as-Judge Implementation Patterns - Read when: building an evaluation pipeline from scratch or integrating LLM judges into CI/CD
• Bias Mitigation Techniques - Read when: evaluation results show inconsistent or suspicious scoring patterns
• Metric Selection Guide - Read when: choosing statistical metrics to validate evaluation reliability
• Evaluation Pipeline Diagram - Read when: designing the architecture of a multi-stage evaluation system

External research:

• Eugene Yan: Evaluating the Effectiveness of LLM-Evaluators - Read when: surveying the state of the art in LLM evaluation
• Judging LLM-as-a-Judge (Zheng et al., 2023) - Read when: understanding position bias and MT-Bench methodology
• G-Eval: NLG Evaluation using GPT-4 (Liu et al., 2023) - Read when: implementing chain-of-thought evaluation scoring
• Large Language Models are not Fair Evaluators (Wang et al., 2023) - Read when: diagnosing systematic bias in evaluation outputs

Related skills in this collection:

• evaluation - Foundational evaluation concepts
• context-fundamentals - Context structure for evaluation prompts
• tool-design - Building evaluation tools

---

Skill Metadata

Created: 2025-12-24 Last Updated: 2026-05-15 Author: Agent Skills for Context Engineering Contributors Version: 2.1.0

Other files in this skill

• references/bias-mitigation.md
• references/evaluation-pipeline.md
• references/implementation-patterns.md
• references/metrics-guide.md
• scripts/evaluation_example.py

references/bias-mitigation.md (verbatim)

Bias Mitigation Techniques for LLM Evaluation

This reference details specific techniques for mitigating known biases in LLM-as-a-Judge systems.

Position Bias

The Problem

In pairwise comparison, LLMs systematically prefer responses in certain positions. Research shows:

• GPT has mild first-position bias (~55% preference for first position in ties)
• Claude shows similar patterns
• Smaller models often show stronger bias

Mitigation: Position Swapping Protocol

async def position_swap_comparison(response_a, response_b, prompt, criteria):
    # Pass 1: Original order
    result_ab = await compare(response_a, response_b, prompt, criteria)
    
    # Pass 2: Swapped order
    result_ba = await compare(response_b, response_a, prompt, criteria)
    
    # Map second result (A in second position → B in first)
    result_ba_mapped = {
        'winner': {'A': 'B', 'B': 'A', 'TIE': 'TIE'}[result_ba['winner']],
        'confidence': result_ba['confidence']
    }
    
    # Consistency check
    if result_ab['winner'] == result_ba_mapped['winner']:
        return {
            'winner': result_ab['winner'],
            'confidence': (result_ab['confidence'] + result_ba_mapped['confidence']) / 2,
            'position_consistent': True
        }
    else:
        # Disagreement indicates position bias was a factor
        return {
            'winner': 'TIE',
            'confidence': 0.5,
            'position_consistent': False,
            'bias_detected': True
        }

Alternative: Multiple Shuffles

For higher reliability, use multiple position orderings:

async def multi_shuffle_comparison(response_a, response_b, prompt, criteria, n_shuffles=3):
    results = []
    for i in range(n_shuffles):
        if i % 2 == 0:
            r = await compare(response_a, response_b, prompt, criteria)
        else:
            r = await compare(response_b, response_a, prompt, criteria)
            r['winner'] = {'A': 'B', 'B': 'A', 'TIE': 'TIE'}[r['winner']]
        results.append(r)
    
    # Majority vote
    winners = [r['winner'] for r in results]
    final_winner = max(set(winners), key=winners.count)
    agreement = winners.count(final_winner) / len(winners)
    
    return {
        'winner': final_winner,
        'confidence': agreement,
        'n_shuffles': n_shuffles
    }

Length Bias

The Problem

LLMs tend to rate longer responses higher, regardless of quality. This manifests as:

• Verbose responses receiving inflated scores
• Concise but complete responses penalized
• Padding and repetition being rewarded

Mitigation: Explicit Prompting

Include anti-length-bias instructions in the prompt:

CRITICAL EVALUATION GUIDELINES:
- Do NOT prefer responses because they are longer
- Concise, complete answers are as valuable as detailed ones
- Penalize unnecessary verbosity or repetition
- Focus on information density, not word count

Mitigation: Length-Normalized Scoring

def length_normalized_score(score, response_length, target_length=500):
    """Adjust score based on response length."""
    length_ratio = response_length / target_length
    
    if length_ratio > 2.0:
        # Penalize excessively long responses
        penalty = (length_ratio - 2.0) * 0.1
        return max(score - penalty, 1)
    elif length_ratio < 0.3:
        # Penalize excessively short responses
        penalty = (0.3 - length_ratio) * 0.5
        return max(score - penalty, 1)
    else:
        return score

Mitigation: Separate Length Criterion

Make length a separate, explicit criterion so it's not implicitly rewarded:

criteria = [
    {"name": "Accuracy", "description": "Factual correctness", "weight": 0.4},
    {"name": "Completeness", "description": "Covers key points", "weight": 0.3},
    {"name": "Conciseness", "description": "No unnecessary content", "weight": 0.3}  # Explicit
]

Self-Enhancement Bias

The Problem

Models rate outputs generated by themselves (or similar models) higher than outputs from different models.

Mitigation: Cross-Model Evaluation

Use a different model family for evaluation than generation:

def get_evaluator_model(generator_model):
    """Select evaluator to avoid self-enhancement bias."""
    if 'gpt' in generator_model.lower():
        return 'claude-4-5-sonnet'
    elif 'claude' in generator_model.lower():
        return 'gpt-5.2'
    else:
        return 'gpt-5.2'  # Default

Mitigation: Blind Evaluation

Remove model attribution from responses before evaluation:

def anonymize_response(response, model_name):
    """Remove model-identifying patterns."""
    patterns = [
        f"As {model_name}",
        "I am an AI",
        "I don't have personal opinions",
        # Model-specific patterns
    ]
    anonymized = response
    for pattern in patterns:
        anonymized = anonymized.replace(pattern, "[REDACTED]")
    return anonymized

Verbosity Bias

The Problem

Detailed explanations receive higher scores even when the extra detail is irrelevant or incorrect.

Mitigation: Relevance-Weighted Scoring

async def relevance_weighted_evaluation(response, prompt, criteria):
    # First, assess relevance of each segment
    relevance_scores = await assess_relevance(response, prompt)
    
    # Weight evaluation by relevance
    segments = split_into_segments(response)
    weighted_scores = []
    for segment, relevance in zip(segments, relevance_scores):
        if relevance > 0.5:  # Only count relevant segments
            score = await evaluate_segment(segment, prompt, criteria)
            weighted_scores.append(score * relevance)
    
    return sum(weighted_scores) / len(weighted_scores)

Mitigation: Rubric with Verbosity Penalty

Include explicit verbosity penalties in rubrics:

rubric_levels = [
    {
        "score": 5,
        "description": "Complete and concise. All necessary information, nothing extraneous.",
        "characteristics": ["Every sentence adds value", "No repetition", "Appropriately scoped"]
    },
    {
        "score": 3,
        "description": "Complete but verbose. Contains unnecessary detail or repetition.",
        "characteristics": ["Main points covered", "Some tangents", "Could be more concise"]
    },
    # ... etc
]

Authority Bias

The Problem

Confident, authoritative tone is rated higher regardless of accuracy.

Mitigation: Evidence Requirement

Require explicit evidence for claims:

For each claim in the response:
1. Identify whether it's a factual claim
2. Note if evidence or sources are provided
3. Score based on verifiability, not confidence

IMPORTANT: Confident claims without evidence should NOT receive higher scores than 
hedged claims with evidence.

Mitigation: Fact-Checking Layer

Add a fact-checking step before scoring:

async def fact_checked_evaluation(response, prompt, criteria):
    # Extract claims
    claims = await extract_claims(response)
    
    # Fact-check each claim
    fact_check_results = await asyncio.gather(*[
        verify_claim(claim) for claim in claims
    ])
    
    # Adjust score based on fact-check results
    accuracy_factor = sum(r['verified'] for r in fact_check_results) / len(fact_check_results)
    
    base_score = await evaluate(response, prompt, criteria)
    return base_score * (0.7 + 0.3 * accuracy_factor)  # At least 70% of score

Aggregate Bias Detection

Monitor for systematic biases in production:

class BiasMonitor:
    def __init__(self):
        self.evaluations = []
    
    def record(self, evaluation):
        self.evaluations.append(evaluation)
    
    def detect_position_bias(self):
        """Detect if first position wins more often than expected."""
        first_wins = sum(1 for e in self.evaluations if e['first_position_winner'])
        expected = len(self.evaluations) * 0.5
        z_score = (first_wins - expected) / (expected * 0.5) ** 0.5
        return {'bias_detected': abs(z_score) > 2, 'z_score': z_score}
    
    def detect_length_bias(self):
        """Detect if longer responses score higher."""
        from scipy.stats import spearmanr
        lengths = [e['response_length'] for e in self.evaluations]
        scores = [e['score'] for e in self.evaluations]
        corr, p_value = spearmanr(lengths, scores)
        return {'bias_detected': corr > 0.3 and p_value < 0.05, 'correlation': corr}

Summary Table

Bias	Primary Mitigation	Secondary Mitigation	Detection Method
Position	Position swapping	Multiple shuffles	Consistency check
Length	Explicit prompting	Length normalization	Length-score correlation
Self-enhancement	Cross-model evaluation	Anonymization	Model comparison study
Verbosity	Relevance weighting	Rubric penalties	Relevance scoring
Authority	Evidence requirement	Fact-checking layer	Confidence-accuracy correlation

references/evaluation-pipeline.md (verbatim)

Evaluation Pipeline Diagram

Visual layout of a production evaluation pipeline.

┌─────────────────────────────────────────────────┐
│                 Evaluation Pipeline              │
├─────────────────────────────────────────────────┤
│                                                   │
│  Input: Response + Prompt + Context               │
│           │                                       │
│           ▼                                       │
│  ┌─────────────────────┐                         │
│  │   Criteria Loader   │ ◄── Rubrics, weights    │
│  └──────────┬──────────┘                         │
│             │                                     │
│             ▼                                     │
│  ┌─────────────────────┐                         │
│  │   Primary Scorer    │ ◄── Direct or Pairwise  │
│  └──────────┬──────────┘                         │
│             │                                     │
│             ▼                                     │
│  ┌─────────────────────┐                         │
│  │   Bias Mitigation   │ ◄── Position swap, etc. │
│  └──────────┬──────────┘                         │
│             │                                     │
│             ▼                                     │
│  ┌─────────────────────┐                         │
│  │ Confidence Scoring  │ ◄── Calibration         │
│  └──────────┬──────────┘                         │
│             │                                     │
│             ▼                                     │
│  Output: Scores + Justifications + Confidence     │
│                                                   │
└─────────────────────────────────────────────────┘

Pipeline Stages

1. Criteria Loader: Loads rubrics and criterion weights from configuration
2. Primary Scorer: Applies direct scoring or pairwise comparison
3. Bias Mitigation: Runs position swaps, length normalization, and other debiasing
4. Confidence Scoring: Calibrates confidence based on position consistency and evidence strength

references/implementation-patterns.md (verbatim)

LLM-as-Judge Implementation Patterns

This reference provides detailed implementation patterns for building production-grade LLM evaluation systems.

Pattern 1: Structured Evaluation Pipeline

The most reliable evaluation systems follow a structured pipeline that separates concerns:

Input Validation → Criteria Loading → Scoring → Bias Mitigation → Output Formatting

Input Validation Layer

Before evaluation begins, validate:

1. Response presence: Non-empty response to evaluate
2. Prompt presence: Original prompt for context
3. Criteria validity: At least one criterion with name and description
4. Weight normalization: Weights sum to 1.0 (or normalize them)

def validate_input(response, prompt, criteria):
    if not response or not response.strip():
        raise ValueError("Response cannot be empty")
    if not prompt or not prompt.strip():
        raise ValueError("Prompt cannot be empty")
    if not criteria or len(criteria) == 0:
        raise ValueError("At least one criterion required")
    
    # Normalize weights
    total_weight = sum(c.get('weight', 1) for c in criteria)
    for c in criteria:
        c['weight'] = c.get('weight', 1) / total_weight

Criteria Loading Layer

Criteria should be loaded from configuration, not hardcoded:

class CriteriaLoader:
    def __init__(self, rubric_path=None):
        self.rubrics = self._load_rubrics(rubric_path)
    
    def get_criteria(self, task_type):
        return self.rubrics.get(task_type, self.default_criteria)
    
    def get_rubric(self, criterion_name):
        return self.rubrics.get(criterion_name, {}).get('levels', [])

Scoring Layer

The scoring layer handles the actual LLM call:

async def score_response(response, prompt, criteria, rubric, model):
    system_prompt = build_system_prompt(criteria, rubric)
    user_prompt = build_user_prompt(response, prompt, criteria)
    
    result = await generate_text(
        model=model,
        system=system_prompt,
        prompt=user_prompt,
        temperature=0.3  # Lower temperature for consistency
    )
    
    return parse_scores(result.text)

Bias Mitigation Layer

For pairwise comparison, always include position swapping:

async def compare_with_bias_mitigation(response_a, response_b, prompt, criteria, model):
    # First pass: A first
    pass1 = await compare_pair(response_a, response_b, prompt, criteria, model)
    
    # Second pass: B first
    pass2 = await compare_pair(response_b, response_a, prompt, criteria, model)
    
    # Map pass2 winner back
    pass2_mapped = map_winner(pass2.winner)  # A→B, B→A, TIE→TIE
    
    # Check consistency
    if pass1.winner == pass2_mapped:
        return {
            'winner': pass1.winner,
            'confidence': (pass1.confidence + pass2.confidence) / 2,
            'consistent': True
        }
    else:
        return {
            'winner': 'TIE',
            'confidence': 0.5,
            'consistent': False
        }

Pattern 2: Hierarchical Evaluation

For complex evaluations, use a hierarchical approach:

Quick Screen (cheap model) → Detailed Evaluation (expensive model) → Human Review (edge cases)

Quick Screen Implementation

async def quick_screen(response, prompt, threshold=0.7):
    """Fast, cheap screening for obvious passes/fails."""
    result = await generate_text(
        model='gpt-5.2',  # Cheaper model
        prompt=f"Rate 0-1 if this response adequately addresses the prompt:\n\nPrompt: {prompt}\n\nResponse: {response}",
        temperature=0
    )
    score = float(result.text.strip())
    return score, score > threshold

Detailed Evaluation

async def detailed_evaluation(response, prompt, criteria):
    """Full evaluation for borderline or important cases."""
    result = await generate_text(
        model='gpt-5.2',  # More capable model
        system=DETAILED_EVALUATION_PROMPT,
        prompt=build_detailed_prompt(response, prompt, criteria),
        temperature=0.3
    )
    return parse_detailed_scores(result.text)

Pattern 3: Panel of LLM Judges (PoLL)

For high-stakes evaluation, use multiple models:

async def poll_evaluation(response, prompt, criteria, models):
    """Aggregate judgments from multiple LLM judges."""
    results = await asyncio.gather(*[
        score_with_model(response, prompt, criteria, model)
        for model in models
    ])
    
    # Aggregate scores
    aggregated = aggregate_scores(results)
    
    # Calculate agreement
    agreement = calculate_agreement(results)
    
    return {
        'scores': aggregated,
        'agreement': agreement,
        'individual_results': results
    }

def aggregate_scores(results):
    """Aggregate scores using median (robust to outliers)."""
    scores = {}
    for criterion in results[0]['scores'].keys():
        criterion_scores = [r['scores'][criterion] for r in results]
        scores[criterion] = {
            'score': statistics.median(criterion_scores),
            'std': statistics.stdev(criterion_scores) if len(criterion_scores) > 1 else 0
        }
    return scores

Pattern 4: Confidence Calibration

Confidence scores should be calibrated to actual reliability:

def calibrate_confidence(raw_confidence, position_consistent, evidence_count):
    """Calibrate confidence based on multiple signals."""
    
    # Base confidence from model output
    calibrated = raw_confidence
    
    # Position consistency is a strong signal
    if not position_consistent:
        calibrated *= 0.6  # Significant reduction
    
    # More evidence = higher confidence
    evidence_factor = min(evidence_count / 3, 1.0)  # Cap at 3 pieces
    calibrated *= (0.7 + 0.3 * evidence_factor)
    
    return min(calibrated, 0.99)  # Never 100% confident

Pattern 5: Output Formatting

Always return structured outputs with consistent schemas:

@dataclass
class ScoreResult:
    criterion: str
    score: float
    max_score: float
    justification: str
    evidence: List[str]
    improvement: str

@dataclass
class EvaluationResult:
    success: bool
    scores: List[ScoreResult]
    overall_score: float
    weighted_score: float
    summary: Dict[str, Any]
    metadata: Dict[str, Any]

def format_output(scores, metadata) -> EvaluationResult:
    """Format evaluation results consistently."""
    return EvaluationResult(
        success=True,
        scores=scores,
        overall_score=sum(s.score for s in scores) / len(scores),
        weighted_score=calculate_weighted_score(scores),
        summary=generate_summary(scores),
        metadata=metadata
    )

Error Handling Patterns

Graceful Degradation

async def evaluate_with_fallback(response, prompt, criteria):
    try:
        return await full_evaluation(response, prompt, criteria)
    except RateLimitError:
        # Fall back to simpler evaluation
        return await simple_evaluation(response, prompt, criteria)
    except ParseError as e:
        # Return partial results with error flag
        return {
            'success': False,
            'partial_results': e.partial_data,
            'error': str(e)
        }

Retry Logic

async def evaluate_with_retry(response, prompt, criteria, max_retries=3):
    for attempt in range(max_retries):
        try:
            result = await evaluate(response, prompt, criteria)
            if is_valid_result(result):
                return result
        except TransientError:
            await asyncio.sleep(2 ** attempt)  # Exponential backoff
    
    raise EvaluationError("Max retries exceeded")

Testing Patterns

Unit Tests for Parsing

def test_score_parsing():
    raw_output = '{"scores": [{"criterion": "Accuracy", "score": 4}]}'
    result = parse_scores(raw_output)
    assert result.scores[0].criterion == "Accuracy"
    assert result.scores[0].score == 4

def test_malformed_output():
    raw_output = 'Invalid JSON'
    with pytest.raises(ParseError):
        parse_scores(raw_output)

Integration Tests with Real API

@pytest.mark.integration
async def test_full_evaluation_pipeline():
    result = await evaluate(
        response="Water boils at 100°C at sea level.",
        prompt="At what temperature does water boil?",
        criteria=[{"name": "Accuracy", "description": "Factual correctness", "weight": 1}]
    )
    
    assert result.success
    assert len(result.scores) == 1
    assert result.scores[0].score >= 4  # Should score high for accurate response

Bias Detection Tests

async def test_position_bias_mitigation():
    # Same response in both positions should tie
    result = await compare(
        response_a="Same response",
        response_b="Same response",
        prompt="Test prompt",
        criteria=["quality"],
        swap_positions=True
    )
    
    assert result.winner == "TIE"
    assert result.consistent == True

references/metrics-guide.md (verbatim)

Metric Selection Guide for LLM Evaluation

This reference provides guidance on selecting appropriate metrics for different evaluation scenarios.

Metric Categories

Classification Metrics

Use for binary or multi-class evaluation tasks (pass/fail, correct/incorrect).

Precision

Precision = True Positives / (True Positives + False Positives)

Interpretation: Of all responses the judge said were good, what fraction were actually good?

Use when: False positives are costly (e.g., approving unsafe content)

def precision(predictions, ground_truth):
    true_positives = sum(1 for p, g in zip(predictions, ground_truth) if p == 1 and g == 1)
    predicted_positives = sum(predictions)
    return true_positives / predicted_positives if predicted_positives > 0 else 0

Recall

Recall = True Positives / (True Positives + False Negatives)

Interpretation: Of all actually good responses, what fraction did the judge identify?

Use when: False negatives are costly (e.g., missing good content in filtering)

def recall(predictions, ground_truth):
    true_positives = sum(1 for p, g in zip(predictions, ground_truth) if p == 1 and g == 1)
    actual_positives = sum(ground_truth)
    return true_positives / actual_positives if actual_positives > 0 else 0

F1 Score

F1 = 2 * (Precision * Recall) / (Precision + Recall)

Interpretation: Harmonic mean of precision and recall

Use when: You need a single number balancing both concerns

def f1_score(predictions, ground_truth):
    p = precision(predictions, ground_truth)
    r = recall(predictions, ground_truth)
    return 2 * p * r / (p + r) if (p + r) > 0 else 0

Agreement Metrics

Use for comparing automated evaluation with human judgment.

Cohen's Kappa (κ)

κ = (Observed Agreement - Expected Agreement) / (1 - Expected Agreement)

Interpretation: Agreement adjusted for chance

• κ > 0.8: Almost perfect agreement
• κ 0.6-0.8: Substantial agreement
• κ 0.4-0.6: Moderate agreement
• κ < 0.4: Fair to poor agreement

Use for: Binary or categorical judgments

def cohens_kappa(judge1, judge2):
    from sklearn.metrics import cohen_kappa_score
    return cohen_kappa_score(judge1, judge2)

Weighted Kappa

For ordinal scales where disagreement severity matters:

def weighted_kappa(judge1, judge2):
    from sklearn.metrics import cohen_kappa_score
    return cohen_kappa_score(judge1, judge2, weights='quadratic')

Interpretation: Penalizes large disagreements more than small ones

Correlation Metrics

Use for ordinal/continuous scores.

Spearman's Rank Correlation (ρ)

Interpretation: Correlation between rankings, not absolute values

• ρ > 0.9: Very strong correlation
• ρ 0.7-0.9: Strong correlation
• ρ 0.5-0.7: Moderate correlation
• ρ < 0.5: Weak correlation

Use when: Order matters more than exact values

def spearmans_rho(scores1, scores2):
    from scipy.stats import spearmanr
    rho, p_value = spearmanr(scores1, scores2)
    return {'rho': rho, 'p_value': p_value}

Kendall's Tau (τ)

Interpretation: Similar to Spearman but based on pairwise concordance

Use when: You have many tied values

def kendalls_tau(scores1, scores2):
    from scipy.stats import kendalltau
    tau, p_value = kendalltau(scores1, scores2)
    return {'tau': tau, 'p_value': p_value}

Pearson Correlation (r)

Interpretation: Linear correlation between scores

Use when: Exact score values matter, not just order

def pearsons_r(scores1, scores2):
    from scipy.stats import pearsonr
    r, p_value = pearsonr(scores1, scores2)
    return {'r': r, 'p_value': p_value}

Pairwise Comparison Metrics

Agreement Rate

Agreement = (Matching Decisions) / (Total Comparisons)

Interpretation: Simple percentage of agreement

def pairwise_agreement(decisions1, decisions2):
    matches = sum(1 for d1, d2 in zip(decisions1, decisions2) if d1 == d2)
    return matches / len(decisions1)

Position Consistency

Consistency = (Consistent across position swaps) / (Total comparisons)

Interpretation: How often does swapping position change the decision?

def position_consistency(results):
    consistent = sum(1 for r in results if r['position_consistent'])
    return consistent / len(results)

Selection Decision Tree

What type of evaluation task?
│
├── Binary classification (pass/fail)
│   └── Use: Precision, Recall, F1, Cohen's κ
│
├── Ordinal scale (1-5 rating)
│   ├── Comparing to human judgments?
│   │   └── Use: Spearman's ρ, Weighted κ
│   └── Comparing two automated judges?
│       └── Use: Kendall's τ, Spearman's ρ
│
├── Pairwise preference
│   └── Use: Agreement rate, Position consistency
│
└── Multi-label classification
    └── Use: Macro-F1, Micro-F1, Per-label metrics

Metric Selection by Use Case

Use Case 1: Validating Automated Evaluation

Goal: Ensure automated evaluation correlates with human judgment

Recommended Metrics:

1. Primary: Spearman's ρ (for ordinal scales) or Cohen's κ (for categorical)
2. Secondary: Per-criterion agreement
3. Diagnostic: Confusion matrix for systematic errors

def validate_automated_eval(automated_scores, human_scores, criteria):
    results = {}
    
    # Overall correlation
    results['overall_spearman'] = spearmans_rho(automated_scores, human_scores)
    
    # Per-criterion agreement
    for criterion in criteria:
        auto_crit = [s[criterion] for s in automated_scores]
        human_crit = [s[criterion] for s in human_scores]
        results[f'{criterion}_spearman'] = spearmans_rho(auto_crit, human_crit)
    
    return results

Use Case 2: Comparing Two Models

Goal: Determine which model produces better outputs

Recommended Metrics:

1. Primary: Win rate (from pairwise comparison)
2. Secondary: Position consistency (bias check)
3. Diagnostic: Per-criterion breakdown

def compare_models(model_a_outputs, model_b_outputs, prompts):
    results = []
    for a, b, p in zip(model_a_outputs, model_b_outputs, prompts):
        comparison = await compare_with_position_swap(a, b, p)
        results.append(comparison)
    
    return {
        'a_wins': sum(1 for r in results if r['winner'] == 'A'),
        'b_wins': sum(1 for r in results if r['winner'] == 'B'),
        'ties': sum(1 for r in results if r['winner'] == 'TIE'),
        'position_consistency': position_consistency(results)
    }

Use Case 3: Quality Monitoring

Goal: Track evaluation quality over time

Recommended Metrics:

1. Primary: Rolling agreement with human spot-checks
2. Secondary: Score distribution stability
3. Diagnostic: Bias indicators (position, length)

class QualityMonitor:
    def __init__(self, window_size=100):
        self.window = deque(maxlen=window_size)
    
    def add_evaluation(self, automated, human_spot_check=None):
        self.window.append({
            'automated': automated,
            'human': human_spot_check,
            'length': len(automated['response'])
        })
    
    def get_metrics(self):
        # Filter to evaluations with human spot-checks
        with_human = [e for e in self.window if e['human'] is not None]
        
        if len(with_human) < 10:
            return {'insufficient_data': True}
        
        auto_scores = [e['automated']['score'] for e in with_human]
        human_scores = [e['human']['score'] for e in with_human]
        
        return {
            'correlation': spearmans_rho(auto_scores, human_scores),
            'mean_difference': np.mean([a - h for a, h in zip(auto_scores, human_scores)]),
            'length_correlation': spearmans_rho(
                [e['length'] for e in self.window],
                [e['automated']['score'] for e in self.window]
            )
        }

Interpreting Metric Results

Good Evaluation System Indicators

Metric	Good	Acceptable	Concerning
Spearman's ρ	> 0.8	0.6-0.8	< 0.6
Cohen's κ	> 0.7	0.5-0.7	< 0.5
Position consistency	> 0.9	0.8-0.9	< 0.8
Length correlation	< 0.2	0.2-0.4	> 0.4

Warning Signs

1. High agreement but low correlation: May indicate calibration issues
2. Low position consistency: Position bias affecting results
3. High length correlation: Length bias inflating scores
4. Per-criterion variance: Some criteria may be poorly defined

Reporting Template

## Evaluation System Metrics Report

### Human Agreement
- Spearman's ρ: 0.82 (p < 0.001)
- Cohen's κ: 0.74
- Sample size: 500 evaluations

### Bias Indicators
- Position consistency: 91%
- Length-score correlation: 0.12

### Per-Criterion Performance
| Criterion | Spearman's ρ | κ |
|-----------|--------------|---|
| Accuracy | 0.88 | 0.79 |
| Clarity | 0.76 | 0.68 |
| Completeness | 0.81 | 0.72 |

### Recommendations
- All metrics within acceptable ranges
- Monitor "Clarity" criterion - lower agreement may indicate need for rubric refinement

Back to muratcankoylan/Agent-Skills-for-Context-Engineering or Agent skills.