> ## Documentation Index
> Fetch the complete documentation index at: https://docs.exulu.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Overview

> Custom evaluation functions to measure and score agent performance

## Overview

`ExuluEval` is a class for creating custom evaluation functions that measure and score agent performance against test cases. Evaluations allow you to systematically test your agents, track quality over time, and identify areas for improvement.

## What is ExuluEval?

ExuluEval provides a framework for defining evaluation logic that:

* **Scores agent responses**: Returns a score from 0-100 based on custom criteria
* **Runs against test cases**: Evaluates agent behavior using structured test inputs
* **Supports any evaluation method**: Custom logic, LLM-as-judge, regex matching, or any scoring approach
* **Integrates with queues**: Can be run as background jobs using ExuluQueues
* **Enables A/B testing**: Compare different agent configurations, prompts, or models

<CardGroup cols={2}>
  <Card title="Custom scoring logic" icon="code">
    Write any evaluation function in TypeScript
  </Card>

  <Card title="Test cases" icon="clipboard-check">
    Structured inputs with expected outputs
  </Card>

  <Card title="LLM-as-judge" icon="scale-balanced">
    Use LLMs to evaluate response quality
  </Card>

  <Card title="Queue integration" icon="layer-group">
    Run evaluations as background jobs
  </Card>
</CardGroup>

## Why use evaluations?

Evaluations help you:

<AccordionGroup>
  <Accordion title="Measure quality">
    Quantify agent performance with consistent scoring criteria across all responses
  </Accordion>

  <Accordion title="Prevent regressions">
    Catch performance degradation when updating prompts, models, or tools
  </Accordion>

  <Accordion title="Compare configurations">
    A/B test different agent setups to find the best performing configuration
  </Accordion>

  <Accordion title="Track improvements">
    Monitor evaluation scores over time to verify that changes improve quality
  </Accordion>

  <Accordion title="Automate testing">
    Build CI/CD pipelines that fail if evaluation scores drop below thresholds
  </Accordion>
</AccordionGroup>

## Quick start

```typescript theme={null}
import { ExuluEval } from "@exulu/backend";

// Create an evaluation function
const exactMatchEval = new ExuluEval({
  id: "exact_match",
  name: "Exact Match",
  description: "Checks if response exactly matches expected output",
  llm: false, // Not using LLM-as-judge
  execute: async ({ messages, testCase }) => {
    const lastMessage = messages[messages.length - 1];
    const response = lastMessage?.content || "";

    return response === testCase.expected_output ? 100 : 0;
  }
});

// Run against a test case
const score = await exactMatchEval.run(
  agent,        // Agent database record
  backend,      // ExuluAgent instance
  testCase,     // Test case with inputs and expected output
  messages      // Conversation messages
);

console.log(`Score: ${score}/100`);
```

## Evaluation types

### Custom logic evaluations

Write any scoring logic in TypeScript:

```typescript theme={null}
const containsKeywordEval = new ExuluEval({
  id: "contains_keyword",
  name: "Contains Keyword",
  description: "Checks if response contains required keywords",
  llm: false,
  execute: async ({ messages, testCase, config }) => {
    const lastMessage = messages[messages.length - 1];
    const response = lastMessage?.content?.toLowerCase() || "";

    const keywords = config?.keywords || [];
    const foundKeywords = keywords.filter(kw => response.includes(kw.toLowerCase()));

    return (foundKeywords.length / keywords.length) * 100;
  },
  config: [
    {
      name: "keywords",
      description: "List of keywords that should appear in the response"
    }
  ]
});
```

### LLM-as-judge evaluations

Use an LLM to evaluate response quality:

```typescript theme={null}
const llmJudgeEval = new ExuluEval({
  id: "llm_judge_quality",
  name: "LLM Judge - Quality",
  description: "Uses an LLM to evaluate response quality",
  llm: true, // Using LLM
  execute: async ({ backend, messages, testCase, config }) => {
    const lastMessage = messages[messages.length - 1];
    const response = lastMessage?.content || "";

    const prompt = `
You are an expert evaluator. Rate the quality of this response on a scale of 0-100.

Test Case: ${testCase.name}
Expected: ${testCase.expected_output}
Actual Response: ${response}

Consider:
- Accuracy: Does it match the expected output?
- Completeness: Does it address all aspects?
- Clarity: Is it well-structured and clear?

Respond with ONLY a number from 0-100.
    `;

    const result = await backend.generateSync({
      prompt,
      agentInstance: await loadAgent(config?.judgeAgentId),
      statistics: { label: "eval", trigger: "system" }
    });

    const score = parseInt(result.text);
    return isNaN(score) ? 0 : Math.max(0, Math.min(100, score));
  },
  config: [
    {
      name: "judgeAgentId",
      description: "Agent ID to use as judge"
    }
  ]
});
```

### Tool usage evaluations

Check if the agent used the correct tools:

```typescript theme={null}
const toolUsageEval = new ExuluEval({
  id: "tool_usage",
  name: "Tool Usage",
  description: "Checks if agent used expected tools",
  llm: false,
  execute: async ({ messages, testCase }) => {
    // Extract tool calls from messages
    const toolCalls = messages
      .flatMap(msg => msg.toolInvocations || [])
      .map(inv => inv.toolName);

    const expectedTools = testCase.expected_tools || [];

    if (expectedTools.length === 0) return 100;

    const usedExpected = expectedTools.filter(tool => toolCalls.includes(tool));

    return (usedExpected.length / expectedTools.length) * 100;
  }
});
```

### Similarity evaluations

Use embeddings to measure semantic similarity:

```typescript theme={null}
import { ExuluEval, ExuluEmbedder } from "@exulu/backend";

const similarityEval = new ExuluEval({
  id: "semantic_similarity",
  name: "Semantic Similarity",
  description: "Measures semantic similarity between response and expected output",
  llm: false,
  execute: async ({ messages, testCase }) => {
    const lastMessage = messages[messages.length - 1];
    const response = lastMessage?.content || "";

    const embedder = new ExuluEmbedder({
      id: "eval_embedder",
      name: "Eval Embedder",
      provider: "openai",
      model: "text-embedding-3-small",
      vectorDimensions: 1536,
      authenticationInformation: await ExuluVariables.get("openai_api_key")
    });

    const [responseEmb, expectedEmb] = await embedder.generate([
      response,
      testCase.expected_output
    ]);

    // Cosine similarity
    const similarity = cosineSimilarity(responseEmb, expectedEmb);

    return similarity * 100;
  }
});

function cosineSimilarity(a: number[], b: number[]): number {
  const dotProduct = a.reduce((sum, val, i) => sum + val * b[i], 0);
  const magnitudeA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0));
  const magnitudeB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0));
  return dotProduct / (magnitudeA * magnitudeB);
}
```

## Test cases

Test cases define the inputs and expected outputs for evaluations:

```typescript theme={null}
interface TestCase {
  id: string;
  name: string;
  description?: string;
  inputs: UIMessage[];                      // Input conversation
  expected_output: string;                  // Expected response
  expected_tools?: string[];                // Expected tool calls
  expected_knowledge_sources?: string[];    // Expected contexts used
  expected_agent_tools?: string[];          // Expected agent tools
  createdAt: string;
  updatedAt: string;
}
```

**Example test case:**

```typescript theme={null}
const testCase: TestCase = {
  id: "tc_001",
  name: "Weather query",
  description: "User asks about weather",
  inputs: [
    {
      role: "user",
      content: "What's the weather like in San Francisco?"
    }
  ],
  expected_output: "Based on current data, it's 68°F and sunny in San Francisco.",
  expected_tools: ["get_weather"],
  expected_knowledge_sources: [],
  expected_agent_tools: [],
  createdAt: "2025-01-15T10:00:00Z",
  updatedAt: "2025-01-15T10:00:00Z"
};
```

## Running evaluations

### Basic evaluation run

```typescript theme={null}
import { ExuluEval, ExuluAgent } from "@exulu/backend";

const eval = new ExuluEval({
  id: "my_eval",
  name: "My Evaluation",
  description: "Custom evaluation",
  llm: false,
  execute: async ({ messages, testCase }) => {
    // Your scoring logic
    return 85; // Score from 0-100
  }
});

// Run evaluation
const score = await eval.run(
  agent,        // Agent DB record
  backend,      // ExuluAgent instance
  testCase,     // TestCase
  messages,     // UIMessage[]
  config        // Optional config
);

console.log(`Score: ${score}/100`);
```

### Batch evaluation

Run multiple evaluations on a test suite:

```typescript theme={null}
async function runEvaluations(
  agent: Agent,
  backend: ExuluAgent,
  testCases: TestCase[],
  evals: ExuluEval[]
) {
  const results = [];

  for (const testCase of testCases) {
    // Generate response
    const response = await backend.generateSync({
      prompt: testCase.inputs[testCase.inputs.length - 1].content,
      agentInstance: await loadAgent(agent.id),
      statistics: { label: "eval", trigger: "test" }
    });

    const messages = [
      ...testCase.inputs,
      { role: "assistant", content: response.text }
    ];

    // Run all evals on this test case
    for (const eval of evals) {
      const score = await eval.run(agent, backend, testCase, messages);

      results.push({
        testCaseId: testCase.id,
        testCaseName: testCase.name,
        evalId: eval.id,
        evalName: eval.name,
        score
      });
    }
  }

  return results;
}
```

## Integration with ExuluQueues

Run evaluations as background jobs:

```typescript theme={null}
import { ExuluEval, ExuluQueues } from "@exulu/backend";

// Create eval with queue config
const eval = new ExuluEval({
  id: "background_eval",
  name: "Background Evaluation",
  description: "Runs as background job",
  llm: true,
  execute: async ({ backend, messages, testCase }) => {
    // Evaluation logic
    return 90;
  },
  queue: Promise.resolve({
    connection: await ExuluQueues.getConnection(),
    name: "evaluations",
    prefix: "{exulu}",
    defaultJobOptions: {
      attempts: 3,
      backoff: { type: "exponential", delay: 2000 }
    }
  })
});

// Queue the evaluation job
// (Implementation depends on your worker setup)
```

## Best practices

<Tip>
  **Start simple**: Begin with basic evaluations (exact match, keyword presence) before building complex LLM-as-judge evaluations.
</Tip>

<Note>
  **Multiple evaluations**: Use multiple evaluation functions to assess different aspects (accuracy, tone, tool usage, etc.).
</Note>

<Warning>
  **Score range**: Evaluation functions must return a score between 0 and 100. Scores outside this range will throw an error.
</Warning>

<Info>
  **Test case quality**: Good test cases are specific, representative of real usage, and have clear expected outputs.
</Info>

## When to use ExuluEval

<AccordionGroup>
  <Accordion title="Agent development">
    Test agent behavior during development to catch issues early
  </Accordion>

  <Accordion title="Prompt engineering">
    Compare prompt variations to find the best performing instructions
  </Accordion>

  <Accordion title="Model comparison">
    Evaluate the same agent with different LLM models (GPT-4 vs Claude vs Gemini)
  </Accordion>

  <Accordion title="CI/CD pipelines">
    Automated testing in deployment pipelines to prevent regressions
  </Accordion>

  <Accordion title="Quality monitoring">
    Continuous evaluation in production to track performance over time
  </Accordion>
</AccordionGroup>

## Evaluation workflow

```mermaid theme={null}
graph TD
    A[Create Test Cases] --> B[Define Evaluation Functions]
    B --> C[Generate Agent Response]
    C --> D[Run Evaluation]
    D --> E{Score >= Threshold?}
    E -->|Yes| F[Pass]
    E -->|No| G[Fail]
    F --> H[Deploy / Continue]
    G --> I[Fix Agent]
    I --> C
```

## Next steps

<CardGroup cols={2}>
  <Card title="Configuration" icon="gear" href="/core/exulu-eval/configuration">
    Learn about evaluation configuration
  </Card>

  <Card title="API reference" icon="code" href="/core/exulu-eval/api-reference">
    Explore methods and properties
  </Card>

  <Card title="ExuluAgent" icon="robot" href="/core/exulu-agent/introduction">
    Create agents to evaluate
  </Card>

  <Card title="ExuluQueues" icon="layer-group" href="/core/exulu-queues/introduction">
    Run evaluations as background jobs
  </Card>
</CardGroup>