> ## Documentation Index
> Fetch the complete documentation index at: https://docs.exulu.com/llms.txt
> Use this file to discover all available pages before exploring further.

# API reference

> Complete method and property reference for ExuluChunkers

## ExuluChunkers namespace

ExuluChunkers is exported as a namespace object:

```typescript theme={null}
import { ExuluChunkers } from "@exulu/backend";

// Access sentence chunker
const sentenceChunker = await ExuluChunkers.sentence.create({...});

// Access recursive chunker
const recursiveChunker = await ExuluChunkers.recursive.function.create({...});

// Access recursive rules
const rules = new ExuluChunkers.recursive.rules({...});
```

## SentenceChunker

### create()

Factory method to create a new SentenceChunker instance.

```typescript theme={null}
static async create(options: SentenceChunkerOptions): Promise<CallableSentenceChunker>
```

<ParamField path="options" type="SentenceChunkerOptions" required>
  Configuration options for the chunker
</ParamField>

<ParamField path="options.chunkSize" type="number" required>
  Maximum number of tokens per chunk
</ParamField>

<ParamField path="options.chunkOverlap" type="number" default={0}>
  Number of tokens to overlap between chunks (default: 0)
</ParamField>

<ParamField path="options.minSentencesPerChunk" type="number" default={1}>
  Minimum sentences per chunk (default: 1)
</ParamField>

<ParamField path="options.minCharactersPerSentence" type="number" default={10}>
  Minimum character length for a sentence (default: 10)
</ParamField>

<ResponseField name="return" type="Promise<CallableSentenceChunker>">
  A callable chunker function that can be invoked with text
</ResponseField>

```typescript theme={null}
import { ExuluChunkers } from "@exulu/backend";

// Create chunker
const chunker = await ExuluChunkers.sentence.create({
  chunkSize: 512,
  chunkOverlap: 50,
  minSentencesPerChunk: 2,
  minCharactersPerSentence: 15
});

// Use chunker
const text = "Your document text here...";
const chunks = await chunker(text);

console.log(chunks.length);        // Number of chunks
console.log(chunks[0].text);       // First chunk text
console.log(chunks[0].tokenCount); // Token count
```

### CallableSentenceChunker

The chunker returned by `create()` is a callable function:

```typescript theme={null}
async (text: string): Promise<Chunk[]>
```

<ParamField path="text" type="string" required>
  The text to chunk
</ParamField>

<ResponseField name="return" type="Promise<Chunk[]>">
  Array of Chunk objects
</ResponseField>

```typescript theme={null}
const chunks = await chunker("Long text to chunk...");

for (const chunk of chunks) {
  console.log(chunk.text);
  console.log(chunk.tokenCount);
  console.log(chunk.startIndex, chunk.endIndex);
}
```

### Properties

The callable chunker also has properties from the SentenceChunker class:

<ResponseField name="chunkSize" type="number">
  Maximum tokens per chunk
</ResponseField>

<ResponseField name="chunkOverlap" type="number">
  Overlap in tokens
</ResponseField>

<ResponseField name="minSentencesPerChunk" type="number">
  Minimum sentences per chunk
</ResponseField>

<ResponseField name="minCharactersPerSentence" type="number">
  Minimum characters per sentence
</ResponseField>

<ResponseField name="tokenizer" type="ExuluTokenizer">
  The tokenizer instance used for counting tokens
</ResponseField>

```typescript theme={null}
console.log(chunker.chunkSize);         // 512
console.log(chunker.chunkOverlap);      // 50
console.log(chunker.minSentencesPerChunk); // 2
```

## RecursiveChunker

### create()

Factory method to create a new RecursiveChunker instance.

```typescript theme={null}
static async create(options: RecursiveChunkerOptions): Promise<CallableRecursiveChunker>
```

<ParamField path="options" type="RecursiveChunkerOptions" required>
  Configuration options for the chunker
</ParamField>

<ParamField path="options.chunkSize" type="number" required>
  Maximum number of tokens per chunk
</ParamField>

<ParamField path="options.rules" type="RecursiveRules" default="default rules">
  Recursive splitting rules (default: paragraphs → sentences → pauses → words → tokens)
</ParamField>

<ParamField path="options.minCharactersPerChunk" type="number" default={50}>
  Minimum character length for a chunk (default: 50)
</ParamField>

<ResponseField name="return" type="Promise<CallableRecursiveChunker>">
  A callable chunker function that can be invoked with text
</ResponseField>

```typescript theme={null}
import { ExuluChunkers } from "@exulu/backend";

// Create with default rules
const chunker = await ExuluChunkers.recursive.function.create({
  chunkSize: 1024,
  minCharactersPerChunk: 75
});

// Or with custom rules
const rules = new ExuluChunkers.recursive.rules({
  levels: [
    { delimiters: ["\n\n"] },
    { delimiters: [". "] },
    { whitespace: true }
  ]
});

const customChunker = await ExuluChunkers.recursive.function.create({
  chunkSize: 1024,
  rules: rules,
  minCharactersPerChunk: 50
});
```

### CallableRecursiveChunker

The chunker returned by `create()` is a callable function:

```typescript theme={null}
async (text: string): Promise<RecursiveChunk[]>
```

<ParamField path="text" type="string" required>
  The text to chunk
</ParamField>

<ResponseField name="return" type="Promise<RecursiveChunk[]>">
  Array of RecursiveChunk objects
</ResponseField>

```typescript theme={null}
const chunks = await chunker("Long text to chunk...");

for (const chunk of chunks) {
  console.log(`Level ${chunk.level}: ${chunk.text}`);
  console.log(`Tokens: ${chunk.tokenCount}`);
  console.log(`Range: ${chunk.startIndex}-${chunk.endIndex}`);
}
```

### Properties

The callable chunker also has properties from the RecursiveChunker class:

<ResponseField name="chunkSize" type="number">
  Maximum tokens per chunk
</ResponseField>

<ResponseField name="rules" type="RecursiveRules">
  The recursive splitting rules
</ResponseField>

<ResponseField name="minCharactersPerChunk" type="number">
  Minimum characters per chunk
</ResponseField>

<ResponseField name="tokenizer" type="ExuluTokenizer">
  The tokenizer instance used for counting tokens
</ResponseField>

```typescript theme={null}
console.log(chunker.chunkSize);              // 1024
console.log(chunker.minCharactersPerChunk);  // 75
console.log(chunker.rules.length);           // Number of levels
```

## RecursiveRules

Class representing recursive chunking rules.

### Constructor

```typescript theme={null}
new RecursiveRules(data?: RecursiveRulesData)
```

<ParamField path="data" type="RecursiveRulesData">
  Configuration for recursive rules
</ParamField>

<ParamField path="data.levels" type="RecursiveLevelData[]">
  Array of recursive levels defining the splitting hierarchy
</ParamField>

```typescript theme={null}
import { ExuluChunkers } from "@exulu/backend";

// Create with default levels
const defaultRules = new ExuluChunkers.recursive.rules();

// Create with custom levels
const customRules = new ExuluChunkers.recursive.rules({
  levels: [
    { delimiters: ["\n\n", "\n"] },
    { delimiters: [". ", "! ", "? "] },
    { whitespace: true }
  ]
});
```

**Default levels:**

1. Paragraphs: `["\n\n", "\r\n", "\n", "\r"]`
2. Sentences: `[". ", "! ", "? "]`
3. Pauses: `["{", "}", '"', "[", "]", "<", ">", "(", ")", ":", ";", ",", "—", "|", "~", "-", "...", "`", "'"]\`
4. Words: `whitespace: true`
5. Tokens: No delimiters

### Properties

<ResponseField name="levels" type="RecursiveLevel[]">
  Array of recursive levels
</ResponseField>

<ResponseField name="length" type="number" getter>
  Number of levels in the rules
</ResponseField>

```typescript theme={null}
const rules = new ExuluChunkers.recursive.rules();

console.log(rules.length);      // 5 (default levels)
console.log(rules.levels[0]);   // First level (paragraphs)
```

### Methods

#### getLevel()

Get a level by index.

```typescript theme={null}
getLevel(index: number): RecursiveLevel | undefined
```

<ParamField path="index" type="number" required>
  The index of the level to retrieve
</ParamField>

<ResponseField name="return" type="RecursiveLevel | undefined">
  The level at the specified index, or undefined if not found
</ResponseField>

```typescript theme={null}
const rules = new ExuluChunkers.recursive.rules();

const firstLevel = rules.getLevel(0);   // Paragraphs level
const secondLevel = rules.getLevel(1);  // Sentences level
const invalid = rules.getLevel(999);    // undefined
```

#### toDict()

Convert rules to a dictionary-like object.

```typescript theme={null}
toDict(): RecursiveRulesData
```

<ResponseField name="return" type="RecursiveRulesData">
  Dictionary representation of the rules
</ResponseField>

```typescript theme={null}
const rules = new ExuluChunkers.recursive.rules({
  levels: [
    { delimiters: ["\n\n"] },
    { whitespace: true }
  ]
});

const dict = rules.toDict();
console.log(dict);
// { levels: [{ delimiters: ["\n\n"], whitespace: false, includeDelim: "prev" }, ...] }
```

#### fromDict()

Create RecursiveRules from a dictionary.

```typescript theme={null}
static fromDict(data: RecursiveRulesData): RecursiveRules
```

<ParamField path="data" type="RecursiveRulesData" required>
  Dictionary representation of rules
</ParamField>

<ResponseField name="return" type="RecursiveRules">
  New RecursiveRules instance
</ResponseField>

```typescript theme={null}
const data = {
  levels: [
    { delimiters: ["\n\n"] },
    { whitespace: true }
  ]
};

const rules = ExuluChunkers.recursive.rules.fromDict(data);
```

#### toString()

String representation of the rules.

```typescript theme={null}
toString(): string
```

<ResponseField name="return" type="string">
  String representation
</ResponseField>

```typescript theme={null}
const rules = new ExuluChunkers.recursive.rules();
console.log(rules.toString());
// "RecursiveRules(levels=[...])"
```

#### Symbol.iterator

The rules object is iterable:

```typescript theme={null}
for (const level of rules) {
  console.log(level.delimiters);
  console.log(level.whitespace);
}
```

## RecursiveLevel

Class representing a single level in the recursive hierarchy.

### Constructor

```typescript theme={null}
new RecursiveLevel(data?: RecursiveLevelData)
```

<ParamField path="data" type="RecursiveLevelData">
  Configuration for the level
</ParamField>

<ParamField path="data.delimiters" type="string | string[]">
  Delimiter(s) to use for splitting at this level
</ParamField>

<ParamField path="data.whitespace" type="boolean" default={false}>
  Whether to split on whitespace (default: false)
</ParamField>

<ParamField path="data.includeDelim" type="'prev' | 'next'" default="prev">
  Whether to include delimiter in previous or next chunk (default: "prev")
</ParamField>

```typescript theme={null}
// Single delimiter
const level1 = new RecursiveLevel({
  delimiters: "\n\n"
});

// Multiple delimiters
const level2 = new RecursiveLevel({
  delimiters: [". ", "! ", "? "],
  includeDelim: "prev"
});

// Whitespace splitting
const level3 = new RecursiveLevel({
  whitespace: true
});

// No delimiters (token-level fallback)
const level4 = new RecursiveLevel();
```

<Warning>
  Cannot use both `delimiters` and `whitespace` in the same level. They are mutually exclusive.
</Warning>

### Properties

<ResponseField name="delimiters" type="string | string[] | undefined">
  Custom delimiters for chunking
</ResponseField>

<ResponseField name="whitespace" type="boolean">
  Whether to use whitespace as a delimiter
</ResponseField>

<ResponseField name="includeDelim" type="'prev' | 'next'">
  Where to include the delimiter
</ResponseField>

```typescript theme={null}
const level = new RecursiveLevel({
  delimiters: [". ", "! ", "? "],
  includeDelim: "prev"
});

console.log(level.delimiters);    // [". ", "! ", "? "]
console.log(level.whitespace);    // false
console.log(level.includeDelim);  // "prev"
```

### Methods

#### toDict()

Convert level to dictionary.

```typescript theme={null}
toDict(): RecursiveLevelData
```

<ResponseField name="return" type="RecursiveLevelData">
  Dictionary representation
</ResponseField>

```typescript theme={null}
const level = new RecursiveLevel({ delimiters: [". "] });
const dict = level.toDict();
console.log(dict);
// { delimiters: [". "], whitespace: false, includeDelim: "prev" }
```

#### fromDict()

Create RecursiveLevel from dictionary.

```typescript theme={null}
static fromDict(data: RecursiveLevelData): RecursiveLevel
```

<ParamField path="data" type="RecursiveLevelData" required>
  Dictionary representation
</ParamField>

<ResponseField name="return" type="RecursiveLevel">
  New RecursiveLevel instance
</ResponseField>

```typescript theme={null}
const data = { delimiters: [". "], includeDelim: "next" };
const level = RecursiveLevel.fromDict(data);
```

#### toString()

String representation of the level.

```typescript theme={null}
toString(): string
```

<ResponseField name="return" type="string">
  String representation
</ResponseField>

```typescript theme={null}
const level = new RecursiveLevel({ delimiters: [". "] });
console.log(level.toString());
// "RecursiveLevel(delimiters=["."], whitespace=false, includeDelim=prev)"
```

## Chunk

Base class for text chunks.

### Properties

<ResponseField name="text" type="string">
  The chunk text
</ResponseField>

<ResponseField name="startIndex" type="number">
  Starting index in the original text
</ResponseField>

<ResponseField name="endIndex" type="number">
  Ending index in the original text
</ResponseField>

<ResponseField name="tokenCount" type="number">
  Number of tokens in the chunk
</ResponseField>

<ResponseField name="embedding" type="number[] | undefined">
  Optional embedding vector for the chunk
</ResponseField>

```typescript theme={null}
const chunk = chunks[0];

console.log(chunk.text);        // "This is the first chunk..."
console.log(chunk.startIndex);  // 0
console.log(chunk.endIndex);    // 245
console.log(chunk.tokenCount);  // 48
console.log(chunk.embedding);   // undefined (or embedding array)
```

### Methods

#### toString()

String representation of the chunk (returns the text).

```typescript theme={null}
toString(): string
```

<ResponseField name="return" type="string">
  The chunk text
</ResponseField>

```typescript theme={null}
console.log(chunk.toString()); // "This is the first chunk..."
```

#### toRepresentation()

Detailed string representation.

```typescript theme={null}
toRepresentation(): string
```

<ResponseField name="return" type="string">
  Detailed representation
</ResponseField>

```typescript theme={null}
console.log(chunk.toRepresentation());
// "Chunk(text='...', tokenCount=48, startIndex=0, endIndex=245)"
```

#### slice()

Get a slice of the chunk's text.

```typescript theme={null}
slice(start?: number, end?: number): string
```

<ParamField path="start" type="number">
  Starting index for the slice
</ParamField>

<ParamField path="end" type="number">
  Ending index for the slice
</ParamField>

<ResponseField name="return" type="string">
  Sliced text
</ResponseField>

```typescript theme={null}
const chunk = chunks[0];
console.log(chunk.slice(0, 50)); // First 50 characters
```

#### toDict()

Convert chunk to dictionary.

```typescript theme={null}
toDict(): ChunkData
```

<ResponseField name="return" type="ChunkData">
  Dictionary representation
</ResponseField>

```typescript theme={null}
const dict = chunk.toDict();
console.log(dict);
// { text: "...", startIndex: 0, endIndex: 245, tokenCount: 48, embedding: undefined }
```

#### fromDict()

Create Chunk from dictionary.

```typescript theme={null}
static fromDict(data: ChunkData): Chunk
```

<ParamField path="data" type="ChunkData" required>
  Dictionary representation
</ParamField>

<ResponseField name="return" type="Chunk">
  New Chunk instance
</ResponseField>

```typescript theme={null}
const data = {
  text: "Sample text",
  startIndex: 0,
  endIndex: 11,
  tokenCount: 3
};

const chunk = Chunk.fromDict(data);
```

#### copy()

Create a deep copy of the chunk.

```typescript theme={null}
copy(): Chunk
```

<ResponseField name="return" type="Chunk">
  Deep copy of the chunk
</ResponseField>

```typescript theme={null}
const original = chunks[0];
const copy = original.copy();

console.log(copy.text === original.text); // true
console.log(copy === original);           // false (different objects)
```

## RecursiveChunk

Extends `Chunk` with recursion level tracking.

### Properties

All properties from `Chunk`, plus:

<ResponseField name="level" type="number | undefined">
  The recursion level at which this chunk was created
</ResponseField>

```typescript theme={null}
const chunk = chunks[0];

console.log(chunk.text);        // "This is the first chunk..."
console.log(chunk.tokenCount);  // 48
console.log(chunk.level);       // 0 (split at top level)
```

**Level interpretation:**

* `0`: Split at first level (e.g., paragraphs)
* `1`: Split at second level (e.g., sentences)
* `2`: Split at third level (e.g., pauses)
* etc.

### Methods

All methods from `Chunk`, with overridden implementations that preserve the `level` property.

## Usage examples

### Basic sentence chunking

```typescript theme={null}
import { ExuluChunkers } from "@exulu/backend";

const chunker = await ExuluChunkers.sentence.create({
  chunkSize: 512,
  chunkOverlap: 50
});

const text = `
  Artificial intelligence is transforming industries worldwide.
  Machine learning enables computers to learn from data without
  explicit programming. Deep learning uses neural networks to
  recognize complex patterns in images, text, and audio.

  The field continues to evolve rapidly. New techniques emerge
  regularly, pushing the boundaries of what's possible.
`;

const chunks = await chunker(text);

console.log(`Created ${chunks.length} chunks`);

for (const [i, chunk] of chunks.entries()) {
  console.log(`\nChunk ${i + 1}:`);
  console.log(`  Text: ${chunk.text.slice(0, 50)}...`);
  console.log(`  Tokens: ${chunk.tokenCount}`);
  console.log(`  Range: ${chunk.startIndex}-${chunk.endIndex}`);
}
```

### Recursive chunking with custom rules

```typescript theme={null}
import { ExuluChunkers } from "@exulu/backend";

// Define custom rules for markdown
const rules = new ExuluChunkers.recursive.rules({
  levels: [
    // Split by headers (keep header with content)
    {
      delimiters: ["\n## ", "\n### "],
      includeDelim: "next"
    },
    // Split by paragraphs
    { delimiters: ["\n\n"] },
    // Split by sentences
    { delimiters: [". ", "! ", "? "] },
    // Split by words
    { whitespace: true }
  ]
});

const chunker = await ExuluChunkers.recursive.function.create({
  chunkSize: 1024,
  rules: rules,
  minCharactersPerChunk: 75
});

const markdown = `
## Introduction

Machine learning is a subset of artificial intelligence.
It enables systems to learn and improve from experience.

## Applications

Recommendation systems use ML to personalize content.
Fraud detection systems identify suspicious patterns.
Autonomous vehicles use ML for navigation and decision-making.

## Future Directions

The field continues to advance rapidly.
New architectures and techniques emerge regularly.
`;

const chunks = await chunker(markdown);

console.log(`Created ${chunks.length} chunks`);

for (const [i, chunk] of chunks.entries()) {
  console.log(`\nChunk ${i + 1} (level ${chunk.level}):`);
  console.log(`  Text: ${chunk.text}`);
  console.log(`  Tokens: ${chunk.tokenCount}`);
}
```

### Analyzing chunk statistics

```typescript theme={null}
const chunker = await ExuluChunkers.sentence.create({
  chunkSize: 512,
  chunkOverlap: 50
});

const text = "Your long document...";
const chunks = await chunker(text);

// Calculate statistics
const tokenCounts = chunks.map(c => c.tokenCount);
const avgTokens = tokenCounts.reduce((a, b) => a + b, 0) / chunks.length;
const maxTokens = Math.max(...tokenCounts);
const minTokens = Math.min(...tokenCounts);

console.log(`Chunks: ${chunks.length}`);
console.log(`Avg tokens: ${avgTokens.toFixed(2)}`);
console.log(`Max tokens: ${maxTokens}`);
console.log(`Min tokens: ${minTokens}`);
console.log(`Total tokens: ${tokenCounts.reduce((a, b) => a + b, 0)}`);

// Distribution
const histogram = {};
for (const chunk of chunks) {
  const bucket = Math.floor(chunk.tokenCount / 100) * 100;
  histogram[bucket] = (histogram[bucket] || 0) + 1;
}

console.log("\nToken distribution:");
for (const [bucket, count] of Object.entries(histogram)) {
  console.log(`  ${bucket}-${parseInt(bucket) + 99}: ${'*'.repeat(count)}`);
}
```

### Inspecting level distribution (recursive)

```typescript theme={null}
const chunker = await ExuluChunkers.recursive.function.create({
  chunkSize: 1024
});

const text = "Your document...";
const chunks = await chunker(text);

// Count chunks by level
const levelCounts = {};
for (const chunk of chunks) {
  levelCounts[chunk.level || 0] = (levelCounts[chunk.level || 0] || 0) + 1;
}

console.log("Chunk distribution by level:");
for (const [level, count] of Object.entries(levelCounts)) {
  const levelName = ["Paragraphs", "Sentences", "Pauses", "Words", "Tokens"][level];
  console.log(`  Level ${level} (${levelName}): ${count} chunks`);
}
```

### Using with ExuluContext

```typescript theme={null}
import { ExuluContext, ExuluChunkers, ExuluEmbedder } from "@exulu/backend";

// Create chunker
const chunker = await ExuluChunkers.sentence.create({
  chunkSize: 512,
  chunkOverlap: 75
});

// Create embedder
const embedder = new ExuluEmbedder({
  id: "openai_embedder",
  name: "OpenAI Embeddings",
  provider: "openai",
  model: "text-embedding-3-small",
  vectorDimensions: 1536
});

// Create context with chunker
const context = new ExuluContext({
  id: "documentation",
  name: "Product Documentation",
  description: "Searchable product documentation",
  embedder: embedder,
  chunker: chunker, // Documents will be chunked automatically
  fields: [
    { name: "title", type: "text", required: true },
    { name: "content", type: "longtext", required: true },
    { name: "url", type: "text", required: false }
  ],
  sources: []
});

// Add document - it's automatically chunked and embedded
await context.createItem(
  {
    title: "Getting Started Guide",
    content: "Very long documentation content...",
    url: "https://example.com/docs/getting-started"
  },
  { generateEmbeddings: true }
);

// Search returns relevant chunks
const results = await context.search({
  query: "How do I install?",
  limit: 5
});

for (const result of results) {
  console.log(`Score: ${result.score}`);
  console.log(`Chunk: ${result.chunk.text.slice(0, 100)}...`);
}
```

## Type definitions

```typescript theme={null}
// Sentence chunker options
interface SentenceChunkerOptions {
  chunkSize: number;
  chunkOverlap?: number;
  minSentencesPerChunk?: number;
  minCharactersPerSentence?: number;
}

// Recursive chunker options
interface RecursiveChunkerOptions {
  chunkSize: number;
  rules?: RecursiveRules;
  minCharactersPerChunk?: number;
}

// Recursive rules data
interface RecursiveRulesData {
  levels?: RecursiveLevelData[];
}

// Recursive level data
interface RecursiveLevelData {
  delimiters?: string | string[];
  whitespace?: boolean;
  includeDelim?: "prev" | "next";
}

// Chunk data
interface ChunkData {
  text: string;
  startIndex: number;
  endIndex: number;
  tokenCount: number;
  embedding?: number[];
}

// Recursive chunk data
interface RecursiveChunkData extends ChunkData {
  level?: number;
}
```

## Best practices

<Tip>
  **Use appropriate chunk size**: Match your embedding model's token limit. Leave 10-20% headroom for metadata.
</Tip>

<Note>
  **Enable overlap for natural language**: Use 10-20% overlap to preserve context at chunk boundaries.
</Note>

<Warning>
  **Monitor chunk count**: More chunks = higher embedding costs. Balance granularity with cost.
</Warning>

<Info>
  **Choose the right chunker**: SentenceChunker for most text, RecursiveChunker for structured documents.
</Info>

## Next steps

<CardGroup cols={2}>
  <Card title="Configuration guide" icon="gear" href="/core/exulu-chunkers/configuration">
    Learn about configuration options
  </Card>

  <Card title="Overview" icon="book" href="/core/exulu-chunkers/introduction">
    Understand chunking concepts
  </Card>
</CardGroup>