Skip to main content

Overview

ExuluAgent represents an AI agent that processes requests using large language models (LLMs). Agents can execute tasks, call tools, maintain conversations, and generate structured outputs. They serve as the primary interface between users and your AI application.

Key features

Multi-provider support

Built-in support for OpenAI, Anthropic, Google, Cerebras, and custom providers

Tool calling

Agents can use tools to query databases, call APIs, and perform actions

Streaming responses

Real-time streaming for progressive response generation

Memory integration

Optional context-based memory for persistent knowledge

Multi-modal support

Process text, images, documents, audio, and video

Structured outputs

Generate validated JSON outputs with Zod schemas

What is an agent?

An agent is an LLM-powered entity that:
  1. Receives input from users or other agents (text, images, files, etc.)
  2. Processes the request using an LLM with custom instructions
  3. Uses tools to query information or perform actions
  4. Generates responses as text or structured data
  5. Maintains conversation history across multiple interactions
Think of agents as intelligent assistants that can understand context, make decisions, and take actions on behalf of users.

Quick start

import { ExuluAgent } from "@exulu/backend";
import { createOpenAI } from "@ai-sdk/openai";

// Create an agent
const assistantAgent = new ExuluAgent({
  id: "assistant",
  name: "Assistant",
  type: "agent",
  description: "A helpful AI assistant",
  provider: "openai",
  config: {
    name: "gpt-4o",
    model: {
      create: ({ apiKey }) => createOpenAI({ apiKey })("gpt-4o")
    },
    instructions: "You are a helpful assistant. Answer questions clearly and concisely."
  },
  capabilities: {
    text: true,
    images: [".png", ".jpg", ".jpeg"],
    files: [".pdf", ".docx", ".txt"],
    audio: [],
    video: []
  }
});

// Register with ExuluApp
const app = new ExuluApp();
await app.create({
  agents: {
    assistant: assistantAgent
  },
  config: { /* ... */ }
});

// Use the agent
const response = await assistantAgent.generateSync({
  prompt: "What is the capital of France?",
  agentInstance: await loadAgent("assistant"),
  currentTools: [],
  statistics: {
    label: "assistant",
    trigger: "api"
  }
});

console.log(response); // "The capital of France is Paris."

Agent types

Exulu IMP includes pre-built agents for major providers:
import { ExuluDefaultAgents } from "@exulu/backend";

const gpt4Agent = ExuluDefaultAgents.GPT_4O;
const gpt5Agent = ExuluDefaultAgents.GPT_5;
const o1Agent = ExuluDefaultAgents.O1;
Available models: GPT-4o, GPT-4o-mini, GPT-4.1, GPT-5, O1, O1-mini, O3-mini

Architecture

AI SDK integration

ExuluAgent is built on Vercel’s AI SDK, providing:
  • Unified interface across multiple LLM providers
  • Streaming support for real-time responses
  • Tool calling with automatic function execution
  • Structured outputs with Zod schema validation
import { streamText, generateText, generateObject } from "ai";

// ExuluAgent uses these internally
const response = await generateText({
  model: agent.model.create({ apiKey }),
  prompt: "Hello!",
  tools: convertedTools
});

Conversation management

Agents maintain conversation history through sessions:
// Messages are stored in the database
const session = "session-123";
const previousMessages = await getAgentMessages({ session, user: userId });

// New messages are appended
const response = await agent.generateStream({
  message: newMessage,
  session: session,
  user: currentUser
});

Tool integration

Agents can use any ExuluTool:
const agent = new ExuluAgent({
  // ... agent config
});

// Tools are provided at runtime
const response = await agent.generateSync({
  prompt: "Search our docs for authentication",
  currentTools: [
    documentationSearchTool,
    databaseQueryTool,
    apiCallTool
  ]
});

Core concepts

Instructions

Instructions define the agent’s behavior and personality:
config: {
  instructions: `You are a customer support agent for Acme Corp.

Guidelines:
- Be friendly and professional
- Answer questions about our products and services
- Escalate billing issues to human agents
- Provide accurate, helpful information

Company info:
- Founded in 2020
- Specializes in enterprise SaaS
- 24/7 support available`
}
Keep instructions clear and specific. Well-defined instructions significantly improve agent performance.

Capabilities

Define what input types the agent can handle:
capabilities: {
  text: true,                          // Text input
  images: [".png", ".jpg", ".jpeg"],   // Image files
  files: [".pdf", ".docx", ".txt"],    // Document files
  audio: [".mp3", ".wav"],             // Audio files
  video: [".mp4"]                      // Video files
}

Memory

Agents can have persistent memory using ExuluContext:
const agent = new ExuluAgent({
  id: "assistant_with_memory",
  name: "Assistant with Memory",
  // ... other config
  config: {
    // ... model config
    memory: "user_interactions" // Context ID for memory
  }
});
When configured, the agent:
  1. Searches relevant memories before responding
  2. Can create new memory items during conversations
  3. Maintains context across sessions

Workflows

Enable background task processing:
workflows: {
  enabled: true,
  queue: await ExuluQueues.register("agent_workflows")
}

Generation modes

Real-time progressive responses:
const { stream } = await agent.generateStream({
  message: userMessage,
  session: sessionId,
  user: currentUser,
  currentTools: tools,
  agentInstance: agentData
});

// Stream to client
for await (const chunk of stream) {
  console.log(chunk);
}
Best for: Interactive chat, long responses, real-time feedback

Usage patterns

As a conversational assistant

const chatAgent = new ExuluAgent({
  id: "chat_assistant",
  name: "Chat Assistant",
  type: "agent",
  description: "Conversational AI assistant",
  provider: "openai",
  config: {
    name: "gpt-4o",
    model: {
      create: ({ apiKey }) => createOpenAI({ apiKey })("gpt-4o")
    },
    instructions: "You are a friendly, helpful assistant."
  },
  capabilities: {
    text: true,
    images: [".png", ".jpg"],
    files: [".pdf", ".txt"],
    audio: [],
    video: []
  }
});

// Stream responses
const { stream } = await chatAgent.generateStream({
  message: newMessage,
  session: sessionId,
  user: currentUser,
  currentTools: [searchTool, calculatorTool]
});

As a data processor

const extractorAgent = new ExuluAgent({
  id: "data_extractor",
  name: "Data Extractor",
  type: "agent",
  description: "Extracts structured data from text",
  provider: "openai",
  config: {
    name: "gpt-4o-mini",
    model: {
      create: ({ apiKey }) => createOpenAI({ apiKey })("gpt-4o-mini")
    }
  },
  capabilities: {
    text: true,
    images: [],
    files: [".pdf", ".docx", ".txt"],
    audio: [],
    video: []
  }
});

// Generate structured output
const data = await extractorAgent.generateSync({
  prompt: "Extract contact information from this email",
  outputSchema: z.object({
    name: z.string(),
    email: z.string().email(),
    phone: z.string().optional()
  })
});

As a tool for other agents

Agents can call other agents as tools:
const specialistAgent = new ExuluAgent({
  id: "sql_specialist",
  name: "SQL Specialist",
  type: "agent",
  description: "Expert at writing SQL queries",
  provider: "openai",
  config: {
    name: "gpt-4o",
    model: { /* ... */ },
    instructions: "You are an expert SQL developer. Write optimized, secure queries."
  },
  capabilities: {
    text: true,
    images: [],
    files: [],
    audio: [],
    video: []
  }
});

// Export as tool
const sqlTool = await specialistAgent.tool(
  "sql_specialist",
  [specialistAgent],
  contexts,
  rerankers
);

// Main agent can use it
const mainAgent = new ExuluAgent({
  id: "main_assistant",
  name: "Main Assistant",
  type: "agent",
  description: "General assistant that can delegate to specialists",
  // ... config
});

const response = await mainAgent.generateSync({
  prompt: "Write a query to find top customers",
  currentTools: [sqlTool], // Main agent can call SQL specialist
  // ... other params
});

Multi-modal capabilities

All agents support text input by default:
capabilities: {
  text: true
}
Agents can process images (PNG, JPG, JPEG, GIF, WebP):
capabilities: {
  text: true,
  images: [".png", ".jpg", ".jpeg", ".gif", ".webp"]
}
Images are passed directly to models that support vision (GPT-4o, Claude Opus, Gemini Pro).
Documents are converted to text before processing:
capabilities: {
  text: true,
  files: [".pdf", ".docx", ".xlsx", ".txt", ".md"]
}
ExuluAgent uses officeparser to extract text from documents.
Audio files can be processed if the model supports it:
capabilities: {
  text: true,
  audio: [".mp3", ".wav", ".m4a"]
}
Video files for models with video understanding:
capabilities: {
  text: true,
  video: [".mp4"]
}

Best practices

Clear instructions: Write detailed, specific instructions. Include examples of desired behavior and edge cases to handle.
Tool selection: Provide only relevant tools to each agent. Too many tools can confuse the agent and increase latency.
API key security: Always use encrypted variables for API keys. Never hardcode credentials in agent configurations.
Context length: Monitor token usage and set appropriate maxContextLength to prevent exceeding model limits.

Next steps