ExuluOtel - Exulu IMP

Overview

ExuluOtel provides built-in OpenTelemetry (OTEL) support for distributed tracing and logging across your Exulu application. It automatically instruments your application to collect traces and logs, helping you monitor performance, debug issues, and understand system behavior in production.

Key features

Distributed tracing

Track requests across services and components

Structured logging

Correlate logs with traces for better debugging

Auto-instrumentation

Automatic instrumentation of Node.js modules

SigNoz integration

Built-in support for SigNoz (cloud and self-hosted)

Component control

Enable/disable telemetry per component

Graceful shutdown

Ensures traces are flushed before exit

What is OpenTelemetry?

OpenTelemetry (OTEL) is an open-source observability framework for collecting traces, metrics, and logs from your applications. It helps you:

Monitor performance: Identify slow operations and bottlenecks
Debug issues: Trace requests through your entire system
Understand behavior: See how components interact
Optimize resources: Find inefficient code paths
Alert on problems: Set up monitoring and alerting

Quick start

Prerequisites

ExuluOtel currently supports SigNoz as the observability backend. You can use:

SigNoz Cloud (hosted solution)
Self-hosted SigNoz (open-source)

Environment variables

Set up your SigNoz credentials:

# .env file
SIGNOZ_TRACES_URL=https://ingest.{region}.signoz.cloud:443/v1/traces
SIGNOZ_LOGS_URL=https://ingest.{region}.signoz.cloud:443/v1/logs
SIGNOZ_ACCESS_TOKEN=your-signoz-access-token

For SigNoz Cloud:

Get your ingestion URL and token from your SigNoz dashboard
Format: https://ingest.{region}.signoz.cloud:443
Regions: us, in, eu

For self-hosted SigNoz:

SIGNOZ_TRACES_URL=http://localhost:4318/v1/traces
SIGNOZ_LOGS_URL=http://localhost:4318/v1/logs
SIGNOZ_ACCESS_TOKEN=  # Optional for self-hosted

Initialize OpenTelemetry

import { ExuluOtel } from "@exulu/backend";

// Create and start OpenTelemetry SDK
const otel = ExuluOtel.create({
  SIGNOZ_TRACES_URL: process.env.SIGNOZ_TRACES_URL!,
  SIGNOZ_LOGS_URL: process.env.SIGNOZ_LOGS_URL!,
  SIGNOZ_ACCESS_TOKEN: process.env.SIGNOZ_ACCESS_TOKEN!
});

// Start tracing
otel.start();

Initialize ExuluOtel before creating your ExuluApp to ensure all operations are traced.

Enable in ExuluApp

Enable telemetry in your application configuration:

import { ExuluApp, ExuluOtel } from "@exulu/backend";

// Initialize OTEL first
const otel = ExuluOtel.create({
  SIGNOZ_TRACES_URL: process.env.SIGNOZ_TRACES_URL!,
  SIGNOZ_LOGS_URL: process.env.SIGNOZ_LOGS_URL!,
  SIGNOZ_ACCESS_TOKEN: process.env.SIGNOZ_ACCESS_TOKEN!
});

otel.start();

// Create app with telemetry enabled
const app = new ExuluApp();

await app.create({
  config: {
    telemetry: {
      enabled: true  // Enable global telemetry
    },
    workers: {
      enabled: true,
      telemetry: {
        enabled: true  // Enable worker telemetry
      }
    }
  },
  contexts: {
    // Your contexts
  },
  agents: {
    // Your agents
  }
});

Configuration

Global telemetry

Enable or disable telemetry globally:

await app.create({
  config: {
    telemetry: {
      enabled: true  // Global switch for all telemetry
    }
  }
});

When enabled:

Express server routes are traced
MCP operations are traced
Logs are correlated with traces

Component-level telemetry

Enable telemetry for specific components:

await app.create({
  config: {
    telemetry: {
      enabled: true  // Global telemetry
    },
    workers: {
      enabled: true,
      telemetry: {
        enabled: true  // Worker-specific telemetry
      }
    }
  }
});

Components:

Workers: BullMQ job processing traces
Express server: HTTP request/response traces
MCP server: Model Context Protocol operation traces

Environment-specific configuration

Different settings for dev/staging/prod:

const isProduction = process.env.NODE_ENV === "production";

await app.create({
  config: {
    telemetry: {
      enabled: isProduction  // Only in production
    },
    workers: {
      enabled: true,
      telemetry: {
        enabled: isProduction
      }
    }
  }
});

What gets traced?

Automatic instrumentation

ExuluOtel uses OpenTelemetry auto-instrumentation to automatically trace:

HTTP requests/responses

Express.js routes
Outgoing HTTP/HTTPS requests
Response status codes and timing
Headers and query parameters

Database operations

PostgreSQL queries
Connection pool usage
Query timing and errors

Redis operations

BullMQ queue operations
Redis commands
Connection management

File system operations

File reads/writes
Directory operations
S3 storage interactions

LLM API calls

OpenAI API requests
Anthropic API requests
Other provider API calls
Token usage and latency

Custom traces

Components automatically create custom traces for:

Agent operations: generateSync(), generateStream()
Context operations: search(), createItem(), updateItem()
Tool executions: Function tool calls
Worker jobs: Background job processing
Embeddings generation: Batch and single embeddings

Viewing traces in SigNoz

After enabling telemetry, visit your SigNoz dashboard to view traces.

Trace structure

HTTP POST /api/agents/run
├─ Agent.generateSync()
│  ├─ Context.search()
│  │  ├─ PostgreSQL: vector search query
│  │  └─ Embedder.generate()
│  │     └─ OpenAI API: create embeddings
│  ├─ Tool.execute()
│  │  └─ External API call
│  └─ OpenAI API: chat completion
└─ Response sent

Key metrics

Duration: How long each operation took
Status: Success, error, or timeout
Attributes: Operation-specific metadata
Events: Important moments in the trace
Logs: Structured logs correlated with spans

Configuration examples

Production setup

import { ExuluApp, ExuluOtel } from "@exulu/backend";

// Initialize OTEL
const otel = ExuluOtel.create({
  SIGNOZ_TRACES_URL: process.env.SIGNOZ_TRACES_URL!,
  SIGNOZ_LOGS_URL: process.env.SIGNOZ_LOGS_URL!,
  SIGNOZ_ACCESS_TOKEN: process.env.SIGNOZ_ACCESS_TOKEN!
});

otel.start();

// Production configuration
const app = new ExuluApp();

await app.create({
  config: {
    telemetry: {
      enabled: true  // Always on in production
    },
    workers: {
      enabled: true,
      telemetry: {
        enabled: true  // Trace background jobs
      }
    },
    MCP: {
      enabled: true
      // Uses global telemetry setting
    }
  },
  contexts: contexts,
  agents: agents
});

Development setup

import { ExuluApp, ExuluOtel } from "@exulu/backend";

// Only enable if env vars are set
const telemetryEnabled = Boolean(
  process.env.SIGNOZ_TRACES_URL &&
  process.env.SIGNOZ_LOGS_URL &&
  process.env.SIGNOZ_ACCESS_TOKEN
);

if (telemetryEnabled) {
  const otel = ExuluOtel.create({
    SIGNOZ_TRACES_URL: process.env.SIGNOZ_TRACES_URL!,
    SIGNOZ_LOGS_URL: process.env.SIGNOZ_LOGS_URL!,
    SIGNOZ_ACCESS_TOKEN: process.env.SIGNOZ_ACCESS_TOKEN!
  });

  otel.start();
}

const app = new ExuluApp();

await app.create({
  config: {
    telemetry: {
      enabled: telemetryEnabled  // Optional in dev
    }
  }
});

Selective component tracing

await app.create({
  config: {
    telemetry: {
      enabled: true  // Global enabled
    },
    workers: {
      enabled: true,
      telemetry: {
        enabled: false  // Disable worker traces (reduce noise)
      }
    }
  }
});

Integration with logging

ExuluOtel automatically correlates structured logs with traces:

import { createLogger } from "@exulu/backend";

// Logger with OTEL integration
const logger = createLogger({
  enableOtel: config?.telemetry?.enabled ?? false
});

// Logs are automatically correlated with active trace
logger.info("Processing request", {
  userId: "user-123",
  operation: "search"
});

// In SigNoz, you can jump from traces to logs and vice versa

Performance considerations

Overhead

OpenTelemetry adds minimal overhead:

CPU: ~1-2% increase
Memory: ~10-20MB for SDK
Network: Batched export reduces impact

Sampling

For high-traffic applications, consider sampling:

// Future: Sampling configuration
// Currently traces all requests when enabled

Batch export

Traces and logs are exported in batches to minimize network overhead:

Traces: Batched every 5 seconds or 512 spans
Logs: Batched every 1 second or 512 records

Troubleshooting

Traces not appearing in SigNoz

Check environment variables

Ensure all required variables are set:

echo $SIGNOZ_TRACES_URL
echo $SIGNOZ_LOGS_URL
echo $SIGNOZ_ACCESS_TOKEN

Verify URLs are correct:

SigNoz Cloud: https://ingest.{region}.signoz.cloud:443
Self-hosted: http://localhost:4318 (or your host)

Verify telemetry is enabled

Check your configuration:

console.log("Telemetry enabled:", config.telemetry?.enabled);
console.log("Worker telemetry:", config.workers?.telemetry?.enabled);

Check network connectivity

Test connection to SigNoz:

curl -X POST https://ingest.{region}.signoz.cloud:443/v1/traces \
  -H "signoz-access-token: your-token" \
  -H "Content-Type: application/json" \
  -d '{}'

Review logs

Check for OTEL initialization:

[EXULU] Setting up OpenTelemetry

Look for errors:

Error terminating tracing: ...

High cardinality warnings

If you see high cardinality warnings in SigNoz:

Avoid using user IDs or request IDs as span names
Use attributes instead of span names for variable data
Limit the number of unique span names

Memory issues

If experiencing memory pressure:

Reduce batch sizes (requires custom configuration)
Enable selective component tracing
Consider sampling for high-traffic routes

Graceful shutdown

ExuluOtel handles graceful shutdown automatically:

// Listens for SIGTERM signal
process.on('SIGTERM', () => {
  sdk.shutdown()
    .then(() => console.log('Tracing terminated'))
    .catch((error) => console.log('Error terminating tracing', error))
    .finally(() => process.exit(0));
});

This ensures:

In-flight traces are flushed
Connections are closed properly
No data is lost on shutdown

Best practices

Initialize early: Start ExuluOtel before creating ExuluApp to capture all traces from initialization.

Use in production: OpenTelemetry is production-ready and adds minimal overhead. Enable it to gain visibility into your application.

Protect credentials: Store SigNoz credentials in environment variables or ExuluVariables, never hardcode them.

Correlate logs: Enable OTEL-aware logging to correlate logs with traces for better debugging.

SigNoz setup

SigNoz Cloud

Sign up at signoz.io
Create a new project
Copy ingestion details from Settings → Ingestion Settings
Set environment variables with your credentials

Self-hosted SigNoz

Install SigNoz using Docker:

git clone https://github.com/SigNoz/signoz.git
cd signoz/deploy
./install.sh

Access SigNoz at http://localhost:3301
Configure environment variables:

SIGNOZ_TRACES_URL=http://localhost:4318/v1/traces
SIGNOZ_LOGS_URL=http://localhost:4318/v1/logs
SIGNOZ_ACCESS_TOKEN=  # Not required for self-hosted

SigNoz installation guide

Future enhancements

Planned features for ExuluOtel:

Support for additional OTEL backends (Jaeger, Zipkin, etc.)
Custom sampling strategies
Metrics collection (in addition to traces and logs)
Advanced span filtering
Custom instrumentation helpers

Example traces

Agent generation trace

Trace: Agent.generateSync
├─ Span: load_agent_config (2ms)
├─ Span: search_contexts (45ms)
│  ├─ Span: generate_embeddings (38ms)
│  │  └─ Span: openai_api_call (35ms)
│  └─ Span: postgres_vector_search (5ms)
├─ Span: execute_tools (120ms)
│  └─ Span: github_api_call (115ms)
└─ Span: llm_generation (850ms)
   └─ Span: openai_chat_completion (845ms)

Total duration: 1,017ms

Context search trace

Trace: Context.search
├─ Span: embedder_generate (42ms)
│  └─ Span: openai_embeddings_api (38ms)
├─ Span: postgres_vector_search (8ms)
└─ Span: reranker_execute (25ms)
   └─ Span: cohere_rerank_api (22ms)

Total duration: 75ms

Worker job trace

Trace: Worker.processJob (queue: embeddings)
├─ Span: fetch_context_item (5ms)
│  └─ Span: postgres_query (3ms)
├─ Span: chunk_text (15ms)
├─ Span: generate_embeddings_batch (250ms)
│  └─ Span: openai_embeddings_api (245ms)
└─ Span: store_embeddings (20ms)
   └─ Span: postgres_bulk_insert (18ms)

Total duration: 290ms

API reference

ExuluOtel.create()

Factory method to create and configure OpenTelemetry SDK.

ExuluOtel.create(options: {
  SIGNOZ_TRACES_URL: string;
  SIGNOZ_LOGS_URL: string;
  SIGNOZ_ACCESS_TOKEN: string;
}): NodeSDK

Parameters:

SIGNOZ_TRACES_URL

string

required

URL for trace export (OTLP/HTTP endpoint)

SIGNOZ_LOGS_URL

string

required

URL for log export (OTLP/HTTP endpoint)

SIGNOZ_ACCESS_TOKEN

string

required

SigNoz access token for authentication

Returns: NodeSDK - OpenTelemetry Node SDK instance Example:

import { ExuluOtel } from "@exulu/backend";

const otel = ExuluOtel.create({
  SIGNOZ_TRACES_URL: "https://ingest.us.signoz.cloud:443/v1/traces",
  SIGNOZ_LOGS_URL: "https://ingest.us.signoz.cloud:443/v1/logs",
  SIGNOZ_ACCESS_TOKEN: "your-access-token"
});

// Start tracing
otel.start();

Next steps

ExuluApp

Learn about ExuluApp configuration

SigNoz docs

Explore SigNoz documentation

OpenTelemetry

Learn about OpenTelemetry

ExuluVariables

Manage telemetry credentials

Getting started

Core classes

Frontend

GraphQL API

Core Types

​Overview

​Key features

Distributed tracing

Structured logging

Auto-instrumentation

SigNoz integration

Component control

Graceful shutdown

​What is OpenTelemetry?

​Quick start

​Prerequisites

​Environment variables

​Initialize OpenTelemetry

​Enable in ExuluApp

​Configuration

​Global telemetry

​Component-level telemetry

​Environment-specific configuration

​What gets traced?

​Automatic instrumentation

​Custom traces

​Viewing traces in SigNoz

​Trace structure

​Key metrics

​Configuration examples

​Production setup

​Development setup

​Selective component tracing

​Integration with logging

​Performance considerations

​Overhead

​Sampling

​Batch export

​Troubleshooting

​Traces not appearing in SigNoz

​High cardinality warnings

​Memory issues

​Graceful shutdown

​Best practices

​SigNoz setup

​SigNoz Cloud

​Self-hosted SigNoz

​Future enhancements

​Example traces

​Agent generation trace

​Context search trace

​Worker job trace

​API reference

​ExuluOtel.create()

​Next steps

ExuluApp

SigNoz docs

OpenTelemetry

ExuluVariables

Overview

Key features

What is OpenTelemetry?

Quick start

Prerequisites

Environment variables

Initialize OpenTelemetry

Enable in ExuluApp

Configuration

Global telemetry

Component-level telemetry

Environment-specific configuration

What gets traced?

Automatic instrumentation

Custom traces

Viewing traces in SigNoz

Trace structure

Key metrics

Configuration examples

Production setup

Development setup

Selective component tracing

Integration with logging

Performance considerations

Overhead

Sampling

Batch export

Troubleshooting

Traces not appearing in SigNoz

High cardinality warnings

Memory issues

Graceful shutdown

Best practices

SigNoz setup

SigNoz Cloud

Self-hosted SigNoz

Future enhancements

Example traces

Agent generation trace

Context search trace

Worker job trace

API reference

ExuluOtel.create()

Next steps