The MCP Server Was a Black Box: Adding OpenTelemetry to FastMCP & TypeScript to See Everything from Agent Reasoning to DB Calls on One Screen

This post is for developers who already run MCP servers or are considering adopting them. When an AI agent calls tools on an MCP server, I honestly had no idea what was happening inside. The entire flow — agent reasoning, tool selection, MCP server execution, and the resulting database calls — was one giant black box. When responses slowed down somewhere, there was no way to tell where the problem was, and when errors occurred, I'd dig through logs and eventually give up. This kept repeating.

The MCP SDK ships with zero built-in observability. You have to wire up OpenTelemetry instrumentation yourself, and once you try to start, it's hard to know where to begin. After reading this post, you'll be able to view full agent traces in Jaeger on a single screen — with almost no changes to existing code — using just a few environment variables for Python (FastMCP) servers, or a single --import flag for TypeScript servers. I've also covered the common traps you'll hit in practice: sensitive data exposure, duplicate spans, and missing sampling.

A few months ago there was almost no material on MCP + OTel, but recently Elastic, Google, and SigNoz have all published instrumentation guides at nearly the same time. OTel v1.39 even added official MCP-specific semantic conventions. Now that agentic workflows are reaching production, running a server without instrumentation is no different from deploying without logs.

Core Concepts

If you'd rather see code before concepts, jump directly to the Practical Application section below.

Why Instrumenting MCP Servers Is Tricky

With a regular HTTP API, attaching one middleware gives you per-request traces instantly. MCP is different. The agent framework (Claude, LangChain, LlamaIndex, etc.) sits above, the MCP server sits below, and JSON-RPC messages travel between them. Stitching the agent-side trace and the server-side trace together under the same trace_id is the core challenge.

On top of that, MCP supports stdio transport in addition to HTTP/SSE, and stdio has no HTTP headers. You can't use the standard OTel W3C TraceContext propagation approach (traceparent header) as-is.

Trace Context Propagation: A mechanism in distributed systems where trace_id and span_id are carried in messages so that different processes can recognize they belong to the same request flow. For HTTP, headers serve as the channel; for MCP, the _meta field is used instead.

The _meta Field: Flowing Context Regardless of Transport

MCP protocol JSON-RPC messages have always had a _meta field. The practice of placing traceparent, tracestate, and baggage values there to propagate context uniformly across HTTP and stdio is what multiple instrumentation libraries, including FastMCP, actually use in practice. The official MCP spec has not fully standardized context propagation via _meta; the community and vendors are in a stage of converging toward this convention.

{
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "name": "search_database",
    "arguments": { "query": "user:123" },
    "_meta": {
      "traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01",
      "tracestate": ""
    }
  }
}

This way, the agent embeds its span information in _meta, and the MCP server extracts it to use as the parent context — completing a single distributed trace.

OTel v1.39 MCP Semantic Conventions

MCP-specific attribute names, first introduced in v1.39 and developed through the current v1.41.0, have been officially standardized. The spec defines the following attributes as standard for tools/call spans.

Note — The GenAI/MCP semantic conventions are currently in Development status (introduced in v1.39, current version v1.41.0). Attribute names may change without a breaking-change notice, so when deploying to production, it's recommended to pin opentelemetry/semantic-conventions to an exact version like 1.39.0.

Practical Application

Example 1: Python — Zero Lines of Server Code Changed with FastMCP 3.0

FastMCP 3.0 has OTel instrumentation built in by default for the HTTP transport, making it almost unbelievably simple. A few environment variables are all it takes (if you also need context propagation over the stdio transport, see the manual implementation approach further below).

# pip install fastmcp opentelemetry-sdk opentelemetry-exporter-otlp
from fastmcp import FastMCP
 
# db, users are existing ORM/client instances
mcp = FastMCP("my-mcp-server")
 
@mcp.tool()
def search_database(query: str) -> str:
    """Searches the database."""
    return db.search(query)
 
@mcp.tool()
def get_user_info(user_id: int) -> dict:
    """Fetches user information."""
    return users.find(user_id)

# Enable OTel via environment variables only — no server code changes needed
OTEL_SERVICE_NAME=my-mcp-server \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
OTEL_EXPORTER_OTLP_PROTOCOL=grpc \
python server.py

FastMCP automatically generates spans for all tool, prompt, resource, and resource_template calls. You don't need to touch a single line of server code.

Should you use async def or def? FastMCP supports both synchronous (def) and asynchronous (async def) handlers, and OTel instrumentation works identically for both. Use async def for IO-bound work like DB calls or external HTTP requests; def is fine for pure computation.

If you need direct control over context propagation or are using stdio transport, you can implement _meta injection and extraction explicitly as shown below.

from opentelemetry import trace, propagate
from opentelemetry.trace import SpanStatusCode
from fastmcp import FastMCP, Context
 
mcp = FastMCP("my-mcp-server")
 
@mcp.tool()
async def search_database(query: str, ctx: Context) -> str:
    # Extract parent context from _meta
    parent_ctx = propagate.extract(ctx.meta or {})
 
    tracer = trace.get_tracer(__name__)
    with tracer.start_as_current_span(
        "search_database",
        context=parent_ctx
    ) as span:
        span.set_attribute("gen_ai.tool.name", "search_database")
        span.set_attribute("mcp.method.name", "tools/call")
        # Caution: do not put sensitive data (API keys, PII) directly into span attributes
        span.set_attribute("db.query.length", len(query))
 
        try:
            result = db.search(query)  # db is an existing client instance
            span.set_attribute("db.result_count", len(result))
            return result
        except Exception as e:
            # Error path: record both the exception and span status so the failure cause is traceable
            span.record_exception(e)
            span.set_status(SpanStatusCode.ERROR, str(e))
            raise

Example 2: TypeScript — Adding Instrumentation to an Existing Server with the --import Flag

When I first opened Jaeger, tools/call spans were appearing twice and I spent a long time puzzled — it turned out the agent framework and the instrumentation library were both creating spans simultaneously. I'll revisit this in the mistakes list below.

If you're already running a TypeScript MCP server, you can add instrumentation without modifying any code.

// npm install @opentelemetry/sdk-node opentelemetry-instrumentation-mcp
// npm install @opentelemetry/exporter-trace-otlp-grpc @opentelemetry/resources
//
// Note: opentelemetry-instrumentation-mcp is a community package,
// not an official OTel package (github.com/theharithsa/opentelemetry-instrumentation-mcp).
// It is recommended to check the license and maintenance status before adopting in production.
 
// instrumentation.ts
import { NodeSDK } from '@opentelemetry/sdk-node';
import { MCPInstrumentation } from 'opentelemetry-instrumentation-mcp';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
import { Resource } from '@opentelemetry/resources';
 
const sdk = new NodeSDK({
  resource: new Resource({
    'service.name': 'my-mcp-server',  // Using string literal since SEMRESATTRS_SERVICE_NAME is deprecated
  }),
  traceExporter: new OTLPTraceExporter({
    url: 'http://localhost:4317',
  }),
  instrumentations: [new MCPInstrumentation()],
});
 
sdk.start();
 
process.on('SIGTERM', () => {
  sdk.shutdown().then(() => process.exit(0));
});

# Compiled to JS (most stable)
node --import ./instrumentation.js your-mcp-server.js
 
# If using tsx (verified on Node.js 22+ and tsx 4.x)
npx tsx --import ./instrumentation.ts your-mcp-server.ts
 
# The ts-node + ESM combination can fail depending on Node.js version and tsconfig settings,
# so try one of the two methods above first

MCPInstrumentation patches the MCP SDK's internal methods to generate a span for each tools/call and also handles _meta context propagation. When you need to record error spans manually, the following pattern serves as a baseline.

import { trace, SpanStatusCode } from '@opentelemetry/api';
 
const tracer = trace.getTracer('my-mcp-server');
 
async function searchDatabase(query: string): Promise<string[]> {
  return tracer.startActiveSpan('search_database', async (span) => {
    span.setAttribute('gen_ai.tool.name', 'search_database');
    try {
      const result = await db.search(query);
      span.setAttribute('db.result_count', result.length);
      return result;
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR, message: String(err) });
      throw err;
    } finally {
      span.end();
    }
  });
}

Example 3: Automatically Deriving RED Metrics with the OTel Collector

Once you're collecting traces, you can automatically extract Rate/Error/Duration metrics using the spanmetrics connector — no separate metrics implementation needed. Export to Prometheus and the metrics are immediately available in a Grafana dashboard.

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
 
connectors:
  spanmetrics:
    histogram:
      explicit:
        # The spanmetrics connector accepts Go duration strings
        buckets: ["10ms", "50ms", "100ms", "500ms", "1s", "5s"]
    dimensions:
      - name: gen_ai.tool.name    # Separate metrics per tool
      - name: mcp.session.id
      - name: service.name
    metrics_expiration: 5m
 
exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
  otlp/jaeger:
    endpoint: http://jaeger:4317
    tls:
      insecure: true
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp/jaeger, spanmetrics]
    metrics:
      receivers: [spanmetrics]
      exporters: [prometheus]

RED Metrics: Stands for Rate (requests per second), Error (error rate), and Duration (response time). The spanmetrics connector automatically calculates all three from trace spans — no separate metrics instrumentation code required.

Pros and Cons

Advantages

Disadvantages and Caveats

Tail-based Sampling: An approach where you look at the full trace after a request completes and decide whether to store it based on whether it had errors or high latency. This lets you discard normal requests and keep only problematic ones, making it effective in high-frequency environments.

The Most Common Mistakes in Practice

1. Deploying without sensitive data filtering: Recording tool arguments wholesale into spans means values like { "api_key": "sk-..." } accumulate in your trace store as-is. It's best to filter them out before collection using the delete action of the Collector's attributesprocessor. This often goes unnoticed in development and is discovered immediately after the first production deployment.

2. Going to production without sampling configured: Collecting everything is convenient in development, but in a production environment where agents call tools hundreds of times per second, storage costs grow far faster than you'd expect. You can easily set 10% sampling with just two environment variables: OTEL_TRACES_SAMPLER=parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG=0.1.

3. Not monitoring for duplicate spans: Agent frameworks like LangChain or LlamaIndex often include their own OTel instrumentation. If you see two tools/call spans for the same tool invocation in Jaeger, the agent framework and the MCP instrumentation library are both creating spans. Check the span scope for each layer and either disable one or narrow its scope.

Wrapping Up

Here are three steps you can take right now.

1. Spin up Jaeger locally: Run docker run -d --name jaeger -p 16686:16686 -p 4317:4317 jaegertracing/all-in-one:latest to start a Jaeger container, and the trace viewer will be available immediately at http://localhost:16686.

2. Attach environment variables to your MCP server: For Python (FastMCP), try running the server with OTEL_SERVICE_NAME=my-mcp-server OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 appended. For TypeScript, create instrumentation.ts from Example 2 and run it with node --import ./instrumentation.js your-mcp-server.js.

3. Call one tool: Invoke a tool via an agent or MCP client, then search for my-mcp-server in the Jaeger UI — you'll see the tools/call span with gen_ai.tool.name and mcp.method.name attributes recorded on it.

If your MCP server is going to production, instrumentation is infrastructure, not a feature. The moment agent reasoning and tool execution are connected in a single trace, you can pinpoint performance bottlenecks immediately and experience a dramatic reduction in the time it takes to trace the cause of errors.

References

Getting Started

• OpenTelemetry - FastMCP Official Docs
• MCP Observability with OpenTelemetry | SigNoz Blog
• Monitor MCP Performance with OpenTelemetry | MCPcat
• How to Instrument MCP Servers with OpenTelemetry for Production Observability | OneUptime Blog

Going Deeper

• How to trace MCP server tool calls with OpenTelemetry and Elastic APM | Elastic Observability Labs
• Distributed tracing for agentic workflows with OpenTelemetry | Red Hat Developer
• FastMCP Distributed Tracing: Transport-Agnostic Context Propagation with _meta
• How OpenTelemetry Traces LLM Calls, Agent Reasoning, and MCP Tools | Greptime Blog

Official Specs and Packages

• Semantic conventions for Model Context Protocol (MCP) | OpenTelemetry Official
• Instrument a self-hosted MCP server with OpenTelemetry | Google Cloud Docs
• opentelemetry-instrumentation-mcp | PyPI
• theharithsa/opentelemetry-instrumentation-mcp | GitHub (TypeScript)