Distributed Tracing Headers - APM

A distributed trace is held together by a header that each service passes on. This page lists what the Atatus APM agents send, what they accept, and what you can expect when part of your system is traced by something else.

The header the agents send

Every Atatus APM agent that propagates uses the W3C Trace Context header:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ^  ^                                ^                ^
             |  trace id (32 hex)                span id (16 hex) sampled flag
             version

Most agents also send tracestate, which carries the sampling rate of the service that started the trace.

Agent	Headers sent	Legacy header	Turn the legacy header off with
Node.js	`traceparent`, `tracestate`	`atatus-traceparent`, off by default	`ATATUS_USE_ATATUS_TRACEPARENT_HEADER`
Go	`Traceparent`, `Tracestate`, `Atatus-Apm-Traceparent`	`Atatus-Apm-Traceparent`, on	`ATATUS_USE_TRACEPARENT_HEADER=false`
Python	`traceparent`, `tracestate`, `atatus-traceparent`	`atatus-traceparent`, on	`ATATUS_USE_ELASTIC_TRACEPARENT_HEADER=false`
Ruby	`Traceparent`, `Tracestate`, `Atatus-Traceparent`	`Atatus-Traceparent`, on	`use_atatus_traceparent_header: false`
Java	`traceparent`, `tracestate`, `atatus-traceparent`	`atatus-traceparent`, on	`use_atatus_traceparent_header=false`
.NET	`traceparent`, `tracestate`, `atatus-traceparent`	`atatus-traceparent`, on	`ATATUS_USE_ATATUS_TRACEPARENT_HEADER=false`
PHP	none	None	None

The legacy header always repeats the traceparent value exactly, so a service that reads either one sees the same trace. It exists for compatibility with older agents; leaving it on costs one header per request and nothing else. The PHP agent does not propagate a trace header. A PHP service still appears in your traces when the One Agent is installed (see below), but a trace that passes through it is joined by the connection rather than by a header, and a trace started in PHP does not continue into the service it calls.

Interoperating with other tracers

Atatus reads traceparent from anyone, so a service traced by OpenTelemetry, Elastic APM or any other W3C tracer continues the same trace. In the other direction, any tracer that reads traceparent continues a trace that started in an Atatus agent. Header formats other than W3C are read by the One Agent on the wire (next section), which means you see the call and the services it connects, but an APM agent of a service in the middle of such a trace will not continue it. If you run Zipkin (b3), Jaeger (uber-trace-id), Datadog (x-datadog-trace-id) or SkyWalking (sw8) tracers next to Atatus agents, put an OpenTelemetry-compatible propagator in front of them or configure them for W3C.

Kafka, RabbitMQ and other messaging

Agents that instrument a message producer put traceparent on the message: Node.js (kafkajs, amqplib), Java (Kafka, RabbitMQ, RocketMQ) and the others. A consumer reads it and continues the trace, so a message shows up as one trace across producer and consumer. Amazon SQS and SNS are the exception: the trace header travels as a message attribute inside the AWS API call, which the One Agent does not read, so producer and consumer appear as separate traces.

What the One Agent adds

The Atatus One Agent watches the kernel, so it sees every request a host makes or serves, including the ones no APM agent instruments: a proxy, a database, a service in a language you have not instrumented yet. It reads the trace header off the wire, which is what lets a kernel-level span join the APM trace of the same request. It reads traceparent and the Atatus and Elastic legacy headers by default, as well as b3, x-b3-traceid, uber-trace-id, the Datadog headers and sw8, on HTTP, HTTP/2, gRPC, Kafka, RabbitMQ, RocketMQ and Dubbo. If your services propagate a header of your own, add its name to the agent’s apm_trace_id and apm_span_id settings; on HTTP, HTTP/2 and gRPC it is then read like the built-in ones. Requests whose header the agent cannot read are not lost: it links them by the connection and the process, so the call still appears, joined to whatever else it can be joined to.

When a trace looks incomplete

What you see	Why
A service is missing from a trace	Its APM agent has `tracing` turned off, so it stores no spans. It still passes the header on, so its callers and callees stay in one trace, and the One Agent shows the calls it made.
A trace ends at a PHP service	The PHP agent sends no trace header. The One Agent shows the calls, linked by the connection.
A call appears under the transaction rather than under the exact code path	The agent reached its per-transaction span limit, so the header carries the transaction’s id instead of the span’s. Raise the span limit if the detail matters.
Producer and consumer of an SQS or SNS message are separate traces	The header travels in the AWS API call, which is not read off the wire.
A trace says only part of the requests were kept	The service was over its request limit. Requests are dropped whole trace by whole trace, so what you see is complete; there are fewer traces, not thinner ones.

​The header the agents send

​Interoperating with other tracers

​Kafka, RabbitMQ and other messaging

​What the One Agent adds

​When a trace looks incomplete

The header the agents send

Interoperating with other tracers

Kafka, RabbitMQ and other messaging

What the One Agent adds

When a trace looks incomplete