Application Insights Distributed Tracing: Following One Request Across Six Services

The hardest part of microservices isn't building them. It's diagnosing why a single user request through six of them is slow. Application Insights' distributed tracing makes this a five-second exercise — if you wire it up properly.

What you get

Every incoming request gets a unique operation_Id. As it calls downstream services, the SDK propagates the ID via the W3C traceparent header. Every dependency, log, exception, and downstream request carries the same ID.

In the App Insights portal, the Application Map blade then shows you:

End-to-end timeline (Gantt chart) of every span in the request.
Which call took 800 ms when it usually takes 40 ms.
The exact log line on the slow service, with the same operation_Id.

The minimal Python setup

pip install azure-monitor-opentelemetry

from azure.monitor.opentelemetry import configure_azure_monitor
configure_azure_monitor(
    connection_string=os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"]
)

That single call instruments Flask, requests, urllib3, redis, psycopg, and more. No per-line tracing code.

Cross-service propagation

If you call another service, your HTTP client must forward the traceparent header. The OpenTelemetry SDK does this automatically. If you're using a custom transport (e.g. raw socket), you must propagate it yourself or the trace breaks at that boundary.

The KQL query that debugs the worst request

requests
| where timestamp > ago(1h) and resultCode != "200"
| top 1 by duration desc
| project operation_Id, name, duration, resultCode
| join kind=inner (
    union dependencies, traces, exceptions
  ) on operation_Id
| order by timestamp asc

One query → every span, log, and exception of the slowest failing request in the last hour. Beats grepping seven log files.

Don't sample everything

At scale, full ingestion gets expensive. Use App Insights adaptive sampling (default) or fixed-rate sampling for non-error traffic. Always force-include errors and slow requests.