AWS X-Ray
Distributed tracing on AWS: traces, segments, and subsegments, the trace map, sampling rules, annotations and metadata, integration with Lambda, API Gateway, load balancers, ECS, and EC2, the move to OpenTelemetry, security, cost, and how X-Ray differs from CloudWatch and CloudTrail.
8 min read · Last reviewed · SAA-C03
Sign in to bookmark tutorials and track what you have read.
On this page
What it is
AWS X-Ray is a distributed tracing service. It collects data about the requests your application serves and follows each request as it moves through your services: from API (Application programming interface)A defined way for one program to ask another for data or actions, such as a set of HTTPS requests and responses. Every AWS service is controlled through its API, which the console, the CLI, and the SDKs call for you.Glossary entry Gateway, to a Lambda function or container, to DynamoDB, SQS, an RDS database, or an external web API. You can then see where time is spent and where errors happen.
Choose X-Ray when a question asks how to find the cause of LatencyThe time between sending a request and receiving the response, usually measured in milliseconds. Distance and processing add to it; caching and serving users from nearby locations reduce it.Glossary entry or errors in a distributed application or MicroservicesAn application built as small, independent services, each owning one business function and its data, that communicate over APIs or events. Each service can be changed, deployed, and scaled on its own.Glossary entry architecture, or to visualize how requests flow between services. It is the exam's only Developer Tools service.
Concepts
- Trace: the full path of one request, made of everything recorded with the same trace ID. The first X-Ray-integrated service that the request reaches adds a tracing header,
X-Amzn-Trace-Id, which is passed downstream so every service records against the same trace. - Segment: the work done by one service for a request, with timing, the request and response, and errors. Subsegments break that work down further, for example one call to DynamoDB or one SQL (Structured Query Language)The standard language for querying and changing data in relational databases. Analytics services such as Amazon Athena and Amazon Redshift also use SQL.Glossary entry query.
- Trace map (formerly the service map): a diagram built from traces, with a node for each service and the latency and error rate on each connection. It now lives in the CloudWatch console, under X-Ray traces.
- Errors: X-Ray separates errors (4xx client errors), faults (5xx server errors), and throttles (429 Too Many Requests).
- Annotations are key-value pairs that X-Ray indexes, up to 50 per trace, so you can search and filter traces by them, for example by customer ID or game name. Metadata holds any other data and is not indexed.
- Filter expressions find traces, for example slow or failed ones. Groups save a filter expression, get their own trace map, and publish CloudWatch metrics.
- Sampling: X-Ray records a sample of requests to control cost. By default it records the first request each second (the reservoir) and 5% of additional requests. Sampling rules change this per service, URL, or method without redeploying code. Sampling is decided once, by the first traced service (for example the API Gateway stage or the front-end service), and downstream services follow that decision, so set rules where traces start. Lambda's own sampling rate can't be configured.
- Insights: X-Ray detects anomalies in fault rates on the trace map and can notify you.
How applications send traces
- Instrument the code so it records segments. AWS now recommends OpenTelemetry, through AWS Distro for OpenTelemetry (ADOT) or the CloudWatch agent. The X-Ray SDK (Software development kit)A set of libraries for a programming language that wraps a service's API and handles details such as signing requests and retrying. AWS SDKs exist for Python (Boto3), JavaScript, Java, Go, and other languages.Glossary entry and daemon entered maintenance mode on February 25, 2026: they get security fixes only, and AWS recommends migrating to OpenTelemetry. Exam questions still describe the SDK and daemon, so know both.
- The X-Ray daemon listens on UDP (User Datagram Protocol)A transport protocol that sends packets without setting up a connection or resending lost ones, trading reliability for low latency. DNS, video calls, and games often use it; Network Load Balancer and AWS Global Accelerator support UDP.Glossary entry port 2000, gathers raw segment data, and relays it to the X-Ray API. The CloudWatch agent can now collect traces in its place.
- Where it runs:
- AWS Lambda: turn on active tracing in the function configuration (
TracingConfig: Mode: Activein CloudFormation,Tracing: Activein AWS SAM); Lambda runs the tracing for you. Without it, a function is in PassThrough mode: it forwards the trace header but sends no traces, even for requests API Gateway sampled, so the function is missing from the trace map. - Amazon API Gateway: turn on X-Ray tracing for a stage; API Gateway applies sampling and adds a node for the stage.
- Amazon ECS: run the daemon (or an OpenTelemetry collector) as a sidecar container next to the application in the task definition.
- Amazon EC2 and On-premisesIn data centers or offices that you run yourself, rather than in the cloud. Hybrid architectures connect on-premises systems to AWS, for example through AWS Direct Connect or a VPN.Glossary entry: install the daemon or the CloudWatch agent.
- AWS Elastic Beanstalk: includes the daemon on supported platforms; you turn it on.
- Application Load Balancers add the
X-Amzn-Trace-Idheader to requests. - SNS, SQS, and Step Functions pass the trace context along, so traces continue across messaging and workflows. SQS carries the trace header in the
AWSTraceHeadermessage attribute, and X-Ray links a producer's trace to the trace of a downstream Lambda consumer automatically; other consumers must read the header and continue the trace in code.
- AWS Lambda: turn on active tracing in the function configuration (
Security, resilience, performance, and cost
Security
- IAM permissions are needed to send traces. Attach a policy such as
AWSXRayDaemonWriteAccessto the Lambda execution role, ECS task role, or EC2 instance profile. Missing permissions are a common reason no traces appear. Turning on tracing in the Lambda console adds them to the execution role for you; a role defined in CloudFormation or another template must grant them itself. - X-Ray always encrypts trace Encryption at restEncrypting data where it is stored, such as on disks, in databases, or in backups, so that it cannot be read without the key. On AWS it usually uses keys in AWS KMS.Glossary entry. To audit or control the key, use the AWS managed key
aws/xrayor a customer managed KMS key. - Don't record secrets or personal data in annotations or metadata.
Resilience
- X-Ray is a managed, Regional service with nothing for you to scale or keep available.
- Use traces to find failing dependencies quickly, and insights to be told about unusual fault rates.
Performance
- Keep sampling rates sensible on very busy services, so tracing doesn't add cost or noise.
- Use the trace map and trace details to find the slowest service or downstream call, then fix that part.
Cost
- Pay per trace recorded and per trace retrieved or scanned. The first 100,000 traces recorded and the first 1,000,000 traces retrieved or scanned each month are free.
- Sampling is the main cost control: record a small share of routine traffic and more of what matters.
Monitoring
- Group metrics and insights notifications appear in CloudWatch.
- CloudWatch Application Signals builds application performance monitoring (APM) dashboards and SLOs on top of traces.
- Transaction Search stores and searches all spans in CloudWatch Logs.
X-Ray, CloudWatch, or CloudTrail?
| Question asks for | Choose |
|---|---|
| The path of a request across services, with latency per hop | AWS X-Ray |
| Which microservice causes errors or slow responses | AWS X-Ray (trace map) |
| Metrics, alarms, dashboards, and application logs | Amazon CloudWatch |
| Who made which AWS API call, and when | AWS CloudTrail |
| Application performance dashboards and SLOs from traces | CloudWatch Application Signals |
| Vendor-neutral instrumentation sent to several tools | AWS Distro for OpenTelemetry |
Reverse mindmap: when the question points here
- Yes, when the question says
- Trace requests across microservices leads to AWS X-Ray
- Find which service adds latency leads to X-Ray trace map
- Visualize calls between Lambda, API Gateway, and DynamoDB leads to X-Ray
- Debug errors in a distributed application leads to X-Ray
- Turn it on
- Lambda function leads to Active tracing
- Lambda missing from the trace map leads to Turn on active tracing (default is PassThrough)
- One view from SQS producer to Lambda consumer leads to Active tracing on the function; traces link automatically
- API Gateway stage leads to Enable X-Ray tracing
- ECS tasks leads to Daemon or collector as a sidecar
- EC2 instances leads to Install the daemon or CloudWatch agent
- No traces appear leads to Check IAM permissions to write traces
- Use it well
- Search traces by customer or order ID leads to Annotations
- Store extra data, no search leads to Metadata
- Record fewer requests to cut cost leads to Sampling rules
- Where a sampling rule takes effect leads to The service that starts the trace
- New instrumentation today leads to OpenTelemetry with ADOT
- No, when the question says
- Audit who called an AWS API leads to CloudTrail
- CPU alarms and log searches leads to CloudWatch
Key takeaways
- X-Ray traces requests end to end across services, using a shared trace ID in the
X-Amzn-Trace-Idheader. - A trace is made of segments and subsegments; the trace map shows latency and errors between services.
- Annotations are indexed for search; metadata is not.
- Default sampling: the first request each second plus 5% of the rest; sampling rules control cost.
- Turn on active tracing for Lambda and tracing for API Gateway stages; run the daemon as an ECS sidecar; grant IAM permissions to write traces.
- The X-Ray SDKs and daemon have been in maintenance mode since February 25, 2026; OpenTelemetry (ADOT) is the recommended instrumentation.
- X-Ray traces requests; CloudWatch handles metrics and logs; CloudTrail records API calls.
Sources
- AWS X-Ray Developer Guide: What is AWS X-Ray?
- AWS X-Ray Developer Guide: AWS X-Ray concepts
- AWS X-Ray Developer Guide: Using the X-Ray trace map
- AWS X-Ray Developer Guide: Configuring sampling rules
- AWS X-Ray Developer Guide: Using filter expressions
- AWS X-Ray Developer Guide: Configuring groups
- AWS X-Ray Developer Guide: Using X-Ray insights
- AWS X-Ray Developer Guide: Instrumenting your application for AWS X-Ray
- AWS X-Ray Developer Guide: X-Ray SDK and Daemon support timeline
- AWS X-Ray Developer Guide: Migrating from X-Ray instrumentation to OpenTelemetry instrumentation
- AWS X-Ray Developer Guide: AWS X-Ray daemon
- AWS X-Ray Developer Guide: Running the X-Ray daemon on Amazon ECS
- AWS X-Ray Developer Guide: Running the X-Ray daemon on Amazon EC2
- AWS X-Ray Developer Guide: Integrating AWS X-Ray with other AWS services
- AWS X-Ray Developer Guide: AWS Lambda and AWS X-Ray
- AWS X-Ray Developer Guide: Amazon API Gateway active tracing support for AWS X-Ray
- AWS X-Ray Developer Guide: Elastic Load Balancing and AWS X-Ray
- AWS X-Ray Developer Guide: Amazon SQS and AWS X-Ray
- AWS X-Ray Developer Guide: Tracing event-driven applications
- AWS Lambda Developer Guide: Visualize Lambda function invocations using AWS X-Ray
- AWS X-Ray Developer Guide: Amazon SNS and AWS X-Ray
- AWS X-Ray Developer Guide: AWS Step Functions and AWS X-Ray
- AWS X-Ray Developer Guide: Data protection in AWS X-Ray
- AWS X-Ray Developer Guide: AWS X-Ray identity-based policy examples
- Amazon CloudWatch User Guide: Application Signals
- Amazon CloudWatch User Guide: Transaction Search
- Amazon CloudWatch pricing