AWS Step Functions

What it is

AWS Step Functions coordinates the steps of a process as a workflow, also called a state machine. You describe the steps, the order they run in, the decisions between them, and what happens when one fails. Step Functions then runs each execution of the workflow, calling AWS services such as Lambda, ECS, Glue, or DynamoDB for each step and keeping track of where it is.

Without it, that coordination logic ends up inside your code: one Lambda function calls the next, retries by hand, and loses track when something times out. Step Functions moves the coordination out of the code, and shows each execution as a diagram in the console, so you can see which step failed and why.

Workflows are written in the Amazon States Language (JSON (JavaScript Object Notation)A text format for structured data made of name-and-value pairs and lists, such as {"bucket": "logs"}. IAM policies, most AWS API responses, and many CloudFormation templates are written in JSON.Glossary entry), or designed visually in Workflow Studio.

States

Each step in a workflow is a state:

State	What it does
Task	Does one unit of work: calls an AWS service or an HTTPS (HTTP Secure)HTTP sent over a TLS-encrypted connection, so that requests and responses cannot be read or changed on the way, and the server proves its identity with a certificate.Glossary entry API (Application programming interface)A defined way for one program to ask another for data or actions, such as a set of HTTPS requests and responses. Every AWS service is controlled through its API, which the console, the CLI, and the SDKs call for you.Glossary entry, or waits for an activity worker
Choice	Picks the next state from the input, like an if-else
Parallel	Runs several branches at the same time and waits for all of them
Map	Runs the same steps for each item in a list
Wait	Pauses for a time, or until a set date and time
Pass	Passes its input to its output, optionally adding or reshaping data
Succeed and Fail	End the execution with success or failure

Calling other services

Step Functions can call over two hundred AWS services directly through AWS SDK (Software development kit)A set of libraries for a programming language that wraps a service's API and handles details such as signing requests and retrying. AWS SDKs exist for Python (Boto3), JavaScript, Java, Go, and other languages.Glossary entry integrations, and some services through optimized integrations. A Task state can call a service in one of three ways:

Request Response (the default): call the service and move on as soon as it replies.
Run a Job (.sync): start a job, such as an ECS task, a Glue job, an AWS Batch job, or an Athena query, and wait until it finishes.
Wait for a Callback (.waitForTaskToken): send a task token to another system, for example in an SQS message, and pause until that system returns the token with SendTaskSuccess or SendTaskFailure. This is how a workflow waits for a human approval or a third-party system. Set HeartbeatSeconds so a lost token does not leave the execution waiting for up to a year.

An example Step Functions workflow for an order. A Task state checks stock, and a Choice state asks whether the item is in stock. If not, the workflow goes to a Fail state. If it is, a Task state waits for a human approval with a callback, and when the task token is returned, a Task state charges the card and the workflow reaches a Succeed state.

Large-scale parallel processing

A Map state in Distributed mode processes a large data set, such as a big CSV or JSON file or millions of objects in Amazon S3, by running each item or batch as a child workflow execution, with up to 10,000 running in parallel. Use it when the data is larger than 256 KiB, or needs more than 40 concurrent iterations. Inline mode is for smaller lists inside the execution.

Standard and Express workflows

You choose the workflow type when you create a state machine, and you cannot change it afterwards.

	Standard	Express
Maximum duration	1 year	5 minutes
Execution model	Exactly once	At least once (asynchronous) or at most once (synchronous)
Best for	Long-running, auditable processes; actions that must not repeat, such as payments	High-volume, short event processing, such as IoT ingestion or streaming transforms
Execution history	Kept by Step Functions for 90 days	Only in CloudWatch Logs, if you turn logging on
`.sync` and callback patterns	Supported	Not supported
Distributed Map	Supported	Not supported
Priced by	State transitions	Number of executions, duration, and memory

Synchronous Express workflows wait for the workflow to finish and return its result, which suits a request from Amazon API Gateway. Asynchronous Express workflows return as soon as they start.

Security, resilience, performance, and cost

Security

Each state machine runs with an IAM execution role, which needs permission for every service the workflow calls. Grant only the actions each step needs.
IAM policies also control who can create, start, and stop workflows.
Encryption at restEncrypting data where it is stored, such as on disks, in databases, or in backups, so that it cannot be read without the key. On AWS it usually uses keys in AWS KMS.Glossary entry: Step Functions always encrypts data at rest with AWS owned keys. For a second layer, choose a customer managed KMS key for a state machine or activity; it then encrypts the definition and execution history, and the execution role needs kms:Decrypt and kms:GenerateDataKey. Encryption in transitEncrypting data while it moves across a network, usually with TLS, so that it cannot be read or changed on the way.Glossary entry uses TLS (Transport Layer Security)The protocol that encrypts traffic between a client and a server and lets the server prove its identity with a certificate. It replaced SSL, though the older name is still common; HTTPS is HTTP over TLS.Glossary entry.
Private access: an interface VPC endpointA private connection from a VPC to an AWS service or another endpoint service without crossing the internet. Gateway endpoints, for Amazon S3 and DynamoDB, are route table targets; interface endpoints are network interfaces that use AWS PrivateLink.Glossary entry (AWS PrivateLink) lets applications in a VPC start and manage executions without the internet, and an endpoint policy can limit what callers do through it.

Resilience

When a state fails and nothing handles the error, the whole execution fails. Handle errors in Task, Parallel, and Map states:
- Retry reruns a state on matching errors, with IntervalSeconds (default 1), MaxAttempts (default 3), BackoffRate (default 2.0), an optional MaxDelaySeconds, and optional jitter.
- Catch sends the execution to a fallback state after retries run out, for example to undo earlier steps or notify someone.
- For a transaction across several services (the saga pattern), each step's Catch routes to compensating steps that undo the steps already done in reverse order, such as refunding a payment and releasing reserved stock.
- States.ALL matches any error; States.TaskFailed matches any error except a timeout.
Set TimeoutSeconds on tasks, and on the whole state machine, so nothing hangs.
Redrive restarts a failed, aborted, or timed-out Standard execution from the step that failed, within 14 days, without rerunning the steps that succeeded.

Performance

The input or output of a state, task, or execution is limited to 256 KiB. Store larger data in Amazon S3 and pass a reference.
A Standard execution's history is limited to 25,000 events; for loops over many items, use Distributed Map or start new executions.
Express workflows have no limit on state transitions per second; Standard workflows have a per-AWS RegionA separate geographic area where AWS runs its services, such as us-east-1 in Northern Virginia, made up of several Availability Zones. Most resources live in one Region; you choose Regions for latency, data residency, and the services you need.Glossary entry quota that you can raise.

Cost

Standard: per state transition; retries count as transitions. The free tier includes 4,000 state transitions a month.
Express: per execution, plus duration (rounded up to 100 ms) and memory used.
For high-volume, short workflows, Express is usually much cheaper. Services that the workflow calls, such as Lambda, are billed separately.

Monitoring

Amazon CloudWatch metrics include ExecutionsStarted, ExecutionsSucceeded, ExecutionsFailed, ExecutionsTimedOut, ExecutionThrottled, and ExecutionTime. Alarm on failed and timed-out executions.
CloudWatch Logs records execution history; Express workflows need it, because Step Functions does not keep their history.
Amazon EventBridge receives an event when a Standard execution changes status, for example to FAILED or TIMED_OUT, so a rule can react.
AWS X-Ray traces requests through the workflow and the services it calls; AWS CloudTrail records Step Functions API calls.

Step Functions or another service?

If the requirement is	Choose
Run steps in order with branching, retries, and error handling	AWS Step Functions
Wait for a human approval in the middle of a process	Step Functions with a callback task token
Process millions of S3 objects in parallel	Step Functions Distributed Map
Only buffer work between two components	Amazon SQS
Fan one message out to many subscribers	Amazon SNS
Start something when an event happens or on a schedule	Amazon EventBridge (it can start a workflow)

Reverse mindmap: when the question points here

Is it AWS Step Functions?

Yes, when the question says
- Coordinate several Lambda functions in order leads to Step Functions
- Branch on a result and retry failed steps leads to Choice state with Retry and Catch
- Wait for a manager's approval leads to Callback with a task token
- Run several steps at the same time leads to Parallel state
- Undo completed steps when a later step fails leads to Catch to compensating steps (saga)
- Wait until a Glue, ECS, or Batch job finishes leads to Task with .sync
- Process each of millions of S3 objects leads to Distributed Map
Choose the workflow type
- Long-running, auditable, must not repeat leads to Standard workflow
- High-volume events, under five minutes leads to Express workflow
- API Gateway needs the result in the response leads to Synchronous Express workflow
Run it well
- Restart a failed execution from the failed step leads to Redrive
- Output larger than 256 KiB leads to Store it in S3 and pass the path
- Encrypt execution history with your own key leads to Customer managed KMS key
No, when the question says
- Decouple two components with a buffer leads to Amazon SQS
- React to events from AWS services or SaaS apps leads to Amazon EventBridge
- Move SaaS data into S3 with no code leads to Amazon AppFlow

Key takeaways

Step Functions orchestrates steps as a state machine: Task, Choice, Parallel, Map, Wait, Pass, Succeed, and Fail states.
Standard workflows run up to a year, exactly once; Express workflows run up to five minutes, at high volume, and cost less for short work.
Retry and Catch handle errors; redrive restarts a failed Standard execution from the failed step.
Callbacks with task tokens wait for humans or other systems; .sync waits for jobs to finish; Distributed Map processes large S3 data sets in parallel.
State input and output are limited to 256 KiB; pass large data through S3.