AWS Lake Formation

What it is

AWS Lake Formation centrally governs, secures, and shares data in a data lakeData lakeA central store for large amounts of structured and unstructured data kept in its original form, usually in Amazon S3, which many tools can analyze. AWS Lake Formation helps build and secure one.Glossary entryClose built on Amazon S3 and the AWS Glue Data Catalog. Its main job is permissions: who can see which databases, tables, columns, rows, and even individual cells.

Lake Formation has its own permissions model, which works alongside IAM. You grant and revoke permissions on Data Catalog resources much as you would in a relational database, instead of writing and maintaining complex Amazon S3 bucket policies and IAM policies for every dataset.

How access works

1. A data lake administrator registers the S3 locations of the data with Lake Formation and grants permissions on Data Catalog databases and tables.
2. A user runs a query in an integrated service, such as Amazon Athena, Amazon Redshift Spectrum, Amazon EMR, AWS Glue, or Amazon Quick.
3. The Data Catalog checks the user's Lake Formation permissions and returns only the metadata the user may see.
4. Lake Formation gives the service temporary credentials to read the data from S3 (credential vending), and the service filters out the columns, rows, or cells the user may not see.

Because the check happens in one place, the same rules apply whichever service the user queries from. CloudTrail records the data access for audits.

Fine-grained access

• Column-level: hide columns such as salary or national ID.
• Row-level and cell-level: data filters restrict a user to certain rows, for example one country's records, or to a combination of rows and columns. These protect personally identifiable informationPII (Personally identifiable information)Information that can identify a person, alone or combined with other data, such as a name, email address, or passport number. Amazon Macie finds PII in Amazon S3.Glossary entryClose (PII).
• Tag-based access control (LF-Tags): attach tags such as classification=confidential to databases, tables, or columns, and grant access by tag. A few tags replace thousands of individual grants as the data lake grows.

Sharing across accounts

Lake Formation can share Data Catalog tables, and the data behind them, with other AWS accounts, organizations, or IAM principalsPrincipalA person or workload that can make requests to AWS, such as an IAM user, a role session, the root user, a federated user, or an AWS service. Resource-based policies name the principals they allow.Glossary entryClose in other accounts, keeping the same fine-grained control. This suits a central data lake account that serves several consumer accounts.

Security, resilience, performance, and cost

Lake Formation is itself a security service, so most of what matters is how it fits with the rest of your security setup.

Security

• Encrypted data: you can register S3 locations encrypted with AWS KMS keys, both customer managed and AWS managed. The IAM role that registers the location must be allowed to use the key, through its own policy or the key policy. For a location encrypted with an AWS managed key, use a custom role instead of the Lake Formation service-linked role.
• Private access: an interface VPC endpointVPC endpointA private connection from a VPC to an AWS service or another endpoint service without crossing the internet. Gateway endpoints, for Amazon S3 and DynamoDB, are route table targets; interface endpoints are network interfaces that use AWS PrivateLink.Glossary entryClose (AWS PrivateLink) lets services in private subnetsSubnetA range of IP addresses in a VPC, inside one Availability Zone. A subnet is public when its route table sends internet traffic to an internet gateway, and private when it does not.Glossary entryClose, such as an EMR or Redshift cluster, call Lake Formation, including the GetDataAccess call that vends temporary credentials, without going through the internet.
• Gradual adoption: hybrid access mode lets you turn on Lake Formation permissions for some databases, tables, or users while others keep using IAM and S3 policies.
• Least privilegeLeast privilegeGiving each person or system only the permissions it needs for its task, and no more. In IAM, this means starting with no access and allowing specific actions on specific resources.Glossary entryClose at scale: grant by LF-Tags rather than by individual tables, and review who holds grantable permissions.

Resilience and performance

• Lake Formation does not store or move your data; it stays in Amazon S3. Availability and query speed come from S3 and from the services that run the queries, such as Athena, Redshift Spectrum, EMR, and AWS Glue.

Cost

• There is no charge for creating permissions or for integrated services using them; you pay those services' normal rates, and the Data Catalog is billed as part of AWS Glue.
• Lake Formation charges only for a few specific features, such as its storage APIAPI (Application programming interface)A defined way for one program to ask another for data or actions, such as a set of HTTPS requests and responses. Every AWS service is controlled through its API, which the console, the CLI, and the SDKs call for you.Glossary entryClose when no other analytics service processes the data.

Monitoring

• AWS CloudTrail records Lake Formation API calls and the data access that integrated services perform through it. When another account reads a shared table, the event is also copied to the owner account's CloudTrail logs.

Lake Formation or something else?

If the requirement is	Choose
Column, row, or cell permissions on data lake tables, applied in every analytics service	Lake Formation
Discover the schema of the data and catalog it	AWS Glue crawlers
Find sensitive data such as PII in S3 buckets	Amazon Macie
Allow or deny access to a whole bucket or prefix	S3 bucket policies and IAM

Reverse mindmap: when the question points here

• Yes, when the question says
  • Column-level access to data lake tables→ leads to Lake Formation permissions
  • Users see only their own country's rows→ leads to Row-level data filters
  • Manage permissions for thousands of tables by classification→ leads to LF-Tags
  • One set of rules for Athena, EMR, Glue, and Spectrum→ leads to Lake Formation
  • Share data lake tables with other accounts→ leads to Cross-account sharing
  • Replace complex S3 bucket policies for analytics access→ leads to Lake Formation
• Set it up securely
  • Data lake encrypted with KMS keys→ leads to Registration role allowed to use the key
  • Clusters in private subnets need credentials→ leads to Lake Formation interface endpoint
  • Move to Lake Formation one dataset at a time→ leads to Hybrid access mode
  • Audit who read which data→ leads to CloudTrail data access events
• No, when the question says
  • Discover schemas and build the catalog→ leads to AWS Glue crawlers
  • Find PII in S3 buckets→ leads to Amazon Macie
  • Run the queries themselves→ leads to Athena, EMR, or Redshift Spectrum

Key takeaways

• Lake Formation is the permissions and governance layer for an S3 data lake.
• It grants access at database, table, column, row, and cell level.
• Integrated services receive temporary credentials and enforce the filters, so one set of rules applies everywhere.
• LF-Tags scale permissions; hybrid access mode allows gradual adoption; cross-account sharing serves central data lake designs.
• Permissions cost nothing extra; CloudTrail records data access for audits.

Sources

• AWS Lake Formation Developer Guide: What is AWS Lake Formation?
• AWS Lake Formation Developer Guide: How it works
• AWS Lake Formation Developer Guide: Data filtering and cell-level security
• AWS Lake Formation Developer Guide: Tag-based access control
• AWS Lake Formation Developer Guide: Cross-account data sharing
• Amazon Macie User Guide: What is Amazon Macie?
• AWS Lake Formation Developer Guide: Registering an encrypted Amazon S3 location
• AWS Lake Formation Developer Guide: Interface VPC endpoints (AWS PrivateLink)
• AWS Lake Formation Developer Guide: Hybrid access mode
• AWS Lake Formation Developer Guide: Logging API calls using AWS CloudTrail
• AWS Glue Developer Guide: Granting cross-account access
• AWS Lake Formation pricing