AWS Shows How to Build an Amazon Bedrock Tokenomics Data Lake

A single view of Amazon Bedrock calls

Amazon Bedrock already provides model invocation logging, which can send full request and response records to Amazon Simple Storage Service or Amazon CloudWatch. According to AWS, the limitation is that this data remains inside individual AWS accounts, making it difficult to understand activity across an organization.

The proposed data lake collects raw invocation logs from linked accounts in a central account. It is designed for teams managing large numbers of Amazon Bedrock workloads and looking beyond an aggregate bill. Queries can show what prompt was sent, what response returned, how many tokens were used, which model handled the request and whether a response was cut off.

The processed data includes the prompt, model response, input and output token counts, model name, stop reason, account, timestamp and AWS Identity and Access Management role. AWS says this changes the view from a total such as the number of tokens used to a call-level picture of what each model invocation contained and how efficiently it ran.

How the five-stage architecture works

The design begins when Amazon Bedrock writes an invocation log to a CloudWatch log group. Subscription filters then forward events from each source account to a CloudWatch Logs destination in the central account.

A Lambda function unwraps the compressed and encoded batches, removes CloudWatch housekeeping messages and moves the caller identity into a simpler field. Amazon Data Firehose then buffers the records, converts them from JSON to Parquet and writes them to an Amazon S3 bucket using date-based paths such as year=2026/month=08/day=27/.

An AWS Glue table tells Amazon Athena how to read the files. Teams can then use SQL to examine token use, model selection, prompts and activity by user or account. AWS says the architecture is deployed as Infrastructure as Code through two AWS CloudFormation templates.

Deployment and cost attribution

The deployment requires a central account and at least one linked account, although AWS says they can be the same account. Users also need the AWS Command Line Interface configured with credentials for each account, CloudFormation permissions, Python 3 with Boto3 and a cloned copy of the AWS Sample Bedrock Invocation Data Lake GitHub repository.

The central CloudFormation stack must be deployed first. Linked account stacks then reference its CloudWatch Logs destination ARN. After that, Amazon Bedrock model invocation logging must be activated separately in each source account because it is an account-level setting rather than a CloudFormation resource.

Once data reaches the lake, teams can connect the processed table to AWS Cost and Usage Report 2.0 data. AWS identifies the IAM ARN as the join key. This can support queries about the cost of prompts by day, team or account, as well as which invocations are the least efficient.

Why the design matters for Bedrock teams

The data lake addresses a gap between account-level billing and detailed operational visibility. By centralizing invocation records, platform and machine learning teams can investigate usage spikes, compare model utilization, identify wasteful prompts and attribute activity to accounts or IAM roles.

AWS provides the CloudFormation templates, Lambda code and SQL queries in its AWS Sample Bedrock Invocation Data Lake GitHub repository. The published design therefore gives organizations a repeatable way to deploy the pipeline across accounts, while leaving teams to query the resulting data through Amazon Athena.