AWS Honey Tokens: The Good, the Bad, and the Ugly

AWS Honey Tokens: The Good, the Bad, and the Ugly

The Good

Imagine you’re an attacker who has gained access to a modern organization’s internal environment. No matter where you land, your first instinct would likely be to search for credentials to escalate your privileges and establish persistent access. This is where honey tokens come in-decoy credentials designed to lure attackers and trigger alerts when used.

Among the most notable honey tokens are AWS Access Key tokens. These long-lived credentials can be strategically placed across systems. When an attacker uses them, they alert the security team to the presence of the attacker or, more broadly, unauthorized behavior within the organization.

Thinkst Canary Tokens are an excellent example of a simple yet effective honey token implementation. The core idea is this: you generate unique tokens for each location you want to monitor and deploy them as "digital tripwires." Although it’s not guaranteed to lure every attacker, as a former Red Teamer, I can attest that these tokens are hard to resist - especially in a locked-down environment with few other attack paths available.

As a team developing a deception platform, we’ve threat-modeled our own honey token solution and compared it to vendor implementations, such as Thinkst’s, to identify limitations in both approaches. The open source Canarytokens project appears to closely resemble the implementation powering canarytokens.org and Thinkst’s commercial offering. We’ll use it as a baseline for our threat model analysis.

The Bad

Diving into how the project works, there are two core Lambda functions:

1. CreateUserAPITokens - Exposed behind API Gateway, this function is used to create a new IAM user, and for that user, creates a corresponding pair of access key credentials.
2. ProcessUserAPITokensLogs - This function is triggered by an AWS CloudWatch Group Subscription Filter, and is used to process any canary AWS user activity, in turn triggering an alert.

Without diving too deeply into the details of CloudTrail (AWS’s service for logging API activity) and CloudWatch Log Group Subscription Filters (which process those logs), we can focus on the core logic: the Lambda function is triggered when a CloudTrail event’s userIdentity.type equals IAMUser:

1resource "aws_cloudwatch_log_subscription_filter" "process_user_api_tokens_logs" {
2  name            = "LambdaStream_${var.process_user_api_tokens_logs}"
3  log_group_name  = aws_cloudwatch_log_group.process_user_api_tokens_logs.name
4  filter_pattern  = "{$.userIdentity.type = IAMUser}"
5  destination_arn = aws_lambda_function.process_user_api_tokens_logs.arn
6}
7

Within the ProcessUserAPITokensLogs function, this decodes the events, and then builds a payload to send to a location constructed from the userIdentity.userName field.

1encoded_info = id_record['userName'].split('@@')
2accessKeyId = id_record['accessKeyId']
3server = encoded_info[0]
4token = encoded_info[1]
5
6url = "http://{}/{}".format(server, token)
7data = {"ip": ip, "user_agent": agent}
8if 'eventName' in msg:
9    data['eventName'] = msg['eventName']
10

An example of the userIdentity.userName field would be canarytokens.com@@cc0pv1gnsc1ohu76fqq40409u. This would then be resolved to http://canarytokens.com/cc0pv1gnsc1ohu76fqq40409u, of which the ProcessUserAPITokensLogs function would then send a HTTP request to with a body including the IP address, user agent, and event name.

To retrieve the username tied to a set of credentials, an attacker would simply need to run aws sts get-caller-identity using the honey token credentials.

1{
2  "UserId": "AIDA475TKMODCRTEGGZJN",
3  "Account": "893192397702",
4  "Arn": "arn:aws:iam::893192397702:user/canarytokens.com@@cc0pv1gnsc1ohu76fqq40409u"
5}
6

It doesn’t take an Advanced Persistent Threat (APT) to recognize this as a honey token and realize detection is likely within minutes. At this point, the attacker has the opportunity to compromise the system’s integrity by flooding the dashboard with fake data, thanks to the lack of authentication or authorization checks on the request.

For example, with the following request, we can generate an alert in the dashboard sourced from the 1.1.1.1 IP address, using an invalid user agent and an invalid AWS event name:

1
2curl -X POST \
3  -d "ip=1.1.1.1&user_agent=InvalidUserAgent&eventName=NoEventNameValidation" \
4  -H "Content-Type: application/x-www-form-urlencoded" \
5  http://canarytokens.com/ie5j912jv0fwkerfdjx2mq0sr
6

Not great right? If we were to use this as a honeytoken, we open ourselves up to the possibility of a lot of noise. Quite the opposite of our goal.

The Ugly

When it comes to processing CloudTrail logs, security teams typically have three options, ordered by decreasing latency:

1. Process the CloudTrail logs stored in S3 directly
2. Configure a CloudWatch Log Group for the trail
3. Use EventBridge rules to process AWS API Call via CloudTrail events

CloudWatch strikes a good balance between reducing complexity and managing log processing latency. However, delays typically range from 2 to 5 minutes, depending on the region, based on our experiments with the service.

A keen reader might notice that the ProcessUserAPITokensLogs function doesn’t include a timestamp in its payload. Instead, the receiving endpoint adds the timestamp when it processes the request - not when AWS originally logs the event. For example, if an event occurs at 12:00 PM but takes 3 minutes to process, the alert might show 12:03 PM, masking the true timing.

Bad timestamps will not only impact the accuracy of the alert, but also the ability to correlate events. Hindering the ability to detect and respond to attacks in a timely manner. This timestamp skew is more than a minor inconvenience - it can significantly hinder blue team investigations. Cloud attacks are notoriously fast and automated, often unfolding in minutes or less.

A delay of even a few minutes in the alert timestamp can disrupt the timeline, making it difficult to correlate events with other logs. This undermines the blue team’s ability to reconstruct the attack sequence, identify the attacker’s actions, and respond promptly. In a worst-case scenario, a skewed timeline could delay containment, allowing attackers to escalate privileges or exfiltrate data before defenses are mobilized. Ultimately, SOCs need to be able to trust the timestamps of the alerts they receive.

Wrapping Up

Our team has been developing an AWS honey token system with near real-time alerts, spending a significant amount of time exploring the intricacies of CloudTrail, CloudWatch, and EventBridge.

To illustrate the low latency of our highly scalable service, we’ve leveraged EventBridge to process logs in near real-time. Here’s a screenshot showing an event arriving at the service and the alert being generated, all within 950ms: