AWS Emulator was written entirely by AI agents. Its own scores say it matches real AWS on 57% of tests

Amazon released AWS Emulator on November 11: a free, offline copy of the AWS APIs that runs on your own machine so coding agents (and people) can test against something that behaves like AWS without touching a real account. The code is public at aws/aws-emulator, dedicated to the public domain under CC0, and Amazon's headline claim is about who wrote it. "Our team hasn't written or reviewed any of the code," Matt Goldberg, the Senior Principal Engineer who started the project, writes in the launch post. Swami Sivasubramanian, in a guest post on Werner Vogels' blog, says it is built from more than 400 API models covering more than 200 services.

What makes this more than a launch is that AWS published the means to check it. Every release ships a scores.json.gz file with the result of each fidelity test, the test suite itself sits in aws/aws-emulator-evaluation-suite, and the agents' entire issue backlog is committed to the repository. We downloaded the scores for the latest release, v1.0.322, and joined them with the suite's manifest.jsonl. The short version: the emulator matches real AWS on 24,243 of 42,857 scored tests, or 56.6%, and the parts most developers would lean on hardest are among the weakest.

How the agents built it

The method is the interesting part, and it is simple to state. AWS has a written specification for every service, the public Smithy API models, and a reference implementation, the AWS cloud itself. Goldberg's first prompt to Kiro, Amazon's coding agent, was three lines: write tests from the AWS CLI, iterate until they pass against real AWS, then build an in-memory implementation that passes the same tests. He says Kiro returned a passable Amazon SQS emulator in about 15 minutes, and the rest of AWS "over the weekend".

The production version splits the work across three teams of agents. One writes fidelity tests from the API models and verifies each against real AWS (in us-west-2, or us-east-1 for global services, per the fidelity docs), so a test records how AWS actually behaves. A second replays the tests against the emulator and files an issue for each group of failures it thinks shares a root cause. A third takes one issue each, fixes it, and has to get through a merge gate: compilation, a coding-standards reviewer, several security agents, and a check that fidelity didn't regress. Goldberg says "hundreds of agents" work concurrently and "thousands of issues" get fixed a day. The result is Rust, one crate per AWS service, 355 services in the current scores file.

Sivasubramanian's own anecdote is the best argument for why this works. He used Kiro to rebuild the replication engine of Alf, the Paxos-based lock service he and Al Vermeulen built two decades ago and that still sits under S3 and DynamoDB. He supervised the agent for four nights before realising it "had no way to check its results". Given a spec and a test environment, the rebuild took about two hours. The lesson is less about the agent than about the oracle: an agent with a ground truth to test against behaves very differently from one without.

What the scores file says

AWS defines a service's fidelity score as the number of its fidelity tests the emulator passes. A test passes only if every call in the workflow matches AWS: status code, resulting state, side effects, and for invalid calls, the exact error. That is a strict bar, which is fair to keep in mind. Here is what v1.0.322, scored on November 11 at commit 7753983, shows for the services people actually use (skipped tests excluded):

Across all 355 services the median is 60.5%. Twelve services pass everything they are tested on, 21 pass nothing, and 133 are below half. A service counts once per test it appears in, so multi-service tests are counted for each service involved.

Three patterns stand out.

Single calls work better than workflows. Tests that touch one service pass 59.2% of the time. The 6,569 scored tests that span two or more services, such as an S3 event landing in SQS or a CloudFormation stack provisioning a Lambda function, pass 41.9%. Those cross-service flows are exactly where a local emulator would save the most time, because they are slow and annoying to set up for real.

The control plane is ahead of the data plane. AWS says this itself: the docs state that the emulator "matches AWS most closely for control plane APIs". The manifest labels let you put a number on it. Of the 29,255 behavioral tests we could match to a manifest entry, control-plane tests (create, describe, update, delete, list) pass 67.3%, data-plane tests (putting and getting objects, sending and receiving messages) pass 53.7%. Creating a bucket is easier to emulate than being a bucket.

Infrastructure as code is the weak spot. CloudFormation, at 19.0% across 4,186 tests, is the most-tested service in the file and close to the bottom. The docs warn that support for CloudFormation, Terraform and the AWS CDK "varies by service", which in practice means a CDK app is one of the riskier things to point at it today.

Permissions are the gap to know about

The README says not to use the emulator "to test security", and the backlog explains why that sentence matters. An issue the agents filed on October 22, still in issues/things-that-do-not-work/ as of this release and marked Issue-Severity: HIGH, is titled "No IAM authorization is evaluated". It describes a role whose only permission is sts:GetCallerIdentity successfully calling iam:CreateUser, and spells out the consequence: "every least-privilege test silently passes" and "a role that is misconfigured in production looks correct against the emulator."

Some of that has been patched service by service since. The v1.0.322 release notes include fixes such as iam-sns-sns-performs-no-iam-authorization-publish and apigatewayv2-deleteportalproduct-performs-no-authorization. But the backlog still holds 118 open issue files with "authoriz" in the name, and a rough cut of the scores, the 737 tests whose IDs mention denial, deny or unauthorized access, shows 337 passing, or 45.7%. That is a keyword filter, not an audit, but it points the same way as the issue text: if your agent writes an IAM policy and checks it against the emulator, a pass tells you little.

This is the practical risk with an emulator built for coding agents in particular. A human who gets a suspicious 200 might go and check. An agent that sees its test pass will usually move on.

"Fidelity tests that pass don't regress"

The fidelity docs make one claim we could test directly: "Fidelity tests that pass don't regress." We compared v1.0.322 with v1.0.290, scored about 17 hours earlier on November 10. Of the 42,537 tests present in both, 651 went from failing to passing. Another 204 went the other way, from passing to failing: 38 involving S3, 33 RDS, 25 SQS, 23 Lambda.

That does not necessarily mean 204 regressions in the code. The same docs say a test can be rebuilt or updated within a test-suite version when AWS changes behavior, and that can turn a pass into a fail. The trouble is that the published scores file records the emulator commit but not the test-suite version, and the notes for v1.0.290 and v1.0.322 don't name one either, although the docs say each release does. From the outside, a rebuilt test and a genuine regression look identical. Net, the day was clearly positive, 447 more passing tests among the shared set, plus 400 new tests and 97 retired ones. But the no-regression promise is not something a user can verify from what is published.

The backlog as a record

The issues/ directory is the most unusual artifact here. It holds 12,708 issue files in things-that-now-work and 6,544 in things-that-do-not-work, each with a severity, the failing test IDs, the commit it was found at, the recorded AWS behavior, and the emulator's behavior. 934 of the open ones still have "stub" in their name, meaning an operation that answers HTTP 501 NotImplemented. A handful of fixed issues carry a human- prefix, presumably filed by people rather than the agents, such as "lambda-invoke-must-run-nodejs20-x-and-nodejs22-x-handlers".

Goldberg says his favourite moment came from an earlier version of the emulator, when the agents "decided to build their own Python interpreter to support AWS Lambda functions." They have since been told to reuse existing libraries. The repository does not take pull requests; outside users are asked to report differences as GitHub issues.

What to make of it

The real achievement is the loop, not the percentage. AWS turned its public API models and its own cloud into an oracle, and an unsupervised agent swarm produced a 355-service emulator with tens of thousands of machine-checkable claims about where it is right and wrong, published down to the individual test.

But "100% built by agents" and "a local copy of AWS" are two different claims, and only the first is fully true today. At 56.6% overall, around two-thirds for the core control-plane services and much lower for CloudFormation, EC2 and multi-service flows, AWS Emulator is a promising test double whose own data tells you where to trust it. Use its fidelity scores, or the aws-emulator agent skill AWS ships to check a project against them, before treating a green test run as proof.