AWS Releases Benchmark to Test AI Agents on Live Cloud Tasks
AWS has released a new open-source tool that grades AI agents on their ability to manage live cloud infrastructure, moving beyond static code tests to measure real-world operational safety and cost.
For two years, cloud providers have promised that artificial intelligence agents could manage complex infrastructure. On July 24, 2026, Amazon Web Services finally provided a way to verify those claims. The company released aws-bench, an open-source research preview that places AI agents into disposable, live AWS accounts to test their ability to troubleshoot and configure resources. This marks a significant shift from theoretical capabilities to measurable, practical performance in a high-stakes environment.
The release was not a major keynote announcement. It appeared quietly in AWS’s news feed and only gained traction in developer circles by late August. This slow rollout reflects the tool’s nature: it is infrastructure for a problem the industry has long worried about but lacked a standard way to measure. The core question aws-bench answers is whether an AI agent can safely operate a cloud account without breaking existing systems or incurring unnecessary costs.
Testing Real-World Cloud Scenarios
The benchmark suite consists of more than 300 tasks derived from real-world usage patterns. These are not simple coding exercises. They include resolving identity and access management policy conflicts, configuring multi-region network peering, and fixing broken Kubernetes deployments. These are the types of issues that typically land in an on-call engineer’s queue during off-hours. By using natural language requests and specific starting states for cloud resources, the tool tests an agent's ability to navigate complex, interdependent systems.
A command-line tool ships with the benchmark, allowing developers to spin up test environments and run agents against these scenarios. The workflow involves selecting an agent framework and a foundation model to see how the pairing performs. Because the tests use live accounts with generated resource IDs, agents cannot simply memorize answers. They must interpret the current state of the system and apply the correct fixes dynamically.
Beyond Simple Correctness Metrics
Traditional benchmarks often rely on static fixtures, such as a frozen codebase or a snapshot of a desktop. An agent gets one shot to produce an output, which is then compared to a known-good answer. This works well for coding tasks but fails to capture the risks of operating a live system. In a live cloud environment, state changes the moment an action is taken. A fix that works in a static test might cause a cascading failure in a production environment.
aws-bench addresses this by scoring performance on three dimensions: task completion, cost efficiency, and security compliance. An agent might resolve a misconfiguration by deleting a resource and rebuilding it from scratch. While this might technically fix the immediate issue, it could incur high costs or create a wider blast radius than a targeted fix. By evaluating cost and security alongside correctness, the benchmark catches gaps that static tests miss. This is crucial for enterprises that need to ensure AI agents do not introduce new vulnerabilities or financial waste while solving problems.
Foundation in Existing Frameworks
The tool is not built from scratch. It is constructed on top of the Harbor agent-evaluation framework, an open-source project also used for other agent benchmarks. This foundation provides the sandboxing and evaluation logic necessary to run agents in isolated environments with scoped credentials. The use of an established framework ensures that the testing environment is consistent and secure, allowing developers to trust the isolation between test runs.
The trade-off for this robust testing capability is complexity. Setting up and interpreting the results requires a deeper understanding of cloud infrastructure than typical software testing. However, for organizations considering delegating operational tasks to AI, this level of scrutiny is necessary. The release signals a maturing industry that is moving from hype to rigorous validation. As AI agents become more common in IT operations, standardized benchmarks like this will be essential for determining which tools are safe for production use.
