AWS Rewrites AgentCore Runtime for Elastic Memory and Adds GPU-Aware Inference Routing
AWS rebuilt AgentCore Runtime for elastic memory and 1.9-second cold starts, and added a GPU-aware SageMaker HyperPod gateway for EKS.
AWS has rebuilt the compute layer beneath its agent platform, pairing a reworked AgentCore Runtime with a GPU-aware routing add-on for Kubernetes. The updated runtime, generally available in five AWS regions, holds cold starts at 1.9 to 2.0 seconds at the 75th percentile across container images ranging from 200 MB to 2 GB, and it reclaims memory the moment a session releases it. Customers pay for what they use rather than the peak their agents once reserved.
The second piece ships alongside it. SageMaker HyperPod Inference Gateway is a Kubernetes-native add-on for Amazon EKS that routes inference requests using live GPU telemetry. AWS puts the reduction in first-token latency at up to 82%, achieved without changes to model servers or client applications.
Both products attack the same bottleneck from opposite ends. The AgentCore Runtime decides where agent sessions execute. The inference gateway decides how model requests find a pod. Together they cover the two points where production agents stall: start-up time and time-to-first-token.
What Changed in the AgentCore Runtime
The previous generation allocated memory for the worst case and charged for it. The V2 runtime reverses that model. Sessions return memory when they finish, and billing tracks actual consumption. For teams running bursty agent workloads, the gap between reserved and used capacity is the gap between provisioning for a Monday morning spike and paying for a Tuesday afternoon lull.
Cold start behavior is the more technically interesting change. AWS holds start times flat regardless of container image size or concurrency, using environment snapshots rather than warming up a complete image. An agent packaged into a 2 GB image starts in roughly the same window as a 200 MB one. That removes a constraint that shaped how teams packaged agents. With a fixed start-time budget, image size stops being a design variable.
Pricing is pay-per-use with no pre-provisioning. The runtime launched on September 18, 2026 in us-east-1, us-east-2, us-west-2, eu-west-1 and ap-northeast-1.
AWS also published a reference architecture for running agent tool calls inside isolated Lambda MicroVMs. Each agent session gets a fresh VM, which keeps credentials, networking and governance inside the customer's own account. The design targets the compliance question that stalls enterprise agent rollouts: who can see the tool call, and where the audit trail lives. A per-session VM gives security teams a boundary they can point to rather than a shared process they have to reason about.
The MicroVM path and the two runtime models together cover three isolation postures, from per-session VM to shared persistent host. Choosing among them is now a workload decision rather than a one-time architecture decision, and it can be revisited as an agent's session profile changes.
Two Runtime Models, Two Workload Shapes
The elastic runtime is not the only option. AWS made AgentCore runtime instances generally available on August 6, 2026, a managed EC2-backed compute option for agents that need to hold state across days. The contrast between the two is the clearest signal of how AWS expects agent workloads to split.
| Dimension | AgentCore Runtime (V2) | Runtime instances |
|---|---|---|
| Session ceiling | 8 hours | Up to 14 days |
| Hardware | Managed, elastic memory | GPU, memory or compute families via capacity provider |
| Pricing | Pay-per-use, no pre-provisioning | Persistent managed EC2 |
| Best fit | Short, bursty, stateless turns | Multi-agent collaboration on shared hosts |
The trade-off is explicit. Elastic sessions cost less when idle and reset cleanly, but they expire. Persistent instances carry the cost of always-on infrastructure and return GPU access, multi-agent coordination on shared hosts, and state that survives a weekend. A support agent handling a single ticket belongs in the first column. A research agent that accumulates context over a multi-day investigation belongs in the second. The 8-hour ceiling on the default microVM runtime is the dividing line, and it is a hard one: an agent that needs hour nine has to move.
Migration cost is low on paper because both runtime models share the same AgentCore APIs, identity controls and observability surfaces. A team that built against the microVM runtime can move a workload onto persistent instances without rewriting its tool layer. That continuity matters more than any single feature, because agent projects often stall when each new infrastructure requirement forces another rewrite.
The GPU-Aware Inference Gateway
HyperPod Inference Gateway attacks a different problem, the gap between a GPU that is busy and a GPU that is merely allocated. Standard Kubernetes routing balances requests without knowing whether a pod's KV cache is saturated, how deep its queue runs, which LoRA adapters are resident, or how well the prefix cache is hitting.
The gateway reads those four signals and sends each request to the pod best positioned to answer it. LoRA adapter affinity matters most for multi-tenant deployments. Keeping a request on the pod that already holds the relevant adapter removes the swap latency that otherwise interrupts inference, which is why AWS describes adapter residency as a routing input rather than a scheduling afterthought.
The architecture runs in two tiers. A per-cluster gateway handles Tier 1 and is available as of September 18, 2026, delivered as an EKS managed add-on. A Global Inference Router forms Tier 2 and is described as coming soon. The gateway is compatible with OpenAI-standard endpoints, which lowers adoption cost for teams already standardized on that interface.
Where the Strategy Points
Read together, the releases describe an attempt to close a gap AWS itself quantifies. On its own build-with-AI page, AWS states that 88% of AI agent pilots never reach production. That number frames everything above. The obstacles that keep agents in notebooks are rarely model quality. They are start-up latency, memory bills that scale with peak rather than use, GPU routing that ignores cache state, and the absence of a session model that matches how long real work takes.
The competitive context sharpens the point. Agent infrastructure is a crowded field, with every major cloud offering some combination of managed runtime, tool gateway and identity layer. The differentiator here is granularity. AWS is selling control over cold starts, memory reclamation and per-request GPU placement rather than one abstracted platform. That is a harder sell to a team that wants a single button and a stronger one to a platform team already burned by opaque scheduling.
The costs deserve naming. Persistent runtime instances reintroduce the always-on spend that serverless pricing was meant to eliminate, so teams need workload-level judgment about which agents deserve a 14-day session. The gateway's most capable tier, the Global Inference Router, is not yet available, which leaves multi-cluster routing a promise rather than a shipped capability. Pay-per-use memory reclamation rewards workloads with clear session boundaries; an agent that holds context indefinitely will not see the savings.
For platform teams, the practical sequence is to audit agent workloads by session length and GPU need, then split them across the two runtime models rather than forcing one. The elastic AgentCore Runtime suits the long tail of short tasks. Runtime instances suit the few agents where state continuity is the product. On the inference side, Tier 1 of the gateway is available now and requires no model-server changes, which makes it a low-risk test for teams already running on EKS.
Why this matters
Agent infrastructure is shifting from whether a company can run an agent to whether it can run thousands of them profitably. The AgentCore Runtime changes move the cost curve and the latency floor at the same time, and the inference gateway addresses the layer above by treating GPU cache state as a routing input rather than an implementation detail.
For decision-makers, the signal is that AWS is competing on operational specifics rather than model access. The teams that benefit first are those already on EKS and Bedrock with measurable latency and memory problems. The ones that should wait are those whose agents have no clear session boundary, since neither pricing model rewards that shape.
Sources
The new AgentCore runtime: Elastic, optimized, and consistently fast starts
Introducing Amazon SageMaker HyperPod Inference Gateway
Amazon Bedrock AgentCore - AWS
AgentCore runtime instances are now generally available - AWS
Migrate agentic workloads to Amazon Bedrock AgentCore | Artificial Intelligence
Amazon Bedrock AgentCore FAQs - AWS
Build & Ship AI Agents: From Pilot to Production – AWS
AI-generated image.
Related Articles
- NVIDIA Brings Nemotron 3 Ultra to AWS to Power High-Efficiency Autonomous Agents
- AWS Rebuilds Amazon OpenSearch Serverless for Agentic AI and Dynamic Workloads
- AWS Debuts Multi-Turn Reinforcement Learning to Refine AI Agent Performance
✔Human Verified
Researched and cross-referenced against primary sources by the Bytevyte editorial team. This article was generated with the assistance of artificial intelligence and reviewed by the Bytevyte editorial team.