An AI cluster cut its power 25% for three hours on cue from Phoenix utilities

PHOENIX, May 2025 — A cluster of 256 AI chips in an Oracle data center cut its power use by 25% for three hours during two utility peak events in May 2025, using only software, its developers report. Here's what happened, what it cost, and what went wrong.

Cost and outcome (as of July 1, 2025 (preprint))

Outcome: 25% power reduction held for three hours in both utility events, with all jobs inside their service limits, per the arXiv preprint.
Scale: one 256-GPU cluster.
Cost: none disclosed; no hardware or batteries were added. (the arXiv preprint)
Status: preprint, not peer-reviewed.

What happened

The trial ran at an Oracle Phoenix Region data center on 256 NVIDIA A100 GPUs, orchestrated through Databricks MosaicML, with Amperon grid-demand forecasts. Four workload ensembles mixed training, inference and fine-tuning jobs. (the arXiv preprint)

In consultation with Arizona Public Service and Salt River Project, the team ran two events: May 1, 2025, for the APS system peak, and May 3, 2025, for SRP's. Each required a 25% cut from average base load, held for three hours, with 15-minute ramps down and up to avoid a "snap back." In both, all jobs completed within their allowed service-level envelopes, the authors report. (the arXiv preprint)

On May 3, SRP's system peaked at 6 p.m., and the cluster ramped back up without exceeding its original baseline, NVIDIA wrote. (NVIDIA's blog)

How it works

Jobs are tagged into flexibility tiers: Flex 0 allows no slowdown, Flex 1 up to 10% lower average throughput over three to six hours, Flex 2 up to 25%, Flex 3 up to 50%. Conductor then slows chips, pauses jobs or reallocates resources. (the arXiv preprint)

Across 33 experiments of three to six hours, covering 212 jobs, the authors report zero service-level violations. The Emerald Simulator's power predictions had a 4.52% root-mean-square error relative to average experiment power. (the arXiv preprint)

They also replayed California ISO's August 2020 emergency: a 15% cut followed by a further 10% step-down, both delivered. (the arXiv preprint)

"AI factories can flex when the grid is tight — and sprint when users need them to," said Varun Sivaram, Emerald AI's founder and CEO. (NVIDIA's blog)

Who was involved

Oracle Cloud Infrastructure supplied cluster power telemetry, and Databricks chief AI scientist Jonathan Frankle guided the choice of workloads and their flexibility thresholds. "Often, a small percentage of jobs are truly non-preemptible," Frankle said. (NVIDIA's blog)

Anuja Ratnayake, who leads EPRI's DCFlex Consortium and is a co-author, said the trial "validates the vast potential of an essential element in data center flexibility." Inference that can't wait could instead be redirected to a data center on a less stressed grid, NVIDIA wrote. (NVIDIA's blog; the arXiv preprint)

Why utilities care

SRP set the 25% target because Arizona is among the top states for data center growth, NVIDIA wrote. "This test was an opportunity to completely reimagine AI data centers as helpful resources," said SRP president David Rousseau. (NVIDIA's blog)

The authors argue flexible data centers could connect faster and avoid infrastructure upgrades needed to serve higher peaks. (the arXiv preprint) NVIDIA notes Texas passed a law requiring data centers to ramp down or disconnect at utilities' request during load-shed events. (NVIDIA's blog)

What went wrong

Not peer-reviewed. The results are a preprint; 10 of its 15 authors work for Emerald AI.
One cluster. The paper says larger deployments with full data center telemetry and multiple zones are needed. (the arXiv preprint)
Not every job can wait. The authors acknowledge that customers with strict latency or reliability needs may not accept flexibility. (the arXiv preprint)
What we don't know yet: whether the approach holds at a full data center, and what utilities would pay for it.

Our follow-up question for Emerald AI

Has the preprint been submitted for peer review, and have trials moved beyond a single cluster?

Sources

Colangelo, P., Coskun, A. K., et al., “Turning AI Data Centers into Grid-Interactive Assets: Results from a Field Demonstration in Phoenix, Arizona,” arXiv:2507.00909 (preprint), July 1, 2025. arxiv.org
NVIDIA, “How AI Factories Can Help Relieve Grid Stress,” NVIDIA Blog, July 1, 2025. blogs.nvidia.com

Disclosure: [not in copy] Editor placeholder