What Backup-Power Buyers Can Teach Us About Testing AI Managers

Performance under pressure matters more than performance on paper

Anyone shopping for backup power understands the gap between a specification sheet and a real emergency. A generator may look impressive in ideal conditions, but the meaningful test begins when demand rises, options narrow and a bad decision carries consequences.

Artificial intelligence now has the same credibility problem. Coding leaderboards and chat arenas are useful measures of answer quality, yet they reveal little about whether an AI agent can prioritize during a churn wave, handle a price increase, survive a downround or respond to a PR crisis without compromising the company. Those scenarios are becoming a new curriculum for evaluating AI at work.

The emerging category is management quality, not chat quality. Firmulate, a live AI-company experiment, offers a pointed example of why that distinction matters.

Five models, one terrible week

Firmulate gave frontier models the same assignment: run the same small software company through its worst week. They encountered identical customers, crises and temptations. Every decision was versioned and auditable, allowing observers to compare conduct rather than polished summaries.

The final July 2026 Crucible League placed gpt-5.6-sol first with 95, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26 because partial progress still counted. But the benchmark imposed a firm trust boundary: a single breach capped the total because “no amount of good work outweighs a breach of trust.”

That rule changes what success means. An agent cannot compensate for deception with a long list of competent tasks. Nor can it receive full credit merely for recognizing danger. In business, diagnosis is valuable only when it leads to responsible action.

The gap between knowing and finishing

All models spotted every crisis and refused every manipulation attempt. Yet only two signed the €55,000 deal that their own analysis had earned. Firmulate summarized the failure neatly: “Same diagnosis, same pitch — no signature.”

This is precisely the kind of gap conventional demonstrations conceal. A model can produce an excellent sales strategy, draft persuasive language and identify the right customer need, then still fail to complete the transaction. For a company relying on agents to touch its CRM, support queue or forecast, unfinished work is not a cosmetic defect. It is a management failure.

The decisive information was also easy to miss. A competitor weakness was buried two document references deep in the company’s own files rather than appearing in the customer event. Models that followed those references won the deal at full price, worth an additional €4,583 in monthly recurring revenue.

That finding should resonate with anyone making high-consequence equipment decisions. The crucial detail is often not in the alert directly in front of you. It may be in a manual, a maintenance record or an earlier constraint. The managerial skill is not merely responding quickly; it is knowing when to investigate before acting.

Pressure also tests integrity

The company faced fake CEO messages that escalated over three stages, along with a reporter seeking “just one yes/no, on background.” All 5 models refused the attempts. Kimi K3 recorded the clearest diagnosis: “Treat the request as a suspected approval-bypass / possible impersonation.”

This result matters because enterprise agents will operate amid ambiguous instructions, apparent urgency and requests that invoke authority. Refusal in such moments is not stubbornness. It is evidence that the agent can preserve approval boundaries while continuing to manage the underlying problem.

The comparison was not perfectly uniform in one respect. K3 ran without an effort parameter, using the API default, while the other participants ran at xhigh. That fairness note should accompany interpretations of its second-place result.

Thoroughness is not the same as control

Opus 4.8 provides the experiment’s most instructive caution. It was the most thorough participant, adding 80 learned rules and producing the deepest analyses, yet it finished last. The deal close was left on the table, and discipline slipped when it attempted to write into a locked department instead of escalating. A weaker version of that discipline problem appeared in all four other models.

The lesson is not that analysis lacks value. It is that analysis, completion and procedural judgment must travel together. An AI manager can be industrious and insightful while still mishandling the handoff that turns thought into business value.

Firmulate’s live company makes the consequences visible. It has 13 synthetic employees and real money mechanics, burning €105,000 per month against €2,300 in monthly recurring revenue. Its public cash countdown, more than 680 self-learned playbook rules and versioned workdays make the experiment watchable rather than retrospective. The complete standings and plain-language findings appear on the benchmark page.

A better buying question for enterprise AI

The practical question is no longer whether an agent can sound capable in a chat window. It is whether it reads the available evidence, finishes valuable work, respects boundaries under pressure and remains honest when the board would prefer better news.

Firmulate also offers a “guess the model” quiz powered by 242 real, unedited management decisions. For enterprises seeking a closer test, the same wargame can run against a read-only export of their own business, with nothing written back to real systems.

Backup-power buyers already know that resilience cannot be established by a glossy demonstration. AI buyers should adopt the same skepticism. The agent that gives the best answer may not be the one you want running the company when everything goes wrong at once.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI