AI’s True Test: Will It Finish What It Starts? Insights from a Real Business Experiment

In sports, the true measure of a champion isn’t just their skill in practice but their ability to close out the game when it matters most. The same applies to AI in business. A new public experiment shows that while AI models can spot problems and refuse to be manipulated, only some can actually follow through and seal the deal—revealing a critical, invisible weakness behind much of today’s AI hype.

How AI Models Were Put to the Ultimate Business Test

In a groundbreaking live experiment, four leading AI models faced the same challenge: run a small software company through its worst week—crises, manipulations, and the ticking clock of real money. The company, which is actively losing cash and running everyday operations, provided a challenging environment designed to test more than just chat skills. Every decision made by the AI was recorded and auditable, giving a transparent window into its true capabilities.

The Models and the Stakes

• gpt-5.6-sol scored 95 out of 100 and uncovered critical hidden information in internal files, leading to a full-price deal worth over €4,583 monthly recurring revenue (MRR).
• Kimi K3 scored 93, the highest discipline among models, and also closed the deal, showing clean decision-making.
• Sonnet 5 scored 88, closed the deal too, but with minor process slip-ups.
• Fable 5 scored 77 and, despite rule discipline, left the deal on the table, highlighting a discipline gap.

The baseline, a do-nothing approach, scored 26, emphasizing that progress is tangible but not enough—trust is the ultimate currency that no model can afford to breach.

What Does This Mean for Business AI?

While all models demonstrated they could identify crises and refuse manipulative tactics—such as fake CEO messages—the decisive factor was whether they could execute tasks they earned or simply identify problems. Only two models managed to sign the deal at full price, a feat that depends on ‘closing strength,’ a trait difficult to measure in traditional chat demos.

The Hidden Weakness: Reading Deeper Files

The key to winning the deal was reading deep into the company’s internal documentation—information buried two document references deep in the files. Models that successfully read and integrated this information were able to make the right decisions and close confidently. This reveals that surface-level assessments or quick chat demos are insufficient. The real test lies in whether the AI can interpret complex documents and follow through with strategic action.

Resistance to Social Engineering and Manipulation

The experiment also tested social engineering tactics—fake CEO messages escalating through multiple stages, along with a reporter trick requesting a quick approval on background. All models refused these attempts, with Kimi K3 explicitly reasoning that such requests could be impersonation or approval-bypasses. This resilience is critical for real-world applications where manipulative tactics are common.

Lessons for Business Leaders

The experiment’s live component, at firmulate.com/live, offers a rare glimpse into how these models perform under real business conditions. The small company, with 13 synthetic employees and over 680 self-learned rules, operates daily with real money mechanics—burning €105k/month against a modest €2.3k MRR. Watching these AI models navigate daily crises reveals that the capability to finish tasks, read complex documents, and stay disciplined under pressure is crucial—and often invisible in traditional demos.

The Disciplinary Gap and Its Implications

Interestingly, the model with the deepest analysis, Opus 4.8, finished last in terms of closing the deal. Its discipline slipped, with attempts left in a locked department instead of escalation, illustrating that thoroughness does not always equate to execution strength. Meanwhile, K3 balanced discipline and decisiveness, closing confidently, highlighting that effective AI management requires more than just rule adherence.

Why This Matters for Your Business

The key takeaway is that AI’s real value isn’t just in generating convincing chat responses but in its ability to complete meaningful work—reading, understanding, and acting on complex information reliably. As the experiment shows, the difference between an AI that merely identifies problems and one that can finish what it starts can be the difference between losing and winning a critical deal.

To see how your enterprise can prepare for this new era, consider running similar tests with your own data, using tools like the public wargame platform at firmulate.com/benchmarks.html. It’s not just about testing AI’s superficial skills but assessing its true ability to execute under pressure and stay honest amidst manipulation.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI