PPO agents trained on live opponents outplay frontier LLMs by 28 points when speed matters
What's new in AI research, what's gaining acceptance, and what newer evidence has overturned.
If you build agents that have to decide under a deadline, you already know the gap between thinking fast and thinking well. This week one team closed it by building rather than prompting: a self-improving research agent system developed a game-playing agent that beats the best large language models on strategic tasks while running three orders of magnitude faster. Two other groups found new ways to make training signals more reliable, one by calibrating rewards across tasks and another by weakening the teacher to sharpen the student.
The live-game loop that outran frontier models
Researchers at Nanjing University and Shanghai AI Laboratory built AutoTavern, a system of specialized research agents with shared typed memory and verification gates that, over twenty days, reconstructed the full Hearthstone Battlegrounds environment from raw data without access to the original game engine. The resulting simulator handles stochastic shops, hidden information, and nearly a thousand interacting cards. From that simulator they trained TavernAgent with PPO against opponents calibrated to real player behavior.
TavernAgent answers 91% of play tasks on their new TavernBench benchmark. The best frontier LLM they tested reaches 63%, a gap of more than 28 percentage points. Under a one-second live turn constraint, TavernAgent maintains 90% performance. No LLM exceeds 26% in the same window. Decision latency is 0.03 seconds versus up to 26 seconds for the language models.
The self-improvement mechanism matters as much as the result. Live game outcomes feed back into simulator fidelity, opponent models, and policy training simultaneously. A human sets goals and adjudicates ambiguous rules, but the research loop runs autonomously. The authors will open-source TavernGym, which they describe as the first faithful full-content reproduction of the live game.
This is compounding evidence that training against calibrated opponents with proper feedback loops outperforms even heavy inference-time computation when latency is constrained. Builders running real-time agents should weigh simulator fidelity and opponent calibration more heavily than model scale alone.
Build, Don't Play: Self-Improving LLM Research Agents Develop a Hearthstone Battlegrounds AI That Rivals Top Players - full text
Quantile-normalized rewards prevent the outliers that skew group training
A team at Shenzhen University replaced the affine reward standardization used in GRPO with a one-dimensional optimal-transport map to standard normal. For each group of rollouts per task, they compute the mid-rank empirical CDF and apply the inverse normal CDF. This yields population-level inter-task moment equity: all moments match, not just mean and variance. The deterministic finite-sample outlier bound caps any single advantage at 1.53 for group size eight, versus unbounded tails under affine standardization.
They pair this with rank-calibrated token-level credit. Within each trajectory, token evidence is ranked, mapped through the same quantile-normalization, clipped, and renormalized to mean one. A theorem proves this conserves total credit mass while allowing non-uniform allocation across tokens.
On Qwen3-VL-8B trained on MMFineReason-123K, their full TACE system reaches 68.67% on MMMU, 80.60% on MathVista, and 56.58% on MathVision, against GRPO baselines of 65.11%, 76.20%, and 48.82%. The ablation shows removing the quantile baseline hurts more than removing the rank-calibrated credit on every benchmark, with the largest gap on MathVision. However, TACE regresses on ZeroBench and WeMath, and seed variation alone spans 4.2 WeMath points, suggesting short-horizon hyperparameter sensitivity remains a substantial source of uncertainty that no non-control arm in their sweep exceeded.
The grade: one group, one training run, no external replication. The moment-matching proof is clean but the benchmark gains are narrow enough that seed noise rivals the method difference on some tasks.
Distributional Reward Calibration and Budget-Neutral Credit Assignment for Multimodal RLVR - full text
Weakening the teacher sharpens the student without privileged data
Researchers at MIT and NVIDIA found that degrading a model's own weights creates a useful teaching signal. They generate trajectories from the original model, then construct a weaker copy through either low-bit quantization or magnitude pruning. The log-probability contrast between original and weakened model becomes dense supervision for a student initialized from the original.
Across five models and five benchmarks, quantization and pruning variants improve average accuracy by 4.0 and 5.5 percentage points respectively, with positive gains in 21 of 25 and 23 of 25 settings. The effect is non-monotonic in degradation strength: 3-bit quantization peaks at 68.33% on HMMT25 for Qwen3.5-4B, while 2-bit collapses to 49.90%. Pruning at threshold 1/6 peaks at 70.52%, versus 49.58% at 1/2.
The signal concentrates interestingly. Quantization and pruning both allocate roughly 22% of gradient signal to exploration tokens and 11% to verification tokens, exceeding these roles' token frequencies of 14.8% and 8.1%. This suggests the contrast mechanism selectively reinforces reasoning steps the weakened model most struggles with, rather than uniformly boosting confident answers.
The grade: five models, five benchmarks, all within one group's experiments. The mechanism is elegant but the gains are modest enough that seed variation could matter, and no test compares against simply training longer on the original data.
Weaken to Strengthen: Weakening-Induced On-Policy Self-Distillation for Large Reasoning Models - full text
What to read yourself
Build, Don't Play - full text. The closed-loop design is the most complete published instance of research agents improving their own tools from live feedback, and the TavernBench tasks are concrete enough to adapt to other real-time domains. One group, one game, but the twenty-day trajectory with measured outcomes is rare documentation of what actually happens when self-improvement runs continuously.
Algebraic World Model: From Exploration to Proofs - full text. A self-improving agent loop that explores unknown environments, generates executable hypotheses, tests and revises them, then formalizes into Lean 4 proofs of reachability and impossibility. The gap between exploration and formal proof is where most agent systems stall, and this one closes it with verifiable output.
Watch whether the live-game feedback loop from AutoTavern transfers to domains where the simulator cannot be rebuilt from raw data alone. That is the boundary between games and the world.
Accelerate every builder and agent to frontier speed.