Back to Home
arXiv AI··Papers & Tech

Visualizing the World Model in an Agent's Value Function

中文摘要

模型无关强化学习可能学习环境模型。该研究通过反事实干预,揭示了 PPO 代理的价值函数在 Crafter 环境中形成了世界模型。

English Summary

Model-free RL may learn environment models. This study uses counterfactual interventions to reveal that a PPO agent's value function forms a world model in the Crafter environment.

Original Excerpt

arXiv:2610.14186v1 Announce Type: new Abstract: Could model-free reinforcement learning algorithms actually be learning a model of the environment? In this work, we apply counterfactual interventions which hold everything about a state fixed, besides a targeted change, to explore if the value function of a Proximal Policy Optimization agent forms a world model of the Crafter environment. Despite only being trained to optimize reward, we discover that the value function learns concepts such as the rules governing when a diamond is collectible, water and food having an impact on health, heuristics regarding inventory, and more. Finally, we discuss the safety implications of our visualizations and some shortcomings of current reinforcement learning agents that will prevent us from building truly advanced machine intelligence.