An Exploratory Analysis of the Multi-Armed Bandit Problem

This paper analyzes individual behavior in multi-armed bandit problems. We use a between-subjects experiment to implement four bandit problems that vary based on the horizon (indefinite or finite) and number of bandit arms (two or three). We analyze commonly suggested strategies and find that an overwhelming majority of subjects are best fit by either a probabilistic ‘win-stay lose shift’ strategy or reinforcement learning. However, we show that subjects violate the assumptions of the probabilistic win-stay lose-shift strategy as switching depends on more than the previous outcome. We design two new “path-dependent” strategies that adapt reinforcement learning and myopic behavior by incorporating a bias towards choosing the previous arm. We find that a majority of subjects are best fit by these strategies, which suggests that subjects respond intuitively to bandit arm outcomes, but also exhibit persistence in their choices