Bayesian Ensemble (BE): Adaptive Ensemble Selection for Bandits and Reinforcement Learning

We address sequential decision‑making under uncertainty, where traditional ensemble‑based Thompson Sampling fixes the sampling distribution over ensemble members, ignoring that members differ in quality as training progresses.

We propose Bayesian Ensemble (BE), a lightweight Bayesian layer added on top of any ensemble. In each round the agent maintains the ensemble parameters θ and a distribution p over member indices z . The index is sampled, the selected member scores actions, and after observing reward both θ and p are updated. Updating p directly with respect to cumulative reward, rather than via a surrogate loss, yields a Bayesian decision loop that learns “which model to trust”.

Bayesian Ensemble Bandit (BEB)

In contextual bandits we replace the uniform member selection with a Beta‑Bernoulli posterior for each member. The Beta parameters are updated conjugately from observed rewards, allowing the index distribution to adapt automatically. Experiments on Neural Testbed (action dimensions d=2,10,50, 20 000 steps) and Mushroom show that BEB reduces regret by 37.0 %, 12.8 % and 42.2 % for the ensemble+ baseline and by 69.8 %, 22.8 % and 30.3 % for the hyper‑model baseline. On the Mushroom dataset the improvement reaches 8.7 % (ensemble+) and 4.8 % (hyper‑model).

Bayesian Ensemble DQN (BE‑DQN)

For reinforcement learning we maintain k independent Q‑networks, each paired with a Beta distribution over its index. At each step a network is sampled, selects the action with maximal estimated reward, and the target Q is constructed as a weighted aggregation of all networks. Theoretical analysis on a one‑direction MDP shows that the Q‑value variance upper bound matches standard DQN while the lower bound matches Ensemble DQN.

Empirical evaluation on MiniGrid (FourRooms, Empty‑6×6, LavaGapS5, GoToDoor‑5×5, MultiRoom‑N2‑S4) compares BE‑DQN with vanilla DQN, Ensemble DQN, Random Ensemble DQN and UAAC. BE‑DQN attains the highest or most stable average reward across all tasks, attributed to its dynamic weighting of Q‑estimators.

Real‑world recommendation (Yahoo!R6B)

On the Yahoo!R6B news‑recommendation log (28 M impressions) we evaluate cumulative clicks. Hypermodel + BEB achieves 50 322 ± 1 487 clicks, surpassing the original hypermodel (49 677 ± 1 356) and beating Random, ε‑Greedy, MC‑Dropout and Gradient‑TS. A 1 M‑sample subset shows ensemble+ + BEB improves regret by ~3 % and scales better as ensemble size grows (regret reduction rises from 28.23 % to 47.97 % when size increases from 25 to 100).

Computation overhead is negligible for ensemble+ + BEB because the Beta‑Bernoulli update is conjugate; hypermodel + BEB adds about 20 % extra time due to variational inference.

Overall, Bayesian Ensemble unifies ensemble learning and Bayesian index selection, delivering consistent gains in bandit, reinforcement‑learning and large‑scale recommendation settings without significant cost.