Reinforcement learning · information theory · Python, JAX IMOL 2025 workshop · extended first-author manuscript in preparation
An open-ended learning agent has to invent its own goals. But with a limited amount of experience, it constantly faces one decision: spend the next steps discovering a brand-new goal, or getting reliably good at a goal it already knows? Existing intrinsic-motivation methods push toward one of these (novelty or competence) but have no principled way to trade them off.
I use empowerment — the mutual information between the goals an agent can choose and the outcomes it actually reaches — as a single currency for that decision. It splits cleanly into two terms, both measured in bits:
Diversity: how many distinct outcomes the agent can reach (raised by discovering a new goal);
Competence: how reliably it lands on the goal it intended (raised by practising a known skill).
Because both are in the same units, the agent can estimate the expected empowerment gain of "discover" versus "improve" and simply pick the larger. The discover-vs-master trade-off becomes an explicit calculation instead of a hand-tuned heuristic.
Environments: a pure-Python GridWorld and a JAX/JIT xland-minigrid backend, with custom key-and-door layouts and per-state action noise to create goals of genuinely different difficulty.
Skills: tabular Q-learning / Dyna-Q, including a shared goal-conditioned policy with hindsight experience replay.
Empowerment estimator: tracks per-goal rollout statistics and computes the ΔEmp predictions that drive goal selection.
Curriculum + evaluation: a phase-based training loop against fair baselines (uniform random, an H-GRAIL-style competence baseline), and a zero-shot evaluation that composes learned skills through a policy graph to reach distant goals.
The selection mechanism demonstrably does the right thing — it stops investing in goals that are already mastered or that are stuck — and fixing two silent correctness bugs (below) reversed several earlier conclusions. The current experimental campaign (multiple seeds per condition, checkpointed reachability-over-training curves) is producing the quantitative results, which will be added here as they are finalized.
Two of the corrections were silent bugs — a tie-breaking rule that made 97–99% of goal selections deterministic, and a noise setting the experiment runner was ignoring. Neither threw an error; the numbers just looked plausible and were wrong. Both were found by logging the distribution of decisions rather than trusting the aggregate outcome, and fixing them reversed several earlier conclusions.
Workshop paper: Empowerment for Goal-driven Open-Ended Learning — N. Catenacci Volpi, K. Archer, E. Dmitruk et al., IMOL Workshop, 2025.
Extended first-author manuscript (CoLLAs) in preparation.
Code will be released with publication.