Bugs that throw are easy. The expensive ones run cleanly and return plausible numbers that happen to be wrong. Two of those turned up in an empowerment-driven RL project; both sat in the goal-selection step, so everything downstream inherited them.
Action choice was an argmax over option scores. Early in training many options score identically, so the tie-break decides everything — and Python's max returns the first maximal element, deterministically. In practice 97–99% of goal selections went to the same option, while the code looked like it was exploring.
Nothing failed. Learning curves still rose. It only showed up when I logged the distribution of selections instead of the aggregate outcome: a mechanism meant to be broadly exploratory was nearly deterministic.
The per-state action noise setting wasn't being parsed by the experiment runner. Every "noisy environment" run was actually deterministic. The value was in the config file; it just never reached the environment, and a missing value fell back to a plausible default, so nothing complained. It surfaced when a result that should have depended on noise turned out completely insensitive to it.
After the fixes, several earlier conclusions reversed — on the noisy environment the full-distribution model went from mediocre to dominant, and the share of decisions actually driven by the empowerment signal rose from ~1% to ~70%. Everything from the buggy runs had to be re-derived.
What I took from it:
Log distributions, not just aggregate outcomes; degenerate behaviour hides in the mean.
If a parameter is supposed to matter, add a check that fails when changing it doesn't change the result.
Plausible is not correct. A number worth trusting is one that survives a deliberate attempt to break it.