AI/Concept/Explore/vs/Exploit
Summary
- Any agent that learns from its own actions faces one recurring choice: act on what it already knows works, or try something whose payoff is unknown. Exploiting collects the reward that present knowledge can reach. Exploring buys information about rewards the agent cannot yet see. Time or attention spent on one is withheld from the other, and neither can be dropped: an agent that only exploits stays stuck on the best thing it happened to find early, and an agent that only explores never cashes in what it learns.
- The tension appears wherever a decision is made repeatedly under uncertainty: a gambler facing a row of slot machines, a lab choosing which experiments to fund, a company weighing refinement of its product against the search for the next one, a reader picking a new book over rereading a loved one, and a knowledge gardener choosing between planting new pages and tending the ones already there.
The Trade-Off
- Choosing to explore means spending a turn on an option whose payoff is unknown, to learn what it is worth.
- Gains — information: a truer map of which options pay, and the chance of finding something better than the current best. The value is highest early, when little is known and a long horizon remains to use what is learned.
- Costs — the reward forgone on that turn, since the known best was passed over, plus the chance that the unknown option is worse. Returns arrive late and uncertain.
- Too far — the perpetual survey: an agent, team or gardener that keeps sampling and never settles, so nothing learned is ever cashed in. March described the result as too many undeveloped ideas and too little distinctive competence.
- Choosing to exploit means taking the option that present knowledge says pays best, and collecting the reward.
- Gains — reward now, with little variance: refinement, efficiency, execution, and the compounding that comes from doing the known thing well.
- Costs — the information forgone: untried options stay unknown, and the estimate of the current best goes unchallenged. The map stops being updated while the world keeps changing.
- Too far — the settled routine: an agent locked onto the best option it happened to find early, a firm polishing a product after the market has left, a garden that is only ever pruned and never planted.
Where the tension came from
- The formal problem is the multi-armed bandit: a row of slot machines with unknown payout rates and a limited number of pulls to spend on them. Allied statisticians worked on it during the Second World War and found it so hard that Peter Whittle joked it should be dropped on Germany to waste enemy scientists' time. Herbert Robbins gave the problem its standard form in 1952 in "Some aspects of the sequential design of experiments".
- In organization theory, James G. March's "Exploration and Exploitation in Organizational Learning" (1991) named the trade-off for firms. Exploitation is refinement, efficiency and execution; exploration is search, variation, risk-taking and discovery. Returns to exploitation are near, certain and fast, while returns to exploration are distant, uncertain and often negative, which is why organizations slide toward exploitation and lose the ability to adapt.
- In AI/Reinforcement Learning the balance is written into the algorithms (Exploration–exploitation dilemma, Wikipedia). Epsilon-greedy takes the best-known action most of the time and a random one otherwise. Upper-confidence-bound methods add an optimism bonus to actions tried less often. Thompson sampling draws a guess at each payoff from what has been observed and acts as if the draw were true. Exploration bonuses turn curiosity into a reward of its own, so the agent can exploit it.
How the balance is struck
- Explore early, exploit late. When little is known, information is cheap and there is a long horizon left to use it. As the horizon shortens, the remaining turns are worth more spent on the known best.
- Keep exploration cheap and reversible, so a bad draw costs little.
- Watch for both failure modes: the settled routine that stopped looking, and the endless survey that never commits.
Through cybernetics
- Seen through cybernetics, a learner is a regulator holding a course through an environment it can only partly see. Exploitation is the corrective action along the known heading. Exploration is the sampling that keeps the map of the environment current, so the corrections stay pointed at the real world. Ashby's requisite variety says a regulator must carry at least as many responses as the world has disturbances, and exploration is how it acquires the variety it lacks. Second-order cybernetics adds the observer to the loop: a gardener steering their own learning is regulating the system that does the steering, which is why the balance is never set once and left alone.
In this garden
- My/Principle/Learn/in Public frames knowledge gardening as calibrating these two forces: exploring adds new things to the garden and new connections to the world, and exploiting tends what is already planted so it becomes more vibrant and real. Explore and Exploit each have a page of their own.