Michael Vassar

Jun 13, 2023 | lesswrong.com | Eliezer Yudkowsky |Paul Crowley |Julian Morrison |Michael Vassar

Steve Omohundro has suggested a folk theorem to the effect that, within the interior of any approximately rational, self-modifying agent, the marginal benefit of investing additional resources in anything ought to be about equal. Or, to put it a bit more exactly, shifting a unit of resource between any two tasks should produce no increase in expected utility, relative to the agent's utility function and its probabilistic expectations about its own algorithms.