Daniel Murfet

Featured in:

Articles

Activation space interpretability may be doomed — LessWrong

Jan 8, 2025 | lesswrong.com | Lucius Bushnaq |Daniel Murfet |Joseph Miller |Matthew Clarke

TL;DR: There may be a fundamental problem with interpretability work that attempts to understand neural networks by decomposing their individual activation spaces in isolation: It seems likely to find features of the activations - features that help explain the statistical structure of activation spaces, rather than features of the model - the features the model’s own computations make use of.
Cognitive Work and AI Safety: A Thermodynamic Perspective — LessWrong

Dec 8, 2024 | lesswrong.com | Daniel Murfet

Introduces the idea of cognitive work as a parallel to physical work, and explains why concentrated sources of cognitive work may pose a risk to human safety. Acknowledgements. Thanks to Echo Zhou for feedback and suggestions. Some of these ideas were presented originally in a talk in November 2024 at the Australian AI Safety Forum slides for which are here: Technical AI Safety (Aus Safety Forum 24)and the video is available on YouTube.
Causal Undertow: A Work of Seed Fiction — LessWrong

Dec 8, 2024 | lesswrong.com | Daniel Murfet

A short story, best enjoyed in-context... try exploring possible meanings of some of the unusual terms with your favourite commercially available Solomonoff inductor. Acknowledgements. Thanks to Simon Pepin Lehalleur and Ziling Ye for feedback and suggestions. This post is the "fun" half of a pair, for the serious version see Cognitive Work and AI Safety. The sand was still cool under Dr. Chen's feet, the sun barely cresting the horizon.
The Queen’s Dilemma: A Paradox of Control — LessWrong

Nov 27, 2024 | lesswrong.com | Daniel Murfet

Our large learning machines find patterns in the world and use them to predict. When these machines exceed us and become superhuman, one of those patterns will be relative human incompetence. How comfortable are we with the incorporation of this pattern into their predictions, when those predictions become the actions that shape the world? My thanks to Matthew Farrugia-Roberts for feedback and discussion.
Selection for error correction in neural networks? — LessWrong

Oct 26, 2024 | lesswrong.com | Lucius Bushnaq |Stefan Heimersheim |Daniel Murfet

Epistemic status: Low confidence. This is a bit of an experiment in lowering my bar for what I make a proper post instead of a shortform. It's just an early stage idea that might go nowhere. It looks to me like neural networks might naturally develop error-correction mechanisms because error correction dramatically increases the volume of solutions in the loss landscape. In other words, neural networks with error correction seem like they’d tend to have much lower learning coefficients.
SLT: The Hessian rank bounds the learning coefficient — LessWrong

Aug 8, 2024 | lesswrong.com | Lucius Bushnaq |Daniel Murfet

TLDR: In a neural network with d parameters, the (local) learning coefficient λ can be upper and lower bounded by the rank of the network's Hessian d1:d12≤λ≤d12+d−d13. The lower bound is a known result. The upper bound is a claim by me, and this post contains the proof for it. If you find any problems with the proof, do point them out. The learning coefficient λ is a measure of loss basin volume and network complexity. You can think of it sort of like an effective parameter count of the model.
Timaeus is hiring! — LessWrong

Jul 12, 2024 | lesswrong.com | Jesse Hoogland |Stan van Wingerden |Alexander Gietelink Oldenziel |Daniel Murfet

TLDR: We’re hiring two research assistants to work on advancing developmental interpretability and other applications of singular learning theory to alignment. About UsTimaeus’s mission is to empower humanity by making breakthrough scientific progress on alignment.
Simple versus Short: Higher-order degeneracy and error-correction — LessWrong

Mar 11, 2024 | lesswrong.com | Daniel Murfet

[minor]I'm a bit confused or I think you need some additional caveats in the intro. I would have said that Bayesian statistics with typical models is well understood as bias toward short description length but there are important caveats in the neural network case, at least conceptually.
Timaeus's First Four Months — LessWrong

Feb 28, 2024 | lesswrong.com | Jesse Hoogland |Daniel Murfet |Stan van Wingerden |Alexander Gietelink Oldenziel

Timaeus was announced in late October 2023, with the mission of making fundamental breakthroughs in technical AI alignment using deep ideas from mathematics and the sciences. This is our first progress update.
Announcing Timaeus — LessWrong

Oct 22, 2023 | lesswrong.com | Jesse Hoogland |Daniel Murfet |Alexander Gietelink Oldenziel |Stan van Wingerden

Timaeus is a new AI safety research organization dedicated to making fundamental breakthroughs in technical AI alignment using deep ideas from mathematics and the sciences. Currently, we are working on singular learning theory and developmental interpretability. Over time we expect to work on a broader research agenda, and to create informed by our research. Our primary focus is research. For now, we're a remote-first organization.