Interrogate the assumptions of AI Safety

“how do we ensure that generally intelligent systems, now and in the future, are beneficial to the moral ambitions of humanity?” A simple, obviously relevant question with no present formalization that does not lose most of the important nuance.

For one thing: it is plainly not a question about computer science, though computer science will be relevant for a number of implementation details. The answers needed before one gets to coding are about the nature of intelligence, about the attributes of complex systems which constitute morality and about what it means to empower a species that disagrees with itself. These are problems regarding life, mind, value and all those things philosophy has sought to grasp since long before machine learning has emerged onto the scene. The stage has not been fully built and we do not have a language in which the script might be written.

Troublingly and simultaneously, this frame-finding work is far more urgent than philosophy is by habit. The artificial systems planning, modelling and acting within the world are already here, regardless of our comprehension, and the speed of their deployment is only increasing.

Serious effort is required to ensure their safety. Economics, control theory and machine learning are brought to bear in the construction of oversight-, evaluation- and interpretability regimes, though these concepts are borrowed and stretched beyond their rightful domains. They creak and strain and the insufficient fit of their assumptions are a thing for intelligence to exploit. Unpredictability and trouble follows the mismatch: how to continuously evaluate a continual learning agent? How to identify the right primitives to describe general agency in the first place?

The research tradition most in touch with these concerns is Agent Foundations, to which reasoning, goals and the titular agency are open problems rather than settled primitives. Though our interests range beyond it, this is where AFFINE, the Agent Foundations Field Network, gets its name.

We allege that the safety work being conducted right now might be built atop the wrong concepts and that at least a fraction of the effort exerted to push the frontier should be directed at careful, rigorous thought about the generator and those unexamined assumptions which allow one to be precise while nonetheless missing the target. This is what it means to be pre-paradigmatic in Kuhn’s sense: to lack the tools with which to discern a correct question from a complicated dead end.

It is imperative to find out early. In a sound frame, the downstream work may still take some competent researchers a number of years but in an unsound frame, the downstream work will waste the time of geniuses for eternity or, more likely, until something catastrophic happens.

Normal research proceeds where the central concepts are locked in place as new ideas and novel data are funneled through them, parcelled, analyzed and incorporated. Where the foundation is sound this approach is entirely unobjectionable and will result in steady, efficient progress.

Occasionally a research trajectory stumbles upon underdefined concepts without noticing and begins to dig fiercely for an insight that does not ultimately exist. The science degrades until someone eventually realizes the mistake. If this is the case in AI Safety it would be wise to survey a few more dig-sites and distribute our eggs across a wider array of baskets.

This, of course, is difficult. How does one evaluate questions for their ability to frame a concept that does not yet exist? Rigorous conceptual thinking is not only slow and thankless, but almost impossible to measure and easy to get wrong. How can we produce it reliably?

The two components we need are simple, but difficult to fund: Space for unhurried thought and people willing to tell you that your framing is wrong instead of running with the prompt. What we want, in short, is a research monastery, so that is what we are trying to build. A setting where the conceptual scaffold can be taken apart carefully by people who have read enough to know what it is intended to do so that the parts may be weighed, rotated and when necessary replaced.

AFFINE grew out of a month-long residential seminar held in Czechia in May 2026, and is now undergoing the same process of refactoring described above. We’re looking through the parts we used at what we were intending to use them for and building something larger and more principled out of those insights: recurring seminars, fellowships, and retreats that bring researchers from across our unsettled field into the same room for long enough to properly disagree about the central problem.

There are many useful things that scholars from adjacent disciplines can bring to a young field of research, from methods to framings to object level results, though chief among them is pointing out where it is confused: The places where it isn’t answering the questions it thinks it is.