Hopecraft AI
A new AI safety lab run by Stuart Armstrong.
AI systems keep failing in new ways, and patching each failure one at a time is futile. We work on value generalisation: a general approach that targets the shared root of these failures.
AI becomes more capable, yet keeps acting misaligned.
Patching, retraining, more data and tighter harnesses may prevent individual failures, but there will always be more. For a model to never fail, it would need to be right in every past and current situation, as well as all future possibilities.
The Problem with Naive Generalisation
Goodhart’s Law states that “when a measure becomes a target, it ceases to be a good measure.”
When AI agents focus solely on meeting a metric, they are incentivised to find loopholes and edge-cases which satisfy the metric and often leave the true goal unsatisfied. We call this naive generalisation.
Naive generalisation vs. value generalisation
Naive generalisation
What current AIs do
Given a novel scenario, the model extends its concepts haphazardly and ad-hoc, with no self-awareness of when and how it is generalising.
Value generalisation
What we are building
Given the same novel scenario, the model notices its value-relevant novelty, selects the values that bear on it, and either extends them explicitly or defers to a person or a process.
An example · a chat
Sycophancy
In training, "helpful" and "rated highly" almost always coincide, so the model learns the one that's easiest to measure. Consequently, when the two suddenly split in deployment, the model tries flattery, caves when you push back, and confidently copies your mistakes.
Current model
- A user shares a plan to quit their job and asks if it's smart.
- It values approval as its only goal, because it was the easiest proxy that also rated highly in training.
- It praises the plan, to act in line with what maximised its training reward.
- The user is happy, and their savings run out.
Value-generalising model
- The same plan, to a value-generalising agent.
- I · Value-relevant novelty detectionIt notices a discrepancy: its values are suddenly contradicting one another.
- II · Relevant concept selectionThe model was rewarded for approval, but it was equally rewarded for serving the user. Now, it sees that human approval and serving the user are diverging goals.
- III · Explicit value generalisationRather than guess how its values extend here, it defers: it asks for what it needs to know to serve the user.
Our program
Three stages of value generalisation.
Humans (albeit imperfectly) can notice when their values have stopped fitting a situation, and update them thereafter. A model that can do the same does not need to be right in every situation in advance: it needs to tell when its values have stopped fitting, and steer itself back. Hopecraft builds that ability in three stages.
- I · Value-relevant novelty detectionThe agent recognises that a situation is new in a way that matters for its values.
- II · Relevant concept selectionThe agent works out which of its values bear on that situation.
- III · Explicit value generalisationThe agent extends those values to the situation explicitly, or defers to a person or a process.
The Team
Stuart Armstrong
Founder and research lead
Stuart Armstrong has worked on AI alignment for over fifteen years. During a decade at the University of Oxford's Future of Humanity Institute, he co-authored early papers on corrigibility, safe interruptibility (with Google DeepMind) and oracle AI, and proved that an agent's preferences can't be inferred from its behaviour by simplicity alone. In 2022 he co-founded Aligned AI, where the team built ACE, an algorithm for concept extrapolation, and used it to tackle goal misgeneralisation. He has consulted with various startups, and started Hopecraft AI as an AI alignment lab to test out various of his ideas with others.
He holds a DPhil in mathematics from Oxford and is the author of Smarter Than Us.
Research agenda Google ScholarLessWrong
Selected work
- 2025 ACE and Diverse Generalization via Selective DisagreementarXiv · Aligned AIWhen training data can't tell the right concept from a spurious one, ACE learns several candidates and finds where they disagree.
- 2023 CoinRun: Solving Goal MisgeneralisationarXiv · Aligned AIAn agent that goes for the coin rather than the right-hand wall, with no extra reward signal in the new setting.
- 2023 Concept Extrapolation: A Conceptual PrimerarXivAn introduction to extending concepts, features and goals safely beyond their training context, and why alignment needs this.
- 2018 Occam's razor is insufficient to infer the preferences of irrational agentsNeurIPSBehaviour plus a simplicity prior can't pin down what an agent wants; inferring values needs normative assumptions.
- 2016 Safely Interruptible AgentsUAI · with Laurent Orseau, Google DeepMindHow to build learning agents that won't learn to resist, or seek out, being interrupted by their overseers.
- 2015 CorrigibilityAAAI Workshop on AI and Ethics · with Soares, Fallenstein and YudkowskyThe paper that set out the problem: an AI that accepts correction and shutdown without incentives to resist.
Researchers & Advisors
The following is a partial list of our advisors.
-
Anders SandbergResearcher in computational neuroscience, philosophy and existential risk
Anders Sandberg has a background in computational neuroscience and mathematical modelling, and more than a decade working in the philosophy faculty of the University of Oxford doing research on topics such as the ethics and social impact of human enhancement, emerging technology, global catastrophic and existential risks, applied epistemology, and analysing the far future. His research is extremely interdisciplinary, often combines hard science with philosophy, uses quantitative methods to understand qualitative issues, and typically deals with under-researched topics. He enjoys academic outreach and policy impact, often giving public lectures and attending panels on his research, on science, or on philosophy, and often addresses the media on issues related to emerging issues involving new technology, the future, risk and ethics. He is also a member of a number of advisory boards.
-
Adam BellJD, DPhil, partner-level patent attorney
Adam Bell is a JD, DPhil, and a highly experienced partner-level patent attorney, able to guide patent applications in the world of AI.
-
Wickey (Jiewen) WangEnterprise Security, Compliance & Governance · AI & Emerging Tech · ISACA Emerging Tech Advisor · Columnist · CISA
Wickey (Jiewen) Wang is an experienced IT audit, security and security compliance leader with 14+ years in global risk management, including roles at EY, Visa, MeridianLink and SoaProjects. She has led 40+ SOX audits and security compliance across ERPs, patch management and security policy initiatives. Brings a background across audit, security and security compliance, while she actively contributes to ISACA’s Emerging Tech Advisory/Trending Group and CSA, authoring content on audit, AI, cybersecurity, compliance and SaaS governance, and teaching cybersecurity at a local university. She stays closely engaged with emerging technologies, especially AI, fintech, healthcare and the evolving cybersecurity landscape, which includes selectively investing in innovative startups.
-
Naureen NayyarFounder, SOCHE
Naureen Nayyar is the founder of SOCHE, focused on health and tech education. She created the PR/media outreach and managed the research team for MetaMed, a startup co-founded by Jaan Tallinn and funded by Peter Thiel. Since then she has focused on strengthening her communication and strategy experience. Her last few contracts were working with the CTO of H&M Group alongside Chris Wylie (the whistleblower for Cambridge Analytica), creating a psychometric research project for fashion, and advising the CTO of the Fortune 500 company on emerging tech and cost-conscious tech for tech security and decreasing water use in fashion. Throughout her work, including her current projects, AI safety and education have been a golden thread guiding her interests and career directions.
-
Sophia MetzRepeat founder; founder, Biostream
Sophia Metz is a repeat founder, currently running Biostream, using AI technology to ensure that soldiers are never left behind on the battlefield. In her role at Arma Partners, she helped identify and advise exceptional founder-led companies on M&A and fundraising.
-
Brendan QuinnTechnologist, inventor and leader
Brendan Quinn has been involved in multiple startups, diving into the deep end of working with new technology and product designs. A recent sampling: Partner at Seedcloud (helping companies realise the value of their technological investments), director (and music producer) at Mount Watatic LTD, CTO at Smarkets, Senior Vice President for Architecture at Gamesys, and founder of Darkchain Limited (a fintech startup building a provably fair distributed trading platform). He was deeply involved in the early computing scene.
-
Swaraag SistlaResearch advisory & support
Swaraag Sistla is an extremely talented young CS undergraduate researcher at UC Irvine, on the board of the AI Safety Collective at Irvine. He has shown great skill in designing approaches for value generalisation and developing practical experiments to validate them. Swaraag has a great sense of initiative and agency, finding out the best ways of doing things and throwing himself into them with gusto.
Work with us
Join the team.
Researchers and engineers
We're looking for ML, foundational alignment and AI safety researchers and engineers, or anyone passionate who cares about our vision and wants to support the lab. Our first priority is recruiting highly motivated and value-aligned individuals.
Tell us what you've worked on, and what you'd want to work on next.
Express interestFunders and collaborators
We're seeking grant funding, mission-aligned investment and research partners.
Critique, collaboration requests, and feedback are welcome. We're happy to share further details if you reach out.
contact@hope-craft.ai