AI Advances and Detection Strategy

April 19th, 2025
airisk, nao

Cross-posted from my NAO Notebook.

This is an internal strategy note I wrote in November 2024 that I'm making public with some light editing.

In my work at the NAO I've been thinking about what I expect to see as LLMs continue to become more capable and get closer to where they can significantly accelerate their own development. I think we may see very large advances in the power of these systems over the next few years.

I'd previously thought that the main impact of AI on the NAO was through accelerating potential adversaries, and so shorter timelines primarily meant more urgency: we needed to get a comprehensive detection system in place quickly.

I now think, however, that this also means the best response involves some reprioritization. Specifically, AI will likely speed up some aspects of the creation of a detection system more than others, and so to the extent that we expect rapid advances in AI we should prioritize the work that we expect to bottleneck our future AI-accelerated work.

One way to plan for this is to imagine what would be the main bottlenecks if we had a far larger staff. Imagine if each senior person had AI support equivalent to all the smart junior people they could effectively manage. Or even (but my argument doesn't depend on this) AI systems that are as capable as today's experienced researchers. I think if in a year or two we found ourselves in this situation we would wish that:

  • We had collected a lot more data, because with a very large virtual computational staff future-AI-assisted-NAO can wring insights out of data far more efficiently than present-NAO.

  • We had started large-scale collection sooner, because, even if AI accelerates sequencing's price decreases and we can collect a lot more data in the future, it can't give us historical data.

  • We had a lot more partnerships for bringing in samples and data, because these take real-human time to scale up.

While I don't think this is the only way things could play out, I think it's likely enough that we should be taking these considerations very seriously in our planning.

April 2025: since initially drafting this we've started an ambitious effort to scale up our pilot system.

Referenced in: Personal AI Planning

Comment via: facebook, lesswrong, mastodon, bluesky, substack

Recent posts on blogs I like:

I did a meditation and it sort of changed my personality?!

a trip report

via benkuhn.net November 8, 2026

The bad part about rape is all the rape

I have recently seen a lot of people casually refer to consensual non-consent orgies as “rape orgies”.

via Thing of Things November 6, 2026

Test and verification technique effectiveness with Astra

When we looked at how well agents used test/verification techniques in August1, we saw that GPT-5.6 Sol generally did what you might call a caricature of various techniques when asked to use a technique or library such as TDD, QuickCheck, Lean 4, etc.; no…

via Posts on November 2, 2026

more     (via openring)