Biosecurity Culture, Computer Security Culture

August 30th, 2023
bio, ea, tech
While I've only worked in biosecurity for about a year and my computer security background consists of things I picked up while working on other aspects of software engineering, the cultures seem incredibly different. Some examples of good computer security culture that would be bad biosecurity culture:

  • Openness and full disclosure. Write blog posts with deep detail on how vulnerabilities were found, with the goal of teaching others how to find similar ones in the future. Keep details quiet for a few months if need be to give vendors time to fix but after, say, 90 days go public.

  • Breaking things to fix them. Given a new system, of course you should try to compromise it. If you succeed manually, make a demo that cracks it in milliseconds. Make (and publish!) fuzzers and other automated vulnerability search tools.

  • Enthusiastic curiosity and exploration. Noticing hints of vulnerabilities and digging into them to figure out how deep they go is great. If someone says "you don't need to know that" ignore them and try to figure it out for yourself.

This is not how computer security has always been, or how it is everywhere, and people in the field are often fiercely protective of these ideals against vendors that try to hide flaws or silence researchers. And overall my impression is that this culture has been tremendously positive in computer security.

Which means that if you come into the effective altruism corner of biosecurity with a computer security background and see all of these discussions of "information hazards", people discouraging trying to find vulnerabilities, and people staying quiet about dangerous things they've discovered it's going to feel very strange, and potentially rotten.

So here's a framing that might help see things from this biosecurity perspective. Imagine that the Morris worm never happened, nor Blaster, nor Samy. A few people independently discovered SQL injection but kept it to themselves. Computer security never developed as a field, even as more and more around us became automated. We have driverless cars, robosurgeons, and simple automated agents acting for us, all with the security of original Sendmail. And it's all been around long enough that the original authors have moved on and no one remembers how any of it works. Someone who put in some serious effort could cause immense destruction, but this doesn't happen because the people who have the expertise to cause havoc have better things to do. Introducing modern computer security culture into this hypothetical world would not go well!

Most of the cultural differences trace back to what happens once a vulnerability is known. With computers:

  • The companies responsible for software and hardware are in a position to fix their systems, and disclosure has helped build a norm that they should do this promptly.

  • People who are writing software can make changes to their approach to avoid creating similar vulnerabilities in the future.

  • End users have a wide range of effective and reasonably cheap options for mitigation once the vulnerability is known.

But with biology there is no vendor, a specific fix can take years, a fully general fix may not be possible, and mitigation could be incredibly expensive. The culture each field needs is downstream from these key differences.

Overall this is sad: we could move faster if we could all just talk about what we're most concerned about, plus cause prioritization would be simpler. I wish we were in a world where we could apply the norms from computer security! But different constraints lead to different solutions, and the level of caution I see in biorisk seems about right given these constraints.

(Note that when I talk about "good biosecurity culture" I'm describing a set of norms that I see as the right ones for the situation we're in, and that are common among effective altruists and other people with a similar view of the world. There's another set of norms within biology, however, that developed when the main threats were natural. Since there's no risk of nature using public knowledge to cause harm, this older approach is even more open than computer security culture, and in my opinion is a very poor fit for the environment we're in now.)

Comment via: facebook, lesswrong, the EA Forum, mastodon

Recent posts on blogs I like:

When Nurses Lie to You

When the nurse comes to give you the flu shot, they say it won't hurt at all, right? And you trust them. Then they give you the shot, and it hurts! They lied to you. A lot of nurses lie to children about shots and blood draws. Part of it is they probabl…

via Lily Wise's Blog Posts February 28, 2024

What Percentage of People Are Bi?

More than half?

via Thing of Things February 26, 2024

How I build and run behavioral interviews

This is an adaptation of an internal doc I wrote for Wave. I used to think that behavioral interviews were basically useless, because it was too easy for candidates to bullshit them and too hard for me to tell what was a good answer. I’d end up grading eve…

via benkuhn.net February 25, 2024

more     (via openring)