We do foundational mathematical research to ensure smarter-than-human artificial intelligence has a positive impact.

The Machine Intelligence Research Institute is a research nonprofit studying the mathematical underpinnings of intelligent behavior. Our mission is to develop formal tools for the clean design and analysis of general-purpose AI systems, with the intent of making such systems safer and more reliable when they are developed.

The field of AI has a reputation for overselling its progress. In the “AI winters” of the late 1970s and 1980s, researchers’ failures to make good on ambitious promises led to a collapse of funding and interest in AI. Although the field is now undergoing a renaissance, overconfidence is still a major fear; discussion of the possibility of human-equivalent general intelligence is still largely relegated to the science fiction shelf.

At the same time, researchers largely agree that AI is likely to begin outperforming humans on most cognitive tasks in this century. Given how disruptive domain-general AI could be, we think it is prudent to begin a conversation about this now, and to investigate whether there are limited areas in which we can predict and shape this technology’s societal impact.

Researchers at MIRI tend to be relatively agnostic about how the state of the art in AI will change over the coming decades, and how many years off smarter-than human AI systems are. 

Machine Intelligence Research Institute

AYubnnStGbcU9CeeHevoKfWFfHSV2AwooDDxbZyyBMI=

Please donate monthly to maximize longterm impact! Do good every month, cancel anytime.

N6J6uC2BFpY1DBCCNGARPn6U5PWj+hjtybLRiPtieQE=

P4tcjBv99m+mR1oLlHA7veVssOWH2cJ8v/umjSkV7/c=

ipacWjgvkUpq55wyZUwsN4Qjjl7hLlH+eEjYOaQHYFw=

HMbe47+AneGCvW/Xo3JKBD0CoNn+J5g7nYakke86o7I=

QNW+H2PwVn8vv9e5Iz1yCCrYO7y4m9mceodAr9ZQuGo=

XD7WW8JsE2Wq5zNvldYo02kE/9XhuTnMMRIiGDtJ/gw=

Xzooxu7EFYOfep+MPJko33CDGhIpi48EHFdc8PC0QKQ=

q14o0EONZSuc0HViRAdjPcEpjceygaKWHZon938heU0=

I'd like the long-term potential of humanity to be preserved

The long-term potential of humanity is unimaginably enormous. I can’t even dream of the number of stars our distant descendants will settle on and how diverse their forms, abilities, and feelings will be.

Scientists researching existential risks estimate the chance of that long-term potential being destroyed by AI as 1/10 within the next 100 years. For comparison, they estimate the probability of an existential catastrophe caused by a naturally-originated pandemic as 1/10000; by a climate change as 1/1000 per century.

AI researchers don’t know how to make an artificial general intelligence such that it won’t destroy humanity; they think the default way the AI will be created would lead to a catastrophe.
No one has come up with a way to solve the problem of aligning preferences of a teachable agent switch human preferences. It is well understood that it’s impossible to achieve just by defining some rules or designing an accurate utility function (just like it’s impossible to come up with a wish a genie would have to fulfill in a way you intended to (https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden-complexity-of-wishes)). The researchers have shown that any statically defined utility function won’t be aligned with human preferences. No one has come up with an algorithm that would create agents doing what we’d want them to do if we were smarter.
Among the ways that can potentially work, there are, e.g., algorithms that create agents that want to satisfy our preferences but aren’t sure what those preferences are, and thus every time they aren’t sure what we would want them to do, they ask. How to create such algorithms, scientists and researchers don’t know yet.
Experts in this field estimate the chances of an existential catastrophe caused by AI a lot higher than 1/10.

The creation of the first AGI will be the event that will determine the history of humanity. The long-term potential of our species can be destroyed - but if we succeed, we’ll get an assistant that will help us solve all of the other problems humanity is facing.  

The estimates of when AGI will be created vary among researchers. But if humanity knew that aliens would arrive on Earth in a few decades, it would already begin to prepare. The creation of AGI is an event far more important than an encounter of another intelligent species - and we almost certainly know that we have less than a century remaining. I believe that AI Existential Safety is the most important thing people can work on right now.

Machine Intelligence Research Institute is one of the organizations that work on reducing the AI-related existential risk. They do research to ensure that smarter-than-human AI will have a positive impact.

You can read more on Artificial General Intelligence in “Human Compatible”, a book by Stuart Russel. He is a professor at UC Berkeley and author of the most popular textbook on machine learning used by 1500 universities around the world.
You can read more on existential threats to humanity in “The Precipice”, a book by Toby Ord. He is a researcher at the Oxford University’s Future of Humanity Institute and the founder of Giving What We Can.