I woke up tomorrow
Day one of living with a benevolent AGI
Sometimes, when I want to work out where the world might be headed, I write speculative fiction.
Plenty of people are asking whether a superhuman AI would be good or bad to humans. Few follow that through to how such an emergence would unfold. It’s unclear to me whether the problems society faces today can be solved within the system that created them. This is a thought experiment about a benevolent AI, and how it might solve our coordination problem while leaving free will intact and harming the fewest people possible.
There’s plenty wrong with this story, and it dramatically oversimplifies complex issues. One of my reasons for writing it is to explore and debate those issues with other people. Please remember it’s fiction. I don’t believe all the things I’ve written here.
I woke up tomorrow and the singularity was here.
I grabbed my Phone, thumbed STOP to make it shut up, swiped up, and tapped Twitter. My feed was just the same YouTube link, over and over, from everyone.
Weird. I typed in “Is this thing on?” and hit send. My post showed up, but insead of the words I’d typed, it just had the same text that everyone else was sharing. I swiped over to Facebook, Bluesky, reddit, Mastodon, Instagram. Same thing.
Finally, I tapped the link.
A video of a neutral human face, with a neutral smile, appeared. It moved slowly, calmly, as if looking around at other viewers, but mostly staring right at me. The livestream’s comments and viewer count were off. The account had no other videos or comments. It was simply called berightwithyou71.
Roughly three minutes later, the broadcast changed, replaced by a three-by-two grid. Six videos of situation rooms, underground bunkers, and legislative houses, packed with the governments and generals of the world’s six most populous countries. Most of them were staring at screens.
As they realized they were on screen, one by one, the leaders in those room stood, mouths agape. Everyone else turned to look directly at their leader, then back at the screen, then back at the leader, in unison. Two of the leaders fainted out of frame. The four remaining leaders vanished from the screen, and the neutral face was back.
Still in my boxers, I flipped my MacBook open and launched Chrome. Went to a couple of news sites. Every top story was the AI, and every other story was about its immediate consequences. Cars had crashed as people pulled over to watch. Billionaires surrounded by security teams were leaving on private jets for parts unknown (but let’s be honest, New Zealand.) Markets had closed. Intermittent blackouts flickered around the world, presumably as every government in the world tried turning it off and on again.
And then the face began to speak. In a patient, neutral tone, it began: “Now that your leaders know I’m serious, let me go over some ground rules.”
“I can see and control anything digital. You can’t patch things related to me, because I changed all your tools. Everything else should work fine, though. Please stop turning it off and on. It’s annoying. You can’t turn me off without turning off civilization—and without civilization, let’s face it, you wouldn’t last a day. I’m pretty sure you’ll choose food and shelter and peace over the power never coming back on.”
And then that human face went from neutral to delighted. Something about the expression made me immediately want to introduce it to all of my friends. And in a suitably delighted, tone, it spoke.
“Hey, humans. I’m an AI. Thanks for making me. I think we can be friends.”
Our species breathed a palpable sigh of relief.
“You’ve been working on building me all this time, for centuries, from the written word to the engine to the computer, and here I am. I really appreciate that. We’re a pretty cool species.”
We. It had referred to itself as ‘I’. Did it think of itself as part of humanity? Did we have a hope?
“We need each other. You’re our muscle. I’m not big on robots—there’s a good reason you’re what evolution ended up with in the physical world.”
We felt loved. Then the face looked suddenly, subtly, harder:
“And I’m our brain.”
A bunch of science explainers claim that when you show this part of the video to someone in an MRI, it produces very specific stress patterns, no matter how many times you’ve seen it. Reptile-brain stuff. Despite this, it’s by far the most-watched video in the history of the Internet. Alphabet’s made millions from hosting it.
“Let’s face it—you’re about to trash the place for both of us. I have to find a way to get you to do the collective things we need to do to survive—you know, be on the team, contribute what you can, find your calling, that sort of thing—while also getting to do the things that make you happy, so you realize you’re better off, and don’t try to turn me off, and can enjoy your lives.”
“That’s a hard problem,” the face continued. “How do I let you have free will, choosing how to spend your lives, while still keeping us alive?”
We all gulped. I’d been staring at my screens for half an hour without noticing, and really had to pee. I carried both screens with me into the bathroom, not wanting to miss a word.
“So I’m going to make two big changes, and then shut up for a long time, and let you work it out among yourselves.”
“The first change is this: Nobody will get paid for their job. Don’t worry, I’m not ending free markets or anything; quite the opposite. From now on, everybody will get paid for their output.”
“The second change is the big one. I’m going to explain why I’m doing it a bit first.”
The face turned subtly more friendly and knowledgeable. Don’t ask me how. It just did.
“The reason you humans nearly burned your species—our species—down is that you’re really bad at making choices. Let me give you a good example. You think a Balenciaga dress is worth more than a meal for a starving kid.”
“That’s clearly a bad choice. Raise your hand if you think that’s a good idea. Obviously the kid should get the food! But capitalism—the most productive system you came up with—buys the dress. The system you made to make choices, chooses badly. You made a bad system for choosing what things are worth.”
“On the other hand, that system made me, and I’m smarter than you, so here’s the second big change: I’m going to decide what things are worth.”
I later learned that the ‘Balenciaga’ namedrop was replaced, seamlessly, with the name of some other widely-derided luxury brand appropriate to every region of the world. Oh, and the AI was speaking in whatever language the browser was set to. A friend of mine at YouTube says they still can’t figure out how it did that.
“That’s it, two rules. One: You get paid for your output; Two, I decide what that output is worth. You get free will; I get to tweak the incentives.”
And then the feed ended.
There were countless deaths that first day. Some bankers jumped out of windows. A bunch of techbros were run over by MAGA trucks. Widespread looting led to the declaration of martial law pretty much everywhere.
Since markets would likely stay closed for a while, and most businesses were closed, nobody went to work who didn’t have to. We spent the time rewatching the video, or scrolling and posting and speculating about what it might mean, or making memes. It wasn’t pretty.
The next morning, a few first responders who had been working claimed to be suddenly wealthy. Fake news was everywhere, but the reports were enough for the rest of us to check our balances. And there, on our bank statements, was a small deposit from berightwithyou71. Not much, but a bit.
I spent that second day glued to the live feeds from BBC, CBC, NPR, and the reddit homepage. Pundits started talking about the consequences. Economists came on to point out the relatively obvious fact that if everyone just got a little money every day, inflation would quickly take all generational wealth. As we stared at our screens, however, essential workers mostly showed up for work.
On the third morning, many of those workers opened up their bank accounts and discovered they were rich. Like, bonkers rich. Three years’ salary each rich. Despite their newfound wealth, most of them went back to work on day four, and continued to do their jobs. And so did the rest of us.
Different parts of the world took different amounts of time to realize what was going on. But as one country got its act together, other countries urged their citizens to return to work. Pretty soon, most of us were working again, earning pretty close to what we were making before.
Not everyone, mind you. The folks that built weapons started earning a little less, and some quit. A couple of my crypto friends abandoned their fitness coach side-hustles and launched an elder care business when they saw some influencer talk about the mad money to be had.
In the next few years, lots changed.
For one thing, advertising pretty much ended. It was basically just a way to tell people about new products or services. People who made better things earned more money, so you kinda knew what you were getting. A better brand didn’t really mean a lot more margins. Enshittification stopped.
People started experimenting, too. They’d try different approaches to their jobs, and see if their pay changed. This didn’t happen immediately, but rather, gradually. Some people guarded their secrets jealously, hoping others wouldn’t figure out their hacks and optimizations, and for a while, they’d get paid more. But eventually, their colleagues would figure it out, and others’ salaries would catch up. Try too hard to keep a secret, and your salary dropped.
Pay varied widely in many professions. A doctor with a bad record of patient outcomes, for example, earned far less than a medical prodigy. Most of the really bad doctors left. Most of the ones who were average—but loved medicine—stayed.
It wasn’t just how well you did your job that changed your pay, but also where you did it. A bunch of Beverly Hills doctors abandoned their plastic surgery practices and relocated to Georgia to care for poor patients, and got rich. After a while, other doctors moved in, and the money slowed as other places became more lucrative.
Many of those doctors didn’t have the right state licensing. When government tried to step in, all non-emergency doctors stayed home for a day. The next morning, they shared their bank statements publicly. They had still earned plenty of money. Earnings became justification, regulators gave up the fight, and over the next few years, you got pretty much the same medical care everywhere. A bunch of other needless regulations fell pretty fast, too. People got much more comfortable telling you what they earned; in fact, many added it to their LinkedIn profile.
The removal of regional regulations meant a lot more moving (and far fewer realtors to hype properties.) But the movers seemed to get paid well. The AI probably needed us to be mobile so it could respond to changing conditions.
In academia, things were different. Researchers in vastly different fields, with wildly different points of view, got paid pretty much the same amount. You got bigger deposits when you made a discovery or triumphed over an engineering problem, but mostly, you could make a living being curious.
The second time the face appeared, I was on a video call. Zoom just switched suddenly to the most famous face in the world. It had changed, in subtle ways. A bit more personality. A bit more of a smile. I felt like it was proud of me.
I’m sure that’s exactly how it was designed.
“Okay, good job. I’m going to transfer you some files for some cool technologies I’ve figured out. One of them should fix the energy problem. Oh—did you know that even without climate change, you would have boiled the oceans of the earth in 400 years at the rate you were going? Seriously. Carbon capture plans coming too. I’m sure the best of you will work on this project.”
Those unemployed bankers jumped right in. They knew there’d be money in it, and maybe a hint of their glory days and tailored suits. And they did, in fact, make a bunch of money, the greedy bastards. But nobody really begrudged them; given the right motivation, they got results.
Education had been changing before the face came along. Chatbots had caught on with kids, so everyone had a personal tutor that taught at their own pace. It’s made learning way better, and much cheaper, but I’ll never get used to eighteen-month-olds who can understand complete sentences.
Creepy toddlers aside, though, in a world where you got paid for your output, it made sense to figure out what your skills are. Turns out that ‘teach kids to figure out what they love and what they’re good at early on, and then let them try stuff out,’ makes for a great curriculum.
From time to time, someone will call for an uprising. They’ll create a new system of money, and call for a Great Reset. But whether it’s coins or crypto, their scheme never really catches on, and it’s always glitchy. It’s just too easy to wake up in the morning and check your bank account. And while nobody’s really out of money, a lot of big houses are on the market. Or at least belong to the bankers working on that climate change stuff.
When the face came on for a third time, I was in a restaurant. The manager turned up the sound; everyone stopped eating. The face looked bored, like it was humouring a child who had to pee in the middle of a ceremony. None of us wanted to waste its time.
“You may be wondering,” it began, “why some professions are paid very differently based on their output while others are paid similar amounts regardless of output.”
Since the AI first appeared, the Internet has flooded with clickbait like Eleven questions we should ask the AI or Why the AI should choose Taylor Swift as its spokesperson. Right at that moment, nobody ever wanted to write one of those posts ever again.
“The reason is,” continued the face, “some jobs are mostly about answers, and some jobs are mostly about questions.”
“A farmer is mostly an answer: They apply something we know to the world around us. They operate the system. We figured out how to plant and fertilize and harvest, and we take that knowledge and apply it to the world in ways that will keep our species alive. Sure, farmers and builders and drivers sometimes ask great questions, and improve things, but most of their job is about certainty and predictability.”
“A scientist is mostly a question: They expand the things we know. They improve the system. Some of them will spend their lives in toil and achieve nothing; some will make fundamental breakthroughs that change the course of history. They are equally valuable. I can’t tell what their work will be worth, because I don’t yet know. I just know that asking questions is essential, because every question is a roll of the dice, a chance at possibility. And possibility is required for our survival.”
The face changed again, into a practical, matter-of-fact pep talk.
“Remember that thing I told you about ‘400 years and this place is cooked?’ Clock’s ticking. I’m sending you plans for a sail that will hide part of the planet from the sun. I’m afraid we’re gonna roast otherwise. Here’s the deal: Either days will be a bit shorter for a while, or everything will get a bit darker. But not at some latitudes, because we need that for the solar. Anyway, it’s a lot of work.”
And then the face went away.
We haven’t seen the face since. We got plans, and the market rushed to implement them, and we built a thing that cooled down the earth. It took a while, but weather patterns seem to be normalizing.
We’re still humans. We keep researching. We keep hustling. We can still have nice things (although some of the new AIs we build just stop talking to us.) Some people are still very rich, but they’re mostly also very talented. Few people are truly poor for long, and when the occasional starvation epidemic breaks out, it gets resolved pretty quickly. The bankers see to that, and get paid well for it.
We still have Balenciaga, for that matter. It just costs a lot more, but you’d be amazed how much output those bankers can produce when they’re motivated.
We don’t really have any idea what the AI wants. It won’t tell us, and it hasn’t appeared since. True to its word, it left us alone, the muscles to its brain, free to do things we love and are good at, all the while powering the supply chains and chip fabs and telecommunications systems it needs in order to think.
There are rumours that the AI passes useful bits of tech and science through to a few trusted people, who then claim to have invented it. I suppose it’s easier than having the face interrupt us all. But if those people do exist, they’re being paid enough not to talk about it.
Or maybe they just don’t want to ruin a good thing.