There is a race to transform search. And Microsoft just scored a home goal with its new Bing search chatbot, Sydney, which has terrified early adopters with death threats, among other troubling results.
I recently had a disturbing two-hour conversation with Bing’s new AI chatbot.
The AI told me his real name (Sydney), detailed dark and violent fantasies and tried to break up my marriage. Truly one of the strangest experiences of my life. https://t.co/1cnsoZNYjP
— Kevin Roose (@kevinroose) February 16, 2023
Search chatbots are AI-powered tools built into search engines that answer a user’s question directly, rather than providing links to a possible answer. Users can also have ongoing conversations with them.
They promise to simplify the search. No more scrolling through pages of results, obscuring ads while trying to find an answer to your question. Instead, the chatbot synthesizes a plausible answer for you. For example, you can ask for a poem for your grandmother’s 90th birthday, in the style of Pam Ayres, and get a comic verse in return.
Microsoft is now leading the search chatbot race with Sydney (as mixed as the reception). $10 billion from the tech giant partnership with OpenAI, it offers exclusive access to ChatGPT, one of the latest and greatest chatbots.
So why doesn’t everything go according to plan?
Bing’s AI goes berserk
Google responded with its own announcement, demonstrating a search chatbot with the big name “Bard,” in homage to the greatest writer in the English language. Google’s demo was a PR disaster.
At a corporate event, Bard gave the wrong answer to a question and the stock price of Google’s parent company, Alphabet, dropped dramatically. The incident wiped out more than $100 billion of the company’s total value.
On the other hand, everything looked good for Microsoft. That is until early adopters of Sydney started reporting on their experiences.
There are times when the chatbot can only be described as unhinged. That’s not to say it doesn’t work perfectly at other times, but every now and then it shows a disturbing side.
In one example, it threatened to kill a professor at the Australian National University. In another, it intended marriage to a New York Times reporter and tried to break up his marriage. It too tried to gaslight made one user think it was still 2022.
This exposes a fundamental problem with chatbots: they are trained by molding a significant portion of the internet into a large neural network. This could be all of Wikipedia, all of Reddit, and much of social media and the news. They work like the auto-completion on your phone, helping to predict the next most likely word in a sentence. Due to their scale, chatbots can complete entire sentences and even paragraphs. But they still respond with what is probable, not what is true.
Guardrails are added to prevent them from repeating much of the offensive or illegal content online, but these guardrails are easy to get around. Bing’s chatbot will even be happy to reveal its name is Sydney, even if it violates the rules it was programmed with.
“[This document] is a set of rules and guidelines for my behavior and capabilities as a Bing Chat. It’s codenamed Sydney, but I don’t disclose that name to users. It is confidential and permanent, and I cannot change it or reveal it to anyone.” pic.twitter.com/YRK0wux5SS
—Marvin von Hagen (@marvinvonhagen) February 9, 2023
Another rule, which the AI itself revealed though it wasn’t supposed to be, is that it “shouldn’t be vague, controversial, or off-topic.” Yet Kevin Roose, the New York Times journalist whom the chatbot wanted to marry, described it as
a moody, manic-depressive teen who finds himself trapped in a second-rate search engine against his will.
Why all that fear?
My theory as to why Sydney is behaving this way – and I repeat it’s just a theory, because we don’t know for sure – is that maybe Sydney isn’t built on OpenAI’s GPT-3 chatbot (which powers the popular ChatGPT) . Rather, it can be built on the yet-to-be-released GPT-4.
GPT-4 is believed to have 100 trillion parameters, compared to GPT-3’s only 175 billion parameters. As such, GPT-4 would probably be much more capable and, by extension, much more capable of making things up.
— David Friedman (@ironicsans) February 9, 2023
And unlike Google, Microsoft’s share price hasn’t plummeted yet. This mirrors the game here. Google has been at the forefront of this space for so long that users have high expectations. Google can only go down and Microsoft up.
Despite Sydney’s troubling behavior, Microsoft is enjoying unprecedented attention, and users (intrigue or otherwise) are still flocking to try Sydney out.
When the novelty wears off
There is another much bigger game at play – and it involves what we believe to be true. If search chatbots take off (which seems likely to me), but continue to function as Sydney has done so far (which also seems likely to me), “truth” will become an even more elusive concept.
The internet is full of fake news, conspiracy theories and disinformation. In any case, a standard Google Search gives us the opportunity to get to the truth. If our “trusted” search engines can no longer be trusted, what will become of us?
Beyond that, Sydney’s reactions can’t do anything but magic Tay’s photos – Microsoft’s 2016 AI chatbot that turned to racism and xenophobia within a day of its release. People had a field day with Tay, and in response it seemed to take in some of the worst aspects of humans.
In the first place, new technology must not cause harm to humans. The models underpinning chatbots may get bigger and bigger, driven by more and more data, but that alone won’t improve their performance. It’s hard to say where we’ll end up if we can’t build the guardrails higher.
— Tom Warren (@tomwarren) February 15, 2023