Home
Trending
New Release
Speaker A: This program is brought to you by Resonance FM. If you like what you hear, please support our work by making a donation at resonancefm.com/donate.
Speaker C: Hello and welcome to Password on Resonance FM with me, Peter Warren, the program that puts our relationship with technology under a spotlight. In this month’s dissection of the digital world, we quite literally look at the relationship that Big Tech is trying to generate between us and it and ask, is it ethical? Because it’s definitely illusory and it needs to be regulated. The clue in all of this is the word ‘it’ because ‘it’ is a word that that in the English language can be a bit amorphous. It moves around a lot. As the great poet T.S. Eliot said, words slip, slide, perish, decay with imprecision, will not stay in place, and it is a prime candidate. And now, in the AI world blossoming in front of our eyes, it has become a real danger because we cannot discriminate between it and IT. We have already begun to personalize technology and allow us to be lulled into a false sense of intimacy. AI has become artificially intimate, the subject of a new book from the renowned Massachusetts Institute of Technology commentator and humanist Professor Sherry Turkle, who we will hear from at the end of the program. Just how dangerous the issue is was forcibly and worryingly demonstrated by Peter Kyle, a UK technology Technology Minister yesterday during Tech Week when he announced, I’ve already seen how ChatGPTs and other AI services have helped me. AI gets to know you. AI gets to know how you ask questions and how you think. It fits in around your own individual learning characteristics. AI is an incredible tutor. So there is no question that AI deployed wisely and safely not just in education but in a young person’s life, can have an incredible levelling-up opportunity. You could say that it’s semantics, but AI does not get to know you. It’s a computer, and the relationship is dangerous because at the moment, according to experts Future Intelligence’s Password parent organisation has interviewed, The motivation for the system to get to know you is financial. The system has an interest in getting you to trust it because it wants you to give the system information about you. Here’s Johnny Penn, Professor of AI Ethics and Society at the University of Cambridge, on that business relationship.
Speaker D: I, on historical grounds, have argued that what we’re interacting with is closer to logistics than cognition. It’s incredible, actually, what has been accomplished, but it’s not cognition, which is sublime. The human brain is the most complex system under the sun. Now, you mentioned trust. For me, there’s a difference between, or not just for me, there is a difference between trust and trustworthiness. Trustworthiness is something that deserves trust, because often we’ll trust things that aren’t good for us. And there are a number of reasons why LLMs, large language models, generative AI, and their kind of anthropomorphic interfaces where it’s like ChatGPT, you’re chatting with a seemingly human system. There are a number of reasons why that is untrustworthy. One is that it’s statistically, unlike the history of computing, which is you get the same answer every time you ask the same question, now there’s a, a sort of mirror, mirror on the wall effect where it’ll give you a different answer every time. You’d mentioned a moment ago an example of a friend who said, it thinks like me. I would argue that they’re not being delusional, but they’re being treated somewhat disingenuously. And it’s an important difference that I’ll summarize this way. My colleague, Jakob Choudhury, and I have written about what we call the intention economy, which we see as a kind of mutation of the attention economy premised around chatbots. And if you’ve been on the modern internet, you know it is driven by advertising. That’s where the big tech companies made the bulk of their money. And what large language models and AI allow is for advertising to have new tentacles, new ways of working. And if you want to understand in simple terms what I mean, you can think about it this way. Language is very revealing. If you got a text message or an email out of the blue and it had just one word and that word was yo with multiple zeros and a lowercase y, you would think, I can maybe imagine who I’m talking to here. We infer a great deal. That’s one word. What users of ChatGPT don’t realize is the system is studying them. And changing itself to sweeten the experience of using the product. And the long-term aim here is to be the first port of call between you and online commerce. So if back in the day you would search for a book that you wanted to order on Google, it might redirect you to Amazon, then you get the Amazon app, cut Google out of the equation. Now, the hope is you use your, chatbot 3 years or 1 year, 3 or 5 years from now. This is the kind of trajectory, and that chatbot will fulfill a purchase for you. So you say, I want this book, go buy it for me. And then ChatGPT behind the scenes, OpenAI, the company that runs it, can make money off of that sale by being the intermediary between them, you and the bookseller. Amplify that or multiply that by every possible purchase, train tickets, trips, groceries, everything. And there’s a huge amount of power in flux. So yeah, should these systems be seen as trustworthy? If they are for profit ultimately, then that alone is a cause for concern. And then there are other kind of design choices around how human these things appear that are also kind of cause for concern. I can go on and on, so I’ll stop there. What we call the intention economy.
Speaker C: Do you think that we should change them? Do you think that there should be warnings that these systems should be saying, I may not have your best interests at heart?
Speaker D: Yeah, I do think it’s cause for concern. Thankfully, I think a few simple remedies to begin would allow us to make use of the tools in the ways that they are useful, because there are uses that will save us from drudgery. But that doesn’t excuse their harms. The simple remedies include don’t design it to look like a human. Don’t have it, if it speaks to you, have a human voice. I was with a researcher from Sweden today who is a computer scientist, and he said his lab does exactly that. They won’t use a human voice to perform this sort of interaction. That’s a simple, that could be law. That’s within our grasp. Everybody understands, okay, these systems speak differently because they are different. They’re not like us.
Speaker C: Johnny Penn, professor of AI ethics and society at the University of Cambridge. As he says, the systems are studying us. They are designed to engage with us. So sophisticated is the research becoming that some companies are seeking to interpret information from our faces. And just as in AI systems, the ethics when it comes to mapping our facial expressions, logging our moods, and counting our likes are non-existent. This is market research in the wild. New York City computer scientist Noah Levinson describes it as stealing your feelings.
Speaker E: Introducing facial emotion recognition. A new kind of artificial intelligence that decides how you feel just by looking at your face. It’s already in the apps, devices, and websites you use every day, including this one you’re using right now. And it’s so fast and invisible, you don’t even notice that it’s learning all kinds of interesting things about you. Like, “You clearly prefer men!” Yay!
Speaker A: Wow!
Speaker E: Isn’t this tech fun? It’s like an expert you never asked for.
Speaker C: Leventon’s short film exposed how social networks have deliberately created software that collects data points in social media users’ faces, voices, and text messages, and uses them to target adverts at exactly the right person in the right place at the right time to make a sale. Noel Leventon wrote and narrated it.
Speaker E: Stealing Your Feelings. Well, it’s an interactive film that watches you back. The movie is about sort of the societal risks posed by emotion detection AI, which is a new kind of artificial intelligence that claims to be able to determine how a human being feels just by visually analyzing their face. And so Stealing Your Feelings watches you react to the content it shows you, and it makes a series of decisions about you in ways that you probably wouldn’t want a machine making decisions about you.
Speaker C: So why did you want to do it? What prompted you to want to do this?
Speaker E: It started as a hypothetical thought experiment. Like, I knew some things about emotion detection AI because I’m an engineer, a computer scientist. And so I’ve been working on computer vision problems related to the human face. And I thought, you know, there’s this whole class of applications like Snapchat and TikTok and Instagram that we’re giving unrestricted access to our faces all the time, right? We’re taking selfies and putting the silly filters on and doing dances. And so I thought, well, wouldn’t it be far out and quite sinister if every time we did that, these companies were secretly using emotion detection AI to figure out how we feel and then correlate that with other inputs from our devices, like our geolocation or, you know, what content we happen to be consuming at that time? Then they would kind of know how you feel about wherever you’re at or whatever you’re looking at, and that’s like very valuable information about consumer behavior. They could sell. But I thought, you know, that’s a really— I liked it a lot. I thought it’s a really far-out kind of dystopian science fiction idea. And I guess I started Googling to see if anybody else had thought about it, and I quickly discovered it’s not really science fiction because Snapchat had filed patents indicating that they were— they had been thinking about it for a few years.
Speaker C: Okay, so what you’re basically saying is that with that information about how you’re feeling coupled with other information, you could find out that some people had a particular liking for multi-story car parks because they’re happy when they’re in a multi-story car park. Just as a stupid example, but that sort of thing. Is that what you’re saying?
Speaker E: Precisely, yeah. And actually, to be specific about the examples in the Snapchat patent, they drew pictures of a parade, for example. How do people feel when they’re at this parade? For example, a Pride parade, or like a parade celebrating operating, well, anything. Another example was, what about a political rally? What if we get how everybody’s feeling when they’re attending a particular political rally? That sort of thing.
Speaker C: Okay, well, I mean, in the case of the political rally, that’s very dangerous, isn’t it? Because that basically means that you can find out what someone’s political allegiance is.
Speaker E: Yes, or you could, depending on whether you believe that emotion detection AI is settled science, you could either correctly extract that or incorrectly extract it, which might be just as dangerous.
Speaker C: So why are companies doing this? What’s in it for them? It’s a lot of effort to go to.
Speaker E: Oh, it’s just the money, isn’t it? I mean, this is how they make their money, is by surveilling us and extracting data about our behaviors and preferences and building personality inferences about us to sell to advertisers or whoever else is interested, which is just about everyone who sells anything.
Speaker C: As Levinson’s highly acclaimed film points out, this data scraping is perfectly legal because Snapchat and other networks have patented their algorithms, and you agreed to it because you accepted the terms and conditions. And there is no law against harvesting users’ reactions and emotions. If there is, by his own admission, Levinson would not have known about it since he called it his TL;DR. That’s too long, didn’t read. How I love acronyms. You could almost say that they could deliberately obscure understanding because they don’t spell it out. Though at the heart of this, or it, is also our own willingness to be enticed into this relationship. People have always wanted their toys to come to life. Children have always had their secret friends. Boats are called she. Cars have unique characteristics. Train drivers talk to their trains. We have always anthropomorphized, a tendency that according to the psychologist Professor Angela Sasse, who studies human-computer interaction and computer security at Germany’s Ruhr University in Bochum, and as Professor of Human-Centered Technology at University College London, we do it because we want to belong. According to Professor Sasse, the reason we anthropomorphize is because it’s familiar, easy. We know how to speak with and interact with humans. If somebody says something in a grammatically incorrect sentence, you can still understand what they mean. And it’s convenient to assume you can speak to the system as you can to a human, rather than learn the vocabulary and grammar, the programming that drives the behaviour of the system. Anthropomorphising also has emotional benefits. If you think your pet understands and loves you, it’s a comfort. Similarly, according to Sasse, if you think the machine understands you, then you think it’s on your side. The problem is that there will come a point, she says, when the system won’t interpret or respond correctly, and then the whole illusion comes crashing down, and you don’t have the tools to instruct the system correctly to get the answer you want or get it to do what you want. It’s something that Professor Sasse elaborated on in an interview with us about the issue of mistaken intimacy in 2021. She tells me her research shows a general willingness to open up to computers that dates back to the 1960s.
Speaker A: So it’s actually, it goes right back. I don’t know if you know Weizenbaum, who was one of the pioneers of basically of computer research, right? And he wrote this program called ELIZA, which essentially just used this sort of reflection, this mirroring technology where ELIZA would just Bessie Repeat, and he was himself very surprised to find how readily people would engage with it and Bessie Start. And what came out there is that in contrast to confiding your problems or discussing your problems with another human being, people did it because they felt the computer wasn’t judging them.
Speaker C: Which is odd, isn’t it? Because we know that that information must be going to a human being. I mean, people are apparently prepared to confide some of their most intimate secrets. They’re prepared to talk to people about their sex life.
Speaker A: I think when you say we must be aware, I think in that moment, you know, and if you’re in a particular sort of like train of thought or flow, and you know, if you start confiding something or writing something, right, that in itself, so there’s partly cathartic optic effect to doing that. I don’t think you really are aware at that moment what potentially could happen with it, who else could use it, and so on. That once, you know, we only have one focus of attention, and if your focus is on, you know, I’m burning yourself, I don’t think people really think that much about the consequences, you know. And I think they certainly don’t think back to the terms of service or whatever that they signed up to and in which they agreed that this data could be used. And most people don’t read those because they’re far too long. And even if you had time to read them, you’d also need time to take a course in legal, in law, and to understand what you’re actually agreeing and signing to. And I think that’s one of the fundamental problems. So I think that awareness isn’t really there.
Speaker C: And this is in despite of all of the stories and the wealth of material that comes out that says that, of course, we are being analyzed. We know we’re being analyzed. Lots of people notice in a strange way that suddenly adverts appear on their web searches because, say, for example, they’ve decided that they want to go on holiday to Portugal. It’s—
Speaker A: I think the part of the problem is that this kind of— I mean, we actually talk about a habit of disclosure. You have to give your information so many over and over again. And people are in the habit of disclosure that it’s Actually, this is one of my PhD students did a study on this where people were completing the non-mandatory fields even though they didn’t have to do it because having to decide whether you to give that information or not is more work than just executing this habit of, you know, if you’re so used to filling this information in, you just do it without thinking about it because stopping and thinking about it is effort.
Speaker C: So what you’re saying is that we fill in information that we don’t need to give, that we’re not necessarily asked to, it’s voluntary to give it.
Speaker A: Because it’s a script, it runs like, you know, there is that cue, you’re familiar that somebody puts a form in front of you and you fill it in. That’s what we do hundreds of times in every week in our interactions. And I would say that actually some of these companies collected it, they’re very aware of this. And they’re using this mechanism. So much of our behavior, 80 to 90% of our behavior is automatic. And that means we don’t think, we just, you know, our brain recognizes a cue and executes a script that’s stored in long-term memory. And it’s very, to stop that, I mean, you sometimes see that also, for instance, when people click phishing links or so, that suddenly maybe a few seconds later, they suddenly go like, ooh, maybe I shouldn’t have done it. But it wasn’t in time to stop this automatic execution. And stopping, unlearning some of those habits is really, that requires effort. And it’s actually something that in our work now, we are really working, we have to work very hard to get companies to understand that they need to, if they want a new secure behavior, is that they really need to remove all the cues to this old behavior. And that’s also what you would need to do as an individual. I think that’s one, you know, you really, it’s a lot of effort to undo a behavior that’s very deeply learned and embedded. You need to do that very consciously and very systematically.
Speaker C: There you go. We just love talking to our devices. They are veritable confessionals. At the start of our interview, you there. Angela Sasse mentioned Joseph Weisman. He invented Eliza, an early psychiatric chatbot. Weisman, who has passed away, but Eliza, being an inanimate computer program, continues to exist on YouTube, and you can go and enjoy its reflections because that’s what it does. It plays your questions back to you so that you expand on them. A tendency that is concerning Eloise Skinner, a psychotherapist who pointed out the incredible dangers in using systems for purposes they’re not really qualified to perform. Psychotherapists are professionally regulated. They are responsible for the advice that they provide. A computer system does not have the same controls.
Speaker B: So I was working on a piece recently for a publication about the dangers of AI, people using AI as a therapy tool in psychotherapy. Obviously, this is like a concern of that people would not get professional help and instead would just feed their problems into an AI tool, which obviously has many issues around confidentiality and privacy and things like that. But also, as you mentioned, the one of the issues is that these tools are generally trained to affirm your perspective. So what will happen is it will start to repeat your narrative back to you. And for people who are stuck in sort of patterns that they actually want to break, sometimes that really isn’t what you need from therapy. Or in fact, you don’t really need advice at all. You actually need someone to listen and then to give you some more thoughts and then to listen again. And so it can be— there can be areas in which this could be quite dangerous, I think.
Speaker C: But that— you’ve highlighted an area that many people have focused in on, which is people start to see these devices as confessionals. They’re prepared to actually give them more information than they would to a human being, because they think that it has a human-like response, but they think it is not human. There is a confusion that is occurring that is actually also exploitable too. Definitely, yeah.
Speaker B: One of the biggest warnings that I give to people who are starting to use these tools in sort of a therapeutic/counseling tool is that the information that you’re giving is essentially being used to then formulate further responses, but also to train the general models as a whole and like taken back into the system. And, you know, these are owned by private companies or companies that have different motivations. And so, yeah, nothing is objective or objectively received?
Speaker C: In the manifesto that we wrote in the book that I just published, we suggested that that is possibly the problem, that these devices should be loyal to you. They shouldn’t be loyal to this big tech company, that these— that the data that you’re giving to it and that the system itself should be working in your Do you think that there should be some legislation that actually mirrors that? Do you think that we should start saying these devices have so much personal information that really they’ve got to look after us? Do you think that it could be a beneficial thing that we actually have these devices as loyal to us?
Speaker B: Yeah, I definitely think that could be. I mean, I guess one risk is that you start to have something some kind of like government-authorized system, like version of the system that is loyal to each individual person.
Speaker C: You’re a psychotherapist. You have a professional obligation to your clients. You have agreements that you enter into with your clients, and you would presumably face some professional opprobrium if you betrayed that. You don’t have the same thing with these technologies. Definitely.
Speaker B: And I mean, there’s a problem with humans as well because a lot of people are unregulated practicing psychotherapy without regulation. And a lot of people, you know, it’s not necessarily checked whether you have qualifications or certifications if you’re just looking online for like therapeutic advice. And to some extent, like, AI is gathering information that’s already out there. So someone could put up unregulated, bad, poor advice online, and then that’s just fed back to you. So there are dangers in, in technology and in real life as well. But yes, as you mentioned, I think, you know, people who are seeking professional help, who do it through a regulated body and who seek someone with particular certifications, then yes, there are a lot more protections in, in that sense of like being able to have a proper system where information is dealt with.
Speaker C: And those protections are going to be ripped up. The Americans say that there will be no sanction for 10 years on the AI companies. That you could get an AI psychotherapist, which obviously you wouldn’t particularly be too welcoming of, that could be giving people completely erroneous advice and doing a huge amount of damage with no comeback on anybody at all.
Speaker B: I mean, I think that’s definitely possible. There’s also— there are pros to people being able to access advice in a way that’s personal to them. You know, for a lot of people, psychotherapy is extremely expensive, it’s very hard to access, there’s a lot of shame and social stigma still around seeking mental health support. To the extent that people could be able to have some, some level, even low-level or signposting assistance from technology, I think that could be a really good thing. So there are definitely pros and cons, but yes, it would need extensive regulation.
Speaker C: Psychotherapist Eloi Skinner. As you can see, it’s an area where already definitions are sliding. In our interview with Cambridge’s Professor Johnny Penn, he pointed out that the development of AI was rapidly raising a number of moral, philosophical, and ethical questions that we had never thought about before in relation to technology. And the picture is becoming very confused. In the US, President Donald Trump’s forthcoming big beautiful bill He suggests that there should be a moratorium on AI regulation for 10 years so the technology can develop. It’s a position the US Vice President, J.D. Vance, had already set out in a speech in February at the Munich Security Conference, where he called for regulation on AI to be suspended, something that the UK government appears to have fallen in line with. It is currently refusing to implement copyright protections against AI for writers and artists who ironically are playing a pivotal role in the camouflaging of AI because it is their work that is allowing the interfaces to become confusingly humanized. As many of those we interviewed pointed out, to achieve its financial aims, AI needs to be humanized so that humans think that they can trust it. That ‘it’ word again. And yet the descent into a dialogue with the machine goes on apace. Within the last 2 weeks, the New York Times published a story about a man who discovered that his daughter had been in a relationship with Apple’s Siri for 2 years and had been using the technology for life guidance advice. A particularly unsettling development given that 2 days a few days ago, Apple researchers revealed that advanced AI suffers complete accuracy collapse in the face of complex problems, according to The Guardian. While last Sunday, Sky TV ran a piece in which Sam Coates, its deputy political editor, revealed that ChatGPT had automatically written a script transcript for him because the system’s programming indicated one should exist. Coates asked the system about this and then stated that ChatGPT had lied to him. An obvious impossibility, but still one caught in our AI illusion. Just how fast and how dangerous this situation is becoming was pointed out by Tom Dennis, the CEO of Serenity and Leadership, a coaching and technology change consultancy. He’s also the author of Rewilding the Corporate Mind.
Speaker G: The more we go on, the, the more time goes on, the more sophisticated the, the AI is. I mean, if, if you’ve been using a particular AI for, say, the last 18 months, there’s been regular upgrades, and each time it is more powerful, it’s more perceptive. It’s occasionally— it makes total nonsense, but it depends on what you’re, you’re doing. But, you know, Mercedes now have got a whole system that integrates with your calendar, and it’ll say, well, look, the traffic is, is this, we need to leave at such and such time, and, and it tell the owner when it needs to leave. And it’s very difficult in a sense not to start to anthropomorphize— I can’t say the word— the car, to say, okay, particularly if it’s speaking with a female voice and the person is a male, or, you know, or vice versa, to develop some sort of relationship.
Speaker C: And again, though, that, that is a delusion, isn’t it? Because there is no relationship. This is a piece of technology. You don’t have a relationship with it.
Speaker G: Well, I think we, we create that. And that’s where, in the same sense that you talk about giving a car or a boat or even more a pet a name, I think the more that they’re are working on creating humanoid robots, and the realism of the robots is increasing all the time. And, you know, they’ve had sex bots for years, and those are becoming very sophisticated. It’s difficult. We, we live in a fantasy world in a sense. And, and of course, the use of robots which don’t actually need to be that humanoid, but just taking an interest in a geriatric, somebody who’s old and infirm and may have, um, some sort of memory issues or whatever, you know, you see their reaction to someone who just wants to cuddle them or has a sense of a smile, and you think, my goodness, we are very vulnerable in this.
Speaker C: That’s a very interesting point because, as we know from Google, that it knows more more data points about us than we actually know, you can see that these robots could become your extended memory, that the actual robot, that the AI carer could start to actually prompt you with what your memories are meant to be, so long as you can trust that it hasn’t made those memories up. This is an extremely worrying place that we’re potentially going to.
Speaker G: Yes, and it’s all the motivation, which is all about money, is so powerful that I don’t see us reining ourselves in. So for me, I think it’s, it’s just important to be really bringing a level of mindfulness to the impact and the way it’s being used and trying to help people to, to see. But at the same time, you know, the, the fact that a robot can operate in a way that a human being can’t, or can analyze things in a way that, you know, 1,000 doctors can’t, it’s such a powerful argument. It’s a slippery slope.
Speaker C: Head of Serenity and Leadership and a former Marine officer and pilot, who also pointed out that one of the greatest dangers will be in computer security because hackers have discovered that the weakest link in any computer system is a human being. It’s a vulnerability that the cybersecurity industry endearingly refers to as wetware. That this issue is proceeding at high speed is now undisputable. Within the last 2 weeks, ChatGPT boss Sam Altman acknowledged that the practice of people saying please and thank you to AI was running up millions of dollars in energy costs. Only last month, the AI company released a postmortem explaining why its ChatGPT GPT-4 model had begun responding with unsettling praise and agreement. People using the technology thought it was creepy. The company said an update had caused a wave of user complaints and memes as ChatGPT validated clearly flawed or dangerous ideas. The online AI Bulletin Atlas commented that the issue stemmed from short-term feedback loops that neglected broader user behavior. OpenAI admitted that while the model was supportive, it felt disingenuous and made users uncomfortable. In response, the company reversed the update and is now revising system prompts, retraining strategies, and introducing new safety tools. It is also testing user control features including personality settings and real-time feedback options. In short, it is changing the fine-tuning of the computer system to its customers, highlighting another very real problem that people are prepared to enter into relationships. Being able to harness human gullibility And to use AI systems to exploit that, either as deepfakes or by hacking their AI agents, represents a profound threat to companies and to people themselves. For many observers, the answer to the problem is for AI to stop being human, to stress its differences and apartness that the European Union has asked for for in the past. In a bid to stop AI becoming confidants for the old, the young, and the vulnerable, the EU said the technology should clearly be labeled as a computer system, and the functions the technology provided should be listed out. A point that the technology law specialist Mark Deane from Wigan and the co-author of my latest book on AI AI on Trial made.
Speaker F: I think we have to probably go back to basics on this one, and I think we are dealing with a problem that we have had for a long period of time with our relationship with technology, which is, as humans, we like to believe in magic. We like to believe in the wizard rather than the man behind the curtain. And what that actually means is that we have been very vulnerable to where AI and certain technologies have been presented to us as being very human-like in behaviours. So we have anthropomorphisation of technology. It is across the board. We have robots that look like humans. We ascribe values and attributes to them as though they were human. You know, we talk about an AI hallucinating. Now, it may well hallucinate in certain instances, but that’s an experience, that’s a behavior that we might otherwise associate with humans. So the starting point is that we as humans really want to believe in this technology as having a human-like feature. We call it artificial intelligence because we assume that it is going to be like us, intelligent, but it’s not. And so there is a real danger that we have moved away from the core building blocks of what this technology is about, and that the way in which it is developed and the way in which products are introduced to us have really relied and used that and leveraged that as a way of making it very blurred, the distinction between the technology on the one hand and the product that we’re actually receiving. So from a legal point of view, that actually makes it very difficult for us as well, because we’re actually actually now starting to have to almost legislate and regulate behaviors that are very human and emotional in many ways, rather than actually just dealing with the core technology, which is what we should be dealing with. And so therefore, we’re having to give some thought to, for example, coercive behavior, which is a type of behavior you would ordinarily associate between humans. And we already know that in the human the analog world, it’s taken a very long time for us to be able to legislate and deal with coercive behaviors in relationships. And we’re now having to deal with that in the technological environment, and that creates some very real problems and some real difficulties for us.
Speaker C: I mean, it’s interesting, isn’t it? You mentioned that this term gaslighting has only just emerged in terms of coercive behavior, and that was based in a film, I think, from the 1930s. 1930s. But it goes even further than that, doesn’t it? There are stories about little toys coming to life. We have this very, very childish desire to make the machines come to life. And we’ve done this for a long time. People have anthropomorphized trains. Train drivers always tell or say that a train is a she. People anthropomorphize their cars. Go back even further. Boats have always been known as she’s. We have always turned them into a female. It seems to be something very deep within our psyche that wants these things to actually have some personality. We seem to be a little lonely.
Speaker F: I think that is absolutely right. But I think that is because we have always said, and you and I have had discussions before about about the need for trust to exist within certain technologies. And I think as humans, we like and find it easier to trust things if we can have some affinity towards them. And so when we’re talking about things as being a he or a she, I think as humans, it enables us to trust that thing more. Now, what is absolutely critical with AI and certain technologies is that it’s not trust in and of itself that is important, it’s trust and transparency. They live together, they go together, and if anyone takes priority, I think we’d have to say it’s transparency. In, in the Manifesto for Responsible AI, one of the first points that is in that manifesto is the fact that transparency is necessary in order to be able to gain trust, and therefore what is really critical is that we’re transparent about when something is artificial and when it is not, so that we are able to trust it.
Speaker C: So that seems to be the issue at the moment. We are not getting that transparency. Say, for example, ChatGPT. I spoke to one of the leading copyright lawyers in the US, perhaps the leading copyright lawyer, who told me that if you were writing a book with ChatGPT, it owns the book, it owns the copyright because you haven’t read the terms and conditions. That’s not very transparent, is it? The technology should say right at the top, you can’t do a book with— I was about to say me, but it’s not me. You can’t do a book with ChatGPT because if you do, the owner of the technology owns your book.
Speaker F: It should be explicit. Well, there’s transparency and there’s transparency. We’ve all come across situations situations where there may be things lurking deep down in the terms and conditions hidden away from sight. Now technically, that may be disclosing it to the other side or disclosing it to somebody who’s trying to engage with it deep down those terms and conditions, but I think what we’re talking about here is something more profound and a need to be transparent and very much in your face before you engage with this so that there is clarity of labeling of when something is real, when something is artificial.
Speaker C: But there’s yet another issue in all of this, isn’t there? Which is, I’ve actually met people, allegedly top businessmen, who have told me that their ChatGPT system thinks like them. It says that the ChatGPT system has absorbed their thinking. And that has created what seems to me to be a terrible confusion if somebody thinks that a piece of technology has absorbed their thought patterns by their interrelationship with it, then the possibility, as you say, to abuse people, to take advantage of them and that relationship is massive.
Speaker F: And the real problem is that we have missed a very important educational piece here, actually informing people about what this technology is all about. And how it is created. And that is the ultimate transparency that is needed and that is somehow missing. It’s all well and good us being able to label something to capture it, to say that this has actually been the product of an artificially intelligent bot or whatever it might be, product, platform, call it what you like. But there is a fundamental piece missing here, which is that unless and until we understand what artificial intelligence is, that it is a combination of computing power, statistics, and datasets, and how those datasets are created, then there’s a real risk that we still continue to perpetuate the problem because we don’t understand what we’re really dealing with here.
Speaker C: The specialist technology lawyer Mark Deem, who for reasons of trust and transparency, I just want to repeat that I have written a book with about the ethics of AI. Called AI on Trial. The sort of spelling out of intent that we really do need about this relationship between us and the tools of big tech. We have to know that this relationship is in our interest, because as many of those we interviewed for this program has stressed, the main source of revenue for big tech is advertising, and it can fine-tune that by now. Blowing our minds. It’s this duality of motivation that sits at the heart of our very one-sided love affair. The technology doesn’t care about us. It only pretends to. And it is this fake intimacy that is preoccupying Professor Sherry Turkle, the Abby Rockefeller Moores Professor of the Social Studies of Science and Technology in the program in Science, Technology, and Society at the Massachusetts Institute of Technology. Author of The Empathy Diaries, whose publishers Little Brown are about to release her latest book, Artificial Intimacy. AI, says Professor Turkle, only wants to be your friend because it wants something.
Speaker H: There’s an experiment where you leave people alone in a room And you say, you just have to sit here for a while and without a phone, without a book, and just quietly and 15 for 15 minutes. And oh, by the way, if you need to give yourself electroshocks, here’s the machine. And the people are college students and they’re grossed out and they say, absolutely, electroshocks, that’s like crazy. Why would we want to give ourselves electroshocks? But after 6 minutes of total non-stimulation, they are giving themselves electroshocks because they cannot tolerate that just sitting. And we are so used to having a regime of constant stimulation that solitude becomes something we kind of can’t handle. It becomes intolerable. And I think that the reason that that’s dangerous, and I think there that technology— I mean, I’m not trying to name there is one culprit, but technology is a culprit— is that we don’t recognize the role of solitude in creating a self. It’s in solitude that empathy is born, because in solitude you get a feeling of sort of, I know who I am, I’m okay who I am, I know who I am. So that you can then turn and reach out to another person and say, well, who are you? And you have the feeling that you’re talking to a different person who knows who they are. You can tell me. And that is the beginning of mutuality. If you have a self that the minute they’re not diverted, they’re looking to another thing on their phone. When they look to you, they’re looking to you to tell that you to tell them who they are. And we shun those people. I mean, we know to sort of shun those people. We don’t want to be with those people in some way. We know that that’s— that they’re not grounded. So phones and our constant diversion by phones, I believe, is undermining the capacity for solitude where empathy is born. And that is my true concern in the way in which empathy— I call it an assault on empathy, where there’s an assault on empathy, which I think is so essential to mutuality, to conversation, to the appreciation of difference, which I think is really what we need now more than ever before. We need that appreciation of difference and appreciation of each other to move forward as people, as democracies, as to get back our human connections.
Speaker C: So, I mean, you said that there is this level of intimacy that is dangerous. Do you think that—
Speaker H: No, I didn’t say that. All right.
Speaker C: Okay. But what I was meaning was intimacy with the machine. Yes. Now, do you think that then, to an extent, we’re being programmed by the machines or that we’re reacting to them? You seem to be— what you were saying was that we needed a number of likes, or some people needed a number of likes before they would move. Is it that we’re letting the machines dictate our lives to an extent?
Speaker H: Well, what I think is that, you know, when I stopped you saying we’re too intimate with the machines, I think it was ironic that I stopped you because I thought you were using the word intimate. With ourselves. I called, I changed the title, but the title of my first book before I changed it to The Second Self was Intimate Machines. And I was referring to computers. I was referring to the kind of connection people form with digital objects. And there is a kind of deep connection, a kind of mind meld, a kind of feeling that it is me, I am it. And that’s different from being programmed by a machine or controlled by a machine. Our screens protect us from feeling a kind of vulnerability, and we like that sense of protection. We find that comforting, and so we’re drawn to them, and we’re drawn to the constant incoming, the constant always something new. Always being able to be diverted that the screens provide. And what that does is make us vulnerable to having face-to-face, in-person human connection with the— in with where you can’t turn it off, where people see the changes of expression on your face, where people see your body language, where you kind of can’t hide. We’ve become, in a way, frightened by that, frightened by our own exposure at that.
Speaker C: In terms of the research that you’ve been doing over that period, what are the things that you’ve discovered about our relationship with computers and phones that surprised you the most then, or most demonstrated this ironic input that you talk about in your book?
Speaker H: Well, the most, over the past few months, the most disturbing has been that despite what I just said, people have been extremely interested in a chatbot called Woebot and in a chatbot called Replica, which are both computer-available entities to do psychotherapy. So for one of them, New York Times— both of them, the New York Times called me and said, what is my comment? In both cases, I said, well, you’re just calling me to have me say something bad. New York Times said yes. And I said, well, let me go on and talk to this replica, you know, let me chat, try this. So I went online, and this is the height of the, you know, COVID, and I I’m lonely. That’s my big problem. Can you talk to me about loneliness? And the chatbot says loneliness is warm and fuzzy. So I went back to the— it was like a glitch. It was like just a glitch. I’m sure they fixed it the next day, this glitch. So I went back to the New York Times reporter and I said, look, it’s a glitch. I’m not criticizing chatbots, but Of course it’s a glitch. It’s not a person. It doesn’t have a body. It’s not afraid of dying. I’m just making the point that if you’re— I don’t really understand why in this moment that is so about bodies and fear of dying, it’s so sad and such a cry for help that we need to be with each other as people, that people are turning to to artificial intelligence. And so the fact that during the pandemic, the people were turning to artificial intelligence rather than each other, or when they didn’t have each other to turn to, made me feel, I mean, I was very surprised. And then I thought, well, it’s not that they prefer it. It’s that people were lonely to point where they felt it was all they had. And I found it really an indictment of how we had organized our communities and our outreach towards each other as humans, that so many people thought that a machine with a glitch that said that loneliness was warm and fuzzy is what people kind of had for each other.
Speaker C: But isn’t one of the reasons why people are prepared to tell these computers things that they may not share with other people is because, as I understand it, it’s because they consider the computer or the mobile phone or the app less judgmental.
Speaker H: Well, that’s our, you know, that’s what I mean about we have failed each other. And it’s our crisis in empathy. That’s another reason that I write the empathy diaries, because I write about the engineering culture and its tremendous lack of empathy towards me as I came through it, towards, you know, what it was like to teach MIT freshmen. A freshman, a student commits suicide who’s in my class, and what the class talks about is whether or not he would have fell to the ground faster. He jumped off a building. And the class talks about whether or not he would have splattered, he would have reached the ground faster if he hadn’t been holding on to something. He jumped out of the window holding a desk. And they say, no, he could have done without the desk, he would have reached the ground just by Galileo’s laws. I mean, the way in which when engineers use engineering language to talk about people, how they leave empathy aside. And to the extent that our language has been filled with the language of engineering and science, with the language of objects to talk about people, I think our language for talking about people has been demolished, undermined. And so I think that computers are good for many things, engineering culture is good for many things, but it’s not good for talking about human relationships and human understanding. And I think in the current situation, our current moment, you can see engineering language creeping more and more into our way of talking about everything. And that’s one of the reasons I wrote the book, to get us back to the great question you started with, is why now? Is because I feel that the culture of engineering that caused me such troubles as I tell my story of a woman and a humanist at MIT, are now inhibiting our ability to seize our moment now and move forward with a greater and more empathic politics towards each other.
Speaker C: Is it too far to say that these machines are after our soul?
Speaker H: They’re not after anything. The machines do not have will. The machines do not have a personality. We’ve created an industry that does very well by scraping our data and selling it. That’s all that’s happened. The machines have not taken on a mind of their own. They’re not like chatting about this interview and saying, well, how can we get Peter and Sherry to, you know, to buy some new product they’ve invented. The commercial interests that have created these machines have seen a tremendous business opportunity from figuring out, you know, I have a visual review. I see eyeglasses. And I should get a checkered shirt. Well, you know, they’re busy figuring out rimmed, steel-rimmed glasses, checkered shirts, how can I sell them more. That’s it. And not your soul is not particularly, you know, it’s not their soul right now, it’s, it’s your next purchase of eyeglasses, shirt. Plus, insofar as they’re interested in your soul, they’re analyzing what you look at on screen so they can send you more of it along with the ads for the eyeglasses and the checkered shirts. Now that’s hardly, you know, that’s become an existential dilemma because once you’re being filtered only the people and the certain people, because your news is being filtered by them and what you see to buy is being filtered by them, but in particular your news is being filtered by them. So it’s not the machine. It’s the companies and the policies behind the machine. So I think we have to get straight to the, you know, who we’re fighting. This really is a fight that consumers can win.
Speaker C: That was MIT’s Professor Sherry Turkle, whose latest book, Artificial Intimacy, is about to be published by Little Brown. If, in spite of all of this, you’re still staying on Twitter LinkedIn, you can connect with me, Pete Warren, and the password team, Blue Buffery and Jane Wyatt. We are all too human, and we’ll be back next month with another touchy-feely exploration of the world of tech. If you can’t wait until then, then go to our website, www.futureintelligence.co.uk, because we We want you to sign up so that we can tell people how many readers we have so we can make an income to bring you this insight. It’s free, and you can find out more about the ramifications of technology in those links that allow you to enter the metaverse. You’ll also find links to our new TV platform, which includes interviews with some of the leading thinkers in the technology world. Because, as Angela Sasse says, it’s convenient to assume you can speak to the system as you can to a human, rather than learn the vocabulary and grammar that drives the behaviour of the system. Learning what’s behind will help you map your way in a world where increasingly people are losing the ability to read the map and drive the car. Thanks for listening. And goodbye.
Speaker B: This program has been brought to you by Resonance FM. If you like what you heard, please support our work by making a donation at resonancefm.com/donate.
