PassW0rd – 11 October 2023 (The Problem With Computer Evidence)

Speaker A: This program is brought to you by Resonance FM. If you like what you hear, please support our work by making a donation at resonancefm.com/donate.

Speaker B: Hello and welcome to Password on Resonance FM with me, Peter Warren. In this month’s programme, we examine the greatest issue of our age: the infallibility of computers. And we’ve found that they really have come up wanting. Two weeks before the Prime Minister Rishi Sunak is due to discuss what controls are necessary to rein in AI development, an investigation by Future Intelligence, Password’s parent organisation, has discovered that the computer systems that develop the data the AI systems depend on are known to be flawed. Even more shockingly, we have found that the Law Commission, the organization that draws up the laws in the UK, was informed of this and yet still went ahead in defining digital evidence as being conclusive in Section 69 of the Police and Criminal Evidence Act. As a result, according to top experts interviewed in an exhaustive investigation you can now hear on Password, innocent people have been convicted, lives have been destroyed, people have committed suicide, and families have been wrecked. Oh, and a lot of people have been bankrupted and lost their income. To find out just how harrowing that experience can be, You only need to look at the experiences of the sub-postmasters wrongly convicted for fraud and false accounting as part of the Horizon scandal. At its root was a computer system created for the Post Office by the UK computer company ICL, which was subsequently taken over by the Japanese company Fujitsu. Horizon had been meant to be a system that allowed the distribution of benefits and which provided a platform for all Post Office services and allowed subpostmasters to balance their books at the end of the day instead of using time-intensive written ledgers. During its development, problems with the system led to the abandonment of the plans to use it for the distribution of benefits, but it was decided to go ahead with it to allow the subpostmasters to balance their books with the central Post Office computer system. Unfortunately, the system didn’t work properly and threw up errors that led to imbalances in the books which the contracts with the subpostmasters required them to make good. Due to this, some 736 subpostmasters were accused of theft by the Post Office, one of the oldest and most powerful institutions in the UK which is allowed to level its own prosecutions and, at that time, one of the most trusted organisations in the UK. The data from the system was presented in court as proof, even though the Post Office and many of its chief witnesses knew that the system was generating accounting errors on a frequent basis. Despite this, the Post Office pursued the cases against the subpostmasters on the basis of the computer system’s assertions relentlessly and pitilessly. Here’s Paul Marshall, a barrister for some of the defendants, on his clients’ ordeal.

Speaker C: To give illustrations of that, I have 3 clients. Tracy Felstead was prosecuted at the age of 19 in 2002 for a shortfall of, of somewhere in the region of about £10,000. I think it was £11,000. It arose while she was on holiday, and she was confronted with it when she came back. She protested she knew nothing about it, and she knew that there were all kinds of balancing problems before she’d gone on holiday and was always told that they’d resolved themselves. But anyway, she was confronted by this. She was suspended, prosecuted. She had an expert witness who was going to be called for her, but because of expense protested about by Fujitsu and the post office, he wasn’t called. A chap called Michael Turner, very expert computer, computer expert and technologist, who, who was an expert witness for years, and he wasn’t called. Unfortunately, he’s now deceased. He wasn’t called. She was convicted in 2002, immediately imprisoned in Holloway at the age of 19, and whilst in prison, she found a fellow inmate hanged traumatically. Her conviction was quashed in 2021, which means that she was convicted— she was subject to a conviction for theft wrongfully for 19 years. It’s completely wrecked her life. Her entire adult life has been dominated by that fact, and it’s caused her long-term and probably permanent mental health illness. And despite her conviction being quashed in 2021, she’s, she’s yet to be compensated. Another client, for example, Janet Skinner, was convicted in 2007. She was encouraged to plead guilty to false accounting instead of a charge of theft on the basis her lawyer told her that she wouldn’t get a custodial sentence. She nonetheless got a custodial sentence. She had two mid-teenage children at the time. She expected to be acquitted and go home to see them. She was imprisoned When she was released from prison, having lost her home, she was again pursued by the Post Office for a sum said to be owing by her and not paid by her, and ended up back in court with the allegation that she had money outstanding owed to the Post Office. In fact, the loss of her home had resulted in there being no equity in her home, and so she’d received no money, and she actually had nothing at all left after the Post Office had finished with her. The consequence of her being prosecuted for the, or arraigned for the second time in the criminal courts, was that she had a complete nervous collapse and was in hospital unable to walk. She had basically an episode of paralysis which lasted for the better part of, well, for certainly more than a month and possibly 5 or 6 weeks. She was unable to walk and they thought she’d never walk again, and she’s never recovered. That’s just two illustrations of the consequences of what the Post Office did, knowing that its computer system was deeply flawed.

Speaker B: Another indirect victim of the Post Office prosecutions was Millie Castleton, the daughter of Lee Castleton, another of the defendants. As Marshall points out, the impact of the case goes much further than just the defendants and should be a source of shame for those who persecuted them.

Speaker C: It’s not something that one expects to happen in the United Kingdom, and one of the most troubling aspects of all of this is that despite what is now known, nothing to date has been done to hold any of those from the board downwards to account for the harm that was and has been inflicted by the Post Office on its victims. And this runs to, because of Families Concerned, thousands of people. And I’ll just give an example of that. Millie-Jo Castleton, who is the daughter of Lee Castleton, who was subject to a civil claim in 2006, has given eloquent testimony in her witness statement to the Post Office inquiry that the consequence for the whole of her life since a child— I think it started when she was about 8— the whole of her life has been dominated by the Post Office. And after her father lost his business and, uh, all his investment, she developed anorexia nervosa She became a pariah at school. She used to pretend to her parents that she had friends because she was so friendless, and she almost died from anorexia on more than one occasion. During her tertiary education, her body weight went down to 5 stone, and she had to take a year out. This is the kind of collateral damage inflicted on families who are closely engaged with those who are the victims of the Post Office. And the scale of the suffering is simply enormous. And to date, no one has been held to account. And I find that as shocking as the events themselves.

Speaker B: Paul Marshall, a barrister representing some of the defendants in the Post Office Horizon scandal. But this doesn’t just end there, because the findings in the Horizon case have hit at one of the foundations of the Police and Criminal Evidence Act 1984, Section 69, which is the admissibility of digital or computer evidence in court. Because, as Stephen Murdoch, Professor of Security Engineering at University College London and head of UCL’s Information Security Research Group points out in no uncertain terms: computers get things wrong all of the time, and their evidence should be viewed as hearsay, and challenging them should not be heresy.

Speaker D: So I’ve been interested in computer evidence for a long time, long before I heard about the Post Office scandal. And that’s because of my work in banking security. And often cases where people lose their money, I end up in court one way or another, or at least end up in a dispute because people care when their money disappears. And often the only thing that could explain what has happened is the computer evidence. And in my experience, there’s been a lot of problems about how this is presented. So Sometimes it’s not there, it’s been deleted. Sometimes it’s there but there’s reasons to suspect something might have gone wrong. There’s some errors not exactly in the part I’m looking but elsewhere. And just knowing from how computers work, computers do go wrong quite often. And the arguments made for why banking systems are correct are very vague hand-wavy arguments that surely we would know if something went wrong, or will we have expensive consultants who told us that these things were secure. And we definitely see cases where they do go wrong.

Speaker B: But I thought this was one of the underpinnings of our new cultural state, that the computer is always right. I thought that this is one of the things that we see out in the world outside that the computer can’t be wrong.

Speaker D: Yeah, it’s quite surprising. Nowadays most people use computers and they see them going wrong all the time. Yet when someone is faced with computer evidence from a large organization, there doesn’t seem to be the same level of skepticism. And sometimes these companies argue that, well, their computers are different, they’re mainframes. But it’s still software, it’s still written by humans, there’s still going to be bugs, and sometimes those bugs will have a material effect on the dispute.

Speaker B: Does this mean that I have to now do the very onerous task of going through my bank statement and adding it all up?

Speaker D: That sort of question depends on how much you’re worried about the sorts of things that might get through. So yeah, there probably will be some problems. On the bank statements. The sorts of things that happen fairly often are transactions get doubled up, which you probably would care about. Sometimes transactions get missed out. Well, if there’s a credit card, maybe you don’t know or care about that, but that happens sometimes. It gets lost. But most people have a fairly good idea of how much they’re expected to pay each month, and if something is dramatically larger, then you care to investigate, then that’s when it’s certainly a good idea to look into more details. But I think a lot of people don’t bother doing that, and some criminals exploit that. So they’ll find some way to steal £10, £20 from a large number of people, and they might get away with it because most people will either not notice, or if they do notice, they’ll just assume it’s some merchant they’ve never heard of. Because we all know in credit card statements the shop you deal with isn’t necessarily the one that’s going to show up in the bill.

Speaker B: So therefore, what you’re saying is we should be a lot more careful, we should pay a lot more attention to our computers, we should, uh, be prepared to challenge them a little more. That doesn’t seem to have been what happened with Section 69 of PACE, does it? I mean, it didn’t seem to be particularly challengeable in the most evident example, which was the Post Office.

Speaker D: Yeah, so there’s now been a lot more written about the way the Law Commission believed things would work, both from a legal perspective and a technological perspective, and their wish of how computer evidence was presented didn’t seem to come through. So for example, they argued that removing the the requirements to produce evidence that a computer is reliable would be fine, because if someone could show there was a problem, they would be able to challenge it. But what we’ve seen in practice is it’s practically impossible to adequately challenge evidence from a computer, because the, the customer rarely has access to anything about that computer system. And in many cases, the organization with the computer system is very reluctant to share anything. And so getting above that bar where you start to introduce doubt is an insurmountable obstacle for many people. And for decades, that was the case in the Post Office scandal.

Speaker B: But there is also— and we saw that to an extent in the Post Office scandal— there is a vested interest from these large technology manufacturers in saying that their systems are perfect. Their brand is based upon the fact that they’re meant to get some of this stuff right, or most of it right.

Speaker D: Yeah, so the first problem that I’ve noticed is the, the argument doesn’t really make sense from a logical perspective. So the common argument made by these organizations is that, well, our system is secure because no one’s found any problems. But then any evidence of problems is dismissed on the basis that the system is reliable. And that sort of circular argument, made with longer words but effectively coming down to that, is the fundamental basis for a lot of computer evidence, and that doesn’t logically make sense. And the second problem is that ordinary people don’t have the access to the records that the companies do use to check whether the system is working, and so they, they can’t latch on to those sorts of weaknesses.

Speaker B: And the companies wouldn’t really want you to have access to that code. They would say that that’s proprietary. They would say that, you know, that they have an interest in that.

Speaker D: Yeah, there’s lots of arguments. So from the banking perspective, the two main arguments come up. The one that is made publicly is that if information was disclosed about the way these systems work, that would harm the security of the bank. Now, I’m not convinced of that. For one, I know how the systems work, but secondly, if that was the case, that would show that the computers are fundamentally flawed because so many people know about these computer systems. And if it became if that knowledge is so dangerous that it can’t be shown in court, then the whole system is flawed from the first perspective. The second argument made is that if it became known that the computer system had problems, then suddenly loads of customers would become criminals overnight and would withdraw money from their account.

Speaker B: Professor Stephen Murdoch of University College London stating quite clearly that the machines are often wrong and should not be relied upon. An unthinkable assertion in the age of the smartphone, machine learning— the name we should use instead of AI— and the internet. Proof that computers are a tool and that we should be very careful about how we use them. Something that the Prime Minister and the other world leaders who are scheduled to meet at Bletchley Park, the much-trumpeted home of British computing where Alan Turing developed Colossus, would do well to mull on. Because intent is one of the crucial factors to weigh up about computer evidence. A point made by James Christie, a former IBM IT auditor and computer expert who has carried out an investigation into the development of the now highly controversial Section 69 of the Police and Criminal Evidence Act, which deals with computer evidence. Christie’s conclusions were damning. The Law Commission, he says, drew up the legislation to make digital evidence proof, partly, he says, because it didn’t want the courts jammed with cases where defendants were challenging whether it was true or not.

Speaker E: But it was so easy for me to uncover what the Law Commission did. It wasn’t that I had to go into any deep detective work, all I had to do was to follow the trail through Google searches, buy a couple of academic papers, buy access to a couple of academic papers, there were, there was no Freedom of Information request, and I just had to make contact with Stephen Castle and Alastair Kelman, and they confirmed what I suspected, that they had been misrepresented. I’ve not been able to get in touch with Professor Tapper, but somebody who knows him very well did review the paper and was happy enough with my arguments that, you know, if you go back to look at the articles that he wrote, he was arguing for the very opposite of what the Law Commission was arguing. So the politicians that took this decision were sold a bum steer by the Law Commission.

Speaker B: What appears to— well, what you appear to have uncovered would seem to be just ignorance, really, isn’t it?

Speaker E: I mean, one of the problems that we’ve been raised from, it’s ignorance combined with arrogance. One of the When I worked, I was a very technical systems developer. I could really work it.

Speaker B: IT expert James Christie on the Law Commission’s misquoting of experts to justify its case for the use of digital evidence in court. But Christie goes even further and points out a charge that is now being leveled more and more at both legal and political bodies. That they do not know enough about the technology that runs our lives.

Speaker E: If they’d consulted and listened properly— they didn’t consult widely enough. They should have gone out to their professional bodies, the British Computer Society, the various engineering bodies, and then they should have brought in some sort of consultant or guide who could have interpreted what the experts were saying and helped them to help the Law Commission to understand the legal implications. They shouldn’t simply have thought that a first in law at Oxford and then a stellar legal career is going to qualify them to talk about engineering or IT topics. Exactly. They then, they knew what they wanted to achieve clearly, which was the repeal of Section 69. Of PACE. Did I say PACE 74 earlier? It was 84, but they wanted the repeal of that, and then they went looking for evidence. And I mean, that was cynical and irresponsible, but where they were just utterly incompetent was they went to sources that told them the opposite, and they just misrepresented them, which left a fairly easy trail for me to follow. To expose what they’d done.

Speaker B: Do you think that there was an objective in all of this, insofar as it could have been that they were saying, hang on, our world is now dependent on computers, so therefore, because we are dependent on computers, then we’ve got to have some means to actually make evidence on them admissible?

Speaker E: Yes, clearly they were concerned about the possibility that courts would get clogged up, where it’d be just too difficult to use computer evidence. It would be too easy to challenge, and any case that hung on computer evidence would just end up in an interminable battle over it. It was clearly the repeal of Section 69 was It was a matter of convenience for the courts, and justice was collateral damage.

Speaker B: James Christie pointing out the terrifying weakness at the heart of our legal system: evidence is allowed to be admitted that the defendant is unable to challenge. It’s a question that has preoccupied Stephen Mason, the man who commissioned Christie’s investigation. Mason is a barrister with an interest in electronic signatures, authentication security, electronic evidence, email and internet use, and the interception and monitoring of communications. He’s also editor of the Digital Evidence and Electronic Signature Law Review and a veteran campaigner against the assumption computer evidence is unchallengeable.

Speaker F: I started doing some little bit of research into electronic evidence and also what we call in England and Wales, or in the law generally, the presumption that computers were reliable. I also looked in my research for any computer-related case law, which actually was quite sparse at that time, or relatively sparse. Perhaps I didn’t look in the right places either. But then I had a case in in 2009, which I represented a Mr. Jobe pro bono, at no cost, through the Bar Pro Bono Unit, because he had a claim of £2,100 which he claimed was taken from an ATM. And in preparing his case, albeit at a late stage, I knew the bank would claim that the ATMs could be relied on to be reliable, and they they could use this presumption. So I took along what little I had written in our textbook about this, because I knew the judge was going to raise this issue, which he did. He did accept my arguments, although he did not find that my client actually proved his case in the circumstances, probably rightly so. But that caused me then to be very careful about looking at this presumption. And I spent probably 18 months of my life researching whether or not computers are reliable, not from the legal perspective, but from the technical perspective. So my entire aim for the second edition of our practitioner text in 2010 was to have a chapter dealing and proving, demonstrating without doubt that this presumption was ridiculous, which I began to do and albeit when I look back at the 2010 edition, it’s a little bit ropey, but nevertheless it improved through to the 5th edition, which is the present edition. James, I, I, um, discovered one of his blogs on this Post Office Horizon case 2 or 3 years ago, and, um, I was so taken and impressed with what he read, I wrote to him and asked if this two-part article he put on his blog, he might be willing to develop for an article for a journal I founded in 2003, the Digital Evidence and Electronic Signature Law Review, which he did, which is very kind of him. And so James actually independently started looking at the background between an initial report produced by the Law Commission and the final report, which I had not done. Obviously, in the days of technology and freedom of information, a lot more information was able to be gleaned as a result. Hence, uh, this article by James where he demonstrates to me quite clearly that the Law Commission knew probably what it was doing and ignored all the signs that the technology was not reliable and recommended this presumption be put in place. And what is more sickening is when you read the appendix to James’s article in which he looks at the debate in the House of Commons passing the relevant statute. It really is quite appalling, and I’ve been canvassing on this ever since 2010.

Speaker B: The barrister Stephen Mason pointing out that in the 21st century, laws must be drawn up by both experts on technology and lawyers if they are to be effective. A long overdue expansion of a debate over technology that now has to go much, much wider. A point tellingly underlined by the former editor of the Daily Telegraph, Charles Moore, biographer of Margaret Thatcher and now Lord Moore. While editing the Telegraph, Moore was told that it was wrong that the Houses of Parliament only contained around 30 MPs with science and technology backgrounds. He angrily responded that “the day a gentleman could not legislate on any issue was the day democracy was dead.” The number of MPs with law degrees still massively outweighs the number of MPs with science-based degrees. It’s a point taken up by the ethicist and author Eve Poole, whose latest book, Robot Souls, examines the new world of the digital As Paul points out, the presumption that computers always present us with perfect data output not only gives us issues related to fake news, but it has now achieved a cultural value. We think the computer is always right.

Speaker A: Well, I think clearly we do. I mean, I know that there’s quite a lot of different examples of different sorts of case and different sorts of digital evidence, but certainly in England Given where we are with precedent at the moment, it does look like some slightly urgent action needs to be taken to make sure there can never be a presumption that the computer is always right. Because the more we learn about computers, the more we know that that is categorically no longer the case, even if with some early, very mechanical computers, that was possibly easier to argue.

Speaker B: It’s very interesting, isn’t it? Because this isn’t just about a sort of a legal position. We’re also attacking a cultural presumption because the world at large assumes that computers must be right. They think that they’re based on calculators, those funny things that Sir Clive Sinclair used to put in the windows in Boots. And that’s an issue too, isn’t it?

Speaker A: Yeah, I think so. It’s partly, I think, because it all seems like magic and because we don’t really understand these things in boxes and they do such marvelous things, we imagine there must be some kind of wizard who’s making it all happen. And, and that gets wrapped up in our heads with all kinds of ideas about truth. And so we assume these things must be infallible. And even though, particularly these days, you know, with gaming and everything else that we do with computers, we know that they’re not just a calculator where you, you do your sums and it’s never wrong because they are so much more interactive than they ever were. But I think with the Post Office scandal, it was also wound up with a whole load of institutional issues that we have where the really big institutions are always seen to be right. And, you know, even with whistleblowing policies, even with everything that we know about institutions and their propensity to go bad, and there is always a presumption that the institution must be right. And if the institution’s technology, it has had that amount of money invested in it, then it must be right as well. And that’s not just the Post Office. We’ve seen that in government, we’ve seen it in you know, a whole load of other areas where the first thing that happens when there is a problem is that the institution asserts its rightness, and then a whole load of actions are taken which are to try and shore up the institution rather than actually stop and pause and interrogate what may or may not be going on. Well, I think that’s— it’s understandable in a way, um, because we’re in very new territory, and I think when you layer into it everything we’re learning about generative AI, there’s a huge panic, which is if you have to stop and pause every time you’re using any kind of technology to check out whether you can trust it or not, then our lives would grind to a halt. And that’s why so much of the emphasis in AI regulation is about transparency and explainability and audit, because we have got to a stage where we really just don’t understand what this technology is doing. So How could we ever actually, hand on heart, say it’s definitely right? Because we don’t know that. And you’re right, I think in the public imagination, it’s still a sort of calculator that can do a couple of whizzy things rather than something which is incredibly different and incredibly difficult to understand. And you would think in Scotland in particular with corroboration, you would never be relying on a sole piece of evidence with the computer that says yes. But, you know, there were miscarriages up here too. In the case of the Post Office. So it’s not even protected in different jurisdictions in terms of that presumption, I think.

Speaker B: But going to your point of, yeah, this is all new, you cannot have something that is built on sand. If we actually cannot get the correct picture about the value of computer evidence, and we are going into that age of AI, then we could be heading for a tremendous problem. The whole edifice could come falling down.

Speaker A: Well, that’s true to a degree, but I mean, I would think in a court you very rarely sit down in front of a screen and say, are you correct? And the computer says yes. You normally have experts. You normally have expert witnesses who come in and interrogate reports or explain about programming. So you’re very, very reliant on those experts.

Speaker C: Experts.

Speaker A: I think there were a whole range of experts involved in these cases, and I think one of the issues is that there is a question about truthfulness and disclosure, which is muddying the waters. So I would have thought in a situation where people are being truthful and you can corroborate evidence by having experts come in and explain what it is the computer is being programmed to do and what is coming out of it, we should be safer. But I think that you should never these days have any presumption in law that the computer would always be right, because we just know that that is no longer a reliable thing to assume.

Speaker B: The author and ethicist Eve Poole, who we can assume knows a bit about ethics because she studied them. Professor Andrew Jones knows a lot about computer forensics, the author of a number of books on the subject and recognized as a leading authority Jones has taught on the subject at a number of universities around the world.

Speaker G: Nothing’s infallible. The real reality is that— and for courts, it’s got to be beyond reasonable doubt, not infallible. You can only go with best evidence. Technology has not reached the point where it’s infallible.

Speaker B: Right, so he Computer is not a perfect calculating device then. Why is that?

Speaker G: Because the programs are written by people.

Speaker B: Right, so it’s the programs that are the problem.

Speaker G: It is close to impossible to scientifically prove that a piece of software does what it says it does and only what it says it does.

Speaker B: Isn’t that a problem for the courts? Then? Surely, how do you go get around presenting computer evidence?

Speaker G: Because it’s to a level that’s acceptable to the courts.

Speaker B: Can you give me an example of that then?

Speaker G: Forensic means acceptable in a court of law.

Speaker B: Well, can you give me an example of what, of how you present evidence in a manner that is acceptable?

Speaker G: Yes, you present the best evidence that’s available and the court has to decide whether or not it’s good enough. And just to pick on a very trivial point, you can prove fairly reasonably that a computer was used to do something. The issue you then have is whose fingers were on the keyboard when that computer was used, and the computer in reality can’t tell you that. Because someone might have gained access to the password or the like. So you then rely, or you look to corroborating evidence. Was there an access control device on the room that the computer was in? Was he the only person in the room at the time? And things like this, you know, is the CCTV that shows the person was there?

Speaker B: So for instance, you could say, okay, I want the GPS data from your telephone. That’ll tell me where you are at a particular point, or that, or that could be some evidence that provides a little more evidence.

Speaker G: Yeah, absolutely. You know, so corroborating evidence. The GPS might show you were in that building at that time, but you know, to actually You can show patterns of activity over a period of time that you can relate to that person, but definitively saying it was his fingers on that keyboard at that time is very difficult. They’re big complex systems. They have millions of lines of code. Checking that all of those lines of code and how they react to each other and interlink to each other As I say, when you get beyond about 100 lines of code, it becomes almost impossible and certainly financially unrealistic to test it to that level.

Speaker B: In a court case, what do you do? Do you say, okay, here is my evidence, this is the evidence, this is the computer evidence that we think makes you guilty? And do you then also present it to the defense so that the defense can say, no, this doesn’t prove that we’re guilty.

Speaker F: It’s—

Speaker G: if I present evidence to a court, I present facts. I don’t present facts that presume guilt or innocence. I present the facts that I’ve been able to discover, and it’s for the court to decide whether it’s guilt or innocence.

Speaker B: But essentially, the point that I’m driving at is that both sides get to see the the facts that you’re presenting?

Speaker G: Yes, absolutely. And the norm is that both sides will present their— before the court case, both sides will present the facts that they found, and they will agree on the things that they have in common. And then the thing that’s debated in court is the things that they have different interpretations on.

Speaker B: Andrew Jones, professor of cybersecurity at the University of Suffolk, on how evidence should be presented in court. As we found from the Horizon case, being told that computer evidence is unchallengeable has led to massive miscarriages of justice. According to the Law Commission in its justification for Section 69, problems with computer evidence would become clear and obvious. It also said that on balance it would operate break fairly. In the Horizon case, the defendants were told that they could challenge the evidence if they could show where the computer coding was wrong in how the data was generated. The Post Office and Fujitsu would not, though, provide access to the system on commercial grounds. According to many of those we have interviewed, the Horizon case is not a one-off. Many other cases have slipped through. In the case of nurses at the Princess of Wales Hospital, Bridgend, nurses were prosecuted because medical devices were providing the wrong data. While the lawyers we interviewed stated that they had represented people who had been accused of massive cash withdrawals from cash machines and of trying to encrypt systems in attacks known as logic bombs. It’s a situation that Lord Timothy Clement-Jones Chair of the All-Party AI Select Committee in the Houses of Parliament, a frequent guest on this program, says points inexorably to reform of Section 68 of the Police and Criminal Evidence Act.

Speaker H: Well, it was a pretty radical move to get rid of Section 69 of the Police and Criminal Evidence Act, and obviously the Law Commission’s report had a huge influence on that. And, you know, when I look back at the paper that’s just come out, it’s the James Christie paper, I think, that tackles it. I mean, it’s a— it’s pretty groundbreaking in terms of how it describes the evidence that the Law Commission took. Now, that was a very long time ago. I don’t think the current Law Commission, which is far more tech savvy, would come to the same conclusions, quite frankly, at all. Especially in the light of the Horizon cases. And, you know, it seems to me that this whole question of computer reliability— and let’s remind ourselves that Section 69 of the Police and Criminal Evidence Act was all about excluding computer evidence, and therefore if you repeal it, it’s all about including computer evidence. But this was done unconditionally without any requirement for reliability or any tests of reliability. So I’m fairly astounded when one looks back, you know, you can’t help feeling that the original Law Commission either didn’t understand at best, or at worst were kind of driving to a conclusion that they’d already decided was the one they wanted to reach. So I don’t think that’s a glorious moment for the Law Commission. Quite honestly. And as I say, I think the really chilling aspect is that of course that led to many of the Horizon cases going the wrong way, defendants being completely unable to contest the evidence. I also think though that it means that ministers, the politicians, had the wool pulled over their eyes because the evidence wasn’t challengeable. So It was— this infected the whole situation, quite frankly.

Speaker B: And in fact, one of the things that Christie said in the interview that he gave to us, he said that there was arrogance and that the Law Commission essentially wanted to come to the conclusion that it came to in terms of evidence. He said that the Law Commission didn’t want to see the courts jammed with companies and individuals challenging the computer evidence.

Speaker H: Yes, well, as a fan of the current Law Commission, that really pains me, you know, because I think that is a falling down of professional standards, if you like, in terms of gathering of evidence and making recommendations, which we know that the current Law Commission, you know, whether it’s on hate crimes or self-driving vehicles and so on. You know, there’s been some really important gathering of evidence which is highly technological in its nature. And actually, it’s done a very good job of late. So whether this was teething troubles or whether it was, in a sense, deliberate is very difficult to judge. But James obviously comes to the conclusions that he does.

Speaker B: And this also does drive the inexorable conclusion to that there must be other cases. And in fact, Alistair Kelman has a few cases on his website. One of them relates to somebody who was accused of putting the equivalent of a logic bomb in computer code. And Kelman said that the most likely person to have done this was the chief witness for the prosecution, principally because the man who was accused had been having an affair with the wife of the chief witness of the prosecution.

Speaker H: Well, there you go. I mean, that is pretty astounding. But I mean, Alastair’s been a great champion for people who’ve been affected by computer evidence. So I take a lot of what he says as gospel, quite honestly. So, you know, that doesn’t surprise me. And It’s uncovering things like that which are so important. And I hope going forward that we get this right, because when you think about it, it’s not just now computer software, it’s something much more sophisticated in the form of AI. And, you know, we might describe it as software, but good heavens, it’s far more powerful and also able to make it up as it goes along, as we know. From some of these large language models. So we’ve got to be ultra wary of that kind of evidence too.

Speaker B: And following that through the black box in the AI, the evidentiary trail is going to be very, very, very difficult to follow.

Speaker H: Absolutely. I think you’re going to need rules of evidence about black box AI very, very soon. I mean, you know, my view, it’s black box and comes to a conclusion and you can’t tell what data it’s used and how it’s used the data and the weighting of the decision-making, I think, you know, you’re very close to saying it’s inadmissible.

Speaker B: I suppose possibly all that one can really say is that we are with computers on a learning curve. This is at the cutting edge of technology and therefore It has to be on the cutting edge of the law, and the law hasn’t been historically that good about being on the cutting edge of things.

Speaker H: No, I quite agree. But let’s face it, the Law Commission report was over 20 years ago when it would have been perfectly possible to come to a different and more robust conclusion about reliability. I mean, after all, 20 years ago, computer software Crashes and so on in our computers, whether they were Apple or MS-based, quite frankly, were legion. So in a way, we should have spotted the reliability issue much more 20 years ago than now. The reliability we’re talking about now relating to AI is much less based on, if you like, the technological system in terms of technical reliability. It’s much more to do with misinformation and the, if you like, the altering of the data within the system. So, you know, we’re faced with a different kind of problem of reliability, but nevertheless, we need to make sure we’ve got the rules in place.

Speaker B: Lord Clement-Jones of the AI Select Committee, as he points out, any future legislation will have to involve consultation with experts, particularly those from the British Computer Society, which is currently awaiting an invitation to the Prime Minister’s Bletchley Park summit on AI development. Sam Da Silva of the law firm CMS is a legal advisor to the British Computer Society and echoes their concerns about the current use of computer evidence by the courts.

Speaker I: Yeah, I think that’s fair to say, yes. I think having that assumption or presumption or reliability is, does cause inadvertent consequences. And, you know, obviously the Post Office scandal was one of those unfortunate outcomes from that.

Speaker B: Do you think that there is a need then for Section 69 to be rewritten, to be overhauled, for people to start looking at it very closely in the light of the fact that we’re now in the 21st century and technology is absolutely center stage in our lives now?

Speaker I: Simple answer is yes, definitely.

Speaker B: Right. One of our other interviewees was Lord Clement-Jones, who is the— he’s the head of the All-Party AI Select Committee. He was very much of the view that if we are to move forward and Rishi Sunak’s talking about AI regulation, then this is one of the things that we’ve got to sort out right at the beginning, isn’t it?

Speaker I: I think so. Although AI is far wider in terms of regulating AI, I think this doesn’t necessarily need to be focused on AI. It’s just there could be a system which is not AI but is generating information and data. So I think wider than AI, I think.

Speaker B: Oh no, I wasn’t for a moment suggesting that. What I was saying was that You can’t really go on to a next stage unless you sort out these elementary and tertiary stages, can you?

Speaker I: Correct. Yeah, AI is much more complex, but you’re right in the sense that IT systems have been around for decades. So yeah, and we need to get that right first.

Speaker B: The conference I was at in San Francisco, people were talking about trust. They were saying that it’s essential to have trust in this new world that we’re going into. You can’t have trust if you can’t trust the digital evidence. You can’t have trust if there’s something wrong with the system, as the Horizon case seems to have pointed out in spades.

Speaker I: Yeah, completely agree with that. I mean, yeah, I guess the AI sort of proposed regulation at EU level is looking at transparency and all that kind of stuff. So it’s a similar concept here in terms of how is that decision made instead of relying on a black box.

Speaker B: Exactly. So how can you have transparency if one side doesn’t understand what is going on?

Speaker I: Very difficult.

Speaker B: One of the points that James Christie makes, he says that there was, and this has been made by many, many of the people that we’ve interviewed, doesn’t necessarily mean that you have to agree, but they’ve been saying that one of the things that this exposes is a lack of technical and technological competence amongst the legal fraternity, and that that has to be addressed too, that there are not enough people in the legislature and in the legal bodies who understand technology. And really, they feel that has to change. I mean, the current head of the Law Commission, apparently people are quite happy with. He was the guy who was the presiding judge in the Horizon case. So he’s got quite a good track record.

Speaker I: Yeah, I think that’s fair to say. I mean, yeah, technology is a continually evolving area, and, you know, people need to get up to speed in terms of what it can do and what it can’t do and just understand the nature of the beast, so to speak.

Speaker B: And given the speed at which it is evolving, unfortunately, the law fraternity is going to have to pick up the pace quite a lot.

Speaker I: I think so, yes.

Speaker B: Sam Da Silva of the law firm CMS and the British Computer Society’s legal advisor on the challenges facing the law in regulating technology. Obviously, the first challenge, even before we start to think about AI, will be to get the basics right. So what needs to be done? In the course of this research, Many people have highlighted the 7 statements made by the barrister Alastair Kelman and Richard Sizer and quoted out of context by the Law Commission when it drew up its original legislation. So we asked Kelman himself to set out what he thinks needs to be done.

Speaker J: Statement 1 was to deal with the qualifications and experience of the person in charge of the computer system. So in those days it used to be the data processing manager. This is to establish that he’s capable of swearing such a document to say that the evidence is reliable or not. Statement 2 would consist of a description of the computer system with reference to each of the components in the system by brand and model number. In the imaginary case I had, I called my computer a Kamikaze DDB-7 with an ASMA 2.62 operating system. But of course today a modern version of that would be something like a running Microsoft Windows 11 Professional version 22H2 for 64, for Intel-based systems, that’s 64-based computers, and the reference number of the latest update running on the system. Now that would enable you to know that you were running the right software and that had been developed and managed professionally. And then if you had custom-written programs were running on that, those would also be properly documented and citing their provenance as well. Statement 3 was intended to be a long statement to deal with the quality of the individual components by reference to the development time involved, so that, for example, reference could be made to any technical literature or manuals which were used. So if you were using Intel-based systems, then of course there’d be documentation from Intel telling you how good and reliable their hardware was in dealing with matters of this sort. So you were actually putting some provenance around that and so on and so forth. And manufacturers of quality products would gladly assist in producing technical evidence of this sort because that’s what their whole business is about. Statement 4 would deal with the testing and documentation standards applied to any custom-written software on the device itself. If the software had been bought in, the software house, if reputable, should be willing to provide information as to the testing documentation standards. But basically, as you will know, if a computer contains 20 decision points, that’s over a million logical paths in what it’s doing. And consequently, in practice, you need to test each of those 1 million logical paths. Now, a lot of that testing does not happen, and people just go by the nod. So this, in a quality operation, would need to address those issues. Statement 5 should deal with the procedures for logging updates in the software and the qualifications of the subordinate staff involved in the computer system. So, you know, this is things like the updates to the running the latest version of the software, latest updates that you get over the internet. That would be what the modern equivalent of things to ensure that you’re using suitably patched and modern software. And statement 6 would deal with the physical and electronic security requirements for the installation. So today that would cover features such as the network attached storage, the NAS, which is a centralized file server which allows multiple users to store and share files over a TCP/IP network or over the internet, things such as that. That topic would be there. And finally, the 7th statement would indicate how the particular computer printout came into existence. And what it purports to show. In that section, the person in charge would be able to say that, for example, no faults manifested themselves during the material time, and then he could be cross-examined on that with the material coming from the other material that was behind it. The idea here was to even up the balance. So it meant that lawyers who weren’t skilled in computer matters would be given enough information to enable them to properly question the evidence in accordance with the trial, and judges and juries could therefore take a view as to how reliable that evidence was. And that was, we believe, to be a fundamental part of the, of the whole justice process. It was a work, but it was necessary that if you were wanting to put in evidence, then you had to ensure that it was reliable.

Speaker B: So it sounds very much as though, to create an analogy, it’s a bit like somebody saying, okay, This is the building. We had it surveyed. There was general agreement that everything was okay. All of the changes that were made to it were made by qualified builders and builders that came from reputable companies. And we know that they work to a particular standard. You seem to be creating something a little similar to that, don’t you?

Speaker J: Indeed. Well, what you— what the idea was here was this was not going to be a terribly bureaucratic activity. This effectively was a kind of audit requirement, somewhat similar to the Sarbanes-Oxley Act in America, where it protects investors and helps them rebuild trust in the financial markets.

Speaker B: The barrister Alastair Kelman, who along with his co-author Richard Syzer developed the Seven Statement Test on digital evidence. In the interests of fairness and balance, We did approach the Law Commission to ask it if it would like to be interviewed for this program. A spokeswoman declined the offer of an interview but did provide the following statement: The Law Commission conducts projects into areas of the law and produces reports which the government is free to accept or reject. The Commission can be asked to look into any area of law again in a new project, but can’t undertake a project unless the Commission and the government agree that the Commission should do it. As part of the public consultation on the 14th program of law reform in 2021, the Commission described a potential project, Justice in the Digital Age, which could include a reconsideration of the presumption of reliability. The Commission has extended the timetable for finalising the 14th programme in view of the government’s focus on priorities for the remainder of this Parliament. It remains the Commission’s intention to agree a new programme with government in due course. We will wait and see, because if the Prime Minister wishes to make the case for the UK being the centre for AI regulation, it would appear that some of the fundamentals of 21st century law need to be established before that happens. We did approach the Labour Party for a comment and received no response whatsoever. You’ve been listening to Password on Resonance FM, written and presented by me, Peter Warren, and produced by Blue Buffery. Thanks for listening. And goodbye.

Speaker A: This program has been brought to you by Resonance FM. If you like what you heard, please support our work by making a donation at resonancefm.com/donate.