1 00:00:01,750 --> 00:00:05,645 Richard, thank you so much for being a part of the podcast. 2 00:00:06,604 --> 00:00:08,399 Yeah, thank you very much, Kevin. 3 00:00:08,399 --> 00:00:09,887 It's a pleasure to be here. 4 00:00:10,658 --> 00:00:17,204 Tell me how you first got into the area of, well, AI more generally initially, and 5 00:00:17,204 --> 00:00:18,585 then responsible AI. 6 00:00:19,416 --> 00:00:24,740 Well, AI more generally, I did my studies and my PhD already in artificial 7 00:00:24,740 --> 00:00:27,082 intelligence in the eighties, in the nineties. 8 00:00:27,082 --> 00:00:31,405 So that's a long time ago when nobody knew what artificial intelligence was. 9 00:00:32,066 --> 00:00:37,130 And then of course, since 2016, when this came back into society and later in 10 00:00:37,130 --> 00:00:39,752 business, that's what I focused on. 11 00:00:39,752 --> 00:00:47,538 So long history in artificial intelligence and in the responsible use of AI, I came 12 00:00:47,538 --> 00:00:48,819 across when in 2000... 13 00:00:49,364 --> 00:00:53,467 17, I was working at a large insurance company. 14 00:00:53,647 --> 00:00:58,751 And in my team, there was a small group who looked at bias and explainability. 15 00:00:58,751 --> 00:01:02,534 And I thought, before that, I just had a business look at it. 16 00:01:02,534 --> 00:01:02,754 Yeah. 17 00:01:02,754 --> 00:01:04,276 So how can you use this for businesses? 18 00:01:04,276 --> 00:01:08,178 But then I said, wow, this is actually important. 19 00:01:08,279 --> 00:01:13,863 And if you if you take this into account, and if this is going be applied 20 00:01:13,863 --> 00:01:14,743 everywhere. 21 00:01:14,852 --> 00:01:20,256 So when I moved back to Telefonica, I took that on board and I brought it into the 22 00:01:20,256 --> 00:01:25,279 company and said, well, if we want to be serious on AI, we should not only look at 23 00:01:25,319 --> 00:01:28,722 business opportunities, but we should also look at responsibility. 24 00:01:28,722 --> 00:01:31,323 And that's how I came in the field. 25 00:01:32,214 --> 00:01:33,676 What was the response in the company? 26 00:01:33,676 --> 00:01:37,662 Because obviously there's some effort that's required and there may be 27 00:01:37,662 --> 00:01:41,348 trade-offs in terms of designing responsible systems, dealing with issues 28 00:01:41,348 --> 00:01:43,230 like fairness and bias. 29 00:01:43,291 --> 00:01:45,333 So what was the response you got? 30 00:01:45,868 --> 00:01:49,150 Well, the response was surprisingly positive. 31 00:01:50,171 --> 00:01:50,291 Yeah. 32 00:01:50,291 --> 00:01:54,954 So for instance, when in 2000, we started this, this effort in 2018 in the 33 00:01:54,954 --> 00:01:55,854 beginning. 34 00:01:56,035 --> 00:02:00,338 And basically because the company was very vocal on big data and artificial 35 00:02:00,338 --> 00:02:02,059 intelligence and the opportunities. 36 00:02:02,059 --> 00:02:07,163 Uh, and at that time there were a lot of those scandals about discrimination, the 37 00:02:07,163 --> 00:02:13,367 compass things, uh, this company's hiring people and only hiring a male gender. 38 00:02:13,547 --> 00:02:14,524 So, um, 39 00:02:14,524 --> 00:02:17,546 I thought, well, it's important to make a statement as well. 40 00:02:17,546 --> 00:02:22,050 So we brought together a very, uh, a set of departments, yeah. 41 00:02:22,050 --> 00:02:25,513 Which you have to put together because this what about, it was an aspirational 42 00:02:25,513 --> 00:02:27,095 statement that we wanted to make. 43 00:02:27,095 --> 00:02:31,238 So we wanted to publish our ethical AI principles. 44 00:02:31,359 --> 00:02:37,364 Uh, and it took us a month of four months or five months to agree on the set of 45 00:02:37,364 --> 00:02:39,746 principles and then how do you formulate them? 46 00:02:39,746 --> 00:02:41,188 Because of course, many 47 00:02:41,188 --> 00:02:45,051 commercial people think, well, if we do this, then we will lose speed, we will 48 00:02:45,051 --> 00:02:50,836 lose revenues, but it was surprisingly fast. 49 00:02:51,337 --> 00:02:55,241 So I think we published them in October 2018. 50 00:02:56,362 --> 00:03:03,790 And a very important lesson, because I've also spoken to many other companies, and 51 00:03:03,790 --> 00:03:06,124 there are two approaches to... 52 00:03:06,124 --> 00:03:09,645 make these statements, publish your ethical principles. 53 00:03:09,645 --> 00:03:11,806 The one is, is a kind of aspirational one. 54 00:03:11,806 --> 00:03:13,987 So this is what we want to do. 55 00:03:13,987 --> 00:03:17,488 And we acknowledge that we're not yet there, but we're gonna work hard to be 56 00:03:17,488 --> 00:03:18,269 there. 57 00:03:19,209 --> 00:03:23,131 And then there are companies that before publishing anything, they want to be 58 00:03:23,131 --> 00:03:28,333 absolutely sure that they are already complying with what they're saying. 59 00:03:28,333 --> 00:03:32,555 Those companies, they are still discussing their ethical principles, because 60 00:03:32,555 --> 00:03:34,140 obviously it's not so... 61 00:03:34,140 --> 00:03:38,346 easy to make sure that you comply with all of them in all aspects. 62 00:03:38,346 --> 00:03:43,292 So we are on a journey as a sector, private sector, public sector, as a 63 00:03:43,292 --> 00:03:47,558 society, and this is going to be incorporated increasingly more, but 64 00:03:47,558 --> 00:03:48,599 gradually. 65 00:03:49,746 --> 00:03:50,406 Absolutely. 66 00:03:50,406 --> 00:03:52,209 Well, this is an important point, though. 67 00:03:52,209 --> 00:03:57,035 How does an organization ensure if they actually get principles out the door, that 68 00:03:57,035 --> 00:04:01,440 they are more than just aspirational, that they can actually influence the 69 00:04:01,440 --> 00:04:02,741 development that goes on? 70 00:04:03,328 --> 00:04:06,308 Yeah, so now you are alluding to ethics washing. 71 00:04:08,469 --> 00:04:14,811 And so any company, any organization that starts on this journey will first start 72 00:04:14,811 --> 00:04:19,752 with principles and there is really nothing behind it. 73 00:04:19,752 --> 00:04:26,134 So you can, you can accuse them as being ethics washing. 74 00:04:26,274 --> 00:04:28,234 But in the end, this is the way how you start. 75 00:04:28,234 --> 00:04:30,695 And then the next year they have to, you have to do something more. 76 00:04:30,695 --> 00:04:32,235 And the next year you have to do something more. 77 00:04:32,235 --> 00:04:32,815 So. 78 00:04:32,844 --> 00:04:33,664 It's a journey. 79 00:04:33,664 --> 00:04:38,687 So we started in, let's say 2018, we published into end of 2018. 80 00:04:39,487 --> 00:04:45,110 And we only approved a formal internal regulation last December. 81 00:04:45,371 --> 00:04:51,114 So it took us like, uh, like, like five years to do that. 82 00:04:51,414 --> 00:04:56,057 And that's not because, uh, we had to convince everybody because there is no 83 00:04:56,057 --> 00:04:59,119 experience whatsoever in the world, how to do that. 84 00:04:59,119 --> 00:05:00,699 So you have to find it out. 85 00:05:01,028 --> 00:05:05,309 yourself and then talk to some other companies who do the same, but there are 86 00:05:05,309 --> 00:05:07,649 very few, especially in the beginning. 87 00:05:07,869 --> 00:05:14,091 So what we did is actually the first step was we did a training course where we 88 00:05:14,091 --> 00:05:17,372 explain our employees and most people most affected. 89 00:05:17,372 --> 00:05:21,013 So we're not speaking about only developing artificial intelligence, but 90 00:05:21,013 --> 00:05:26,335 also about procuring it on the market and also about using it internally and also 91 00:05:26,335 --> 00:05:27,555 about selling it. 92 00:05:27,555 --> 00:05:28,955 So all those people. 93 00:05:28,960 --> 00:05:32,642 And that's a big, that's a, that are thousands of people in our company. 94 00:05:33,242 --> 00:05:35,104 And they need to understand something about this. 95 00:05:35,104 --> 00:05:35,844 Yeah. 96 00:05:35,844 --> 00:05:37,505 So that's what we started with. 97 00:05:37,765 --> 00:05:41,968 Then next step we started about a kind of thinking about the governance. 98 00:05:41,968 --> 00:05:45,570 So these are the principles, but so who is responsible for it? 99 00:05:45,570 --> 00:05:47,412 What are the functions? 100 00:05:47,412 --> 00:05:52,394 Do we need a new role in the organization to be able to deal with this? 101 00:05:53,848 --> 00:05:58,011 We, of course, like everybody, we started to make a questionnaire where we do this 102 00:05:58,011 --> 00:06:00,072 ethics impact assessment. 103 00:06:01,334 --> 00:06:06,238 And I think we went through like 15 different versions based on experiences 104 00:06:06,238 --> 00:06:07,338 with people. 105 00:06:07,519 --> 00:06:14,124 Then we, we brought it out in the wild and we tested it with real, uh, real use cases 106 00:06:14,124 --> 00:06:16,366 and people who are not so involved. 107 00:06:17,027 --> 00:06:20,149 And then finally we decided to run some pilots with a governance model. 108 00:06:20,149 --> 00:06:23,332 So do we need an AI ethics committee or do we need. 109 00:06:23,332 --> 00:06:28,014 a special role, like we call it the responsible AI champion. 110 00:06:28,515 --> 00:06:32,198 And how do you organize this completely distributed, decentralized? 111 00:06:32,198 --> 00:06:34,219 And what are the things? 112 00:06:34,219 --> 00:06:36,561 How, what is the workflow? 113 00:06:36,561 --> 00:06:38,382 How is compliance involved? 114 00:06:38,382 --> 00:06:39,462 How is ESG involved? 115 00:06:39,462 --> 00:06:43,265 So we did really a pilot, uh, in four different business units. 116 00:06:43,265 --> 00:06:47,348 And based on that experience, we, we thought we were ready to write really 117 00:06:47,348 --> 00:06:51,911 something up that can be then, uh, complied with, and that is what is 118 00:06:51,911 --> 00:06:53,091 currently the case. 119 00:06:53,228 --> 00:06:56,129 So it's a journey that took us many years. 120 00:06:56,129 --> 00:06:59,011 Of course, today, if you start a journey, there is much more experience. 121 00:06:59,011 --> 00:07:01,513 So it will take you maybe three years. 122 00:07:01,873 --> 00:07:05,956 And anyway, if you're in Europe, you have to do something anyway, because of the 123 00:07:05,956 --> 00:07:12,099 upcoming AI Act, which will be, which will enter in force in, in two years from now. 124 00:07:14,122 --> 00:07:18,909 So in that journey, how do you evaluate what's success? 125 00:07:18,909 --> 00:07:22,174 You talked about a number of choices, a number of options. 126 00:07:22,174 --> 00:07:25,297 How does the organization decide what the right things are to implement? 127 00:07:26,634 --> 00:07:32,677 Well, I think success is that you move forward in a possible way. 128 00:07:32,677 --> 00:07:38,781 And most of my experience is sometimes the process is more important than the result. 129 00:07:38,781 --> 00:07:43,423 So of course, the result is that all the systems that you launch and actually, so 130 00:07:43,423 --> 00:07:49,587 we have more than 500 systems every day running in our in our corporation 131 00:07:49,607 --> 00:07:50,567 globally. 132 00:07:50,864 --> 00:07:54,826 So it's not like 10 systems or 15 or 40. 133 00:07:54,826 --> 00:07:59,408 It's really a huge amount of systems. 134 00:07:59,408 --> 00:08:04,571 So you really have to do something seriously in order to understand that. 135 00:08:05,652 --> 00:08:10,835 Of course, what we now have to do, we start with the new systems, where by 136 00:08:10,835 --> 00:08:17,038 design we take this into account, but then slowly we have to do retrospective work on 137 00:08:17,038 --> 00:08:20,139 looking at all the systems that we have. 138 00:08:22,094 --> 00:08:26,656 But if you have a question like, for example, do we have a centralized AI 139 00:08:26,656 --> 00:08:31,279 review board for ethics issues versus do we decentralize it out? 140 00:08:31,760 --> 00:08:34,901 How do you get to answer some of those questions at an organization? 141 00:08:37,620 --> 00:08:40,302 Yeah, that's a good point. 142 00:08:41,843 --> 00:08:45,787 It depends of course, whether I think one of the first things that is important is, 143 00:08:45,787 --> 00:08:51,292 are you a provider of artificial intelligence, mainly, or are you a user of 144 00:08:51,292 --> 00:08:52,553 artificial intelligence? 145 00:08:52,553 --> 00:08:57,657 A company like Telefonica or energy companies or financial companies, they are 146 00:08:57,657 --> 00:09:03,983 all users, pharmaceuticals, and companies like Microsoft or Google, et cetera, they 147 00:09:03,983 --> 00:09:05,203 are providers. 148 00:09:05,444 --> 00:09:06,185 of this technology. 149 00:09:06,185 --> 00:09:09,408 So I think that makes a very different thing. 150 00:09:09,408 --> 00:09:15,054 If you are a provider, and you're a big tech serving thousands or millions of 151 00:09:15,054 --> 00:09:19,598 customers across the globe, I think you have to be very centralized. 152 00:09:19,698 --> 00:09:23,121 You can't leave this locally because then many things will happen. 153 00:09:23,462 --> 00:09:29,047 If you are a user of this technology, it depends a bit on how 154 00:09:29,236 --> 00:09:30,797 unified you are as a corporation. 155 00:09:30,797 --> 00:09:35,039 I mean, if you have acquired, if you exist because of an emerging acquisition of 30 156 00:09:35,039 --> 00:09:39,041 different companies, it's not so easy to do centralized. 157 00:09:39,461 --> 00:09:44,544 But if you are a centralized company to some extent, then it's always good to have 158 00:09:44,544 --> 00:09:45,865 a mixed approach. 159 00:09:45,865 --> 00:09:46,125 Yeah. 160 00:09:46,125 --> 00:09:53,469 So for instance, we have a, let's say at the corporation at headquarters where we 161 00:09:53,469 --> 00:09:58,171 run several of our global businesses and our support functions. 162 00:09:58,964 --> 00:10:05,129 We have a central model where we have an AI ethics committee, and we have certain 163 00:10:05,129 --> 00:10:09,292 persons in the central business units, because there are also central business 164 00:10:09,292 --> 00:10:09,412 units. 165 00:10:09,412 --> 00:10:16,238 There is, of course, finance, there is HR, but there is also digital transformation, 166 00:10:17,019 --> 00:10:20,021 let's say all our networks from a global perspective. 167 00:10:20,381 --> 00:10:21,462 We have all that centralized. 168 00:10:21,462 --> 00:10:26,426 And then, of course, we have operations in 15 companies across the world. 169 00:10:26,486 --> 00:10:28,167 And then what we tell them. 170 00:10:28,396 --> 00:10:32,538 is here you have a role model which you can use. 171 00:10:32,538 --> 00:10:34,780 And if you want to use it, we can help you. 172 00:10:34,980 --> 00:10:40,524 But given your situation locally, you are free to set up an alternative way of doing 173 00:10:40,524 --> 00:10:44,606 this as long as this is approved globally. 174 00:10:45,207 --> 00:10:51,031 So it's a kind of you cannot impose everything from the top because it's not 175 00:10:51,031 --> 00:10:52,492 one size fits all. 176 00:10:52,572 --> 00:10:55,474 But there are certain principles that you have to follow. 177 00:10:58,008 --> 00:11:04,591 that you have a figure called, actually a role, it's a function of another job 178 00:11:04,591 --> 00:11:06,772 called responsible AI champion. 179 00:11:06,772 --> 00:11:12,134 That is in every business unit that uses, develops, buys, or sells artificial 180 00:11:12,134 --> 00:11:15,276 intelligence needs to have such a function. 181 00:11:15,416 --> 00:11:21,619 And this person is the go-to person for AI ethical responsibility questions across 182 00:11:21,619 --> 00:11:22,999 that unit. 183 00:11:24,524 --> 00:11:29,328 Now, if you don't use any AI in a certain area, of course you don't need such a 184 00:11:29,328 --> 00:11:30,168 person. 185 00:11:30,349 --> 00:11:34,913 If you use a lot, maybe you need three and it's not even full-time. 186 00:11:34,913 --> 00:11:42,419 So it's just assignment of a new real refunction to an existing job position. 187 00:11:42,860 --> 00:11:48,765 And so that is completely distributed because it sits in the businesses. 188 00:11:49,126 --> 00:11:51,088 And then of course you need some... 189 00:11:51,088 --> 00:11:55,051 obstruct you to organize, to bring those people together, to train them. 190 00:11:55,632 --> 00:11:58,034 You need an AI ethics committee. 191 00:11:58,034 --> 00:12:02,338 And then of course you can have the same, you can have for each big area of 192 00:12:02,338 --> 00:12:06,441 business, you can have a separate AI ethics committee, or you can have one 193 00:12:06,441 --> 00:12:08,223 centralized for the company. 194 00:12:08,223 --> 00:12:09,904 What is the best situation? 195 00:12:10,525 --> 00:12:11,886 So actually there are two factors. 196 00:12:11,886 --> 00:12:15,569 One is, do you have enough experts to make such statements? 197 00:12:15,569 --> 00:12:17,110 Because it's not an easy. 198 00:12:18,220 --> 00:12:22,943 Of course, it sounds like anybody has an opinion about ethics and about AI, but 199 00:12:22,943 --> 00:12:23,824 that's not what you want. 200 00:12:23,824 --> 00:12:30,349 You want people with a mix of profiles, of backgrounds, but you also want to have 201 00:12:30,349 --> 00:12:32,711 some expertise in the area. 202 00:12:32,711 --> 00:12:36,294 So you don't have always the luxury to have like five committees in the different 203 00:12:36,294 --> 00:12:39,337 business because simply it's not yet there over time. 204 00:12:39,337 --> 00:12:39,497 Yeah. 205 00:12:39,497 --> 00:12:43,340 There will be much more experts, but today that's a difficult question. 206 00:12:43,800 --> 00:12:47,603 And, and the other part is. 207 00:12:49,180 --> 00:12:51,241 whether how much AI there is. 208 00:12:51,241 --> 00:12:52,061 Yeah. 209 00:12:52,061 --> 00:12:56,424 So if there is a hundred systems, maybe with one committee, because it's 210 00:12:56,424 --> 00:13:02,387 sufficient, because it's not a, so you have also two ways of using an AI, AI 211 00:13:02,387 --> 00:13:03,628 ethics committee. 212 00:13:03,628 --> 00:13:08,610 One is permission-based and the other is problem-based. 213 00:13:08,631 --> 00:13:13,833 So permission-based means anything that will be done by the company first goes to 214 00:13:13,833 --> 00:13:14,374 the committee. 215 00:13:14,374 --> 00:13:16,115 There is a discussion, yes, we can do that. 216 00:13:16,115 --> 00:13:17,975 No, we cannot do it or whatever. 217 00:13:18,724 --> 00:13:20,505 And that's permission based. 218 00:13:20,505 --> 00:13:24,488 And problem based is, look, we're working with this AI product. 219 00:13:25,209 --> 00:13:28,412 We are discussing about the impact, and we see there are some doubts. 220 00:13:28,412 --> 00:13:31,555 Should we use gender as a variable in the model? 221 00:13:31,555 --> 00:13:32,175 Yes or no? 222 00:13:32,175 --> 00:13:34,117 Now, that's a problem. 223 00:13:34,117 --> 00:13:37,080 And then you bring it to the AI ethics committee, and they discuss. 224 00:13:37,080 --> 00:13:39,201 Only about ethics, never about business. 225 00:13:39,482 --> 00:13:42,004 And then they highlight, and they give recommendations. 226 00:13:42,064 --> 00:13:43,925 And then they go back to the team. 227 00:13:44,646 --> 00:13:46,308 Now, if the team agrees, then they 228 00:13:46,308 --> 00:13:50,190 they go forward, if they don't agree, is escalated to an ad hoc committee where you 229 00:13:50,190 --> 00:13:52,472 bring together ethics and business. 230 00:13:52,472 --> 00:13:54,633 And in the end, the company takes a decision. 231 00:13:54,633 --> 00:13:58,436 The important thing is an informed decision, it's an explicit decision, and 232 00:13:58,436 --> 00:14:00,037 it's a documented decision. 233 00:14:00,037 --> 00:14:05,341 So that's the difference by just doing things. 234 00:14:05,341 --> 00:14:11,826 So that works quite well from the ethics perspective. 235 00:14:11,826 --> 00:14:14,607 Now, if you are a technology provider, 236 00:14:15,468 --> 00:14:20,270 where you have billions of customers, maybe is more a permission-based. 237 00:14:20,370 --> 00:14:27,013 So we want to build an AI system that analyzes all electronic communications in 238 00:14:27,013 --> 00:14:28,213 enterprises. 239 00:14:28,614 --> 00:14:36,757 And then we can find hubs of knowledge and isolated hubs of departments that are not 240 00:14:36,757 --> 00:14:37,858 in contact. 241 00:14:38,718 --> 00:14:42,059 So that is a product that has, well. 242 00:14:42,060 --> 00:14:47,942 potentially serious implications for workers and some ethical questions. 243 00:14:47,942 --> 00:14:52,484 So then it's good actually, so if you provide this product to billions of 244 00:14:52,484 --> 00:14:55,906 people, then it's good to have a permission base because the impact is 245 00:14:55,906 --> 00:14:56,046 huge. 246 00:14:56,046 --> 00:15:01,988 So it's again, it's not the same if you're mostly a user of this technology where 247 00:15:01,988 --> 00:15:04,509 you're a big tech provider. 248 00:15:04,630 --> 00:15:08,671 Startups is a different world because they are smaller. 249 00:15:09,756 --> 00:15:15,893 And they, of course, they have so many resources, but still, if it's high risk, 250 00:15:15,893 --> 00:15:18,479 it's important to think those things in advance. 251 00:15:19,370 --> 00:15:23,095 Does the fact that telephonica is in telecommunications, which is a regulated 252 00:15:23,095 --> 00:15:27,381 industry, make it easier to adopt these kinds of practices? 253 00:15:31,592 --> 00:15:36,373 Well, of course, but there are many regulated industries, yeah, and 254 00:15:36,373 --> 00:15:39,514 telecommunications is regulated, but finance and insurance is even more 255 00:15:39,514 --> 00:15:42,955 regulated and healthcare is enormously regulated. 256 00:15:42,955 --> 00:15:52,718 So, of course, the culture of compliance, yeah, of having to having processes and 257 00:15:52,718 --> 00:15:57,139 systems in place that check things, that is... 258 00:15:57,140 --> 00:15:58,060 already in place. 259 00:15:58,060 --> 00:16:00,282 So you can a lot of piggyback on that. 260 00:16:00,402 --> 00:16:04,146 If you are a company that is not affected by any regulation, of course, then it 261 00:16:04,146 --> 00:16:08,289 probably is harder to start with. 262 00:16:08,389 --> 00:16:12,493 But take into account that we started from the business. 263 00:16:13,194 --> 00:16:20,460 We didn't start from compliance because at that moment, this was not on the radar of 264 00:16:20,460 --> 00:16:21,540 compliance. 265 00:16:21,561 --> 00:16:24,883 Today, companies who are aware of upcoming regulation 266 00:16:25,007 --> 00:16:27,587 Of course, they will start from compliance. 267 00:16:29,398 --> 00:16:34,596 How does generative AI change the landscape for responsible AI practices in 268 00:16:34,596 --> 00:16:35,317 your view? 269 00:16:38,901 --> 00:16:42,983 Technically speaking, not too much. 270 00:16:43,564 --> 00:16:52,310 I mean, ethical AI, responsible use of AI is about estimating the ethical and social 271 00:16:52,310 --> 00:16:55,292 impact on human rights by design. 272 00:16:55,292 --> 00:17:02,678 I don't know if that is machine learning with structured data or whether that is 273 00:17:02,678 --> 00:17:05,659 generative AI with unstructured data. 274 00:17:06,184 --> 00:17:13,909 You still have to evaluate the impact on human rights. 275 00:17:13,909 --> 00:17:18,833 Now the difference is that, let's say traditional AI is narrow AI. 276 00:17:18,833 --> 00:17:19,053 Yeah. 277 00:17:19,053 --> 00:17:22,355 So you have a use case and you evaluate the use case. 278 00:17:22,495 --> 00:17:25,657 Now, generative AI is kind of general purpose to some extent. 279 00:17:25,657 --> 00:17:25,878 Yeah. 280 00:17:25,878 --> 00:17:27,419 You can use it for different tasks. 281 00:17:27,419 --> 00:17:32,522 You can use it for very innocent tasks and also for very serious tasks. 282 00:17:33,303 --> 00:17:34,643 And that means that. 283 00:17:36,100 --> 00:17:42,504 Um, it is different because for narrow AI, it makes sense to evaluate use cases and 284 00:17:42,504 --> 00:17:43,385 not the technology. 285 00:17:43,385 --> 00:17:49,429 You don't, uh, you don't evaluate a recommendation algorithm or a random 286 00:17:49,429 --> 00:17:52,651 forest algorithm in terms of ethical impact. 287 00:17:52,651 --> 00:17:57,815 Once you apply it to a situation, then you, uh, evaluate the impact. 288 00:17:58,796 --> 00:18:04,339 And with, with generator to vie is this, this border is not so clear cut. 289 00:18:04,844 --> 00:18:08,706 Because I can bring to the market, and this is what the companies do, they have 290 00:18:08,706 --> 00:18:14,590 the big tech do, a general purpose AI that I can use as a user or as a business for 291 00:18:14,590 --> 00:18:15,970 many different things. 292 00:18:16,051 --> 00:18:23,535 I can use it for generating marketing texts, which is probably not so impactful. 293 00:18:23,535 --> 00:18:30,759 But I can also use it to analyze natural language of patient dossiers. 294 00:18:31,148 --> 00:18:35,049 and then make a decision for certain patients in terms of healthcare decisions, 295 00:18:35,049 --> 00:18:37,509 which is very high impact. 296 00:18:37,730 --> 00:18:46,352 I can record interviews for HR automatically and then have them analyzed 297 00:18:46,512 --> 00:18:52,013 by Generative AI to see whether I should hire the people that has a lot of impact. 298 00:18:52,154 --> 00:18:58,655 So it's difficult to put an ethical impact assessment on chat GPT. 299 00:18:59,208 --> 00:19:00,468 as such. 300 00:19:00,468 --> 00:19:02,570 So you have to do it. 301 00:19:02,570 --> 00:19:08,614 And as a user, that doesn't change because we never use chat GPT or generative dive 302 00:19:08,614 --> 00:19:10,155 for whatever. 303 00:19:10,155 --> 00:19:16,039 So we use it to improve this or in our legal area or in our whatever area. 304 00:19:16,039 --> 00:19:21,503 So we analyze the use case of this technology. 305 00:19:21,503 --> 00:19:22,744 But if you are a provider, 306 00:19:22,744 --> 00:19:23,844 then it makes a difference. 307 00:19:23,844 --> 00:19:29,889 And that's also where the AI Act has found a lot of difficulties in coming to 308 00:19:29,889 --> 00:19:31,089 agreement. 309 00:19:31,390 --> 00:19:39,135 And because if you want to regulate it as one system, let's say as a high risk, 310 00:19:39,135 --> 00:19:44,399 impactful system, then you also regulate a lot of innocent use cases and the other 311 00:19:44,399 --> 00:19:45,279 way around. 312 00:19:46,666 --> 00:19:47,126 Absolutely. 313 00:19:47,126 --> 00:19:50,631 But let's say you're in a company like Telefonica that's not the one developing 314 00:19:50,631 --> 00:19:54,014 the large language models, but which has a lot of these use cases. 315 00:19:54,215 --> 00:19:59,942 And so a product group says we want to use a chat bot for customer service, 316 00:20:00,743 --> 00:20:02,045 generative AI. 317 00:20:02,045 --> 00:20:06,029 How do you think about the addressing those kinds of ethical issues? 318 00:20:08,780 --> 00:20:12,000 Well, there is beyond the ethical question. 319 00:20:12,000 --> 00:20:14,561 There is also just a performance question. 320 00:20:14,741 --> 00:20:23,724 So if I use a chat bot externally with customers and I based that large part on 321 00:20:23,724 --> 00:20:29,625 existing systems like Lama2 or like, like chat GPT, then I have a significant 322 00:20:29,625 --> 00:20:35,947 business risk, not even speaking about ethical because those things can answer 323 00:20:35,947 --> 00:20:36,967 you whatever. 324 00:20:37,280 --> 00:20:39,660 whatever they can go into a direction. 325 00:20:40,520 --> 00:20:47,722 So what companies have to do is kind of how do you tame or limit what those 326 00:20:47,722 --> 00:20:50,283 systems can say if they are talked to with your customers. 327 00:20:50,283 --> 00:20:53,704 And that's even only from a business perspective. 328 00:20:53,704 --> 00:20:54,344 Yeah. 329 00:20:54,424 --> 00:20:57,685 You don't want, so if we have a chat bot where we can, we can talk with our 330 00:20:57,685 --> 00:21:02,186 customers, you want them to ask questions about the, about the elections in the U S 331 00:21:02,346 --> 00:21:06,440 and that we make a statement that George chat bot makes a statement about. 332 00:21:06,440 --> 00:21:07,820 candidate or so. 333 00:21:08,000 --> 00:21:11,182 Now, how do you limit that? 334 00:21:12,443 --> 00:21:16,945 Of course, if you manage to do that, then there are technologies for that, rack 335 00:21:16,945 --> 00:21:23,869 technologies, whatever, where you take your own content, and then you use that to 336 00:21:23,869 --> 00:21:28,131 generate the information, and then you use ChetGBT to have the dialogue and the 337 00:21:28,131 --> 00:21:31,853 conversation to put everything together in a natural way. 338 00:21:31,893 --> 00:21:36,075 That reduces significantly the hallucination part. 339 00:21:36,760 --> 00:21:38,840 But still it can still have a lot of bias. 340 00:21:38,840 --> 00:21:46,082 Yeah, so like bias professional bias in terms of professions typical male or 341 00:21:46,222 --> 00:21:53,004 Female process professions or it can even have Yeah, let's say use language that we 342 00:21:53,004 --> 00:21:59,126 don't want them to use insinuating Things that are not inclusive. 343 00:21:59,146 --> 00:22:02,747 So there's still quite some challenges from an ethical perspective. 344 00:22:02,747 --> 00:22:04,627 But then what you have to do is 345 00:22:04,644 --> 00:22:10,285 This is my knowing what is your target audience and then say, okay, suppose that 346 00:22:10,285 --> 00:22:12,066 something is going wrong in this direction. 347 00:22:12,066 --> 00:22:16,187 So what is the impact on the people it's working with? 348 00:22:16,207 --> 00:22:19,988 Is that impact severe or light? 349 00:22:21,108 --> 00:22:23,309 What's the probability that it happens? 350 00:22:23,309 --> 00:22:26,330 Is it 0.1% or is it 20%? 351 00:22:26,330 --> 00:22:29,391 And then how many people are involved? 352 00:22:29,391 --> 00:22:32,784 Is it 10, 100, a million, a full nation? 353 00:22:32,784 --> 00:22:36,967 So all those things matter in estimating what is the impact. 354 00:22:36,967 --> 00:22:42,492 And if you do that by design and in a systematic way, then the risk that 355 00:22:42,492 --> 00:22:46,395 something comes out that you don't want is much less. 356 00:22:46,395 --> 00:22:48,737 Of course, you can never avoid it, yeah? 357 00:22:48,737 --> 00:22:50,898 Because some things you never know in advance. 358 00:22:51,139 --> 00:22:55,763 But still, I mean, it's much better to do it in such a way than just do it as quick 359 00:22:55,763 --> 00:23:02,107 as possible, put it in the market and then discover, oops, we didn't know this. 360 00:23:03,234 --> 00:23:11,549 What about cultural context, language, how specific do these kinds of mechanisms have 361 00:23:11,549 --> 00:23:12,249 to be? 362 00:23:14,500 --> 00:23:20,225 Well, it is true that because those machines, this is massive brute force 363 00:23:20,225 --> 00:23:22,806 calculations with statistical models. 364 00:23:23,708 --> 00:23:28,171 Of course, the dominant culture on the internet is English and US. 365 00:23:30,013 --> 00:23:35,398 So of course, many things that it answers, even though in a different language, still 366 00:23:35,398 --> 00:23:41,343 the culture that is encoded implicitly in the model is a specific culture. 367 00:23:42,116 --> 00:23:48,040 And that has as a disadvantage that smaller cultures can be, yeah, let's see, 368 00:23:48,181 --> 00:23:49,842 have no, no role. 369 00:23:49,842 --> 00:23:51,443 Uh, not so much. 370 00:23:51,443 --> 00:23:55,787 Of course, in the Spanish speaking world, there are still a 600 million Spanish 371 00:23:55,787 --> 00:23:57,008 speaking people in the world. 372 00:23:57,008 --> 00:23:59,430 So still a very large language. 373 00:23:59,430 --> 00:23:59,810 Yeah. 374 00:23:59,810 --> 00:24:03,974 Not as, of course, by far, not as big as English, but definitely bigger than the 375 00:24:03,974 --> 00:24:05,955 Dutch and from the Netherlands. 376 00:24:05,975 --> 00:24:09,318 So only 20 million people in the world who speak that. 377 00:24:09,318 --> 00:24:11,096 So, um, 378 00:24:11,096 --> 00:24:17,638 From a language perspective, in terms of words, that can be worked on, because 379 00:24:17,778 --> 00:24:23,320 there are initiatives, local initiatives, where people who love their language 380 00:24:23,321 --> 00:24:25,902 always put focus on that. 381 00:24:25,902 --> 00:24:28,383 So I'm not so worried about that. 382 00:24:28,383 --> 00:24:35,165 But the cultural aspect is something that has to be taken into account. 383 00:24:35,506 --> 00:24:40,267 As far as I know, that is not a solved problem, but it's a research problem. 384 00:24:42,702 --> 00:24:47,988 So you mentioned a couple of times the AI Act that was recently adopted in Europe. 385 00:24:47,988 --> 00:24:54,496 How do you see that or the broader move globally to adopt AI regulations impacting 386 00:24:54,496 --> 00:24:55,977 what companies are doing? 387 00:25:00,024 --> 00:25:02,366 I think it's a very good step, first step. 388 00:25:02,366 --> 00:25:06,648 It's a good step in the right, an important direction. 389 00:25:08,570 --> 00:25:13,193 I think global regulation or global agreements are very important. 390 00:25:13,373 --> 00:25:19,477 That's why the EU act is important, because it covers 27 countries. 391 00:25:19,477 --> 00:25:21,498 It's not for one country only. 392 00:25:21,638 --> 00:25:23,579 Of course, the world is much larger. 393 00:25:24,016 --> 00:25:29,239 than 27 companies and there of course there is the US and there is the UK, there 394 00:25:29,239 --> 00:25:31,461 is China, Russia. 395 00:25:32,281 --> 00:25:38,506 In that perspective I'm a fan of the UNESCO Ethical AI recommendation because 396 00:25:38,506 --> 00:25:49,453 it covers 193 plus one countries including the US, China, Saudi Arabia, Iran, so 397 00:25:49,954 --> 00:25:52,475 countries that are very hard to get together. 398 00:25:53,396 --> 00:25:58,457 and they are together and they have signed up to an ethical use of artificial 399 00:25:58,457 --> 00:25:59,517 intelligence. 400 00:25:59,558 --> 00:26:02,278 Of course, this is just a sign up, yeah, there's no sanction. 401 00:26:02,278 --> 00:26:07,620 If you still do many different things, nothing happens and that's the situation 402 00:26:07,620 --> 00:26:08,140 today. 403 00:26:08,140 --> 00:26:11,221 But that situation is also happening with human rights. 404 00:26:11,541 --> 00:26:16,523 There are countries who signed an international human rights declaration and 405 00:26:16,523 --> 00:26:19,023 are not respecting human rights as they should. 406 00:26:19,023 --> 00:26:19,623 So... 407 00:26:20,044 --> 00:26:24,325 This is a necessary step towards a more formal regulation. 408 00:26:25,845 --> 00:26:29,726 So I think the AI Act is a good thing. 409 00:26:30,106 --> 00:26:36,508 But of course, it also can be improved to some extent and is still on paper. 410 00:26:36,568 --> 00:26:41,449 There's very little experience with what it means in practice. 411 00:26:41,530 --> 00:26:47,431 I think a very important thing is this is risk-based and that the majority of the 412 00:26:47,431 --> 00:26:48,891 use cases of AI... 413 00:26:49,108 --> 00:26:50,488 are not high risk. 414 00:26:50,608 --> 00:26:53,589 They're limited list, so they have no regulation at all. 415 00:26:53,589 --> 00:27:00,012 But there are certain areas where there is potential risk and those areas are 416 00:27:00,012 --> 00:27:01,032 regulated. 417 00:27:01,753 --> 00:27:07,475 So from very light regulation in terms of if you make a deep fake, you have to say 418 00:27:07,475 --> 00:27:09,996 it's a deep fake. 419 00:27:10,036 --> 00:27:12,677 You can't claim that it is for real. 420 00:27:13,698 --> 00:27:15,939 There's transparency obligation. 421 00:27:15,939 --> 00:27:18,159 If it's high risk in terms of 422 00:27:19,380 --> 00:27:20,441 impacting human rights. 423 00:27:20,441 --> 00:27:26,286 So if I want to use AI to hire people or to fire people, or to accept students, or 424 00:27:26,286 --> 00:27:31,630 to give a loan, determine the interest rate, or whether I accept people for a 425 00:27:31,630 --> 00:27:36,414 loan, a mortgage, medical applications. 426 00:27:37,115 --> 00:27:39,717 So everything that is essential for people. 427 00:27:39,897 --> 00:27:46,042 If you use AI for those specific things, then you need to comply with certain 428 00:27:46,943 --> 00:27:47,983 requirements. 429 00:27:48,744 --> 00:27:51,286 before putting it in the market. 430 00:27:51,286 --> 00:27:56,230 So the data should not have bias, should have quality, should have human oversight, 431 00:27:57,432 --> 00:27:59,313 the adequate human oversight. 432 00:28:01,655 --> 00:28:06,519 You should log the system in the sense that if something happens later, you can 433 00:28:06,519 --> 00:28:08,681 investigate whether there was a problem. 434 00:28:08,681 --> 00:28:14,166 So it's a set of requirements that you have to take into account, but only for 435 00:28:14,166 --> 00:28:16,347 high risk use cases. 436 00:28:16,768 --> 00:28:22,049 And then there is unacceptable risk and those things are outright forbidden to do. 437 00:28:22,049 --> 00:28:27,571 If you do that, you have to pay 35 million or 17% of your global annual review. 438 00:28:27,571 --> 00:28:31,832 So, so, and then there is the part of general purpose AI. 439 00:28:31,832 --> 00:28:35,913 So generative AI, which they have put in later. 440 00:28:36,714 --> 00:28:42,535 Um, and they have struggled a lot with the fact that 441 00:28:42,628 --> 00:28:46,950 The AI Act was based on use cases and suddenly there is a system that you can 442 00:28:46,950 --> 00:28:49,051 use for many use cases. 443 00:28:49,511 --> 00:28:57,616 So I think that if you use generative AI for high risk, then you just do the same 444 00:28:57,616 --> 00:29:00,057 thing as you have to do for any other system. 445 00:29:00,057 --> 00:29:03,659 You have to assess the impact, you have to comply with certain things. 446 00:29:03,839 --> 00:29:10,763 Now, if you use a third party for generative AI, like OpenAI or Microsoft, 447 00:29:10,963 --> 00:29:12,463 for your use case, 448 00:29:12,720 --> 00:29:19,902 of little risk, nothing to do from the user perspective, from the company 449 00:29:19,902 --> 00:29:20,622 perspective. 450 00:29:20,622 --> 00:29:23,843 But if you are using it for high risk, then you have to comply. 451 00:29:23,843 --> 00:29:28,964 And that means there is a transparency obligation towards the provider that the 452 00:29:28,964 --> 00:29:32,725 provider has to say, okay, this system is trained on this data, I've done this 453 00:29:32,725 --> 00:29:36,366 checks on bias, I can monitor, etc. 454 00:29:36,366 --> 00:29:40,307 So that is a kind of tension that is written in the law. 455 00:29:40,516 --> 00:29:45,159 but we don't know how it spells out in practice. 456 00:29:45,660 --> 00:29:52,526 And then if you are a provider of such a model like OpenAI or like Microsoft, then 457 00:29:52,526 --> 00:29:55,869 they speak about systemic risk systems. 458 00:29:55,869 --> 00:30:02,214 And that means systems that are so big that they actually can create an impact in 459 00:30:02,214 --> 00:30:03,855 society at large. 460 00:30:04,777 --> 00:30:07,179 And there are some requirements on top of the system. 461 00:30:07,179 --> 00:30:08,344 So they have to... 462 00:30:08,344 --> 00:30:12,344 have a system of notification, they have to alert system, they have to check 463 00:30:12,605 --> 00:30:14,545 periodically the data, et cetera. 464 00:30:15,005 --> 00:30:17,726 And they have to report on those things to the European Commission. 465 00:30:17,726 --> 00:30:22,407 But how they check, what they check is left to the industry. 466 00:30:22,407 --> 00:30:28,249 So it's a kind of hybrid self-regulation, formal regulation, and that is definitely 467 00:30:28,249 --> 00:30:33,530 a compromise between different perspectives. 468 00:30:33,530 --> 00:30:37,591 And in my view, and that's my personal view, is a... 469 00:30:38,024 --> 00:30:45,267 incorrect discussion, because it speaks about either you innovate, or you are 470 00:30:45,267 --> 00:30:46,427 responsible. 471 00:30:46,707 --> 00:30:51,389 Yeah, so if you want to innovate, you cannot be responsible, because there are 472 00:30:51,389 --> 00:30:55,871 other countries, yeah, who are not responsible, and they can innovate faster. 473 00:30:56,031 --> 00:31:02,294 So that's the kind of discussion that's put up like the US innovates, China 474 00:31:02,294 --> 00:31:05,635 copies, and the EU regulates. 475 00:31:06,555 --> 00:31:07,656 Now, 476 00:31:07,828 --> 00:31:15,210 If you speak about responsible AI by design, so regulation by design for high 477 00:31:15,210 --> 00:31:20,531 risk things, that means that in all the way, in all the process that you build 478 00:31:20,531 --> 00:31:25,072 things like chat GPT or whatever, you start thinking already, or what could be 479 00:31:25,072 --> 00:31:27,953 the impact if things go wrong or what about hate speech? 480 00:31:27,953 --> 00:31:28,733 How do I train? 481 00:31:28,733 --> 00:31:31,254 So you think in advance about all those things. 482 00:31:31,254 --> 00:31:37,200 And then in the end, what comes out of it is probably a more responsible. 483 00:31:37,200 --> 00:31:40,362 project rather than correcting it once in the market. 484 00:31:40,943 --> 00:31:45,807 And also think, if you launch something to the market and you discover a problem, 485 00:31:45,807 --> 00:31:50,551 which has happened several times with Google, with the generative AI's they put 486 00:31:50,551 --> 00:31:56,236 in the market, actually impact and the cost of repairing that is much higher from 487 00:31:56,236 --> 00:32:00,619 all perspectives than if you do it while you are designing. 488 00:32:00,619 --> 00:32:01,279 Yeah. 489 00:32:02,612 --> 00:32:07,714 And then there are actually several business benefits from doing innovation in 490 00:32:07,714 --> 00:32:08,575 a responsible way. 491 00:32:08,575 --> 00:32:12,817 So it's not doing either you innovate or a responsible, no, you innovate in a 492 00:32:12,817 --> 00:32:13,758 responsible way. 493 00:32:13,758 --> 00:32:20,481 And if you do that, society is more and more requiring companies to be responsible 494 00:32:20,481 --> 00:32:21,622 beyond profit. 495 00:32:21,622 --> 00:32:23,443 We see that with climate change. 496 00:32:23,803 --> 00:32:26,024 Climate change is actually a business factor. 497 00:32:26,024 --> 00:32:31,847 And now at this moment, investors look at your climate rankings and ratings. 498 00:32:32,000 --> 00:32:34,922 and emissions before they invest in you. 499 00:32:34,922 --> 00:32:40,866 So there is a trend that even your customers value, not only your profits, 500 00:32:40,866 --> 00:32:43,948 but also value your responsibility. 501 00:32:43,948 --> 00:32:47,811 So if you're responsible and you show that, and actually you do it actually in 502 00:32:47,811 --> 00:32:50,292 practice, that's a positive point. 503 00:32:51,933 --> 00:32:53,954 The same for your employees. 504 00:32:54,075 --> 00:33:00,059 So employees become more and more aware of those things and they want companies that 505 00:33:00,059 --> 00:33:00,940 not only 506 00:33:00,940 --> 00:33:05,422 make money, but it also have a responsible role in society. 507 00:33:06,363 --> 00:33:10,005 So that also is actually a business benefit. 508 00:33:10,005 --> 00:33:13,666 If you are a responsible company and you've thought about the things, actually 509 00:33:13,727 --> 00:33:20,370 governments who are also your customers, they talk to you more. 510 00:33:20,370 --> 00:33:24,573 They ask you for support when this goes about regulation of AI. 511 00:33:24,573 --> 00:33:25,813 They want to have your opinion. 512 00:33:25,813 --> 00:33:29,955 So you create actually a better relationship. 513 00:33:31,341 --> 00:33:35,924 with the public administration, which always is a large buyer of many companies. 514 00:33:35,924 --> 00:33:40,528 So, of course, those are not so intangible that they appear on the balance sheet, but 515 00:33:40,528 --> 00:33:44,771 I think those are all recognized factors that are positive. 516 00:33:47,018 --> 00:33:50,980 You recently left Telefonica, although I know you're still very involved in this 517 00:33:50,980 --> 00:33:53,941 whole area of responsible AI in various contexts. 518 00:33:53,941 --> 00:34:00,545 Tell us what you see as the biggest issues going forward that companies and other 519 00:34:00,545 --> 00:34:02,165 organizations need to focus on. 520 00:34:04,524 --> 00:34:10,567 I think on the short time for companies, it is, I mean, depending of course, where 521 00:34:10,567 --> 00:34:14,849 they are, but in Europe is be prepared for the regulation. 522 00:34:15,109 --> 00:34:19,631 And that for some companies is going to be like the GDPR was a lot of work. 523 00:34:19,892 --> 00:34:25,815 So it will be a lot of work, but it is not something completely new. 524 00:34:25,815 --> 00:34:32,218 We kind of copying the process we did for GDPR with a different content, some maybe 525 00:34:32,218 --> 00:34:33,559 a bit more difficult. 526 00:34:34,500 --> 00:34:41,363 But so it will, that is, I think, one of the things that is coming from a company 527 00:34:41,363 --> 00:34:42,543 perspective. 528 00:34:43,223 --> 00:34:47,906 I think another thing that is, should be coming and is maybe not yet coming are 529 00:34:47,906 --> 00:34:56,589 more the indirect and more global effects of the use of artificial intelligence. 530 00:34:56,770 --> 00:35:01,388 So what happens to the future of work is in the end. 531 00:35:01,388 --> 00:35:05,409 Creating more work or is it creating less work if it creates less? 532 00:35:05,609 --> 00:35:08,530 So what does it mean for the financial sustainability of states? 533 00:35:08,530 --> 00:35:11,011 Because there will be less income taxes. 534 00:35:11,011 --> 00:35:13,452 Is this the time for Robo tax? 535 00:35:13,472 --> 00:35:15,853 Is it the time for UBI? 536 00:35:15,873 --> 00:35:22,616 So those discussions are not dealt with at the moment with regulation. 537 00:35:23,477 --> 00:35:24,937 But I think they are important. 538 00:35:24,937 --> 00:35:29,199 There is also an issue on equality and gaps. 539 00:35:29,199 --> 00:35:31,079 So this is. 540 00:35:31,764 --> 00:35:33,644 very powerful technologies. 541 00:35:33,644 --> 00:35:40,526 It helps increase global GDP, it helps increase national GDPs, and also market 542 00:35:40,526 --> 00:35:42,747 caps and profits of companies. 543 00:35:42,747 --> 00:35:47,928 But if you look at how this wealth is distributed, then it's ending up in a few 544 00:35:47,928 --> 00:35:51,329 countries and in a few companies. 545 00:35:51,349 --> 00:35:59,491 So there is a challenge on how to make that a fair system in the long run. 546 00:36:00,000 --> 00:36:04,742 Otherwise, if we keep going as we are, so the companies who are doing this now, they 547 00:36:04,742 --> 00:36:08,943 will become bigger and bigger and bigger and more powerful and more powerful, and 548 00:36:08,943 --> 00:36:14,906 they will become richer and richer and richer at the expense of other people. 549 00:36:14,906 --> 00:36:15,066 Yeah. 550 00:36:15,066 --> 00:36:22,409 So the question of leaving nobody behind is very important here and is not dealt 551 00:36:22,409 --> 00:36:23,049 with either. 552 00:36:23,049 --> 00:36:28,371 So regulation takes things into account of the direct impact of AI. 553 00:36:28,932 --> 00:36:32,715 within the scope of individual organizations. 554 00:36:33,136 --> 00:36:40,165 But the global issues that survive the use of this technology at large, they are 555 00:36:40,165 --> 00:36:43,087 still only part of discussions. 556 00:36:44,262 --> 00:36:46,068 Okay, much to think about. 557 00:36:46,068 --> 00:36:50,181 Richard, I want to thank you so much for joining me on the road to accountable AI. 558 00:36:50,984 --> 00:36:53,955 Okay, it was a pleasure to be here, Kevin.