You're welcome.
It's so great to talk to you.
It's so great to be here, Kevin.
Thanks for having me.
You've been involved for a long time at a number of different organizations, public
sector, private sector, nonprofit, on issues around internet policy, privacy,
and so forth.
How is it different or is it different in your perspective what's happening now with
AI?
I think that I just heard a ding, so I'm going to have to start over again to
silence my notifications.
Okay.
You know, Kevin, I think there are a lot of conversations right now about whether
this is truly different and kind of the really monumental change in how we live
our lives.
I'm a little bit of a skeptic about that in that I guess perhaps because you and I
have been working in this area for so long, we've seen
the trains of large language models and automated decision making and just
software that runs really fast in the background.
Why I think this feels really different to people is that, first of all, this has
come into the wild.
I mean, it's come into the public use very quickly and that it seems to be alive.
People seem to feel like this is making decisions about them or it's talking back
to them.
And although we know cognitively it's not alive, having a converse with you, having
an inanimate object suddenly talk your language feels different than previous
iterations of technology.
Why I think it's important that we scrutinize it is, and I've said this
before, it's not only the data that we're putting into it, it's the decisions it's
making about us in the world.
What about the development of AI governance and policy?
How have you seen the regulatory developments in this area around AI
relative to what we saw a decade ago or two decades ago with the internet and
privacy?
I think because it's moved so quickly from not known to known in the public, the
policymaking world is trying to catch up as usual to technology and doing it
quickly.
What I've also seen is that good companies, I think, are ready to apply
governance principles to new technologies because this is not the first one.
We have internet, we have...
social media, we have lots of policies around other uses of technologies by our
own, what we call employees, our associates.
And so we really, we in particular, my company, we're ready.
We had already thought about and started to build and started to benchmark with
other companies.
What does good governance look like?
for these tools.
And of course, you know, just say AI, it's almost like saying sidewalks, right?
It encompasses so much, right?
But to be general, kind of looking at emerging technologies, biometrics, AI, and
a whole bunch of other de novo uses of technology in the real world, proximate to
human beings, we did set about looking at what the tech companies were offering,
looking at what other companies were recommending, looking at what trade
associations and think tanks were saying about,
governance and the possibilities of governance.
I also sometimes like to call it trying to govern the ungovernable, right?
Because it's so opaque in so many ways, but it's also, we shouldn't engage in that
kind of tech exceptionalism that we are so special and so new and so different that
we shouldn't play by some reasonable rules or that it's just too hard to figure out.
If it's too hard to figure out, we shouldn't be using it, right?
So what we did was look at ordinary processes that companies put into place.
what the company's principles are, and then how we apply those as people.
And it's usually our engineering, our data scientists, our tech folks who want to
deploy new tools that will affect human beings.
We pull those out of the work streams and say, if this tool is going to be making
consequential decisions about a human being, their rights, their knowledge,
their abilities to move forward, in most particular,
their employment and advancement, that tool is going to require additional close
scrutiny by technologists and by lawyers and policy people before it goes live.
And then at waypoints, you know, six months a year, and then perhaps annually
after that, because as you know, these tools drift, they change, that, you know,
it is a constant governance model.
And we're still building towards automation in our governance and
monitoring of tools of all kinds.
And, but again, I don't think just because it's new and funky and cool, it should
escape some kind of scrutiny.
In fact, maybe, maybe more so because of that.
So you issued a Walmart Responsible AI pledge in October of 2023.
And first of all, I'm curious, it's interesting that you call it a pledge.
It's not just a set of principles.
Tell me a little bit about the process of developing that.
And like all of our kind of internal policies, statements, pledges, it's really
a multifunctional endeavor.
I don't think the lawyers should sit in a room and write things.
I don't think the technologists should sit in a room and write things.
The comms people shouldn't be working by themselves.
It was really a kind of collaborative approach that a number of our teams came
to my team and said, we'd like to say something publicly about the guardrails
and the guidelines and really the values that we live by as a company.
And I said, really, who's going to listen to that?
Again, somewhat the skeptic.
But in all seriousness, the pledge is really to our customers and associates,
people who we are engaging with to both be transparent and accountable in our uses.
And that sort of really reflects one of the things I say to my team all the time.
When we are introducing new technologies into the environment of human beings,
whether it's online or in the stores, in the real world,
in offices and in warehouses anywhere our associates work, we have a duty to explain
because in many cases it may be the first time someone's come into contact with
this.
And so I always say to our team, inform, educate, entertain.
And let me tell you what I mean by that.
Inform, obviously notice that, you know, that something is happening in the
environment that is different.
It's particularly in the stores, which were, you know, 20 years ago, you could
walk in, you could walk out, you could pay cash, you would be unobserved to any
system in the world.
That's changing, obviously, with your cell phones, with electric cash registers, with
cameras in the stores for security and safety.
So I think we have a duty to inform.
We are meeting people where they are in their comfort level with technology, in
their educational level, in their language abilities.
And we have to speak to all of those people.
So that's inform.
Educate.
Tell them a little more.
Tell them a little more about what the technology does, how it works.
When I grew up in New York, there was a company called Sims.
I actually had a summer job there one summer and their phrase was, an educated
consumer is our best customer.
I believe the same thing about technology and the technology and responsible
technology use of our company.
If they know the responsible choices we're making about these technologies, it will
engender trust and mutual respect and hopefully really benefit the ongoing
customer store relationship.
And the last part is entertain.
And honestly,
This is fun.
You and I were talking a little while ago, you know, I, in all of the years we've
been working, we, I get to learn something new almost every day.
And, you know, certainly every year there's a new, you know, or every few
years there's some monumental change in the technology.
That's a joy.
It's really a joy to be a part of that and to learn and to be surrounded by people
who are just so, so smart and dedicated and both want to do the right thing and
also want to use the latest technology to do whatever it is their job.
requires.
So and it should be fun.
We had a long time ago, we had these sweepers in some of the stores.
And if you go into Sam's Club, you might still see them.
And they forget the brand name, but they were about the height of a, you know, I
don't know, 10 or 12 year old boy.
I'm thinking about my son is probably, you know, a few years ago.
And but almost almost eye level.
And they had cameras.
And the purpose of the camera was actually to scan the shelves to say what's out of
stock so that
people in the back store room would go, okay, on aisle seven, we need to get more
cereal.
But a reasonable person would think, was that camera looking at me?
What's that camera?
What's it recording?
What's it gonna do with that information?
So I said to the guys, first of all, I thought we should have put vests on them
and made them, anthropomorphize them and also have disclosures right there on the
machine that says, I'm not looking at you.
In fact, we had blurred out any facial.
features of any human being to just be looking at boxes and QR codes and the
like.
But it can be fun.
I have no problem with anthropomorphizing the technology if it helps people
understand and feel comfortable with it.
me just stop a second.
Do you want to see if you can shut off that other thing that's dinging?
If it's too much of a pain, it's not a big deal.
This is not Hollywood, but yeah.
I don't know what that is.
okay.
how do I shut off the notifications?
Let's take a look on my
Okay, I think it's gone.
I think it's all off now.
Alright, thank you.
Yeah, it was not a big deal, but...
No, no, we don't want it in the background.
Okay.
Back at it.
Okay.
Hope that works.
No, OK, we'll see.
All right.
That's a good.
first.
no, no, you're okay.
Yeah, it's, I mean, whatever this, this tool Riverside is recording each of us
individually and uploading it.
So it should be fine.
Even if the.
Okay.
Yeah, it's interesting that one of the values that is in that pledge is customer
centricity, which is not typically what people see in terms of AI.
You've got things like fairness and so forth.
But why is customer centricity for Walmart a value in terms of responsible AI?
I think so many times companies roll out technologies that they say are for the
customer, but are really for the company or really have some other purpose.
And in this case, when we say customer centricity, it really goes to what I was
saying before.
Does the customer understand?
Do they feel comfortable with it?
Is it a use case that enhances their experience?
That's, I think, some of the values I think about when I think about customer
centricity.
But ultimately, it's about does this deployment...
serve the humans with which it is proximate.
And if it doesn't, why are we doing it?
And if those customers or associates don't understand it, then we have failed in our
duty of care, our duty to inform, educate, and explain.
So I think it really goes to, we have recognized that people will be impacted
even just having it in their environment, having it nearby, and that...
There may be different levels of comfort with it and that we have a responsibility
to those, to all of our customers, regardless of their comfort.
What else do you think is distinctive about the way you approach these AI
issues, given that Walmart is a retailer that touches literally hundreds of
millions of customers directly as opposed to enterprise service providers, for
example?
Well, again, it's because it's a frontline human machine interface, right?
We get to live in the real world and physical world as well as the internet and
see how people feel comfortable with these tools.
It's funny that we were focused on customers because I would say our most
effective rollouts initially were in tools that support our associates in what
they're doing.
You know, I give the example and I don't think it's a real one, maybe a real one
now.
when I was doing a Sam's Club tour and they let me decorate cupcakes, which I
think is hilarious.
Now, anyone who's had a young child knows when you're ordering a birthday cake, you
want to make sure you get the red one or the blue one or the right number of roses
or the right number of decorations.
And so there are guidebooks that tell associates, this is how many roses you put
on an eight by 11 sheet cake and this is what it should look like for a new
associate or even for someone experienced who hasn't done this particular task
before.
That means they run to the back room, they have to flip through the thing, they have
to page through and say, you know, what is, and then read and then decorate and
then read.
Just one example of all the many thousands of things associates do in a day.
With AI, there is a potential and I believe it's deployed already in some
places in the country to simply say, and we have a tool called Ask Sam, we have
another tool, we have tools named a number of things that are kind of in the Walmart
legend and language.
And say, you know, Sam, how do I decorate an eight and a half by 11 cheesecake for a
young, you know, for a boy, for a girl, for, you know, in a fire truck, whatever.
And that will save time, saves effort.
They can say, you know, repeat that.
It will really be an efficiency builder for an associate.
And again, that's a very, I think, fun and upbeat example, but there are lots of, you
know, paper based or knowledge worker examples as well that improve efficiency.
One of the things we have put online are a number of our employee manuals, our
benefits manuals, basic information.
When you've got millions of associates, many of them are going to be asking the
same question, right?
So saves time, makes sure the information is accurate, every associate is getting
the same information.
That's a good example of where regular old AI is a good enough tool.
do not need generative AI because you actually don't want it learning.
You don't want it learning and changing the content.
You want that content to be static and accurate and correct for everybody.
That's, I think, one of the other points I would make is people have gotten really
excited about generative AI.
And it's a fascinating tool.
And there's lots of good and bad about it.
But you don't necessarily, it's not necessarily the right tool for every job
that you might think of.
And so fit for purpose, to me, is a really big question.
you know, and accuracy and truth of the information to me, those are some of the
biggest worries I have about gender AI.
How do you structure the work on responsible AI in your organization or
your organization relative to the rest of the company?
Thanks for asking.
So digital citizenship is, as we define it, the responsible use of technology and
data, period, full stop.
That's the mission.
We have now got the privilege of leading a team of several hundred people.
And that includes lawyers, technologists, data scientists, policy people.
It's basically half kind of legal and half compliance and operational.
And so we went about setting up a team.
I really think that the woman who has to lead that team, when we created it,
had one of the best jobs in the company because her first year was doing all the
research and the benchmarking and finding all the best practices and then standing
up the team for us.
And again, it is the people who created the original policy went right up to the
leadership of the company to say, these are the rules for rolling out an AI tool
that has an impact on a human being.
And then created a kind of a gate in our rollout processes to
to try to explain it without too much jargon, that says, is this an AI tool?
Does it impact humans?
If so, you've got to be reviewed and answer some questions and engage with the
Digital Values Team.
That's the name of the team.
And again, that team includes, it is led right now by a data scientist, and they
will kind of probe and scrutinize the algorithm itself, the purpose for the
tool, who's rolling out, who is the responsible kind of business owner of it.
and then schedule, not just do the initial scrutiny, but schedule kind of follow up
monitoring and oversight.
So kind of all the tools of a traditional compliance program, but applied to this
new technology.
And so far, so good.
And we've worked with quite a few teams.
I don't have an exact number of how many tools they've scrutinized, but the team is
busy and it's growing.
So I think the metric is, you know, as we do more in this area, they have more work
to do.
But the biggest, I think the biggest challenge and the biggest opportunity is
really one of education, of getting out and making sure that all of our
technologists understand what's expected of them and how we expect them to think
about the tools that they're deploying from the beginning.
So, you know, if we educate everybody, actually it will cut down on the work for
this team, but all to the good because there's plenty of other work to do.
And I really want...
each of our associates to think of themselves as a responsible user of the
technology and data.
This is always a challenge with compliance though.
How do you convince them that this is valuable and it's not going to slow down
what they're trying to do?
Such a good question.
And it's funny because I don't say I bristle at those questions.
It's always been a perennial, like is privacy getting in the way of productivity
or of profits or whatever.
We really do believe in customer trust and associate trust and putting people first.
So I really posited as an issue of trust and responsibility and that that engenders
a better relationship with.
the company and that we are in a competitive environment where this is a
benefit.
It's considered a good thing, I think, by our customers.
We see more and more educated customers about technology and asking the questions
about how it's being used.
So I really don't try.
I don't think of it as as antithetical, but rather as kind of enhancing our
relationship with our customers.
And we do try to be lean.
Obviously, we try to be really efficient and not
every AI tool that's just deciding the example I always use and it is a real one
is, are the bananas on the conveyor belt right?
We've got AI that looks at fruit and vegetables and meat and things and says,
based on what it has learned over the last how many years and what a good thing,
right?
We want good quality food to be eaten and safe and that sort of thing.
That we do not consider, it's impactful to humans obviously indirectly, but we're not
making judgments about humans.
They do not need to go through the digital.
value scrutiny.
But I think frankly, in my experience, most people want to do the right thing.
They're not always sure how their job, you know, affects that right thing and the
service to the customer and associate.
So it's a big company.
And so education and outreach across multiple languages, multiple parts of the
world, multiple job descriptions, that to me is an exciting educational challenge.
Yeah, I mean, it sounds like you've got a screen that is similar to what many of the
governments are coming up with in terms of there's high -risk AI or whatever it's
described at.
This is what we really need to worry about.
Then there's all kinds of other uses of AI that maybe don't need the same scrutiny.
I hope people do look at it as a spectrum that not everything deserves or merits or
needs the same level of scrutiny, but some of it very much does.
And in the corporate world, the impactful decision -making that we would have would
obviously be in hiring, advancement, people's rights and responsibilities as an
employee.
In the customer world,
it's actually been even more fun because it's things like our AI chat bots and
putting together different fashion pieces on the website and saying, you bought a
red sweater, maybe this blue skirt goes with it.
This is fun.
Obviously, we want to be careful.
We don't want to get it right.
But it is not a consequential decision, depending on where you're going that day,
whether the skirt matches the sweater.
But again, of a different ilk than financial or career or other things.
How does the team stay up to date with the fast changing nature of technology in this
area?
That is such a great question.
And it is, I think, a constant part of everyone's job to be reading, to be
engaged.
We are members of quite a few, not only professional trade associations in retail,
in law, and that sort of thing.
I do find the IAPP, the International Association of Privacy Professionals, to
be increasingly aware.
And they just today put out a big report on AI governance.
and we're members of something called the Data and Trust Alliance, which was out
front very early on.
We were one of the founding members in looking at cross -industry guidance.
So it's not just one area of industry, not just one size of institution, but how do
we make governance real for all companies?
And in fact, we, as well as many small and medium enterprises, sometimes are
procuring things from third parties, from vendors.
And...
just like every other new technology, we see a flourishing of vendors coming in and
saying, this is the key thing, this is the perfect tool for everything, solution for
everything you've ever needed.
And many people don't know how to scrutinize that, right?
And we've had salespeople in our office saying, we can't really disclose the
algorithm and we can't really explain how it works.
And we're like, well, come back when you can, because that's not gonna be good
enough for our customers or associates.
So.
So rules around how to engage with vendors and how to scrutinize those tools as well
as the ones you're building internally I think are really important and that's a
role that DTA has played.
Do you have benchmarks or goals or a way to assess whether you're being successful
in these efforts?
I think benchmarking, especially when it's in the policy world, is hard.
But certainly some of our benchmarking is how many tools have gone through, how many
secondary reviews we've done.
Customer complaints are also a metric if things are making the wrong decision or
the right decision.
And so that's what we're looking at right now.
But if you've got any ideas, I'm all ears.
Well, no, it's an interesting challenge because on the one hand, there's value in
coordination and standardization.
And on the other hand, what makes sense for an organization at the scale of
Walmart is not going to be at the right thing for a startup or for a company
that's in a completely different industry.
And so that's why I'm curious about how different organizations think about and
see progress in those areas.
Yeah, that's such a good point.
I think there's some core values that probably cut across in terms of
transparency and kind of fit for purpose or, you know, scrutinizing high risk tools
more than others.
But we recognize also the place of privilege that I sit in and being able to
stand up an entire team to think about and then go do and create a whole new
compliance program around this.
And so well aware.
And that's why DTA set about creating kind of products and workbooks and things for
smaller medium enterprises that.
They're kind of off the shelf guidance as well.
Are there any major gaps that you see either in terms of what resources or tools
are available to you or in any other area?
I think we can always do more.
Again, back to the comment about governing the ungovernable, we can all do more to
really think about how to automate and really probe the tools in a consistent
fashion.
And like we were saying before, there's obviously a lot of concern around bias and
about accuracy.
But how do you prove that it is doing the thing that it's supposed to do and that
it's not doing the thing it's not supposed to do?
There's not a lot of standardization in that yet.
And so standard setting bodies, where are you?
I'm sure someone's working on a framework for us to all follow at some point.
And I know you and I were going to talk about legislation and government
interaction.
I get very concerned when I think government actors are saying, we have to
regulate this one thing.
Again, as we were saying, AI is, you know.
it's like the air or the, you know, the sidewalks.
It's just a tool and a platform for doing something else.
And I'm usually the one saying, is there a law that prevents this already?
You know, is there something on the books that simply hasn't been enforced in this
area or hasn't been applied to tools like this yet?
If it's really, really that new, you know, certainly it may need some level of
regulation or legislation.
But I'm usually the skeptic saying,
We may have rules about this already, about being unfair to people based on race
or gender.
And let's look at what's on the books already.
Yeah, I mean, that's a big challenge that we're seeing a lot where I talk to lots of
people who say, well, companies can't be trusted and their incentives are not
aligned.
And we know this technology is so new.
So how do you convince people that the resources and the mechanisms are there to
address the problems that arise?
I certainly hear the same things and I certainly know that big companies are just
whatever, big companies are made out of people, right?
And so...
It's not been my experience, not here and not in other big and small places I've
worked that people want to do the wrong thing.
In fact, I think our incentives are entirely aligned because if we lose the
trust of our customers, if we lose the trust of regulators in the countries in
which we do business, then the consequences are much greater than just
failing in one program or product.
It's a...
you know, an enterprise wide impact.
So I, again, I'm skeptical.
I understand it's good to have a healthy skepticism of large institutions in
general.
And I, and I respect that.
And that's why part of back to why we came out with the AI pledge is like, that is
something that we feel is worth talking about that we are committed to walking our
values forward in the digital world, in this tool, in this setting and in others.
And if people feel like we're not doing it.
They should let me know, they should let our CEO know, and they do occasionally,
including some of our own associates, which I actually love.
I love it when people raise their hand and say, this doesn't feel quite right to me.
And I've got a letter once from a pharmacist in Michigan, and he said, I
don't like that policy.
And it actually started a whole review from the ground up of how we were dealing
with a particular issue.
So I think we try to be very responsive.
Something I've tried to talk about within the company is,
and all of the good governance we've put in on these tools and others, mistakes may
still happen.
That's the really big challenge I find, especially for a company that prides
itself on its reputation and doing things right.
It's not an environment where we'd like to fail fast and move on.
We really do kind of bet our reputation on a lot of things.
And what I try to encourage my team and others in leadership is to realize that...
If a mistake happens, we're going to have to work hard and fast to fix it.
But I think people will give us grace if we are transparent about it.
If we say, this is what happened.
Here's what we're doing to fix this incident.
And here's what we're putting in place to have it not happen again.
So especially in kind of generative AI, when it moves and talks and it moves fast,
we need to be ready to both be humble and be nimble in that area.
Yeah, I mean, one of the challenges with generative AI is what you said touches so
many people that now it's not just the data scientists who are building things.
What does trust really mean to you in that context?
I think it's a really important and valuable point that this is all about
creating and promoting trust.
But what does that actually boil down to?
On the website, it means we've correctly described a product.
Say you've got a child who's allergic to peanut butter, you better be sure that the
product you've bought does not have peanut butter in it.
That's really important to us, and that was important to us before there was
generative AI.
But now if you have generative AI creating product descriptions, you've got to make
sure they're right.
And as we know, generative AI is not always right.
So in some tools we're just about to put out in the...
We used outside consultants to red team, to probe it, to try to crash it, and
things still do go wrong, but I think putting in the effort before it goes live
to really try to stress test it and then to stay, again, really vigilant about how
the AI might change is essential.
I feel, again, a deep sense of responsibility, not just because it's the
catchphrase of our team, but I think...
all kind of purveyors of technology should be thinking about the impact it's having
on the world and on the people it comes into contact with.
And going forward, are there either developments that you're looking to or
anything that you see coming in the next few years that are going to be
particularly important in this area?
Well, I can tell you the things I'm the most worried about, unfortunately, it's
not the way to start to answer that question.
But the things that I worry about are the appearance of accuracy and truth when it
is not there.
It's too cute to say fake news, but just simply the authoritative answers that
often are provided by these tools that are simply incorrect.
Do all of our citizens in this country or elsewhere?
have the ability to be good consumers of this kind of media savvy or media, you
know, digital kind of citizenship in the true sense of can we consume and analyze
information for its content and for its provenance?
And do we have the tools to help us do that?
I saw a great study years ago about kind of government issued propaganda and
Americans were more likely to click on something that was, it kind of seemed to
be true, but it was a little bit off than many parts of Eastern Europe.
And I thought, wow, that kind of shocks me.
You know, we have free press and we have all this stuff.
And it was because they had lived with propaganda, perhaps in more recent times
than we had.
And I thought, well, how do we get to, you know, a place where we have really
empowered?
Internet users of all ages, shapes, sizes, everything to scrutinize the information
or perhaps have penalties for people who put out things knowingly that are
incorrect.
That's to me one of the hardest questions for the Internet community, for the world,
and one of the consequences of gendered AI that worries me the most.
The other one is our kids using it to write term papers.
But that's me.
The exciting thing to me is always, not just the general utility for human beings,
but I have a son who's on the spectrum.
We have lots of associates who are disabled or in some way we have many
customers.
We've just started, you may have seen quiet hours in the stores where every
morning for two hours, everything in the sound wise and the lights are lowered.
And it's been a real, it's been a joy to watch that evolve and people really get
excited about it.
I always call it kind of the ancillary effects as well.
People of all kinds who may not fit in its traditional kind of disability community
have said, some people just like it quiet.
My husband says I would much rather shop when it's quiet.
And whatever.
And so the thing I'm the most excited about, not just about AI and gender AI,
but technology is the ability to assist humans who are physically, cognitively
otherwise.
diverse or challenged in some way.
And Gen .AI, especially in voice enabled, is just a boon for many, many people.
And I'm so excited about that.
What that means for our workforce, enabling more people to join our
workforce, generally join the workforce, and enabling customers to find the
information they want and the goods and services they want.
I think that's one of the best and highest uses of technology.
No question.
No, it's all tremendously exciting and it's sort of the paradoxes in and of this
technology that what makes it exciting is also what in some ways makes it scary.
I mean, you talked about sort of two sides of citizenship in both the corporate and
the public governmental context where this technology can be so great at
personalizing, but that can also be abused.
So true, so true.
And transparency is necessary but not sufficient answer.
Absolutely.
Nuala, really great to talk to you.
Thanks so much for sharing your experiences and what you're doing.
Delighted to be here.
Thanks for having me.