Welcome, Scott.
FICO was in some ways one of the original AI companies, at least in the sense of
business analytics, going back to the invention of the credit score back in the
50s.
How does that long history influence the way that today you think about deploying
AI?
Yeah, thank you, Kevin.
You know, it's part of our legacy, right?
We've always focused on having a responsible view on how we use analytics.
Originally, back in the 50s, right, we decided that, you know, decisions could be
made more ethically and more fairly through the use of mathematics, right?
And that started what was, you know, the use of analytics and credit scoring, which
evolved to the PICO credit score.
which later on in the nineties, we also branched into AI with our fraud solutions.
And so we've always had at the center of our mind a focus on the fact that we
develop models that impact people.
And therefore those models, whether they be AI or something else, right?
Or machine learning or something else needs to be built carefully and
responsibly and reproducibly and with a level of accountability around it.
And so that's been our big focus for us.
given the gravitas to some of the decisions that get made with these types
of analytics and models that we develop at FICO.
So is it really from the consequences because many technologists for a long time
said, well, this is just math, it's neutral, it's not biased.
And we'll get into that.
I think nowadays there's a lot more recognition of the problems.
But what got the organization, what got you personally early on to put such an
emphasis on these questions of ethics and responsibility?
So in the early days of credit risk, and it still prevails today, there are a lot
of challenges with respect to discriminatory usages of models and of
lending.
And so back in the, before we used math and models and making decisions about
credit, whether you got a loan or not could be based on whether or not we went
to the same church, did we go to the right, live in the right part of town,
right?
And these are kind of very subjective sort of measures.
And so it was very, you know, from the very get -go and the launch of the way we
developed analytics, we understood that we were actually combating a problem, which
is, you know, the data back then and in the data even today is full of bias.
And therefore, you know, we go in with this view that data is dirty, data is
bias.
And so when we develop a model, we have to do that in such a way that we can, you
know,
fared out where those biases could be, remove them and build the most fair and
ethical model possible.
So I think, you know, that was really part of our origin story, if you will, in terms
of why math would be more empirical, more fair, right?
We still have challenges today and how math gets used, right?
How these scores get used.
And that's a big part of the infrastructure, right?
Of how decisions get made.
But we wanted to make sure that the models or scores that are used,
to help aid in making fair decisions are built to the highest level of standard
possible from an ethics and fairness perspective.
challenge today as the models have gotten so much more sophisticated and we moved to
AI and deep learning and it's not as clear even how you get from the inputs to the
outputs.
So I think you've hit on it exactly right.
I mean, one of the biggest sort of challenges we have today that wasn't the
case when I started here at the company 25 years ago is that we somehow all accept
that, you know, AI is used to make all these decisions and the models have become
more and more complex.
There've been more and more parameters.
There've been people enamored with how quickly they can build models and how
large these models can be.
And they don't focus on the physics.
Like I'm a PhD physicist and I think about physics, right?
And I think the world is defined by, you know, laws of physics and types of
behaviors that are relatively simple, right?
So, you know, we don't need models with hundreds of thousands or millions of
parameters to make it a credit decision, right?
What we need is we need level of transparency in terms of the types of
models.
But...
You know, as an, as an industry, right.
we, we've kind of adopted these big models and somehow bought into bigger is better
without that and sacrificing the ability to actually understand what drives those
outcomes.
Now, if you look at things like, you know, the FICO score that we produce, we still
use a generative additive models for that.
And primarily we do that because when we, let's say, Kevin, give you a decision,
you, you, a decision gets made based on a score and you say, well, I don't
understand why that decision was made.
We need to be tremendously factual in terms of what are the reasons or why you
got the better score or the worst score.
Cause you, you're going to need to act on that, right?
Because you're still going to need to get that loan.
And so for us, it's not just like, okay, yes, let's use explainable AI, which I
like to talk about as being good, bad, and ugly with most of it being bad and ugly,
right?
Let's focus on interpretability as job one.
Because if I tell you, these are the three reasons why you didn't get the great.
the best score, then you could go and act on them.
And if you did that, right, we would see that consequential sort of improvement in
the score.
And that's the level of sort of visibility, which is these scores are not
just meant to block, right?
They're meant to assist and provide understanding of what the outcome was and
maybe how you improve that outcome in the future.
Clearly that matters in credit scoring where as you're talking about their human
consequences and it's in many cases regulated.
Do you think that in general that companies have gone too far with the big
non -explainable models?
What if you're using AI for something like sales forecasting?
Say, does it really matter if you can explain what the rationale was?
I think it still does.
I think you have to look at the consequence of the decision.
So if we're going to produce a picture of a five -legged elephant with generative
AI, it probably doesn't matter.
That's art.
And if you don't like the picture, we throw it away.
There's no consequence there.
But if I'm going to do a sales forecast, and I have shareholders that are impacted,
and I have employees that are impacted, I have bonuses that are impacted, then
that's a very consequential decision and one where, at the very least,
you know, I should have understanding and confidence in, you know, how the analytic
reached an outcome, right, or a recommendation.
And so I'd say, you know, that this sort of concept on models that are more
interpretable and maybe simpler so that we understand and we feel comfortable in the
assumptions that went into those models or what was learned by those models to make
the decision.
But we understand that so we can have confidence in the models, right?
One of the things that I think about, Kevin, is like over the last year, right,
a customer,
confidence in models, AI in particular, has gone down considerably.
Consumers, like 60 % roughly, depending on which survey, have a lot of concern about
use of AI.
Businesses themselves have concerns about using AI because they don't feel like they
have it under control, right?
And even in the financial sector, same thing.
So I think, you know, we are at a sort of like tipping point where folks are going
to say, well, that's all cool that we can, you know, use, you know, use a thousand
GPUs to create a model, but...
you know, do we need to and what do we lose?
Do we lose our ability to actually interpret or have trust in the model?
And I think a lot of people run into that, even if it's not a regulated environment
that they are sort of contemplating the use of.
So how should an organization think about the trade -offs in terms of, you know,
they adopt a large language model or something that's not today particularly
interpretable versus the alternatives?
Yeah.
So, you know, I think one needs to figure out what is the best tool for the job.
Right.
And so, you know, I've had this conversation with many executives where,
you know, I try to explain to them that, you know, there are different AIs for
different purposes.
Right.
And, you know, in general, right, we should use the simplest and minimalistic
solution, right, to, to.
to solve a problem, right?
So if I can solve the problem with a relatively simple interpretable model,
right?
Or even a linear model, maybe that's the best thing because I want to provide a lot
of confidence in that decision and I need that level of transparency, right?
If you really need to goose up the performance, right?
The detection capabilities, then maybe you get more complicated and you'll have like
an interpretable neural network, which is something that we focus on, which allows
that level of learning those imputed relationships that drive.
decisions but can still be interpreted.
I would argue there's very few use cases probably where, at least in business
context, that you have to go to completely unexplainable models.
Even if you look at interpreting text and classifying text, you can do that without
full generative AI models that hallucinate and that are unexplainable.
And so I think just stepping back and making sure that...
you say, what is the minimalistic sort of solution that would solve this business
problem and treat it as a business problem, not as a science problem where
you're interested in science.
We're all interested in cool science.
But if we're asking a business problem, people will say, where can I get to 85,
90, 95 % of a solution and still have an ability to rationalize this, explain it to
my board, explain it to my manager and my governance team.
And as soon as we put on our business hats,
You guys very, you know, the worst versus reward is very much like a typical CRO
sort of conversation that you'd have in terms of what risks you're willing to take
and, you know, getting that extra, you know, a little bit of detection, but
potentially, let's say, discriminating against a group of individuals, not a
great trade off, right?
And that's where I think some of those decisions come back down to just making
those solid business decisions.
What are some of the things that you've done at FICO to make your AI systems more
accountable?
So what we focused on is establishing what is our model governance standards at the
company.
So we have a firm belief that, and we use the term responsible AI versus
accountable.
I like the term accountable AI, but in my mind, accountable AI kind of implies that
I'm accountable for the wrongs that I've done.
And that's true, right?
But I would rather have a responsible strategy that says this is the right way
or this is the company way of doing it.
So at the very...
start of it is a model governance document that model development governance standard
for how all models get developed, right?
Because if I have, you know, 200 or 300, 400 data scientists, I can't have them all
making decisions around how to, you know, choose an algorithm or what's the right
way to explain a model or how do we deal with ethics, right?
So we have these are the models you're allowed to use.
This is how we explain the models.
This is how we do the ethics.
And so,
The first step is having that model development standard that everyone
follows.
The second major step then is having an enforcement tool.
And it sounds a little bit like we're restricting creativity of data scientists.
And we are to a certain extent because customers and our customers' customers are
not lab experiments.
We need to make sure that we are using technology that is proven and understood.
And so we enforce that today through the use of blockchain, right?
Where decisions get made that where we show that like these are the 27 things you
need to show done as part of the model development standard, you know, and then
have a proof of record that each were done, tested, and then approved.
And that allows us to release models that meet a model development standard.
And, you know, the consequences are great, right?
You know, we'll...
One major sort of advantage is that everyone follows the same standard, right?
And so you have less errors and issues when models get released.
The other major piece is all that energy that people put into like governing
models, right?
Where there might be thousands of models in a thousand different conversations all
get funneled down into let's improve the model development standard, right?
And then use things like blockchain to enforce.
So it's really gotten us much more tactical.
And actually much more efficient in how we develop high quality models at scale in an
understandable, predictable way.
And then we focus our, you know, back to the data science creativity, we focus our
innovation on improving the standard, right?
And then, you know, then we treat your productions, developing models as a
production process off of that standard.
So let me unpack those two pieces.
You talked about, you start with the standards and the ethics.
So where do those come from?
Yeah, so the standards are ones that are typically defined by someone like myself
within an organization that has like a broad purview of the problems that we
solve, that the pluses and minuses of different algorithms and then, you know,
making them hard decisions, right?
There's certain algorithms that we just do not allow to be used at FICO, right?
Because, you know, they're not deemed explainable or they're not deemed ethical
or, you know, or they have so many parameters that just...
science would dictate that you can't collect enough data to actually specify
all those parameters properly and therefore you have low confidence in
models.
And so, I'm part of that, part of that would be our legal teams and ethics people
and also business owners, right?
That it might have very, very keen views on how regulated AI specifies that certain
regulation needs to be met.
And so that all gets folded in into that model development standard.
It gets kind of...
through that sort of set of stakeholders and then wants to find, right, then it's
operationalized.
And then on the second piece, why blockchain?
And I actually do a lot of work on blockchain and digital assets.
So I was very excited to hear about the work you're doing there and understand
you've gotten patents and awards for it, but can you expand a little bit more about
what the value is of using a distributed ledger for that kind of audit trail?
Yeah, so we started this responsible AI journey a very long time ago.
And we started with this concept of what's we call analytic tracking documents.
And I treated it as a contract back then.
And it was a contract between myself and the business owners and my scientists.
And there were named resources and tasks that needed to be done.
And we would enforce it through weekly stand -ups, sprint reviews, agile
development processes.
And we made a lot of progress there, but things could slip through the cracks,
right?
Because if I didn't ask the right questions or if something wasn't presented
sufficiently, right?
We would basically go through that process in a human way to ensure that it's
enforced.
With the blockchain though, right?
We're able to automate all of that, right?
And the reason why we use the blockchain is number one,
we have that immutable chain, right?
Cause I want to see the history of what we did well and what we struggled with,
right?
And maybe, you know, I completed a task and one of my scientists tested it and
said, no, Scott, you're wrong.
You didn't meet that task.
I want that on the blockchain.
And so the immutability means that people take things seriously.
So back 15 years ago, when they actually had a sign, a piece of paper with me, like
a little mini mock contract, when you put your signature on a piece of paper still
means something, right?
Same with the blockchain.
As soon as I commit it as done on the blockchain, it's there, it's immutable.
I take it seriously, right?
And then, you know, the people after me, they're checking that work and verifying
that work.
Their names are on the ledger, right?
And they're taking it seriously.
And so it provides a sort of level of accountability because we know that it's
going to be on that ledger.
We know that it's immutable and that we're not going to be able to change it.
And it also establishes tremendous trust, right?
Because I can open that distributed ledger or private ledger to regulators, to
governance teams where they see exactly what was there, right?
And if Scott was particularly bad on a task and he messed it up seven times and
seven times his data scientists rejected it, right?
And maybe eventually Scott was taken off that task and was replaced by Julie,
that's fine, right?
But that is the level of sort of...
clarity and transparency and seriousness around meeting that standard.
And if you look at regulators, they don't have that level of transparency.
So it drives the right behaviors in a data science program.
It also provides that level of trust when it comes to explaining how the model is
built, because there's no disagreement.
There's no like, well, I think it does this.
If we say, for example, and I say, well, these are the latent features that are
learned, they're persistent in the blockchain.
There's no...
I think it is, or I thought it might be, no, if the requirement is I have to tell
you what all the latent features are and what the relationships that were learned,
that's on the blockchain.
And if later on, or a governance person or ethics person, lawyer, regulator wants to
look at that and they say, ooh, we don't like that latent feature, that's fine.
That's the conversation that needs to be had, right?
And so that's one of the reasons, but it's enforcement, it's accountability.
And frankly, you know, it's...
you'd think data scientists wouldn't like it, but it actually provides tremendous
comfort because unfortunately most data scientists are put in situations where
they're ill -equipped to take on that level of responsibility.
So it also protects them.
So it's a sort of happy harmony of kind of benefits.
And then, you know, frankly, after that, when the model gets released, it only gets
released when all those requirements are met and accepted and you have that whole
history.
That blockchain goes with the model to where it gets deployed.
And it's also the thing that lives with the model so that later on a year from
now, two years from now, when there's questions about what that model does and
who built it, why they built it, why is it behaving this way?
You can go look into that blockchain, understand what's inside the model.
And moreover, today we actually put in the entire requirements on how you monitor it,
right?
Which is another big issue in AI, right?
How do you monitor these models?
And so the only person that really knows how to do that is the person who built it.
Might as well codify that at the time it was built and say, these are the operating
parameters for my model.
And then you run it in production.
It sounds like a lot of overhead though.
How do you build all that into the workflows?
So there is work.
And so we have a UI that we essentially codify all those requirements.
And so as part of the model development process, they have to list that in.
I don't think it's any incremental effort.
It's more about organizing.
So you think about my desk is actually, you can't see it, but it's a big mess.
And so like,
It's very cluttered, right?
And there's order that I understand, but I don't think you'd understand necessarily
unless you're as cluttered as I am.
So we're structuring the work in that, that the requirements get met.
There is overhead, but without that, we'd go have to go back and maybe answer
questions that should have been answered, you know, a month ago or six weeks ago.
And so we actually find that it's more efficient because, you know, we've
actually codified exactly what.
you know, a governance team will be looking for exactly how we're going to
report the reasons for how a model behaved.
And so, you know, there's more work upfront to organize that work into, you
know, a structured way or on that model governance standard, but there's so much
less work after, right, or during the model build because you've already
satisfied all the requirements.
You know, one of the things that's interesting to me is there are still a lot
of organizations that have model governance teams, right?
And, you know, model governance team,
generally operates that I build the model, I throw it over to you, you don't know
anything about the model, you start asking questions, right?
The questions may or may not be relevant, you may or may not have the same level of
expertise in what I've done.
These governance processes can take nine months, right?
I would rather say when the model is released, you and I, if you're the
governance person Kevin, you and I are agreed on what you need to see.
And as proof of work, when that model is released, it's shown.
You can go look at the blockchain.
And so in some sense, it drives all that governance into the model development
process.
So it's not an entirely different process that oftentimes takes twice as long or
three times as long as actually developing the model in the first place.
Yeah, do you think there will be more pressure for that kind of approach?
Certainly in banking, we have model risk management standards and those kinds of
requirements.
But as we talk now, especially with generative AI, where you've got companies
that are building the models, other companies that are adapting them to
applications, and then companies are taking those, are doing the rag chain and
so forth.
Now you have a lot more organizations that are somehow touching or dependent on the
model, and then something goes wrong.
Yeah.
is that this kind of audit trail you think going to be ultimately what organizations
need to go to?
I think it most absolutely will be.
And so you look at the work that people do in LLMs.
I look at different proof of concepts that people want to do.
And we obviously do proof of concepts here at FICO.
We also do core research in this area.
We've been doing it for 20 years in generative AI.
But there's a lot of imprecision in what people understand.
For example, like RAG.
You talk about this augmentation.
But like,
you know, what, how, what's the parameter on what gets used, what doesn't get used?
How do you do the embedding?
What, you know, what's the chunking?
What's the overlap?
These all really, really matter from a business context perspective, but a lot of
it's just kind of, you know, glossed over, right.
In terms of looking at the outputs.
And I think, you know, if people are going to go leverage some of these technologies
that are a little bit more on the fringe and in terms of responsible AI or, then.
for sure they're gonna have to keep track of these as decisions, right?
So they can revisit it down the road if they find unsatisfactory results in the
use of it.
And I think that'll be a big part of where people need to get to.
Otherwise, you'll have a model and even simple models are really hard to
understand.
So these large applications of generative AI most definitely will have to have a lot
of structure and just picking apart things like RAG.
There's a lot, a lot of decisions.
And then there's a lot of things that, you know, people take for granted that they
don't truly understand how it works.
Right.
And so from that perspective, just codifying that as a risk or delving in to
where they will use, let's say, you know, a private LLM sort of capabilities versus
building their own will be critically important in terms of where they can, you
know, faithfully say, listen,
Yeah, we only use our documentation answering the question or maybe we don't.
Maybe there is hallucination going on there.
And I think some of those things have to be codified so that there's some
governance around.
Are there areas where you've changed your mind or you've realized that you were
going in the wrong direction in terms of governance and responsibility in recent
years?
Well, that's a great question.
I think it's an evolving process.
I'll put it that way.
I think with blockchain, I'm very confident that's the direction forward.
In fact, we're so confident at FICO, we have this AI decisioning platform, which
was a leader in a Forrester wave from 2023.
And we're using blockchain beyond just models.
We're using it to kind of keep track of the entire decision process.
And so,
I think this blockchain was a great sort of approach.
I think, to your point about what is the right sort of heftiness of a process, this
is still a refinement for us in terms of the pluses and minuses.
So I don't think there's anything where we backed away and said, let's take an
entirely different direction.
But what we have tried to do is try to figure out how to make it as...
less burdensome as possible.
That's why, you know, a big part of it, the challenge wasn't a model development
center for FICO, because that's part of our business and our challenge wasn't
blockchain technology, you know, that's one you can get your hands around.
You know, our challenge was like, you know, how to come up with a UI so the data
scientists don't spend a lot of time having to worry about that.
And we did that, right?
And so that's one of my core focuses is like,
Okay, you know, making sure that that user experience for a data scientist is good.
I don't like if I have a world -class data scientist, I don't need her being a world
-class blockchain sort of, you know, whisperer either, right?
I mean, and this is where, you know, most machine learning, you know, teams and AI
teams today that build models have these areas of specialty.
And I think that's been one of the major learnings is just no one human being can.
can encompass all that in one head, right?
And so figuring out what that MLOps needs to look like, and nowadays maybe,
regulated AI ops, if you will, right, is really important.
What are you hearing from customers or when you go out there and talk to people
about their thoughts on responsibility in AI?
Yeah.
So the customers are really intrigued by the use of, let's say, the audible AI
concepts that we have and blockchain.
They're under pressure.
More than 50 % of, in one of our survey we did a couple of years back, 50 % of
financial institutions in the United States don't have model standards.
And roughly 60 % of banks,
you know, say that they're very concerned, they don't have a handle on how to control
bias and, you know, and fairness and development of models when it comes to AI.
And so, you know, we see customers that are a little bit nervous about their
abilities to do this in a very tactical way and in a very certain way.
In addition to that, right, they see this as one way that they're going to have to,
you know, respond to upcoming AI regulation, right?
We saw the EU AI Act.
is marching ahead, right?
Many organizations are starting to already codify how they're going to respond to
that AI regulation, right?
Which comes down to, let's go through all the little bullet points of things that we
need to demonstrate, right?
And then how do we do that, right?
And I think they recognize for themselves, particularly if they don't have a model
development standard, that it's all over the place.
And so they find this as one way to kind of, you know, herd cats, if you will, and
get more efficient.
And I think organizations that do that will find benefits.
It might be a little bit of time to do all that work, but frankly, where AI
regulation is going today and consumer trust is heading right now, it's a
necessary sort of task that people need to do.
But once they do it, you'll have teams that are supercharged at doing it and
doing it very efficiently.
And...
You know, I think that's where you start to get these multiples of, you know,
productivity, just because people are so used to doing that versus you and I
sitting there and looking at a particular type of model and figuring out how the
hell we're going to explain this thing.
Right.
And then, you know, being able to sleep at night and make sure that we were able to
do it, right.
Is a challenge versus, you know, having a tried and true proven method.
Right.
Again, right.
When we saw business problems and that's what a lot of our customers do, right.
It's not about.
algorithm A versus algorithm B, it's about outcomes.
And so, you know, them just having that path is really important.
And then being able to show that they follow the path appropriately and defend
it, super important to them.
What do you think is the biggest challenge going forward?
I think one of the biggest challenges is the AI regulation that's coming.
So when generative AI, well, the large language model thing popped up, it took
almost all the oxygen out of the room.
And I was involved in so many conversations that were just utter
nonsense about, we don't need data scientists anymore.
And these things are super intelligent.
And I think now, right?
we have gotten to a more sensible place where people understand that, you know,
these large language models are cool and they're fun and they're interesting,
right?
They're also not meant for many regulated environments.
They also predict the next token in a sequence of tokens.
They're not some Uber intelligence, right?
Even if we feel that they are based on how they talk to us, communicate.
So we are all smartening up and letting the hype cycle die down.
But we lost like a year and a half or two years on responsible AI journeys.
So like when I mentioned earlier,
you know, about 50 % of organizations don't have an AI governance standard at
all, right?
Which means Jack and Jill are building models entirely differently at their
organizations.
And there might be hundreds or thousands of Jacks and Jill's, right?
That's a huge problem when regulation is going to insist that you write a report
for every single model and you can justify that, you know, you follow the
regulations, let's say.
And I think this is our biggest sort of challenge.
In some sense, it's a positive because responsible AI will get, have to get real
really, really quickly.
But for those laggards, right, in defining those standards, there's going to be a lot
of work to do, which means they need to figure out who in the industry is leading
in those spaces and try to mirror some of that so they can kind of catch up and
define those governance standards themselves for how they develop and
operationalize models.
So that's interesting.
You think that all the excitement around LLMs and Gen .AI got organizations to take
their eye off the ball somewhat in terms of responsibility?
Absolutely.
Yeah.
And frankly, right.
The general, all that LM gender of AI, you know, height also caused regulation to
speed up.
Right.
And so like, like the Biden administration and putting out that their AI guidance,
right.
A lot of it was predicated not on, you know, using AI and machine learning to
make, let's say a credit risk decision.
And a lot of it was based on.
Wow, look at all these LMX, you know, live experiments on consumers, right?
Where organizations, including the organizations that, you know, big
organizations that build these models can't explain it.
And I think that created a lot of that trust issue and it also created a lot of
concern in the regulatory environment.
So I think, you know, it most definitely people took their eye off the ball, but it
also raised this sort of heightened concern about, whoa, which direction are
we going here?
Right?
And, you know,
That's why like if you look at the EU regulation in detail, right, you got a
very high risk and high risk and different classifications of AI for the purposes of
just saying, okay, you know, there are some types of models where we're going to
have to be much more pragmatic and it might take three, four months to build
that model with two or three people, because we're going to build it very, very
carefully, right?
It's not about spin the model on the set of GPUs and be done with it, right?
And I think...
I think that's one of the challenges that we've had with, with generative AI, but I
do think it's all coming around and they'll, you know, if regulation can use
good direction, it is, there'll be no choice, right?
You'll have to kind of catch up.
but, but most definitely, right.
Those were not the conversations for, for many different organizations.
And it has been frustrating for some where they see that, you know, people are more
enamored with what a generative AI will produce than trying to, you know,
properly developed models in a trustworthy and accountable way and having some of
those hard decisions about a model of illness standard.
But I'm glad to hear that after all of that, you're still optimistic, it sounds
like, that organizations are going to get to where they need to go.
So many important issues.
It's really been a pleasure speaking with you, Scott.
Thank you so much.
pleasure Kevin, thank you.