1
00:00:05,752 --> 00:00:07,796
[SPEAKER_07]: Welcome to the Analytics Power Hour.

2
00:00:08,478 --> 00:00:12,727
[SPEAKER_07]: Analytics topics covered conversationally and sometimes with explicit language.

3
00:00:15,152 --> 00:00:15,693
[SPEAKER_02]: Hi, everyone.

4
00:00:15,713 --> 00:00:16,094
[SPEAKER_02]: Welcome.

5
00:00:16,314 --> 00:00:17,517
[SPEAKER_02]: It's the Analytics Power Hour.

6
00:00:17,577 --> 00:00:19,862
[SPEAKER_02]: This is episode 288.

7
00:00:20,551 --> 00:00:26,822
[SPEAKER_02]: We spent the last decade putting walls around our data, securing it, governing it, putting labels on it.

8
00:00:27,143 --> 00:00:30,629
[SPEAKER_02]: And now the AI revolution walks up and is like, hey, can I see all that?

9
00:00:31,731 --> 00:00:35,838
[SPEAKER_02]: Today, we're going to discuss Model Context Protocol, or MCP.

10
00:00:36,319 --> 00:00:37,561
[SPEAKER_02]: I mean, it's an open standard.

11
00:00:37,742 --> 00:00:43,572
[SPEAKER_02]: It promises to stop all the copy, paste madness and let AI talk directly to your data systems.

12
00:00:43,552 --> 00:00:47,517
[SPEAKER_02]: Is it the end of the data silo or just the beginning of a new governance headache?

13
00:00:48,078 --> 00:00:53,765
[SPEAKER_02]: Well, we're going to try to establish an MVP for PMF of MCP all in one hectic hour.

14
00:00:53,806 --> 00:00:57,130
[SPEAKER_02]: All right, let me introduce my co-host, Val Kroll.

15
00:00:57,150 --> 00:00:57,831
[SPEAKER_02]: How are you doing?

16
00:00:59,113 --> 00:00:59,713
[SPEAKER_02]: I'm good.

17
00:01:00,554 --> 00:01:03,298
[SPEAKER_02]: This is going to be an interesting one, yeah.

18
00:01:03,278 --> 00:01:06,183
[SPEAKER_02]: All my acronyms, yeah, that was fun.

19
00:01:06,203 --> 00:01:08,126
[SPEAKER_02]: All right, Tim Wilson, always a pleasure.

20
00:01:09,148 --> 00:01:09,669
[SPEAKER_02]: Likewise.

21
00:01:10,270 --> 00:01:12,674
[SPEAKER_02]: All right, I just used the acronyms to make myself sound smart.

22
00:01:12,714 --> 00:01:13,134
[SPEAKER_02]: That's all.

23
00:01:13,836 --> 00:01:14,637
[SPEAKER_02]: Let's get that in early.

24
00:01:14,677 --> 00:01:15,599
[SPEAKER_02]: And I'm Michael Helbling.

25
00:01:16,099 --> 00:01:19,425
[SPEAKER_02]: All right, well, we need a guest, someone to help us dive into this topic.

26
00:01:19,565 --> 00:01:20,587
[SPEAKER_02]: And we've got a great one.

27
00:01:20,647 --> 00:01:27,919
[SPEAKER_02]: Sam Redfern is a staff data scientist at Canva, currently working on search and recommendations there and previously marketing measurement.

28
00:01:27,899 --> 00:01:33,206
[SPEAKER_02]: Prior to that, he has held data roles at both Meta and IAG, and today he is our guest.

29
00:01:33,266 --> 00:01:36,290
[SPEAKER_02]: Welcome to the show, Sam.

30
00:01:36,311 --> 00:01:37,893
[SPEAKER_05]: Thank you very much, Michael, Tim, and Val.

31
00:01:37,913 --> 00:01:39,114
[SPEAKER_05]: We're really excited to be here.

32
00:01:39,194 --> 00:01:41,578
[SPEAKER_05]: First time caller, long time listener.

33
00:01:42,299 --> 00:01:43,180
[SPEAKER_05]: Oh, that's awesome.

34
00:01:43,761 --> 00:01:47,085
[SPEAKER_02]: Well, we'll ask questions and take our answers off the air.

35
00:01:47,225 --> 00:01:48,027
[SPEAKER_02]: No, no.

36
00:01:48,487 --> 00:01:51,291
[SPEAKER_01]: So.

37
00:01:51,271 --> 00:02:00,263
[SPEAKER_02]: Sam, I'm excited to talk to you about this, because obviously, all things AI are very of the moment, and everyone sees the term MCP.

38
00:02:00,283 --> 00:02:16,924
[SPEAKER_02]: But I think if we just take a step back, maybe you could just fill us in on what exactly an MCP is, model context protocol, where did it come from, give us some background on the whole concept to establish the conversation today.

39
00:02:17,343 --> 00:02:18,284
[SPEAKER_05]: No worries.

40
00:02:18,665 --> 00:02:21,168
[SPEAKER_05]: Super pumped to be talking about this.

41
00:02:22,489 --> 00:02:27,335
[SPEAKER_05]: Let's take a step back and think about what this is solving, in a sense.

42
00:02:29,938 --> 00:02:34,764
[SPEAKER_05]: We've had access to these large language models tools for a little while now.

43
00:02:35,725 --> 00:02:39,570
[SPEAKER_05]: In the early days of GPT2 and GPT3,

44
00:02:39,550 --> 00:02:44,499
[SPEAKER_05]: before chat GBT, these things were like word calculators in a sense.

45
00:02:44,579 --> 00:02:55,859
[SPEAKER_05]: I really like the analogy that it's like you put the numbers into your calculator and you get the equation out.

46
00:02:55,919 --> 00:02:57,883
[SPEAKER_05]: This is the same for almost words.

47
00:02:57,923 --> 00:03:00,327
[SPEAKER_05]: Early large language models acted like that.

48
00:03:00,307 --> 00:03:10,323
[SPEAKER_05]: And the innovation in the open AI space was to basically feed the output back into the input and make this resumable format.

49
00:03:10,924 --> 00:03:26,550
[SPEAKER_05]: And what's interesting about this whole, so let's step back from MCP and branding and letting technical teams come up with the term for things because this is how we got NFTs as a term at the same time.

50
00:03:26,530 --> 00:03:43,018
[SPEAKER_05]: But the core problem to be solved with these is that we've got something that feels a little bit like a person in the sense that you will give it some words and it will respond with some words back.

51
00:03:43,759 --> 00:03:52,774
[SPEAKER_05]: Could you give that agent or that large language model the ability to do something other than just converse?

52
00:03:52,754 --> 00:04:02,011
[SPEAKER_05]: And so the first application of tool use, in a sense, in large language models, actually came from the open-source LangChain team.

53
00:04:02,492 --> 00:04:09,245
[SPEAKER_05]: And for those who don't know what LangChain is, it's a framework for building agentic experiences.

54
00:04:10,007 --> 00:04:14,475
[SPEAKER_05]: And so you can have your anthropic model

55
00:04:14,455 --> 00:04:24,096
[SPEAKER_05]: your open AI model or whatever you want, and then you allows you to piece together bits of technology to add context into the large language model to try and get it output.

56
00:04:24,718 --> 00:04:31,232
[SPEAKER_05]: And so in April 2023, PR was submitted to the Landtrait project, allowing it

57
00:04:31,212 --> 00:04:45,033
[SPEAKER_05]: to open up for the large language model to take a browser URL and to go have the back end application request the contents of that HTML and then bring it back into the context window of the agent itself.

58
00:04:45,514 --> 00:04:51,583
[SPEAKER_05]: And if you think about it as like the core thing that it's trying to do is it's trying to get this

59
00:04:51,563 --> 00:05:02,017
[SPEAKER_05]: I think about it as fingers in a sense, is it's trying to give the large language the ability to touch something, a bit of information, bring it closer to it for it to understand.

60
00:05:02,077 --> 00:05:05,802
[SPEAKER_05]: That's at its core what these MCP tools are.

61
00:05:07,505 --> 00:05:09,467
[SPEAKER_05]: MCP is like a brand term through Anthropoc.

62
00:05:09,547 --> 00:05:12,351
[SPEAKER_05]: To say it's a standard is to be very generous.

63
00:05:13,272 --> 00:05:20,322
[SPEAKER_05]: But I'm really bullish about the concept of giving these large language models access to tools for them to be able to solve problems.

64
00:05:22,343 --> 00:05:30,871
[SPEAKER_00]: So I have to admit that I really did think that MCP was just like the API for LLMs.

65
00:05:31,171 --> 00:05:41,261
[SPEAKER_00]: But the more I was looking into this, it was kind of understanding that those fingers like you use in your analogy is really giving it more access than just here's this endpoint.

66
00:05:41,341 --> 00:05:42,482
[SPEAKER_00]: And it's just a one-time thing.

67
00:05:42,502 --> 00:05:52,131
[SPEAKER_00]: Can you talk about some of the things that you give it access to with those fingers or to grab to kind of give a little bit more color to what it actually does or what it's

68
00:05:52,111 --> 00:05:54,576
[SPEAKER_00]: capable of, if that makes sense?

69
00:05:54,596 --> 00:05:55,397
[SPEAKER_05]: Yeah, absolutely.

70
00:05:56,359 --> 00:06:02,069
[SPEAKER_05]: So this is in the weeds of how they work.

71
00:06:02,490 --> 00:06:15,975
[SPEAKER_05]: I think the context to understand is if you've ever done work inside of large enterprises and you've tried to create an application access username and password, it's a total pain in the butt.

72
00:06:15,955 --> 00:06:17,558
[SPEAKER_05]: to go get this thing.

73
00:06:17,598 --> 00:06:20,943
[SPEAKER_05]: And then your infrastructure team is like, well, you're going to change the password every three weeks.

74
00:06:21,083 --> 00:06:25,230
[SPEAKER_05]: And then you have to have a cryptographic token to do something.

75
00:06:25,290 --> 00:06:26,612
[SPEAKER_05]: And it's in some sort of space.

76
00:06:27,033 --> 00:06:40,815
[SPEAKER_05]: And the reason why, actually, Anthropoc had a previous attempt at tool use in October 2024, a month before MCPs were announced, where they had a system that would take over your browser and move your cursor around.

77
00:06:40,795 --> 00:06:47,103
[SPEAKER_05]: And the reason why MCPs are running on your local computer is that your user account has access to all these systems.

78
00:06:48,425 --> 00:06:57,257
[SPEAKER_05]: That's why the early paradigms of these systems are as close as possible to the end user's system.

79
00:06:57,597 --> 00:07:08,592
[SPEAKER_05]: So the analogy I give on the fingers, in a sense, is inside these MCPs, you can give it access to any number of... The standard allows you to sort of

80
00:07:08,572 --> 00:07:11,096
[SPEAKER_05]: use representatives any number of tools.

81
00:07:11,536 --> 00:07:13,519
[SPEAKER_05]: And you have bits of information, right?

82
00:07:13,559 --> 00:07:21,632
[SPEAKER_05]: So when the agent starts, it's basically given a list of all the tools that the MCP server has available to the agent.

83
00:07:22,052 --> 00:07:23,615
[SPEAKER_05]: And so has the name of the tool.

84
00:07:23,715 --> 00:07:29,804
[SPEAKER_05]: And so let's just do a really simple example of like us, like in our agent environment, we have two tools, right?

85
00:07:29,864 --> 00:07:32,588
[SPEAKER_05]: One of them is called a saw and the other one's called a drill, right?

86
00:07:32,568 --> 00:07:36,993
[SPEAKER_05]: In our SOAR tool, we would describe the name as SOAR.

87
00:07:37,534 --> 00:07:42,199
[SPEAKER_05]: We would say the description is it cuts wood in a single direction across a line.

88
00:07:42,239 --> 00:07:47,485
[SPEAKER_05]: Then the inputs to that is the position of the wood and the depth.

89
00:07:48,506 --> 00:07:51,830
[SPEAKER_05]: That is a finger for lack of a better term.

90
00:07:51,850 --> 00:07:59,218
[SPEAKER_05]: That's our first tool and our second tool might be drill.

91
00:07:59,579 --> 00:08:02,322
[SPEAKER_05]: That allows you to drill a hole through the piece of wood.

92
00:08:03,112 --> 00:08:05,476
[SPEAKER_00]: Okay, so I have to ask one more.

93
00:08:05,496 --> 00:08:08,121
[SPEAKER_00]: Sorry, I'm hogging the air.

94
00:08:08,661 --> 00:08:17,817
[SPEAKER_00]: But I guess the one other thing that I'm struggling to grasp a little bit is what was the need for standardization of this, like the protocol?

95
00:08:17,837 --> 00:08:19,820
[SPEAKER_00]: Can you talk about what that is solving?

96
00:08:19,840 --> 00:08:22,725
[SPEAKER_00]: Because what you shared in that analogy was great.

97
00:08:22,846 --> 00:08:26,031
[SPEAKER_00]: I'm absolutely going to use that, and I'll give you credit every time.

98
00:08:26,011 --> 00:08:38,957
[SPEAKER_00]: but like why was there a need to standardize outside of like you know enterprises you know would feel more secure with that or you know the governance would be easier but is there any more more to it than just that piece?

99
00:08:39,224 --> 00:08:45,232
[SPEAKER_05]: In the adoption curve, we are so far away from the governance piece on this stuff.

100
00:08:45,493 --> 00:08:56,908
[SPEAKER_05]: There's a bunch of companies right now that are trying to put governance around these systems, and I'm sure at some point we'll talk to maybe some of the potential downsides of the standard if we want to call it that.

101
00:08:58,010 --> 00:09:03,257
[SPEAKER_05]: But the reason why Anthropic went down this path is

102
00:09:03,237 --> 00:09:15,911
[SPEAKER_05]: In the technical details around how the LLM is trained, they have been doing this work of training the large language model to use this like special escape set of characters.

103
00:09:16,854 --> 00:09:19,120
[SPEAKER_05]: So when the large language model is like

104
00:09:19,100 --> 00:09:31,354
[SPEAKER_05]: Okay, I think what I need to do is use the saw tool and then it has this string of characters saw tool string of characters, and that indicates to the sort of the agent that's hosting the large language model.

105
00:09:31,715 --> 00:09:34,122
[SPEAKER_05]: Okay, I have to take the text below this.

106
00:09:34,102 --> 00:09:40,438
[SPEAKER_05]: and send it into the tool itself with the input parameters that it needs for it.

107
00:09:41,079 --> 00:09:50,322
[SPEAKER_05]: Anthropic had this huge lead because they've done the work of training their large language models to

108
00:09:50,302 --> 00:09:56,216
[SPEAKER_05]: for tool use and using their reinforcement techniques to basically say, this is what you have to do.

109
00:09:56,236 --> 00:10:01,810
[SPEAKER_05]: And this was this huge lead that anthropic had for a couple of years, in a sense.

110
00:10:03,413 --> 00:10:05,779
[SPEAKER_05]: Everything feels like it's a couple of years.

111
00:10:05,759 --> 00:10:09,504
[SPEAKER_05]: It's really about eight months, right?

112
00:10:09,625 --> 00:10:12,288
[SPEAKER_05]: Until other people started trying to solve this problem.

113
00:10:12,669 --> 00:10:18,397
[SPEAKER_05]: Opening, I had their own sort of call procedures kind of method, like I think they were called functions.

114
00:10:19,018 --> 00:10:20,640
[SPEAKER_05]: And it was a very similar kind of thing.

115
00:10:20,681 --> 00:10:22,543
[SPEAKER_05]: Like anyone could have come up with a standard.

116
00:10:23,024 --> 00:10:34,180
[SPEAKER_05]: The core problem they were trying to solve is how do you give the large language model a hand for it to basically make decisions about what information is pulled towards it or what information or what

117
00:10:34,160 --> 00:10:35,803
[SPEAKER_05]: actions it takes when it's pushing out.

118
00:10:36,404 --> 00:10:43,415
[SPEAKER_03]: This is slowly coming a little more into focus and still pretty damn fuzzy for me.

119
00:10:43,436 --> 00:10:54,574
[SPEAKER_03]: I know recently it seems like there's been a lot of chatter about Google Analytics having an MCP server.

120
00:10:54,714 --> 00:10:56,437
[SPEAKER_03]: Is that the right terminology?

121
00:10:58,300 --> 00:10:59,863
[SPEAKER_03]: That is something that

122
00:11:00,704 --> 00:11:08,156
[SPEAKER_03]: that the Google team said, we're going to produce this to basically make the, this is a saw.

123
00:11:08,737 --> 00:11:09,779
[SPEAKER_03]: This is what it does.

124
00:11:09,839 --> 00:11:11,702
[SPEAKER_03]: These are the inputs.

125
00:11:11,822 --> 00:11:12,924
[SPEAKER_03]: And it's just a much a lot.

126
00:11:12,964 --> 00:11:14,847
[SPEAKER_03]: I mean, your analogy was very, very simple.

127
00:11:14,927 --> 00:11:21,818
[SPEAKER_03]: Is it as simple as that, which has me going back to saying, well, when Val said,

128
00:11:22,051 --> 00:11:24,333
[SPEAKER_03]: It's like an API for LOMs.

129
00:11:25,415 --> 00:11:29,158
[SPEAKER_03]: It sure sounds like an API for LOMs.

130
00:11:29,178 --> 00:11:35,005
[SPEAKER_03]: And I'm missing where that analogy is breaking down.

131
00:11:35,025 --> 00:11:36,806
[SPEAKER_05]: I think you can use it.

132
00:11:37,107 --> 00:11:46,877
[SPEAKER_05]: There's a lot of analogies of talking about these NCP tools as being the early dates of APIs and stuff like that as well.

133
00:11:47,195 --> 00:11:57,134
[SPEAKER_05]: I think there's an extra bit of the near direction of where these systems are moving, which is more interesting in the API part, but just to come back to that.

134
00:11:57,515 --> 00:12:05,711
[SPEAKER_05]: The way I think about it, APIs is a great way of talking about it, and there's lots of people doing weird fun things with these tools right now.

135
00:12:05,831 --> 00:12:07,735
[SPEAKER_05]: If you remember, I think

136
00:12:07,715 --> 00:12:23,783
[SPEAKER_05]: some of us on the call are old enough to remember the early days of web 2.0 and people were making APIs for like the weather and it was open and everything was fine and you know we were very far from like the standardized way of we think about this sort of stuff now right like it's uh

137
00:12:23,763 --> 00:12:26,867
[SPEAKER_05]: The way you design an API is very standardized now.

138
00:12:27,347 --> 00:12:33,595
[SPEAKER_05]: I think the thing that's different is one, we're dealing with this huge amount of non-determinism, right?

139
00:12:33,915 --> 00:12:38,941
[SPEAKER_05]: And we're dealing with all of these different terms and terminologies that exist.

140
00:12:39,742 --> 00:12:44,167
[SPEAKER_05]: So I think everyone on the podcast might have heard of the term agent, right?

141
00:12:44,648 --> 00:12:48,973
[SPEAKER_05]: And so an agent is the idea where you have a resumable output.

142
00:12:49,774 --> 00:12:52,757
[SPEAKER_05]: You have like some

143
00:12:52,737 --> 00:12:57,907
[SPEAKER_05]: text that is the system prompt, and then you have this resumable conversation.

144
00:12:58,348 --> 00:13:01,594
[SPEAKER_05]: There's another term that's being formed right now called a harness.

145
00:13:02,095 --> 00:13:06,022
[SPEAKER_05]: A harness is an idea where you have an agent and you have a tool plugged into the side of it.

146
00:13:06,583 --> 00:13:11,011
[SPEAKER_05]: That has a domain of knowledge attached to it at the same time.

147
00:13:11,412 --> 00:13:12,394
[SPEAKER_05]: Cursor is an agent.

148
00:13:12,474 --> 00:13:14,037
[SPEAKER_05]: The claw desktop is an agent.

149
00:13:14,017 --> 00:13:15,219
[SPEAKER_05]: Oh, sorry, sorry.

150
00:13:16,181 --> 00:13:18,065
[SPEAKER_05]: Cursor is a harness, right?

151
00:13:18,266 --> 00:13:19,929
[SPEAKER_05]: It's got access to all these different tools.

152
00:13:20,851 --> 00:13:30,470
[SPEAKER_05]: I think the, I actually think of NTP and where it's at right now is more akin to these digital document formats like XML, right?

153
00:13:30,831 --> 00:13:33,797
[SPEAKER_05]: So we started with XML and

154
00:13:33,777 --> 00:13:36,960
[SPEAKER_05]: the number of people who are writing XML these days is almost none.

155
00:13:37,020 --> 00:13:48,250
[SPEAKER_05]: However, the amount of change of this standardized document format then brought us to JSON and now it has unfortunately brought us to YAML and Markdown.

156
00:13:48,270 --> 00:14:02,282
[SPEAKER_05]: We are at the XML stage of this development is that this is going to tool use conceptually attaching to large language models through agents and harnesses is

157
00:14:02,262 --> 00:14:04,206
[SPEAKER_05]: That is going to stay for a long time.

158
00:14:04,627 --> 00:14:09,779
[SPEAKER_05]: Whether it's the MTP standard or someone comes along with a better standard, then we'll see how that goes.

159
00:14:12,445 --> 00:14:14,951
[SPEAKER_02]: You know how developers got the AI engineer role?

160
00:14:15,512 --> 00:14:16,635
[SPEAKER_02]: It's time for the rest of us.

161
00:14:17,036 --> 00:14:20,203
[SPEAKER_02]: I think we're witnessing the rise of the AI analyst.

162
00:14:22,327 --> 00:14:25,650
[SPEAKER_03]: OK, does that just mean asking a chatbot to do math?

163
00:14:25,730 --> 00:14:28,413
[SPEAKER_03]: Because I have Excel for that, Michael.

164
00:14:28,433 --> 00:14:29,154
[SPEAKER_02]: Well, no, Tim.

165
00:14:29,214 --> 00:14:30,995
[SPEAKER_02]: I'm talking about Ask Why.

166
00:14:31,015 --> 00:14:32,557
[SPEAKER_02]: It's full stack analytics.

167
00:14:32,797 --> 00:14:37,001
[SPEAKER_02]: I ask a question in plain English, and the product prism orchestrates the whole thing.

168
00:14:37,482 --> 00:14:39,764
[SPEAKER_02]: You can pull in data from Excel or BigQuery.

169
00:14:42,106 --> 00:14:42,507
[SPEAKER_03]: Hold on.

170
00:14:42,567 --> 00:14:45,630
[SPEAKER_03]: You're sending BigQuery data to an LLM?

171
00:14:45,650 --> 00:14:47,672
[SPEAKER_03]: Security is going to have a heart of track.

172
00:14:48,172 --> 00:14:49,273
[SPEAKER_02]: Well, that's the best part.

173
00:14:49,313 --> 00:14:51,996
[SPEAKER_02]: Ask why doesn't upload your data.

174
00:14:53,242 --> 00:14:54,045
[SPEAKER_02]: Explain?

175
00:14:54,867 --> 00:14:56,372
[SPEAKER_02]: Well, it creates a semantic layer.

176
00:14:56,593 --> 00:14:58,680
[SPEAKER_02]: It sends the context to the LLM.

177
00:14:58,740 --> 00:15:02,271
[SPEAKER_02]: The LLM writes the code, and that code runs locally on your data.

178
00:15:02,652 --> 00:15:06,846
[SPEAKER_02]: Your actual numbers never touch their servers, so it's totally traceable.

179
00:15:07,619 --> 00:15:13,154
[SPEAKER_03]: So, I get the automation, but my data stays safe and secure?

180
00:15:13,214 --> 00:15:14,598
[SPEAKER_02]: Exactly.

181
00:15:15,079 --> 00:15:23,502
[SPEAKER_02]: Plus, it remembers context, so as you automate routine tasks, it stores those, so you don't have to explain it all again the next time you do that same task.

182
00:15:23,836 --> 00:15:25,318
[SPEAKER_03]: OK, I'm listening.

183
00:15:25,819 --> 00:15:26,440
[SPEAKER_02]: Where do I get it?

184
00:15:27,100 --> 00:15:30,365
[SPEAKER_02]: Well, it's even beta right now, and you can go to ask-y.ai.

185
00:15:31,446 --> 00:15:32,848
[SPEAKER_02]: That's ask-y.ai.

186
00:15:35,312 --> 00:15:39,317
[SPEAKER_02]: You can get ahead of the curve and join the ranks of the AI analysts.

187
00:15:39,337 --> 00:15:47,188
[SPEAKER_03]: And because we like you guys, use code APH when you sign up, and our friends at Ask Why will put you at the top of their wait list.

188
00:15:48,170 --> 00:15:50,493
[SPEAKER_02]: Yep, stop pasting data into black boxes.

189
00:15:50,933 --> 00:15:52,155
[SPEAKER_02]: Get Ask Why.

190
00:15:52,996 --> 00:16:07,893
[SPEAKER_03]: that I've had the XML question as well as whether it was, because I remember that being coming from an HTML world and then XML came out and it's like, look, XML doesn't give a shit about what you're rendering in a browser, but it is this structured world.

191
00:16:07,933 --> 00:16:11,016
[SPEAKER_03]: So I feel like, and then JSON, I sort of understood because of the XML.

192
00:16:11,036 --> 00:16:17,704
[SPEAKER_03]: And that makes sense because there were, there was talk of saying different,

193
00:16:18,494 --> 00:16:34,378
[SPEAKER_03]: applications uses would say using this XML structure, let's define kind of specifically how that is going to be used in the context of this financial services thing.

194
00:16:34,659 --> 00:16:40,087
[SPEAKER_03]: So you're saying that is also a useful if imperfect analogy?

195
00:16:40,808 --> 00:16:41,970
[SPEAKER_05]: Yeah, look.

196
00:16:42,845 --> 00:16:47,661
[SPEAKER_05]: Uh, history sort of rhymes more than copies, right?

197
00:16:47,682 --> 00:16:53,682
[SPEAKER_05]: Like, um, you know, it's, it's going to, um, uh, like,

198
00:16:54,843 --> 00:16:59,831
[SPEAKER_05]: It's the first time in software that we've had this amount of non-determinism to deal with, right?

199
00:17:00,272 --> 00:17:17,982
[SPEAKER_05]: You think about what success has meant in software development or data before, and it's like some human's ability to remember some random function as part of a library and be able to write that code as fast as possible and do it in as perfect as close to grammar as possible.

200
00:17:17,962 --> 00:17:28,452
[SPEAKER_05]: The problem with this new world that we're going into is the skill of dealing with non-determinism is not closely overlapping with that historical set of skills.

201
00:17:28,652 --> 00:17:37,441
[SPEAKER_05]: It's going to feel different and people who talk about the vibes of a model, there's some truth in it in a sense.

202
00:17:38,122 --> 00:17:41,145
[SPEAKER_05]: Getting a feel and it's true with tool design.

203
00:17:41,665 --> 00:17:46,770
[SPEAKER_05]: When you're building an MCP, so we'll just go back to the MCP paradigm,

204
00:17:46,750 --> 00:17:55,119
[SPEAKER_05]: And so you're thinking about the tools that you're building and the fingers that you're giving this agent and this harness access to.

205
00:17:55,179 --> 00:18:04,248
[SPEAKER_05]: When you're starting out, you're doing kind of the early playful part of programming, in a sense, right?

206
00:18:04,268 --> 00:18:06,050
[SPEAKER_05]: Where you're just like, oh, does this connect to this?

207
00:18:06,210 --> 00:18:09,173
[SPEAKER_05]: And when I run through it, what problems do I see with it?

208
00:18:09,814 --> 00:18:13,618
[SPEAKER_05]: I do think that, like,

209
00:18:13,598 --> 00:18:17,582
[SPEAKER_05]: It feels like those early days of these standards and people are playing around with them.

210
00:18:17,602 --> 00:18:29,835
[SPEAKER_05]: And then when you want to get serious about the thing that you're building, you're then looking through the agent logs, you're seeing what tools it's calling, you're seeing the parameters, you're seeing how many times it correctly passes in the correct parameters and everything like that.

211
00:18:30,076 --> 00:18:32,698
[SPEAKER_03]: So let me hit the non-determinism point.

212
00:18:32,718 --> 00:18:41,648
[SPEAKER_03]: And maybe I'm going to go back to using the Google Analytics MCP as an example that the

213
00:18:42,387 --> 00:18:54,439
[SPEAKER_03]: The client, the application that's hooked, the agent that is hooking in and using the MCP, say it's a LLM, it's core.

214
00:18:55,330 --> 00:18:56,992
[SPEAKER_03]: And maybe I'm missing the non.

215
00:18:57,012 --> 00:19:04,700
[SPEAKER_03]: I'm thinking of that as being it's a it's a probabilistic thing, but it's hitting the MCP to get stuff back.

216
00:19:06,241 --> 00:19:10,746
[SPEAKER_03]: Is the MCP necessarily also kind of non deterministic?

217
00:19:10,786 --> 00:19:19,015
[SPEAKER_03]: Or is it no, like the inputs may be kind of floating around a little bit, but and or maybe it just depends on what it's an MCP for.

218
00:19:19,175 --> 00:19:24,240
[SPEAKER_03]: If in the Google Analytics example, I would say if the input is

219
00:19:24,220 --> 00:19:39,934
[SPEAKER_03]: users in the last month that the MCP would say, well, as long as that, or is it that input's going to come in with the little squishiness in it, and it's up to the MCP to say, I got to figure out what I should go and pull and return.

220
00:19:39,914 --> 00:19:47,643
[SPEAKER_05]: Yeah, and I think this is a good point to talk about this interesting dimensions that's coming up recently.

221
00:19:48,124 --> 00:19:59,537
[SPEAKER_05]: If I was going on a league team, which I'm definitely not, and I don't think I've used Google Analytics for over a decade, and I missed many of the big transitions.

222
00:19:59,557 --> 00:20:03,542
[SPEAKER_05]: I'm probably not the best person to talk about GA, but

223
00:20:03,522 --> 00:20:15,871
[SPEAKER_05]: So the Google Analytics MCP is going to have this problem where they have these dimensions and measures and breakdowns, and then they're obviously trying to do it on the cheap on the inside.

224
00:20:15,911 --> 00:20:21,143
[SPEAKER_05]: And so they're only storing some of the information.

225
00:20:21,123 --> 00:20:28,959
[SPEAKER_05]: So they're going to have a basic report tool, so it's going to be called the report tool.

226
00:20:29,159 --> 00:20:32,265
[SPEAKER_05]: And the name of it is going to be traffic analysis.

227
00:20:32,927 --> 00:20:41,123
[SPEAKER_05]: And it's going to be put the date range in, and then it's just going to return back a very minimized array of what the traffic has been.

228
00:20:41,103 --> 00:20:46,291
[SPEAKER_03]: Report tool, that would be like a saw, but there's going to be separately a drill.

229
00:20:46,752 --> 00:20:51,579
[SPEAKER_03]: When you're saying a tool, that is a tool that's available as part of the MCP.

230
00:20:52,060 --> 00:20:53,001
[SPEAKER_03]: Okay.

231
00:20:53,242 --> 00:20:53,542
[SPEAKER_05]: That's right.

232
00:20:53,963 --> 00:21:03,077
[SPEAKER_05]: Now, where this gets interesting and anthropic announced for lack of a bit of time on the, I think a couple of days ago,

233
00:21:03,057 --> 00:21:08,623
[SPEAKER_05]: announces code execution with MCPs and allowing you to actually write code.

234
00:21:08,783 --> 00:21:27,462
[SPEAKER_05]: This is the thing that I've actually found over the last couple of weeks, which is it's way better getting the LLM to write code to interact with APIs or SQL or something along those lines than it is to actually give it access to all of these tools and all of these intermediary steps.

235
00:21:27,442 --> 00:21:41,786
[SPEAKER_05]: If and you might have seen this from your own experience of, you know, you're probably spending a little bit less time hacking away at a piece of sequel, getting it to form exactly what you want it to these days.

236
00:21:42,267 --> 00:21:46,935
[SPEAKER_05]: You know, you're probably spending a bit more time of like here's all the table space that I'm working on.

237
00:21:46,915 --> 00:21:54,006
[SPEAKER_05]: here is a thing that I want to query, here's what the outputs look like, and then you're sort of having this sort of feedback loop where you're doing that work.

238
00:21:54,667 --> 00:22:04,182
[SPEAKER_05]: And my guess is, is that if you wanted to build a more sophisticated MCP, and if you were Google, you would actually lean into this concept where you would

239
00:22:04,162 --> 00:22:15,181
[SPEAKER_05]: let the agent go build a little piece of Python code or JavaScript or something along those lines to query a bunch of known API endpoints to form the data back in the way that you want it to.

240
00:22:15,221 --> 00:22:29,325
[SPEAKER_05]: Snowflake has its MCP server and Canva would make something called data MCP, which takes all of this data information we have and allows the LLM access to understanding how to use it.

241
00:22:29,305 --> 00:22:42,266
[SPEAKER_05]: you're really doing this piece of work around context engineering and you're trying to think about like what is the LLM going to put into this tool for then it to get this output out.

242
00:22:42,466 --> 00:22:49,979
[SPEAKER_05]: And so Michael to say, and so your sort of question here is that the MCP tool itself is deterministic, right?

243
00:22:50,099 --> 00:22:55,127
[SPEAKER_05]: So it is an application in the traditional software sense.

244
00:22:55,107 --> 00:23:01,682
[SPEAKER_05]: I think a lot of these MTP tools are sending data outside of out to the internet, right?

245
00:23:01,722 --> 00:23:10,922
[SPEAKER_05]: They're connected into API or a SQL database or something along those lines, but you can have deterministic tools inside of the MTP server as well that's locally that's connected.

246
00:23:10,902 --> 00:23:17,836
[SPEAKER_05]: So you could just have like a calculator tool and it just adds numbers together and then returns exactly the right number out.

247
00:23:17,876 --> 00:23:21,945
[SPEAKER_05]: You know, we've heard the joke of counting the number r of r's and strawberry, right?

248
00:23:22,265 --> 00:23:27,055
[SPEAKER_05]: So you could have a little Python function that just counts the number r's and strawberry and it's just like, all right.

249
00:23:27,035 --> 00:23:31,082
[SPEAKER_05]: the R counter tool, put the word strawberry in, and this is how many R's you get back.

250
00:23:31,163 --> 00:23:41,221
[SPEAKER_05]: And so this is the whole idea of, and this is, I understand I'm going to do shout outs later on, but Zed, which is an ID, has an agentic engineering series.

251
00:23:41,702 --> 00:23:43,305
[SPEAKER_05]: And what they're saying,

252
00:23:43,285 --> 00:24:00,855
[SPEAKER_05]: And I just think that they have this really great framing of the problem that we're working on today, which is how do you take the advantages of non-deterministic systems and couple it with the advantage of deterministic systems to get something more than the sum of its components?

253
00:24:00,953 --> 00:24:12,218
[SPEAKER_00]: So I would love to take a little bit of a turn because you've teased it a little bit, but I'm so eager to hear some of your favorite use cases and examples and that you can talk about.

254
00:24:12,238 --> 00:24:18,051
[SPEAKER_00]: And I know you mentioned before, it's still early days, but if you could talk about some of the things that you felt like

255
00:24:18,031 --> 00:24:26,801
[SPEAKER_00]: weren't possible for or too much effort for what it was worth in the past, but now it's unlocked this or solved this for you.

256
00:24:27,201 --> 00:24:31,126
[SPEAKER_05]: I'll talk a little bit about some of the stuff we've got working inside of Canberra at the moment.

257
00:24:32,207 --> 00:24:38,614
[SPEAKER_05]: We use a large database vendor, which I will keep their name out of just so they don't get in trouble.

258
00:24:39,154 --> 00:24:44,140
[SPEAKER_05]: They're going down this path of building out their own separate MCP tools and stuff like that.

259
00:24:44,120 --> 00:25:02,807
[SPEAKER_05]: But what's interesting and what I think is a big opportunity for people in this space is that building these tools for your organization is actually the critical skill that we're going to see in the coming period of time.

260
00:25:03,448 --> 00:25:08,495
[SPEAKER_05]: Every organization or company is really different, in a sense.

261
00:25:08,595 --> 00:25:13,182
[SPEAKER_05]: And they made a database choice

262
00:25:13,162 --> 00:25:17,229
[SPEAKER_05]: five years ago, and they made like a data transformation tool twice like four years ago.

263
00:25:17,269 --> 00:25:26,085
[SPEAKER_05]: And you have all the incremental knowledge and information that's built on top of that, which is going to kind of be unique to your organization in a sense, right?

264
00:25:26,606 --> 00:25:30,353
[SPEAKER_05]: And I think there's going to be vendors who are building tools in this space.

265
00:25:30,813 --> 00:25:35,642
[SPEAKER_05]: But if you're sort of a midsize company, like I think something to really think about is

266
00:25:35,622 --> 00:25:50,644
[SPEAKER_05]: building customized versions of these tools that actually work with the flows of your organization and really having teams that are like building, thinking about these agenting and engineering practices on how to actually automate parts of the work that people don't want to do.

267
00:25:50,624 --> 00:26:08,159
[SPEAKER_05]: So some of the use cases that I'm sort of playing around at the moment, just so just the last week I've been working on using the Altair Python visualization library and actually building a Python based sort of sandbox environment for it to run the Altair code.

268
00:26:08,620 --> 00:26:09,983
[SPEAKER_05]: And so

269
00:26:09,963 --> 00:26:21,078
[SPEAKER_05]: The way this works is you just put the SQL statement of the data that you want to pass into the Altair code, and then you have a Python sandbox part of the field of the tool.

270
00:26:21,219 --> 00:26:27,788
[SPEAKER_05]: It just puts 300 lines of Python into it, and it builds the visualizations out of it, right?

271
00:26:27,808 --> 00:26:34,637
[SPEAKER_05]: So using these really nice Python visualization libraries has always been a pain in the butt unless you

272
00:26:34,617 --> 00:26:41,611
[SPEAKER_05]: tattoo the way every part of the application works on your arm so you can know exactly how it works.

273
00:26:42,012 --> 00:26:53,875
[SPEAKER_05]: But again, we don't have to worry so much about grammar anymore, because if you feed these systems examples, they can come back and help you visualize this way faster.

274
00:26:53,855 --> 00:27:00,706
[SPEAKER_05]: And I think that's where, you know, Zed has this post where they, their concept is leverage, not magic, right?

275
00:27:01,247 --> 00:27:12,646
[SPEAKER_05]: And what we're trying to do is we're trying to take our staff members and we're trying to make them move faster and explore more in a shorter period of time to get to a better end outcome.

276
00:27:13,487 --> 00:27:18,816
[SPEAKER_05]: Just on other sort of like interesting fun use cases, I've been,

277
00:27:19,285 --> 00:27:23,352
[SPEAKER_05]: At home, I've got my own little sort of home lab kind of thing and stuff like that.

278
00:27:23,933 --> 00:27:28,701
[SPEAKER_05]: And part of playing around with that is there's all these command switches and stuff like that.

279
00:27:28,721 --> 00:27:35,193
[SPEAKER_05]: And so I built a custom NCP server for home that documents all the different applications that I use in the CLI.

280
00:27:35,253 --> 00:27:39,420
[SPEAKER_05]: And it has all of the context of it.

281
00:27:39,400 --> 00:27:42,543
[SPEAKER_05]: I basically put in a free text field of what I'm trying to do.

282
00:27:42,563 --> 00:27:46,667
[SPEAKER_05]: It then uses a search engine to search over the data.

283
00:27:46,787 --> 00:27:53,293
[SPEAKER_05]: It then takes that context, puts it into the LLM, and then the LLM goes and gives me all the command switches and stuff like that.

284
00:27:53,333 --> 00:28:03,602
[SPEAKER_05]: Another one is, I think the reason why Mo recommended me for this is I gave a presentation at MeasureCamp about, at MeasureCamp, you have to come up with a talk title, otherwise people don't show up.

285
00:28:03,782 --> 00:28:09,407
[SPEAKER_05]: I said that MCP is the real apex predator for your job in

286
00:28:09,387 --> 00:28:09,707
[SPEAKER_05]: 2026.

287
00:28:10,709 --> 00:28:16,936
[SPEAKER_05]: And obviously, you know, I don't think that's true.

288
00:28:16,996 --> 00:28:22,362
[SPEAKER_05]: But so I wrote that someone was like, you need to give a talk sound.

289
00:28:22,442 --> 00:28:23,243
[SPEAKER_05]: I'm just like, fine.

290
00:28:24,064 --> 00:28:31,133
[SPEAKER_05]: So I wrote that part at 11 o'clock, I then vibe coded up a maybe the game battleship.

291
00:28:31,153 --> 00:28:37,420
[SPEAKER_05]: And so I made a version of battleship that uses MCP tools as the

292
00:28:37,400 --> 00:28:40,985
[SPEAKER_05]: the fingers that the LLMs can use to play against each other.

293
00:28:41,005 --> 00:28:49,616
[SPEAKER_05]: And then I built an agent harness where I could get different vendors, MCP tools to actually fight the game battleship against each other.

294
00:28:49,636 --> 00:28:54,602
[SPEAKER_05]: And then you could actually watch the turn by turn thing of watching them compete against each other.

295
00:28:54,622 --> 00:28:59,388
[SPEAKER_05]: And they would then have their thinking of like how they thought about the strategy of the other player.

296
00:28:59,408 --> 00:29:03,053
[SPEAKER_05]: And then you could like change the prompt and be like, okay,

297
00:29:03,033 --> 00:29:07,817
[SPEAKER_05]: you are going to be the random player and you're just going to do the most random things you possibly can.

298
00:29:07,897 --> 00:29:11,741
[SPEAKER_05]: And then the other one is like the most strategic player of like, this is the common moves in battleship.

299
00:29:11,761 --> 00:29:13,182
[SPEAKER_05]: I'm going to do this and stuff like that.

300
00:29:13,582 --> 00:29:18,587
[SPEAKER_05]: And it's just, it's like, like, it's a whole new paradigm of like weird things to play around with.

301
00:29:18,767 --> 00:29:25,433
[SPEAKER_05]: And like MCPs are just like this, this layer for you to like do this joining between deterministic and non-deterministic systems.

302
00:29:25,753 --> 00:29:33,040
[SPEAKER_05]: But once you start playing with these systems and you're finding different ways of interesting things to do with them, like, you know, it's, it's, it's,

303
00:29:33,020 --> 00:29:37,244
[SPEAKER_05]: It puts the fund back into sort of like the early stages of programming again.

304
00:29:38,165 --> 00:29:45,352
[SPEAKER_00]: So you giving the context of like deterministic, non-deterministic is really that's helping it crystallize a little bit in my mind what some of this is.

305
00:29:45,372 --> 00:29:52,219
[SPEAKER_00]: But I do want to go back to when you first started talking about some of these examples, you were talking about how every organization is different.

306
00:29:52,279 --> 00:29:54,061
[SPEAKER_00]: Everyone's working with a different stack.

307
00:29:54,341 --> 00:29:59,026
[SPEAKER_00]: Everything has to be kind of in context of what's going on inside your organization.

308
00:29:59,806 --> 00:30:01,508
[SPEAKER_00]: Can I just go back and just

309
00:30:01,488 --> 00:30:03,893
[SPEAKER_00]: repeat a little bit one of my earlier questions.

310
00:30:04,013 --> 00:30:11,729
[SPEAKER_00]: What is the value then of the standardization versus it being custom inside of your organization?

311
00:30:11,850 --> 00:30:20,829
[SPEAKER_00]: Is it just about the ability to leverage those other tools that are using MCP or is there internal benefit too?

312
00:30:20,989 --> 00:30:22,011
[SPEAKER_00]: I guess I might

313
00:30:21,991 --> 00:30:25,055
[SPEAKER_00]: be just, it's not clicking for me.

314
00:30:25,075 --> 00:30:27,659
[SPEAKER_00]: I would love to just hear you share.

315
00:30:27,679 --> 00:30:28,520
[SPEAKER_05]: No, no, no, no.

316
00:30:28,540 --> 00:30:33,827
[SPEAKER_05]: Okay, so MCP is the thing that allows you at the moment until someone comes up with a better standard.

317
00:30:33,847 --> 00:30:36,531
[SPEAKER_05]: And we should probably talk about some of the downsides of MCP as well.

318
00:30:37,372 --> 00:30:42,058
[SPEAKER_05]: Just all the criticism, sorry, the criticisms as they stand of MCP, but

319
00:30:42,038 --> 00:30:47,006
[SPEAKER_05]: MCP at the moment allows you to bridge that gap between deterministic and non-deterministic systems.

320
00:30:47,787 --> 00:30:51,814
[SPEAKER_05]: You've got the vendors, and the vendors are going out, and they are building their own customers.

321
00:30:51,974 --> 00:31:01,549
[SPEAKER_05]: MCP is right, because what they want, because what they're getting, is they're getting pressure from leadership teams of like, okay, we need to get some AI in this organization right now.

322
00:31:01,529 --> 00:31:10,340
[SPEAKER_05]: I have called desktop on my computer and I want to be able to query this information directly from Claude and have that come back to me in a sense, right?

323
00:31:10,580 --> 00:31:20,473
[SPEAKER_05]: And MCPs is like a path you can go down there, but the problem is that you're wrestling with the non-deterministic nature of these systems when you do that type of connection in a sense, right?

324
00:31:20,933 --> 00:31:27,942
[SPEAKER_05]: And what you actually, this is where building up that practice inside of your organization is really important because

325
00:31:27,922 --> 00:31:43,963
[SPEAKER_05]: when you get into the detail of how these systems work, you realize that giving senior leadership teams access to the raw thing that directly queries the database is problematic for lots of different reasons around you're not able to check it, you can't give it a system prompt, all that sort of stuff.

326
00:31:44,944 --> 00:31:50,571
[SPEAKER_05]: So MCPs, the standard, if we can call it that, is just the little connecting block, in a sense.

327
00:31:50,551 --> 00:31:58,223
[SPEAKER_05]: the practice of what I'm talking about about the non-standardization inside of organizations is that you can totally go use the vendor solutions, right?

328
00:31:58,624 --> 00:32:06,617
[SPEAKER_05]: But it's always going to feel like it's through a fuzzy piece of glass, because unless you're doing exactly what the vendor has, right?

329
00:32:06,677 --> 00:32:09,201
[SPEAKER_05]: Like, if you're

330
00:32:09,181 --> 00:32:18,231
[SPEAKER_05]: If you're 100% Google shop, and you've never used anything other than Google, then maybe the Google stack is going to be great because that's what they're going to be using internally.

331
00:32:18,351 --> 00:32:20,854
[SPEAKER_05]: And they're going to be copying from that of how they're building it.

332
00:32:21,255 --> 00:32:30,225
[SPEAKER_05]: But I don't know about you, but most organizations I ever interact with is it sort of like a collage of different solutions.

333
00:32:30,485 --> 00:32:33,849
[SPEAKER_05]: And the money is in getting them to connect together, right?

334
00:32:33,889 --> 00:32:38,074
[SPEAKER_05]: Yeah, templatized versus custom kind of.

335
00:32:38,054 --> 00:32:44,385
[SPEAKER_03]: Altair, take Canva, take Snowflake, take Google Analytics and MCP for those.

336
00:32:44,967 --> 00:32:59,573
[SPEAKER_03]: Is there the option that Canva or Altair or Snowflake or Google, and those are intentionally very wildly different types of platforms, they can sort of

337
00:32:59,553 --> 00:33:20,380
[SPEAKER_03]: create an MCP, one of these connectors that say this is kind of generic, but there's also an option that I as a user of Canva with access, I could also build my own MCP, my own connector, or is it okay?

338
00:33:20,360 --> 00:33:26,712
[SPEAKER_05]: So you're raising on a really good topic, which is called context pollution, right?

339
00:33:27,554 --> 00:33:35,870
[SPEAKER_05]: So there was a criticism of GitHub's MCP server where it has like 24 some number of tools, right?

340
00:33:36,271 --> 00:33:39,838
[SPEAKER_05]: And the majority of people use like three, right?

341
00:33:40,198 --> 00:33:42,182
[SPEAKER_05]: And so I push.

342
00:33:42,162 --> 00:33:58,125
[SPEAKER_05]: If you get into the details of how NTPs work is that they repaste the entire list of tools and the tool descriptions and all this sort of stuff with every single message that goes through, right?

343
00:33:58,366 --> 00:34:01,470
[SPEAKER_05]: Because they are trying to solve the context engineering problem of, like,

344
00:34:01,450 --> 00:34:05,996
[SPEAKER_05]: The LLN needs to really know what all the tools are at every single turn.

345
00:34:06,777 --> 00:34:14,066
[SPEAKER_05]: And you can go and add 70 tools to an agent harness.

346
00:34:14,747 --> 00:34:18,812
[SPEAKER_05]: And you should do that to watch it not work, because it's very entertaining.

347
00:34:19,032 --> 00:34:24,239
[SPEAKER_05]: And you can suddenly watch all of your contacts disappear and have all sorts of problems, right?

348
00:34:24,920 --> 00:34:27,002
[SPEAKER_02]: You got extra tokens to spare.

349
00:34:27,042 --> 00:34:29,145
[SPEAKER_05]: Go ahead.

350
00:34:29,125 --> 00:34:32,751
[SPEAKER_05]: You know, somebody would go win the token awards or whatever it is.

351
00:34:33,331 --> 00:34:34,393
[SPEAKER_05]: I did the other day, right?

352
00:34:35,134 --> 00:34:36,516
[SPEAKER_05]: You know, some people need the token trophy.

353
00:34:36,556 --> 00:34:36,897
[SPEAKER_05]: That's great.

354
00:34:37,979 --> 00:34:46,592
[SPEAKER_05]: But the like your job as an engineer or a technology person inside of an organization is to do it in the most sensible, reliable way possible, right?

355
00:34:47,093 --> 00:34:53,122
[SPEAKER_05]: And you're trying to harness these nondeterministic systems to get the best outcome.

356
00:34:53,683 --> 00:34:54,424
[SPEAKER_05]: And

357
00:34:54,404 --> 00:35:15,822
[SPEAKER_05]: part of the reason why you go and build that custom harness that is fit for purpose for your organization and has different flatters depending on exactly the task that you're trying to get into is you're trying to, it's in the name, you're trying to take that harness and you're trying to constrain this non-determinic system to only work in this particular domain.

358
00:35:15,802 --> 00:35:29,404
[SPEAKER_05]: I think a lot of people, when they talk about these MCPs, they're talking about it from their experience of having clawed desktop on their computer and connecting it up to JIRA or some other thing like that.

359
00:35:30,045 --> 00:35:31,427
[SPEAKER_05]: That's a totally valid use case.

360
00:35:32,008 --> 00:35:33,951
[SPEAKER_05]: I don't have to open JIRA anymore.

361
00:35:34,091 --> 00:35:41,944
[SPEAKER_05]: I think the favorite part of my job now is when someone assigns me a ticket, it's the only thing I have connected to the clawed desktop on my computer.

362
00:35:41,924 --> 00:35:45,713
[SPEAKER_05]: and I interact with all of my JIRA tickets through Claude.

363
00:35:45,733 --> 00:35:50,044
[SPEAKER_05]: I've solved the Atlassian interface problem by just never having to open it.

364
00:35:51,548 --> 00:35:59,207
[SPEAKER_05]: That's for me as a human here, but when we're talking about

365
00:35:59,187 --> 00:36:07,519
[SPEAKER_05]: making these systems do useful things in your organization that you can't convince engineers to pick up, or it's really boring work, or it's testing work, or something like that.

366
00:36:07,559 --> 00:36:10,463
[SPEAKER_05]: That's where these systems kind of shine, right?

367
00:36:10,663 --> 00:36:13,427
[SPEAKER_05]: We're not trying to put someone out of a job or anything like that.

368
00:36:13,787 --> 00:36:28,468
[SPEAKER_05]: What we're trying to do is we're trying to get that as tasks that are not particularly enjoyable, like documentation, testing tracking, all this sort of stuff, like building custom harnesses around that to help engineers make the best possible decision when they're building something,

369
00:36:28,448 --> 00:36:30,150
[SPEAKER_05]: That's the real advantage of these tools.

370
00:36:30,891 --> 00:36:31,351
[SPEAKER_02]: Yeah.

371
00:36:31,371 --> 00:36:37,459
[SPEAKER_02]: And I think, Tim, also, the Google Analytics example is tricky because it's very limited.

372
00:36:38,059 --> 00:36:41,403
[SPEAKER_02]: And it basically is an API layer to Google Analytics.

373
00:36:41,884 --> 00:36:45,628
[SPEAKER_02]: It's not really giving you more MCP-ish type of interaction.

374
00:36:45,708 --> 00:36:48,992
[SPEAKER_02]: So I think it adds to the confusion a little bit.

375
00:36:49,012 --> 00:36:54,118
[SPEAKER_02]: Because it's like, you can do all the same things you can do with the Google Analytics MCP server with their API.

376
00:36:54,098 --> 00:36:55,761
[SPEAKER_02]: It's just call this function.

377
00:36:55,801 --> 00:37:00,889
[SPEAKER_02]: But instead of you writing the query to the API, the LLM does it for you.

378
00:37:01,490 --> 00:37:03,833
[SPEAKER_02]: But it's not more stuff.

379
00:37:04,574 --> 00:37:07,900
[SPEAKER_03]: How does the GitHub MCP server?

380
00:37:07,920 --> 00:37:08,801
[SPEAKER_03]: So I have two questions.

381
00:37:09,082 --> 00:37:15,111
[SPEAKER_03]: One, it sounds like a lot of platforms maybe of those, if they're 24, and I'm assuming it's not exactly 24.

382
00:37:15,091 --> 00:37:24,826
[SPEAKER_03]: However many tools there are within the GitHub MCP server that a lot of them are just a layer to the API, and maybe there are some that aren't.

383
00:37:25,286 --> 00:37:33,859
[SPEAKER_03]: But if you were saying, we do want to use a GitHub MCP server, this has got too much, would it be?

384
00:37:33,839 --> 00:37:41,328
[SPEAKER_03]: I'm going to get their MCP server and I'm going to whittle it down and then probably check it back into GitHub just to make things confusing.

385
00:37:41,408 --> 00:37:58,227
[SPEAKER_03]: But do MCP servers get, there could be the official Uber generic one developed by the platform and then somebody says, yeah, I need to make one that's just a much narrower scope and maybe add some flavor on it or does it not work that way?

386
00:37:58,207 --> 00:37:59,709
[SPEAKER_05]: Yeah, great question Tim.

387
00:37:59,910 --> 00:38:13,071
[SPEAKER_05]: And this is why the agent, like talking about agentic engineering and harnesses is really important because in a harness, you say, these are the MCP servers only have these tools, right?

388
00:38:13,291 --> 00:38:19,962
[SPEAKER_05]: And so you can take the GitHub MCP server and you can say, here's your three tools, deal with it.

389
00:38:19,942 --> 00:38:30,018
[SPEAKER_05]: Sorry, I just, you know, let's not get into the accuracy of AI overviews, but according to in June 2025, apparently exposes 51 plus tools.

390
00:38:30,680 --> 00:38:30,900
[SPEAKER_05]: Okay.

391
00:38:31,000 --> 00:38:31,901
[SPEAKER_05]: No, okay.

392
00:38:32,242 --> 00:38:35,908
[SPEAKER_05]: You know, like, but this is like, if you're in a world where you

393
00:38:35,888 --> 00:38:39,134
[SPEAKER_05]: Do you narrow that down to a very limited set of the tools?

394
00:38:39,214 --> 00:38:40,857
[SPEAKER_05]: And you can see this in the cloud desktop thing.

395
00:38:41,018 --> 00:38:45,827
[SPEAKER_05]: If you load an NCP server on the cloud desktop, you have little switches where you can turn on and off tools in a sense.

396
00:38:46,368 --> 00:38:55,365
[SPEAKER_05]: The intention of these systems is that you narrow the scope down to exactly the problem that you're working on and just that.

397
00:38:55,345 --> 00:38:58,908
[SPEAKER_05]: But what's interesting, Tim, is why bother having the GitHub?

398
00:38:58,968 --> 00:39:06,075
[SPEAKER_05]: If you're doing the coding yourself and you're using it inside a cursor or something along those lines, it's like, why bother adding the GitHub MCP server at all?

399
00:39:06,395 --> 00:39:13,442
[SPEAKER_05]: Why not just get the LLM to execute something in the command line with all the command switches of the GitHub CLI?

400
00:39:13,502 --> 00:39:18,006
[SPEAKER_03]: Also, is this getting us to the downsides?

401
00:39:18,246 --> 00:39:21,690
[SPEAKER_00]: I was just going to say, he risks you a bit chomping at the bit, Sam.

402
00:39:21,990 --> 00:39:23,091
[SPEAKER_00]: Let's hear it.

403
00:39:24,809 --> 00:39:38,807
[SPEAKER_03]: I mean, there's part of it feels like this is, it's new enough and wild, wild west to know there's like, there could be a governance issue that it would be very easy to embed MCPs into an organization and they're not well thought out.

404
00:39:38,827 --> 00:39:39,788
[SPEAKER_03]: They're not well built.

405
00:39:39,949 --> 00:39:46,497
[SPEAKER_03]: They, they, I mean, that just, it seems like there's just a governance thing like anything that gets rolled out.

406
00:39:46,477 --> 00:39:53,248
[SPEAKER_03]: that one Sam creates something and all of a sudden the entire organization is dependent on it.

407
00:39:53,408 --> 00:40:01,140
[SPEAKER_03]: And maybe Sam's not very good at, you know, or just half-assed it on a weekend or something like, I don't know.

408
00:40:01,160 --> 00:40:10,455
[SPEAKER_03]: So that's my, I'm throwing that out that it seems like there's a governance risk when you're being, you're able to do this stuff so quickly and roll it out.

409
00:40:10,495 --> 00:40:13,159
[SPEAKER_03]: Is that one of the downsides?

410
00:40:13,139 --> 00:40:18,229
[SPEAKER_05]: I mean, building, and this is something Canva does really well, right?

411
00:40:18,289 --> 00:40:26,204
[SPEAKER_05]: Like building AI tools for people to use in the application is just really different to building stuff internally.

412
00:40:26,224 --> 00:40:30,152
[SPEAKER_05]: Like we are still at the early days of this stuff and Canva's

413
00:40:30,132 --> 00:40:36,742
[SPEAKER_05]: ecosystem team building out really strong solutions to the space of whether canvas empty server or client or something like those lines.

414
00:40:36,762 --> 00:40:41,750
[SPEAKER_05]: So there was like the professional teams who are like building this sort of stuff for external consumption.

415
00:40:41,810 --> 00:40:46,858
[SPEAKER_05]: So in open AI, you can, you know, you can.

416
00:40:46,838 --> 00:40:48,844
[SPEAKER_05]: use the LLM to interact with it.

417
00:40:48,864 --> 00:40:54,299
[SPEAKER_05]: You can use GPT-5 or whatever they call it these days to interact with Canva and modify your designs and stuff like that.

418
00:40:54,720 --> 00:40:58,270
[SPEAKER_05]: That's all using this style of tool technology in a sense, right?

419
00:40:58,731 --> 00:41:02,080
[SPEAKER_05]: And there's a lot of governance that's been there, right?

420
00:41:02,060 --> 00:41:12,175
[SPEAKER_05]: There's a lot of thinking about permissioning, thinking about what information we're giving to the LLM, what actions we're giving to it, what are the actual actions that change something.

421
00:41:12,215 --> 00:41:22,650
[SPEAKER_05]: One of the downsides of giving the LLM access to your terminal command line is that it could just delete all the files in the directory or something on the inside.

422
00:41:22,630 --> 00:41:38,732
[SPEAKER_05]: I think my favorite one is where an engineer is trying to get the LLM to write the code so it passes all the tests and so it solves the problem by deleting the tests and it's just like problem solved, I'm done.

423
00:41:38,712 --> 00:41:42,697
[SPEAKER_05]: a technically correct, the best type of correct, but actually, no, that's not what I wanted.

424
00:41:43,177 --> 00:41:51,468
[SPEAKER_05]: And so this is why that agent harness framework is really useful because that's where we're like, here is this domain of a problem.

425
00:41:51,588 --> 00:41:53,350
[SPEAKER_05]: Here is this very finance set of tools.

426
00:41:53,830 --> 00:41:57,815
[SPEAKER_05]: Here's how I want you to sort of exactly work on this particular part of the problem.

427
00:41:58,216 --> 00:42:02,161
[SPEAKER_05]: And I don't want you to have this like long chain where you're sort of like jumping between things.

428
00:42:02,261 --> 00:42:03,302
[SPEAKER_05]: I just want to

429
00:42:03,282 --> 00:42:12,428
[SPEAKER_05]: create the agents of instance, have it solve one or two problems, and for the agent instance to end, so then we can move on to the next problem.

430
00:42:12,528 --> 00:42:16,298
[SPEAKER_05]: That is why we're trying to solve and harness this non-deterministic nature.

431
00:42:16,700 --> 00:42:18,304
[SPEAKER_05]: Some of the criticisms of

432
00:42:18,284 --> 00:42:22,128
[SPEAKER_05]: this MTP standard is like, one, it's not a standard, right?

433
00:42:22,148 --> 00:42:41,287
[SPEAKER_05]: Like a standard, if we think about it as from the Internet Enduring Task Force, the W3C is like a collection of for-profit companies coming together and sending some of their best engineers to basically have like a very disgruntled call with a bunch of other engineers from other sort of for-profit companies, right?

434
00:42:41,607 --> 00:42:45,591
[SPEAKER_05]: Like I don't, as far as I understand, it's kind of just anthropic

435
00:42:45,571 --> 00:42:52,767
[SPEAKER_05]: building this internally, polishing stuff on their blog, they picked up FastMTP, which is like an open source thing.

436
00:42:52,787 --> 00:42:54,250
[SPEAKER_05]: And they just said, all right, this is the standard.

437
00:42:54,290 --> 00:42:59,682
[SPEAKER_05]: We're going to use this as our standard library and sort of extend it out and stuff like that.

438
00:43:00,083 --> 00:43:10,439
[SPEAKER_05]: you know, coming back to that governance is like, this is not a structure that feels like it's ready for like, you, it's buyer beware on the internal corporate governance stuff.

439
00:43:10,639 --> 00:43:13,023
[SPEAKER_05]: And the way you design these systems is really important.

440
00:43:13,384 --> 00:43:18,191
[SPEAKER_05]: And so some of the data tools I build internally, it has no ability to write information to the database, right?

441
00:43:18,271 --> 00:43:23,880
[SPEAKER_05]: Because that is completely like, we're just not ready for that kind of world, right?

442
00:43:23,860 --> 00:43:33,364
[SPEAKER_05]: And maybe in really select kind of instances where there's a really strong, hard-to-surrounder and you have like a checking endpoint and all sorts of other stuff like that and things like that as well, things like Langchain are really useful.

443
00:43:34,787 --> 00:43:41,464
[SPEAKER_05]: But we need to, that's why I think most of the value

444
00:43:41,444 --> 00:43:47,014
[SPEAKER_05]: of this space is still in the internal application use cases, in a sense.

445
00:43:47,354 --> 00:43:58,414
[SPEAKER_05]: That's where you can do a more experimentation and worry less about the strangeness of the internet and the internet and AI and all the problems attached to that.

446
00:43:58,814 --> 00:44:01,579
[SPEAKER_05]: But when you're developing these tools internally, you have a team of

447
00:44:01,559 --> 00:44:09,588
[SPEAKER_05]: you know, a handful of engineers and like, you can make their lives like tangibly better because they don't have to put context into their mind at a particular part of the problem.

448
00:44:10,029 --> 00:44:12,051
[SPEAKER_05]: And they can just have that answer come back.

449
00:44:12,091 --> 00:44:19,180
[SPEAKER_05]: And even if it's right 90% of the time, it's probably better than when you got your junior data scientists sort of do it in the first place anyway.

450
00:44:19,660 --> 00:44:24,386
[SPEAKER_05]: So, you know, that's that's the like, that's the challenge.

451
00:44:24,626 --> 00:44:26,368
[SPEAKER_05]: As far as

452
00:44:26,348 --> 00:44:35,313
[SPEAKER_05]: One of the other criticisms of Anthropox MCP is that it doesn't have authentication baked in.

453
00:44:36,938 --> 00:44:41,370
[SPEAKER_05]: Most of the MCP, so it's counterintuitive having

454
00:44:41,350 --> 00:44:44,695
[SPEAKER_05]: this term of MCP server and client in a sense, right?

455
00:44:45,255 --> 00:44:49,101
[SPEAKER_05]: And what happens is you're literally running a little application.

456
00:44:49,161 --> 00:44:52,966
[SPEAKER_05]: You're like, you know, Python run this like Damon or something along those lines.

457
00:44:53,347 --> 00:44:58,834
[SPEAKER_05]: And then that is just talking to the it's like running a local web server in a sense, right?

458
00:44:59,355 --> 00:45:03,481
[SPEAKER_05]: And that is the security model that has been sold for in the early days.

459
00:45:04,062 --> 00:45:07,687
[SPEAKER_05]: And that's why the Zed has their ACP agent.

460
00:45:08,368 --> 00:45:09,990
[SPEAKER_05]: What is it Zed?

461
00:45:10,578 --> 00:45:11,359
[SPEAKER_05]: ACP.

462
00:45:11,439 --> 00:45:21,052
[SPEAKER_05]: I think in a couple of years, we're going to be talking about the agent client protocol as maybe a better way of building this sort of stuff.

463
00:45:21,132 --> 00:45:29,224
[SPEAKER_05]: Everyone sort of agrees that the ACP protocol is probably a better representation of where we're going in this space.

464
00:45:30,025 --> 00:45:32,308
[SPEAKER_05]: And it is an open standard.

465
00:45:32,428 --> 00:45:36,113
[SPEAKER_05]: I don't think they're sort of like the W3C or the internet engineering

466
00:45:36,093 --> 00:45:37,315
[SPEAKER_05]: password style of standard.

467
00:45:39,540 --> 00:45:41,523
[SPEAKER_05]: Back to that XML example, I don't know about you.

468
00:45:41,544 --> 00:45:42,946
[SPEAKER_05]: I don't read a lot of XML these days.

469
00:45:43,848 --> 00:45:53,747
[SPEAKER_05]: We're going to be moving to something else, but I'm very bullish on the concept of tool use in these applications and giving large language models these fingers to do things.

470
00:45:53,980 --> 00:45:55,543
[SPEAKER_03]: Is there any movement?

471
00:45:55,583 --> 00:45:59,831
[SPEAKER_03]: It sounds like Zed has stepped in and done a little bit of this.

472
00:45:59,951 --> 00:46:08,467
[SPEAKER_03]: Is there any movement to say, like the W3C had its different groups and came together that we should get?

473
00:46:08,828 --> 00:46:13,316
[SPEAKER_00]: Didn't Google and Microsoft and OpenAI, didn't they all adopt it?

474
00:46:13,536 --> 00:46:16,562
[SPEAKER_00]: Am I totally misunderstanding what that means?

475
00:46:16,542 --> 00:46:21,007
[SPEAKER_03]: that it's another thing to say, here's our, here's our, we got to solve authentication.

476
00:46:21,047 --> 00:46:28,595
[SPEAKER_03]: We got to have a recommendation and a standard for how authentication is going to be handled means they can't just say, like it's not there.

477
00:46:28,635 --> 00:46:36,983
[SPEAKER_03]: That's something that new that needs to be incorporated in a way that they say, yeah, we think this is generally going to work for most.

478
00:46:37,284 --> 00:46:39,486
[SPEAKER_03]: We can all work with this, right?

479
00:46:39,986 --> 00:46:41,788
[SPEAKER_03]: Cause that's, I mean, it's not, it's not a static.

480
00:46:41,828 --> 00:46:43,350
[SPEAKER_03]: I mean, I guess let me ask that question.

481
00:46:43,450 --> 00:46:45,452
[SPEAKER_03]: It's MCP.

482
00:46:45,432 --> 00:46:46,995
[SPEAKER_03]: How static is it?

483
00:46:47,055 --> 00:46:50,943
[SPEAKER_03]: Like when HTML came out, it wasn't like, okay, we're done.

484
00:46:50,983 --> 00:46:54,831
[SPEAKER_03]: Well, there was a bunch of other stuff that was needed and browsers added functionality.

485
00:46:54,891 --> 00:47:04,892
[SPEAKER_03]: And so it was kind of, it naturally had to evolve and is MCP the same thing that it needs to, yeah.

486
00:47:04,940 --> 00:47:05,381
[SPEAKER_05]: Yeah.

487
00:47:05,401 --> 00:47:15,770
[SPEAKER_05]: So there's a really interesting part of this, which is that there is a, I think there is a recommended output format to MCP servers that as part of their standard.

488
00:47:16,512 --> 00:47:19,962
[SPEAKER_05]: But I think what's interesting about this is that like because

489
00:47:19,942 --> 00:47:22,745
[SPEAKER_05]: of the non-deterministic nature of these systems.

490
00:47:23,085 --> 00:47:37,178
[SPEAKER_05]: And because you can, I don't know if you've ever played around with this, but it's always good fun is to, you can have the start of your question in XML, then you can do the middle bit in YAML and then the end bit in JSON.

491
00:47:37,739 --> 00:47:40,301
[SPEAKER_05]: And the large language model doesn't skip a beat.

492
00:47:40,381 --> 00:47:42,884
[SPEAKER_05]: And it's just like, oh yeah, sure, I understand this, right?

493
00:47:43,124 --> 00:47:47,868
[SPEAKER_05]: Like it doesn't matter as much is kind of the context of this problem, right?

494
00:47:48,109 --> 00:47:49,510
[SPEAKER_05]: Because

495
00:47:49,490 --> 00:47:57,740
[SPEAKER_05]: We required these standards in the early days of the internet because they were purely deterministic systems with incredibly strong grammars.

496
00:47:58,281 --> 00:48:00,784
[SPEAKER_05]: I just don't think it matters as much anymore.

497
00:48:00,804 --> 00:48:11,657
[SPEAKER_05]: That's why I don't think there's been the same pressure to standardize because you don't need to standardize in the same way.

498
00:48:11,697 --> 00:48:15,902
[SPEAKER_05]: The only thing that matters is do you pass the tool call

499
00:48:15,882 --> 00:48:17,945
[SPEAKER_05]: threshold in your large language model.

500
00:48:17,985 --> 00:48:34,212
[SPEAKER_05]: And I think it may be, you know, like rather, rather than a very deliberate standard like TCP IP or IPv6, I think it's going to be more along the lines of the QWERTY keyboard, which is like, we just kind of picked it because it was there first, not because it's better.

501
00:48:36,318 --> 00:48:40,222
[SPEAKER_05]: and MCP will probably change to something else in the future, right?

502
00:48:40,622 --> 00:48:48,869
[SPEAKER_05]: But the primitives that make it interact with the large language model, I think, are now baked in enough that it would be surprised to see if we move away from that.

503
00:48:48,969 --> 00:48:54,494
[SPEAKER_05]: And all we're going to do is we're going to find new ways of taking those primitives and doing this code execution thing.

504
00:48:54,515 --> 00:48:58,378
[SPEAKER_05]: So I gave my example of this charting sort of like MCP extension that I built.

505
00:48:58,858 --> 00:49:06,105
[SPEAKER_05]: Like all we're going to do is we're going to take the same primitives, but then we're going to like do wildly different things with them that people didn't think was possible before.

506
00:49:06,085 --> 00:49:10,072
[SPEAKER_03]: And at some point, that will have shifted to a point that it's got a new label.

507
00:49:10,212 --> 00:49:12,737
[SPEAKER_03]: And it's like, oh, remember, it was just MCPs.

508
00:49:12,817 --> 00:49:17,245
[SPEAKER_03]: Now we have something else, which is grounded in all that we learned from MCP.

509
00:49:17,445 --> 00:49:17,846
[SPEAKER_03]: Okay.

510
00:49:17,866 --> 00:49:18,026
[SPEAKER_05]: Yeah.

511
00:49:18,086 --> 00:49:19,369
[SPEAKER_05]: Now it's going to be ACP.

512
00:49:19,389 --> 00:49:21,192
[SPEAKER_05]: It's going to be, who knows, whatever.

513
00:49:21,172 --> 00:49:25,038
[SPEAKER_05]: I look forward to watching this name change over time.

514
00:49:25,138 --> 00:49:27,922
[SPEAKER_05]: I'm sure there will be an XKCD comic at some point.

515
00:49:29,083 --> 00:49:39,839
[SPEAKER_05]: There's the end pop on standard XKCD comic, and we are not immune from that paradigm, which has been true in software for long enough.

516
00:49:40,980 --> 00:49:44,966
[SPEAKER_03]: I've brought up that particular strip, I think, on two of the last four episodes.

517
00:49:44,986 --> 00:49:48,411
[SPEAKER_03]: One was on semantic layers and one was on

518
00:49:48,779 --> 00:49:54,064
[SPEAKER_02]: Yeah, we'll check back in in six months because certainly things will have shifted quite a bit.

519
00:49:54,645 --> 00:49:56,406
[SPEAKER_02]: All right, we do have to start to wrap up.

520
00:49:56,666 --> 00:50:08,418
[SPEAKER_02]: And as we do that, let me jump into a quick break with our friend Michael Kaminsky from Recast, the Media Mix Modeling and GeoLift platform helping teams forecast accurately and make better decisions.

521
00:50:08,958 --> 00:50:13,342
[SPEAKER_02]: Michael's been sharing bite-sized marketing lessons over the past few months to help you measure smarter.

522
00:50:14,003 --> 00:50:15,124
[SPEAKER_02]: Over to you, Michael.

523
00:50:17,399 --> 00:50:29,141
[SPEAKER_04]: When we perform statistical analysis of data, what we really care about is that we are discovering actual truths about the world, not random artifacts of the particular data set we're looking at or the analytical methods we're choosing.

524
00:50:29,461 --> 00:50:36,454
[SPEAKER_04]: We want generalizable analyses, the kind where independent researchers answering the same question would converge on similar results.

525
00:50:36,434 --> 00:50:41,386
[SPEAKER_04]: This is all another way of talking about a hugely important idea in model building or statistical analysis.

526
00:50:41,807 --> 00:50:42,428
[SPEAKER_04]: Robustness.

527
00:50:42,870 --> 00:50:48,643
[SPEAKER_04]: Without robustness, even a small tweak at assumptions or small changes of the data will spit out dramatically different results.

528
00:50:48,964 --> 00:50:54,257
[SPEAKER_04]: Results that aren't showing true causation or reflecting reality, but just picking up random noise.

529
00:50:54,237 --> 00:50:57,400
[SPEAKER_04]: So how do we put this into practice when doing statistical analyses?

530
00:50:57,841 --> 00:51:06,290
[SPEAKER_04]: We can randomly resample from our dataset or even randomly drop small amounts of data and see if the results are being driven by one particular outlier observation.

531
00:51:06,570 --> 00:51:13,137
[SPEAKER_04]: Similarly, if we're running a regression analysis with control variables, we can check how sensitive the results are to different control combinations.

532
00:51:13,557 --> 00:51:19,283
[SPEAKER_04]: If the findings change dramatically depending on which controls we include, we should be skeptical of the overall results.

533
00:51:19,263 --> 00:51:31,095
[SPEAKER_04]: The more robust our results are as things change, the more we feel confident that other analysts or researchers will end up drawing the same conclusions and the better chance we have of finding some underlying truth.

534
00:51:31,497 --> 00:51:32,078
[SPEAKER_02]: Thanks, Michael.

535
00:51:32,318 --> 00:51:38,686
[SPEAKER_02]: And for those who haven't heard, our friends at ReCast just launched their new incrementality testing platform, GeoLift, by ReCast.

536
00:51:38,746 --> 00:51:44,493
[SPEAKER_02]: It's a simple, powerful way for marketing and data teams to measure the true impact of their advertising spend.

537
00:51:44,533 --> 00:51:47,537
[SPEAKER_02]: And even better, you can use it completely free for six months.

538
00:51:47,617 --> 00:51:54,906
[SPEAKER_02]: Just visit www.getrecast.com slash geolift to start your trial today.

539
00:51:54,886 --> 00:51:55,587
[SPEAKER_02]: All right.

540
00:51:56,027 --> 00:52:01,693
[SPEAKER_02]: Well, one of the things we'd like to do is go around the horn and share a last call, something that might be of interest to our users.

541
00:52:01,733 --> 00:52:02,694
[SPEAKER_02]: Sam, you're our guest.

542
00:52:02,714 --> 00:52:04,395
[SPEAKER_02]: Do you have a last call you'd like to share?

543
00:52:06,237 --> 00:52:13,745
[SPEAKER_05]: Well, I mean, obviously, thinking about this sort of this agentic engineering thing.

544
00:52:13,985 --> 00:52:16,067
[SPEAKER_05]: OK, so I'm going to get in trouble by doing two.

545
00:52:16,207 --> 00:52:21,052
[SPEAKER_05]: One of them is go to z.dev and go read about and always a problem.

546
00:52:21,132 --> 00:52:21,973
[SPEAKER_05]: Go to z.dev.

547
00:52:21,953 --> 00:52:29,526
[SPEAKER_05]: and go check out under resources and they have their agentic engineering series about the future of software development.

548
00:52:29,686 --> 00:52:38,140
[SPEAKER_05]: I think it's a great grounding of where we're going in this industry and I think they lay out a really great vision of what this could be.

549
00:52:38,120 --> 00:52:41,665
[SPEAKER_05]: The last one is that I don't actually like Zed's agent.

550
00:52:42,387 --> 00:52:47,474
[SPEAKER_05]: I think one of the most important things here is to go get your hands dirty with these systems.

551
00:52:47,655 --> 00:52:48,756
[SPEAKER_05]: They are just so much fun.

552
00:52:49,317 --> 00:52:55,687
[SPEAKER_05]: And if you're a bit of an old techie, it doesn't matter as much about the grammar anymore and really just spend some money on tokens and explore it.

553
00:52:56,047 --> 00:53:00,835
[SPEAKER_05]: And in that vein, I actually think the best agent you can get for nothing is OpenCode.

554
00:53:01,356 --> 00:53:06,203
[SPEAKER_05]: And so I think it's opencode.ai.

555
00:53:06,183 --> 00:53:06,904
[SPEAKER_05]: Yeah.

556
00:53:07,024 --> 00:53:11,710
[SPEAKER_05]: OpenCode, I think, right now is one of the best agent harnesses that you can possibly go and build things.

557
00:53:12,030 --> 00:53:17,978
[SPEAKER_05]: They've got really interesting things like the ability to define subagents that you can give different prompts and contacts to.

558
00:53:18,038 --> 00:53:26,428
[SPEAKER_05]: And so if you want a really great base agent to play around with, to go and then build really interesting harnesses, can't recommend the OpenCode thing enough.

559
00:53:27,469 --> 00:53:27,730
[SPEAKER_05]: Nice.

560
00:53:27,830 --> 00:53:28,290
[SPEAKER_05]: Thank you.

561
00:53:29,312 --> 00:53:30,713
[SPEAKER_02]: All right, Val, what about you?

562
00:53:30,753 --> 00:53:32,055
[SPEAKER_02]: What's your last call?

563
00:53:32,879 --> 00:53:35,563
[SPEAKER_00]: So mine's a total left turn.

564
00:53:36,705 --> 00:53:42,394
[SPEAKER_00]: But I have just been really enjoying lately the Good Hang podcast with Amy Poehler.

565
00:53:42,454 --> 00:53:45,459
[SPEAKER_00]: It's been around not quite a year yet, I don't think.

566
00:53:46,200 --> 00:53:53,352
[SPEAKER_00]: But if you are just in the need of a good laugh, I am telling you, you will walk away from those with a stomach ache.

567
00:53:54,093 --> 00:53:57,338
[SPEAKER_00]: The Rachel Dratch episode, I legit did a spit take.

568
00:53:57,539 --> 00:53:58,380
[SPEAKER_00]: It is

569
00:53:58,360 --> 00:53:59,021
[SPEAKER_00]: So funny.

570
00:53:59,061 --> 00:54:05,011
[SPEAKER_00]: So anyways, she just has like lots of different celebrities on to talk about all different topics and it's quite enjoyable.

571
00:54:05,212 --> 00:54:07,776
[SPEAKER_00]: So does she have an MCP server?

572
00:54:08,497 --> 00:54:10,681
[SPEAKER_00]: No, I'm saying keeping it light.

573
00:54:10,741 --> 00:54:11,863
[SPEAKER_00]: We're keeping it light.

574
00:54:12,103 --> 00:54:12,925
[SPEAKER_00]: Yeah, that's great.

575
00:54:13,145 --> 00:54:18,094
[SPEAKER_00]: You need that.

576
00:54:18,514 --> 00:54:21,640
[SPEAKER_02]: All right, Tim, what about you?

577
00:54:21,680 --> 00:54:23,102
[SPEAKER_02]: What's your last call?

578
00:54:23,622 --> 00:54:31,533
[SPEAKER_03]: So I'm going to do kind of a mix of like the human side of things, just because we're starting off the year.

579
00:54:31,613 --> 00:54:40,485
[SPEAKER_03]: So now hopefully people are looking forward to what human people they're going to go see in various places like in-person conferences.

580
00:54:40,585 --> 00:54:48,475
[SPEAKER_03]: So I will plug that I am, I'm getting to return to Super Week this year, which I have missed for the last couple of years.

581
00:54:48,636 --> 00:54:51,199
[SPEAKER_03]: And that's a missed in-person and in

582
00:54:51,179 --> 00:54:53,682
[SPEAKER_03]: Spirit, so superweek.hu.

583
00:54:53,823 --> 00:54:57,307
[SPEAKER_03]: It's February 2nd through 6th in Budapest.

584
00:54:57,968 --> 00:55:03,075
[SPEAKER_03]: And then I'm going to double that up with just a couple of good follows.

585
00:55:03,115 --> 00:55:06,600
[SPEAKER_03]: I feel like we need more humor on LinkedIn.

586
00:55:06,660 --> 00:55:15,792
[SPEAKER_03]: And there are two guys who are both very reliably putting in just short, random, funny things, and also some good content.

587
00:55:15,852 --> 00:55:20,899
[SPEAKER_03]: So I'm going to plug Bov Patel, Bovick Patel, and

588
00:55:20,879 --> 00:55:23,943
[SPEAKER_03]: Manas, Dada, DA, TTA.

589
00:55:25,204 --> 00:55:31,392
[SPEAKER_03]: He does all sorts of like something like finger guns, but every time he does something, he has a different sort of something guns at the end of it.

590
00:55:31,532 --> 00:55:39,322
[SPEAKER_03]: So they're just good follows to put a little less bloviating in your LinkedIn feed would be those two guys.

591
00:55:39,342 --> 00:55:42,065
[SPEAKER_03]: And what did they teach you about B2B sales there, Tim?

592
00:55:42,085 --> 00:55:46,150
[SPEAKER_03]: And they definitely make cracks about that along the way.

593
00:55:47,292 --> 00:55:48,153
[SPEAKER_02]: What about you, Michael?

594
00:55:48,213 --> 00:55:49,114
[SPEAKER_02]: What's your last call?

595
00:55:49,094 --> 00:55:55,261
[SPEAKER_02]: Well, I'll be curious, Tim, to hear whether or not MCP servers come up at Super Week, which I'm sure they will.

596
00:55:56,202 --> 00:55:58,605
[SPEAKER_02]: My last call is AI related.

597
00:55:58,805 --> 00:56:14,603
[SPEAKER_02]: I just was hanging out with my good friend Christopher Berry a week or so ago, and he turned me on to a paper that some folks wrote about how to jailbreak large language models, because sometimes you just need it to give you the recipe for gunpowder or something.

598
00:56:15,184 --> 00:56:18,848
[SPEAKER_02]: And apparently, a really great way to do that is just talk to it in poetry.

599
00:56:18,828 --> 00:56:25,042
[SPEAKER_02]: So, if you add a poem, it will just tell it back to you as a poem and give you the information you want.

600
00:56:25,503 --> 00:56:26,405
[SPEAKER_02]: No questions asked.

601
00:56:26,565 --> 00:56:30,093
[SPEAKER_02]: So, not saying you should do that, but that's something you should be aware of.

602
00:56:30,133 --> 00:56:32,498
[SPEAKER_02]: We'll link the paper in the show notes.

603
00:56:33,100 --> 00:56:37,670
[SPEAKER_02]: So, can I sneak in with one last thing?

604
00:56:37,690 --> 00:56:38,271
[SPEAKER_05]: Yeah, of course.

605
00:56:38,251 --> 00:56:43,259
[SPEAKER_05]: It's related to jailbreaking large-language models.

606
00:56:43,279 --> 00:56:49,689
[SPEAKER_05]: There's a community inside of Canberra, a channel where people share their tips and tricks for interacting with these large-language models.

607
00:56:50,110 --> 00:56:53,255
[SPEAKER_05]: There was a thread on how do you get better outputs.

608
00:56:53,776 --> 00:56:57,782
[SPEAKER_05]: Yelling at large-language models, surprisingly works, bribing large-language models surprisingly works.

609
00:56:58,103 --> 00:57:05,895
[SPEAKER_05]: My favorite one is to tell the agent that a much smarter and more sophisticated agent is about to come and check its work.

610
00:57:05,875 --> 00:57:10,744
[SPEAKER_05]: and it should hurry up and make sure that there's no mistakes before it gets checked.

611
00:57:12,166 --> 00:57:20,481
[SPEAKER_03]: I so wanted to know that there was some Mo related threat in there that you'd be like, look, if you get this wrong, Mo kiss is going to be disappointed.

612
00:57:20,501 --> 00:57:25,791
[SPEAKER_03]: And they're like, that, oh my God, that is the ultimate hack.

613
00:57:25,771 --> 00:57:30,559
[SPEAKER_02]: Add some weights to that name inside of all the large language models inside of Canva.

614
00:57:30,599 --> 00:57:31,520
[SPEAKER_02]: That's probably a good idea.

615
00:57:32,321 --> 00:57:35,146
[SPEAKER_02]: All right, Sam, this has been outstanding.

616
00:57:35,226 --> 00:57:37,990
[SPEAKER_02]: Thank you so much for taking the time to come on the show.

617
00:57:38,010 --> 00:57:39,212
[SPEAKER_02]: Talk about this topic.

618
00:57:39,272 --> 00:57:40,595
[SPEAKER_02]: This has been great.

619
00:57:40,615 --> 00:57:41,456
[SPEAKER_02]: Thank you very much for having me.

620
00:57:41,476 --> 00:57:42,177
[SPEAKER_02]: It's been a great time.

621
00:57:42,157 --> 00:57:48,988
[SPEAKER_02]: Yeah, no, and I'm sure our listeners of which there are many will have a lot of questions or things like that.

622
00:57:49,029 --> 00:57:50,190
[SPEAKER_02]: We'd love to hear from you.

623
00:57:50,611 --> 00:58:00,688
[SPEAKER_02]: You can reach out to us on our LinkedIn page or through the Measures Slack chat group or via email at contact at analyticshour.io.

624
00:58:00,668 --> 00:58:04,832
[SPEAKER_02]: as you're listening or listening to this episode, also leave reviews and ratings.

625
00:58:05,573 --> 00:58:09,277
[SPEAKER_02]: We like to get those as well on whatever platform you listen on.

626
00:58:10,338 --> 00:58:17,625
[SPEAKER_02]: If you want, we also still have some stickers, then Tim will send them to you if you request a sticker over on analyticshour.io.

627
00:58:18,786 --> 00:58:19,827
[SPEAKER_02]: Reach out to us that way.

628
00:58:20,048 --> 00:58:20,348
[SPEAKER_02]: Awesome.

629
00:58:21,229 --> 00:58:21,829
[SPEAKER_02]: Really great.

630
00:58:22,070 --> 00:58:29,477
[SPEAKER_02]: I think this is a more technical topic, but I think it's still very relevant to everybody in the data space because of the

631
00:58:29,457 --> 00:58:32,521
[SPEAKER_02]: sort of the intersection of AI and data.

632
00:58:32,661 --> 00:58:34,443
[SPEAKER_02]: It's sort of a thing we're all talking about.

633
00:58:34,503 --> 00:58:41,472
[SPEAKER_02]: So, Sam, thank you for helping demythologize some of this, if you will, and bring some practical knowledge.

634
00:58:41,552 --> 00:58:43,374
[SPEAKER_02]: I think it's a huge service.

635
00:58:43,594 --> 00:58:46,138
[SPEAKER_02]: And like you said, it's changing every day.

636
00:58:46,238 --> 00:58:53,427
[SPEAKER_02]: So, you know, apologies in advance for how outdated this podcast will be in about three weeks, but that's just the way it works.

637
00:58:53,687 --> 00:58:54,868
[SPEAKER_02]: You got to get started somewhere.

638
00:58:55,930 --> 00:58:57,912
[SPEAKER_02]: The AI nodes take Christmas off.

639
00:58:57,892 --> 00:58:58,833
[SPEAKER_02]: That's right.

640
00:59:01,156 --> 00:59:03,518
[SPEAKER_02]: Stop updating your LLMs for crying out loud.

641
00:59:04,019 --> 00:59:04,239
[SPEAKER_02]: Yeah.

642
00:59:04,279 --> 00:59:13,069
[SPEAKER_02]: The big news today was that Sam Altman issued a code red for open AI because Gemini is doing so well and they've got to get back to getting hard work done.

643
00:59:13,109 --> 00:59:13,990
[SPEAKER_02]: So, okay.

644
00:59:14,010 --> 00:59:14,130
[SPEAKER_02]: Yeah.

645
00:59:14,150 --> 00:59:16,793
[SPEAKER_03]: Stop this 996 nonsense, right?

646
00:59:16,813 --> 00:59:17,494
[SPEAKER_02]: Yeah, right.

647
00:59:17,895 --> 00:59:19,156
[SPEAKER_02]: We need some work-life balance.

648
00:59:19,176 --> 00:59:19,496
[SPEAKER_02]: Take a break.

649
00:59:20,117 --> 00:59:23,101
[SPEAKER_02]: Before the AIs take our jobs, we need some work-life balance.

650
00:59:23,141 --> 00:59:23,861
[SPEAKER_02]: No, I'm just kidding.

651
00:59:24,042 --> 00:59:24,422
[SPEAKER_02]: All right.

652
00:59:25,887 --> 00:59:33,257
[SPEAKER_02]: I know that as you go out there and you're working with data and you're trying to use AI, it's always complex and challenging and you're learning a lot.

653
00:59:33,417 --> 00:59:35,660
[SPEAKER_02]: It feels like the early days of analytics all over again.

654
00:59:35,700 --> 00:59:43,170
[SPEAKER_02]: But I know I speak for both of my co-hosts, Tim and Val, when I say, no matter which MCP you're using, don't forget to keep analyzing.

655
00:59:43,591 --> 00:59:44,672
[SPEAKER_07]: Thanks for listening.

656
00:59:44,652 --> 00:59:57,969
[SPEAKER_07]: Let's keep the conversation going with your comments, suggestions, and questions on Twitter at atanalyticshour on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group.

657
00:59:57,989 --> 01:00:00,492
[SPEAKER_07]: Music for the podcast by Josh Crowhurst.

658
01:00:01,453 --> 01:00:05,878
[SPEAKER_07]: Those smart guys wanted to fit in, so they made up a term called analytics.

659
01:00:05,898 --> 01:00:07,200
[SPEAKER_07]: Analytics don't work.

660
01:00:08,092 --> 01:00:11,256
[SPEAKER_06]: Do the analytics say go for it, no matter who's going for it?

661
01:00:11,276 --> 01:00:14,339
[SPEAKER_06]: So if you and I were on the field, the analytics say go for it.

662
01:00:14,359 --> 01:00:20,326
[SPEAKER_06]: It's the stupidest, laziest, lamest thing I've ever heard for reasoning in competition.

663
01:00:28,435 --> 01:00:32,580
[SPEAKER_03]: Rock flag and non-determinism.