1 00:00:17,200 --> 00:00:20,720 Hey everyone, this is Jack Smith. Welcome to IT 2 00:00:21,220 --> 00:00:24,360 Horror Stories podcast. Good to have you back as 3 00:00:24,860 --> 00:00:28,530 always. Welcome, welcome, welcome. And. And this month Bob 4 00:00:29,030 --> 00:00:32,090 is back. Hey, everyone. Hey, Bob. Bob, 5 00:00:32,590 --> 00:00:36,090 you came to me, you wanted to talk about something. Yes, 6 00:00:36,410 --> 00:00:40,050 go ahead. I wanted to talk about the bane of my existence. 7 00:00:40,550 --> 00:00:43,690 And honestly, I think the bane. The bane 8 00:00:44,190 --> 00:00:47,770 of the existence of a lot of IT people. And that's the dreaded topic 9 00:00:48,270 --> 00:00:51,130 of shadow it. Oh no. Oh yes, 10 00:00:53,850 --> 00:00:57,530 I was. Bane of your existence. I was going to reply something else, 11 00:00:58,030 --> 00:01:00,650 but shadow it. I hear you. Yes, go ahead, 12 00:01:01,290 --> 00:01:04,930 Shadow it. For those who are. Who aren't 13 00:01:05,430 --> 00:01:09,130 really familiar with term, first of all, praise yourself. Lucky you don't know how 14 00:01:09,630 --> 00:01:12,410 good you have it. True. Can confirm. Exactly. 15 00:01:12,730 --> 00:01:16,090 And what is shadow it? Shadow it 16 00:01:16,590 --> 00:01:20,290 is essentially business doing IT stuff without the 17 00:01:20,790 --> 00:01:24,250 knowledge of the IT department. And yeah, I know that a lot of you are 18 00:01:24,750 --> 00:01:28,130 currently screaming. I hear you, I feel you. 19 00:01:28,290 --> 00:01:32,450 I would be screaming as well. But yeah, that's. That's not really an all 20 00:01:32,950 --> 00:01:36,130 that interesting episode. So I'm going to. I'm going to tell 21 00:01:36,630 --> 00:01:40,490 you a few stories of what I. Of the. 22 00:01:40,990 --> 00:01:43,570 More. How do I best put this? 23 00:01:43,810 --> 00:01:47,250 The. The ones that stuck with me. Let us say it like that. 24 00:01:47,410 --> 00:01:51,010 The ones that deserve to be retold for future generations. 25 00:01:51,170 --> 00:01:54,390 Yes. As a warning as it is, Shadow it is 26 00:01:54,890 --> 00:01:58,390 always. There is not a single Shadow IT story that ends well. 27 00:01:59,270 --> 00:02:03,430 That's true. That's true. And you 28 00:02:03,930 --> 00:02:06,470 mentioned when business does it, 29 00:02:07,590 --> 00:02:10,950 but imagine that it's your local IT department 30 00:02:11,510 --> 00:02:14,950 doing their local IT in the global environment. So when it 31 00:02:15,450 --> 00:02:18,630 does shadow it, it gets worse. But this is your 32 00:02:19,130 --> 00:02:23,140 story first. And I'll pick in. Yeah, indeed. Indeed. And I think it 33 00:02:23,640 --> 00:02:27,220 doing shadow it, that's an all different episode. True. 34 00:02:28,180 --> 00:02:32,060 Let's first stick with business does it. So that's fun enough. I'm going 35 00:02:32,560 --> 00:02:36,700 to set the scene. We are, I think around 10 36 00:02:37,200 --> 00:02:39,860 years ago at a major multinational company, 37 00:02:40,420 --> 00:02:43,740 global company, very well known. And I'm going to leave it at 38 00:02:44,240 --> 00:02:47,580 that. And essentially it was just an average day. I was managing 39 00:02:48,080 --> 00:02:51,960 the IT department as per usual. I was managing the application 40 00:02:52,460 --> 00:02:56,040 side of things. And suddenly I get a call of the CEO. 41 00:02:56,540 --> 00:02:59,280 Never a good thing if you get a call of CEO out of the blue. 42 00:02:59,780 --> 00:03:03,600 And essentially he started shouting and yelling at me because that was who 43 00:03:04,100 --> 00:03:08,040 he was. He liked to shout and yell that apparently the 44 00:03:08,840 --> 00:03:12,200 sales department couldn't do their job. 45 00:03:12,280 --> 00:03:14,440 Oh no, it was our fault. 46 00:03:15,480 --> 00:03:19,720 There's a power plug attached to it. Of course it's your fault. First I imagined 47 00:03:20,220 --> 00:03:23,850 it was Something like that. And I wished it, it would have been something 48 00:03:24,350 --> 00:03:27,570 like that. No, my first risk was, okay, 49 00:03:27,730 --> 00:03:31,290 why can't sales sell? Because honestly, they need to 50 00:03:31,790 --> 00:03:35,370 talk to customers and do their thing. So what 51 00:03:35,870 --> 00:03:39,850 was he talking about? Apparently he was talking about the sales 52 00:03:40,350 --> 00:03:43,410 reports. So the sales reports, the projections, 53 00:03:43,970 --> 00:03:47,010 the revenue streams, all the sales funnels, 54 00:03:47,170 --> 00:03:51,130 all of that shit, that it didn't work anymore. That was my 55 00:03:51,630 --> 00:03:54,760 fault. So I went over there, spoke with 56 00:03:55,260 --> 00:03:58,800 the VP of sales, asked him to show me what they were 57 00:03:59,300 --> 00:04:02,960 talking about. And I was glad that I did. Because being 58 00:04:03,120 --> 00:04:06,480 a large multinational company, of course 59 00:04:06,800 --> 00:04:10,800 everybody expected you use the software solutions that 60 00:04:11,120 --> 00:04:14,160 are enforced by the head office. 61 00:04:14,880 --> 00:04:18,320 And they did use those, but only the bare minimum. 62 00:04:18,550 --> 00:04:22,230 Because apparently what they did was they 63 00:04:22,790 --> 00:04:26,710 had over the course of the years, some technical 64 00:04:26,790 --> 00:04:30,990 people working in the sales department who were quite 65 00:04:31,490 --> 00:04:34,710 prolific in Access and Excel. Oh dear, oh dear. 66 00:04:36,150 --> 00:04:39,590 I sense a lot of job security for the people having 67 00:04:40,090 --> 00:04:43,670 to fix this. Those people who initially set it up were long gone. 68 00:04:44,080 --> 00:04:47,960 Yes, the moment you said somebody started doing this in 69 00:04:48,460 --> 00:04:52,000 Excel and access maintenance is not an issue because it doesn't exist. 70 00:04:52,240 --> 00:04:55,680 Yes, indeed, indeed. They were long gone. And they were 71 00:04:56,180 --> 00:04:59,600 used by the entire sales department, which was around 50 people, 72 00:04:59,680 --> 00:05:02,720 so quite a big department fiddling around in those. 73 00:05:02,880 --> 00:05:06,200 My initial thought was, okay, let's figure out 74 00:05:06,700 --> 00:05:09,960 what's going on here logically. Logically, indeed. I asked 75 00:05:10,460 --> 00:05:13,950 him to show me, and he pointed me to a 76 00:05:14,450 --> 00:05:17,870 network share where there were various files or where there should 77 00:05:18,370 --> 00:05:21,390 have been various files, but the files weren't there anymore. So oops. 78 00:05:21,550 --> 00:05:25,150 Oops indeed. But at the same time, network share, what's the biggie? 79 00:05:25,650 --> 00:05:28,710 Everything is backed up. To quote yourself, I hear a 80 00:05:29,210 --> 00:05:33,390 but coming. Yes, there is a big but that's 81 00:05:33,890 --> 00:05:37,270 just over the horizon. When I went to the network team 82 00:05:37,770 --> 00:05:41,310 and I said, hey guys, can you please restore whatever is on particular network share? 83 00:05:41,810 --> 00:05:45,280 They asked me what network share. They they were not aware of a specific network 84 00:05:45,360 --> 00:05:48,800 share, and indeed, on the storage system 85 00:05:48,880 --> 00:05:52,560 that particular network share did not exist. So that was problem 86 00:05:53,060 --> 00:05:56,360 one. Where the hell was that network share? Where was it coming from? Where was 87 00:05:56,860 --> 00:06:00,560 it living? After I think two days, we figured out that 88 00:06:00,960 --> 00:06:04,400 network share was essentially a local server 89 00:06:04,480 --> 00:06:07,800 they bought and they had running somewhere in closet in 90 00:06:08,300 --> 00:06:11,920 their office, which was patched in into the the 91 00:06:12,420 --> 00:06:16,200 regular desk office network. So as far as we were concerned, 92 00:06:16,700 --> 00:06:19,960 it was just a regular desktop or it was plugged 93 00:06:20,460 --> 00:06:23,480 into a desktop network outlet. We weren't 94 00:06:23,980 --> 00:06:27,439 aware of that particular server. Apparently a server that was running 95 00:06:27,939 --> 00:06:31,680 already for eight years without any Software 96 00:06:32,180 --> 00:06:35,520 updates without any security, without any 97 00:06:36,020 --> 00:06:39,600 OS updates. Really. It was just installed there and 98 00:06:39,840 --> 00:06:43,040 left alone. I have to ask, was it 99 00:06:43,540 --> 00:06:46,920 server hardware or did they just scavenge a desktop from somewhere? No, 100 00:06:47,420 --> 00:06:50,000 no, it was an actual server hardware. 101 00:06:50,480 --> 00:06:53,720 And since it was sales that bought it on the sales budget 102 00:06:54,220 --> 00:06:57,600 at the time, it was the most over the top powered 103 00:06:58,100 --> 00:07:02,000 machine possible. It cost. At the time that they bought it, it cost several 104 00:07:02,400 --> 00:07:06,630 million. Of course, because salespeople 105 00:07:07,130 --> 00:07:10,030 to salespeople will sell what salespeople want to sell. 106 00:07:10,350 --> 00:07:14,110 Exactly. And they needed a lot of storage. And essentially there was a lot 107 00:07:14,610 --> 00:07:18,270 of storage on there. We discovered that the machine had around 4 108 00:07:18,770 --> 00:07:22,350 terabytes of storage 10 years ago. 10 years ago. Wow. 109 00:07:22,590 --> 00:07:25,950 As I said, it was an expensive bike. 110 00:07:26,450 --> 00:07:29,150 That's impressive. Yes, indeed. Very impressive. 111 00:07:30,580 --> 00:07:35,380 Can I do a nerd question? Yes. Were these 15k 112 00:07:35,700 --> 00:07:39,780 RPM fibre channel discs? I never opened the machine 113 00:07:40,500 --> 00:07:44,820 because that would explain the million euro price tag. 114 00:07:45,139 --> 00:07:48,580 Honestly, there's a high likelihood. But yeah, exactly. 115 00:07:49,080 --> 00:07:51,860 But I refuse to open the machine. 116 00:07:52,360 --> 00:07:55,060 Yeah. Because the second you touch it, you're responsible for it. 117 00:07:55,390 --> 00:07:58,910 Exactly. You don't even want to open the cabinets. It's in. 118 00:07:59,790 --> 00:08:03,790 You had to. I had to. But I did not open the physical machine and 119 00:08:03,870 --> 00:08:07,310 I just. Once I discovered that I prayed to God that the machine 120 00:08:07,810 --> 00:08:11,310 was still running. Luckily, it was running. Okay. 121 00:08:11,870 --> 00:08:14,030 However, it ran out of disk space. 122 00:08:15,150 --> 00:08:18,430 I have many questions, but I'm not going to ask them right now. It ran 123 00:08:18,930 --> 00:08:22,950 out of disk space, so the machine blue screened and 124 00:08:23,450 --> 00:08:27,770 it couldn't boot up anymore. Okay. Yep. I see multiple operating 125 00:08:28,270 --> 00:08:31,970 system reasons for this happening. Yes. And since it didn't boot up anymore, 126 00:08:32,470 --> 00:08:35,210 it wasn't available on the network anymore and they couldn't. 127 00:08:35,290 --> 00:08:38,490 Could no longer access all the files. And the share was there for. 128 00:08:39,130 --> 00:08:42,450 And the share was. Was indeed exposed to the sales 129 00:08:42,950 --> 00:08:44,730 guys in their own local. 130 00:08:46,090 --> 00:08:49,610 What's it called again? Vlan. So, yeah, we. 131 00:08:51,900 --> 00:08:55,860 It was relatively decently set up at the time, but I 132 00:08:56,360 --> 00:09:00,300 still don't know how those guys managed to have the necessary 133 00:09:00,620 --> 00:09:03,860 rights to do the setup. That was my question and I 134 00:09:04,360 --> 00:09:07,620 was gonna ask you, shall I keep my security stuff for later or do I 135 00:09:08,120 --> 00:09:10,540 throw it in now? Because I have multiple questions here. 136 00:09:11,980 --> 00:09:15,740 You can throw it in now. Yeah. So was it a domain member? 137 00:09:16,260 --> 00:09:20,580 No. Okay. And then they either had one generic 138 00:09:20,900 --> 00:09:24,260 user or they used password pass through and had to change 139 00:09:24,760 --> 00:09:27,460 their domain password on the thing every now and then. 140 00:09:27,620 --> 00:09:31,380 Essentially it used the same password. It used the login of the VP. 141 00:09:31,540 --> 00:09:35,460 Oh. I am now scanning my brain for VPs. 142 00:09:35,540 --> 00:09:37,300 Being fired by catapult, 143 00:09:39,940 --> 00:09:43,940 we choose not to have any subscription, model nor sponsoring in order to keep 144 00:09:44,440 --> 00:09:48,240 our stories accessible for each and everyone to support us. Please check 145 00:09:48,740 --> 00:09:52,480 our merchandise at shop IThorrorStories EU or buy 146 00:09:52,980 --> 00:09:56,720 us a coffee at co-.com IThorrorStories. 147 00:10:05,840 --> 00:10:09,560 You wish. But no, apparently the CEO is completely aware and fine with 148 00:10:10,060 --> 00:10:13,920 it because it gave them the flexibility 149 00:10:14,080 --> 00:10:17,680 to do what Head office didn't allow 150 00:10:18,180 --> 00:10:20,940 them to. And of course, then as long as you had the sales figures, 151 00:10:21,100 --> 00:10:24,700 it was all good. Exactly. Okay, please continue. 152 00:10:25,420 --> 00:10:28,940 So, yeah, essentially we got it cleaned up, 153 00:10:29,420 --> 00:10:33,100 we removed the old data because apparently they used it to store every 154 00:10:33,420 --> 00:10:37,100 little document. Everything you can imagine, 155 00:10:37,600 --> 00:10:40,780 they stored on there. Okay, so you got it back up again? 156 00:10:40,860 --> 00:10:44,780 Yes, we got it back up again. But then it was the next question, 157 00:10:45,020 --> 00:10:48,940 what do we do with this? Yeah. Because you have 158 00:10:49,440 --> 00:10:54,260 now officially made a discovery 159 00:10:54,760 --> 00:10:58,380 that is a huge liability for so 160 00:10:58,880 --> 00:11:01,780 many reasons. Yep. You can. Yes. This is. 161 00:11:02,419 --> 00:11:05,300 You don't even know where to start. Exactly. 162 00:11:06,660 --> 00:11:10,100 Officially, I had to inform head office of the situation. 163 00:11:10,260 --> 00:11:13,460 Yes. Head office would just say power it down, 164 00:11:14,090 --> 00:11:17,770 period. That is, anyone with at 165 00:11:18,270 --> 00:11:21,210 least one active brain cell insecurity would say, 166 00:11:21,690 --> 00:11:24,930 switch it off and forget about it. Yeah. 167 00:11:25,430 --> 00:11:28,090 Yes. So those were my responsibilities on one hand. 168 00:11:28,410 --> 00:11:31,850 On the other hand, of course, we have the CEO, the local 169 00:11:32,350 --> 00:11:36,010 CEO and vp and the entire sales team said, yeah, we absolutely 170 00:11:36,510 --> 00:11:39,970 need this because the data is not available in the system of 171 00:11:40,470 --> 00:11:44,050 the head office and the reports that we get from there aren't tailor 172 00:11:44,550 --> 00:11:48,090 made to exactly look and feel how we want them 173 00:11:48,590 --> 00:11:51,770 to be. So that's fair enough. That's essentially between a 174 00:11:52,270 --> 00:11:56,050 rock and a hard place. And then you have the CIO because 175 00:11:56,550 --> 00:11:59,330 of. Yes. Oh dear. Oh dear, indeed. 176 00:11:59,810 --> 00:12:02,850 Essentially, the cio, he was not aware of the situation. 177 00:12:03,250 --> 00:12:06,410 So that's an interesting board meeting where you want to be a fly on the 178 00:12:06,910 --> 00:12:11,090 wall. Yeah, yeah. But he wasn't aware of the situation. 179 00:12:11,170 --> 00:12:14,850 But yeah, he was even in a worse place than me because he 180 00:12:15,630 --> 00:12:19,150 locally responsible for security and risk. 181 00:12:20,270 --> 00:12:22,350 You also need to report the CEO. 182 00:12:23,710 --> 00:12:27,550 So what essentially we landed on was 183 00:12:28,050 --> 00:12:31,669 that we would have a migration project from all 184 00:12:32,169 --> 00:12:36,070 the data that's on there to migrate it into the system 185 00:12:36,570 --> 00:12:40,670 of head office. And then at the same time, the CIO 186 00:12:41,170 --> 00:12:45,120 and the CEO lobbied Head office to have reports that 187 00:12:45,620 --> 00:12:49,120 looked more like the ones that they were using locally, which was a very 188 00:12:49,360 --> 00:12:53,360 back office backroom politics happened because company 189 00:12:53,600 --> 00:12:57,200 was German and you know how they can be sticklers for rules. 190 00:12:58,480 --> 00:13:03,440 Essentially we had to have very senior German directors 191 00:13:03,760 --> 00:13:06,920 agree on the fact that we did not immediately 192 00:13:07,420 --> 00:13:11,560 power down those machines. We migrated all the data into the 193 00:13:12,060 --> 00:13:14,820 data of the platform, the head office, 194 00:13:14,900 --> 00:13:18,580 and on top of that, that the existing reports needed to 195 00:13:19,080 --> 00:13:22,340 be rebuilt. Essentially all of that happened over the course of around 196 00:13:22,840 --> 00:13:26,260 12 months. So quite successfully, quite happy. Until. 197 00:13:26,420 --> 00:13:30,140 Because the story doesn't end there until I suddenly 198 00:13:30,640 --> 00:13:33,940 get. I think we were nine months in, in that migration. 199 00:13:34,340 --> 00:13:36,660 I get a call from the VP of sales, 200 00:13:37,060 --> 00:13:39,780 hey, we have an issue with one of the new reports. 201 00:13:40,100 --> 00:13:43,660 Okay, fine, but we don't do those. Those are done by head 202 00:13:44,160 --> 00:13:47,700 office. Yeah, but they don't want to help us with these reports. 203 00:13:48,100 --> 00:13:50,020 There's going to be a reason for that, I feel. 204 00:13:50,740 --> 00:13:54,020 Yes. So essentially what they had been 205 00:13:54,520 --> 00:13:58,500 doing ever since we started migrating the data into the corporate system 206 00:13:58,980 --> 00:14:02,580 was dumping that data into a 207 00:14:02,660 --> 00:14:06,340 Power BI data lake. And there they started 208 00:14:06,580 --> 00:14:10,260 creating new reports again. So they basically made 209 00:14:11,270 --> 00:14:14,670 a copy of what was provided and then worked on the copy. Yes, 210 00:14:15,170 --> 00:14:18,310 exactly. And yeah, sorry to jump in. Do I 211 00:14:18,810 --> 00:14:22,470 want to know where that server was running? Well, that was 212 00:14:22,970 --> 00:14:25,270 running on the corporate network in corporate cloud. 213 00:14:26,389 --> 00:14:30,550 Surprised? Yeah. So all of those things were good. Let us 214 00:14:31,050 --> 00:14:34,470 say people from compliance and Audit were 215 00:14:34,710 --> 00:14:38,510 not aware about the fact that commercial data was now living 216 00:14:39,010 --> 00:14:43,350 in a much more publicly available or in a much more publicly 217 00:14:43,850 --> 00:14:46,910 available data space. Yeah, but yeah, they just said, 218 00:14:47,410 --> 00:14:50,830 okay, we go from Access and Excel, we go to Power bi 219 00:14:51,330 --> 00:14:54,630 and we just keep on doing what we were doing. That was their plan 220 00:14:55,130 --> 00:14:59,150 until the moment that they noticed that they did not have sufficient Power 221 00:14:59,650 --> 00:15:02,990 BI knowledge to rebuild the reports. Because they started building reports 222 00:15:03,150 --> 00:15:06,410 because corporate set for quite a big bunch of their reports. 223 00:15:06,890 --> 00:15:10,810 There's no need to have those reports. You have an existing report 224 00:15:10,890 --> 00:15:14,730 where you can activate a filter and you have exactly what you need. 225 00:15:14,810 --> 00:15:18,810 But no, they prefer to have 50 different reports each 226 00:15:19,310 --> 00:15:23,410 for one client instead of one report where you then just selected 227 00:15:23,910 --> 00:15:26,250 the client. That was too much of a bother. Oh, 228 00:15:27,210 --> 00:15:30,570 yeah, okay. Yeah, I see what they were doing and why they were doing 229 00:15:31,070 --> 00:15:34,610 it. And I also see why head office would 230 00:15:35,110 --> 00:15:38,330 think otherwise. Yes, indeed. So, yeah, then they 231 00:15:38,830 --> 00:15:42,810 came to us about Power BI thing. I just cleared. We don't do Power bi, 232 00:15:43,310 --> 00:15:47,850 period. And when I left that 233 00:15:48,350 --> 00:15:52,610 company, they were recruiting a Power BI specialist 234 00:15:52,770 --> 00:15:57,170 that would be living in the sales department specifically 235 00:15:57,670 --> 00:16:01,260 for that. And in parallel, they were trying to fight off the 236 00:16:01,760 --> 00:16:06,220 audit and compliance department of head office that were 237 00:16:06,720 --> 00:16:10,300 informed about the fact that the data was. They will 238 00:16:10,700 --> 00:16:14,420 get to it because once you get to the corporate systems, you will 239 00:16:14,920 --> 00:16:18,820 have some kind of Audit running over it and then it will pop up with 240 00:16:19,320 --> 00:16:23,340 a thing. And on that thing I have a question that 241 00:16:23,660 --> 00:16:27,060 how could that system remain undetected for so 242 00:16:27,560 --> 00:16:30,870 long, even in user network? Of course it's 243 00:16:31,370 --> 00:16:34,950 10 years ago you wouldn't be as proactive security wise as you 244 00:16:35,450 --> 00:16:39,070 are today. Exactly. But today your average 245 00:16:39,470 --> 00:16:43,070 generic network sweep will find a device 246 00:16:43,570 --> 00:16:46,750 that's not in your domain, in user network and then pick that up. 247 00:16:47,250 --> 00:16:50,590 Hopefully. Hopefully indeed. Plus if you do network 248 00:16:50,750 --> 00:16:54,430 authentication, any device not in the domain cannot get onto the 249 00:16:54,930 --> 00:16:58,990 network. Yeah. So I assume 250 00:16:59,490 --> 00:17:02,670 that in this day and age it would be significantly more difficult. But at the 251 00:17:03,170 --> 00:17:07,230 same time this was not set up without the compliance of somebody 252 00:17:07,730 --> 00:17:11,990 from it. Let's be honest, it couldn't have been set up, 253 00:17:12,390 --> 00:17:15,510 especially with patching and the fact that it was in a closet and all those 254 00:17:16,010 --> 00:17:19,030 things. It was professionally installed. 255 00:17:19,110 --> 00:17:21,670 Somebody knew. Somebody knew at what one time? 256 00:17:22,710 --> 00:17:26,270 And honestly I think that's the biggest thing here. Somebody knew. 257 00:17:26,770 --> 00:17:30,070 Somebody did not inform the right people or the right people were 258 00:17:30,570 --> 00:17:34,310 informed and they did not inform their successors afterwards. 259 00:17:34,470 --> 00:17:37,910 Yeah. But business essentially they thought they were 260 00:17:38,410 --> 00:17:42,470 doing everything above the board. Yeah. Which I can see it happen that 261 00:17:42,550 --> 00:17:46,110 local, it said okay, they have a thing here and we will do 262 00:17:46,610 --> 00:17:50,110 it properly, we will put it away, we'll do some basic setup, 263 00:17:50,610 --> 00:17:54,510 yada yada. Yeah. And they do the things to avoid being detected by 264 00:17:55,010 --> 00:17:58,550 corporate because let's be honest, a local CEO still has the 265 00:17:59,050 --> 00:18:02,230 power to fire you even if you say no for the right reasons. That's always 266 00:18:02,730 --> 00:18:06,470 a thing. And then so on the CEO was that country level or regional 267 00:18:06,970 --> 00:18:10,390 level? That was country level. Country level. Okay. So not that super high up 268 00:18:10,890 --> 00:18:13,780 in the food chain, but locally some mandate and yeah, 269 00:18:14,280 --> 00:18:18,180 essentially locally he could decide everything as 270 00:18:18,680 --> 00:18:22,020 long as it in theory it was aligned the corporate rules, the corporate 271 00:18:22,520 --> 00:18:25,900 guy. So yeah, but yeah, that's something 272 00:18:26,220 --> 00:18:30,460 that hopefully today would not be as obvious. Certainly not in larger corps 273 00:18:30,960 --> 00:18:34,380 where they have woken up on their cybersecurity level 274 00:18:34,880 --> 00:18:38,620 that most of them should do internal sweeps and internal 275 00:18:39,120 --> 00:18:42,610 scans and network authentication. But I can imagine that 276 00:18:43,170 --> 00:18:46,130 10 years ago this was not as much the case. 277 00:18:46,610 --> 00:18:49,890 So. Yeah, but at the same time, 278 00:18:51,730 --> 00:18:55,090 if sufficient people are complicit 279 00:18:55,590 --> 00:18:58,530 in the story, it can happen even in this day and age. 280 00:18:58,610 --> 00:19:01,850 Yes, there's sufficient tooling 281 00:19:02,350 --> 00:19:06,250 where it can be detected. But if you do the right things, it's still 282 00:19:06,750 --> 00:19:09,950 not all that obvious. No, no. That is, 283 00:19:10,590 --> 00:19:14,110 you will need a bit more accomplices. But when you 284 00:19:14,610 --> 00:19:18,270 do play the local versus regional versus global 285 00:19:18,770 --> 00:19:22,350 card, you will Find a lot of folks that say, yeah, okay, but we, 286 00:19:22,430 --> 00:19:25,750 we need this locally and we're not getting it from higher up, so we will 287 00:19:26,250 --> 00:19:30,510 do it ourselves. That is a very common 288 00:19:30,670 --> 00:19:33,990 reflex to have, actually. Yeah. And origin stories 289 00:19:34,490 --> 00:19:38,650 of any shadow RTS that is created is exactly because 290 00:19:39,150 --> 00:19:42,850 of that. Somebody has a specific need and the need 291 00:19:43,350 --> 00:19:47,250 is not fulfilled by the rest of the organization. 292 00:19:47,810 --> 00:19:51,250 So they're going to try and find a solution. Yeah. That on its own 293 00:19:51,750 --> 00:19:55,450 is good. You know, you want to move ahead in whatever 294 00:19:55,950 --> 00:19:59,210 you need to do and what you're doing. But yeah, then in the end it's 295 00:19:59,710 --> 00:20:03,650 the non existence of the run phase. It was set up once and they 296 00:20:03,730 --> 00:20:07,520 basically for all these years forgot about it because that network 297 00:20:08,020 --> 00:20:11,520 share was always there and it worked. From what I understood 298 00:20:12,020 --> 00:20:15,600 is that there was no data loss because once you added some disks to 299 00:20:16,100 --> 00:20:19,920 the container or the RAID array, the thing booted up again and 300 00:20:20,160 --> 00:20:23,560 everything came back online. So they were extremely lucky that 301 00:20:24,060 --> 00:20:27,600 there was no data loss there. Yeah. Honestly, 302 00:20:27,760 --> 00:20:31,040 what I just did was just cleared out the temp files. 303 00:20:31,540 --> 00:20:35,150 Okay. Yeah. So you go into, you boot into maintenance mode, you free up some 304 00:20:35,650 --> 00:20:39,110 space. Yes. And I can imagine from 10 years you will have some 305 00:20:39,670 --> 00:20:42,990 megabytes of temp space available that the page file can extend 306 00:20:43,490 --> 00:20:46,750 again. Exactly. And then I told them, okay, now you have to clean 307 00:20:47,250 --> 00:20:50,630 up everything that you no longer legally need. Just purge it. 308 00:20:50,710 --> 00:20:54,230 Yeah. So would it be because it's basically 309 00:20:54,470 --> 00:20:58,750 a file share? Yeah. Was it suggested as a workaround? 310 00:20:59,250 --> 00:21:03,470 Hey, you have your own system here on your own little hidden file server that 311 00:21:03,970 --> 00:21:07,550 you actually move it to a corporate file server and then you 312 00:21:08,050 --> 00:21:11,350 start using it there and then you work out the politics with her doctor in 313 00:21:11,850 --> 00:21:15,310 Germany later, because it's a German company. Was that considered or an option or 314 00:21:15,470 --> 00:21:18,670 was it just too far fetched from my brain? No, no, it was 315 00:21:19,170 --> 00:21:23,070 asked. But essentially then the way the file shares were 316 00:21:23,570 --> 00:21:27,070 set up was that you had your local personal network share 317 00:21:27,310 --> 00:21:30,950 and then essentially you only had a file share used for transfer, 318 00:21:31,450 --> 00:21:35,630 for data transfer. Okay. You did not any, any shared 319 00:21:36,130 --> 00:21:39,670 data was more. Was more stored on SharePoint and, 320 00:21:40,170 --> 00:21:44,630 and stuff like that. And yeah, SharePoint didn't allow for the amount 321 00:21:45,130 --> 00:21:48,870 of file sizes that they were using and the transfer was 322 00:21:49,370 --> 00:21:53,390 purged on a weekly basis. I do get it. It is the same moral penalty 323 00:21:53,890 --> 00:21:56,950 of using Excel as a database as it is to use SharePoint as a 324 00:21:57,450 --> 00:21:57,590 file server. 325 00:22:02,550 --> 00:22:05,950 This podcast features Jack Smith and guests. We say thank 326 00:22:06,450 --> 00:22:10,630 you to our demoscene friends who helped making this podcast possible. Commander Homer for editing 327 00:22:10,710 --> 00:22:15,270 Danko for music and audio advice, Ned Poet for additional voiceovers. 328 00:22:24,920 --> 00:22:28,520 Exactly. And even so, the file size were just too 329 00:22:29,020 --> 00:22:32,520 big for SharePoint at the time that was set up in that SharePoint environment 330 00:22:32,760 --> 00:22:36,280 was 50 megabytes. Not a single report I know is under 331 00:22:36,780 --> 00:22:40,240 that. Exactly. I do understand their, I wouldn't call 332 00:22:40,740 --> 00:22:45,000 it business reason, but that's exactly what it is. Yeah. But essentially 333 00:22:45,080 --> 00:22:49,120 their business reason was that they did not like the new 334 00:22:49,620 --> 00:22:53,330 system. No. Because if I then asked back, it coincided 335 00:22:53,830 --> 00:22:57,090 with the rollout of the new sales system that corporate rolled out. 336 00:22:58,210 --> 00:23:01,690 We're not doing this. Exactly. They said we are not doing this. We are 337 00:23:02,190 --> 00:23:05,690 just entering the basic minimum. So corporate is happy and we will 338 00:23:06,190 --> 00:23:09,890 just keep on using our Excels and access files that we have. 339 00:23:11,730 --> 00:23:14,770 And since they were no longer allowed to do it on a 340 00:23:15,270 --> 00:23:18,990 network share like they did before and then they put their own, their own 341 00:23:19,490 --> 00:23:23,230 little box in. Exactly. They saw a problem and they 342 00:23:23,730 --> 00:23:27,430 solved it from their perspective with all the risks and everything attached to 343 00:23:27,930 --> 00:23:31,270 it and everybody just forgot about it because it was just there. Yeah, exactly. 344 00:23:31,430 --> 00:23:35,510 Specifically, if you like thinking that, well, if when the CEO said 345 00:23:35,750 --> 00:23:38,990 well yeah, no, it's running there, there's no technical background 346 00:23:39,490 --> 00:23:42,790 left, suddenly it dies and then it becomes your problem. 347 00:23:42,950 --> 00:23:45,400 Like what chair, what server? 348 00:23:45,720 --> 00:23:49,120 I think they were extremely lucky that it was just running out of 349 00:23:49,620 --> 00:23:53,840 disk space and not one of the disk containers dying 350 00:23:54,340 --> 00:23:56,840 and they effectively having lost all their data. 351 00:23:57,080 --> 00:24:00,680 Because then I don't think corporate would have let this 352 00:24:01,180 --> 00:24:04,720 one slide. No, no, honestly I wouldn't really say that corporate 353 00:24:05,220 --> 00:24:08,880 wouldn't have let it slide. The fact that that happened because from 354 00:24:09,380 --> 00:24:12,120 corporate perspective, all sales data that needed to be there was there. 355 00:24:12,420 --> 00:24:15,620 Indeed. And sales were going all good and sales was going all good. 356 00:24:15,780 --> 00:24:19,380 I think the only issue corporates would have had with that 357 00:24:19,880 --> 00:24:23,700 particular situation was that if there would have been a drop in sales, 358 00:24:24,660 --> 00:24:28,459 but as long as sales would have just delivered 359 00:24:28,959 --> 00:24:32,300 and hit their target, they couldn't care less that the additional data 360 00:24:32,800 --> 00:24:36,180 that corporate didn't really was interested in that that was lost. 361 00:24:36,680 --> 00:24:39,540 But this was mainly the entire local sales organization was, 362 00:24:39,930 --> 00:24:43,650 was organized to use all that specific data to give reports to 363 00:24:44,150 --> 00:24:47,890 clients and do client health analysis and all of those 364 00:24:48,390 --> 00:24:51,050 things, those were based on that additional local data. 365 00:24:51,770 --> 00:24:55,649 And then you turn out surprised how it gets 366 00:24:56,149 --> 00:24:59,770 frustrated if the entire system is basically working against their own 367 00:25:00,270 --> 00:25:03,370 policies and everybody is happy as long as nothing goes 368 00:25:03,870 --> 00:25:08,290 wrong and when things go wrong, it's it's fault. Yes. That's a wonderful summary. 369 00:25:08,610 --> 00:25:11,410 Yeah, absolutely. Absolutely. And honestly, 370 00:25:12,450 --> 00:25:15,650 this is Not a rarity, folks. Let's be honest. This is 371 00:25:16,150 --> 00:25:19,810 a common story. There are plenty of you guys around 372 00:25:20,310 --> 00:25:23,770 and I'm sure that know of a desktop sitting on their desk 373 00:25:24,270 --> 00:25:27,770 or laptop somewhere. In a desk that is running 374 00:25:28,270 --> 00:25:31,730 a daily job or that is running a file share 375 00:25:32,230 --> 00:25:35,660 or a web server or whatnot. Because somebody 376 00:25:36,160 --> 00:25:39,540 at one time decided that that was the most convenient way forward 377 00:25:40,020 --> 00:25:43,860 and that they would get around to doing it properly 378 00:25:44,260 --> 00:25:47,300 later on. You get expensive oopsies. Yes, 379 00:25:48,820 --> 00:25:52,900 expensive oopsies with consequences that 380 00:25:53,060 --> 00:25:56,100 can cost you dearly. Oh, absolutely. 381 00:25:56,600 --> 00:26:00,880 Yeah. Maybe I'll just end on that. Sure. There's one small anecdote 382 00:26:01,380 --> 00:26:04,960 about Shadow IT by business that I want to share with each 383 00:26:05,460 --> 00:26:09,080 and every one of you. And I was working in a company and we just 384 00:26:09,580 --> 00:26:12,920 had a change of senior management, okay. So the 385 00:26:13,420 --> 00:26:16,800 entire board was replaced for commercial reasons. 386 00:26:17,280 --> 00:26:20,800 Reasons are reasons, for reasons are reasons. 387 00:26:20,960 --> 00:26:25,120 And the director at that particular time said, 388 00:26:25,460 --> 00:26:29,140 made a very strong statement. He said, in my previous 389 00:26:29,300 --> 00:26:32,660 company, we suffered tremendously from Shadow it. 390 00:26:32,820 --> 00:26:36,540 So I vouch here that I will no longer 391 00:26:37,040 --> 00:26:40,500 tolerate any Shadow IT being done. Okay. In between now 392 00:26:41,000 --> 00:26:45,060 and the next three years, all Shadow IT will be thrown out 393 00:26:45,140 --> 00:26:49,540 the window and will be replaced by properly maintained 394 00:26:50,040 --> 00:26:52,990 systems. Okay, good, great. We were very happy. 395 00:26:53,490 --> 00:26:56,910 Yeah. As an IT department. Yes. Until the next 396 00:26:57,410 --> 00:27:00,550 part of his speech. Game. Next part of his speech. I don't 397 00:27:01,050 --> 00:27:04,350 remember it word for word anymore. It's been too long. 398 00:27:04,850 --> 00:27:08,550 But he essentially said, because somebody asked and what 399 00:27:09,050 --> 00:27:13,150 about the needs that Shadow IT helps 400 00:27:13,470 --> 00:27:17,470 fulfill? And he said, yeah, of course, those needs remain. 401 00:27:17,710 --> 00:27:21,990 So we will empower all business users to develop 402 00:27:22,470 --> 00:27:26,110 their own little programs and scripts in order to fulfill 403 00:27:26,610 --> 00:27:29,270 those needs and those will be supported by it. 404 00:27:31,030 --> 00:27:34,630 This is intentional silence from my end now. Yeah, it's like. 405 00:27:39,190 --> 00:27:41,110 Alrighty then. Yes, 406 00:27:43,270 --> 00:27:44,790 yes. Oh dear. 407 00:27:46,710 --> 00:27:50,070 So I think that's one other thing you should 408 00:27:50,570 --> 00:27:54,010 also be concerned about. Shadow IT is not always recognized as Shadow 409 00:27:54,510 --> 00:27:57,730 rt, but also sometimes it's known under user 410 00:27:58,230 --> 00:27:59,810 empowerment. Absolutely. 411 00:28:02,050 --> 00:28:05,090 Good, Bob. Let's round off here. This was very, 412 00:28:05,170 --> 00:28:09,170 very insightful. I think from every possible end. 413 00:28:09,329 --> 00:28:13,410 Thanks everyone again for tuning in. As always. You can 414 00:28:13,910 --> 00:28:17,730 find our previous episodes on ithorrorstories.eu. you can 415 00:28:18,230 --> 00:28:21,810 find us where you find. Find your podcast that is on 416 00:28:22,310 --> 00:28:25,690 Apple Music, on YouTube, on Spotify, on Deezer, and wherever 417 00:28:26,190 --> 00:28:29,610 you find your panic attacks, reach out to 418 00:28:30,110 --> 00:28:33,650 us. HelloThorrorStories. EU check our merchandise. 419 00:28:34,150 --> 00:28:37,570 It helps paying for all the costs we do here with the podcast. 420 00:28:38,070 --> 00:28:41,650 Otherwise we are just in it for the likes and the subscribes and your comments. 421 00:28:41,810 --> 00:28:45,130 So if you like this episode please give us a little 422 00:28:45,630 --> 00:28:49,510 thumbs up or a heart. Any merge is shop.it HorrorStories 423 00:28:49,910 --> 00:28:52,550 EU. Thank you for tuning in. 424 00:28:53,030 --> 00:28:55,990 See you next time. Any last words Bob from your end? 425 00:28:56,950 --> 00:29:00,230 Yeah, pretty much the same as you. Thanks for 426 00:29:00,730 --> 00:29:04,310 listening everyone. And yeah, I'm not going to do the like and 427 00:29:04,810 --> 00:29:08,390 subscribe come on YouTube thing, but if you 428 00:29:08,890 --> 00:29:12,030 think that somebody else in your acquaintances might 429 00:29:12,530 --> 00:29:16,470 like these stories, don't forget to share. That's highly 430 00:29:16,970 --> 00:29:20,790 appreciated. That's a good one. Thanks a lot. Cheers everyone. Bye bye. 431 00:29:21,290 --> 00:29:21,590 Cheers guys. 432 00:29:27,030 --> 00:29:30,550 The content of this podcast is intended for entertainment purposes 433 00:29:31,050 --> 00:29:35,030 only and is meant to humorously explore various tech related situations. 434 00:29:35,270 --> 00:29:38,810 Any resemblance to actual events or real persons, living or dead 435 00:29:38,880 --> 00:29:42,720 instead is purely coincidental. We ridicule situations, 436 00:29:42,800 --> 00:29:45,880 never individuals or groups. Listener discretion 437 00:29:46,380 --> 00:29:50,400 is advised and we encourage everyone to approach technology with a sense of humor 438 00:29:50,900 --> 00:29:51,520 and an open mind.