Hey everyone, this is Jack Smith. Welcome to IT
Horror Stories podcast. Good to have you back as
always. Welcome, welcome, welcome. And. And this month Bob
is back. Hey, everyone. Hey, Bob. Bob,
you came to me, you wanted to talk about something. Yes,
go ahead. I wanted to talk about the bane of my existence.
And honestly, I think the bane. The bane
of the existence of a lot of IT people. And that's the dreaded topic
of shadow it. Oh no. Oh yes,
I was. Bane of your existence. I was going to reply something else,
but shadow it. I hear you. Yes, go ahead,
Shadow it. For those who are. Who aren't
really familiar with term, first of all, praise yourself. Lucky you don't know how
good you have it. True. Can confirm. Exactly.
And what is shadow it? Shadow it
is essentially business doing IT stuff without the
knowledge of the IT department. And yeah, I know that a lot of you are
currently screaming. I hear you, I feel you.
I would be screaming as well. But yeah, that's. That's not really an all
that interesting episode. So I'm going to. I'm going to tell
you a few stories of what I. Of the.
More. How do I best put this?
The. The ones that stuck with me. Let us say it like that.
The ones that deserve to be retold for future generations.
Yes. As a warning as it is, Shadow it is
always. There is not a single Shadow IT story that ends well.
That's true. That's true. And you
mentioned when business does it,
but imagine that it's your local IT department
doing their local IT in the global environment. So when it
does shadow it, it gets worse. But this is your
story first. And I'll pick in. Yeah, indeed. Indeed. And I think it
doing shadow it, that's an all different episode. True.
Let's first stick with business does it. So that's fun enough. I'm going
to set the scene. We are, I think around 10
years ago at a major multinational company,
global company, very well known. And I'm going to leave it at
that. And essentially it was just an average day. I was managing
the IT department as per usual. I was managing the application
side of things. And suddenly I get a call of the CEO.
Never a good thing if you get a call of CEO out of the blue.
And essentially he started shouting and yelling at me because that was who
he was. He liked to shout and yell that apparently the
sales department couldn't do their job.
Oh no, it was our fault.
There's a power plug attached to it. Of course it's your fault. First I imagined
it was Something like that. And I wished it, it would have been something
like that. No, my first risk was, okay,
why can't sales sell? Because honestly, they need to
talk to customers and do their thing. So what
was he talking about? Apparently he was talking about the sales
reports. So the sales reports, the projections,
the revenue streams, all the sales funnels,
all of that shit, that it didn't work anymore. That was my
fault. So I went over there, spoke with
the VP of sales, asked him to show me what they were
talking about. And I was glad that I did. Because being
a large multinational company, of course
everybody expected you use the software solutions that
are enforced by the head office.
And they did use those, but only the bare minimum.
Because apparently what they did was they
had over the course of the years, some technical
people working in the sales department who were quite
prolific in Access and Excel. Oh dear, oh dear.
I sense a lot of job security for the people having
to fix this. Those people who initially set it up were long gone.
Yes, the moment you said somebody started doing this in
Excel and access maintenance is not an issue because it doesn't exist.
Yes, indeed, indeed. They were long gone. And they were
used by the entire sales department, which was around 50 people,
so quite a big department fiddling around in those.
My initial thought was, okay, let's figure out
what's going on here logically. Logically, indeed. I asked
him to show me, and he pointed me to a
network share where there were various files or where there should
have been various files, but the files weren't there anymore. So oops.
Oops indeed. But at the same time, network share, what's the biggie?
Everything is backed up. To quote yourself, I hear a
but coming. Yes, there is a big but that's
just over the horizon. When I went to the network team
and I said, hey guys, can you please restore whatever is on particular network share?
They asked me what network share. They they were not aware of a specific network
share, and indeed, on the storage system
that particular network share did not exist. So that was problem
one. Where the hell was that network share? Where was it coming from? Where was
it living? After I think two days, we figured out that
network share was essentially a local server
they bought and they had running somewhere in closet in
their office, which was patched in into the the
regular desk office network. So as far as we were concerned,
it was just a regular desktop or it was plugged
into a desktop network outlet. We weren't
aware of that particular server. Apparently a server that was running
already for eight years without any Software
updates without any security, without any
OS updates. Really. It was just installed there and
left alone. I have to ask, was it
server hardware or did they just scavenge a desktop from somewhere? No,
no, it was an actual server hardware.
And since it was sales that bought it on the sales budget
at the time, it was the most over the top powered
machine possible. It cost. At the time that they bought it, it cost several
million. Of course, because salespeople
to salespeople will sell what salespeople want to sell.
Exactly. And they needed a lot of storage. And essentially there was a lot
of storage on there. We discovered that the machine had around 4
terabytes of storage 10 years ago. 10 years ago. Wow.
As I said, it was an expensive bike.
That's impressive. Yes, indeed. Very impressive.
Can I do a nerd question? Yes. Were these 15k
RPM fibre channel discs? I never opened the machine
because that would explain the million euro price tag.
Honestly, there's a high likelihood. But yeah, exactly.
But I refuse to open the machine.
Yeah. Because the second you touch it, you're responsible for it.
Exactly. You don't even want to open the cabinets. It's in.
You had to. I had to. But I did not open the physical machine and
I just. Once I discovered that I prayed to God that the machine
was still running. Luckily, it was running. Okay.
However, it ran out of disk space.
I have many questions, but I'm not going to ask them right now. It ran
out of disk space, so the machine blue screened and
it couldn't boot up anymore. Okay. Yep. I see multiple operating
system reasons for this happening. Yes. And since it didn't boot up anymore,
it wasn't available on the network anymore and they couldn't.
Could no longer access all the files. And the share was there for.
And the share was. Was indeed exposed to the sales
guys in their own local.
What's it called again? Vlan. So, yeah, we.
It was relatively decently set up at the time, but I
still don't know how those guys managed to have the necessary
rights to do the setup. That was my question and I
was gonna ask you, shall I keep my security stuff for later or do I
throw it in now? Because I have multiple questions here.
You can throw it in now. Yeah. So was it a domain member?
No. Okay. And then they either had one generic
user or they used password pass through and had to change
their domain password on the thing every now and then.
Essentially it used the same password. It used the login of the VP.
Oh. I am now scanning my brain for VPs.
Being fired by catapult,
we choose not to have any subscription, model nor sponsoring in order to keep
our stories accessible for each and everyone to support us. Please check
our merchandise at shop IThorrorStories EU or buy
us a coffee at co-.com IThorrorStories.
You wish. But no, apparently the CEO is completely aware and fine with
it because it gave them the flexibility
to do what Head office didn't allow
them to. And of course, then as long as you had the sales figures,
it was all good. Exactly. Okay, please continue.
So, yeah, essentially we got it cleaned up,
we removed the old data because apparently they used it to store every
little document. Everything you can imagine,
they stored on there. Okay, so you got it back up again?
Yes, we got it back up again. But then it was the next question,
what do we do with this? Yeah. Because you have
now officially made a discovery
that is a huge liability for so
many reasons. Yep. You can. Yes. This is.
You don't even know where to start. Exactly.
Officially, I had to inform head office of the situation.
Yes. Head office would just say power it down,
period. That is, anyone with at
least one active brain cell insecurity would say,
switch it off and forget about it. Yeah.
Yes. So those were my responsibilities on one hand.
On the other hand, of course, we have the CEO, the local
CEO and vp and the entire sales team said, yeah, we absolutely
need this because the data is not available in the system of
the head office and the reports that we get from there aren't tailor
made to exactly look and feel how we want them
to be. So that's fair enough. That's essentially between a
rock and a hard place. And then you have the CIO because
of. Yes. Oh dear. Oh dear, indeed.
Essentially, the cio, he was not aware of the situation.
So that's an interesting board meeting where you want to be a fly on the
wall. Yeah, yeah. But he wasn't aware of the situation.
But yeah, he was even in a worse place than me because he
locally responsible for security and risk.
You also need to report the CEO.
So what essentially we landed on was
that we would have a migration project from all
the data that's on there to migrate it into the system
of head office. And then at the same time, the CIO
and the CEO lobbied Head office to have reports that
looked more like the ones that they were using locally, which was a very
back office backroom politics happened because company
was German and you know how they can be sticklers for rules.
Essentially we had to have very senior German directors
agree on the fact that we did not immediately
power down those machines. We migrated all the data into the
data of the platform, the head office,
and on top of that, that the existing reports needed to
be rebuilt. Essentially all of that happened over the course of around
12 months. So quite successfully, quite happy. Until.
Because the story doesn't end there until I suddenly
get. I think we were nine months in, in that migration.
I get a call from the VP of sales,
hey, we have an issue with one of the new reports.
Okay, fine, but we don't do those. Those are done by head
office. Yeah, but they don't want to help us with these reports.
There's going to be a reason for that, I feel.
Yes. So essentially what they had been
doing ever since we started migrating the data into the corporate system
was dumping that data into a
Power BI data lake. And there they started
creating new reports again. So they basically made
a copy of what was provided and then worked on the copy. Yes,
exactly. And yeah, sorry to jump in. Do I
want to know where that server was running? Well, that was
running on the corporate network in corporate cloud.
Surprised? Yeah. So all of those things were good. Let us
say people from compliance and Audit were
not aware about the fact that commercial data was now living
in a much more publicly available or in a much more publicly
available data space. Yeah, but yeah, they just said,
okay, we go from Access and Excel, we go to Power bi
and we just keep on doing what we were doing. That was their plan
until the moment that they noticed that they did not have sufficient Power
BI knowledge to rebuild the reports. Because they started building reports
because corporate set for quite a big bunch of their reports.
There's no need to have those reports. You have an existing report
where you can activate a filter and you have exactly what you need.
But no, they prefer to have 50 different reports each
for one client instead of one report where you then just selected
the client. That was too much of a bother. Oh,
yeah, okay. Yeah, I see what they were doing and why they were doing
it. And I also see why head office would
think otherwise. Yes, indeed. So, yeah, then they
came to us about Power BI thing. I just cleared. We don't do Power bi,
period. And when I left that
company, they were recruiting a Power BI specialist
that would be living in the sales department specifically
for that. And in parallel, they were trying to fight off the
audit and compliance department of head office that were
informed about the fact that the data was. They will
get to it because once you get to the corporate systems, you will
have some kind of Audit running over it and then it will pop up with
a thing. And on that thing I have a question that
how could that system remain undetected for so
long, even in user network? Of course it's
10 years ago you wouldn't be as proactive security wise as you
are today. Exactly. But today your average
generic network sweep will find a device
that's not in your domain, in user network and then pick that up.
Hopefully. Hopefully indeed. Plus if you do network
authentication, any device not in the domain cannot get onto the
network. Yeah. So I assume
that in this day and age it would be significantly more difficult. But at the
same time this was not set up without the compliance of somebody
from it. Let's be honest, it couldn't have been set up,
especially with patching and the fact that it was in a closet and all those
things. It was professionally installed.
Somebody knew. Somebody knew at what one time?
And honestly I think that's the biggest thing here. Somebody knew.
Somebody did not inform the right people or the right people were
informed and they did not inform their successors afterwards.
Yeah. But business essentially they thought they were
doing everything above the board. Yeah. Which I can see it happen that
local, it said okay, they have a thing here and we will do
it properly, we will put it away, we'll do some basic setup,
yada yada. Yeah. And they do the things to avoid being detected by
corporate because let's be honest, a local CEO still has the
power to fire you even if you say no for the right reasons. That's always
a thing. And then so on the CEO was that country level or regional
level? That was country level. Country level. Okay. So not that super high up
in the food chain, but locally some mandate and yeah,
essentially locally he could decide everything as
long as it in theory it was aligned the corporate rules, the corporate
guy. So yeah, but yeah, that's something
that hopefully today would not be as obvious. Certainly not in larger corps
where they have woken up on their cybersecurity level
that most of them should do internal sweeps and internal
scans and network authentication. But I can imagine that
10 years ago this was not as much the case.
So. Yeah, but at the same time,
if sufficient people are complicit
in the story, it can happen even in this day and age.
Yes, there's sufficient tooling
where it can be detected. But if you do the right things, it's still
not all that obvious. No, no. That is,
you will need a bit more accomplices. But when you
do play the local versus regional versus global
card, you will Find a lot of folks that say, yeah, okay, but we,
we need this locally and we're not getting it from higher up, so we will
do it ourselves. That is a very common
reflex to have, actually. Yeah. And origin stories
of any shadow RTS that is created is exactly because
of that. Somebody has a specific need and the need
is not fulfilled by the rest of the organization.
So they're going to try and find a solution. Yeah. That on its own
is good. You know, you want to move ahead in whatever
you need to do and what you're doing. But yeah, then in the end it's
the non existence of the run phase. It was set up once and they
basically for all these years forgot about it because that network
share was always there and it worked. From what I understood
is that there was no data loss because once you added some disks to
the container or the RAID array, the thing booted up again and
everything came back online. So they were extremely lucky that
there was no data loss there. Yeah. Honestly,
what I just did was just cleared out the temp files.
Okay. Yeah. So you go into, you boot into maintenance mode, you free up some
space. Yes. And I can imagine from 10 years you will have some
megabytes of temp space available that the page file can extend
again. Exactly. And then I told them, okay, now you have to clean
up everything that you no longer legally need. Just purge it.
Yeah. So would it be because it's basically
a file share? Yeah. Was it suggested as a workaround?
Hey, you have your own system here on your own little hidden file server that
you actually move it to a corporate file server and then you
start using it there and then you work out the politics with her doctor in
Germany later, because it's a German company. Was that considered or an option or
was it just too far fetched from my brain? No, no, it was
asked. But essentially then the way the file shares were
set up was that you had your local personal network share
and then essentially you only had a file share used for transfer,
for data transfer. Okay. You did not any, any shared
data was more. Was more stored on SharePoint and,
and stuff like that. And yeah, SharePoint didn't allow for the amount
of file sizes that they were using and the transfer was
purged on a weekly basis. I do get it. It is the same moral penalty
of using Excel as a database as it is to use SharePoint as a
file server.
This podcast features Jack Smith and guests. We say thank
you to our demoscene friends who helped making this podcast possible. Commander Homer for editing
Danko for music and audio advice, Ned Poet for additional voiceovers.
Exactly. And even so, the file size were just too
big for SharePoint at the time that was set up in that SharePoint environment
was 50 megabytes. Not a single report I know is under
that. Exactly. I do understand their, I wouldn't call
it business reason, but that's exactly what it is. Yeah. But essentially
their business reason was that they did not like the new
system. No. Because if I then asked back, it coincided
with the rollout of the new sales system that corporate rolled out.
We're not doing this. Exactly. They said we are not doing this. We are
just entering the basic minimum. So corporate is happy and we will
just keep on using our Excels and access files that we have.
And since they were no longer allowed to do it on a
network share like they did before and then they put their own, their own
little box in. Exactly. They saw a problem and they
solved it from their perspective with all the risks and everything attached to
it and everybody just forgot about it because it was just there. Yeah, exactly.
Specifically, if you like thinking that, well, if when the CEO said
well yeah, no, it's running there, there's no technical background
left, suddenly it dies and then it becomes your problem.
Like what chair, what server?
I think they were extremely lucky that it was just running out of
disk space and not one of the disk containers dying
and they effectively having lost all their data.
Because then I don't think corporate would have let this
one slide. No, no, honestly I wouldn't really say that corporate
wouldn't have let it slide. The fact that that happened because from
corporate perspective, all sales data that needed to be there was there.
Indeed. And sales were going all good and sales was going all good.
I think the only issue corporates would have had with that
particular situation was that if there would have been a drop in sales,
but as long as sales would have just delivered
and hit their target, they couldn't care less that the additional data
that corporate didn't really was interested in that that was lost.
But this was mainly the entire local sales organization was,
was organized to use all that specific data to give reports to
clients and do client health analysis and all of those
things, those were based on that additional local data.
And then you turn out surprised how it gets
frustrated if the entire system is basically working against their own
policies and everybody is happy as long as nothing goes
wrong and when things go wrong, it's it's fault. Yes. That's a wonderful summary.
Yeah, absolutely. Absolutely. And honestly,
this is Not a rarity, folks. Let's be honest. This is
a common story. There are plenty of you guys around
and I'm sure that know of a desktop sitting on their desk
or laptop somewhere. In a desk that is running
a daily job or that is running a file share
or a web server or whatnot. Because somebody
at one time decided that that was the most convenient way forward
and that they would get around to doing it properly
later on. You get expensive oopsies. Yes,
expensive oopsies with consequences that
can cost you dearly. Oh, absolutely.
Yeah. Maybe I'll just end on that. Sure. There's one small anecdote
about Shadow IT by business that I want to share with each
and every one of you. And I was working in a company and we just
had a change of senior management, okay. So the
entire board was replaced for commercial reasons.
Reasons are reasons, for reasons are reasons.
And the director at that particular time said,
made a very strong statement. He said, in my previous
company, we suffered tremendously from Shadow it.
So I vouch here that I will no longer
tolerate any Shadow IT being done. Okay. In between now
and the next three years, all Shadow IT will be thrown out
the window and will be replaced by properly maintained
systems. Okay, good, great. We were very happy.
Yeah. As an IT department. Yes. Until the next
part of his speech. Game. Next part of his speech. I don't
remember it word for word anymore. It's been too long.
But he essentially said, because somebody asked and what
about the needs that Shadow IT helps
fulfill? And he said, yeah, of course, those needs remain.
So we will empower all business users to develop
their own little programs and scripts in order to fulfill
those needs and those will be supported by it.
This is intentional silence from my end now. Yeah, it's like.
Alrighty then. Yes,
yes. Oh dear.
So I think that's one other thing you should
also be concerned about. Shadow IT is not always recognized as Shadow
rt, but also sometimes it's known under user
empowerment. Absolutely.
Good, Bob. Let's round off here. This was very,
very insightful. I think from every possible end.
Thanks everyone again for tuning in. As always. You can
find our previous episodes on ithorrorstories.eu. you can
find us where you find. Find your podcast that is on
Apple Music, on YouTube, on Spotify, on Deezer, and wherever
you find your panic attacks, reach out to
us. HelloThorrorStories. EU check our merchandise.
It helps paying for all the costs we do here with the podcast.
Otherwise we are just in it for the likes and the subscribes and your comments.
So if you like this episode please give us a little
thumbs up or a heart. Any merge is shop.it HorrorStories
EU. Thank you for tuning in.
See you next time. Any last words Bob from your end?
Yeah, pretty much the same as you. Thanks for
listening everyone. And yeah, I'm not going to do the like and
subscribe come on YouTube thing, but if you
think that somebody else in your acquaintances might
like these stories, don't forget to share. That's highly
appreciated. That's a good one. Thanks a lot. Cheers everyone. Bye bye.
Cheers guys.
The content of this podcast is intended for entertainment purposes
only and is meant to humorously explore various tech related situations.
Any resemblance to actual events or real persons, living or dead
instead is purely coincidental. We ridicule situations,
never individuals or groups. Listener discretion
is advised and we encourage everyone to approach technology with a sense of humor
and an open mind.