Hey, everyone. Welcome back to it Horror Stories with Jack Smith.
I am Jack Smith, and this time across the table, I am joined
by Philip. Good evening, sir. Hi. Good evening. How are
you doing? Pretty well. Things as usual. Well, looking at
the story you brought, things as usual does not sound really motivating.
Yeah, it's been a wild ride. Yeah.
Because we did the DevOps royale with cheese
episode 8 and we did then the shadow it reports episode 9 a
few months ago, and you basically came up with something
similar, but yet very different. Yeah, that I did want
to throw in here because the original plan was to. Yeah, let's do more
about shadowit because it is so broad and even I didn't see this one
coming. So I am going to give you
the microphone. I am going to jump in left and right wherever I can.
And I think I can be interesting, which is every five seconds. Everybody be
warned. The microphone, sir Philip, is yours.
Okay, so to give you some backstory,
I was working as a contractor for a subcontractor for Telco.
It was, you could say, a pretty big name
around these parts. Okay. But company I worked for was basically
a subcontractor for them. And the reality was that that
was more of a. Yay. We're a telecommunication. Telecommunications company.
Yeehaw. So if you say a subcontractor,
like an entire service provider, as in, we'll take care of your help desk,
we'll take care of your field services. It was mostly
field services, so we didn't have anything to do with end user of said Telco
company, but it was more of the installation of the
infrastructure and things like that. So I am now confused and I have questions
about you guys. Maybe not you yourself. You were going out on the streets in
setting up, say, the infrastructure so the modems, and you don't deal with the end
user. How did you manage that? Well, it was the backend infrastructure,
so. Oh, okay. Yeah, yeah. So the end users, they,
to be fair, they have subcontractors for that as well,
but I see, yeah. So you were doing services for
a different contractor that dealt with the end users. Yes,
exactly. Isn't the wonder. Isn't the wonderful world of outsourcing
overly needlessly complicated? Oh, it's. Yeah, it's overly
needlessly complicated by design and I think they really want to keep it that
way. Wonderful, wonderful framing in
complexity. Yeah, please go on. This particular subcontractor
was what I like to call cowboy it. So I think that's a phrase
that your listeners might be familiar with. I'm familiar
with it. Which is. Which.
Yep. So you're. You're basically shooting it from the hip.
Yep, basically. That kind of. What could go wrong? Yeah,
exactly. There was this particular. I think it was a Tuesday or a
Wednesday. I don't remember that in detail, but. But I came
in in the morning as usual, you know, motivated and everything.
Motivated and everything. I went to the coffee
machine, got my, my coffee to start
the day properly. And yeah, then I noticed that a lot of the help
desk people were pretty frantic. I think
about half an hour after I arrived, there were some murmurs of,
yeah, there's a problem with the platform. So the
platform, quote, unquote, the platform about half an
hour after I started. Because that was also the time that the
field engineers would be. Oh, yeah, so you would come in early and then
once they get to the first on site customer, you're there just in case.
Yes, exactly. Okay. And then, well, poof, it was
getting a bit more busy than usual. They got a lot of calls,
a bit more than they usually get. They do get a lot of calls usually,
but this was this time it was. Yeah, it was a lot different. I heard
some of the help desk engineers say, oh, yeah, but the platform is down.
And then another one responded, but, no, it's not down. It's always this slow.
They just need to be patient and blah, blah, blah. The fact that it was
slow is important, so keep that in mind. But they were saying, yeah,
it can be down. It was working perfectly yesterday and nothing changed.
And nothing changed, of course, nothing changed. Famous last words,
obviously. Well, at least you're not updating in
production on a Friday. So at least it was on a Monday then. Yeah,
yeah, exactly. I see no problem. No further problem. That's kind of
the advantage of being in a cowboy IT environment. They kind of
learned their lesson in a way that they shouldn't update things on Friday,
especially not when there's happy hour on Friday. So priorities,
priorities. Exactly. Lovely.
It started to get more busy. A lot more
calls were coming in and there were a lot of. A lot of people saying,
but yeah, it is down, it is down and this is a disaster, and blah,
blah, blah. As one does now, to give you some context, the platform
they were talking about was a ticketing application for the field engineers. So basically
There were about 200 field engineers and they would log into this specific
application. It was basically to follow up on their work.
So if they had to do an installation, they would do the logging in there.
They would get their tickets there to Start the work and you know,
basically their entire day planning was in there. Yes, basically.
Yeah, yeah, yeah. I can imagine. If that's not
reachable, it's not going to be a good day. Yeah, yeah, you could say
that. It was fairly quote unquote, business critical. They were used
to it being slow in the sense that. Slow. I don't know if your
listeners are of the age that they were still using 56k modems,
but you can kind of compare it with that. Our beloved statistics on
YouTube, Spotify and Apple Music confirm
the age bracket. So yes, we know. So you can imagine about two or three
minutes per page refresh that wasn't out of the ordinary. That's not
even 56K modem. That is. That is male pigeon speed.
Yes, exactly. Actually, I think there was this study that
pigeons might have been faster actually. But why does my brain now goes
to avian carrier networks over tribute? Yes, exactly.
You know exactly what I'm talking about. Oh yes, we have discussed this multiple times.
It was even that bad some days. So that
you could almost prepare a full English background. And then it's logical that they tell
you, oh, just wait a few minutes, it'll get there. Yeah, exactly.
I guess we're going to come back to this, but I don't
want to know the hardware platform or the setup or the whatever, because too many
questions now. And we are not in a third call
at the moment, so I'm not going to care yet. You could say that
it would be surprising for a telco company to have a setup like this.
But more on that later. We have mentioned mainframes. Fair enough.
In a few episodes. So I'm ready for
most of things, like not sure that I want to talk about a
VAX or a Wang, but this time instead
of just it being slow, there were some people
that were getting NGINX pages. So they were getting.
For those familiar with nginx, you can use it
as proxy. So that means that you can get error pages from NGINX
itself. So you can get 404s and 504s from NGINX.
It's not just a normal browser saying this page cannot be
found. It's something in the middle says whatever I need to
talk to is not there. Yeah. This endpoint that doesn't exist, please investigate.
So it's going to be something else. Yeah, exactly. Whoopsie daisy.
Yeah. So you can imagine that the project manager on duty was kind of
sweating bullets and having an aneurysm. Probably on the verge of starting
with A drinking problem. But you can imagine the chaos that
was ensuing at that point. So. Oh, wonderful day.
Yeah, exactly. So just a regular Tuesday.
Or not. So we tried pinpointing the
issue. The help desk people were
frantically running around like headless chickens. At that point,
a junior I remember very well was completely
panicking saying that, yeah, I can't even ping the machine.
It's not reachable. I don't know what's going on. Another one was frantically
tapping his keyboard, trying to find out from the logs from the NGINX
machine what was going on and basic troubleshooting steps,
but things weren't going that well. Kind of an understanding. Yeah, because you
have described headless chickens so that basically everybody is doing something
and there's zero coordination going on. Yes, exactly. So of
course I was there as a contractor. So you
tend to shut up. I tend to shut up.
I just want to see the world burn. If something comes to me, I'll give
my opinion. Yeah, exactly. But even, even at some point it
was like, okay, this is just getting too much now. Mind you, I wasn't
fully aware of their infrastructure because I said, not my
job. You're not. You're there to do one thing, not manage the infrastructure. Yes,
exactly. Yeah. I was more of the development side at that point.
So I was just working on my stuff locally. So whatever,
they'll solve it, I guess at some point. Yeah, at the server it
is not your problem as such, you're not impacted. But can I take
an estimated guess here to what you've mentioned already?
NGINX says I can't reach the machine. Somewhere someone
found out the internal IP and said, well, it's not
responding. Yes. If you're still running around like a bunch of headless chickens after
that point, instead of trying to find out where Utopic idea.
Your documentation says that IP address is located.
Yeah. So you can stop
running around the chicken coop. Yeah, there's your root. Cause the IP is down.
And then why it is down? Well, let's go put on your data helmet
and let's go digging. Yes, exactly. Now, to be fair, those hell
deskers, they were not really on the operations side
of things. So it was just the go in between. Is the hell of
desk guys. They don't know how the backend works. They know the tool, they know
their pages and that's all what their pay bracket is there for.
So absolutely nothing against the hell of this guy.
We choose not to have any subscription, model nor sponsoring in order to
keep our stories accessible for each and Everyone to support us,
please check our merchandise@shop.it horror stories.us or
buy us a coffee@co dashee.com it horror stories.
Yeah, exactly. So I can imagine them not
being, oh, yeah, we need to check the server itself and blah,
blah, blah. Usually you would go to the application owner now and say,
hey, your thing is down. And then in your escalation matrix, you can find
this person's contact details and hope he's not on vacation for three weeks. Yeah,
exactly. Or create a priority one ticket and throw it up the chain, basically.
So if you can't find anyone, you escalate up and maybe somebody knows someone.
Indeed. Of course. I was in the emergency meeting and
I kind of saw things unfold. And then I
being a little bit proactive, I was like,
yeah, but where is this machine located?
Is it cloud hosted? Is it in a data center somewhere?
And then one of the other DevOps people said,
oh yeah, but this particular server is in our data
center. And I was like, okay, sure, but that's kind of my question. Where is
this data center located? Because if you can't ping the machine, then something
is wrong there. Apparently the data center was local in the
sense that it was in the same building. Oh, that's a good one.
It was basically the next room over from where we were sitting.
This is going too easy. As you
can imagine. I was already half intrigued,
half horrified. So I suggested to let's have
a look at that machine. Are we able to access that room? Is there a
possibility that we can just go there and have a look
with our own eyes? What's going on? Are the blinking lights still blinking? Would be
your first check. Is there something. Yeah, yeah, yeah. Indeed. The project
manager was, yeah, that's not a bad idea. And it's like, yeah,
okay, I sometimes have those. So we went to investigate
on how to access that room because it's like
a broom closet. So I mean, we're looking for the
key for this specific room. And that was a point where
we went to the receptionist, our dearest
Linda, the gatekeeper high priest. Pieces of the key cabinet.
I know the type. Yeah. So after
a difficult and perilous search through the drawer of mystery,
she emerged victorious with small and unassuming
metallic key. We managed to get into the quote unquote
data center. So no keypad, no badge to enter security
through obscurity. You'll never guess that this holds a data room.
Yeah, exactly. And the only thing stopping
you from entering was basically just Linda Wielder of the one Key to rule them
all. So, so yeah,
we, we entered this so called data center.
Now at this point, obviously I wasn't expecting any biometric scanners
or high level security detail or. No, no,
you're. No, we've, we have, we have passed this level exactly.
But when we entered this specific
room, it was honestly even worse than I expected.
So my expectations were already low. But let's
just say that we entered that room and instead of a data center, we found
ourselves in a glorified storage closet with commitment issues.
Okay, so how many people were on the floor usually?
So are you talking 20, 50, 100 people? This was
not the main office of this specific telco. Obviously at any given time
I would say about maybe 30 people. Okay, yeah,
yeah. So 30 people would put you into the thing of. Unless there's a
security audit coming up, we're not gonna do anything with the data room
and just forget it exists. Yes, basically that. Yeah.
And then if you get an audit, you put up some
plywood walls, you know, put in the door and say, well, it's separated
now. Yeah. So this is officially a storage closet. Now here's a broom,
here's a bucket, you know, and here's. Well, so. But were the machines
in a rack? Yes, Funnily enough, they did have two small,
I want to say 42U racks there.
42 is full size. They were filled with a lot
of hardware. I'll get back to that later. But. Well,
let's just say that there were still some machines in business there that were
old enough to have a driver's license. So what you often see is
a machine or a service will get decommissioned, but then they forget the
machine and it's left powered on, screaming for NT4 Service Pack
6 updates. Yes. In the distance with two of the
three RAID disks in error. So lots of screaming from piezo
speakers going on. Yeah, you know, it's bad when the servers look like this.
They've seen things. I can imagine the,
the range of compact servers in front of me.
Yeah, you're, you're that have seen things. Yes. Yeah,
you're not far off. So, I mean a couple of them were
still blinking away. But like in that I'm trying my best,
please let me retire kind of way, somebody shoot me.
Somebody shoot me. Exactly. So we went ahead and fans
groaning left and right aside, we went to investigate the machine that was
allegedly running this specific application. Allegedly.
But of course, yeah. My hopes of it still being up and running
were lower than the remaining warranty of any machine in that room. So you can
kind of imagine the situation at this point. Yeah, when you enter the room,
you see all the old hardware and you see more red than green lights blinking,
running. And finally the RAID container died or
the thing just gave up and is now blue screened
or Colonel panicked. Whatever platform you're using, did you pray
to Saint Rebootius?
To be honest, I think if you would reboot any of those machines,
they wouldn't start up anymore. So even praying to Saint Rebootius at
that point would be proven useless, I suppose.
Well, okay, so that
wasn't done. What was next? Well, next we were trying
to locate this specific machines and honestly
I was surprised that those machines were labeled at all,
seeing the state of things. But I think we've been looking for
like an hour or so. Meanwhile, helpdesk still frantically
running around like headless chickens. But nothing in the rack really matched at
what we were looking for because you had an IP address, therefore you hopefully had
a DNS name. Then we are looking for SRV005.
Exactly. So we were basically checking the labels at this
point. But the one that we had on file, spoiler alert,
the file that I'm referring to was basically an Excel sheet that was on
some local share hosted somewhere. But yeah,
we couldn't really find it. Oh, we've had discussions
here in previous episodes where we would have given a lot of money for
and for basic Excel sheets somewhere. So that's okay.
Okay, so yeah, that was one thing that they kind of did. Okayish.
Nice. We were staring at a rack full
of cobwebs, ancient hardware, a lot of cable, spaghetti and sadness.
And I just happened to stare closer into a
suspicious opening about the size of two units tall. So the server
that it was supposed to running on was missing
in action. As in it was gone. Instead, what I saw was
just the saddest pair of cables that I've ever seen in a
quote unquote professional environment. A VGA, a PS2 and
a power cable. Oh no, not even that. Oh dear.
What I saw there was a patch cable. So so
far, nothing unexpected. And the other cable that I saw, there was
a micro USB cable. Just.
That's it. Okay, awkward silence moment.
Micro usb. USB C. USB A.
Because if it was USB C, could have been a laptop. No, it wasn't USB
C. And this was pre USB C Laptop charger days. Oh dear.
Yes. Please don't tell me that they did that. Yeah, I think
you can kind of see where this is going. So eventually the cats came crawling
out of the bag. And keep in mind, this was around the time that things
like home automation and stuff like that and tinkering
with hardware was getting popular. So apparently one of the engineers,
bless his enthusiasm, had taken the machine that was
connected their home to experiment and play around with.
So it was basically running on a Raspberry
PI. This poses many questions, but also gives a lot of answers.
Yes. As to why it was going so slow.
Yes, exactly. It was just a basic Raspberry PI. I think it
was one of the first models even that was running the ticketing system,
which was needed for about 200 field engineers,
where they logged in every single day to get to their
tickets. So if I understand correctly, you had
200 engineers on a first second
generation Raspberry PI doing your ticketing
for all your field services. Yeah, pretty much. So I think the
thing that actually saved the fact that it was under
this kind of load was the nginx proxy doing the caching.
Yeah, basically. So at least that's my assumption for now,
because I don't know the exact details of that setup,
but sounds very plausible nonetheless. Nothing says production grades,
infrastructure, hosting an app for 200 field engineers like a €35
hobby board normally used to teach kids Python.
This podcast features Jack Smith and guests. We say thank you to
our demoscene friends who helped making this podcast possible. Commander Homer
for editing, Danko for music and audio advice, NetPoet for
additional voiceovers.
To be fair, that engineer also assumed that no one would be insane
enough to go from test deployment on a PI to production because it works.
I can't be too upset of at that specific engineer,
to be honest. Well, you don't know who set it live there. It could be
the engineer, could be someone else that says, oh, this was made in test,
it works. Let's just push it out and we'll move it later. Yeah, exactly.
So finding the guilty person for this will be a different endeavor.
Yeah, I think there's better use of your time than trying to
figure out what exactly. As we have mentioned a few times in
episode 10 when Anonymous attacks on the DDoS, your focus
is to get back online and not to finger point in a period of
crisis. So yeah, indeed. Starting to point fingers
during your emergency meetings and trying to already
throw people under the bus. Wrong place, wrong time. Let's get your
200 field guys back online now.
Going on the ledge here. Because it's a Raspberry PI.
Do I have to wish for a backup?
Because the storage went with the device. Yeah,
so funny thing about that is that we were
kind of just in time, I think, to get the SD card.
Yes, the SD cards back. So it's really fun to run a SQLite
database on an SD card. So you found the actual SD card?
Yes, where the engineer was like,
oh yeah. Because that's how we found out that the engineer took it home.
Is like oh, so the guy that took it home was still there.
Yes, sure,
yeah. Like I said, cowboy it. I mean it's.
Yeah, this goes for a Yeepik I.
Yeah, exactly. So yee haw and let's go. But we
got the SD card back, which thankfully wasn't formatted.
That specific engineer was like, I'm just going to use a different SD
card, you know, just in case. So again, the guy wasn't incompetent
or anything like that. But it's. Oh no, no, no, it' oh yeah,
this is my old test Raspberry PI. I'll just take it out and put it
back in the drawer and I have a bigger SD card for it now,
so I'll just take out the old one. Yeah, pretty much that. So. Oh dear,
we got that SD card back. Of course, I do not want to know the
write cycles on that thing, especially since it's a SQLite database.
Well, yes. And then we ended
up migrating the app to a dedicated VM with
backups this time and postgres database.
Like a real life growing up application on an actual ESX
server that stayed in the building. The funny thing is that we also managed to
convince finance to fork over the cache for a
semi decent box to migrate all the other legacy applications
to the poor hardware that was running there for over 25
years could finally be put to rest. The atypical
scenario of no, no, it's all ok, okay, it's fine, we don't need a maintenance
contract. And then when things go poof. Yeah,
suddenly everything is possible because they do realize that.
Oh yeah, this is useful. Yes, exactly.
So it's an FAFO moment. Kind of so. And that's
basically how it went as far as I know. I think this
specific application is still running unless they move to something completely different.
But when I left there, it was running happily on that new machine. So.
Well, as long as the guy who maintains it is still there or
is reachable, I can imagine that indeed it is going to be still
in use. And well, as you also move to more
powerful hardware. Did the field engineers come back with, wow,
this is now going so blazingly fast. Yeah, I think they had some
very good feedback when it came to the speed
of the application. So I'm pretty sure that that the field engineers were
happy at that point. Yeah, I'm now wondering as to people listening,
suddenly realizing oh crap, we have a Raspberry PI in
the data center. And now it's okay
because I've used them as well for a network probe or logging
that you just plug it into the network and you plug it into a switch
where you switch port where you give all the VLANs that you can do tracking
for performance and everything and just some logging that's perfectly fine and even yep
if you want to test something out quickly and you have something specifically with the
later PI 3s and 4s, they're okay ish to
do that. But yeah, you should never run that into production. Although I see many
social media posts and YouTubes on I made myself
a Raspberry PI file server cluster with some additional boards
where you then put SSDs on it or similar. I'm not sure
that is a good idea. Although yes, it has more CPU power than your
average Synology. Yeah, and there have been some issues with Synology
lately as well to give an example. But yeah, that's a completely
different story. Of course I am an avid Synology user myself.
Yes, it is not perfect, but it is as idiot
proof that I can work with it. Fair enough. So to jump back on the
Raspberry PI. Okay, it happens. It happened. It was resolved.
Yes, but again only when things went poof. Yeah, always take
precautions before it's too late. And this is as I mentioned
it earlier, having been in well, not with subcontractors, but when you're
doing mager and acquisition, you also analyze
the local data center. Is it a closet? Is it a cabinet? Is it secure?
Is there air conditioning? Is there a backup? And very
often you come back with no. It's only when you're starting to
get into a few hundred people organization that there is time and
budget for that. Otherwise you start out with a company 5 people,
10 people and you it grows organically and you end
up with this spaghetti and old servers standing there waiting
to be shut down. Which never happens until you know, somebody comes up and
cleans up or until something goes really really bad as this
was the case here. If I want to blame not
blame, but if I would have been the guy writing the report here,
question would be okay, why hasn't this been noticed?
Why didn' the telco that this subcontractor was working for perform an
audit of its subcontractors IT systems to ensure that their other subcontractor
had their application available when it needed to be. And you find that out the
moment you do a site visit and you see your data room guarded
by Linda, the key master. That is one big red flag that
needs to be addressed. And that didn't happen or wasn't picked up. In my
experience, what usually happens here, if your quote unquote data center
is a glorified storage room with a matching broomstick and bucket
and it can be opened with a key from a filing cabinet. That's straight from
the 90s. We've had in was it episode five,
Paris rooftops, we had network going out because.
Exactly. In the maintenance cabinet, the cleaning crew would
unplug the switch for the mopping machine of the floor and then plug it back
in. So, yes, this is what happens.
Unfortunately, that's what you're saying now is something that I've heard
way too much before. So wait until you've lived through it. Yeah,
a few times. For the love of God, don't host production
applications on a Raspberry PI. I mean, like you said, they're good for like monitoring
and stuff like that. And I actually use one at home for a headless
server that I'm running, which I use as a VNC server or if it's completely
down or like a jump host. Yeah, exactly. So for
those things, perfectly fine, even for things like home automation,
but that's in a home setting. I've got a few Raspberry PIs left and right
on customer locations where they're being exactly used as
a jump host with a VPN server on it so that if need be,
it will save you a two hour drive to walk in and restart the
service. Yeah, so yes, that's what they're ideal for. And give your guys
a checklist once every month, log in, do get update so that
everything is patched and everything. And then you're okay. Of course, not in
enterprises, but small businesses, no problem. It works. Everything is better
than a mini tower case under a desk. Yeah, of Linda.
Exactly. Good, good. Filip and Linda,
thank you for your input and sharing everything here. Yeah, you're very welcome.
Absolutely. A pleasure having you here. So this has been it, Horror stories
with Jack Smith. As always, you can find us on Spotify, Apple Music,
YouTube, Deezer and wherever you find your podcast. New episodes
at least once a month. We have merchandise. You can find the merch@shop.it
horrorstories, EU or support us via Ko Fi
if you are in the US you can change your shop region. At the bottom
of the page we have social media, Instagram, TikTok,
Facebook, LinkedIn, Blue sky and Mastodon and all
other links are on our website. IT Horror Stories eu.
Thank you for tuning in. Until next time. And don't forget you're
one of us. Bye bye.
The content of this podcast is intended for entertainment purposes
only and is meant to humorously explore various tech related situations.
Any resemblance to actual events or real persons, living or
dead, is purely coincidental. We ridicule situations,
never individuals or groups. Listener discretion is
advised and we encourage everyone to approach technology with a sense
of humor and an open mind.