I work at a medium size company with hundreds of Linux servers and none of them get updated. Because it’s more important that they keep running as they are than to have the latest updates. I bet this is very common for most companies.
There is nothing more important than security patches on a system.
I used to work at an FMI, which’s motto was “keep things stable”. Even the ciso department bought that crap. Until we hired a white hat hacker. The only thing given was the name of the company. He managed to get into the building, access an employee’s workstation and install a root kit on one of the most important financial message tracking systems (you know, the one that instructs other systems to transfer money), using a security bug, which would have been patched if they kept a regular (security) update cycle. After shit hit the fan, many people were fired and an update cycle was introduced.
No system is important enough to not patch. And if you believe it is, you’re wrong.
Yeah, but that just takes way too much work. You think I really care about the company’s/bank’s money if I’m not getting paid enough for that job? Security patches can also introduce new problems, like x changes, so y doesn’t work, so the main app doesn’t work… and what, then I have to manually edit code, introduce the thing that x relied on so that y can work again.
I’m sorry, but this is not your average IT department’s job… or if it is, I expect a damn good compensation for it.
I’ve updated and rolled back snapshots because of shit like this… nah, not gonna try and figure out what the problem was… at least not for the salary I’m currently getting paid. If it burns, it burns, so be it.
I’d be surprised if you actually saw anything change from security updates tbh, I don’t think I’ve ever seen anything break from a quick patch.
Dist upgrades are when things might break, but they’re only once every few years. Leave them too long though and you may end up with compatibility issues if you need to make changes.
Fair enough if you’re not getting paid enough, the company should hire more people to stay on top of that though.
Usually you upgrade everything though, not just sec patches. And it’s a risk that something stops working, and nobody wants to spend time on that…
Exactly… because it’s tidious and time consuming… and I won’t get extra pay for it. Meanwhile I’m also expected to do everything else I do… sorry, just not gonna bother at all.
“Way too much work” — if you ever said that where I work I’d fire you or not hire you in a heartbeat. An administrator’s role is not only to the stability of the system but the security too. You’re a hackers wet dream.
One, you have no idea how much or little I’m getting paid. Two, you have no idea where I live and the struggles I have to face every day. Three, even if I do work “as expected”, I won’t get paid more (agan, you don’t know where I live).
It’s real easy to bitch about work ethics on a full stomach while getting back from work in a nice car with heated leather seats.
Every individual concern you’ve brought up in this thread is valid and not uncommon. The important part that @ramble81@lemm.ee is focusing on is that your mindset on how you approach these challenges is incredibly unprofessional.
Having more to do than you realistically have time to do is the reality of working in IT. Everything the business does ultimately comes back to IT at some point in the process, so we naturally have to work with every single branch of the business.
Being underpaid is a reality of work for most people in the modern world. The professional thing to do is to decouple your feelings of how you’re being paid from individual tasks or duties that are expected of you. “I’m not paid enough to deal with this” should be limited to tasks outside of your scope of work. If you’re ultimately not paid enough to do your job, complaining about individual tasks that are part of your job being bnove your paygrade is just saying you aren’t willing to do your job. I can’t tell you the best solution to your pay situation. Maybe changing jobs or even changing industries will help, but also changing your mindset can do wonders for your mental health. For example shifting to instead saying to yourself “I’m woefully underpaid but at least I’m working in IT and not at X” can greatly help ease the pain until you reach whatever milestone which does help improve your situation
What you need to do regarding your workload is have a conversation with your manager/superior about prioritization. You say “hey, I have this this this and this that need to get done right now and I can’t realistically do them all, what do you want me to prioritize and deprioritize” and if something important hasn’t been given priority in a long time (such as patches) you need to then push back and say “we haven’t been able to apply security patches in quite a while, I think we should reprioritize this so we can put some time into patching this week” this is how you manage a gigantic workload is by shifting priorities. The longer important maintenance tasks are ignored, the larger the impact and the harder it will be to complete the tasks
The important part that @ramble81@lemm.ee is focusing on is that your mindset on how you approach these challenges is incredibly unprofessional.
I know it is.
Now, ask me why that is.
The professional thing to do is to decouple your feelings of how you’re being paid from individual tasks or duties that are expected of you. “I’m not paid enough to deal with this” should be limited to tasks outside of your scope of work.
Lol 😂, no one has actually told me to keep the servers up to date, the only thing I was ever told was “keep shit running”. I’ve done updates on my own incentive, since I’m the senior IT engineer in the company. When things turn to shit after an update, hey, I’m rolling back a snapshot, I did not sign up for this 🙌.
And that is basically it 🤷… I get the same salary regardless if I do them or not. After a few failed ones, I just gave up. F it, not worth the time or the effort.
If you’re ultimately not paid enough to do your job, complaining about individual tasks that are part of your job being bnove your paygrade is just saying you aren’t willing to do your job.
Nope, I’m saying “Fuck you, pay me!”.
You think they care about updates and security? I’ve mentioned it a few times at meetings… “yeah, we’ll talk about that later”. OK 🤷. They obviously have no idea how fucked up things can get if you’re not up to date regarding security… but hey, they have been warned more than a few times 🤷.
What you need to do regarding your workload is have a conversation with your manager/superior about prioritization. You say “hey, I have this this this and this that need to get done right now and I can’t realistically do them all, what do you want me to prioritize and deprioritize” and if something important hasn’t been given priority in a long time (such as patches) you need to then push back and say “we haven’t been able to apply security patches in quite a while, I think we should reprioritize this so we can put some time into patching this week” this is how you manage a gigantic workload is by shifting priorities. The longer important maintenance tasks are ignored, the larger the impact and the harder it will be to complete the tasks
🤦… dude, you really have no idea where I live 😂🤣😂… otherwise you wouldn’t be saying this.
Things have been said more than once… I have asked, have pleeded for more personel… deaf ears. I have put it in writing, no use. Fine, then I just keep things running and that’s basically it 🤷.
Oh, and regarding workload, I already prioritize. The priority is to keep shit running, not to be up to date (obviously)… so, I just keep things running.
The phrase “Fuck you, pay me” comes to mind.
Cheapskates don’t get top of the line security hardening. Pay more now or suffer a breach and pay contractors $1000/hr to fix your broken shit because you paid minimum wage for an administrator position and wanted them to do 5 jobs at once.
This guy gets it… and probably doesn’t live in the US, cuz he knows the term “work 5 jobs at once”.
Surely you meant the opposite? Working multiple jobs is a very USian thing. Now I’m curious, where are you from?
No no no no, I meant what I said. Working 5 jobs AT ONCE (at one postion, one place)… AT THE SAME TIME. I do hardware, software, scripting, installing, configuration, maintenance (hardware and software… the whole shbang, DB included), Linux, Windows, BSD, servers, workstations (over 800 of them, and it’s me and 2 other guys under me!)… I even do freaking rig dusting! And ALL that, for 700 freaking euros a month! Not to mention pirating, that is also included cuz… they’re too cheap to pay for licenses for… well, anything really.
So, excuse me if I don’t care if the servers are up to date, mmmk.
From Macedonia BTW.
Jup same here. We have a colleague that constantly reminds everyone that we’re not properly patched (even running eol versions) but there’s always something to be done that’s a higher priority.
Exactly. Shit needs to just work, period. Why? Because otherwise, I’m the one getting 2AM calls… and I would be OK with that if I’m properly compensated for it… which I’m not.
Typically monthly or quarterly patching depending on severity and DMZ exposures. When log4j or shellshock hit it was patch once the patch was released and tested
Not at all.
Updates in Linux are far more tolerable. There’s really no reason to delay Debian stable, imo, unless you absolutely can’t risk some downtime.
Server rats excepted, it’s just a process that goes in the background and at most, you have to reboot the kernel.
There’s no staring at the Blue Screen of Boredom while windows update holds your machine hostage.
Do you work for the North Korean government or something OP? Why discourage people from keeping their systems secure?
What they are referring to is people just don’t update their server because during that time they wouldn’t be able to make a profit. This goes more to middle siszed businesses but happens rather often
I was making a joke
Joke transfer unsuccessful. Server crashed. Time to update the joke server.
Blows my mind, lol. Usually means no redundancy that allows one set to be done while the other set handles the traffic.
Yeah, it is quite common, I can confirm… well, at least around here it is.
Security is an art… the art of not giving a fuck about your data
-Op, probably
Me with my ‘homelab’ nas:
system (user-facing) package has an update? It’ll auto-update overnight
dockerized service has feature updates? Let watchtower handle it with the weekly schedule
dockerized service with security patch? yeah, let’s hit that this afternoon
actual system update? EVERYTHING IS GOING OFFLINE -4 SECONDS AGO FOR THIS
The system is going down NOW.
I find this to be least acurate with debian… on other distros a patch may or may not install a new version of that package. that can bring changes to the behavior. On debian stable the security issues are backported. So you can patch and be sure that there is no changes to the behavior of the system. It is basically the reason all vm’s i manage are debian stable.
It is also true they never crash. But that is expected of linux. It is the extreme reliabillity that is the debian killer feature for me.Unatended-upgrades keeps all systems securly patched. But there is a need for a reboot for kernel updates now and then.
I remember when Linux fan boys would give Windows users shit for needing to restart for updates.
Linux does not need to reboot, if you use KLP.
Debian updates are not usually that big of a deal especially if you have HA configured
True except for the one BOFH admin on the team who actually cares about best practices.
And yes, most distros have painless updates, the devs and everyone else don’t care.
Hi. It’s me. The guy bitching about best practices every other meeting. Sorry, but some of my past and present coworkers are clowns.
But Debian has security updates and you can set up unattended upgrades.
Meirl
