How Do You Protect Servers From a Rogue Admin? 219
Treborto writes "I work with a non-profit that has an extensive collection of photos and videos. These are used in publications and on the web. We have several levels of privileges: read-only of small, watermarked images; read-only of large, clean images; edit of the site; and admins who can confer privileges. It has happened that people leave the organization in anger. So far, no Admin has done so. Is there a back-up, site mirroring, privilege, or other strategy you'd recommend so we have protection from an Admin gone bad?"
backups and snapshotting (Score:2, Insightful)
FS snapshotting and backups are the only way, but make sure your backups are protected (locked up) etc.
Re:backups and snapshotting (Score:5, Insightful)
And one more thing to add - extensive logging of anything done with administrative privileges.
I worked at a place where everyone had sudo privileges; but any command done using it was logged to a couple different remote servers not administered by the same person. Worked out well; and anyone misusing it (say, running sudo bash) got noticed and talked to pretty quickly).
Re: (Score:2)
I never got the deal with why `sudo bash` is bad. It doesn't stop sone from doing something bad, and in this case, I'll just sudo vi and then :!bash from there. Nice an obscured.
Re: (Score:2)
Yeah "sudo emacs", etc and anything similar was frowned upon just as much as sudo bash.
The policy was that if they couldn't tell that you were doing something useful that needed sudo, they'd come interrogate you; so sed one-liners became a habit.
I'm sure you could subvert it somehow (sneak in your own perl script called sed or something silly); but for the most part just the idea that they trusted you with root; and also watched what people did with it; kept honest people honest.
Re: (Score:2)
wouldn't sudo script be an acceptable way of working around this?
It would seem a lot less painful.
What's the real problem? (Score:4, Insightful)
If people routinely leave your non-profit organization in anger, then the organization's leaders probably need to address a more fundamental problem than server administrative rights.
Re:What's the real problem? (Score:5, Interesting)
Those problems may be why the non-profit _exists_. People passionately involved in political or social issues are often _very_ political and social. Excited, eager volunteers can far too easily become disillusioned and angry: this certainly happens in the open source community all the time. After all, OpenBSD was created when Theo de Raadt had issues with the rest of the NetBSD development group. You can try to weed out all dangerous emotional issues from your agenda, you can try to filter out over-passionate members, but then you lose the very ability to create or to change the world that non-profits are created for.
With that in mind, the admins can also be passionate about issues and often are. Often underpaid and administered by people confused about technology, keeping things working with limited non-profit budgets is an artform, and I applaud and learn fascinating tricks from such personnel, and try to share knowledge with them to both of our advantages. In this case, the knowledge is about protocols for password management, protecting email backups, arranging reliable and recoverable and _thorough_ offsite backups and restoration procedures, and how to detect malicious behavior early.
Giving good advice requires some background of the operating systems and amount of data involved. Are there databases involved? Personal information such as credit cards and home addresses? Email from the board of directories? Is it on an Exchange mail server, or GMail services? The details matter a lot.
Re:What's the real problem? (Score:5, Insightful)
Author didn't say people routinely leave in anger, just that it happens. I've worked with a non profit charitable in the past, that had to make a decision whether to fund an alternative to planned parenthood, called choices. From what we saw, choices wasn't offering a lot of choice. They wanted to provide more of an alternative to abortions, and show women how adoptions could be a possible solution, and I really can't fault them for that, but they didn't want to provide information on preconception birth control, only abstinence, and in actual practice, they were tending to also push this message that not getting a ring from the male involved first made it all the woman's fault. Surely you can see how issues such as those can lead to angry resignations and workers who feel there's no compromise with management possible, and who might even break privacy laws as a result. Not all the risk is juvenile attitudes and L33Tspeak hacker volunteers who might get into petty arguments and storm out, much of it if is from people who sincerely think the issues are critical and worth bending a few rules over, and that the people who don't agree are all somehow stupid or hypocritical or venial, justified targets for anger.
Re: (Score:2)
Re: (Score:2)
Re: (Score:3, Informative)
And it's worth noting that people can get angry for reasons that don't have anything to do with the job. I don't care how wonderful the work environment is. Someone having trouble with life and a bit of mental illness can get angry anyway.
Re: (Score:3, Interesting)
While I dont fully agree with those claiming this is completely "off topic" it doesnt really answer the question at all.
The issue might be that the admins work in an organization with shitty leadership but that is not really something an admin can reasonably be expected to 'fix'.
What can be done though is to set up systems that mitigate the risk and damage of someone going batty. That is the question presented, not how to fix bad management!
Re: (Score:3)
Not to keep beating this poor deceased equine, but it doesn't just answer the question, it provides the only answer.
Someone needs to manage the backups. Someone needs to grant permissions, even if they have no other administrative role. Someone needs god-like powers to keep everything running smoothly. And if that someone decides to cause damage on their way out, they can and will.
Re: (Score:2)
Asking how to prevent that damage misses the point - You can't.
Yeah. The only thing I can suggest would be to have two totally separate systems (possibly at two sites) with two admins. That way you have another system and administrator to fall back on if things go bad. But that might be an expensive solution.
Re: (Score:3)
Non-profits can frequently employ volunteers with limited skill sets like interpersonal skills and empathy and are sometime very attractive to strongly Narcissistic personalities. Then add into the mix some aspies who are superman technically but naive socially, some parolees from the halfway house and a couple of work-study interns from the mental-health and you have a pretty volatile mixture of personalities that would tax the best of managers. It's probably not a question of people going batty, but keepi
Re: (Score:2)
Create a snapshot archive of your server (Score:5, Funny)
Create a encrpyted password protected snapshot archive of your server and name it something catchy like angie jolie secret sextape 1-29-2011 and upload it to piratebay. Safe secure lifetime backup retention online.
You have to trust someone (Score:5, Interesting)
Re: (Score:2)
And usually that's the admins. Most admins gone bad would be smart enough to bone the backups if they were going to do deliberate damage
The best bet is several admins. One manages the backups, another manages the "live" data, then you can have admin who oversees them (or more than one if you have the staff). If you maintain a few versions of backup data then you can minimize a rogue admin trashing your data.
Of course a determined person can still mess up your live data and all of the backups if they act over a long enough time. Hopefully the overseer can catch a long-term problem before it corrupts even your backups.
Oh and make sure everyth
Re: (Score:3)
Indeed.
We enforce the multi-admins at several levels here, and it means basically that no admin is god.
No admin has super powers, if you prefer.
So that means, there's:
1 admin (or more) who can administrate other admins and security rights. He need the express allowance from the user admin to unlock his powers, for 1 hour.
1 admin (or more) who can administrate users, but that's all. (he can disable other admins but cannot grand admin powers)
1 admin (or more) who can administrate backups, but that's all.
1 adm
Re: (Score:3)
Oh I forgot to mention that every admin has log read access, and append access, none has erase/overwrite/regular write access.
A separate group of people are securing the physical room and need 2 admins to inspect the system physically, +1 of the physical security dudes.
It sounds complicated but if you're organized it's actually pretty straight forward.
Re: (Score:3)
Who has the budget for that many technically savy people?
Don't Trust The Bosses (Score:4, Interesting)
At a small company I used to work for ("used to" being the key phrase here), the bosses, who both insisted on full admin rights, had a bit of a difference with each other. One of the bosses came in one Saturday night, killed the backup (they never took my advice of having multiple backups, including one off-site), and ran off with the server.
I tried recovering the backup, but he did a remarkable job in killing it.
The company didn't exist for more than a week after that.
Re: (Score:2)
"One of the bosses came in one Saturday night, killed the backup (they never took my advice of having multiple backups, including one off-site)"
Then they overlook your advice of having a backup *at all*. If there's not an off-site disconnected copy, then it's not a backup. You can call it "hot copy", "security copy", "near storage", whatever, but don't call it a backup for it isn't (now you obviously know why it is not).
Re: (Score:2)
When you're in school and need cash, you're not that picky about where you work. Especially if the location is conveniently located in the nicest part of town.
But anyway, these guys didn't listen to a thing I said about anything. They didn't have an actual IT guy, and I was only needed on-call. It was a total IT fiasco. Every computer in the place had local admin rights, due to their shitty software (all it did was link to a database on the server... but it wouldn't run without local admin rights), basicall
Re: (Score:2)
And admins have the keys. You are still fucked.
Re: (Score:3)
Read again.
Having the keys matters not. You still cant destroy the backup that is no longer in your possession. You -can- however release the information in the backup if you release the keys.
A fairly simple and common procedure is to have a sealed envelope with master encryption keys in a safe somewhere that the admins do not have access to.
Hell, in my previous job I didnt have access to the physical location where backup tapes were stored. I could ship stuff there, but not retrieve without a process of fi
Re: (Score:2)
Simple solution start corrupting the backups, before they make it to tape. Lots of ways to do that. If you do this for a whole tape rotation they will really be boned. This would mean going to quarterlies.
Re: (Score:2)
"Having the keys matters not. You still cant destroy the backup that is no longer in your possession."
The *real* question is: Unix-like systems are at C1 orange book level. Windows are a bit more C2-ish but a real nightmare to deal with them in such fashion.
Too technical? I'll tell it straight then. Both in unix-like and Windows environments there will be at least one admin with god-like privileges; deal with it.
The poster of the question doesn't even tell us what does "protect" means to him, so it's ver
Re: (Score:3)
No you don't. If your data is important, it's common to back-up off-site to a place, where the admins only have read/append access. A cheap way of doing that is to agree with another company to "swap backups"; they back up at your site and you at theirs. Naturally, the back-ups are encrypted.
"Criss-cross."
"Huh?"
"You bone our backups, I bone yours. No one would ever suspect until it's too late. Criss-cross."
"What, are you high? I like my job. I'll trash your backups for a crate of beer, but don't touch my company's backups."
"So we're agreed?"
Peer review of changes? (Score:2)
On a serious note if its cost effective to safeguard against a possible bad admin implement a peer review system of the workload each admin does.
rsnapshot (Score:3)
rsnapshot on a regular basis to a off-site service, that's read-only to the organization. I run that kind of service for several organizations for exactly that reason.
Re:rsnapshot (Score:5, Funny)
Ah, but what do they do when you decide to go rogue?
It's just rogue admin turtles all the way down...
Re: (Score:2)
But then... (Score:2)
you leave in anger and ruin the backup...
There should be more than one person worrying about this, keep the physical media in somebody's hands (preferably management)
Good backups and minimal access elevation (Score:2)
The best thing you can do is plan to mitigate any damage done. Of course this is easiest by not giving anyone any rights at all, but when you do have to give someone any kind of power try to wall them in as much as possible, so what damage they can do is very limited. Offsite backups that they dont have access to is best for recovery, especially if they have physical access to the site. I know some people will complain that treating everyone like a criminal will encourage destructive behaviour, but at the s
Tips for "rouge" admin defense (Score:5, Informative)
Rogue admins are extremely rare. So rare that there are many other more likely threats you will encounter, such as hackers or data breach. Worry about those first.
The reality is that most people work in a spirit of cooperation and don't want the black mark on their reputation. They would rather walk away without burning bridges.
That being said, bad admins (and employees in general) spring from two causes: bad treatment and pre-existing jerks.
The best way to handle both situations is to talk to your employees regularly, and find out how they feel. If you know that some policy or other is bothering them, you can avert a crisis very easily if you know about it beforehand.
Some people are just jerks. Don't let these people continue in your organization, even if they are brilliant and highly capable, and even if you don't have an equally brilliant replacement. A mediocre replacement who can work well with others will be much more productive.
(Often said: About 15% of your productivity comes from innate ability, 85% from working with others.)
That having been said, if you're really worried about someone doing you in, make sure you have regular backups and that you personally have access to the backup system. Reformatting a disk and copying data is easy - position yourself so that you can recover completely from the maximum damage they can do.
Re: (Score:3)
Yes, you generally only give your most trusted men the keys to the kingdom. But it doesn't mean it never, ever happens. Of course you can expect major chaos, backdoors, deleted data but it's nice if not everything goes up in flames. I'd say there's two things you need:
1) A backup system the admin doesn't have access to
2) A plan for a clean rebuild/restore of the core systems.
3) Don't tell him that's why you're doing it...
The backup can pretty much be explained by wanting to have an offsite backup with someo
Re: (Score:2)
In my eyes it's not about doubting the admins. There can always be rogue ones, even if few, you never know and you shouldn't spend time finding who's who (especially that you can be wrong).
The problem is that you don't know who is using the admin rights, how and what for. That's why you must split the admin rights into admin sets, to several very separate persons/accounts. (aka split the powers, or divide to reign, however you like to hear it)
If one admin is compromised by a hacker, or is rogue, or anything
Classic case of insider misuse (Score:2)
Coercion through shared guilt. (Score:2)
Same As Always (Score:5, Informative)
Re: (Score:2)
1. Create regular full backup of production system.
2. Verify that the backups are ok. Preferably by multiple people and/or external personnel.
3. Ship said backup on physical media to an off-site location where admin staff has no access.
Now... Tell me just how the lone rogue admin is going to fuck up this system?
Re: (Score:2)
That is easy to bone. I just make sure the backups all contain a nice backdoor for me. Or that they check the date when they boot and then nuke itself. It the backup is only data, then just mess it up a little. I can easily make 1 in every 1000 fields in an xls wrong, have fun finding them.
Re: (Score:2)
Re: (Score:2)
How do you protect servers from rogue admins, they same way you protect passengers jets from rogue pilots...
By having a copilot on the flight deck next to them? Or did you mean by making sure that their aware that if they crash the plane, they don't get fired, or sued, or even jailed, but rather that they die? (It's not 100% effective, but it's pretty good.)
the same way you protect patients from rogue doctors..
By surrounding them with highly-trained colleagues and subordinates -- other doctors and nurses -- who monitor their conduct, who have received thorough and ongoing training, and who will get in their way if they try to do something dangerous? Or did yo
Re: (Score:2)
"Food Salary"
Egg white omlet in the morning, please.... (I hear the food at Quilted Girraffe is good....)
Re: (Score:2)
Don't let clueless PHB's run IT (Score:3)
Don't let clueless PHB's run IT.
Don't make so there only 1 guy doing the network admin
Don't ask for admin password over a conference call
More so than a rogue admin (Score:3)
What is you backup method. Many more things can happen than a rogue admin messing up files. Disks fail, equipment gets stolen, users accidentally delete items - all of which point to having a robust, redundant backup strategy. Absent that, rogue admins are the least of your worries.
We've kept rolling backups - i.e several weeks worth, on duplicate media. On-site for fast access and off site for ensuring its availability if something happens on-site. I know others that mirror the entire operation to another secure location.
My suggestion - figure out how much data needs to be backed up, how often does it change, and then develop a redundant backup strategy with teh ability to roll back several generations.
You can't protect against any and all employee actions, but at least you can make it hard to totally destroy your data.
Also - as others pointed out - find out why people leave mad and fix the underlying cause.
Re: (Score:2)
Backups are a pretty good answer, but there are some problems to consider. First, deleting files is not the only thing an admin can do. They can screw with your data without deleting it. They can configure something so that it will fail spectacularly at an inopportune moment. They can screw with your backups and make them inaccessible. They can leave access for themselves back into your network so they can sabotage things later.
Re: (Score:2)
Re: (Score:3)
Ask the sys admins there to come up with a method; most folks working non-profit do it for the work not the pay, and many techs like the responsibility and challenge. By asking them to help solve the problem, you reduce the stress that would otherwise make them think they are the bad guy, and give them the merit that they do know what they are doing. Even if they cant come up with a reasonable solution, if you pick a third party, they wont be so miffed about it.
Re: (Score:2)
Every suggestion posted so far mentions making extra backups, using third party software for audit and tracking to adding extra, bureaucratic steps into the mix that will do just that: piss someone off.
If making redundant off-site backups pisses you off, you really shouldn't work in IT!
Secure, read-only, off-site backups are the best option for this, they have multiple purposes, and anyone who is pissed off by their existence is either a control freak who I would be scared to work with, or is actually planning to do harm which is even worse.
I worked at a place where the off-site backup policy was that I handed the weekly tape backups to the owner of the company and he took them home. I was never offended
Re: (Score:2)
No, you could have corrupted the date on the tapes as it was written. He had no way to test that. The best corruption is not total fail, but random small stuff that would take forever to fix. You can do a lot of damage just by making random changes to accounting documents.
Re: (Score:2)
I could have, but it would involve a long term plan to hurt the company, that form of damage is very difficult to protect against, but also very easy to detect (anyone can verify the backup at any time by restoring it to something). The fact that I was only one of a team of admins also made it more difficult, as I wasn't always the one handling the backups.
The most common form of damage from a "rogue admin" is going to be created in a short period of time between the time they get really pissed off, and sec
Re: (Score:2)
In short, an admin may turn hostiled
You cant ! (Score:2)
Re: (Score:2)
You need more than one (Score:2)
Auditing and consequences (Score:4, Informative)
No matter what solutions you use for backups, the admin will be able to corrupt or bypass them in some way given enough thought and motivation.
However, for sane though disgruntled people it would be sufficient for them to have the common sense understanding that malicious actions will have strict consequences - people generally don't risk going to jail just to annoy a manager or company. And in the cases where someone would really be prepared to risk that, I'd rather worry about them coming to office with a gun, not tampering with a pile of pictures.
What was the aftermath of the previous cases you say of people leaving in anger and presumably doing something damaging? Your previous reaction in these cases forms the expectations in your admins about what they can get away with when leaving in anger.
Levels of Privileges (Score:2)
I see there's escalating levels of access, but it doesn't sound like those levels are tied to law. They probably should be, i.e. it's not so much file size as whether the file is about an adult person or a minor, whether the file contains medical information or not, and such things that should be the first consideration in defining those privileges. A single dental photo sounds like a small image under your definition, but its treatment depends on HIPAA first and foremost, never size or image format.
Logical problem (Score:2)
Before you get to any details, there's a sort of logical problem in protecting against admins: Who are you going to get to set up the protections? If you hire me as an admin and then ask me to secure the network against myself, there's nothing to prevent me from putting in some kind of alternate access (i.e. a secret backdoor). If you hire someone else to secure the network against me, then there's nothing to prevent that person from keeping some alternate means of access. That's before you even get to t
Off-site backups (Score:2)
Send regular backups tapes off-site to someplace like Ironmountain. Only give authority to retrieve tapes to collection managers and/or company executives, not to server admins. This also protects your collection in case your office or coloc goes up in flames.
Keep at least 6 months of tapes off-site so you have 6 months to discover a time-bomb or hidden corruption left behind by the rogue sysadmin.
Test restores regularly.
Outsource backups and perform audits (Score:2)
If you truly are concerned about the trustworthiness of your systems administrator; you definitely don't have the right person in place and you need to take steps NOW to ensure the continuity of your systems. Start implementing strict documentation standards for everything - passwords, system maintenance procedures, run books, network diagrams, etc... This information then needs to be stored in location accessible by senior executives and audited by an external firm to ensure completeness and validity. You
Think more of the legal ways (Score:3)
Don't worry about your infrastructure so much. Having been in this position, I noticed that companies seem to worry quite a lot of it.
But it seems to me that it's an unlikely situation. Let's suppose there's an admin really pissed off at you for some reason. What could they do to your photo collection?
All those options are pointless and ultimately suicidal for the admin involved. All you need to do is to have readonly off-site backups (which you should have anyway, what if the building gets flooded or burns down?). If properly done the rogue admin can't screw that up, and while the things above might hurt, they'll be perfectly survivable. Even the torrent isn't a big deal. A serious publication isn't going to touch an illegal collection with a 10 foot pole. As a public organization they're an easy and profitable target.
However, those things are terribly stupid and suicidal for the rogue admin. Who will be the first suspect in line when any of the above happens? The recently fired angry admin. Law enforcement treats such things harshly, and word of mouth gets around and it's unlikely they'll get another job after that.
All the admins I've seen leave (and I took note and did it myself when leaving a job) tried to leave in an as non-threatening way as possible. For instance on my last day on one job I discussed with a coworker what I had been doing, where the files were, what was unfinished, the lists of passwords and access control methods to be changed, etc. I did everything I could to make sure that nothing in my departure could be interpreted in a "screw you" of any kind, and to make sure my successor could take over.
Now, what should you be worried about? The legal ways an ex-employee can screw you over. For instance, the BSA. It's easy to report to them. From what I hear they're most eager to show up, offer rewards to the reporter, and it's very hard to deny them entry. And I hear that their visits can be very expensive. So make extra sure you're in perfect licensing compliance (which is pretty hard), or switch to Free Software.
Re: (Score:2)
If they are the ones making the backups there is no protection in shipping them offsite.
Re: (Score:2)
But then you're talking about somebody planning over a long term, and not just breaking things in a rage after being fired.
First, you don't have good backups if you're not testing them once in a while. A test should make it evident whether the right stuff is being backed up, and whether it's being backed up properly. With that in place it's hard to hiddenly screw something up.
Second, it's still monumentally stupid. It's a dangerous trick to pull off. Things can easily go wrong (what if the admin gets asked
two man rule (Score:2)
Version Control Everything (Score:2)
While it is not yet standard practice, there is absolutely no reason why your server cannot be completely under version control. The only point of contention is the password/groups file. Aside from that, you should be able to use something like TinyCoreLinux to get a minimalist boot image, with a version control system, (SVN, CVS, etc) configure the version control and save that image. Then once you boot the image, you issue a get/sync/update command which gets the most recent version of everything.
TCL Linu
Re: (Score:2)
While the above is slanted towards Linux, there is no reason why this can't work on windows too.
TinyCoreLinux - http://www.tinycorelinux.com/ [tinycorelinux.com]
It is more up-to-date than DSL, and has an easy-to-use package manager.
One thing I loved about NDS (Score:2)
How does that help? Well, if LDAP or some other free-as-in-b
This comes up almost daily on PHB websites... (Score:5, Insightful)
The ways you mitigate the issue of "rogue" admins, is vet them, listen to what they are saying in terms of technology, don't micro-manage them, and pay them well. The good ones without a doubt will know the technology better than their manager/management structure will ever know it. The reason the admin says something about the setup/configuration/technology is almost always because it is needed change. If you can't afford to make those changes, then you need to explain that is the reason, don't make up some BS about how you want things to stay the way the are, or you want to change the organization/structure to something else, because they will "call" you on it. Again, they know the technology better than you ever will.
The other thing to do is to pay them appropriately. You are trusting them with running some of the most complex systems in your entire company, as well as safe-guarding your data, your processes, and your daily operations. The reason why you don't see many rogue CEO's is because he/she is being paid well to run the company, choose its path, and steer the ship, so to say. The system admins in today's information based businesses are the guys keeping your entire company running. If your servers/data were all destroyed, and your business would not survive, then you might want to consider paying the people who keep that data/servers a more appropriate amount of compensation since they are so vital to your business.
Again, there are very few admins who go rogue, and even fewer who did not do so after being mistreated by their bosses/management. If people want to point out at the case of Terry Childs, they need to get a clue. Were mistakes made, sure. Did Terry have some issues? Yes. Did he actually go rogue? No. In his eyes, he was protecting the network from idiots and incompetents, and following the rules as currently defined. He wouldn't give out the passwords in a room of strangers, over the phone, or via email where it can easily be intercepted and then misused, as well as be cause for firing him because policy stated not to do any of those things. So he was placed into a situation where he would be fired if he handed out the passwords, or fired if he didn't. And once fired, he really had no obligation at all to give it out anymore, why? Because he didn't work there. Same as if you fired your top salesman, or stock broker, or process manager. They don't have any obligation to tell you anything about the contacts/client relationships/methods for picking stocks/how things work. If you fired them before you obtained that information, then you should have been fired. In the Childs case, were they trying to obtain that information, sure. But in the wrong way according to policy. They should have taken Terry into a one on one conversation, in a private room, with no one the phone and asked in that setting. Even then, he might have refused to have the manager have the password because the manager didn't have the knowledge or skill to know how to properly vet someone as being capable of having the password. The only thing that would happen is that it will cause someone to screw up the settings and create work for Terry since he will be the one called in to fix it, and most likely not paid for that extra time he had to spend fixing someone else's screw up.
Again, it comes down to properly compensating the admins, listening to them, and not trying to play office politics with them. You treat them well, and they will do whatever it takes to keep the systems running because they take pride in their work. You treat them like crap, blindly disregard their expertise in terms of operating the servers/network because "you know better than they do", you are asking for th
Re: (Score:2)
First, you need to stop drinking the coolaid. You are paying the sys-admin to keep your systems up and running. They do have "the keys to the kingdom", because you are paying that person to hold them. If you don't trust that person to hold the keys, then you shouldn't have hired them in the first place.
This discussion has got me wondering just what kind of person is always expecting everyone to stab him in the back. I understand caution and reasonable preparation for calamities human and otherwise, but the PHB crowd really fixates on the paranoia.
angry workers (Score:2)
If you have workers who are leaving in anger then you have a organizational issue that security is not going to fix.
Re: (Score:2)
When a company is large enough there will always be angry people, be it for a good or no reason at all - it's human nature, as bad we know it is.
While this should be prevented as much as possible, the company going bankrupt (and affecting a hundred souls) should be prevented as well.
Just keep sysadmins happy (Score:2)
Is there a back-up, site mirroring, privilege, or other strategy you'd recommend so we have protection from an Admin gone bad?
Sounds like you already have a technical solution for cleanup. If it were me, I'd have two locked server rooms, and each sysadmin is only allowed into one server room. Each room has half of the original servers, and half of mirrored servers from the other room. The mirrored servers rsync from the original servers regularly, with a resticted user account with sudo access only to rsync (plus the options in /etc/sudoers to restrict rsync to only backup particular directories, otherwise it could overwrite /e
Re: (Score:2)
Defence in-depth, distributed authority (Score:2)
While a lot of people have complicated methods... (Score:2)
...there's actually a pretty easy method.
Simply set up a file server somewhere that the admin do not have physical access to. Setup a server in a locked office. Put it in the president's office, it makes him fell important. (Of course, don't get him any login to it or even attach a screen.) It's so simple and does so little, you don't have to worry about overheating or anything.
No remote login or anything. All it does is have one file sharing point (SFTP or something), that gets logged into and files uplo
Re: (Score:2)
...there's actually a pretty easy method.
I really thought you'd say "put a gun on their familly's head and say if data is gone, they're gone too".
But then you started writing about something not as easy!
Re: (Score:2)
I also thought that Duplicity should be mentioned. It uses librsync, its dead easy to setup for backups and supports everything you can think of (encryption, deltas, recovery per time period, various upload means going from regular copy to sftp, scp, and the list goes on for a while)
Excellent tool http://duplicity.nongnu.org/ [nongnu.org]
Re: (Score:2)
Yeah, that's something that could probably handle the entire backup, and just, at the end of it, copy it somewhere else.
Hell, if the backups are encrypted, that's one less thing to worry about on this server.
Then you just need a script like in cron:
mv /backups/shared/ /backups/`date +%m-%d` /backups/shared/ /backups/ -maxdepth 1 -type d -ctime +30 -exec rm -r {} \; /backups/ |mail admin@example.com
mkdir
find
ls
Or, you know, something a bit more complicated, but essentially that. Just moving the backups
Re: (Score:2)
Cool, then all they have to do is messup the backups before they get to that server.
Re: (Score:2)
Yes, if you have admin behaving maliciously for a month and no one notices, you'll be in trouble.
Of course, if they can do that, you're pretty much fucked anyway.
Go down before the overman... (Score:2)
Prison (Score:2)
The protection you're looking for is called "prison." Wiping the files on the way out would be like burning down the building or returning with a gun. It does, rarely, happen and those folks go to jail. The only things you can do about it ahead of time are: treat your staff well and, when it is necessary to fire someone, make very sure that both their privileges have been thoroughly revoked and you have a current, tested backup.
Backups and commercial offsite storage (Score:2)
Ha Ha (Score:2)
Well, to make the same point in more sober fashion, here we see an important boundary condition of the security problem. Somebody has to build and maintain these systems. Ultimately, the privilege necessary to do that is the same privilege necessary to defeat any security measures they might embody. That's the state of ultimate guardianship. And then we must ask, as Juvenal did a couple of millennia ago, who is to guard the guardians?
The answer, though imp
Looking at this from a different standpoint, (Score:2)
I have always marveled that there are so *few* rogue admins. In most companies we have access to *everything* -- Employee records, financials, AP/AR, and whatever anyone is doing on any PC or server. Yet reports of rogues intent on stealing or causing damage are rare. It usually goes the other direction -- the rogue takes action out of a concern that management isn't taking security seriously enough. I bet you could think of a few examples.
The question I would ask is: What is it about the job, or ab
"Need to Know" Privs (Score:2)
The method to use depends on RPO/RTO (Score:2)
If an admin deletes all the files.... how quickly do you need every single photo back?
The longer your business can live without them, the less expensive a solution you will require, and the more reliable a solution you can pick.
Some of the least expensive solutions are.... burn every new file to DVD and backup every new file to tape or traditional film. Translation: backup as you go.
If you want your collection to survive thermonuclear war and EMPs, then record every frame to film using a Film recorder [wikipedia.org];
Depends on the lengths you want to go to (Score:3)
I worked for a bank auditing company for a while, and installing anything (or any administrative work) was a pure PITA. There was a mandatory "four eyes" principle in effect. Logging in without a second person (every admin login caused a text message to go to all admins, just in case you're wondering whether nobody did it "stealthily") was grounds for instant firing. You would grab a fellow admin (or, if nobody was around, anyone who could "supervise"), fill out a form that you and him are going to log in, then you started a protocol (pencil and paper type) of what you are going to do. Every keystroke, every click of the mouse, was to be written down, then executed. Installing a program or an update by protocol could well take an hour or two, and certainly not 'cause the machines were slow. Termination was told to you the moment you were let go, the same moment two admins were sent with high priority order to revoke your admin privs. On the upside, you were let go instantly, i.e. take your stuff, do not log in, you may spend the rest of your working days at home (i.e. effectively another 1 month of paid vacation). If you had to clean up anything on your machine, two admins did it for you.
This is a level of security and paranoia that borders on insane. Personally, I'd say it's a wee bit beyond insane already. But it gives you an idea that banks tend to take security and the threat of rogue admins VERY serious.
But there is one thing you should definitely do when firing an admin: Revoke his admin privs INSTANTLY the moment he learns that he is gone and send him home. Even if laws demand that you have to tell him 2 weeks before firing him, send him home on 2 weeks of paid vacation. It's cheaper than the threat of having him do something to retaliate at you.
Costs, Risk Mitigation (Score:2)
To be serious about security, you have to eliminate every last single point of failure. Although I seriously doubt a non-profit would have the cash to justify paying rather than simply trusting, if they were serious about limiting the damage an admin could do, they would outsource the backup, requiring that the backup be regularly monitored for suspicious changes and tested both by the outsource and by someone within the company.
Re:Change Root Passwords to Your Box (Score:4, Informative)
Re: (Score:2)
You keep using that word. I do not think it means what you think it means.
Re: (Score:2)
Think about Back Doors... (Score:2)
Re: (Score:2)
Have fun proving it was on purpose. Impossible task if the corruption is done correctly. Heck, the you in this case probably lacks the no how on how to even test said tapes.
Re: (Score:2)