What Would You Want In a Large-Scale Monitoring System? 342
Krneki writes "I've been developing monitoring solutions for the last five years. I have used Cacti, Nagios, WhatsUP, PRTG, OpManager, MOM, Perl-scripts solutions, ... Today I have changed employer and I have been asked to develop a new monitoring solution from scratch (5,000 devices). My objective is to deliver a solution that will cover both the network devices, servers and applications. The final product must be very easy to understand as it will be used also by help support to diagnose problems during the night. I need a powerful tool that will cover all I need and yet deliver a nice 2D map of the company IT infrastructure. I like Cacti, but usually I use it only for performance monitoring, since pooling can't be set to 5 or 10 sec interval for huge networks. I'm thinking about Nagios (but the 2D map is hard to understand), or maybe OpManager. What monitoring solution do you use and why?"
I Name My Devices After Al Qaeda Members (Score:5, Funny)
Publish them in DNS, and have the NSA monitor them for me!
Re: (Score:3, Funny)
I was going to suggest he should ask the UK government, but I like your idea better.
Re:I Name My Devices After Al Qaeda Members (Score:5, Funny)
The only drawback to this comes in the form of UAVs.
Re:I Name My Devices After Al Qaeda Members (Score:5, Funny)
No need to. We are doing it anyway.
Your NSA.
Re: (Score:2)
What Would You Want In a Large-Scale Monitoring System?
uncle sam is that you?
Re: (Score:2)
Good idea, except they stopped monitoring when they found out your sites are all still under construction [osamabinladen.com].
Hyperic HQ (Score:2, Informative)
Hyperic HQ may be worth checking out.
rule based DSS (Score:2)
Don't assume that you can successfully diagnose the problem based on your understanding of the indicators. You don't know my institutional context. Instead, give me a decision support system that I can use by adding rules that key off the monitored indicators and inject some of our own expertise into the diagnostic process.
OpenNMS (Score:5, Informative)
That's all you should need. For 5000 devices I don't know that any of the options you listed would be appropriate.
OpenNMS is much more than monitoring, but I think that you'll appreciate the other features as well.
http://www.opennms.org/
Enjoy.
Re: (Score:2)
Mod parent up, OpenNMS rules.
Re:OpenNMS (Score:5, Interesting)
I use OpenNMS as well. I actually migrated off of Nagios to OpenNMS. Tried out Zenoss and Cacti as well. While any of these are better than OpenView IMHO, I liked OpenNMS's full suite of functionality without having to pay for the 'commercial' version.
Re:OpenNMS (Score:5, Insightful)
I've only tried OpenNMS. It looks very powerful, but wasn't at all hard to get installed and configured on Ubuntu - it figures out the type of node it has discovered and shows useful data through SNMP, and can also do uptime monitoring, and is generally very scalable and configurable if needed.
Re: (Score:2)
I'll second this.
Re: (Score:3, Informative)
Re: (Score:3, Informative)
That would definitely be a bug. I'll look into it... :)
A more interesting question (Score:5, Insightful)
Re:A more interesting question (Score:4, Insightful)
What limitations exist in current solutions that justifying developing a new one from scratch ?
Exactly! Too often people just jump in and redo everything without actually investigating what needs to be fixed. Quote from George Santayana "Those who cannot learn from history are doomed to repeat it,' seems very appros here.
Re:A more interesting question (Score:4, Informative)
He said he was asked to "develop a new solution" - which most likely means he gets to pick and choose what to implement, whether parts of it are custom developed or off the shelf. I would imagine a good solution would be a core product plus custom built extensions for the features he needs that the product doesn't implement itself.
Re: (Score:3, Informative)
Re: (Score:2)
Exactly, I need a good core product that I'll evolve over time.
Why do you want to "evolve" it (by which I'm assuming you mean modify in depth rather than "configure") it at all ? What's missing that you need ?
System monitoring isn't exactly a fresh and new field. There are numerous well-established and quite comprehensive products already out there.
Re: (Score:2)
Not all the vendor respect open standards, so you have to guess how to get the info from an UPS, Printer,
Re:A more interesting question (Score:4, Interesting)
The big questions are:
Will your solution need to support snmp v3?
Do the devices you talk to have published oids?
Do you need source code to extend it?
If yes to these, OpenNms is a great bet.
Re: (Score:2)
No, monitoring is inside the intranet, so there is no need for SNMP v3, besides they are configured for read only and access from 1 IP only.
Do the devices you talk to have published oids?
Some do, some don't.
I'll check OpenNms
Re: (Score:2)
He said he was asked to "develop a new solution" [...]
From TFSummary:
Today I have changed employer and I have been asked to develop a new monitoring solution from scratch [...]
Where I come from, "from scratch" doesn't mean "configure existing solutions to my needs".
Re: (Score:3, Funny)
Yea, from scratch, first I'd develop the tools needed to mine the raw materials of silicone, iron, and other needed elements. Then I'd refine them and produce the needed components for memory and processors and storage. as well as develop the new networking, power, form factor etc... Then start working on the boot code and a core kernel, hmm should it be micro/macro or hybrid...? Then I'd start working on interface tools or user space or something along those lines. Once I got this part done I'd start
Re: (Score:3, Insightful)
Obviously by "from scratch" he means his company has nothing in place he has to build on; he is free to build a system on whatever tools he likes.
Re: (Score:2)
s/develop/deploy/ (Score:2)
He's just asking what to use
Before I get flamed... (Score:4, Interesting)
I am going through this right now and am using and have used all the above mentioned solution. We are leaning towards System Center Operation Manager. http://www.microsoft.com/systemcenter/operationsmanager/en/us/default.aspx If you had told me 6 months about that it would be the way to go, I would have said over my dead body, but it has come a very long way in terms of usability and ease of setup.
Re: (Score:2)
I forsee myself in your shoes in the next few months. Application-level awareness of our key Microsoft applications (Exchange, MOSS, AD, etc..) are very high on our need list, and so SCOM is a natural best-of-breed pick. However, I *really* want a single integrated solution that also covers our unix/linux systems. Is unix/linux monitoring part of your requirements? If so, could you briefly describe the capabilities (and requirements on the monitored systems) that SCOM currently has in this regard?
Re: (Score:2, Informative)
SCOM R2 integrates native unix and linux agent [microsoft.com], supported systems are :
HP-UX 11i v2 and v3 (PA-RISC and IA64)
Sun Solaris 8 and 9 (SPARC) and Solaris 10 (SPARC and x86)
Red Hat Enterprise Linux 4 (x86/x64) and 5 (x86/x64) Server
Novell SUSE Linux Enterprise Server 9 (x86) and 10 SP1 (x86/x64)
IBM AIX v5.3 and v6.1
For application awareness, you can check bridgeways management packs [bridgeways.ca].
Re:Before I get flamed... (Score:4, Informative)
As an aside, SCOM is a good product, but be sure you have (and are willing to invest) the time to configure it to match your environment. Just because it's also made by MS and has management packs for all of their products doesn't mean you can just flip the on switch and have everything monitored. You will almost certainly be flooded with useless alerts, and not alerted for things that you do care about.
Re:Before I get flamed... (Score:4, Informative)
Unless they start to optimize the mess I'm not sure I want to use it.
Zabbix (Score:5, Informative)
Re:Zabbix (Score:5, Informative)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
No reliability issues? (Score:2)
Which revision?
i tried it for a couple of months, and rather like it, but it'd simply stop monitoring stuff, triggers wouldn't fire reliably etc.
Re: (Score:2)
Haven't had any monitoring problems, we interface to a master Remedy system for stuff due to ITIL, and we have triggers that run scripts that try to fix mundane problems (which usually works).
Re:Zabbix (Score:5, Informative)
We're using Zabbix at work and I'm doing daily backups of the database with a simple mysqldump command. Since the tables are InnoDB and not MyISAM, you can use the --single-transaction switch. That way, it takes a virtual snapshot of the db at the start of the backup process and the writes can still keep going (they are still happening but they aren't commited until the transaction finishes). Granted, our DB isn't that big (10GB only), but it's been working fine and restore tests also seem to work fine.
Here's the daily cron:
mysqldump -u blah -pblah --single-transaction --opt --skip-lock-tables zabbix | gzip > /backup/zabbix_db.sql.gz
Re: (Score:2)
Sounds like it is either using MyISAM for tables, or you aren't using the --single-transaction option of mysql-dump for INNODB.
Zabbix is easy to maintain and flexible (Score:4, Interesting)
Zabbix allows you to build some fairly powerful rulesets and chains of overrides using its web gui. It's not perfect, but it keeps improving and the attitude of the developers is friendly unlike some of the other projects.
GKrellM (Score:5, Funny)
You can pry my GKrellM from my cold, dead hands!
Yeah, for 5000 devices, the displays start to take up quite a bit of screen space, but that's what video walls are for!
*cough*
Re: (Score:2)
Re: (Score:2)
To be fair, one of the two mods knew it was +11 Funny... er I mean +1 Funny...
One of today's articles has a thread deploring people who read the article, the summary, the title, or the posts before posting. So in my defense, I only read 4 words of the title and posted, so I saw Would Want Monitoring System. Naturally I thought of GKrellM.
Now if I had read 4 different words, I'd have thought it was spam and deleted it. "What You Want Large"
Re: (Score:2)
Re: (Score:2)
So what you're saying is, GKrellM needs plugins for Zenoss and Nagios and whatever else?
Damnit, now my +1 Funny is starting to sound practical. Quick, somebody add another +1 Funny! This has to be stopped!
The Dangers of averaging (Score:5, Insightful)
MRTG does it right...most of the others do it wrong
When rolling up a days worth of data (averaging), you loose the peak information on most monitoring systems
So your 380Mbps peak that you had an hour ago is fine on today's graph
But tomorrow, when you look at "yesterdays" graph...the peak is down to 100Mbps
and next week, when you look at "last weeks" graph...there's a little 50Mbps peak
Damnit... I want to keep information on my peaks for capacity planning!
Zenoss (Score:5, Informative)
I was really impressed by Zenoss [zenoss.com], which has all the slick features that cost the earth from vendors like HP for Openview. You get automatic discovery, CMDB inventory, availability monitoring, alerting, and performance graphs all in a web portal.
You get open source, commercial support, and a good community of users and plug-in developers. The best of both worlds IMHO.
Re: (Score:2, Interesting)
A little tough to setup new SNMP devices, I thought, but overall a great product. Even the free version gets you quite far.
Re:Zenoss (Score:5, Informative)
ZenOSS may be great, but a word of warning. We've had 3 failed attempts at implementing it in our shop. What we tried to achieve was mainly host and service-monitoring, with some slight network-monitoring on the side. Nothing fancy, just some 20 hosts, maybe 30 network-devices, and a variety of services.
One of the major parts we've found missing in most open-source solution was proper event-management (recieving syslog + snmp traps, and apply some intelligence to it regarding flow control, dispatching, archival and that stuff.) ZenOSS is on paper, and throughout the initial evaluation one of the best open source tools to do this.
However, during our three attempts to get it up and running, we've always encountered some major obstacle (usually after a while of operation), forcing us to start all over from scratch. The problems we had was always in the same category, strange and unexplainable errors, often hard to reproduce, and in general it resulted in a very flaky experience. Some of the problems have been service-checks showing both false positives and false negatives, and in the last problem ZenOSS refused to import new SNMP MIB:s, complaining about some IP-address that could not be found anywhere in the config, and grepping ultimately found the IP to be only present somewhere in the opaque zope-database, where evidently it could not easily be removed, nor even found exactly what the ip-address was for. (It was something auto-discovered in a remote network segment out of our control, but advertised throughout the routers.)
So, while ZenOSS can do all kinds of things, and does a LOT of things really well, it's extremely complex, not in all parts on solid foundation (such as all network objects in a non-accessible Zope-database that the devs themselves recommends not touching since it may upset things more). If you plan on implementing ZenOSS, I would not go without the support, which I assume is great, since there seems to be quite some dark pits to fall in on your own.
I dont know how come we had so much obstacles and strange problems when others seem to have a smooth ride. Maybe one explanation is what were the final nail in the coffin for ZenOSS in our deployment. When I started asking around about these problems (and ZenOSS has a really helpful community, no problems there), I realised that many users claimed to have gotten into similar problems that we had, but their solution were to just keep daily backups, and revert to a backup when they ran into these problems. For us, the monitoring data is basis for a lot of 3d-party agreement, and loosing even days worth of monitoring and logging is completely unacceptable due to these reasons. We do backup everything, but in case of rare disasters, and we must be able to rely on the monitoring system giving us a clear view through those disasters.
Re: (Score:2)
Yes. Seriously.
The other problem I have with Zenoss is the reporting is basically non-existant. It may sound like I'm being hyper-critical, but it's only because I've looked at Zenoss and I so wanted it to be the NMS for me (
Re: (Score:2)
From the same page you linked:
>** Volume pricing:Volume discounts, site licenses, and corporate-wide licensing are available for environments larger than 1,000 managed resources. Contact Zenoss Sales to learn more.
So I call BS on the $500,000-$900,000/year number you gave.
Re: (Score:2)
You missed the volume and site licensing options available for networks of 1000 nodes+
Their pricing seems in line with their competition, a quick search finds the follow pricing for HP offerings:
Network Node Manager, $6,000; OpenView Operation, $17,995; OpenView Internet Services, starts at $12,449 for 5 targets. Additional targets: 5, $2,038; 25, $10,207; 250, $65,160.
I am not familiar with how their product line works, and I'm sure they also have volume licensing agreements for large customers, bu
Re: (Score:2)
Next, Zenoss's offering is not comparable to OpenView, sorry. HP OpenView is a massively more mature and robust product, and is really in a totally different league than Zenoss.
Re:Zenoss (Score:5, Interesting)
And this is why we (OpenNMS) don't play the per-node. It's not any harder to run OpenNMS when managing 1000 nodes than when managing 100, you only need to scale hardware appropriately. Per-node pricing is an artificial limitation.
We also don't play the "you get a special price behind closed doors" game, our support prices are public, fair, and the same for everyone [opennms.com] -- and that's only if you need commerical support -- our prices are $0 if you don't need or want support.
If you do the math, it's $0 for the software, plus $14,995/year for support for any number of nodes, and the software is 100% open-source and fully capable of replacing or exceeding OpenView [opennms.org]. ;)
Spiceworks? (Score:2)
http://www.spiceworks.com/ [spiceworks.com]
Not sure how far it scales but I have played with it on some small installations, very easy to manage.
I have used Cacti but never felt it was mature or robust enough for very large environments
SCOM, System Center Operations Manager we are deploying now for our enterprise, however I would be afraid to manage IT on my own as it is a large system on to it self, yet very powerfull.
A couple of other options (Score:3, Informative)
I use Nagios and some custom rolled scripts myself.
For some other options, Nagios has now been forked, so if that is "close" to what you want, you may want to contribute to Icinga [icinga.org].
Reconnoiter [omniti.com] also looked pretty kewl, but they haven't released anything yet, but it looks like they are planning it to be very scalable.
No humans being monitored! (Score:2)
That is what I would want! ^^
Re:No humans being monitored! (Score:4, Funny)
I find a human monitoring system to be the most reliable. There is always someone to fire, if something goes wrong.
OpsView (Score:2)
Nagios Might Work (Score:3, Interesting)
I spent last year converting a shop from OpenView to Nagios. They were in the same neighborhood as you (~5000 devices).
If you do not like the Nagios UI, you could create something else. The native Nagios UI is CGI based and implemented in C. The documentation is good and the sources are well commented.
The hardest decision about Nagios is how to implement the monitoring. I went w/SNMP (polling, not traps) for the most part. Sorting out all the Nagios plugins is something of a chore and many of them seem incomplete and abandoned.
MRTG also integrates w/Nagios, which can be useful.
Good luck.
ZenOSS all the way (Score:5, Interesting)
Cons:
For me, that's easy (Score:2)
I want lots of buttons and dials! And flashing lights!
The mistake (Score:4, Insightful)
The mistake is trying to monitor thousands of devices on a 2-D map. I'll look pretty to the suits, but be useless for the users. Nothing but endless slow clicky clicky clicky.
Give them a text screen of whats currently down ... that'll work.
Re: (Score:2)
I want the help desk to know what is the problem before our clients calls us.
Re: (Score:2)
Similar Slashdot thread (Score:3, Informative)
Here's a similar thread from a while back that covers most of the options: http://linux.slashdot.org/article.pl?sid=07/03/05/1812247 [slashdot.org]
Re: (Score:2)
Roll you own... (Score:2, Interesting)
I built and managed a software system for me previous employer called the SMS (Server Management System). It basically tracked 50 of our web servers, database servers, and Endeca (full text search) farms at data centers spread around the country. It was pretty simple.
The syst
Pandora FMS (Score:2)
As one of the core devs and large user, I can tell you it scales well, develops easy and has a lot prefab. The system does everything you're asking for. Let me know if you need help or paid support.
not sure if this is helpful, but... (Score:2)
I'm a software developer and, sadly, my knowledge of hardware systems isn't always what it should be. When I write an application to run on a server and it starts to get slow, I want to know where the bottleneck is. Is my application CPU-bound? I/O-bound? Memory-bound? Do I need more memory? Faster storage? More cores or faster processor speed? Is it the network that's causing the problem? I can usually figure this out using various linux command-line programs like netstat and top and all that, but
Re: (Score:3, Informative)
JMX Support (Score:2)
What I'd like to see is a good monitoring support for JMX-capable Java services.
It'd be nice to set up an alarm based on time spent in garbage collector in a JVM running our application, for example.
Re: (Score:2, Informative)
You certainly want to check ZenOSS out then. Our instance monitors JVMs across our deployment for everything from heap size to open file descriptors to uptime for the jvm specifically-- all of which can be alerted on if desired.
Nagios, Munin, GKrellm (Score:2)
I think Nagios should provide a good start.. they've recently added a lot of scalability features. Though it has a high learning curve and all of its configuration is done in text, I've always found it worth the time and effort. I currently use it to monitor services on a couple hundred machines.
Munin is a bit simpler, but I like the graphs it provides which occasionally are more useful than the data Nagios provides. In some cases, Nagios might tell me that a server went down, but I'd look at Munin and
SNMPc (Score:2)
It's *not* open-source, but it IS inexpensive. When I worked at a NOC, we used it to monitor hundreds of routers, switches, mainframes, Tandem systems, UNIX boxes, etc. It takes SNMP traps and displays them graphically on a 2D map, and the 2D map is very nicely implemented. You can have your top level view made up only of groups of devices, so if a group goes red you double-click that group to view its members and see which device actually has the error. IIRC, you can nest groups, so it ends up being a
I Hate War Rooms (Score:5, Interesting)
I really don't like the "War Room" video wall concept. I suspect such walls are made to look cool rather than to monitor.
What you want in large-scale monitoring is:
Etcetera. These are some of the things that make sane large monitoring systems. I don't think any open source product has all of them, alas.
Re: (Score:2)
Re: (Score:3, Informative)
When you have your parent/child relationships and your dependencies set up properly, Nagios does this very well. A properly configured Nagios system will alert you only for that switch that died, not for the 200 services behind tha
Re: (Score:3, Informative)
"What you want in large-scale monitoring is:"
Let's see:
"The ability to map complex relationships. I don't want 50 alerts that I can't reach host X, host Y, etc. I want one alert that I can't reach router A."
Nagios do this. I know, I configured mine that way.
"Even better, I want to map things so that I can say "end user application XYZ is not accessible in Kansas due to X being down"."
Nagios can do that, while I never deployed it that way.
"I want my monitoring solution to understand HA and service degredati
Don't be like Tivoli, OpenView, etc (Score:3, Insightful)
Focus on usability and rapid deployment rather than wide-ranging featuresets that sit on the shelf for a decade. Nearly all products in this space really, really suck.
Intermapper (Score:2)
Big fan of intermapper (www.dartware.com) ... It can use nagios plugins as well.
It's fairly cheap.. We monitor about 1250 devices at the moment with it... can be set all way down to 5 seconds.
Server and Client are both in Java... so more or less it runs on any platform.
They give out 30 day demo keys.
Cacti w/plugins (Score:3, Interesting)
I use Cacti, with THold [cacti.net] and weathermap [cacti.net] plugins.
But then I'm biased.
If you would write one from scratch... (Score:2)
Then I advise you to let all the connection checks do by a machine dedicated for this task.
The statistical checks on each of the 5000 machines itself 24 hours a day while pushing/pulling data at intervals to that dedicated machine.
I advise 2 or more measuring machines each on a separate network each doing the same task and synchronizing their data for redundancy.
Intellipool Network Manager (Score:2)
You can always try INM (http://www.intellipool.se/ [intellipool.se]).
It's quite feature rich and it's worth a look.
OpenNMS (Score:3, Insightful)
That fact that it is also NRPE compatible was a plus - I could use all the Nagios plugins and check scripts I'd written.
I was also planning on using it to launch a more aggressive webmin-style management solution - since OpenNMS built this great database of data about my devices and hosts, I could use it to do actual management - change data/settings.
Cons: It's a Java/Tomcat tool, as much as that is really a con. It's not like you need to run Jboss or Websphere to use it (though I suppose you could).
What I Lack in Open Source Monitoring Solutions (Score:5, Insightful)
I just did a quick survey and evaluation of the open source monitoring-market for my company, and found a few shortcomings/frustrations in a few aspects where none of the evaluated system seems to get it 100% right.
Transparent Planned Design
Many solutions out there seems to have been developed in what can only be described as an "organic" process. I.E. a few scripts were used from start, were hooked up with some other scripts, were slammed into a web-interface, got some more features, then something central were ripped out and replaced to allow yet more features and so on and so forth. (Read: Nagios) While this is of course often the best way to get something working for a particular need, and on a tight budget, it makes adoption really hard unless you happen to have exactly the same need.
Event management
Does anyone know a solution that can both receive from syslog and decode traps with a given MIB, and then do some simple logic, like squashing repeats, displaying on a web-page with archival-options, and dispatch to mail/sms based on configurable rules? Except for ZenOSS (and ZenOSS have other problems), I haven't found a single sensible system that does this.
Modularity/Seamless Integration
Since much of the monitoring systems out there doesn't seem to have a clear design, it's often very hard to add missing features. I.E. project X missing an event manager, or is the builtin not satisfactory? No probs, I'll just, ehh, where does this wire come from? Is this really a socket? Did anyone really connect that? It's ok with blackbox-solutions, as long as they serve all my needs, and have clear interfaces to combine with other solutions that serves related needs, but sadly no solution evaluated does everything we need it to and we end up struggling with manual routines to compensate for it.
Complexity
There are a few really neat systems that does almost everything one can ask for. (Short of flying cars). Unfortunately, the ones we've tried have always turned out to be very complex, and also do a lot of things we didn't want. Since it's then often not very modular, it hard to get it stop doing the things we don't want, or change the things we need implemented slightly differently. Also the huge codebase that comes along with trying to scratch everyones itch seems to get it's share of bugs, and troubleshooting in large more or less opaque systems is not a fun task.
The Perfect Monitoring System
After evaluating all options we could find, we've come to the conclusion that none of the systems we've looked at or tested really fits our needs (Although ZenOSS came close, we encountered just too many bugs and oddities to keep investing time in it). Furthermore, we could not find a combination of systems that integrates well, and together fits our needs, which I personally see as a bigger problem.
What I would really want to see in the world of Open Source Monitoring, is an eco-system of monitoring apps with an overarching design/architecture. Design a framework where different entities and steps in the monitoring are clearly defined and interfaced with each other, but still allows for differing implementations, and integration with unforeseen needs. For example, at our shop, we continuously analyze roughly 700mbit of streaming video for availability and quality. Noone designing a monitoring system could probably forsee this as an appliance, but in The Perfect Monitoring System, it should be clear for the average-skilled hacker how to integrate it.
Use the Tivoli architecture and rewrite it (Score:2)
As you've discovered, the free systems will fall over and die once they're past a certain size. I've worked with Tivoli customers that have tens of thousands of servers, and for all of its problems, Tivoli is scalable.
They way they do it is, obviously, divide and conquer. There are specific ways that they do it. I'm mixing up the architecture and terminology on purpose, because the Tivoli terminology will confuse you.
* there are agents on every box that do the monitoring
* they report to a region
* those regi
Re: (Score:3, Interesting)
How about... (Score:5, Informative)
Needed features in random order: ...and a system for annotating the above. Raw data is neat, annotated data is even better.
* Scalability - few k machines is minimum. This probably means smart, decentralized collection and aggregation of data.
* Flexible whitebox monitoring - for given class of devices, I should be able to configure how to fetch this device's data (http, smnp, ssh+command, rpc, you-name-it) and how to interpret it ("read the status page there, get this and that value").
* Flexible blackbox monitoring - for given class of devices, I should be able to configure a set of actions that should be performed on it (fetch a page, ssh into, ping) and how results of that action should be interpreted (ok/nok, time to complete, etc.).
* Easy way to tag (source/machine/network segment) and aggregate (max/min/mean/stddev/%ile/sum) of the monitoring data.
* Some language to easily calculate derivative values from the data above.
* Interface for defining graphs, using collected data.
*
* Alerting subsystem, which should allow for defining different destinations, together with escalation rules. And custom alerts - using the .
* (nice to have) HTTP server with a simple HTML templating, to allow for easy creation of arbitrary dashboards.
* (if you have the above) predefined templates for most of common things. Both detailed ("everything about device X") and general ("if the background of the page is green, you're fine! If it's not, here you'll find a concise list of what's broken").
* hooks/libraries to use collected data "outside" of the system
I realize that's a lot, but boy, such system would be very useful and flexible.
Monitoring on scale (Score:3, Interesting)
If your monitoring something of that scale, you should probably look into a profession solution.
I use Zenoss (open source) and like it quite a bit. It takes time to customize for your setup, but unless you have a bland network, that is almost always the case. I will say this, it's much easier to setup the Nagios was a couple of years ago when I was using Nagios. Though I've heard there has been some improvement.
Clu (Score:2)
If you want a monitor that can display useful information about thousands of nodes on a single display try clumon [illinois.edu]. We use it for our 1000+ node clusters. The software was developed in-house but is available under the University of Illinois/NCSA Open Source License Copyright (noticeware). If you're just going to use this in-house, the license shouldn't be an issue.
You can see a sample clumon display of a working cluster at NCSA Linux Cluster Monitor [teragrid.org].The clumon page for that cluster shows you each the job s
Real world alarm capability (Score:2)
I know I'm late to the party, but I haven't seen anyone bring this one up yet: Real-world alarm/notification capability (pager, buzzer, a machine that goes bing, [youtube.com] something like that)
My reasoning: I run a small IT business with various support contracts. I, and probably quite a few others, can't afford to pay someone to sit at a monitor and watch a screen (or a bunch of screens) whilst tied to a desk.
Most of the monitoring solutions (Nagios, others) are capable of off-site notification, but it's the "last
Re: (Score:2)
Real-world alarm/notification capability (pager, buzzer, a machine that goes bing, [youtube.com] something like that)
...sorry, I mentioned "pager" as a real world alarm, then panned the idea--I meant "pager" as in "Mr. Jones, please check the server logs..."
Groundwork (Score:2)
We are using http://www.groundworkopensource.com/ [groundworkopensource.com] for our monitoring. It is working pretty well, and we can use existing Nagios scripts with it.
If you have Linux servers, don't use OpManager! (Score:3, Informative)
We used OpManager in production for over a year. It has terrible Linux support. None of their built-in plugins worked properly for monitoring even basic parameters like disk space, free memory, CPU usage, etc. When we pointed this out to their support people, they said we should build our own plugins with SNMP OIDs. Um....no. Not for the amount of money we paid for that steaming POS. We finally kicked OpManager to the curb about a month ago, and have our entire environment, Windows and Linux servers being monitored with Nagios. Nagios scales well, we are currently watching several hundred hosts and about 3500 services.
OpenNMS is also a good tool, its ability to map servers back to switch ports is extremely handy.
Re: (Score:2)
Sounds like you need a centralized syslog server. It could do more for you than just log commands.....