By Katherine Chalmers
January 2006

The Dirty Little Secret Behind Incident Management

For many companies incident management is a natural first step toward adopting ITIL. Unfortunately the dirty little secret behind incident management is that while companies are beginning to adopt new best practices like ITIL and proactive processes, many are neglecting to make the most of the monitoring infrastructures they already have in place that could make these initiatives much easier.

A recent survey of IT professionals about their performance management solutions, confirms that underutilization of monitoring tools is common – but often necessary to avoid information overload. Recently, Channel Source Direct conducted telephone interviews of 195 IT professionals responsible for these systems in an Incident Management survey sponsored by Netuitive. They found that 39% of respondents said that they purposely set performance metric baselines at higher than optimal levels simply to reduce the number of alarms they receive from the system. Forty-seven percent of respondents who receive more than 100 alerts per day say that they employ this tactic to manage floods of alarms.

This means that these users are getting far less benefit than their performance monitoring software is intended to provide. By setting thresholds abnormally high, they risk missing critical early notifications of impending problems.

In fact, despite setting thresholds high, many IT organizations still struggle to avoid information overload when their performance monitoring solutions send alarms for normal system fluctuations. Overall 30% of the survey respondents said that more than half of the alerts they receive are non-critical or false. Sixteen percent reported that 70% or more of the alarms they receive are non-critical or false.

“Properly configuring and maintaining performance monitoring software is a real challenge,” explains Mike Wise, a senior solution architect and ITIL consultant at Netuitive. “To truly optimize one of these systems, you’d have to analyze and identify the normal operating ranges of hundreds of system metrics on every component in your infrastructure – plus you’d have to know how each metric typically fluctuates every hour, every day, and every month. It’s humanly impossible to optimize all this with manual thresholds or even with custom scripting.”

Many companies find that even aggressive threshold administration fails to completely eliminate their false alerting problem. Among respondents who devote 50 or more hours per quarter to thresholding, 43% say half or more of their alerts are false or non-critical and 20% say more than 70% are.

“Effective monitoring and clean, accurate problem data is crucial,” Netuitive’s Wise continues. “It’s garbage in, garbage out.’ If you fuel your ITIL processes and BSM tools with data from an underutilized performance monitoring solution, you’re just moving junk from one place to another and adding more complexity to the mess.”

How do companies overcome information overload and make the most of their systems? “To be honest – simply measuring service levels is not really what it’s about – since service levels may capture only a small part of the performing infrastructure,” explains Dennis Drogseth, Network Services Practice Leader at analyst firm Enterprise Management Associates. “Broad analytic systems designed to validate effective performance in the face of change are coming of age on the market – but, one would have to say from this that too few IT organizations are using them yet.”

Three of the new analytics systems currently available to help companies enhance their monitoring solutions are BMC Performance Manager Intelligent Alerts from BMC Software, AppManager Performance Profiler from NetIQ, and Netuitive SI from Netuitive.

Whether they use traditional manual performance monitoring system administration techniques or new real-time analysis systems, many companies should definitely re-examine how they are using their existing incident management systems. There may be hidden value in those existing systems.