How to Accelerate Your Application Problem Resolution Process

With the cost of application downtime ranging from nearly $100,000 per hour

in the retail industry, to literally millions per hour in the finance industry it’s not

surprising that companies devote significant resources to resolving application

problems. What is perhaps surprising (and, fortunately, correctable) is how much

money and effort the process demands. Why is application problem resolution

Firstly, the laborious process of problem resolution for production applications

is still performed manually by staff from the application development, IT

operations, and help desk teams, which ties up company resources.

Secondly, root cause analysis is still an iterative process of information gathering,

debugging, and attempting to recreate the problem in a lab, leading to finger-

pointing among departments, long resolution times, exorbitant downtime

costs, and frustrated users. During QA/testing and deployment, it can mean

Accelerate the problem resolution process throughout the application lifecycle,

and you can eliminate some of the costs, business impact, and improve

This paper examines the challenges of application problem resolution and presents

three (3) case studies that demonstrate how leading global companies in the

banking, telecommunications, and healthcare industries are using best practices to

automate their application problem resolution processes, eliminate the need to

recreate problems in the lab, and reduce their IT support costs significantly–all

Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Applications demand a new kind of support . . . . . . . . . . . . . . . . . . . . . . . . 4

Downtime costs big time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Finance Industry Case Study: Danske Bank . . . . . . . . . . . . . . . . . . . . . . . . . 6

Telecom Industry Case Study: Cox Communications . . . . . . . . . . . . . . . . . . 8

Healthcare Industry Case Study: Cerner Corporation . . . . . . . . . . . . . . . . . . 9

Identify’s AppSight Application Problem Resolution System . . . . . . . . . . . . 11

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

About Identify Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Applications demand a new kind of support

Software is the backbone of business today. You would think that as enterprise

applications became indispensable, they would likewise become more and more

dependable. Instead, the rising tide of systems complexity and interdependency

are making enterprise application problems inevitable, intractable, and more elusive.

At one major insurer, for example, a payables application mysteriously kept

crashing on startup, but only for one unlucky group of users, and only on two

of six servers. After investing over a month and $50,000 in a multi-engineer

analysis, the organization still had not discovered a cause or cure.

At an upscale retail specialty store, cashier queries for customer records were

hanging up the registers during the wait for overdue server responses. Then,

every time an impatient cashier rebooted an individual register, this locked

up all other registers, requiring a 30-minute server reboot that suspended all

transactions storewide. Nine technicians worked the problem for three months

to no avail, costing over $150,000–not counting lost sales and goodwill.

A leading hospital management organization faced recurring, intermittent

processing delays in a Web-based application. When initial resolution efforts

proved fruitless, the organization deployed a 15-person crisis team that cut

across all application and infrastructure domains. This team spent some 3,000

hours–$300,000 in fully loaded costs–and then, still stymied, looked outside for

These are not isolated examples. They represent business as usual (or the

interruption of same) throughout the Fortune 2000 today.

The characteristics that make today’s enterprise applications so capable and

productive when all goes well, make them nearly inscrutable when things go

wrong. Contemporary enterprise applications encompass millions of lines of

code, never fault-free in such quantities. They serve myriad users who, being

human, predictably do unpredictable things with them. They span multiple

platforms and networks, run by multiple groups who wield fragmentary

expertise, authority, and views of the environment.

* See p.12 to see how these problems were resolved.

The net result is mission-critical organisms with millions of moving parts. At this

scale of complexity, processes inevitably collide and conflict, with elusive root

With the cost of application downtime in major organizations ranging from

nearly $100,000 per hour in retail, to literally millions per hour in banking and

brokerage, it’s not surprising that companies devote the necessary resources to

resolving application problems. What is perhaps surprising (and, fortunately,

correctable) is how much money and effort the process demands.

Throughout the 7.2-year average lifespan of major enterprise applications,

software maintenance and support costs represent between 60-80% of total

software lifecycle costs. This limits developers’ ability to advance new initiatives,

and goes far to explain why corporations tend to view IT operations as cost centers

rather than the strategic enablers and centers of value they can and should be.

Software maintenance and support is so costly because the largely manual,

trial-and-error-driven approaches that prevail cannot cope with today’s complex

enterprise applications. Consider the course of a typical application malfunction:

An end user reports a problem to the help desk, initiating a mutually frustrating

exchange. The responder tries to learn everything possible, but the user is hard

pressed to help because–until problem intruded–he or she was focused on

business objectives, not keystrokes and system states.

Even when end users’ trouble reports are impeccable, the problems that absorb

the bulk of problem resolution team resources present themselves as mysteries.

IT-dependent Manufacturing $1.5M Transportation $0.09M

Source: META Group, Oct. 2000 and The Yankee Group, March 2004

In addressing these, fully 80% of the effort goes not into resolving the problems

per se, but into replicating them so their root cause can be identified. It’s

painful to spend resources re-creating problems no one wanted in the first

place. But it’s just as your car has to reproduce that indescribable, intermittent

noise when the mechanic is listening or you can’t get the help you need.

On average, problem replication efforts involve 6.5 experts, who must replicate

the problem 4.8 times (with fingerpointing, headscratching, and further grilling

of exasperated end users) before they establish the root cause.

Following are examples of how three top organizations in different industries

are using an innovative technology approach that enables them to eliminate the

cumbersome problem replication process, so they can meet the problem resolution

challenge head-on–accelerating the problem resolution process and reducing

Founded in 1871, Danske Bank is the largest bank in Denmark and a leading

player in the Scandinavian financial markets, offering a wide range of financial

services, including insurance, mortgage finance, asset management, brokerage,

credit card, real estate, and leasing services. In 2000, Danske Bank’s IT organization

designed Business PC to provide its customers throughout Europe with a

reliable, easy way to perform financial transactions securely and cost-effectively.

A lifecycle approach to application problem resolution

Ebbe Skak Larsen, Chief Architect at Danske Bank, recognized the need for a

better infrastructure for application problem resolution throughout Business PC’s

lifecycle. After initiating an in-house project to build a comprehensive system

support solution, Larsen found Identify Software’s AppSight Black Box system

and Danske Bank has been using it ever since.

With AppSight, the support team can easily pinpoint the root cause of application

failures, such as performance bottlenecks, configuration problems, and functional

errors, without having to recreate the problems, saving them significant time

and costs. Today, Business PC is offered in 11 localized versions and serves over

15,000 companies and 50,000 users in across Europe and the US. AppSight

Black Box is an integral part of the Business PC package, enabling Danske Bank’s

support team in Aarhus, Denmark to provide users with technical support as

Eliminating the need to recreate problems in a lab

According to Larsen, even with good QA, errors will occur in any complex

application. “Production errors are typically difficult to recreate in our test and

customer support labs and might result in our customers experiencing costly

application downtime,” Larsen explained. AppSight helps Danske Bank’s support

operation work more effectively, streamlining the process of capturing and

communicating problems, and providing rapid root cause analysis for problem

resolution. “It can cost a company tens of thousands of dollars to pinpoint a

single application error, recreate the problem, and resolve the issue, in addition

to lost employee productivity and customer confidence,” said Larsen. “AppSight

enabled us to identify the cause of problems in a few hours, rather than a few

weeks. We have solved some very elusive problems we may never have found

Danske Bank adopted AppSight Black Box technology because it ensured the

highest levels of customer support for Business PC. Prior to implementation,

Larsen estimated that using AppSight would save the bank more than $900,000

over a period of five years. A year after implementation, Larsen calculated that

return on investment for the first year of operation alone amounted to more

Telecom Industry Case Study: Cox Communications

Cox Communications is among the largest broadband communications companies

in the U.S., serving over six million customers in more than 20 states and

Cox.com is the primary interface between Cox’s 16,000 employees and its

customers. Through Cox.com, Cox Communications provides the latest service

information, sells new services, and provides state-of-the-art electronic bill

payment. The Cox technical support team supports 30 internal applications and

40 intranet applications for each of the company’s remote customer-facing

sites–the complete system comprises over 25,000 nodes.

Maintaining responsiveness, while containing costs

Rapid application error resolution is crucial to generating revenues through

Cox.com. “We need to provide our customers with real-time responses to their

requests, regardless of any errors that a transaction has generated,” notes James

Hampton, Cox’s Manager of Configuration Management and Systems. “Every

time a customer signs on to Cox.com, we want to be able to offer him promotions

and special discounts according to his profile, as well as update his account

history. As a result, a single transaction touches between 2 and 6 servers.”

technology. “Given the complexity of our system, resolving application errors is

very costly and time-consuming,” explains Hampton. “Recently, for example,

we had a problem with a specific transaction failing to complete its update

sequence through our system. It took us three days, involving two people each

from three functional groups to determine that the error was the result of an

Oracle database being taken offline for regular maintenance.”

“Getting to the root cause of that problem took us over 140 man-hours. Now

Getting to the root cause of that problem

enables us to record the exact movement of

a transaction and the associated activity.

We now solve this kind of problem in less

- James Hampton, Manager of Configuration Management Systems,

AppSight enables us to record the exact movement of a transaction and the

associated activity. We now solve this kind of problem in less than an hour,”

Comparing configurations to pinpoint problems

Cox is finding that AppSight is effective in finding the root cause of transaction

errors in their web applications, and in pinpointing more complex issues, such

as periodic performance degradation on specific servers. For example, Hampton

recently applied AppSight to understand why one database server slowed down

periodically in the morning. “AppSight provides us with the ability to compare

the configurations of a server that is working properly with that of a problematic

server, all from the application’s perspective,” says Hampton. “We combined

this comparison with log samples during these slow periods and discovered

scheduling issues with a group of COM objects. We adjusted schedules and

achieved significant performance improvement. Typically, the technical support

team spent over 50% of its time recreating a problem scenario—AppSight

eliminates this step. You want to minimize the fingerpointing and get to actual

problem solving as fast as possible,” Hampton notes.

Providing value throughout the application lifecycle

Using AppSight in production provided Cox with huge savings, but Hampton

sees even greater value in applying the technology throughout the application

lifecycle–from development and testing to production. “AppSight provides a

comprehensive log of how an application actually behaves in the production

system. I would like the development team to review those logs and see what

kinds of errors and conflicts emerge from a classic set of sessions. This final

check would make our application rollouts to production much smoother and

could dramatically reduce the number of undetected problems in our applications.”

As a world leader in healthcare software solutions, Cerner is replacing paper

charts with intelligent, interactive electronic forms designed to improve patient

care and business management by “eliminating error, variance, waste, delay,

and friction.” Because the lifeblood of modern medicine is information, and

healthcare institutions increasingly depend on software solutions to manage and

deliver it, downtime and application problems can bring patient care to a costly

Minimizing downtime requires rigorous quality assurance and it’s typically a

manual, laborious, and costly process. “Given the labor it takes to assure the

software quality demanded in healthcare, where lives are literally at stake, we

were immediately intrigued by Identify’s AppSight Black Box technology,”

recalls Owen Straub, Cerner’s VP of Engineering.

Cerner first used AppSight to accelerate its Quality Assurance process, and

found that the more than 40 engineers in its QA group gained two hours

per person, per day. Because Cerner’s QA engineers also provide level-three

customer support, they soon saw the difference between solving problems in

the lab with AppSight and the challenge of troubleshooting problems in the

Cerner now embeds Identify’s Black Box technology in its Cerner Millennium

healthcare solutions platform. The Black Box is transparent to users and has no

effect on system performance. But its effect on customer support is dramatic.

Either running proactively and feeding log data into cyclical, FIFO buffers, or

running on-demand when problems emerge, the Black Box captures field

employing AppSight in production virtually eliminates the need to recreate

problems in the lab, saving us significant time and costs,” says Straub.

For example, a Cerner client in radiology reported a baffling problem:

a transcriptionist found that random paragraphs kept mysteriously disappearing

from her transcriptions. The problem escalated to the point that the client needed

it fixed immediately, since avoidably incomplete information in healthcare

records poses great potential danger. Cerner sent its level-3 SWAT team on-site,

as stipulated in the client’s service agreement. When the transcriptionist stepped

out for a moment, the problem happened again, so the Cerner engineer activated

the Black Box. Then, when the problem quickly recurred, the engineer was

able to study the log and discover that the transcriptionist, in cleaning up raw

transcription files, kept hitting a combination of keys that the relevant program

interprets as “delete paragraph.” With the root cause revealed, the fix was

simple, and nearly instantaneous. Had AppSight Black Box technology not been

used to trap this problem “in the wild,” Straub estimates that replicating such

an unforeseen and idiosyncratic problem “could have taken weeks.”

Along with reducing the duration of client site visits and dramatically shortening

downtimes, AppSight has reduced the number of site visits needed because

Cerner engineers can now solve more problems remotely. “Before AppSight,”

says Straub, “tough customer problems pulled our developers away from new

product work for intolerably long stretches. Now that AppSight reveals the root

causes, they can be instant heroes and get right back on task.”

AppSight creates a win/win situation: Cerner delivers superior product performance

and better, timelier customer support while spending less money to do so.

Cerner clients deliver superior patient care and gain business efficiencies while

enjoying increased confidence in their information solutions and the company

Identify’s AppSight Application Problem Resolution System

For most enterprises and software vendors, problem resolution remains a manual,

error-prone, communications-intensive activity that involves multiple members

of the application team-architects, developers, testers, and operations personnel.

The problem resolution team typically goes though a lengthy and costly process

that includes iterative attempts to gather information about the problem and

multiple attempts to recreate the problem before even beginning the root cause

analysis phase. Identify’s AppSight automates the entire process.

AppSight leverages patented Black Box software technology to monitor application

execution and capture a synchronized, real-time log of user actions, system

events, performance metrics, configuration data, and code execution flow–

much like the “black box” flight recorder on an aircraft captures a real-time

record of a flight. The AppSight Black Box log can be replayed and analyzed to

quickly pinpoint the root cause of all kinds of Microsoft Windows, .NET, and

J2EE application problems, whether related to performance, configuration

By capturing actual problem history in a centralized repository, AppSight provides

the basis for team collaboration and communication. By allowing each member

of the team to quickly analyze problem information using role-based views,

rather than recreate the problem, AppSight eliminates up to 70% of the cycle

time traditionally consumed by root cause analysis.

AppSight complements and synergizes the existing, more narrowly-focused

tools that organizations use to support their enterprise systems, such as systems

management products like Unicenter, Patrol, Tivoli, and MOM, which provide

monitoring and troubleshooting for the system infrastructure, and software

development tools that enable developers to test

software functions and hardware/software interactions

To appreciate the difference that AppSight makes,

let’s revisit the three problems we mentioned earlier:

When the healthcare insurer installed AppSight,

they had already invested $50,000 into a stalled

application problem analysis. With AppSight,

they were able to solve the problem in less

than two hours. Analysis of the Black Box log

pinpointed the root cause of the problem to

AppSight enabled the specialty retailer to solve a cash register problem that had

eluded them for more than three months–at a cost of $150,000 and counting–in

just one day. Resolving this problem not only stopped a $50,000-per-month

cash burn, but also enabled the retailer to resume scheduled new store openings

that had been placed on indefinite hold for the duration of the crashing cash

Finally, the hospital administration organization that had fruitlessly deployed a

15-person, ad-hoc crisis resolution team to troubleshoot its web-based application

found that AppSight helped them narrow the problem down–for the first time

since it emerged–in just a few hours. By isolating the root cause to specific

components in the application server tier, the company was able to return most

crisis team members to their normal responsibilities. Within the next several

days, the remaining engineers pinpointed the problem to threading issues with

Had the company deployed AppSight from the beginning, they could have

reduced their investment in engineering hours from 3,000 to less than 100–

saving over $290,000 in this one incident alone!

The characteristics that make today’s enterprise applications so powerful and

productive when all goes well, make them nearly inscrutable when things go

wrong. Contemporary enterprise applications encompass millions of lines of

code and serve myriad users who interact with them in ways that cannot be

Throughout the 7.2-year average lifespan of major enterprise applications,

software maintenance and support costs represent between 60-80% of total

software lifecycle costs. It is so costly because the largely manual, trial-and-error-

driven approaches that prevail cannot cope with today’s complex enterprise

New technology approaches, such as Identify’s AppSight application problem

resolution software, are helping top organizations in the finance, retail, healthcare,

manufacturing, insurance, communications, and many other industries accelerate

the problem resolution process by eliminating the cumbersome process of problem

replication–along with the fingerpointing and frustration that go with it–to

improve application performance and availability, reduce application support

is the leader in application problem resolution software, helping hundreds

of enterprises and software vendors speed application delivery, increase application

quality, performance, and availability, and reduce application support costs.

The company’s AppSight Application Problem Resolution System leverages

unique, patented Black Box Application Flight Recorder technology to capture,

communicate, and determine the root cause of application problems, thereby

dramatically accelerating problem resolution processes across the application

lifecycle. Founded in 1996, Identify is a global organization, with operational

headquarters in New York and offices in Israel and throughout the U.S., Europe,

and Asia-Pacific. For more information about Identify, please visit

www.identify.com or call 800.364.5467 or +1 919.878.3717.

I de nti fy i s a r egi s ter ed t ra dema rk and A ppSi ght is a tra dema rk o f I dent ify So ft war e L td. Al l o the r na mes co nt ai ned here in are the ma rks o f t hei r r es pec ti ve ho lde rs .