How to Accelerate
Your Application
Problem Resolution
Process
3 Best Practice Case Studies
pg_0002
How to Accelerate Your Application
Problem Resolution Process
© 2005 Identify Software, Ltd.
2
Abstract
With the cost of application downtime ranging from nearly $100,000 per hour
in the retail industry, to literally millions per hour in the finance industry it’s not
surprising that companies devote significant resources to resolving application
problems. What is perhaps surprising (and, fortunately, correctable) is how much
money and effort the process demands. Why is application problem resolution
so costly?
Firstly, the laborious process of problem resolution for production applications
is still performed manually by staff from the application development, IT
operations, and help desk teams, which ties up company resources.
Secondly, root cause analysis is still an iterative process of information gathering,
debugging, and attempting to recreate the problem in a lab, leading to finger-
pointing among departments, long resolution times, exorbitant downtime
costs, and frustrated users. During QA/testing and deployment, it can mean
missed launch dates and internal strife.
Accelerate the problem resolution process throughout the application lifecycle,
and you can eliminate some of the costs, business impact, and improve
customer service.
This paper examines the challenges of application problem resolution and presents
three (3) case studies that demonstrate how leading global companies in the
banking, telecommunications, and healthcare industries are using best practices to
automate their application problem resolution processes, eliminate the need to
recreate problems in the lab, and reduce their IT support costs significantly–all
with a rapid ROI.
pg_0003
How to Accelerate Your Application
Problem Resolution Process
© 2005 Identify Software, Ltd.
3
Table of Contents
Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Applications demand a new kind of support . . . . . . . . . . . . . . . . . . . . . . . . 4
Downtime costs big time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Finance Industry Case Study: Danske Bank . . . . . . . . . . . . . . . . . . . . . . . . . 6
Telecom Industry Case Study: Cox Communications . . . . . . . . . . . . . . . . . . 8
Healthcare Industry Case Study: Cerner Corporation . . . . . . . . . . . . . . . . . . 9
Identify’s AppSight Application Problem Resolution System . . . . . . . . . . . . 11
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
About Identify Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
pg_0004
How to Accelerate Your Application
Problem Resolution Process
© 2005 Identify Software, Ltd.
4
Applications demand a new kind of support
Software is the backbone of business today. You would think that as enterprise
applications became indispensable, they would likewise become more and more
dependable. Instead, the rising tide of systems complexity and interdependency
are making enterprise application problems inevitable, intractable, and more elusive.
Business interruptus reigns
At one major insurer, for example, a payables application mysteriously kept
crashing on startup, but only for one unlucky group of users, and only on two
of six servers. After investing over a month and $50,000 in a multi-engineer
analysis, the organization still had not discovered a cause or cure.
At an upscale retail specialty store, cashier queries for customer records were
hanging up the registers during the wait for overdue server responses. Then,
every time an impatient cashier rebooted an individual register, this locked
up all other registers, requiring a 30-minute server reboot that suspended all
transactions storewide. Nine technicians worked the problem for three months
to no avail, costing over $150,000–not counting lost sales and goodwill.
A leading hospital management organization faced recurring, intermittent
processing delays in a Web-based application. When initial resolution efforts
proved fruitless, the organization deployed a 15-person crisis team that cut
across all application and infrastructure domains. This team spent some 3,000
hours–$300,000 in fully loaded costs–and then, still stymied, looked outside for
further help.
These are not isolated examples. They represent business as usual (or the
interruption of same) throughout the Fortune 2000 today.
More power, more players, more problems
The characteristics that make today’s enterprise applications so capable and
productive when all goes well, make them nearly inscrutable when things go
wrong. Contemporary enterprise applications encompass millions of lines of
code, never fault-free in such quantities. They serve myriad users who, being
human, predictably do unpredictable things with them. They span multiple
platforms and networks, run by multiple groups who wield fragmentary
expertise, authority, and views of the environment.
* See p.12 to see how these problems were resolved.
pg_0005
How to Accelerate Your Application
Problem Resolution Process
© 2005 Identify Software, Ltd.
5
The net result is mission-critical organisms with millions of moving parts. At this
scale of complexity, processes inevitably collide and conflict, with elusive root
causes and costly results.
Downtime costs big time
With the cost of application downtime in major organizations ranging from
nearly $100,000 per hour in retail, to literally millions per hour in banking and
brokerage, it’s not surprising that companies devote the necessary resources to
resolving application problems. What is perhaps surprising (and, fortunately,
correctable) is how much money and effort the process demands.
Throughout the 7.2-year average lifespan of major enterprise applications,
software maintenance and support costs represent between 60-80% of total
software lifecycle costs. This limits developers’ ability to advance new initiatives,
and goes far to explain why corporations tend to view IT operations as cost centers
rather than the strategic enablers and centers of value they can and should be.
Software maintenance and support is so costly because the largely manual,
trial-and-error-driven approaches that prevail cannot cope with today’s complex
enterprise applications. Consider the course of a typical application malfunction:
An end user reports a problem to the help desk, initiating a mutually frustrating
exchange. The responder tries to learn everything possible, but the user is hard
pressed to help because–until problem intruded–he or she was focused on
business objectives, not keystrokes and system states.
Problem replication is the roadblock
Even when end users’ trouble reports are impeccable, the problems that absorb
the bulk of problem resolution team resources present themselves as mysteries.
Brokerage
$4.5M Healthcare
$0.5M
Telecommunications
$3.0M Hospitality/Travel $0.5M
Energy
$3.0M E-Commerce $0.1M
Banking
$2.6M Retail
$0.09M
IT-dependent Manufacturing $1.5M Transportation $0.09M
Business impact of downtime, per hour
Source: META Group, Oct. 2000 and The Yankee Group, March 2004
pg_0006
How to Accelerate Your Application
Problem Resolution Process
© 2005 Identify Software, Ltd.
6
In addressing these, fully 80% of the effort goes not into resolving the problems
per se, but into replicating them so their root cause can be identified. It’s
painful to spend resources re-creating problems no one wanted in the first
place. But it’s just as your car has to reproduce that indescribable, intermittent
noise when the mechanic is listening or you can’t get the help you need.
On average, problem replication efforts involve 6.5 experts, who must replicate
the problem 4.8 times (with fingerpointing, headscratching, and further grilling
of exasperated end users) before they establish the root cause.
Following are examples of how three top organizations in different industries
are using an innovative technology approach that enables them to eliminate the
cumbersome problem replication process, so they can meet the problem resolution
challenge head-on–accelerating the problem resolution process and reducing
application support costs significantly.
Finance Industry Case Study: Danske Bank
Founded in 1871, Danske Bank is the largest bank in Denmark and a leading
player in the Scandinavian financial markets, offering a wide range of financial
services, including insurance, mortgage finance, asset management, brokerage,
credit card, real estate, and leasing services. In 2000, Danske Bank’s IT organization
designed Business PC to provide its customers throughout Europe with a
reliable, easy way to perform financial transactions securely and cost-effectively.
While Business PC would
provide new levels of
service and cost efficiency,
application problems
constituted a major risk.
A single lost or delayed
transaction could prove
both expensive, and
damaging to the
bank’s reputation.
A lifecycle approach to application problem resolution
Ebbe Skak Larsen, Chief Architect at Danske Bank, recognized the need for a
better infrastructure for application problem resolution throughout Business PC’s
lifecycle. After initiating an in-house project to build a comprehensive system
support solution, Larsen found Identify Software’s AppSight Black Box system
and Danske Bank has been using it ever since.
Return on investment for the first year
of operation alone amounted to more
than $1,000,000.
- Ebbe Skak Larsen, Chief Architect,
Danske Bank
pg_0007
How to Accelerate Your Application
Problem Resolution Process
© 2005 Identify Software, Ltd.
7
With AppSight, the support team can easily pinpoint the root cause of application
failures, such as performance bottlenecks, configuration problems, and functional
errors, without having to recreate the problems, saving them significant time
and costs. Today, Business PC is offered in 11 localized versions and serves over
15,000 companies and 50,000 users in across Europe and the US. AppSight
Black Box is an integral part of the Business PC package, enabling Danske Bank’s
support team in Aarhus, Denmark to provide users with technical support as
problems emerge.
Eliminating the need to recreate problems in a lab
According to Larsen, even with good QA, errors will occur in any complex
application. “Production errors are typically difficult to recreate in our test and
customer support labs and might result in our customers experiencing costly
application downtime,” Larsen explained. AppSight helps Danske Bank’s support
operation work more effectively, streamlining the process of capturing and
communicating problems, and providing rapid root cause analysis for problem
resolution. “It can cost a company tens of thousands of dollars to pinpoint a
single application error, recreate the problem, and resolve the issue, in addition
to lost employee productivity and customer confidence,” said Larsen. “AppSight
enabled us to identify the cause of problems in a few hours, rather than a few
weeks. We have solved some very elusive problems we may never have found
without AppSight.”
Bottom-line benefits
Danske Bank adopted AppSight Black Box technology because it ensured the
highest levels of customer support for Business PC. Prior to implementation,
Larsen estimated that using AppSight would save the bank more than $900,000
over a period of five years. A year after implementation, Larsen calculated that
return on investment for the first year of operation alone amounted to more
than $1,000,000.
pg_0008
How to Accelerate Your Application
Problem Resolution Process
Telecom Industry Case Study: Cox Communications
Cox Communications is among the largest broadband communications companies
in the U.S., serving over six million customers in more than 20 states and
Cox.com is the primary interface between Cox’s 16,000 employees and its
customers. Through Cox.com, Cox Communications provides the latest service
information, sells new services, and provides state-of-the-art electronic bill
payment. The Cox technical support team supports 30 internal applications and
40 intranet applications for each of the company’s remote customer-facing
sites–the complete system comprises over 25,000 nodes.
Maintaining responsiveness, while containing costs
Rapid application error resolution is crucial to generating revenues through
Cox.com. “We need to provide our customers with real-time responses to their
requests, regardless of any errors that a transaction has generated,” notes James
Hampton, Cox’s Manager of Configuration Management and Systems. “Every
time a customer signs on to Cox.com, we want to be able to offer him promotions
and special discounts according to his profile, as well as update his account
history. As a result, a single transaction touches between 2 and 6 servers.”
The Cox.com system
ensures that the
customer gets a
payment receipt, even
for a failed transaction.
The order is then
reconciled offline by
the back-office group
in a time-consuming,
manual process.
To reduce application
support costs, Cox
selected AppSight for
its state-of-the-art
problem resolution
technology. “Given the complexity of our system, resolving application errors is
very costly and time-consuming,” explains Hampton. “Recently, for example,
we had a problem with a specific transaction failing to complete its update
sequence through our system. It took us three days, involving two people each
from three functional groups to determine that the error was the result of an
Oracle database being taken offline for regular maintenance.”
“Getting to the root cause of that problem took us over 140 man-hours. Now
© 2005 Identify Software, Ltd.
8
Getting to the root cause of that problem
took us over 140 man-hours. Now AppSight
enables us to record the exact movement of
a transaction and the associated activity.
We now solve this kind of problem in less
than an hour.
- James Hampton, Manager of Configuration Management Systems,
Cox Communications
pg_0009
How to Accelerate Your Application
Problem Resolution Process
AppSight enables us to record the exact movement of a transaction and the
associated activity. We now solve this kind of problem in less than an hour,”
says Hampton.
Comparing configurations to pinpoint problems
Cox is finding that AppSight is effective in finding the root cause of transaction
errors in their web applications, and in pinpointing more complex issues, such
as periodic performance degradation on specific servers. For example, Hampton
recently applied AppSight to understand why one database server slowed down
periodically in the morning. “AppSight provides us with the ability to compare
the configurations of a server that is working properly with that of a problematic
server, all from the application’s perspective,” says Hampton. “We combined
this comparison with log samples during these slow periods and discovered
scheduling issues with a group of COM objects. We adjusted schedules and
achieved significant performance improvement. Typically, the technical support
team spent over 50% of its time recreating a problem scenario—AppSight
eliminates this step. You want to minimize the fingerpointing and get to actual
problem solving as fast as possible,” Hampton notes.
Providing value throughout the application lifecycle
Using AppSight in production provided Cox with huge savings, but Hampton
sees even greater value in applying the technology throughout the application
lifecycle–from development and testing to production. “AppSight provides a
comprehensive log of how an application actually behaves in the production
system. I would like the development team to review those logs and see what
kinds of errors and conflicts emerge from a classic set of sessions. This final
check would make our application rollouts to production much smoother and
could dramatically reduce the number of undetected problems in our applications.”
Healthcare Industry Case Study: Cerner
As a world leader in healthcare software solutions, Cerner is replacing paper
charts with intelligent, interactive electronic forms designed to improve patient
care and business management by “eliminating error, variance, waste, delay,
and friction.” Because the lifeblood of modern medicine is information, and
healthcare institutions increasingly depend on software solutions to manage and
deliver it, downtime and application problems can bring patient care to a costly
and hazardous halt.
Minimizing downtime requires rigorous quality assurance and it’s typically a
manual, laborious, and costly process. “Given the labor it takes to assure the
software quality demanded in healthcare, where lives are literally at stake, we
© 2005 Identify Software, Ltd.
9
pg_0010
How to Accelerate Your Application
Problem Resolution Process
were immediately intrigued by Identify’s AppSight Black Box technology,”
recalls Owen Straub, Cerner’s VP of Engineering.
Cerner first used AppSight to accelerate its Quality Assurance process, and
found that the more than 40 engineers in its QA group gained two hours
per person, per day. Because Cerner’s QA engineers also provide level-three
customer support, they soon saw the difference between solving problems in
the lab with AppSight and the challenge of troubleshooting problems in the
field without it.
Zooming in on root causes
Cerner now embeds Identify’s Black Box technology in its Cerner Millennium
healthcare solutions platform. The Black Box is transparent to users and has no
effect on system performance. But its effect on customer support is dramatic.
Either running proactively and feeding log data into cyclical, FIFO buffers, or
running on-demand when problems emerge, the Black Box captures field
problems when, where, and
how they happen. This
enables Cerner’s engineers to
analyze customer Black Box
logs with AppSight, swiftly
pinpoint the root cause of a
problem, and proceed
directly to a solution–saving
the hours, days, or even
weeks involved with problem
replication. “We found that
employing AppSight in production virtually eliminates the need to recreate
problems in the lab, saving us significant time and costs,” says Straub.
For example, a Cerner client in radiology reported a baffling problem:
a transcriptionist found that random paragraphs kept mysteriously disappearing
from her transcriptions. The problem escalated to the point that the client needed
it fixed immediately, since avoidably incomplete information in healthcare
records poses great potential danger. Cerner sent its level-3 SWAT team on-site,
as stipulated in the client’s service agreement. When the transcriptionist stepped
out for a moment, the problem happened again, so the Cerner engineer activated
the Black Box. Then, when the problem quickly recurred, the engineer was
able to study the log and discover that the transcriptionist, in cleaning up raw
transcription files, kept hitting a combination of keys that the relevant program
interprets as “delete paragraph.” With the root cause revealed, the fix was
simple, and nearly instantaneous. Had AppSight Black Box technology not been
© 2005 Identify Software, Ltd.
10
We found that employing AppSight in
production virtually eliminates the need
to recreate problems in the lab, saving
us significant time and costs.
- Owen Straub, VP of Engineering
Cerner Corp.
pg_0011
How to Accelerate Your Application
Problem Resolution Process
used to trap this problem “in the wild,” Straub estimates that replicating such
an unforeseen and idiosyncratic problem “could have taken weeks.”
Along with reducing the duration of client site visits and dramatically shortening
downtimes, AppSight has reduced the number of site visits needed because
Cerner engineers can now solve more problems remotely. “Before AppSight,”
says Straub, “tough customer problems pulled our developers away from new
product work for intolerably long stretches. Now that AppSight reveals the root
causes, they can be instant heroes and get right back on task.”
AppSight creates a win/win situation: Cerner delivers superior product performance
and better, timelier customer support while spending less money to do so.
Cerner clients deliver superior patient care and gain business efficiencies while
enjoying increased confidence in their information solutions and the company
standing behind them.
Identify’s AppSight Application Problem Resolution System
For most enterprises and software vendors, problem resolution remains a manual,
error-prone, communications-intensive activity that involves multiple members
of the application team-architects, developers, testers, and operations personnel.
The problem resolution team typically goes though a lengthy and costly process
that includes iterative attempts to gather information about the problem and
multiple attempts to recreate the problem before even beginning the root cause
analysis phase. Identify’s AppSight automates the entire process.
AppSight leverages patented Black Box software technology to monitor application
execution and capture a synchronized, real-time log of user actions, system
events, performance metrics, configuration data, and code execution flow–
much like the “black box” flight recorder on an aircraft captures a real-time
record of a flight. The AppSight Black Box log can be replayed and analyzed to
quickly pinpoint the root cause of all kinds of Microsoft Windows, .NET, and
J2EE application problems, whether related to performance, configuration
issues, user mistakes, or code errors.
By capturing actual problem history in a centralized repository, AppSight provides
the basis for team collaboration and communication. By allowing each member
of the team to quickly analyze problem information using role-based views,
rather than recreate the problem, AppSight eliminates up to 70% of the cycle
time traditionally consumed by root cause analysis.
AppSight complements and synergizes the existing, more narrowly-focused
tools that organizations use to support their enterprise systems, such as systems
management products like Unicenter, Patrol, Tivoli, and MOM, which provide
monitoring and troubleshooting for the system infrastructure, and software
© 2005 Identify Software, Ltd.
11
pg_0012
How to Accelerate Your Application
Problem Resolution Process
development tools that enable developers to test
software functions and hardware/software interactions
under lab conditions.
To appreciate the difference that AppSight makes,
let’s revisit the three problems we mentioned earlier:
When the healthcare insurer installed AppSight,
they had already invested $50,000 into a stalled
application problem analysis. With AppSight,
they were able to solve the problem in less
than two hours. Analysis of the Black Box log
pinpointed the root cause of the problem to
be access rights to the required Windows
registry key, due to an inadvertent and
undetected change.
AppSight enabled the specialty retailer to solve a cash register problem that had
eluded them for more than three months–at a cost of $150,000 and counting–in
just one day. Resolving this problem not only stopped a $50,000-per-month
cash burn, but also enabled the retailer to resume scheduled new store openings
that had been placed on indefinite hold for the duration of the crashing cash
register nightmare.
Finally, the hospital administration organization that had fruitlessly deployed a
15-person, ad-hoc crisis resolution team to troubleshoot its web-based application
found that AppSight helped them narrow the problem down–for the first time
since it emerged–in just a few hours. By isolating the root cause to specific
components in the application server tier, the company was able to return most
crisis team members to their normal responsibilities. Within the next several
days, the remaining engineers pinpointed the problem to threading issues with
their COM object, and solved it.
Had the company deployed AppSight from the beginning, they could have
reduced their investment in engineering hours from 3,000 to less than 100–
saving over $290,000 in this one incident alone!
Summary
The characteristics that make today’s enterprise applications so powerful and
productive when all goes well, make them nearly inscrutable when things go
wrong. Contemporary enterprise applications encompass millions of lines of
code and serve myriad users who interact with them in ways that cannot be
adequately tested in a lab.
© 2005 Identify Software, Ltd.
12
pg_0013
How to Accelerate Your Application
Problem Resolution Process
© 2005 Identify Software, Ltd.
13
Throughout the 7.2-year average lifespan of major enterprise applications,
software maintenance and support costs represent between 60-80% of total
software lifecycle costs. It is so costly because the largely manual, trial-and-error-
driven approaches that prevail cannot cope with today’s complex enterprise
applications.
New technology approaches, such as Identify’s AppSight application problem
resolution software, are helping top organizations in the finance, retail, healthcare,
manufacturing, insurance, communications, and many other industries accelerate
the problem resolution process by eliminating the cumbersome process of problem
replication–along with the fingerpointing and frustration that go with it–to
improve application performance and availability, reduce application support
costs, and enhance business service.
About Identify Software
Identify
®
is the leader in application problem resolution software, helping hundreds
of enterprises and software vendors speed application delivery, increase application
quality, performance, and availability, and reduce application support costs.
The company’s AppSight Application Problem Resolution System leverages
unique, patented Black Box Application Flight Recorder technology to capture,
communicate, and determine the root cause of application problems, thereby
dramatically accelerating problem resolution processes across the application
lifecycle. Founded in 1996, Identify is a global organization, with operational
headquarters in New York and offices in Israel and throughout the U.S., Europe,
and Asia-Pacific. For more information about Identify, please visit
www.identify.com or call 800.364.5467 or +1 919.878.3717.
I de nti fy i s a r egi s ter ed t ra dema rk and A ppSi ght is a tra dema rk o f I dent ify So ft war e L td. Al l o the r na mes co nt ai ned here in are the ma rks o f t hei r r es pec ti ve ho lde rs .