The clues were all right there. An application that had been through intense performance testing was getting regular complaints about its online response times. Because it couldn't be reproduced in the lab and the logging only stored exceptions, the lackluster performance was initially blamed on older browsers or other user technology problems. The Help Desk was inundated with frustrated user calls, and one of the Help Desk reports included the comment that the "10 and 4 o'clock rush hours" were seemingly getting worse.

This clue helped one of the more spirited of the application team track down the problem. In a conversation with the application DBA, it was mentioned that another application was using some of the data, "but that couldn't be a problem because they only needed one table." Further research indicated that late requirements changes and odd programming choices had resulted in the other application siphoning much more data than that during two scheduled processes that ran at, you guessed it, 10 a.m. and 4 p.m.

Applications don't exist in a vacuum. Tight integration across the enterprise exists almost everywhere, and the impact of some of this integration on performance is difficult to measure. Performance testing frequently focuses on one application, and its impact on others is not easy to measure. In reality, statistical analysis on enterprise production performance is needed, and Altaworks Panorama provides that capability. Panorama provides the necessary leads for tracking down cross-application, shared resource impacts that otherwise are very difficult to resolve.

Among other performance monitoring capabilities, Panorama also provides detailed performance analysis, alert notification, and key monitoring views through a Web-based user interface.

Panorama provides a series of low system-overhead Dynamic Sampling Agents that are installed on each system. Specific data adapters then collect information about the application or environment (see Table 1).

The Dynamic Sampling Agent controls this collection, consolidates the information, and sends it to the Service Management Platform (SMP).

The SMP then cycles through all of that data, performing complex statistical summarization, pattern analysis, and cross-component correlation. Alerts can be set that notify the Panorama Web-based user interface of key events for tracking.

I had a chance to test these features within Panorama using their Guided Tour, which runs on your PC. I recommend trying to get this, as it does a great job of demonstrating the key capabilities of the product.

Following install, I started it up with the Guided Tour, which set up a series of Data Adapters and began collecting information about my laptop and a sample application. From there, I was able to start and stop a series of simulations that allowed me monitor realistic changes in my system.

The user interface provides a series of tabs: "Performance", for live viewing of key performance statistics; "TopN", for ranking of performance data; "Deviation", which shows components' performance against expected; "Correlation", which allows multiple components to be analyzed together; "Events", where threshold limits for notification can be viewed; "Reports", where Crystal Reports against the data can be viewed; and "Configuration" for setup. A series of "Jump To" buttons exist that allow like data to be viewed among the tabs without having to set up the query again.

While testing a series of servlet calls, I was able to view the overall performance and then drill down on the performance to view specific servlet behavior. I used the "Top N" view (see Figure 1) to see the servlet performance ranked by peak servlet calls completed/second. Similarly, Panorama can enable method-level collection and ranking for EJBs. You can also do this for generic Java classes via an additional configuration file.

All of this information can be analyzed together to determine any performance correlation between the various components of an application. For example, I showed the correlation between a series of system calls and the percentage of processor time being used. The metric Correlator can be set up to eliminate metrics that have little impact on current performance. Panorama's Analysis Engine also provides a catalog of known performance-impact relationships that ease problem identification.

The performance data is collected every second by the DSA from the individual Data Adapters. When live data is being viewed, it is retrieved every 10 seconds from the appropriate DSA. The SMP harvests consolidated data from the DSAs every 15 minutes. This data is stored in the database, either SQL Server or Oracle. Historical information can then be accessed easily for additional analysis and review. The Time Control button on the left controls the time frame to retrieve.

Another feature that provided some real benefit was the ability to view the data through deviation scores. Deviation scores are roughly defined as "% outside of dynamic limit range for that time interval." This scoring technique lets you compare metrics from any system or application tier, even if they have different units.

I tried creating a few new views of the data being collected, but the process was confusing. A recommendation from Panorama is to use the TopN, Correlation, or Performance view to find the activity that interests you, pause screen refresh, press the Correlation button at the bottom of the page, and save the appropriate view you get with just those metrics. This method worked well with me.

It is also very easy to select and filter out information as needed. A series of dropdown boxes at the top of every view provides this functionality. Drilling down on the metrics is intuitively provided for by double-clicking on the metric that interests you. Setting up event notification is also fairly straight-forward. Panorama also provides "HedzUp" Event notification, which automatically calls out potential problems for the user.

Conclusion
Panorama provides the ability to monitor a wide variety of performance data across many tiers and applications. It also provides the ability to view this information in an easy-to-use browser-based UI. Most important, Panorama comes with a variety of statistical analysis tools that identify the impacts of this data across tiers and applications. It employs this intelligent analysis to identify and report significant events, data patterns, and abnormal behavior. This provides an enterprise-wide view into your entire system or suite of applications, instead of an application-by-application view that doesn't mirror how the enterprise really works.