Testing Cognos 8 BI Scalability

By Jennifer Hanniman, Product Marketing, Platform and Architecture, Cognos Inc., and Dean Browne, Product Manager, Platform, Cognos Inc.

The test results are in: Cognos 8 BI Version 8.2 demonstrates predictable scalability and mission-critical reliability required to enable customer success.

To ensure that Cognos solutions meet performance and scalability requirements for enterprise business intelligence, we regularly test our solutions at the IBM Innovation Center for Business Partners. In our most recent scalability test, conducted in March 2007 on an IBM System x platform, Cognos 8 BI Version 8.2 delivered the best Cognos 8 BI results to date!

Here are some of the test highlights:

Cognos 8 BI delivers proven, predictable scalability. Testing demonstrated that Cognos 8 BI scales linearly across the three principles of scalability testing - user, horizontal and proportional scalability.

Cognos 8 BI delivers mission-critical performance and scalability when deploying BI capabilities to a large number of users across the organization. Testing was conducted on up to 80,000 users performing tasks that varied from simple report delivery to complex analysis.

Cognos 8 BI delivers scalability on real-life, enterprise BI deployments. Cognos 8 BI achieved a scaling efficiency of 85%, which exceeded unsolicited customer requirements. Moreover, as users and hardware were proportionally increased, response times remained exceptionally stable, within a range of 2%. This means that as Cognos 8 BI scales across an enterprise, it delivers predictable and reliable performance.

Cognos 8.2

While at IBM Innovation Center, an apples-to-apples comparison of Cognos 8 BI Version 8.2 and Cognos 8 BI Version 8.1 was performed across the three principles of scalability. Those tests demonstrated Cognos 8 Version 8.2 improvements across all business use cases within a "Typical BI Day" scenario.

Two highlights from this test phase are:

• Cognos 8 BI Version 8.2 demonstrated an average performance improvement of approximately 90% over Cognos BI Version 8.1 for the mixed case scenario. This means that response times were nearly twice as fast as in the previous release.
• Cognos 8 BI Version 8.2 demonstrated an average performance improvement of 200% over Cognos 8 BI Version 8.1 when looking specifically at the analysis use case.

While it is important to remember that actual customer response times will vary, Cognos is committed to ensuring solution scalability. Cognos 8 BI Version 8.2 delivers improved quality and performance to meet your enterprise scalability requirements.

Testing the "Typical BI Day"

In order to address the needs of ever-growing enterprise BI deployments, a solution platform needs to address the three key principles of scalability – user, horizontal, and proportional scalability.

User scalability: Setting initial performance expectations when hardware is constant - given a constant configuration (e.g. eight CPUs) as you increase workload (users, reports, unit of work) a scalable product means there will be a proportional change in some performance metric, e.g. graph linearly increasing trend, until the system is saturated

Horizontal scalability: Meeting performance expectations using resources - given a constant workload (e.g. 100 concurrent users) as you increase capacity (e.g. eight to 16 CPUs) a scalable product means proportional decreases in response times and increases in throughput, e.g. 8 CPUs 100 concurrent users response time of 10 seconds, versus 16 CPUs 100 concurrent users response of approximately 5 seconds

Proportional scalability: Maintaining performance expectations as the user community grows - given a constant configuration/workload ratio (e.g. 10 concurrent users per CPU,) a scalable product means you can expect to maintain performance by scaling the hardware. For example, if there are 500 concurrent on 4 CPUs with a user base that is expected to grow to 1,000 users, performance can be maintained by doubling the hardware.

To narrow the performance test options, Cognos worked with customers, analysts, and professional services to represent a "Typical BI Day" - including simulating multiple users performing activities such as viewing reports, generating reports and performing analysis. The intent was not to test all of the permutations and combinations of hardware and users, simply to probe how Cognos 8 BI can perform and scale to meet your needs, in your IT environment, for your user communities, and in your enterprise deployments.

To accurately model typical deployments the test case reflected common BI requirements including:

• Scale for massive end user reach – BI deployments are expanding to embrace a broader, more casual end-user community including users that do not necessarily live and interact with BI everyday, yet require mission-critical information to be delivered to their fingertips. Therefore these tests focused on performance for delivering pre-generated content, and robust and predictable scalability to enable growing user communities.
• Access simple reporting on current information – As needs grow for operational insight and access, users want simple access to reports with the most up-to-date information. Therefore these tests focused around on-demand report execution against a mix of sources including dimensional-modeled relational and OLAP.
• Address complex and challenging reporting – Many businesses require interactive reporting to combine multiple sets of data in a single view, and provide users the ability to explore data through drilling, ranking and enhancing the content. Therefore these tests examined interactivity of prompted reports, drilling and ranking content and complexity of multi-query reports, WISIWIG layouts, and master-detail formats.
• Address the exploration and research needs of information experts – Most analysts aren’t satisfied with pre-formatted reporting, but prefer in-depth analysis and the ability to explore and traverse data to derive insight. Therefore these texts focused on exploration tasks in analysis of drilling, rotating, comparative analysis and addressing the needs of the most demanding analyst community.

Using the Results in Your Environment

Each deployment has a unique combination of factors that influence the performance and scalability of your BI solution, including your user community, solution requirements and infrastructure. The IBM Innovation Center scalability test results cannot be used to set performance expectations for your specific deployment, however the scalability tests do confirm that Cognos 8 BI performs predictably and meets expectations for large scale deployments. As you plan, deploy and manage your BI solutions, this means you can:

• Meet your performance SLA: Predictable performance allows you to effectively plan deployment hardware requirements to meet your performance SLAs upon implementation.
• Manage user expectations: Understanding BI performance helps you effectively communicate with users about performance considerations prior to implementation. An effective communication strategy ensures user expectations are set before deployment.
• Maximize your IT investment: Demonstrating proven BI performance levels helps your organization to plan and manage your IT infrastructure investment and eliminate last-minute purchases to handle unexpected performance results.