my Siebel sandpit

Many Siebel projects use automated tools for performance and stress testing. Here are some hints and tips for conducting effective and meaningful performance tests.

Setting goals

Set clear goals for the performance testing activity. Is the aim to prove that the platform can support the intended number of users ? Or to see how many users the platform can support ? What is the expected response time for a given operation ?

Test scenarios

It is unlikely that all users will be using the application in exactly the same way. Different classes of users may use the application very differently. Try to simulate the main types of application usage. Carefully consider all real-life users of the application. Many Siebel applications are deployed in a central data centre to a global community. If there are some sales users in Eastern Europe with older desktop PC's and ISDN connections, include those in the test scenarios. If the application uses real-time feeds, data quality, Workflow, Assignment Manager or Actuate Reports, ensure these elements are incorporated into the testing activity.

Think time

The 'think-time' is the interval between the user clicking a button in the Siebel application. However, think-times can vary considerably. A application where data entry clerks are simply transcribing details from paper application forms may have a relatively short think-time. Conversely, the think-time for agents working in a call centre talking to customers on the phone is likely to be higher. Performance testing tools like LoadRunner are expensive and may be licensed by the number of Virtual Users. Do not be tempted to simulate 1,000 end users by purchasing 500 licenses and reducing the think-time. 500 users with 15 second think time does not equate to 1,000 users with a 30 second think-time.

Validate the tests

Walk through the test scenarios in single user mode and identify any obvious low hanging fruit before the formal load testing activity. Increase Siebel logging and use database monitoring tools to identify any long running SQL statements.

Caching

Siebel is an N-tier application with scope for caching at many levels. Ensure that most of the recommended configuration changes for optimal performance have been implemented. The Siebel Performance Tuning Guide contains many valuable recommendations. Most application vendors prefer to simulate normal usage where all caches are warm. Consequently, most test scripts are recorded twice and the second test capture is then replayed to ensure that the performance tests fully utilise caching. Consider whether a comparative 'cold' test is useful. Some sites stop and restart the Siebel Enterprise and database weekly which means on a Monday morning all caches are empty.

Ramp up

The actual login process is fairly expensive (OM initialisation and a database connection). A useful load test is simply to login all users at regular intervals until all users are logged in. This test will also ensure that various configuration parameters (MaxTasks for the Object Manager and database connections) are set appropriately.

Representative testing

Try to ensure the application usage is realistic and representative by using a full volume database for the testing activity and each simulated Siebel user has an appropriate position. Do not re-use virtual users during the test as this can skew the test results. For example, if the test creates an activity which is displayed on the home page, then the home page will grow and grow (and performance will degrade) after each test iteration. A useful test is to stress a single application server to determine how many users can be supported on a single physical server. This test also eliminates the complexity and overhead of clustering and/or load balancing from the architecture. The main output from performance testing is a report including a plethora of colourful charts and detailed metrics summarising the performance of each tier in minute detail. It is always useful to continue to monitor the application by maintaining a session and working normally to observe the responsiveness of the application.