In this Quick Start Guide, you will lean how to use PREMIER to create performance test cases. At this moment, PREMIER is only available from the Subversion server. Once a binary distribution is available, this page will be updated. Also, PREMIER is still in alpha version, so expect changes in PREMIER, too. If you have comments or feature requests, please post them in the project page.

Although you can create performance test cases with solely PREMIER, it is recommended to use an automated test framework with PREMIER to automate the test process. JUnit is used in this quick start guide. You may choose your favorite test framework.

• Terms Used in PREMIER
• A Simple Performance Test Case
• Adding Some Random Factors
• Composing Different Requests

Terms Used in PREMIER

Before we get into any thing about PREMIER, let's see the terms that are used here (and throughout this web site):

• Job: A job is a scenario that needs to be done by the software.
• Request: A request is issued from outside of the software system. A user sends a request to the software system, and then the software system performs the job.
• Task: A task defines the frequency and request pattern for a job.
• Task Composite: A task composite is, as the name implies, a composition of multiple tasks.

Take an online shopping website for example. A single credit card transaction is a job. A task can be "credit card transactions according to a poisson process with a mean request interval of 2 seconds." A task composite can include the credit card transaction task among others.

A Simple Performance Test Case

In the simplest test case, you will need to use the following classes/interfaces:

• IJobFactory: You will need to provide a job factory class that implements the IJobFactory interface. The job factory can create instances of Runnable objects. These Runnable objects (or jobs in this famework) are the scenarios that will be executed in the test case.
• ITaskListener: A task listener tracks the events such as the start and end of a job, and the end of a task.
• Task and TaskComposite: These classes represent the conceptual task and task composite.

First let's create a job factory. The simplest job factory only needs to provide two information: the name of this type of job and the definition of the scenario. Here is an example:

class SearchCatalogScenario implements IJobFactory {
    public String getJobName() {
        return "Search Catalog";
    }

    public Runnable createNewJob() {
        return new Runnable() {
            public void run() {
                IPage page = WebFramework.createPage("search");
                IForm form = page.getForm("search");
                form.setInput("pid", "A12100123");
                IPage result = form.submit();
            }
        };
    }
}

We can now use this job factory to write test cases. Please note that a "virtual web framework" is used here. In this example, if we want to make sure that a single search request is finished within 0.5 second, we can do this:

public void testSearchCatalogSpeed() {
    DescStatisticsListener listener = new DescStatisticsListener();    1
    Task task = new Task();    2
    task.setInterval(new Once());    3
    task.addListener(listener);
    task.setJobFactory(new SearchCatalogScenario());    4

    TaskComposite composite = new TaskComposite("example");  5
    composite.setTimeLength(10000);  6
    composite.setCompositeTotal(200);  6
    composite.addTask(task);
    composite.start();  7
    listener.waitTillComplete();  8
    assertTrue(listener.getAverage() < 500);  9
}

There you go. A performance test case which tests if a scenario takes more time than expected. This example might seem more complicated than other similar test frameworks, but as you will see, we can modify this test case to perform much more complicated performance testing with little change.

Adding Some Variable Factors

We may run the "once test" several times, and perform statistics analysis. The problem with this approach is that the results from different test runs will be very similar, if the environment configuration is similar. This might be OK, if the software is a single-user system. However, in a multiple-user system, we need to see how the system behaves with different request patterns (or workloads). Under such situations, introducing random factors to the request patterns is important. This is because how the requests come in the system is mostly unpredictable. Using a deterministic workload for performance testing is unrealistic in most cases.

PREMIER includes a Poisson process generator. We can use the random process to test the search catalog scenario. For example, we can use the following test case to show that, during the 10-second test time, 200 search catalog requests come in the system, and 95% of them are completed within 0.5 second:

public void testSearchCatalogSpeed() {
    DescStatisticsListener listener = new DescStatisticsListener();
    Task task = new Task();
    task.setInterval(new ExpInterval());    1
    task.addListener(listener);
    task.setJobFactory(new SearchCatalogScenario());

    TaskComposite composite = new TaskComposite("example");
    composite.setTimeLength(10000);
    composite.setCompositeTotal(200);
    composite.addTask(task);
    composite.start();
    listener.waitTillComplete();
    assertTrue(listener.getSuccessRate(500) > 0.95);  2
}

From the perspective of performance modelling (for example, queueing network modelling), the time required to finish a job is more important than the time a request is completed. However, most if not all servers perform multiple jobs simulteneously. This makes modelling the server behavior very difficult. Compare to precise server modelling, measuring the time required to finish a request is much easier. By changing only two lines, this test case brings more information than the "run once" test case.

Composing Different Requests

We can make the test case more realistic by composing different types of requests. For example, in addition to "search catalog," the most requested feature in the website might be "check out", "log in", and "search related item." If we know that "search catalog" accounts for 30% of the incoming requests, we can use the following test case to show that 95% of the "search catalog" requests finish within 0.5 second:

public void testSearchCatalogSpeed() {
    DescStatisticsListener listener = new DescStatisticsListener();
    Task task1 = new Task();
    task1.setInterval(new ExpInterval());
    task1.addListener(listener);
    task1.setPortion(0.3);  1
    task1.setJobFactory(new SearchCatalogScenario());

    Task task2 = new Task();
    task2.setInterval(new ExpInterval());
    task2.setJobFactory(new CheckOutScenario());

    Task task3 = new Task();
    task3.setInterval(new ExpInterval());
    task3.setJobFactory(new LogInScenario());

    Task task4 = new Task();
    task4.setInterval(new ExpInterval());
    task4.setJobFactory(new SearchRelatedItemScenario());

    TaskComposite composite = new TaskComposite("example");
    composite.setTimeLength(10000);
    composite.setCompositeTotal(200);
    composite.addTask(task1);
    composite.addTask(task2);
    composite.addTask(task3);
    composite.addTask(task4);
    composite.start();
    listener.waitTillComplete();
    assertTrue(listener.getSuccessRate(500) > 0.95);
}

In this test case, different tasks are specified. PREMIER will compose different requests according to the distributions. This test case demonstrates the performance of the scenario of interest under a more realistic work load.