Scalability Test Strategy

Description of the approach we plan to take in testing Mifos® to ensure it can scale to 1 million clients.

Objective

MFIs that have deployed Mifos are growing rapidly, and we need to verify our current architecture  and software design can accommodate that growth. For example, by March 2010, Grameen Koota expects to be at over 500,000 clients, and over 1 million clients in 2011. The goal of this document is to outline the approach for testing Mifos in a controlled test environment and use those results to resolve issues blocking customers.

The tasks in this document will span more than one release.  Tasks slated for specific releases will be noted below.

Approach

For scalability testing, we will test Mifos 1.4.x and the latest "trunk" versions in a performance lab.  For flexibility and access, we will target using a 'cloud' environment like Amazon EC2.

Environment

We want to locate the performance test environment in the cloud because:

  • It allows many Mifos contributors from around the world to help build and run the tests, and easily analyze results.
  • It allows us to easily scale the test load by adding new servers.
  • It is cost-effective, since we don't have to pay for the lab when we are not using it.

Tools

Scalability testing against the UI will be done with JMeter, an open source performance testing tool.  The reason we want to use open source tools are:

  • We want to scale the test load in a cost effective manner.
  • We want to build a system that anyone can use and contribute to.
  • We would like to be an example of good performance testing practices.

We will test the application at the UI level (except batch jobs), driving the application via test automation. Several measurements will be recorded to compare the relative scalability of Mifos at 300,000 of customers, 550,000 number of customers, and eventually over 1 million customers.

Scalability work will be broken up into specific tasks or "stories":

Scalability Testing Stories

  1. Interview consortium MFIs for confirmation of current Mifos bottlenecks.
  2. Identify and setup test tools, test environment.
  3. Prioritize test scenarios and estimate tasks.
  4. Document scalability test scenarios and concurrent usage pattern. Planned concurrent usage should be average of best and worst case usage.
  5. Create test scripts for each performance scenario to be used for test execution.
  6. Run a specific "smoke" tests to compare response of application with existing GK data size and 550,000 client data set.
  7. Execute each of the test iterations mentioned below.

Assumptions and Open Questions

Assumptions:

  • Testing will focus on mission critical areas of Mifos, not entire application.
  • Focus of testing is to ensure scalability with larger database. Not focusing on network speeds, hardware variations, or software variations.  Some of these other variations may be recommended to the MFI, but the focus of the testing will be to measure changes to the software or software configuration settings.
  • Scalability testing tools will be open source.
  • Tests will be portable - able to run in a local lab environment or in a hosted environment like Amazon EC2.
  • Test data sets and performance scripts will be written by SunGard.

Open Questions:

  1. Do we test with Windows platform (current GK environment), Linux, or both?
  2. What is the specific release criteria for each test?  Response time within x% of the previous execution?
  3. How many concurrent users is appropriate for test iterations?

Test iterations

Gazelle "D" timeframe

  1. Run tests with current GK database size - single user
  2. Run tests with current GK database size - tbd concurrent users
  3. Run tests with 2X GK database size (approx. 550,000 clients)- single user
  4. Run tests with 2X GK database size (approx. 550,000 clients)- tbd concurrent users
  5. Rerun tests with 2X database with additional monitoring tools like slow query logging.
  6. Rerun tests with 2X database after changes are made to application and or database to measure improvements - single user.
  7. Rerun tests with 2X database after changes are made to application and or database to measure improvements - tbd concurrent users.
  8. Rerun tests with 2X database after changes are made to optimize Tomcat, MySQL settings. - tbd concurrent users.

   Note: In case of batch jobs, concurrent vs. single user does not apply.

Future Projects

  1. Once 2X tests have passed criteria, run tests with 4X GK database size (approx. 1.1 million clients)- single user
  2. Once 2X tests have passed criteria, run tests with 4X GK database size (approx. 1.1 million clients)- tbd concurrent users

 

 Measurements

  1. Measure response time for each defined scalability test scenario. Start/stop times to be defined for each scenario.
  2. Monitor database activity - slow queries.
  3. Measure size of tables, disk usage space on database system.
  4. Measure CPU usage, memory utilization
  5. For batch jobs, overall execution time and average execution time per record for identified batch jobs.

Test Environment

Tests will be developed and excuted locally initially, but for ongoing scalability testing, we will utilize the Amazon EC2 cloud to host application server, database server, and simulated user load.

We will create EC2 images that contain the necessary environment, tools, and data to execute a test run.

Test results will be stored on a stable location since the EC2 test systems will be shutdown after test execution.

Test environment needs to be archived and documented to allow comparison testing to occur with future Mifos releases.

Costs

  1. Additional engineering effort to learn new tools, create, and execute scalability testing
  2. Staff time to coordinate and plan test scenarios based on MFI interviews.
  3. Amazon EC2 time
  4. Staff time for analysis of test results
  5. Staff development time to respond to inquires, experiment with possible solutions, and make code improvements.

Tools

  • Mifos platform
    • Java 6 JRE
    • Apache Tomcat
    • MySQL 5.1
  • Load testing tool - JMeter
  • Monitoring tools?
  • Reporting collection tool?

Release Criteria

Test results will document pass/fail result when evaluated against each test scenario's acceptance criteria.  As part of test planning, we will need to define the acceptance criteria for each scalability test scenario.  For example, "no more than 10% degradation in processing rate (meetings generated per client) for Generate Meeting Schedule batch job when 555,000 clients are present in database". 

last modified 2010-02-05 14:41