• View
• Edit
• History
• Print

Software Testing - Part 2

A Software Engineering Lecture by Steven Choy

Lecture Overview:

In this lecture, we will continue to learn Software Testing. It covers System Testing, Regression Testing, and Documenting Testing. We will also have a brief touch on how JUnit is used to automate unit testing in Java.

While unit and integration testing focus on finding faults in individual components and the interfaces between the components, system testing ensures that the complete system complies with the functional and nonfunctional requirements. System testing includes functional testing, performance testing, pilot testing, acceptance testing, and installation testing.

Regression testing is to ensure that a program has not regressed; the functionality that was working yesterday is still working today. In other word, regression testing is to ensure that we do not introduce new bugs while removing existing bugs. A program still works correctly after changes were made to it.

Reading: Chapter 11 of the textbook Testing

System Testing

System Testing Overview

• System testing ensures the system, as a whole, complies with both functional and non-functional requirements
• Activities involves:
  • Functional Testing
  • Performance Testing
  • Pilot Testing
  • Acceptance Testing
  • Installation Testing

Functional Testing

• Also known as Requirement Testing
• As the name reveals, it checks if the system conforms with the functional requirement
• Test cases are identified from use case model and requirement specification

Performance Testing

• What are our goals?
  • Check if our system performance meets user's requirement
  • Find out the details of the system in terms of the following metrics:
    • Throughput (no. of request/sec)
    • CPU/Memory/IO/Network Utilization
    • Maximum number of client/request it can handle before breaking the system
    • Time for fail-over of systems
    • Security level (How vulnerable is our system for attack?)
    • etc
• Load Testing
  • Load the system with an expected number of data/request and check if the system can handle
• Stress Testing
  • Push the system beyond the limit to evaluate its robustness
  • Load the system with an unreasonable large number of request/data, trying to stress the system to the breaking point
• Timing Testing
  • Attempt to find behaviours that violate timing constraints described by the non-functional requirement
• Security Testing
  • Try to find out the security flaws of the system
• Recovery Testing
  • Evaluates the ability of the system to recover from erroneous state (e.g. hardware/network failure)

Performance Test Tools

• ApacheBench from Apache HTTP server
• Rational Performance Tester from IBM (Website)
• LoadRunner from Mercury (Website)
• QALoad from Compuware (Website)

Acceptance Testing

• Does your client accept the system? ⇒ Acceptance Testing
  • Acceptance testing is performed by clients, not developers
  • The client designs and performs series of tests to check if it conforms with the requirement
• Typically, majority of bugs are found during this stage (WHY?)
• Besides, after trying out the system, clients may request for change of requirement in this stage

Pilot Testing

• Pilot testing is "Let end-users really try out the system before its official release"

Software release life cycle

Does Testing end Here?

Regression Testing

• Object-oriented software development is an iterative process
• Fix one bug may introduce another bug somewhere in the code
• Adding/modifying a feature may also introduce side effect to the existing code
• A system that fails after the modification of a component is said to be regress
• Tests that try to produce such failure ⇒ Regression Test
• Regression testing is testing that a program has not regressed; the functionality that was working yesterday is still working today. In other word, regression testing is to ensure that we do not introduce new bugs while removing existing bugs. A program still works correctly after changes were made to it.
• So, what's the best way to ensure the modified component does not introduce side effect to the existing system?
  • Retest the whole system with all the test cases!
• Strategies to streamline the process:
  • Retest dependent components
  • Retest risky components
  • Retest frequently-used components

Automate Your Testing

• Unit/Integration Test Automation
  • JUnit – an open-sourced test framework for Java
  • Latest version is 4.0 but we only focus on v3.8
• User Interface Testing Automation
  • QARun from Compuware
  • QuickTest from Mercury
  • Rational Robot from IBM (website)
  • AutomatedQA

A Very Simple Example

    public Calculator() {
    }

    public int add(int num1, int num2) {
        return num1 + num2;
    }

    public int minus(int num1, int num2) {
        return num1 - num2;
    }

Unit Test in a manual way

• If you need to perform unit testing on that piece of code, you probably do it using "System.out"…

    public static void main(String[] args) {
        Calculator calculator = new Calculator();
        int result = calculator.add(100, 200);
        System.out.println("Result for (100+200) = " + result);
        result = calculator.minus(200, 100);
        System.out.println("Result for (200-100) = " + result);
    }

• Console Output

Automate Unit Testing by JUnit

• JUnit is developed to automate unit testing in Java
• A regression testing framework, written by Erich Gamma and Kent Beck

JUnit Example

    private Calculator calculator = null;

    public CalculatorTest(String name) {
        super(name);
    }

    protected void setUp() {
        calculator = new Calculator();
    }

    public static Test suite() {
        TestSuite suite = new TestSuite(CalculatorTest.class);
        return suite;
    }

    /** Test of add method, of class Calculator */
    public void testAdd()  {
        /* Test case #1: Adding two positive integers together */
        int result = calculator.add(100, 200);
        assertEquals(300, result);

        /* Test case #2: Adding two negative integers together */
        result = calculator.add(-100, -200);
        assertEquals(-300, result);
    }

    /** Test of minus method, of class Calculator */
    public void testMinus()  {
        /* Test case #1: Subtract two positive integers */
        int result = calculator.minus(200, 100);
        assertEquals(100, result);

        /* Test case #2: Subtract two negative integer */
        result = calculator.minus(-200, -100);
        assertEquals(-100, result);
    }

Unit Testing Quotes

• Unit tests can be tedious to write, but they save you time in the future by catching bugs after changes [William Wake]

JUnit References

• Download JUnit at:
  • http://www.junit.org
• Learn more about JUnit at:
  • http://junit.sourceforge.net/doc/testinfected/testing.htm

Test Documentation

• Test Plan
  • Describe what to test
  • A document describes what to be tested, the scope, approach, resources and schedule of the test
• Sample Test Plan Outline

    1) Test Plan Identifier
    2) References
    3) Introduction
    4) Test Items
    5) Software Risk Issues
    6) Features to be Tested
    7) Features not to be Tested
    8) Approach
    9) Item Pass/Fail Criteria
    10) Suspension Criteria and Resumption Requirements
    11) Test Deliverables
    12) Remaining Test Tasks
    13) Environmental Needs
    14) Staffing and Training Needs
    15) Responsibilities
    16) Schedule
    17) Planning Risks and Contingencies
    18) Approvals
    19) Glossary

• Test Specification
  • Describe how to test
  • Describe how each test case is run
• Sample Test Specification Outline

• Test Report
  • Present the test result

Extra Materials about Software Testing

Suggested Readings on JUnit:

• JUnit Test Infected: Programmers Love Writing Tests
• JUnit Reloaded (by Ralf Stuckert on 12/07/2006)
• Get Acquainted with the New Advanced Features of JUnit 4
• JUnit.org Resources for Test Driven Development

Some Tools for Performance Testing:

• apachebench gateway by codeflux.com
• IBM Rational Performance Tester
• Mercury LoadRunner (Load Testing Software - Automated Performance Testing and Web Testing Software)
• Compuware QALoad - Testing for application performance under heavy load

Some Tools for Testing Automation

• Compuware QARun - Functional testing for quality enterprise applications
• Mercury QuickTest Professional
• IBM Rational Robot
• AutomatedQA - Tools for Software Testing and Quality Assurance

Other useful resources:

• uTest - Software Testing Service

• Software QA Testing and Test Tool Resources

• Open source software testing tools, news and discussion

• Open source performance testing tools

• Selenium - from OpenQA

• The Grinder, a Java Load Testing Framework

• Apache JMeter

• The Best Unit Testing Resources: Beginner to Expert : Codeulate

• Sample Acceptance Test Plan

Thanks for Reading

If you would rather like to have this lecture note in printed format, please click the print action link in the top right corner.

If you find any problem in this lecture note, please feel free to tell Steven by steven@findaway.hk