Testing with ScalaTest

We have talked before about about incremental testing of our functions. We will now create more systematic tests for our objects and methods.

To use ScalaTest, you must have installed it as explained in the installation instructions.

ScalaTest allows us to write test suites. A test suite is a class that contains an arbitrary number of tests, checking the correctness of our code.

A simple test suite

Let's start with a simple example (test1.scala):

import org.scalatest.FunSuite

class AdditionCheckSuite extends FunSuite {
  val two = 2
  val three = 3

  test("one plus one") {
    assert(1 + 1 == two)
    assert(1 + 1 != three)
  }

  test("one plus one with result") {
    assertResult(two) { 1 + 1 }
  }
}

We compile this class:

> fsc test1.scala 

If you now check the contents of the current directory, you'll find a new file AdditionCheckSuite.class there. We can run the test suite as follows (note that we are running the suite through a different object provided by the library:

> scala org.scalatest.run AdditionCheckSuite

The output shows that everything went alright, all tests were passed:

> scala org.scalatest.run AdditionCheckSuite
Run starting. Expected test count is: 2
AdditionCheckSuite:
- one plus one
- one plus one with result
Run completed in 163 milliseconds.
Total number of tests run: 2
Suites: completed 1, aborted 0
Tests: succeeded 2, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

To demonstrate what it looks like when tests fail, I will change the definition of two and three in my test (test2.scala):

import org.scalatest.FunSuite

class AdditionCheckSuite extends FunSuite {
  val two = 3
  val three = 2

  test("one plus one") {
    assert(1 + 1 == two)
    assert(1 + 1 != three)
  }

  test("one plus one with result") {
    assertResult(two) { 1 + 1 }
  }
}

Again I compile and run the test:

> fsc test2.scala 
> scala org.scalatest.run AdditionCheckSuite
Run starting. Expected test count is: 2
AdditionCheckSuite:
- one plus one *** FAILED ***
  2 did not equal 3 (test2.scala:8)
- one plus one with result *** FAILED ***
  Expected 3, but got 2 (test2.scala:13)
Run completed in 183 milliseconds.
Total number of tests run: 2
Suites: completed 1, aborted 0
Tests: succeeded 0, failed 2, canceled 0, ignored 0, pending 0
*** 2 TESTS FAILED ***

You see that the first and second test check exactly the same behavior. Using assertResult can be clearer when you have a computation and want to check that the result matches what you expected.

Checking that exceptions are thrown

To check that our code correctly handles input where we need to throw an exception, we need to write tests where we indicate that an exception is actually expected. This is done using the intercept method (test3.scala):

import org.scalatest.FunSuite

class AdditionCheckSuite extends FunSuite {
  val two = 2
  val three = 3

  test("one plus one") {
    assert(1 + 1 == two)
    assert(1 + 1 != three)
  }

  test("one plus one with result") {
    assertResult(two) { 1 + 1 }
  }

  test("division by zero") {
    intercept[ArithmeticException] {
      1 / 0
    }
  }
}

Here is the output:

> fsc test3.scala 
> scala org.scalatest.run AdditionCheckSuite
Run starting. Expected test count is: 3
AdditionCheckSuite:
- one plus one
- one plus one with result
- division by zero
Run completed in 168 milliseconds.
Total number of tests run: 3
Suites: completed 1, aborted 0
Tests: succeeded 3, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

If we change 1/0 to 1/1, so that no exception will be thrown, the test suite correctly detects the error:

> scala org.scalatest.run AdditionCheckSuite
Run starting. Expected test count is: 3
AdditionCheckSuite:
- one plus one
- one plus one with result
- division by zero *** FAILED ***
  Expected exception java.lang.ArithmeticException to be thrown, but no exception was thrown. (test3.scala:17)
Run completed in 173 milliseconds.
Total number of tests run: 3
Suites: completed 1, aborted 0
Tests: succeeded 2, failed 1, canceled 0, ignored 0, pending 0
*** 1 TEST FAILED ***

Testing our own classes

For the polynomial calculator project, I have written a class Polynomial implemented in the source file polynomial.scala.

Let's now write a test suite for the Polynomial class. This is much better than testing it interactively, because when a test fails, you can modify your code and run all tests again. If you ever need to add features to your classes or change them in any way, it is very useful to have access to all the tests you used previously: You can run them again to make sure that you have not broken anything.

It is a good habit to create test suites for all interesting classes that your write. You can use the test suite to have confidence that the class you implemented is correct. And even more useful, it gives you confidence that when you make a change, you do not break some part of the class by mistake.

Here is a test suite for my Polynomial class. Note that it simply uses the Polynomial class for its tests (checkpoly.scala):

import org.scalatest.FunSuite
import scala.language.implicitConversions

class PolynomialCheckSuite extends FunSuite {
  implicit def intToPolynomial(a: Int): Polynomial = {
    new Polynomial(Array(a))
  }
  
  test("creating polynomials") {
    val p1 = new Polynomial(Array(3))
    assert(p1.degree == 0)
    assert(p1.toString == "3")
    val p2 = new Polynomial(Array(-1, 3, -4, 0, -6))
    assert(p2.degree == 4)
    assert(p2.toString == "-6 * X^4 - 4 * X^2 + 3 * X - 1")
    val p3 = new Polynomial(Array(0, 0, 1))
    assert(p3.degree == 2)
    assert(p3.toString == "X^2")

    val p0 = new Polynomial(Array(0))
    assert(p0.degree == -1)
  }

  test("addition and subtraction") {
    val p1 = new Polynomial(Array(3))
    val p2 = new Polynomial(Array(-1, 3, -4, 0, -6))
    val p3 = new Polynomial(Array(5, 0, 4, 0, -6))

    val q1 = p1 + p2
    val q2 = p2 - p3
    assert(q1.toString == "-6 * X^4 - 4 * X^2 + 3 * X + 2")
    assert(q2.degree == 2)
    assert(q2.toString == "-8 * X^2 + 3 * X - 6")
  }

  test("multiplication and power") {
    val p1 = new Polynomial(Array(3))
    val p2 = new Polynomial(Array(-1, 3, -4, 0, -6))
    val p3 = new Polynomial(Array(0, 0, 5))
    val p4 = new Polynomial(Array(2, -4, 6, 8))

    val q1 = p1 * p2
    val q2 = p1^5
    val q3 = p3^5
    val q4 = p2 * p3
    val q5 = p2 * p4
    assert(q1.toString == "-18 * X^4 - 12 * X^2 + 9 * X - 3")
    assert(q2.degree == 0)
    assert(q2.coeff(0) == 3*3*3*3*3)
    assert(q3.degree == 10)
    assert(q3.toString == "3125 * X^10")
    assert(q4.degree == 6)
    assert(q4.toString == "-30 * X^6 - 20 * X^4 + 15 * X^3 - 5 * X^2")
    assert(q5.toString == "-48 * X^7 - 36 * X^6 - 8 * X^5 - 12 * X^4 + 26 * X^3 - 26 * X^2 + 10 * X - 2")
  }

  test("creating polynomials using X") {
    val X = new Polynomial(Array(0, 1))
    assert(X.toString == "X")
    val p4 = -1 * (X^5) + 3 * (X^3) - (X^2) + 5
    assert(p4.toString == "-X^5 + 3 * X^3 - X^2 + 5")
    val p5 = (X - 1) * (X - 3) * (X + 5)^2
    assert(p5.toString == "X^6 + 2 * X^5 - 33 * X^4 - 4 * X^3 + 319 * X^2 - 510 * X + 225")
  }

  test("evaluation") {
    val X = new Polynomial(Array(0, 1))
    val p1 = new Polynomial(Array(3))
    val p2 = new Polynomial(Array(-1, 3, -4, 0, -6))
    val p3 = new Polynomial(Array(0, 0, 1))
    val p4 = -1 * (X^5) + 3 * (X^3) - (X^2) + 5
    val p5 = (X - 1) * (X - 3) * (X + 5)^2

    assertResult(3.0) { p1(5) }
    assertResult(-1.0) { p2(0) }
    assertResult(4.0) { p3(2) }
    assertResult(0.0) { p5(-5.0) } 
  }
}

To run the test suite, we of course first have to compile the Polynomial class, then the PolynomialCheckSuite class, and finally we can run the test suite:

> fsc polynomial.scala 
> fsc checkpoly.scala 
> scala org.scalatest.run PolynomialCheckSuite
Run starting. Expected test count is: 6
PolynomialCheckSuite:
- creating polynomials
- addition and subtraction
- multiplication and power
- creating polynomials using X
- evaluation
- derivatives
Run completed in 172 milliseconds.
Total number of tests run: 6
Suites: completed 1, aborted 0
Tests: succeeded 6, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

When you work on the project and make changes to the Polynomial class and companion object, you can run the test suite after each change to make sure everything is still working. (You should also add new tests to the test suite for the new methods that you create.)

Starting with the test suite

We can take this one step further. We can think about a test suite as an executable specification of the correct behavior of the class that we want to create. So instead of writing code first and later testing it using the test suite, we first write down a list of tests that our code must satisfy to be correct. Then we start implementing the methods of the object one by one. After each change, a few more tests will pass correctly. When all tests are passed, we are done with the implementation.

Let's go back to our Polynomial object and assume that it is not already given to us: We should implement it from scratch.

This time we start with the test suite checkpoly.scala. Here we immediately encounter a problem: Since PolynomialCheckSuite uses the Polynomial object, it is impossible to compile checkpoly.scala without a Polynomial class.

So what we will do is to create an empty framework for the Polynomial class that shows all the methods and their types. Using this framework, we can not only compile the test suite, but we now also have a full documentation of the methods that we plan to implement.

So the first version of my Polynomial class looks like this (polynomial1.scala):

class Polynomial(coeffs: Array[Int]) {

  def degree: Int = ???

  def coeff(i: Int): Int = ???
  
  override def toString: String = ???
  
  def + (rhs: Polynomial): Polynomial = ???
  
  def - (rhs: Polynomial): Polynomial = ???
  
  def * (rhs: Polynomial): Polynomial = ???

  def ^ (ex: Int): Polynomial = ???

  def apply(x: Double): Double = ???
}

I've defined all the public methods of the class that I plan to implement, including the types of arguments and the result type. Each method is defined to return ???. This code can already be compiled:

> fsc polynomial1.scala 

And so now we can already run our test suite:

> fsc checkpoly.scala 
> scala org.scalatest.run PolynomialCheckSuite
Run starting. Expected test count is: 6
PolynomialCheckSuite:
- creating polynomials *** FAILED ***
  scala.NotImplementedError: an implementation is missing
 ... and many more failed tests

Of course all tests will fail, because our class does not do anything yet. In particular, if you try to evaluate ???, it raises a NotImplementedError. (??? is a special function with result type Nothing. Since Nothing is a subtype of every Scala type, it is legal to write ??? instead of any Scala type. However, there is no object of type Nothing, and so the function must throw an exception.)

Now we are set up to build our Polynomial class. I start by adding a constructor (polynomial2.scala):

class Polynomial(coeffs: Array[Int]) {
  private val A = createCoeffs(coeffs)
  
  private def createCoeffs(A: Array[Int]): Array[Int] = {
    var s = A.length - 1
    while (s >= 0 && A(s) == 0)
      s -= 1
    A take (s+1)
  }
  
  def degree: Int = A.length - 1

  def coeff(i: Int): Int = if (i < A.length) A(i) else 0
  
  override def toString: String = {
    var s = new StringBuilder
    var plus = ""
    var minus = "-"
    for (i <- degree to 0 by -1) {
      if (coeff(i) != 0) {
	var c = coeff(i)
	s ++= (if (c > 0) plus else minus)
	plus = " + "; minus = " - "
	c = c.abs
	if (i == 0)
	  s ++= c.toString
	else {
	  if (c != 1)
	    s ++= c.toString + " * "
	  if (i > 1)
	    s ++= "X^" + i.toString
	  else 
	    s ++= "X"
	}
      }
    }
    s.toString
  }
  
  def + (rhs: Polynomial): Polynomial = ???
  
  def - (rhs: Polynomial): Polynomial = ???
  
  def * (rhs: Polynomial): Polynomial = ???

  def ^ (ex: Int): Polynomial = ???

  def apply(x: Double): Double = ???
}

We compile the Polynomial class and run the test suite again:

> fsc polynomial2.scala 
> scala org.scalatest.run PolynomialCheckSuite
Run starting. Expected test count is: 6
PolynomialCheckSuite:
- creating polynomials
- addition and subtraction *** FAILED ***
  scala.NotImplementedError: an implementation is missing
  ...
Run completed in 170 milliseconds.
Total number of tests run: 6
Suites: completed 1, aborted 0
Tests: succeeded 1, failed 5, canceled 0, ignored 0, pending 0
*** 5 TESTS FAILED ***

The first test has now passed: We can correctly construct Polynomial objects. There are only five failed tests left. I can continue to implement missing methods, running the test suite after each update of my code. Bit by bit I construct a correct and fully implemented class.

Note that we do not need to compile the test suite again: It is sufficient to compile it using the empty Polynomial framework at the beginning! (Of course you may discover later that you want to add some more tests, in which case you will have to compile checkpoly.scala again.)