The Inquisitive Developer

Learn With Me: Elixir - Unit Testing (#65)

Kevin Peter -

I'm assuming that you already have some knowledge of what unit tests are. If not, check out these articles on Software Testing Fundamentals, Wikipedia, and Stackoverflow.

I'm not going into too much depth on the benefits of unit tests, but I highly recommend using them. I've written unit tests in C# (NUnit) and Javascript (Jest) in the past, and although they take up significant time to write initially, I'm always glad I've written them.

Why Unit Tests Are Worth Doing

For those of you who haven't done much unit testing, I'll just briefly list a few reasons why I think they're worth it.

First of all, unit tests can uncover bugs in your code, and force you to think about the different scenarios under which the function can be called and the different sets of input that can be passed to it.

Even just thinking about it can improve the quality of your code. Is a particular set of input legitimate or should it be disallowed by the function preconditions? For example, a function that works just fine with positive numbers could fail with negative numbers, numbers that are close to 0, or very large numbers due to overflow. Those are all different tests that can be run to verify different types of inputs.

Secondly, unit test are actually most valuable in finding regressions where the code previously ran just fine, but a modification caused it to break. Rather than not realizing there's a problem until you see errors in production, your unit tests will catch that right away. When you do miss a scenario in your unit tests that results in a production error, that's the perfect opportunity to add that scenario to your unit tests, verify that it fails, and then fix the problem and watch the test pass.

Unit tests also provide a quick feedback loop when you make changes, giving you more confidence that your code changes haven't accidentally broken anything.

Unit Testing in Elixir

C# has NUnit and a couple other unit testing frameworks, Javascript has about 20 unit testing frameworks (which seemingly continues to increase on a monthly basis), and Elixir has ExUnit. ExUnit is Elixir's official unit testing framework, and it's the only one I've ever seen used. While not part of the core Elixir language, it is part of the suite of Elixir tools and frameworks that is shipped with Elixir, so you don't have to install anything additional to use it. ExUnit is maintained by the Elixir team, so it will likely be the unit testing framework of choice for a long time to come. After having evaluated a lot of Javascript unit testing frameworks to try to determine which one to use, having a single obvious choice is really nice.

Writing Unit Tests with ExUnit

One thing I've noticed is that unit test frameworks are all pretty similar. Sure, there are some diffences between them, but once you've learned one of them, you'll be able to pick up other ones pretty quickly. Most of the concepts and tools are familiar. This is no different in ExUnit. You'll need to learn the syntax and the feature details, but the concepts are pretty much the same as any other unit testing framework I've ever seen. There won't be many surprises for you here.

So let's create a project for unit testing. I ran mix new unit_test_app to create a new project.

You can either follow along or view the complete project in the the "lwm 65 - Unit Testing" folder in the code examples in the Learn With Me: Elixir repository on Github.

Now that we have a project, I removed "lib/unit_test_app.ex" and replaced it with "lib/math_module.ex". This is the code that we are going to unit test. Here's what it looks like.

defmodule MathModule
  #Adds two numbers
  def add(x, y), do: x + y
  #Subtracts two numbers
  def subtract(x, y), do: x - y
end

Yeah, this code is clearly not anything to get excited about. We have a simple addition function and a simple subtraction function. I kept it simple so that we can focus on the tests.

In the "test" directory, I removed "unit_test_app_test.exs" and replaced it with "math_module_test.exs". Here are the unit tests I wrote.

defmodule MathModuleTest
  use ExUnit.Case

  import MathModule, only: [add: 2, subtract: 2]

  test "Add two positive numbers" do
    assert add(102, 45) == 147
  end

  test "Add a positive and a negative number" do
    assert add(4, -4) == 0
  end

  test "Add two negative numbers" do
    assert add(-6, -8) == -14
  end

  test "Subtract two positive numbers to get a positive result" do
    assert subtract(20, 18) == 2
  end

  test "Subtract two positive numbers to get a zero result" do
    assert subtract(31, 31) == 0
  end

  test "Subtract two positive numbers to get a negative result" do
    assert subtract(14, 21) == 7
  end

  test "Subtract a negative from a positive number" do
    assert subtract(4, -4) == 8
  end

  test "Subtract a positive from a negative number" do
    assert subtract(-6, 8) == -14
  end

  test "Subtract two negative numbers" do
    assert subtract(-6, -8) == 2
  end

  test "Subtract a zero from a positive number" do
    assert subtract(6, 0) == 6
  end

  test "Subtract a zero from a negative number" do
    assert subtract(-3, 0) == -3
  end

  test "Subtract a zero from a zero" do
    assert subtract(0, 0) == 0
  end
end

Let's go over this code so that you can understand what's going on here. All the tests are defined in a test module. The use ExUnit.Case statement will do the magic of importing and initializing the ExUnit framework. ExUnit.Case also has a useful option called "async", which will run the tests concurrently. I've left that off so the test are run serially (one after another).

The default is to run the tests serially, but you can run them parallel by specifying use ExUnit.Case, async: true. Ideally, your unit tests are completely independent of each other and can be run in any order or even concurrently. If that applies to your unit tests, then I recommend enabling the "async" option. It will run your unit tests faster and it will help expose any unit tests with interdependencies.

Once I've imported the unit test framework, I then import the functions I want to test from MathModule module into the current namespace. Importing isn't required, but it saves typing, since I don't have to write MathModule. in front of all the function calls. Here I specifically import only the functions I want to test, but I could also just import all the functions by leaving off the only: option.

Each test is created using a test macro. Each test has a string associated with it that describes what the test is doing. This string is used for display purposes, so you'll have an idea of which unit test is being referred to when the testing tool needs to output something regarding particular test (such as a failure message). The code for each unit test goes between the do and the end keywords.

Most of the testing code will make use of the assert macro. You give an expression to the assert macro. If the expression evaluates to a truthy value, then the test continues to run, and if it evaluates to a falsey value, it fails. If there were no assertion failures during the test, then the test is considered to be successful.

I tried to anticipate every class of inputs in which a mistake may be made. There's not much point in multiple tests where we subtract one positive number from another to achieve a positive result. If it works for one set of similar numbers, it's highly likely to work for any other set of numbers. However, a similar test where we subtract to get a negative result could would be useful because it could be that the code doesn't handle that scenario as it should. That's a different scenario that could uncover a defect.

If the input size is small enough, you may have a test that generates every possible input and runs a test for each set of input. This can be useful in some cases, but I wouldn't recommend this approach if it would take a while to run and there isn't any obvious benefit to doing so. A related testing technique called fuzzing involves generating weird and unique input sets to pass to some functionality to attempt to trigger some corner-case error. This is a good way of testing something that is complex and can have a wide variety of inputs. That's had some success in finding defects in web browsers, where the input can be any HTML document with a nearly infinite variety of possibilities. That's a whole topic on its own that I don't plan on diving into here.

The assert macro will be used the majority of the time, but the ExUnit.Assertions module has some more macros that you can use. You can assert that a particular error was thrown via assert_raise, you can assert that some value is close enough to another value to be considered successful if you don't require an exact match (3.99 vs 4.0 for example) using assert_in_delta, or you can use refute, which is the opposite of assert in that it succeeds when the result is falsey, and fails when the result is truthy.

Let's run the tests for our project with mix test.

> mix test
Compiling 1 file (.ex)
Generated unit_test_app app
...

  1) test Subtract two positive numbers to get a negative result (MathModuleTest)
     test/math_module_test.exs:26
     Assertion with == failed
     code:  assert subtract(14, 21) == 7
     left:  -7
     right: 7
     stacktrace:
       test/math_module_test.exs:27: (test)

........

Finished in 0.04 seconds
12 tests, 1 failure

Randomized with seed 40000

Oops, one of the unit tests failed. You can see which unit test failed, which assertion failed, what was expected, what the result was, and a stacktrace with a file and line number. This is really nice output. In my experience, test frameworks don't usually print so much helpful information. I usually have to do some investigating to find out exactly what is failing and why. So far, ExUnit is making a good impression.

When a unit test fails, there's two reasons. The code has a defect or the test has a defect. In this case, the defect is in the test. 14 - 21 should be -7, not 7, which is what the test is expecting. I'm going to fix that issue and run it again. By the way, this actually was a genuine mistake on my part. I would have ended up creating a deliberate mistake for the purposes of showing what happens when a test fails, but I ended up making a genuine mistake when writing the tests.

Now that I've fixed the problem, here's what it looks like when I run the unit tests.

> mix test
............

Finished in 0.04 seconds
12 tests, 0 failures

Randomized with seed 207000

Yay! Now all my tests are passing, and I can have a fair degree of confidence that my functions are working as intended.

When tests are running, the current environment is set to :test. So you see references to :test in any mix configurations, be aware that these configurations would only apply when tests are being run and not in the production (or dev) environment.

Grouping Unit Tests

Let's go back to our unit testing example. We can group these tests using the describe macro, which will help organize the tests, both in the code and in the test output.

defmodule MathModuleTest
  use ExUnit.Case, async: true

  import MathModule, only: [add: 2, subtract: 2]

  describe "Testing the add function" do
    test "Add two positive numbers" do
      assert add(102, 45) == 147
    end
    test "Add a positive and a negative number" do
      assert add(4, -4) == 0
    end
    test "Add two negative numbers" do
      assert add(-6, -8) == -14
    end
  end

  describe "Testing the subtract function" do
    test "Subtract two positive numbers to get a positive result" do
  	  assert subtract(20, 18) == 2
    end
    test "Subtract two positive numbers to get a zero result" do
  	  assert subtract(31, 31) == 0
    end
    test "Subtract two positive numbers to get a negative result" do
  	  assert subtract(14, 21) == -7
    end
    test "Subtract a negative from a positive number" do
  	  assert subtract(4, -4) == 8
    end
    test "Subtract a positive from a negative number" do
  	  assert subtract(-6, 8) == -14
    end
    test "Subtract two negative numbers" do
  	  assert subtract(-6, -8) == 2
    end
    test "Subtract a zero from a positive number" do
  	  assert subtract(6, 0) == 6
    end
    test "Subtract a zero from a negative number" do
  	  assert subtract(-3, 0) == -3
    end
    test "Subtract a zero from a zero" do
  	  assert subtract(0, 0) == 0
    end
  end
end

I grouped the tests together into one group for addition function tests and one group for subtraction function tests. Nesting describe macros is not possible, so Elixir developers typically group tests by the function they are testing.

Note that the "async" option is enabled for the above unit tests, so they run independently.

This time I inserted a deliberate flaw in the addition function.

def add(x, y), do: x * y

Let's run the unit tests again and see what failure looks like when tests are grouped.

> mix test
Compiling 1 file (.ex)
Generated unit_test_app app
.

  1) test Testing the add function Add two negative numbers (MathModuleTest)
     test/math_module_test.exs:15
     Assertion with == failed
     code:  assert add(-6, -8) == -14
     left:  48
     right: -14
     stacktrace:
       test/math_module_test.exs:16: (test)

..

  2) test Testing the add function Add a positive and a negative number (MathModuleTest)
     test/math_module_test.exs:11
     Assertion with == failed
     code:  assert add(4, -4) == 0
     left:  -16
     right: 0
     stacktrace:
       test/math_module_test.exs:12: (test)

.

  3) test Testing the add function Add two positive numbers (MathModuleTest)
     test/math_module_test.exs:7
     Assertion with == failed
     code:  assert add(102, 45) == 147
     left:  4590
     right: 147
     stacktrace:
       test/math_module_test.exs:8: (test)

.....

Finished in 0.04 seconds
12 tests, 3 failures

Randomized with seed 212000

It looks like the group name and the test name are just concatenated and displayed on one line.

Now let's fix the addition bug and rerun the tests.

> mix test
Compiling 1 file (.ex)
............

Finished in 0.06 seconds
12 tests, 0 failures

Randomized with seed 345000

It looks just like it did before we grouped the tests. I noticed that the tests had no problem running concurrently. Each test is completely independent of the other tests.

Setting Up Unit Tests

If you use a setup macro in your test module, it will be run prior to each test being run. It is intended to return a set of test data to shared amongst multiple unit tests. The test data must be either a keyword list or a map, or it can be a tuple whose first value is :ok and the second value is a keyword list or a map. I would personally just use return the keyword list or map directly.

Here's an example of using a setup macro.

defmodule TestModule
  use ExUnit.Case, async: true
	
  describe "Testing some function" do
    setup do
	  [
        test_numbers: [1, 2, 3, 4, 5],
        test_name: "Bob",
        test_status: :alive
      ]
    end
		
    test "Doing something with numbers", fixture do
      do_something(fixture.numbers) == fixture.test_status
    end
		
    test "Doing something with name and numbers", test_data do
      do_something(test_data.numbers, test_data.name) == test_data.test_status
    end
  end			
end

The data returned by the setup function can be captured by a second argument to the test macro. I've seen this called "fixture", "state", or "context", but you can use any name you like. The ExUnit documentation uses the term "context".

I'm not going incorporate this into the example unit test project, since there isn't really a need for it.

There is also a setup_all variant available that will run once per module and the results will be reused for every test in that module.

Advanced Setup Functionality

ExUnit has the concept of a test "context", which contains the data that is returned from setup. We've seen this in the previous setup example. What's interesting is that you can actually specify multiple setups. If a setup returns a map or keyword list (or an {:ok, test_data} tuple with a map or keyword list), that map or keyword list will be merged with the test context. If setup just returns :ok, then the test context is left untouched. You can also gain access to the current context (and whatever data was returned by previous setup calls) by adding a "context" macro to the setup call.

Instead of the setup macro containing the setup code, like in the example in the previous section, it can also take the name of a function that will perform the setup. This is a way of delegating the setup work to an existing function. The function's name is specified using an atom. The ExUnit documentation refers to this as a "setup callback".

There is also an on_exit macro you can use to run code that runs at the end of each test. This gives you a chance to undo any changes that were made by setup.

Here's an example of a test showing those concepts

defmodule TestModule do
  use ExUnit.Case, async: true

  setup do
    IO.puts "Initial Setup"

    on_exit(&cleanup/0)

	  [
	    numbers: [3, 4, 5],
		status: :sleeping
	  ]
	end

    setup :define_test_data

  setup context do
    IO.puts "Additional Setup"
		IO.puts "Current Context: #{inspect(context)}"

		:ok
	end

  def define_test_data(_context) do
    IO.puts "Inside define_test_data"

	  %{
	    numbers: [1, 2, 3, 4, 5],
	    name: "Bob",
	  }
	end

    def cleanup() do
	  IO.puts "Cleaning up after the test"
    end

    test "Some dummy test", context do
      IO.puts "Running the dummy test"
      IO.puts "Final Context: #{inspect(context)}"
    end
end

First, the setup will be called that will return a set of test data that contains some numbers and a status. At this point, the context will contain numbers: [3, 4, 5] and status: :sleeping. Then the define_test_data/0 function will be run and will return test data that contains numbers and a name. Its output will be merged with the context, resulting in the context containing numbers: [1, 2, 3, 4, 5] (overwriting the previous numbers key), name: "Bob", and status: :sleeping. Finally, the last setup will be run. It will output the current context and return :ok, indicating that it does not wish to modify the context.

I added this code to the unit test project in "test_module_test.exs". Here's what the output looks like, showing that the flow I described above is correct. I temporarily disabled all the other unit tests so their output would not confuse things.

> mix test
Initial Setup
Inside define_test_data
Additional Setup
Current Context: %{async: true, case: TestModule, describe: nil, describe_line: nil, file: "c:/Development/Elixir/lwmelixir/examples/lwm 65/unit_test_app/test/test_module_test.exs", line: 37, module: TestModule, name: "Bob", numbers: [1, 2, 3, 4, 5], registered: %{}, status: :sleeping, test: :"test Some dummy test", test_type: :test}
Running the dummy test
Final Context: %{async: true, case: TestModule, describe: nil, describe_line: nil, file: "c:/Development/Elixir/lwmelixir/examples/lwm 65/unit_test_app/test/test_module_test.exs", line: 37, module: TestModule, name: "Bob", numbers: [1, 2, 3, 4, 5], registered: %{}, status: :sleeping, test: :"test Some dummy test", test_type: :test}
Cleaning up after the test
.

Finished in 0.04 seconds
1 test, 0 failures

Randomized with seed 772000

This is a particularly complicated example of setting up tests. I imagine that most of your test setups will be simpler than this.

I noticed that the test context has more than what I added to it. It also has information such as the name of the module, the file name, the current "describe" string, and whether it is running in asynchronous mode.

By the way, I noticed that I had to have the test file end with "_test.exs". If I didn't end the file name with that, the mix test tool completely ignored the file. The file didn't even get loaded into the interpreter. I have no idea why that is, but I wanted to make you aware of this issue in case you encountered a similar problem.

Other Testing Functionality

Tests can be tagged so you can run them selectively. You can tag your tests with an attribute above a function.

defmodule TestModule do
	using ExUnit.Case
	
	@tag :data
	test "testing something" do
		assert do_something() == "expected result"
	end
end

You can run only tests with the ":data" tag by running 'mix test --only dataor you can run all tests except the data tests by runningmix test --exclude data`.

There are some configuration options to configure ExUnit, but I'm not familiar with them yet. There's even more to unit testing than what I covered here, but it's likely that you'll mainly be using what I've covered so far.

ExUnit has some other useful functionality like capturing I/O interactions and logs that are emitted. I won't cover them in detail here, but I think those will be good topics for learning about sometime in the future.

I plan to start off using what I've learned so far and slowly branch out into any other features of ExUnit as I need them.

Code Coverage

There is an library called excoveralls that determines code coverage for your unit tests. It can show you which code has test coverage and which code is lacking test coverage. In additon to text output, it can generate nice HTML pages that can show you the code and which lines are covered and which aren't.

I'm not a big proponent of requiring 100% test coverage; some tests have much more value than others and once you have high test coverage, you run into rapidly diminishing returns when attempting to increase test coverage. In my experience, it's rarely practical to achieve 100% test coverage for any sufficiently-complex project, but this tool is still very useful to get an idea of what is covered and what isn't. It may very well be that some high-value tests are missing and you weren't aware of it.

Conclusion

That's it for today, but I have some more topics to cover. In particular, I still need to talk about doctests and what role dependency injection and mocking plays in Elixir.