6 July 2020
Faster Tests on your Elixir/Phoenix project and what to do when you get them
by Chris Nelson
I recently started a project from a client where we’ve been replacing a system built on a no-code environment that the client has outgrown. Elixir and Phoenix have been perfect choices for this project for a number of reasons. This might be worthy of another post, but what I’d like to share in this post is what I’ve learned from an afternoon I spent speeding up the unit tests on our project. The TL;DR: it was relatively easy to get faster tests, but I had some other issues with GenServers to fix when I got them.
What even is a unit test?
This, as they say, immediately precipitates a seething cauldron of debate. For my purposes, I’m talking about all of our tests that aren’t end to end browser automation tests (we tag them as feature tests). The browser automation tests use Wallaby, and are expected to be considerably slower to run. When I say unit tests, I’m talking about everything else. They do hit the database, interact with processes, etc. On our project they had started getting slower than I thought they should be, and I wanted to know why.
The culprit: encrypting user passwords
I had a theory before I started that encryption was the chief culprit. I observed that tests that created more users were getting slower, whereas other tests that did similar things (interacting with the DB, etc) did just fine. We are using Argon2 for encrypting passwords, as it seemed to be the most highly recommended solution in the community. We basically are using two functions. The first, Argon2.add_hash/2, takes a password and an optional key and returns a map with the hashed password in it. The second, Argon2.check_pass/3 takes a map (the user), a password, and an optional key name for the hashed password, and returns true if the password matches the hashed password. Replacing it followed a pattern I like to use for faking out behaviour I don’t want to call in my tests: setting up a module attribute with a default but configurable value that points at functions I want to fake. In practice, I swap out the encryption in my User module:
defmodule MyApp.User
@add_hash Application.get_env(:my_app, :hash_password, &Argon2.add_hash/2)
defp put_hashed_password(%Ecto.Changeset{valid?: true, changes: %{password: password}} = changeset) do
change(changeset, @add_hash.(password, hash_key: :hashed_password))
end
end
Then, in my account module I swap out checking password:
defmodule MyApp.Account
@check_password Application.get_env(:frayt_elixir, :check_password, &Argon2.check_pass/3)
def authenticate(email, password) do
user= Repo.get_by(User, email: email)
case @check_password.(user, password, hash_key: :hashed_password) do
{:ok, user} -> {:ok, user}
{:error, _} -> {:error, :invalid_credentials}
end
end
end
In test.exs, we can change our config to point at fake versions of these:
config :my_app,
hash_password: &MyApp.Test.FakePassword.hash_password/2,
check_password: &MyApp.Test.FakePassword.check_password/3
And writing this fake is pretty simple too:
defmodule MyApp.Test.FakePassword do
def hash_password(_, _), do: %{hashed_password: "gobbledygook"}
def check_password(user, "password", _), do: {:ok, user}
def check_password(user, "secretpassword", _), do: {:ok, user}
def check_password(user, "ABC456", _), do: {:ok, user}
def check_password(user, "somesupersecretstuff", _), do: {:ok, user}
def check_password(_, _, _), do: {:error, :invalid_credentials}
end
Lo and behold, after this change our unit test run went from 90 seconds to 3!
I have 99 test problems, but slowness ain’t one
My tests ran fast alright, but failed in unpredictable ways. It didn’t take long to understand why: I had some GenServer processes that were being started as children in application.ex. Previously, by happy? accident my tests had been slow enough for the server processes to exit when tests finished. But now that the tests were faster, the processes were still around between test runs and mass chaos ensued. At least, that was my theory. I had to find some way to quit starting my GenServers in application.ex and only start them for the tests that needed them while making sure my GenServers were terminated at the end of each test. At that point, if my tests were passing predictably I could be reasonably sure I was correct.
Time to kick your children out of your application
The first thing I realized is that it didn’t seem like it would work if I had the application start my child processes in the test environment. I haven’t seen much about this topic, so it’s quite possible I’m missing something obvious. If so, by posting this I’m sure people in the Elixir community will patiently show me the error of my ways and I will update this post :) In any event, here’s what I did in application.ex:
...
children = [
# Start the Ecto repository
MyApp.Repo,
# Start the endpoint when the application starts
MyApp.Endpoint,
# Starts a worker by calling: MyApp.Worker.start_link(arg)
# {MyApp.Worker, arg},
{Guardian.DB.Token.SweeperServer, []}
] ++ Application.get_env(:my_app, :child_processes, [
MyApp.ImportantServer,
MyApp.OtherImportantServer,a
])
This means that I start my application as normal in dev and production (and ci, which we’ll talk about shortly) and my application functions normally. In test, I added:
config :my_app,
child_processes: []
This prevents my GenServers from interfering with my unit tests. At this point, I only had a few failing tests, and perhaps more importantly they were the same ones each time. Those tests were tests that cover portions of the system that rely on the GenServer processes. Next, I had to figure out to have them started in a safe way.
start_supervised to the rescue
A little googling turned up a solution: an ExUnit callback named start_supervised. start_supervised takes a child spec and starts it, as well as guaranteeing the process will exit before the next test starts. For my app, I created a test helper to start the processes I needed like so:
defmodule MyApp.Test.StartImportProcesses do
use ExUnit.CaseTemplate
def start_important_processes(_) do
{:ok, _pid} = start_supervised(ImportantProcess1)
{:ok, _pid} = start_supervised(ImportantProcess2)
:ok
end
end
Then, in the “unit” tests that happen to need them:
import MyApp.Test.StartImportProcesses
setup :start_important_processes
Probably I could make this even easier and make a common test case a la DataCase, but I’ll leave that as an exercise for the reader. Or for my future self. Whoever gets to it first :)
What about functional tests?
This all worked great, but I had one last problem to solve: for true end to end feature tests, I needed all the processes launched at application startup. But in the test environment, I had turned them off. What do? I had already tagged all my feature tests so excluding them from the main build wasn’t a problem. What I needed was a fourth environment in additional to dev, test, and prod. I called it, unoriginally, ci. It seemed like this would be easy, peasy: copy my test.exs to ci.exs and comment out the child_processes setting, run feature tests with MIX_ENV=ci. There was one thing I missed though, which a quick google search clued me in on. I needed to define the compile path for ci in mix.exs:
# copied the test entry
defp elixirc_paths(:ci), do: ["lib", "test/support"]
Victory at last
And finally, I had achieved my goal. My hundreds of unit tests now ran in a few seconds, as God intended. And I had a clean build on my CI server which ran both unit and feature tests. I’d love to hear any thoughts on all this. I’m not sure we have commenting turned for our blog posts, but I’m superchris on the twitters. Happy testing! Happy Elixiring! Stay healthy friends.
