Unit-testing in Ü
Now Ü has built-in unit-testing functionality. Writing a unit-test is as simple as this:
// Function to be tested.
fn Mul( i32 x, i32 y ) : i32 { return x * y; }
// Test for function above.
fn_test( MulTest )
{
halt if( Mul( 1, 3 ) != 3 );
halt if( Mul( 0, 255 ) != 0 );
halt if( Mul( 5, 7 ) != 35 );
halt if( Mul( -2, 9 ) != -18 );
// Further checks
// ...
}
halt is used for handling test failures, but it’s recommended to use some macro (like assert) which is able to print into stderr, what exactly went wrong.
Running such tests is possible via simple command test supported now by the Ü build system.
Implementation details
fn_test is just a special syntax, which allows defining a unit-test function.
Unit-test functions are almost like regular ones, but there are small differences in the way how they are built.
They get internal encoded names, so that it’s not possible to reference a unit-test (call it manually).
They always have external linkage and unit-test functions defined in imported files get comdat, so that they are deduplicated.
Also unit-test functions are marked with special metadata, so that it’s possible to identify them and process later.
What happens later is more interesting. While the Ü build system links regular build targets, unit-test functions are internalized and thus optimized-out, so that they are not present in result machine code. But it also creates executables containing unit-tests for each build target. This is done by launching the Ü compiler with special flag, which enables building of a table where all unit-test functions are listed. Then the result machine code containing this table is linked against a special library, which knows how to use it.
How this unit-testing library is implemented? The simplest implementation would be just to loop over all tests in the unit-tests table and to call them. Such approach is fine, but only in cases where all unit-tests are successful. As long as at least one unit-test fails, the whole unit-testing process crashes and thus no further unit-tests can be run and their status can’t be reported. So, some way of failure-recovery is necessary.
Languages like C++ often use exceptions for reporting test failures. This approach works, generally, even if it isn’t ideal. Rust has no exceptions, but it has stack unwinding, which is usually executed on panic being triggered. It functions almost exactly like unwinding in C++ happening after an exception is thrown. But Ü has no exceptions or even unwinding. It was a design decision not to implement it. Exceptions make things harder by obscuring control-flow and breaking local reasoning. Unwinding isn’t implemented, since it’s again breaks local reasoning and requires writing code in an unwinding-safe way, which is hard to do.
So, Ü needed a way to report test failures in some other way.
First I tried experimenting with setjmp/longjmp approach and signal handlers.
This is already not a good way of doing things, since recovering after a failure using longjmp doesn’t allow calling destructors, which leads to memory leaks, files left in opened state or even perpetually-locked mutexes.
But I thought that it’s a lesser evil to use them rather than having no proper way of handling failures in unit-tests at all.
I did a couple of experiments with such approach.
setjmp with longjmp executed in signal handler worked as intended on Linux systems.
But on Windows it didn’t work as expected.
And it’s not clear whether it’s correct to call longjmp from a signal handler, C standard doesn’t describe this clearly.
So, eventually I abandoned this approach.
There was last alternative - use proper per-process isolation. It allows handling failures gracefully, if a process crashes, it (almost always) frees resources associated with it. I experimented with running each unit-test in its separate process. This did work well on Linux, but with some noticeable overhead (roughly 1ms for starting each child process and waiting for it to finish). On Windows it did work too, but order of magnitudes slower than on Linux - with 15ms overhead for each process (slightly less if antivirus-scanning is disabled in system settings). So, it works, but it may be too slow if many unit-tests (hundreds to thousands) are present.
In the end I decided to use a more tricky approach. A separate process is started for running unit-tests. Many unit-tests are run sequentially inside it, as long as no test failure/crash is detected. In case of test failure or other crash, the parent process restarts the child to continue running tests. A block of shared memory is used between the parent and child processes to track already finished tests or tests which have been started, but failed. Since restarting a child process isn’t cheap, there is a limit on how many times it can be done, like 32 times. So, only up to 32 failed test-cases can be reported, which isn’t ideal, but mostly fine. It’s enough work for a programmer to identify what went wrong and to fix failed tests.
Each unit-tests executable creates a child process for running actual tests.
On Unix fork call is used for that.
On Windows a unit-tests executable runs itself using CreateProcessW with a special command-line flag to indicate, that it’s a child instance.
Using process-based tests isolation has one more advantage.
Not only halt usages are caught as test failures.
Any other process crash can be detected and reported, this includes access violation, integer division by zero, stack overflow and others.
This allows catching failures caused by incorrectly-written unsafe code, incorrectly-used external code, infinite recursion and similar cases not handled directly by Ü safety mechanisms.
The only thing which is (for now) can’t be handled are deadlocks, which are tricky to detect (it’s unclear whether there is a deadlock or just long computation).
The approach with running unit-tests in a separate process has one significant disadvantage. Debugging failed tests (with a debugger) isn’t that easy, since debuggers (usually) don’t debug child processes. As a workaround for this issue a small trick has been implemented - a unit-tests executable detects if a debugger is present, and if it’s the case, no child process is spawned and all tests are run in the current process instead.
The Ü build system has test command, which runs all unit-tests. But there are some caveats in it. It’s assumed, that the target triple specified for a project is compatible with host target triple. In other words, it’s assumed, that running test executables can be possible on host system. It’s not true in case of cross-compilation, like if building a project for Windows on Linux machine or building a project for iOS from OS X. But even if cross-compilation is performed, tests are built anyway. They can be still useful, if one finds a way to run them (somehow) on target system, even if running them on host using the test command isn’t possible.
Unit-test functions are always built. There is no way to skip building them. It’s done in order to be able to fix compilation errors in them as quickly as possible, if code on which they depend has been changed. It’s always good to fix compilation errors early, rather then to perform such forgotten fixes later.
Unit-test executables are generated for each build target by-default. It’s done in this way to encourage usage of unit-testing. Building unit-tests is pretty cheap. For each build target its unit-tests executable is linked from the same bitcode files obtained by compiling target’s source files, which are later used for linking this build target. So, no double work is performed. Linking unit-test executables and generating machine code for them isn’t free, but it’s almost free if no unit-tests at all are present or if there are only a few of them testing only a small part of the target’s codebase (dead code is eliminated before generating machine code).
Motivation for integrated inline unit-tests
One may ask, why using special syntax with fn_test?
Why having separate support of unit-testing in the build system?
Many other languages don’t have such built-in functionality, but unit-testing in them still can be used.
And why unit-test building and linking is turned-on by-default?
The answer is: friction reduction.
Creating unit-tests in Ü is as easy as writing a fn_test function and running the test command of the build system.
No extra setup is necessary.
No thirdparty unit-testing framework needs to be integrated.
No separate custom steps are needed for building and launching unit-tests.
With such low friction there is little to no reason not to write unit-tests, since it’s that easy.
And since unit-test linking is turned-on by-default, there is no danger to forget by mistake that no unit-tests are actually run.
It’s assumed, that such low-friction may encourage writing unit-tests. And more unit-tests means better overall quality of the result software written in Ü.
Results
After unit-testing functionality has been implemented, some unit-tests have been added in various places of Ü codebase. This includes parts of Compiler1 and the build system. And they seem to work well. A lot of functionality remains uncovered by such unit-tests, primary because it has been already written and debugged properly and because there are other ways for testing it. But I expect, that new unit-tests may be added for new pieces of functionality in future.
There is no drastic increase in build times caused by linking unit-test executables. It’s not entirely free, but fast enough in order to not care about it. And if in some rare case it creates some undesirable slow-down, one can still disable unit-tests for a particular build target.