Testing MirageVPN against OpenVPN™
As our last milestone for the EU NGI Assure funded MirageVPN project (for now) we have been working on testing MirageVPN, our OpenVPN™-compatible VPN implementation against the upstream OpenVPN™. During the development we have conducted many manual tests. However, this scales poorly and it is easy to forget testing certain cases. Therefore, we designed and implemented interoperability testing, driving the C implementation on the one side, and our OCaml implementation on the other side. The input for such a test is a configuration file that both implementations can use. Thus we test establishment of the tunnel as well as the tunnel itself.
While conducting the tests, our instrumented binaries expose code coverage information. We use that to guide ourselves which other configurations are worth testing. Our goal is to achieve a high code coverage rate while using a small amount of different configurations. These interoperability tests are running fast enough, so they are executed on each commit by CI.
A nice property of this test setup is that it runs with an unmodified OpenVPN binary. This means we can use an off-the-shelf OpenVPN binary from the package repository and does not entail further maintenance of an OpenVPN fork. Testing against a future version of OpenVPN becomes trivial. We do not just test a single part of our implementation but achieve an end-to-end test. The same configuration files are used for both our implementation and the C implementation, and each configuration is used twice, once our implementation acts as the client, once as the server.
We added a flag to our client and our recently finished server applications, --test, which make them to exit once a tunnel is established and an ICMP echo request from the client has been replied to by the server.
Our client and server can be run without a tun device which otherwise would require elevated privileges.
Unfortunately, OpenVPN requires privileges to at least configure a tun device.
Our MirageVPN implementation does IP packet parsing in userspace.
We test our protocol implementation, not the entire unikernel - but the unikernel code is a tiny layer on top of the purely functional protocol implementation.
We explored unit testing the packet decoding and decryption with our implementation and the C implementation. Specifically, we encountered a packet whose message authentication code (MAC) was deemed invalid by the C implementation. It helped us discover the MAC computation was correct but the packet encoding was truncated - both implementations agreed that the MAC was bad. The test was very tedious to write and would not easily scale to cover a large portion of the code. If of interest, take a look into our modifications to OpenVPN and modifications to MirageVPN.
The end-to-end testing is in addition to our unit tests and fuzz testing; and to our benchmarking binary.
Our results are that with 4 configurations we achieve above 75% code coverage in MirageVPN. While investigating the code coverage results, we found various pieces of code that were never executed, and we were able to remove them. Code that does not exist is bug-free :D With these tests in place future maintenance is less daunting as they will help us guard us from breaking the code.
At the moment we do not exercise the error paths very well in the code.
This is much less straightforward to test in this manner, and is important future work.
We plan to develop a client and server that injects faults at various stages of the protocol to test these error paths.
OpenVPN built with debugging enabled also comes with a --gremlin mode that injects faults, and would be interesting to investigate.