Themis Development Security Practices
Testing during the build process
Our approach to testing Themis:
- Every Themis commit is checked with CircleCI and Bitrise, which runs a set of tests on the whole code.
- Every major Themis release is manually tested by building it on a number of platforms (including with intentionally building Themis with wrong parameters).
- Every time some new security-related features are introduced to Themis/summarised in a Themis release, they undergo a brief 3rd party code review.
In CI, every commit gets checked by:
- Valgrind as a dynamic analysis tool that detects memory leaks and memory management problems because we understand that poor memory management is often the source of the most catastrophic bugs.
- Splint as a static analysis tool that detects potential poor coding practices.
We have a special cryptographic test suite that contains test initialisation vectors for testing the donor cryptographic libraries with known parameters and results. These donor libraries are: OpenSSL, LibreSSL, BoringSSL, Libsodium, BearSSL.
The NIST-developed test is used for testing the quality of pseudo-random number generators in the donor libraries.
For symmetric encryption, we run a selected set of NIST-recommended tests for AES256. Such test sets contain both the initialsation vectors and the expected output.
The only cryptographic implementation hardcoded into Themis (ed25519-driven Socialist Millionaire Protocol used in Secure Comparator) is tested against known attacks via a separate test.
Testing Builds and Major Changes
We strive to build the most complete and explicit tests for Themis. They include both the functional and the API tests (that also check if correct error messages are returned for unexpected/invalid values).
For the core functionality and every language wrapper, including mobile, we created a unit test suit, placed in tests folder.
Besides, we built an integration test suit, to check:
- compatibility between language wrappers (f.e. encrypting using rubythemis, decrypting using gothemis);
- previous and current versions of Themis (backwards compatibility checks);
- running Themis on different architectures (x86, x86_64, ARM);
- running Themis on different desktop OS (Ubuntu, Debian and CentOS).
On many important occasions, we run Themis through:
- Several source code analysis tools like Cppcheck (after an issue #130 raised by awesome briongloden) to provide different static analysis output to verify against the output provided by Splint. Cppcheck is particularly good at identifying undefined behaviour.
- GCC and Clang in maximum warning verbosity mode to see all the possible warnings. The two compilers used side by side return the most complete tests and bridge the majority of the possible testing gaps.