Acra documentation

Start here

Welcome to the official documentation for Acra — database security suite for cryptographic protection, intrusion detection, and greater security of distributed apps.

We also recommend checking out the Advanced and Tutorials & How Tos documentation chapters.

If you want to skip ahead to practice, you can:

What is Acra

Acra – database security suite with protection for sensitive and personal data.

Acra provides selective encryption, multi-layered access control, database leakage prevention, and intrusion detection capabilities in a convenient, developer-friendly package. Acra was specifically designed for web and mobile apps with centralised data storage, including with distributed, microservice-rich applications.

Cryptographic design of Acra ensures that no secret (password, key, etc.) leaked from the application or database will be sufficient for decryption of the protected data chunks that originate from it.

Acra was built with specific user experiences in mind:

  • Quick and straightforward integration of security instrumentation.
  • Easy to try: you can experience the full might of Acra without committing to its installation using Docker containers (you can also request access to Acra Live Demo to play around with Acra without installing a thing).
  • Compatibility with encryption-demanding compliances: Acra can run on certified crypto-libraries (FIPS, GOST).
  • Cryptographic protection of data: to compromise an Acra-powered app, the attacker will need to compromise a separate compartmented server, AcraServer - more specifically - its key storage and database; until AcraServer is compromised, the data is safe.
  • Cryptography is hidden under the hood: you're safe from the risk of selecting a wrong key length or algorithm padding.
  • Secure default settings to get you going.
  • Intrusion detection to give you an early warning about suspicious behaviour.
  • SQL injections prevention through a built-in SQL firewall.
  • Ops-friendly: Acra can be easily configured and automated using a configuration automation environment.

Acra prevents plaintext data leakage from a database because the data is encrypted using context-dependent keys before it gets to the database. The database server has no means to decrypt the data and has no access to decryption keys. The data is encrypted with keys that are unique for each client application (or even data entity), making sure that even if one decryption key is leaked, only a minimum and amount of data can be decrypted as a result. AcraServer is the only entity responsible for data decryption and during the decuryption, it combines the key from pieces – using the keys stored in key storage (i.e. Amazon KMS) and context from application.

Acra prevents data leakage from application middleware because no keys accessible to middleware would be enough to decrypt the data and client applications can only encrypt the data. The data is transferred between the micro-services in ecnrypted form and it is put into the database in the end (encrypted). Cracking and tampering with the client application won't lead to data leakage.

Acra is able to limit SQL injection and insider risk using SQL filtering and poison records. AcraCensor is a separate customisable SQL firewall module for AcraServer that checks every incoming request. AcraCensor will let the allowed queries through to the database and return an error on forbidden ones. Posion records are records specifically created in such a way that they wouldn't be queried by a user under normal circumstances, but will be included in the outputs of SELECT * requests. If the database response contains poison records, AcraServer will detect them and inform you of untypical behaviour.

What's inside Acra

Acra consists of a client-side library and several server-side applications. Server-side Acra components should run as separate services/servers.

Client-side:

Server-side:

  • AcraServer — a separate daemon that runs in an isolated environment (separate virtual machine or physical server). AcraServer is responsible for holding all the secrets required to decrypt the data and for actually decrypting this data.

  • AcraCensor — a firewall-like component that sits inside AcraServer and checks every SQL query to the database.

Acra also has AcraWebConfig — a lightweight HTTP web server for managing AcraServer's certain configuration options.

You can check out the Protecting data using Acra documentation article to gain a better understaing of the way data protection processes work in Acra.

Platforms and databases

With the exception of AcraWriter, which is a library that developers integrate into the client-side application, all Acra components are distributed as binaries.

Acra’s binaries are built for CentOS 7, Debian Stretch, Debian Jessie, Ubuntu Bionic, Artful, Xenial, Trusty.

Acra currently supports PostgreSQL v9.4 - v11 and MySQL 5.7+.

Design and special features

Acra is a database protection suite with special features:

  • multi-layer access compartmentation to secrets for better security and early-stage intrusion detection;
  • special architecture for better developer usability and security of the in-app data;
  • single point of trust for minimal attack surface, inspecton of traffic that touches attack surface, and consistent risk management.

One of the key design goals for Acra was to ensure that engineering behind the security guarantees is compatible with modern software development and reliability engineering practices, i.e.:

  • with the typical division between application and database,
  • with an arbitrary number of application hosts, consumer microservices, and database nodes,
  • high level of abstraction in the database usage patterns in the app code.

Security design goals are rather simple (the guarantees are further explained and explored in Security design):

  • No amount of data and/or secrets leaked from AcraConnector and database server could be sufficient for decryption of the data stolen from the database. This should be achieved through cryptography rather than through general system design.
  • Sometimes even a well-behaved and trusted client (which is supposed to receive decrypted responses from AcraServer) might become compromised. However, any alteration of the app's data consumption in an unauthorised manner should be noticed, reported and prevented:
    • Acra provides a number of alarm triggers to detect potentially suspicious behaviour and requests from the application. Those triggers are configurable and allow creating unique sets of detection rules specific to the workflow and the app.
    • Acra provides a number of panic scripts, which either shut down the decryption, shut down the database connection, or provide other threat isolation methods.
    • These features are controlled programmatically and are built for modern automated environments.

In some Acra's use cases, the application can store encrypted data as separate blobs (files that are not in a database - i.e. in the S3 bucket, local file storage, etc.). In this case, you can use AcraTranslator — a lightweight server that receives AcraStructs and returns the decrypted data.

Acra is a continuously developing security tool. And as any proper security tool, it requires enormous human efforts for validation of the methods, code, and finding possible infrastructural weaknesses. We're constantly enhancing and improving Acra as we go. This is done to ensure that the provided security benefits are not rendered useless through implementation problems or increased complexity.

Cryptography in Acra

Acra relies on Themis cryptographic library, which implements high-level cryptosystems based on the best available open-source implementations of the most reliable ciphers.

Themis' cryptographic services, used via Go wrapper in AcraServer and AcraConnector, and via corresponding language wrappers in AcraWriters, are high-level compositions of carefully chosen, well-known cryptographic primitives. Acra does not contain any self-made cryptographic primitives or obscure ciphers. To deliver its unique guarantees, Acra relies on the combination of well-known ciphers and a smart key management scheme.

Currently, you can build Themis with using OpenSSL, LibreSSL, and Google's BoringSSL (a number of experimental build methods for LibSodium, BearSSL, and even CommonCrypto are available, too). They provide better security, use the implementations that have passed industry-specific compliance audits, and/or fit the hardware acceleration on your platform.

The enterprise version of Acra can run on the certified crypto-libraries of your choice (i.e. FIPS, GOST), drop us an email to get a quote.

How Acra helps with GDPR

Acra can help reach better compliance with GDPR for your product(s) as it covers some of the demands of the articles 25,32,33, and 34. Read more in "Acra and GDPR compliance".

Live Demos — try Acra without coding

The fastest way to start out with Acra is to try it with Docker, but there is an even easietr way to get aquianted with Acra and its capabilities. In fact, there are two. It is the Acra Live Demo(request the access to the demo by following the instuctions under the link) and Acra Engineering Demo.

Read more about each of Acra Live Demos below.

Acra Live Demo (try Acra in one click)

Acra Live Demo is a web-based demo that demonstrates the work of Acra (data protection, intrusion detection, rollback, and a number of other capabilities) in a typical web-infrastructure (deployed on our servers for your convenience).

Acra Live Demo infrastructure contains: Django-based application, PostgreSQL database, AcraServer with AcraCensor, log monitor. Sensitive data is encrypted in a Django application, stored in a database, and decrypted through Acra.

From the users' perspective, the website's work is unchanged. However, the data is securely protected so that even hacking the web application won't lead to data leakage.

The available actions include:

Requirements: Chrome, Firefox, or Safari browser.

Note: We create a separate playground for each user, that's why we ask for your email; you'll receive an invitation email.

Request an invitation to access to Acra Live Demo.

Acra Engineering Demo (whole infrastructure in one command)

Acra Engineering Demo illustrates the integration of Acra data protection suite into existing applications: Django-based web application and Python CLI application. We took well-known applications and added an encryption layer to them.

Data protection is completely transparent for the users and requires minimal changes in the infrastructure.

Developers and Ops friendly:

  • run a single command to deploy the application, database, Acra's components, logs, and dashboards;
  • read the code changes and see how little it takes to integrate encryption into the client application;
  • learn how Acra works by reading logs, monitoring metrics in Prometheus, and watching Grafana dashboards;
  • inspect Docker-compose files, architecture schemes, database tables, and much more.

Requirements: Linux or macOS terminal.

Run the Acra Engineering Demo!.

Otherwise — proceed to Getting started to install and try Acra in your own infrastructure.

Getting started

Installing Acra is fairly simple:

Requirements

  • A client application written in Ruby, Python, PHP, Node.js, or GoLang, which runs on a Linux machine.
  • The application talks to PostgreSQL or MySQL database via some language-specific framework (you can use SSL or non-SSL mode between AcraServer and AcraConnector for Acra; in non-SSL mode Secure Session will be used).
  • Themis and client application language wrapper are installed on your machine (in most cases it will be installed automatically during AcraWriter installation process).
  • Also, if you're installing Acra manually from the GitHub repository, you need to have Themis' dependencies and libssl-dev package installed, and libcrypto.so needs to be available in $PATH.

Setup process

  • Generate keypair, which will identify this AcraConnector using acra-keymaker utility. AcraServer will recognise this AcraConnector instance by the (previously) specified <client_id>.
  • Deploy AcraConnector:
  • Add another system user on the machine that runs your application (for AcraConnector).
  • Deploy AcraConnector binaries to run under that user.
  • Distribute the keys: put the keys generated by acra-keymakerclient_id, client_id_server.pub, and client_id_storage.pub – into the corresponding .acrakeys folder.
  • Run AcraConnector as a service from under this user.
  • Point the database functions in your applications to talk to localhost:9494 (default port) or any other port you've chosen.
  • Deploy AcraServer to a separate machine:
  • Place the keys generated by acra-keymaker client_id.pub, client_storage, and client_id_server to the .acrakeys folder.
  • Copy AcraServer public key / Zone key to the .acrakeys folder belonging to:
    • AcraConnector's user,
    • Any user that runs AcraWriter code.
  • Integrate AcraWriter:
  • Make client_id_storage.pub accessible to AcraWriter code.
  • Integrate AcraWriter closely following the examples and using either AcraConnector (if available on this machine) or direct database connection.
  • Test everything:
  • Connect AcraConnector to AcraServer, send a regular request to your database through AcraConnector.
    Result: If you see the answer, AcraConnector and AcraWriter are able to connect and forward signals back and forth. It means that the network and the keys are fine.
  • Upon integrating AcraWriter into your code, try generating an AcraStruct from some payload.
    Result: If you succeed in running AcraWriter code, Themis library is installed properly and some of the keys are located in the expected place.
  • Write a row with AcraStruct into the database, either directly or through AcraConnector. Request this row through AcraConnector.
    Result: If you see decrypted payload in the response, the scheme works properly.
  • Thoroughly read the documentation.
  • Use Acra!
  • P.S. Feel free to raise an Issue or contact us at info@cossacklabs.com.

Quickstart

Note: Please read the documentation before starting out with Acra! There are some fundamental concepts that we highly advise you to understand before you proceed. Pay special attention to the Architecture.

Also, remember that:

  • AcraServer and Acra's dependencies need to stay on a separate server / virtual machine.
  • AcraConnector can run in a container or under separate user on the same machine as the main application.

Choose your weapon!;) There are different ways to start out with Acra. You can either:

Using Acra with Docker

Note: This way of using Acra is for testing purposes only and should never be used for real-life production environments.

For a really quick start, we recommend trying Acra with Docker first. Using only two commands you will get all Acra components and database up and running, with a secure transport layer between them. We prepared several typical infrastructures to play with.

The detailed instructions for installing and operating Acra with Docker can be found in the Trying Acra with Docker documentation page. Bear in mind that this way of using Acra is for testing purposes only and should never be used for real-life production environments.

If you want to try Acra in your real-life setting, please use the installation instructions below and either install Acra from our official Cossack Labs repository or from the GitHub Acra repository.

Installing Acra from the Cossack Labs repository

Debian / Ubuntu

1. Import the public key used by Cossack Labs to sign packages:

wget -qO - https://pkgs.cossacklabs.com/gpg | sudo apt-key add -

Note: If you wish to validate the key fingerprint, it is: 29CF C579 AD90 8838 3E37 A8FA CE53 BCCA C8FF FACB.

2. You may need to install the apt-transport-https package before proceeding:

sudo apt-get install apt-transport-https

3. Add the Cossack Labs repository to your sources.list.
You should add a line that specifies your OS name and the release name:

deb https://pkgs.cossacklabs.com/stable/$OS $RELEASE main
  • $OS should be debian or ubuntu.
  • $RELEASE should be one of Debian or Ubuntu release names. You can determine this by running lsb_release -cs, if you have lsb_release installed.

We currently build packages for the following OSs and RELEASE combinations:

  • Debian "Jessie" (Debian 8),
  • Debian "Stretch" (Debian 9),
  • Ubuntu Trusty Tahr (Ubuntu 14.04),
  • Ubuntu Xenial Xerus (Ubuntu 16.04),
  • Ubuntu Artful Aardvark (Ubuntu 17.10).

For example, if you are running Debian 9 "Stretch", run:

echo "deb https://pkgs.cossacklabs.com/stable/debian stretch main" | \
  sudo tee /etc/apt/sources.list.d/cossacklabs.list

4. Reload local package database:

sudo apt-get update

5. Install the package

sudo apt-get install acra

CentOS / RHEL / OEL

Note: We only build RPM packages for x86_64.

1. Import the public key used by Cossack Labs to sign packages:

sudo rpm --import https://pkgs.cossacklabs.com/gpg

Note: If you wish to validate the key fingerprint, it is: 29CF C579 AD90 8838 3E37 A8FA CE53 BCCA C8FF FACB.

2. Create a Yum repository file for Cossack Labs package repository:

wget -qO - https://pkgs.cossacklabs.com/stable/centos/cossacklabs.repo | \
  sudo tee /etc/yum.repos.d/cossacklabs.repo

3. Install the package:

sudo yum install acra

That's it! You've successfully installed Acra from the Cossack Labs repository.

Installing from GitHub

These are the instruction for installation of AcraServer from the Cossack Labs' GitHub repository for Acra. You will need the same set of dependencies for each component.

Installing the dependencies

Install the dependencies for Acra:

sudo apt-get install git golang libssl-dev make build-essential

Set up your $GOPATH to some place where you will store the code.

Install Themis

git clone https://github.com/cossacklabs/themis.git
cd themis
make
sudo make install

Build the key generator and generate the keys

go get github.com/cossacklabs/acra/cmd/acra-keymaker

Then generate the keys and distribute them across the infrastructure.

Remember to generate ACRA_MASTER_KEY and assign it to the environmental variable!

Set up the environment for AcraServer

On a separate machine, create a user for AcraServer:

sudo useradd -m acra-server
sudo su acra-server
cd ~/
export GOPATH=`pwd`

Build AcraServer

go get github.com/cossacklabs/acra/cmd/acra-server

Place someid.pub, someid_storage and someid_server keys to .acrakeys directory for AcraServer. Now you can finally launch the AcraServer.

Launching AcraServer

Running AcraServer is easy, just point it to the database:

$GOPATH/bin/acra-server --db_host=127.0.0.1

If you see an error message "master key is empty", it means that you haven't generated ACRA_MASTER_KEY, please return to the Key Generation step.

You can complement the command above with --db_port=5432 -v to adjust the listener port and add logs to get going quickly. For all the available CLI parameters, please refer to the corresponding section in How AcraServer works.

You can also run with the options from config. You can copy the example config ($REPO_DIR/configs/acra-server.conf or from GOPATH $GOPATH/src/github.com/cossacklabs/acra/configs/acra-server.conf) or generate it yourself:

$GOPATH/bin/acra-server --dump_config --config_file=acra-server.conf

Also run:

$GOPATH/bin/acra-server --config_file=acra-server.conf
Logging
$GOPATH/bin/acra-server --db_host=127.0.0.1 -v
Explicit server_id
$GOPATH/bin/acra-server --db_host=127.0.0.1 --server_id=some_name

It is used as an identifier for Secure Session between AcraConnector and AcraServer and can also be used for checking the correctness of AcraConnector's setup.

Listen on a custom port:
$GOPATH/bin/acra-server --db_host=127.0.0.1 --incoming_connection_port=3000

AcraServer listens to port 9393 by default.

Switch PostgreSQL driver to escape_bytea (hex mode by default)
$GOPATH/bin/acra-server --db_host=127.0.0.1 --pgsql_escape_bytea

Before PostgreSQL 9, only escape_bytea mode was present. Now the hex mode is present by default during install.

Additional features:

AcraConnector

Create a user for AcraConnector:

sudo useradd -m acra-connector
sudo su acra-connector
cd ~/
export GOPATH=`pwd`

Build AcraConnector

go get github.com/cossacklabs/acra/cmd/acra-connector

Put someid and someid_server.pub keys into .acrakeys directory for AcraConnector.

Run AcraConnector

$GOPATH/bin/acra-connector --acraserver_connection_host=127.0.0.1 --client_id=someid -v

If you see error message similar to "Configuration error: AcraConnector private key .acrakeys/someid doesn't exists", it means that you haven't generated keys or keys are place in a wrong folder, please return to the Key Generation step.

AcraConnector is now listening on the localhost port 9494.

Write an app with AcraWriter

Read the introductory info about AcraWriter here, or read one of our tutorials: Django project, Rails project.

Create a user for the app:

sudo useradd -m application
sudo su application
cd ~/
export GOPATH=`pwd`

Copy AcraServer's public key with suffix _storage for the app. Write your own or adapt our examples to your settings.

go get github.com/cossacklabs/acra/examples...
nano src/github.com/cossacklabs/acra/examples/golang/src/example/examples.go
or
nano src/github.com/cossacklabs/acra/examples/ruby/example.rb
or
nano src/github.com/cossacklabs/acra/examples/nodejs/examples_with_zone.js

Congratulations! Acra is ready.

Architecture and data flow

Acra components


Acra-based application architecture using Themis SecureSession transport encryption.

  • AcraWriter — a client-side library that integrates into the app's workflow either through ORM or directly and provides the means for encryption of the sensitive data by generating AcraStructs (Go, Python, Ruby, PHP, Node.js).

  • AcraConnector — a client-side daemon that runs under a separate user / in a separate container, and which acts as a database listener that redirects all the queries to AcraServer and, upon receiving the results, feeds it back into the app. AcraConnector is an optional component, required in systems that are using extra transport encryption layer via Themis SecureSession.

  • AcraServer — a separate daemon that runs in an isolated environment (separate virtual machine or physical server). AcraServer is responsible for holding all the secrets required to decrypt the data and for actually decrypting this data.

  • AcraCensor – is a firewall-like component that sits inside AcraServer, and checks every SQL query to the database.

  • AcraWebConfig – a lightweight HTTP web server for managing AcraServer's certain configuration options.

Using Themis SecureSession in Acra

Data flow: reading data

Acra's design values simplicity as much as security. Reading the data should not require much tweaking — just point the database address to AcraConnector and you're good to go:

  1. The application sends database request to AcraConnector (which pretends to be a database listener) using the standard PostgreSQL/MySQL protocol.
  2. AcraConnector sends that request to AcraServer using Secure Session (socket protection protocol).
  3. AcraServer sends a request to the database using the regular PostgreSQL/MySQL protocol and receives an answer. If this request is an SQL query, AcraServer checks with AcraCensor and blocks or allows the query to pass through depending on the AcraCensor settings.
  4. If AcraServer detects the presence of AcraStruct while parsing the answer, AcraServer will attempt to decrypt it and replace AcraStruct with the resulting plaintext in the answer, adjusting the size of the answer. If decryption fails, AcraServer just forwards the answer as is (in encrypted form).
  5. AcraServer returns the data to AcraConnector (with Secure Session), which in turn returns it to the application as if it had come from the database itself.

Data flow: writing data

Writing data requires:

  • Encrypting your sensitive piece of data with create_acrastruct, ORM, or a pre-created statement handler somewhere in your code. (Django example, Ruby on Rails example).
  • Actually writing the data:
  • Send the data via AcraConnector connection through AcraServer if you need a single entry point into your database.
  • Send the data directly into the database if you'd like to decouple writes from reads and have a lower quantity of sensitive secrets in this part of the code.

The beauty of Acra's design is that you don't need AcraServer and all its components to write the data. When dealing with microservice-oriented architecture, you can push writes directly to the database where it's convenient or do it through AcraServer where it's present:

Data flow with Zones: reading the data

During the request, the application has to include Zone Id explicitly somewhere in the 'select' SQL query before the data that is to be decrypted.

For example, table1 with sensitive data is in the data column and Zone Id is equal to client_id (stored in the client column). To make AcraServer decrypt everything properly, you need SQL statement like: SELECT client, data FROM table1;. You can also store the identifier elsewhere, separately from the data, and include it in a request statement, i.e.: SELECT '00fv8gIV41'::bytea, data from table1;, where '00fv8gIV41' is zone/client ID.

Upon detecting ZoneID, AcraServer will use a corresponding private key to decrypt the next detected AcraStruct. AcraServer will ignore the AcraStruct if no Zone Id is detected before the arrival of AcraStruct.

Data flow with Zones: writing data

  • Add a zone through Acra (either via console utility or using HTTP API through AcraConnector);
  • Encrypt your sensitive piece of data with create_acrastruct somewhere in the code, ORM, or in the prepared statement handler using zone key + Id as an argument (Django example, Ruby on Rails example);
  • Write the result to the database (similarly to a default zone).

Using TLS/SSL in Acra

Data flow: reading data

If you rely on TLS/SSL in your system design and you are not using Zones, you might omit AcraConnector at all. Just point the database address to the AcraServer:

  1. The application sends database request to AcraServer (which pretends to be a database listener) using the standard PostgreSQL/MySQL protocol over SSL.
  2. AcraServer sends a request to the database using the regular PostgreSQL/MySQL protocol over SSL and receives an answer. If this request is an SQL query, AcraServer checks with AcraCensor and blocks or allows the query to pass through depending on the AcraCensor settings.
  3. If AcraServer detects the presence of AcraStruct while parsing the answer, AcraServer will attempt to decrypt it and replace AcraStruct with the resulting plaintext in the answer, adjusting the size of the answer. If decryption fails, AcraServer just forwards the answer as is (in encrypted form).
  4. AcraServer returns the data to the application as if it had come from the database itself.

Data flow: writing data

Writing data is the same as in the description above:

  • Encrypting your sensitive piece of data with create_acrastruct, ORM, or a pre-created statement handler somewhere in your code. (Django example, Ruby-on-Rails example).
  • Actually writing the data:
  • Send the data via AcraServer if you need a single entry point into your database.
  • Send the data directly into the database if you'd like to decouple writes from reads and have a lower quantity of sensitive secrets in this part of the code.

The beauty of Acra's design is that you don't need AcraServer and all its components to write the data. When dealing with microservice-oriented architecture, you can push writes directly to the database where it's convenient, or do it through AcraServer where it's present:

Zones

Zones are the way to cryptographically compartmentalise records in an already-encrypted environment. Zones rely on different private keys on the server side.

The idea behind Zones is very simple (yet quite specific to some use-cases): when we store sensitive data, it's frequently related to users / companies / some other binding entities. These entities could be described through some real-world identifiers, or (preferably) random identifiers, which have no computable relationship to the protected data.

Acra uses this identifier to also identify, which key to use for decryption of a corresponding AcraStruct.

How Zones work

When analysing the database output stream, AcraServer searches for certain strings called Zone Ids (also ZoneId and zoneid/zone_id in code). They let Acra know that within this record a private key corresponding to the Zone Id should be used for the actual decryption of AcraStructs. It will only work if the user explicitly formats the output for Zone Id to precede the AcraStruct.

Default zone

If you extend the Zone concept to all Acra's behaviours, you'll see that a zoneless mode is actually using one default zone, which doesn't require a Zone Id (zone_id).

Running AcraServer in Zone mode

For AcraServer to be actually able to decrypt something, your AcraStructs-generating code must be written with Zones in mind.

$GOPATH/bin/acra-server --db_host=127.0.0.1 --zonemode_enable

More information about Zones

  • Data flow: Simplified data flow for Zone-based encryption/decryption processes.
  • Tuning Acra: How Zones affect certain tuning strategies for increased performance / security.
  • Using Zones on client: A practical How-To.

AcraStruct

Understanding AcraStruct

AcraStruct is a container format. Before generating each AcraStruct, AcraWriter generates a keypair of throwaway keys that are used in the encryption process and then get zeroed (turned into zeros) in the memory once the process is over.

AcraStruct = Begin_Tag + Throwaway_Public_Key + Encrypted_Random_Key + Data_Length + Encrypted_Data

  • Begin_Tag[8] — 8 bytes, header tag (can be changed);
  • Throwaway_Public_Key[45] — temporary public key generated by AcraWriter;
  • Encrypted_Random_Key[84] — encrypted Random Key by using SMessage (see next);
  • Data_Length[8] — length of the Encrypted data (see next);
  • Encrypted_Data[Data_Length] — payload encrypted with Random Key.

Generating AcraStruct

AcraWriter is used to generate AcraStruct, but the generation process is quite simple and can be implemented in any custom writer:

  • AcraWriter generates a key pair of throwaway keys using Themis EC key generator:
    Throwaway_Keypair = (Throwaway_Public_Key, Throwaway_Private_Key).
  • Generates Random Symmetric Key (RK), 32 bytes long.
  • Encrypts RK using Secure Message with Throwaway_Private_Key and Acra_Public_Key(or Zone key – see Zones):
    Encrypted_Random_Key = SMessage(RK, Throwaway_Private_Key, Acra_Public_Key).
  • Encrypts the payload with Secure Cell in Seal mode:
    Encrypted_Data = SCell(RK, payload).
  • Erases/fills with zeros memory area with the RK.
  • Calculates the encrypted payload length and transforms it to little endian 8 bytes long (Data_Length).
  • Connects attributes together as described in the original formula.
  • Erases/fills with zeros memory area with the Throwaway_Keypair and original payload.

You can see how AcraStruct is generated in acra-writer.go, acrawriter.py, acrawriter.rb or acrawriter.php.

Decrypting AcraStruct

AcraServer, upon receiving and detecting valid AcraStruct, is able to:

  • Extract Throwaway Public Key (TPK).
  • Decrypt asymmetric envelope with TPK and Acra's Private Key (or Zone key).
  • Extract Random Key (RK) for Secure Cell container out of a decrypted envelope;
  • Decrypt Secure Cell, extract payload;
  • Reconstruct database answer in such a way that AcraStruct is replaced by decrypted data.

Check implementation in decryptor/base/utils.go.

Zone Ids

Although not directly related to AcraStructs, Zone Ids are used for matching Zones to keys. The current format is 8 bytes "begin tag" + 16 symbols a-zA-Z.

Storage Models

The storage model modes used in Acra are WholeCell and InjectedCell.

In WholeCell mode, AcraStruct represents a complete piece of data (i.e. database cell, a file, or some data transmitted into AcraTranslator). In this mode it is expected that the encrypted data will look something like:

  1. <AcraStruct>,
  2. <AcraStruct>,
  3. <AcraStruct>.

In InjectedCell mode, AcraStruct is stored inside some piece of data. i.e. inside some file or in a database cell with a file inside, with AcraStruct as a piece of that file, not the whole file. In this mode, the encrypted data will look something like this:

  1. <Some AcraStruct data, some other AcraStruc data>,
  2. <AcraSctruct>,
  3. <File containing AcraStruct alongside other data>.

The main difference between these modes lise in performance. In the WholeCell mode, AcraStructs are simply decrypted. In InjectedCell mode, AcraServer needs to find AcraStructs inside some other data element first and then decrypt them, which will obviously take longer. The process of searching for the necessary piece of data takes place as the data is going through Acra. Acra will look for AcraStructs in every piece of data in InjectedCell mode.

Which mode should you choose?

Let’s consider an example where we’re storing an email in a database and we’d like to encrypt it, “wrapping” it into an AcraStruct. We’d get a table:

Email Column 2 Column 3
Column2Value Column3Value
Column2Value Column3Value

In this case, an AcraStruct takes up a whole cell and we are trying to decrypt it as is, without searching for anything. But things can be different. In InjectedCell mode a binary MsgPack or protobuf could be stored in a table, and it is possible that in those data entities only one field is encrypted.

In this case, such data entity wouldn’t be a single AcraStruct - it would be a data entity that contains an AcraStruct or several AcraStructs. This means that in the InjectedCell mode we stop assuming that the database cells can only contain complete AcraStructs. AcraStructs can be inside some other pieces of data and that’s where we’re starting to look for them. This consequently slows down the processing speed. However, not every task needs the InjectedCell mode, which is why one can switch between modes, depending on what you’re encrypting and how you’re storing it.

By default, the WholeCell mode is active (--acrastruct_wholecell_enable). You can switch between the modes using startup parameters:

--acrastruct_injectedcell_enable
    Acrastruct may be injected into any place of data cell
--acrastruct_wholecell_enable
    Acrastruct will stored in whole data cell (default true)

Please also see the CLI reference.

Key management

There are three types of keys used in Acra:

  • Transport key pair (see Secure Session) for protection of transport between AcraServer and AcraConnector, or AcraTranslator and AcraConnector. The two parties should exchange public keys, keeping the private keys to themselves.

  • Storage key pair: AcraWriter uses the public key for data encryption, AcraServer/AcraTranslator use private storage key for data decryption.

  • Authentication storage key for encryption/decryption credentials of AcraWebConfig users.

Storage keys can be represented by either:

  • One keypair for each unique client_id(default Zone mode), which is used for encryption by AcraWriter and decryption by AcraServer/AcraTranslator (so, in fact, there can be many such keypairs, one for each client_id).
  • A set of Zone keys (multiple Zone mode). Each Zone represents a unique user or type of users and has corresponding encryption and decryption keys. Using Zones complicates an unwanted decryption: the attacker needs not only to get the decryption key, but to use a correct Zone Id as well.

Key names and locations

Each Acra component has to have its own key folder (by default it is .acrakeys) that contains a set of keys necessary for the correct functioning of the components:

  • AcraConnector needs to have its own transport private key and other party transport public key to establish a Secure Session connection. You should put transport public key of AcraServer or AcraTranslator into AcraConnector's key folder, depending on the connection type.

  • AcraServer should have:

    • AcraConnector's transport public key and AcraServer's own transport private key. This is necessary for accepting connections via Secure Session from the clients.
    • Storage private key(s) for decrypting AcraStructs that are generated by AcraWriter.
  • AcraTranslator should have:

    • AcraConnector's transport public key and AcraTranslator's own transport private key. This is necessary for accepting connections via Secure Session from the clients.
    • Storage private key(s) for decrypting AcraStructs generated by AcraWriter.
  • AcraWriter should have storage public key(s). They are necessary for encrypting data into AcraStructs in such a way that would only be read by AcraServer/AcraTranslator.

  • If you're using AcraWebConfig HTTP server to configure the AcraServer remotely, the users' credentials are stored encrypted by authentication storage key and are decrypted on AcraServer upon logging in the user. AcraServer needs to have an authentication storage key.

Keys table

Purpose Private key Stays on Public key Put to
Transport AcraConnector clientid AcraConnector clientid.pub -> AcraServer/
AcraTranslator
Transport AcraServer clientid_server AcraServer clientid_server.pub -> AcraConnector
Transport AcraTranslator clientid_translator AcraTranslator clientid_translator.pub -> AcraConnector
AcraStruct encryption clientid_storage.pub -> AcraWriter
AcraStruct decryption clientid_storage AcraServer/
AcraTranslator
AcraStruct encryption with Zones zoneid_zone.pub -> AcraWriter
AcraStruct decryption with Zones zoneid_zone AcraServer/
AcraTranslator
Authentication storage key
for AcraWebConfig's users
auth_key AcraServer

Generating all the Acra keys in one go

We described many keys here and the private keys are (obviously) stored in an encrypted form. The main Acra key is ACRA_MASTER_KEY, which is used for decryption of private keys for every Acra service. Keep an eye on this key!

1. Generating ACRA_MASTER_KEY

Use acra-keymaker to generate master key into master.key file and assign it into the environment variable on the corresponding server:

go install ./cmd/acra-keymaker
$GOPATH/bin/acra-keymaker --generate_master_key=master.key
export ACRA_MASTER_KEY=`cat master.key | base64`

2. Generating transport and encryption keys

After this, the master key will be used by acra-keymaker for encryption of private keys before those are written to the disk.

Now, let's generate keypairs for Acra services:

$GOPATH/bin/acra-keymaker --client_id=someid

Carry out these operations on the machine running AcraServer to make sure that the private keys for Acra never leak outside from the server.

The generator will generate and place 8 keys into the .acrakeys directory (you can change this with --keys_output_dir argument):

.acrakeys/someid
.acrakeys/someid.pub
.acrakeys/someid_server
.acrakeys/someid_server.pub
.acrakeys/someid_storage
.acrakeys/someid_storage.pub
.acrakeys/someid_translator
.acrakeys/someid_translator.pub

Note: For acra to work properly, the key folders have to have proper permissions as set by acra-keymaker:

  • folder 700.
  • private keys 600.

3. Key exchange

3.1 Exchange transport keys between AcraConnector and AcraServer
Component should contain keys named like these
AcraConnector transport public key of AcraServer
transport private key of AcraConnector
.acrakeys/someid_server.pub
.acrakeys/someid
AcraServer transport public key of AcraConnector
transport private key of AcraServer
.acrakeys/someid.pub
.acrakeys/someid_server
3.2 Exchange transport keys between AcraConnector and AcraTranslator
Component should contain keys named like these
AcraConnector transport public key of AcraTranslator
transport private key of AcraConnector
.acrakeys/someid_translator.pub
.acrakeys/someid
AcraTranslator transport public key of AcraConnector
transport private key of AcraTranslator
.acrakeys/someid.pub
.acrakeys/someid_translator
3.3 Exchange AcraStruct encryption/decryption keys between AcraWriter and AcraServer/AcraTranslator
Component should contain key named like this
AcraWriter storage public key .acrakeys/someid_storage.pub
AcraServer storage private key .acrakeys/someid_storage
AcraTranslator storage private key .acrakeys/someid_storage
3.4 Exchange AcraStruct Zone keys between AcraWriter and AcraServer/AcraTranslator

Generating Zone keys is different from generating usual AcraStruct encryption keys. You should run acra-addzone on AcraServer to generate zone:

go install ./cmd/acra-addzone
$GOPATH/bin/acra-addzone

When you run it, you should receive a JSON object that looks something like this:

{"id":"DDDDDDDDQHpbUSOgYTzqCktp","public_key":"VUVDMgAAAC3yMBGsAmK/wBXZkL8iBv/C+7hqoQtSZpYoi4fZYMafkJbWe2dL"}

By decoding a base64-wrapped public key, you get a binary Zone key which should be provided (placed into) to AcraWriter for generating AcraStructs. Zone keys are placed into the key folder and are named zoneid_zone for private key and zoneid_zone.pub for public key.

Component should contain key named like this
AcraWriter zone public key .acrakeys/zoneid_zone.pub
AcraServer zone private key .acrakeys/zoneid_zone
AcraTranslator zone private key .acrakeys/zoneid_zone

Read more about using Zones in the AcraWriter guide.

(Re)Generating only some of the keys

If you want to update or re-generate only some particular keys, it can be easily done via CLI parameters for acra-keymaker:

  • --generate_acraconnector_keys – generates AcraConnector transport keypair;
  • --generate_acraserver_keys – generates AcraServer transport keypair;
  • --generate_acratranslator_keys – generates AcraTranslator transport keypair;
  • --generate_acrawriter_keys – generates data storage keypair to be split between AcraWriter and AcraServer/AcraTranslator;
  • --generate_acrawebconfig_keys – generates a symmetric key for encrypting credentials of AcraWebconfig' users;
  • --generate_master_key – generates new ACRA_MASTER_KEY, all private keys should be re-generated after changing master key.

Client side: AcraConnector and AcraWriter

AcraConnector and AcraWriter

AcraConnector is a (separate) service running alongside your application — it pretends to be a database listener, relays all the requests to AcraServer, receives the responses, and returns them to the app, just like a normal database listener would do.

To talk to AcraServer, you'll need to run AcraConnector on the same host as your application, in a separate container or as a separate user. You'll also need to route database requests to its address.

To talk to AcraTranslator you'll need to do the same: run AcraConnector on the same host as your application, in a separate container or as a separate user, and use its URL as destination URL from your application.

Why do we need a special piece of software to talk to your other piece of software?

Acra needs a trusted agent on the application's side to protect the sensitive decrypted responses, to provide basic channel authentication, and to enforce certain behaviours.

AcraConnector acts as a local proxy that receives requests from the app and returns decrypted answers. AcraConnector provides an encrypted and authenticated connection to AcraServer, which, in turn, fetches the response to the request from the database and decrypts all the data. AcraServer then returns the data to AcraConnector via a secure connection.

AcraConnector works in a similar fashion with AcraTranslator, redirecting AcraStructs from application to AcraTranslator and delivering the decrypted response back.

This enforces maximum secrecy, employing authentication that is easy to manage (pre-shared private keys), and requires minimal intervention into your code for successful implementation!

Getting started with AcraConnector

Method 1A. Launch AcraConnector using Docker (the fastest way to try AcraConnector)

Note: Using Docker is recommended for testing purposes only. Please don't rely on Docker in real-life production settings.
Note: The following examples focus on using AcraConnector and AcraWriter with PostgreSQL, but Acra also supports MySQL.

Clone the Acra repository, build images, and start Docker compose with PostgreSQL, AcraServer, AcraConnector, and Secure Session between them:

git clone https://github.com/cossacklabs/acra.git
make docker
docker-compose -f docker-compose.pgsql-nossl-server-ssession-connector.yml up

Okay, now the basic infrastructure is all set up, all components are connected, and keys are distributed into the appropriate folder.

Now, proceed to step launching AcraConnector. Or install AcraConnector manually:

Method 1B. Manual launch

Note: Skip this if you've used the Docker method described above.

  • Install dependencies: Themis cryptographic library:
git clone https://github.com/cossacklabs/themis.git
cd themis
make
sudo make install
  • Install AcraConnector:
go get github.com/cossacklabs/acra/cmd/acra-connector

Note: Hereinafter all the commands starting with 'go' are meant to be executed from the 'acra' folder (the folder with the repository code).

  • Install key generation utility:
go get github.com/cossacklabs/acra/cmd/acra-keymaker
  • Use acra-keymaker to generate master key into master.key file and assign it into the environment variable like this:
$GOPATH/bin/acra-keymaker --generate_master_key=master.key
export ACRA_MASTER_KEY=`cat master.key | base64`

Read more about the different types of keys used in Acra in the Key Management section of the documentation.

  • Generate the "client" proxy keypair:
$GOPATH/bin/acra-keymaker --client_id=client_name --generate_acraconnector_keys

The name of the key should be longer than 5 characters. It is also used as an identifier for the Secure Session connection between AcraConnector and AcraServer.

The generated keypair client_name and client_name.pub will appear in .acrakeys (or anywhere you ask with --keys_output_dir=/path/to/dir argument).

  • Exchange public keys:

  • You should put public key client_name.pub into the corresponding folder (.acrakeys) on AcraServer.

  • You should put AcraServer's public key (called client_name_server.pub) to AcraConnector's key folder (.acrakeys or anything you chose in --keys_output_dir).

Now, proceed to launching AcraConnector.

Launching AcraConnector

By default, AcraConnector is ready to talk to AcraServer. You need to use a one-line command:

$GOPATH/bin/acra-connector --client_id=client_name --acraserver_connection_host=acra.server.host

To point AcraConnector to AcraTranslator, configure an appropriate connection host/port/string and mode:

$GOPATH/bin/acra-connector --client_id=client_name --acratranslator_connection_host=acra.translator.host --mode=acratranslator

For security reasons, consider configuring your firewall to allow the connections only from legit AcraConnector IPs. If an attacker compromises your client application and AcraConnector, filtering IP addresses might prevent DoS.

AcraConnector CLI reference

--acraserver_api_connection_port
  Port of Acra HTTP API (default 9090)
--acraserver_api_connection_string
  Connection string to Acra's API like tcp://x.x.x.x:yyyy or unix:///path/to/socket
--acraserver_connection_host
  IP or domain to AcraServer daemon
--acraserver_connection_port
  Port of AcraServer daemon (default 9393)
--acraserver_connection_string
  Connection string to AcraServer like tcp://x.x.x.x:yyyy or unix:///path/to/socket
--acraserver_securesession_id
  Expected id from AcraServer for Secure Session (default "acra_server")
--acraserver_tls_transport_enable
  Use tls to encrypt transport between AcraServer and AcraConnector/client
--acraserver_transport_encryption_disable
  Use raw transport (tcp/unix socket) between acraserver and acraproxy/client (don't use this flag if you not connect to database with ssl/tls
--acratranslator_connection_host
  IP or domain to AcraTranslator daemon (default "0.0.0.0")
--acratranslator_connection_port
  Port of AcraTranslator daemon (default 9696)
--acratranslator_connection_string
  Connection string to AcraTranslator like grpc://0.0.0.0:9696 or http://0.0.0.0:9595
--acratranslator_securesession_id
  Expected id from AcraTranslator for Secure Session (default "acra_translator")
--client_id
  Client ID
--config_file
  path to config
-d  Log everything to stderr
--dump_config
  dump config
--generate_markdown_args_table
  Generate with yaml config markdown text file with descriptions of all args
--http_api_enable
  Enable connection to AcraServer via HTTP API
--incoming_connection_api_port
  Port for AcraConnector HTTP API (default 9191)
--incoming_connection_api_string
  Connection string like tcp://x.x.x.x:yyyy or unix:///path/to/socket (default "tcp://127.0.0.1:9191/")
--incoming_connection_port
  Port to AcraConnector (default 9494)
--incoming_connection_prometheus_metrics_string
  URL which will be used to expose Prometheus metrics (use <URL>/metrics address to pull metrics)
--incoming_connection_string
  Connection string like tcp://x.x.x.x:yyyy or unix:///path/to/socket (default "tcp://127.0.0.1:9494/")
--jaeger_agent_endpoint
  Jaeger agent endpoint that will be used to export trace data (default "127.0.0.1:6831")
--jaeger_basic_auth_password
  Password used for basic auth (optional) to jaeger
--jaeger_basic_auth_username
  Username used for basic auth (optional) to jaeger
--jaeger_collector_endpoint
  Jaeger endpoint that will be used to export trace data (default "127.0.0.1:14268")
--keys_dir
  Folder from which will be loaded keys (default ".acrakeys")
--logging_format
  Logging format: plaintext, json or CEF (default "plaintext")
--mode
  Expected mode of connection. Possible values are: AcraServer or AcraTranslator. Corresponded connection host/port/string/session_id will be used. (default "AcraServer")
--tls_acraserver_sni
  Expected Server Name (SNI) from AcraServer
--tls_auth
  Set authentication mode that will be used in TLS connection with AcraServer/AcraTranslator. Values in range 0-4 that set auth type (https://golang.org/pkg/crypto/tls/#ClientAuthType). Default is tls.RequireAndVerifyClientCert (default 4)
--tls_ca
  Path to root certificate which will be used with system root certificates to validate AcraServer's certificate
--tls_cert
  Path to certificate
--tls_key
  Path to private key that will be used in TLS handshake with AcraServer
--tracing_jaeger_enable
  Export trace data to jaeger
--tracing_log_enable
  Export trace data to log
--user_check_disable
  Disable checking that connections from app running from another user
-v  Log to stderr all INFO, WARNING and ERROR logs

Changing configuration options for AcraConnector

You can run with the options from config or you can use the CLI parameters described above.

Copy the example config ($REPO_DIR/configs/acra-connector.yaml or from GOPATH $GOPATH/src/github.com/cossacklabs/acra/configs/acra-connector.yaml) or generate the config yourself:

$GOPATH/bin/acra-connector --dump_config --config_file=<path_to_config_file/acra-connector.yaml>

and run:

$GOPATH/bin/acra-connector --config_file=<path_to_config_file/acra-connector.yaml>

AcraConnector will start listening on port 9494, and will attempt to connect to AcraServer on port 9393.

Configuring incoming connections
--incoming_connection_host
--incoming_connection_port
--incoming_connection_string

You can change the host and port on which AcraConnector will be listening to receive the data from AcraServer/AcraTranslator:

For example, setting the listening port to 5432:

$GOPATH/bin/acra-connector --client_id=client_name --acraserver_connection_host=acra.server.host --incoming_connection_port=5432
--http_api_enable

AcraServer supports two modes: binary data and HTTP API commands.

Start AcraConnector with HTTP API port open to accept API requests for AcraWebConfig and new Zones' requests by sending a request to http://127.0.0.1:9191/getNewZone. With the former, you will receive a json file {"id": "zoneid", "public_key": "base 64 encoded zone public key"} in the response.

$GOPATH/bin/acra-connector --client_id=client_name --acraserver_connection_host=acra.server.host --http_api_enable
--incoming_connection_api_port
--incoming_connection_api_string

You can change the port and host for listening to AcraServer responses made through HTTP API:

$GOPATH/bin/acra-connector --client_id=client_name --acraserver_connection_host=acra.server.host --http_api_enable --incoming_connection_api_port=12345

If you're running AcraConnector from the same machine that contains your app's code (and remember - you must run AcraConnector from a different user!), you can connect to locahost:5432 from your app to have a normal database experience.

Configuring connections to AcraServer
--acraserver_connection_string
--acraserver_connection_port
--acraserver_api_connection_string
--acraserver_api_connection_port

You might specify custom binary data and API connection strings (or a host/port pair) to connect to AcraServer:

$GOPATH/bin/acra-connector --client_id=client_name --acraserver_connection_host=127.0.0.1 --acraserver_connection_port=10000
--acraserver_securesession_id
--acraserver_tls_transport_enable

Set SecureSessionId to use Themis Secure Session between AcraConnector and AcraServer.

Alternatively, you can use TLS: set acraserver_tls_transport_enable to true to enable using TLS between AcraConnector and AcraServer. AcraConnector requires the path to keys and certificates: tls_ca is the path to root certificate; tls_cert path to the TLS server certificate and tls_key is the path to the TLS server key.

Configuring connections to the database
--tls_acraserver_sni

If the client application is connecting to the database with SSL enabled (i.e. sslmode values either allowed or required), you should define tls_acraserver_sni as the database hostname that AcraConnector checks during TLS handshake on the connection attempt.

Configuring connections to AcraTranslator
--mode=acratranslator
--acratranslator_connection_host
--acratranslator_connection_port
--acratranslator_connection_string

To connect to AcraTranslator, you should specify the exact mode and provide the connection parameters:

$GOPATH/bin/acra-connector --client_id=client_name --mode=acratranslator --acratranslator_connection_host=127.0.0.1 --acratranslator_connection_port=10000
Other important configuration options:
--keys_dir

Sets a custom key directory location. By default, AcraConnector is looking for keys in .acrakeys, but you can specify another location.

--disable_user_check

Turns off the test that checks if AcraConnector has started from under a user different from the one running the app. This option was only added to simplify your tests!

$GOPATH/bin/acra-connector --client_id=client_name --acraserver_connection_host=127.0.0.1 --disable_user_check

To change the logging format to plaintext, CEF, or json, use:

--logging_format      

Read more about analysing logs here.

AcraWriter

After you have configured AcraConnector, your application can keep using your database handling code as before — all the extra work will now be taken care of by Acra's components.

AcraWriter is a library for your code that can be used anywhere within your app whenever you need to encrypt sensitive records.

Under the hood, it is basically the Themis library generating AcraStructs with the keys you've provided to AcraWriter.

To start protecting your data, pick a place in your code where you would like to integrate AcraWriter (or any self-made AcraStruct encrypter).

You can use our libraries for Golang, Python, Ruby, PHP, Node.js.

Or you can take a look at our examples: Golang, Python, Ruby, PHP, Node.js.

If you plan to use Acra with: - PostgreSQL — use BYTEA binary type; - MySQL — use mysql binary type.

Acra example usage scenario

You can encrypt the sensitive data by generating AcraStructs with AcraWriter anywhere across your app. Send INSERTS/UPDATES to the database either through AcraConnector or directly via a separate connection.

You can decrypt AcraStructs by sending database requests to AcraConnector and receive responses that went through AcraServer.

Acra does not dictate a way of storing database requests in the database. If your code is a monolith running on one server as one service, it might make sense to point the writes into the same connections as reads, i.e. AcraConnector.

However, if your code is a huge family of microservices where some of them just write data and some just read it, it is essential to be able to do it the way you want it. You can write AcraStructs generated by AcraWriter directly into the database so that you don't need to carry all the AcraConnector keys with every piece of code on every machine.

For example, we're aware of a setup where AcraStructs are sent down the Kafka stream and emerge in the database at some further point in the future.

Client-side without Zones

This is an instruction for PostgreSQL.

Writes

When you need to store some sensitive data:

  • before doing INSERT/UPDATE on data, you should use AcraWriter libraries specific to your language, encrypting sensitive data into AcraStruct with AcraServer's public key with suffix _storage (some_client_id_storage.pub) that is different from the keypair used by AcraConnector for communicating with AcraServer.
  • then you can proceed with doing INSERT/UPDATE, either through AcraConnector or directly.

Reads

Read requests are the same as the regular reads, but their remote address is AcraConnector instead of your database listener. A typical PostgreSQL connection address would look like this: postgresql://user:password@127.0.0.1:9494/db_name.

Note for PostgreSQL: In the current layout, you aren't required to use SSL when trying to connect to the database. Transport protection for sensitive (decrypted) data is provided between AcraServer and AcraConnector via Secure Session. However, you can setup using SSL connection, too.

Client-side with Zones

Description of the client-side with Zones for PostgreSQL.

Zone-based encryption is the best way to cryptographically compartmentalise the data that comes from different sources, following the user's choice. Cryptographically, the Zones are a mechanism for modifying the AcraServer key according to the Zone Id and for mapping Zone's private keys during the decryption. In theory, you could use a new key for every record. However, it would introduce an unwanted overhead on AcraServer's performance.

Writing

To write AcraStruct with Zone keys, you need to:

  • generate the Zone keys on AcraServer,
  • acquire Zone key and Zone Id from AcraServer,
  • encrypt data using this key as AcraServer key in a standard AcraStruct process.

There are two ways you can generate the keys:

  1. First, make sure that you already generated ACRA_MASTER_KEY and out it into an environmental variable, or generate it like this:
export ACRA_MASTER_KEY=$(echo -n "My_Very_Long_Key_Phrase_ge_32_chars" | base64)

Read more about different types of keys used in Acra on Key Management page.

  1. Then build AcraAddZone utility in AcraServer from Acra repository:
go build acra-addzone
./acra-addzone

When you run it, you should get JSON object that looks something like this:

{"id":"DDDDDDDDQHpbUSOgYTzqCktp","public_key":"VUVDMgAAAC3yMBGsAmK/wBXZkL8iBv/C+7hqoQtSZpYoi4fZYMafkJbWe2dL"}

By decoding base64-wrapped public key, you get the binary Zone key, which can be used for generating AcraStructs.

  1. AcraConnector locally provides HTTP API to add Zones and return public keys on 127.0.0.1:9191/getNewZone. You can change the port via --incoming_connection_api_port argument.

Both approaches provide identical results.

Reading

Reading with Zones takes place just as it does without Zones. However, to point AcraServer to the exact Zone key, it has to use, you need to structure your query in such a way that AcraStruct is preceded by the corresponding Zone Id in its answer. For example, you may want to store the Zone Id in the preceding column and execute the query in the following way:

SELECT id, lastname, zone, data, datetime FROM table;

where zone contains Zone Id and data stands for AcraStruct.

If you keep the Zone Id in some other place (i.e. configuration file, other tables, or other database storage), you need to explicitly state it in the select, for example:

SELECT id, lastname, 'DDDDDDDDQHpbUSOgYTzqCktp'::bytea, data, datetime FROM table;

Note: It's worth mentioning that ::bytea conversion is necessary for PostgreSQL to work properly.

Building AcraWriter for your language

Acra uses Themis for performing cryptographic computations. AcraWriter is basically Themis + high level code that generates AcraStructs from input data and public key.

There are many languages / architectures supported by Themis, and eventually, most of them will support Acra, too.

Common steps for all languages

Install dependencies

sudo apt install git libssl-dev make build-essential

Install Themis

git clone https://github.com/cossacklabs/themis.git
cd themis
make
sudo make install
cd -

Get Acra sources

git clone https://github.com/cossacklabs/acra.git
cd acra

Building AcraWriter for Python

Before building AcraWriter for Python, make sure you've installed Themis as system library!

Installation

sudo apt install python-pip
pip install acrawriter

Testing

from pythemis.skeygen import GenerateKeyPair, KEY_PAIR_TYPE
from acrawriter import create_acrastruct

keypair = GenerateKeyPair(KEY_PAIR_TYPE.EC)
create_acrastruct(b'some data', keypair.export_public_key())

Output

'""""""""UEC2\x00\x00\x00-\xc2\xda\xbc\x03\x03`/\xc5\x8d.|DuX\xc4\x1c\xe5\x11\xd2>.\xc7.\xf3\xe5\xa5\xec\x83\xdbC\xd9\xdb/\x85K\x19\xe5 \'\x04&T\x00\x00\x00\x00\x01\x01@\x0c\x00\x00\x00\x10\x00\x00\x00 \x00\x00\x00\xb1o\xde\xbf\xcc\xa1\xe6\x13\x00MT)\x98\x95@Z?\xd1\xea\xa2n\xa1\xf7\x93\x9dD\xf4Gr\n?8J\x94\x1c$\x86\x91\xdc\x05^\x1d\xe0>\x18ru\xa4A\x04\xf9\x98X]\x13\x9c$\x97~\x145\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01@\x0c\x00\x00\x00\x10\x00\x00\x00\t\x00\x00\x00D\xe7\xafw\x82\xb3\xff"q3\x94\x17X\x14+\xe6e\x07\xecR>\xc9\x9d\xd4\xa8"T\xf5(\xcf:\x83\xe8\x82\x02\xcf\x16'

Examples

With Zones and without Zones.

Building AcraWriter for Ruby

Before building AcraWriter for Ruby, make sure you've installed Themis as system library!

Installation

gem install acrawriter

Test

gem install --user-install minitest rake
rake test

Output

...

# Running:

.

Finished in 0.003829s, 261.1501 runs/s, 0.0000 assertions/s.

1 runs, 0 assertions, 0 failures, 0 errors, 0 skips

Examples

With Zones and without Zones.

Building AcraWriter for Nodejs

Before building AcraWriter for NodeJS, make sure you've installed Themis as system library!

Installation

sudo apt install node npm
sudo apt install nodejs-legacy
sudo npm install -g npm
cd wrappers/nodejs
npm install nan
npm install

Testing

node test.js

Output

work

Examples

With Zones and without Zones.

Building AcraWriter for PHP

Before building AcraWriter for PHP, make sure you've installed Themis as system library!

Installation

sudo apt install php5 php5-dev
cd themis
sudo make phpthemis_install
# extension=phpthemis.so
sudo nano /etc/php5/cli/php.ini
# check that extension is available
php --ri phpthemis
cd -

Testing

php acra/wrappers/php/test.php

Output

work

Now use acra/wrappers/php/acrawriter.php in your project. In future acrawriter will contain composer.json and will be added to packagist.

Examples

With Zones and without Zones

Building AcraWriter for Go

Before building AcraWriter for Go, make sure you've installed Themis as system library!

Installation

sudo apt install golang
# leave your GOPATH or set some
export GOPATH=$HOME/work
go get github.com/cossacklabs/acra/acra-writer

Testing

go test github.com/cossacklabs/acra/acra-writer

Output

ok      github.com/cossacklabs/acra/acra-writer 0.112s

Examples

With Zones and without Zones

Building AcraWriter for C++

Before building AcraWriter for C++, make sure you've installed Themis as system library!

Dependencies

Additionally, install ThemisPP (Themis C++ wrapper) as system library from Themis source folder:

cd themis
make themispp_install
cd -

Installation

Grab acrawriter.hpp source file and add it to your project. Link themis and crypto libs.

Example of CMake file:

project(my_project)
set(CMAKE_CXX_STANDARD 14)

//path to acrawriter.hpp file
set(ACRAWRITER ../../wrappers/cpp)
include_directories(${ACRAWRITER})

set(
    SOURCE_FILES
    ${ACRAWRITER}/acrawriter.hpp
    main.cpp
    )

add_executable(my_project ${SOURCE_FILES})
target_link_libraries(my_project themis crypto)

C++ examples and tests

See CLion project example in examples/cpp for generating AcraStruct with and without Zones, and tests.

Building AcraWriter for iOS

Installation via CocoaPods

Update your Podfile with:

pod 'acrawriter'

iOS examples and tests

See iOS project example in examples/objc for generating AcraStruct with and without Zones, and decrypting them using AcraTranslator via HTTP API.

Building AcraWriter for Android

Installation

Installation is manual: just grab AcraWriter and AcraStruct java files into your project. AcraWriter depends on Themis, so you should build and install Themis for Android as well.

See step-by-step guide in examples/android_java/Readme.

Android examples and tests

See Android project example in examples/android_java for generating AcraStruct with and without Zones, and decrypting them using AcraTranslator via HTTP API.

Server side: AcraServer

How AcraServer works

AcraServer is a very important part of Acra — it is the server responsible for decrypting all the database responses and forwarding them back to clients.

How AcraServer works:

  • Upon the first AcraConnector connection, AcraServer does not automatically connect to the database and sits in the waiting mode instead. The type of the database is defined by setting postgresql_enable or mysql_enable fields to true. By default, AcraServer is trying to connect to PostgreSQL.
  • If the client application is connecting to the database where SSL is enabled (sslmode values either allowed or required), you should define tls_db_sni field for AcraServer. It indicates database hostname that AcraServer checks during the TLS handshake on the connection attempt.
  • When the first AcraConnector connection arrives, AcraServer initialises secure communication.
  • By default, communication is based on Secure Session. During the initialisation stage, AcraConnector sends its Id within the Secure Session protocol, whereas AcraServer tries to find the public key <id>.pub and private key <id>_server, which should be in .acrakeys or wherever the keys_dir parameter points to. If such keys are found, AcraServer is able to establish the connection.
  • Another option is to use TLS connection (it can be enabled by setting acraconnector_tls_transport_enable flag to true). AcraServer requires the path to keys and certificates: tls_ca is the path to root certificate; tls_cert is the path to TLS server certificate and tls_key is the path to TLS server key. Bear in mind that you need explicitly set the client_id as currently, AcraServer can only process zoneless connections when using a single client_id.
  • After a successful initialisation of the session, AcraServer creates a database connection via --db_host/--db_port parameters and starts forwarding all the requests coming from AcraConnector into the database. Upon receiving the answer, AcraServer attempts to decrypt zone_id and AcraStruct. If Zones are disabled, the stream only gets parsed for AcraStruct. If AcraStruct was not detected, the response of the database will be passed 'as is'.
  • If AcraStruct was recognised (with or without a Zone), AcraServer will attempt to unpack the AcraStruct and to decrypt the payload. After that, AcraServer will replace the AcraStruct with the decrypted payload, change the packet's length, and return the answer to the application via AcraConnector. Acra allows users to combine several Zones and Zone-dependent AcraStructs into one query — the only rule is that for each AcraStruct an attempt at decryption will be made (using the Zone key that corresponds to the latest preceding matching Zone_id).
  • If AcraServer detects a poison record within the AcraStruct's decryption stream, AcraServer will either shut down the decryption, run an alarm script, or do both, depending on the pre-set parameters (poison_run_script_file, poison_shutdown_enable).
  • Every incoming request to AcraServer is passed through AcraCensor (Acra's firewall). AcraCensor will pass allowed queries and return error on forbidden ones. Rules are configured and stored in yaml file.
  • The cycle loops and repeats ∞.

Getting started with AcraServer

1a. Launch using Docker (the fastest way to try AcraServer)

Note: Using Docker is recommended for testing purposes only. Please don't rely on Docker in real-life production settings.
Note: The following examples focus on using AcraServer, AcraConnector and AcraWriter with PostgreSQL, but Acra also supports MySQL.

Clone the Acra repository, build images and start Docker compose with PostgreSQL, AcraServer, AcraConnector, and Secure Session between them:

git clone https://github.com/cossacklabs/acra.git
make docker
docker-compose -f docker-compose.pgsql-nossl-server-ssession-connector.yml up

Okay, now the basic infrastructure is all set up, all components are connected, and keys are placed in the appropriate folder.

Now, move to step 2 – launch AcraServer.

1b. Manual launch

Note: Skip this if you've used the Docker method described above.

  • Install the dependencies: Themis cryptographic library:
git clone https://github.com/cossacklabs/themis.git
cd themis
make
sudo make install
  • Install AcraServer:
go get github.com/cossacklabs/acra/cmd/acra-server

Note: All the commands starting with 'go' are meant to be executed from the 'acra' folder (the folder with the repository code).

  • Install the key generation utility:
go get github.com/cossacklabs/acra/cmd/acra-keymaker
  • Use acra-keymaker to generate master key into the master.key file and assign it into the environment variable like this:
$GOPATH/bin/acra-keymaker --generate_master_key=master.key
export ACRA_MASTER_KEY=`cat master.key | base64`

Read more about different types of keys used in Acra on the Key Management page.

  • Generate transport keys and storage keys:
$GOPATH/bin/acra-keymaker --client_id=client_name --generate_acraserver_keys
$GOPATH/bin/acra-keymaker --client_id=client_name --generate_acrawriter_keys

The client_id of the key should be longer than 5 characters. It is used as an identifier for the Secure Session connection between AcraConnector and AcraServer.

The generated transport keypair client_name_server and client_name_server.pub will appear in .acrakeys (or anywhere you ask with --keys_output_dir=/path/to/dir argument). It's used to establish Secure Session connection between AcraConnector and AcraServer.

The keypair for AcraWriter has two keys: public, that is used on AcraWriter side to encrypt data into AcraStruct, and private, used by AcraServer to decrypt AcraStructs. Their names are: client_storage and client_storage.pub.

  • Exchange public keys:

  • You should take public key of AcraConnector client_name.pub and put it into the corresponding folder (.acrakeys) on AcraServer.

  • You should put AcraServer's transport public key (called client_name_server.pub) to AcraConnector's key folder (.acrakeys or anything you chose in --keys_output_dir).
  • You should embed AcraServer's storage public key (called client_storage.pub) into AcraWriter client library.

AcraServer reads decryption keys from the key folder and stores them in memory in encrypted form. It uses LRU cache to increase performance by keeping only the actively used keys in memory. The size of the LRU cache can be configured depending on your server's load.

2. Launching AcraServer

Launching AcraServer is a one-line command that takes many parameters :).

By default, AcraServer is ready to talk with AcraConnector and PostgreSQL database:

$GOPATH/bin/acra-server --client_id=client_name --incoming_connection_host=acra.connector.host --incoming_connection_port=9393 --db_host=postgresql.db.host --db_port=5432

AcraServer is just one part of the whole, read about the AcraConnector setup to put it all together.

AcraServer CLI reference

--acracensor_config_file
  Path to AcraCensor configuration file
--acraconnector_tls_transport_enable
  Use tls to encrypt transport between AcraServer and AcraConnector/client
--acraconnector_transport_encryption_disable
  Use raw transport (tcp/unix socket) between AcraServer and AcraConnector/client (don't use this flag if you not connect to database with ssl/tls
--acrastruct_injectedcell_enable
  Acrastruct may be injected into any place of data cell
--acrastruct_wholecell_enable
  Acrastruct will stored in whole data cell (default true)
--auth_keys
  Path to basic auth passwords. To add user, use: `./acra-authmanager --set --user <user> --pwd <pwd>` (default "configs/auth.keys")
--client_id
  Expected client ID of AcraConnector in mode without encryption
--config_file
  path to config
-d  Log everything to stderr
--db_host
  Host to db
--db_port
  Port to db (default 5432)
-ds
  Turn on http debug server
--dump_config
  dump config
--generate_markdown_args_table
  Generate with yaml config markdown text file with descriptions of all args
--http_api_enable
  Enable HTTP API
--incoming_connection_api_port
  Port for AcraServer for HTTP API (default 9090)
--incoming_connection_api_string
  Connection string for api like tcp://x.x.x.x:yyyy or unix:///path/to/socket (default "tcp://0.0.0.0:9090/")
--incoming_connection_close_timeout
  Time that AcraServer will wait (in seconds) on restart before closing all connections (default 10)
--incoming_connection_host
  Host for AcraServer (default "0.0.0.0")
--incoming_connection_port
  Port for AcraServer (default 9393)
--incoming_connection_prometheus_metrics_string
  URL (tcp://host:port) which will be used to expose Prometheus metrics (<URL>/metrics address to pull metrics)
--incoming_connection_string
  Connection string like tcp://x.x.x.x:yyyy or unix:///path/to/socket (default "tcp://0.0.0.0:9393/")
--jaeger_agent_endpoint
  Jaeger agent endpoint that will be used to export trace data (default "127.0.0.1:6831")
--jaeger_basic_auth_password
  Password used for basic auth (optional) to jaeger
--jaeger_basic_auth_username
  Username used for basic auth (optional) to jaeger
--jaeger_collector_endpoint
  Jaeger endpoint that will be used to export trace data (default "127.0.0.1:14268")
--keys_dir
  Folder from which will be loaded keys (default ".acrakeys")
--keystore_cache_size
  Count of keys that will be stored in in-memory LRU cache in encrypted form. 0 - no limits, -1 - turn off cache
--logging_format
  Logging format: plaintext, json or CEF (default "plaintext")
--mysql_enable
  Handle MySQL connections
--pgsql_escape_bytea
  Escape format for Postgresql bytea data
--pgsql_hex_bytea
  Hex format for Postgresql bytea data (default)
--poison_detect_enable
  Turn on poison record detection, if server shutdown is disabled, AcraServer logs the poison record detection and returns decrypted data (default true)
--poison_run_script_file
  On detecting poison record: log about poison record detection, execute script, return decrypted data
--poison_shutdown_enable
  On detecting poison record: log about poison record detection, stop and shutdown
--postgresql_enable
  Handle Postgresql connections (default true)
--securesession_id
  Id that will be sent in secure session (default "acra_server")
--tls_auth
  Set authentication mode that will be used in TLS connection with Postgresql. Values in range 0-4 that set auth type (https://golang.org/pkg/crypto/tls/#ClientAuthType). Default is tls.RequireAndVerifyClientCert (default 4)
--tls_ca
  Path to root certificate which will be used with system root certificates to validate Postgresql's and AcraConnector's certificate
--tls_cert
  Path to tls certificate
--tls_db_sni
  Expected Server Name (SNI) from Postgresql
--tls_key
  Path to private key that will be used in TLS handshake with AcraConnector as server's key and Postgresql as client's key
--tracing_jaeger_enable
  Export trace data to jaeger
--tracing_log_enable
  Export trace data to log
-v  Log to stderr all INFO, WARNING and ERROR logs
--zonemode_enable
  Turn on zone mode

AcraServer Configuration files

All these parameters are available in a yaml-file which you can feed to the system through --config_file CLI argument. You can take a look at an example configuration file in config/acra-server.yaml. You can copy the example config ($REPO_DIR/configs/acra-server.yaml or from GOPATH $GOPATH/src/github.com/cossacklabs/acra/configs/acra-server.yaml). You can also generate one yourself. Use:

$GOPATH/bin/acra-server --dump_config > acra-server.yaml

and then run:

$GOPATH/bin/acra-server --config_file=acra-server.yaml

AcraCensor — Acra’s firewall

AcraCensor is a separate SQL firewall module for AcraServer that checks every incoming request. AcraCensor will let the allowed queries through to the database and return an error on forbidden ones. AcraCensor can be configured for your exact use case, allowing to setup a whitelist and a blacklist for tables, query patterns, and exact queries. AcraCensor works with both MySQL and PostgreSQL protocols. It is built on top of this SQL parser, which works slightly better with MySQL protocol.

AcraServer starts AcraCensor if a configuration rules' file is present. You should set up the path for AcraCensor's configuration file using --acracensor_config_file parameter in acra-server.yaml configuration. The acra/configs folder stores an example of the configuration settings for a firewall created based example from configs/acra-censor.example.yaml. You can use this example config as a template for your own, using the database of your choice.

Starting AcraCensor using exact configuration file:

$GOPATH/bin/acra-server --acracensor_config_file=configs/acra-censor.example.yaml --client_id=client_name --incoming_connection_host=acra.connector.host --db_host=postgresql.db.host 

Refer to AcraServer if you have problems with passing configuration parameters.

Configuring AcraCensor rules

This is an example configuration file for AcraCensor. The file describes several handlers. Their position in the file represents their priority of processing.

We suggest the next structure of handlers for better security: put query_capture first to log every query into the censor_log, then put query_ignore to ignore some database-specific control queries that occur when the database is starting and when the database drives check connections, then put the blacklist handler to block all the unwanted queries, and finally put the whitelist handler to allow some specific "normal" queries.

ignore_parse_error: false
handlers:
  - handler: query_capture
    filepath: censor_log

  - handler: query_ignore
    queries:
      # mysql queries AcraCensor will ignore
      - ROLLBACK
      - SET AUTOCOMMIT = 0
      - SHOW VARIABLES LIKE 'sql_mode'
      - show collation where `Charset` = 'utf8' and `Collation` = 'utf8_bin'

  - handler: blacklist
    queries:
      - select data_raw from plaintext_table
    tables:
      - acrarollback_output
    patterns:
      - "SELECT password from users %%WHERE%%"
      - "SELECT * from users where type=private"

  - handler: whitelist
    queries:
    tables:
      - pets
    patterns:
      - "SELECT * from users where type=public"

Next, see the explanation for each handler below, in order of importance.

Ignoring specific queries

It might be useful not to apply whitelist/blacklist rules to some exact queries, for example, the SQL queries on starting up the database.

We suggest looking into tests/acra-censor_configs to see which PostgreSQL and MySQL queries we've added to ignore list for running integration tests.

AcraCensor compares each input query with queries from the query_ignore list. If there is a match, the query is being allowed through and no blacklist/whitelist rules are applied.

Whitelist and blacklist

There are two basic settings types for AcraCensor:

  • The whitelist, which allows something specific and restricts/forbids everything else.
  • The blacklist, which allows everything and forbids something specific.

For each list, there are settings that regulate:

  • Queries,
  • Tables,
  • Patterns.

You can configure the whitelist and the blacklist separately or simultaneously. The order of priority for the lists is defined by their order in the configuration file. Priority of work for one of the lists is the following: queries, followed by tables, followed by rules.

Queries

Queries work in the following manner: the queries addressed to the database are alternately checked (lines are compared) for the queries matching the ones that the firewall needs to block/allow. If there is a match, the query addressing the database is blocked and a corresponding error is returned (if it's AcraCensor's blacklist at work) or the query is allowed through (if it matches a record in the whitelist of AcraCensor).

There are no restrictions for the query types (SELECT, INSERT, UNION, etc.).

Tables

Tables work in the following manner: the names of the tables are alternately extracted from the queries that address the database and each table is compared to the list of tables from the configuration file. If a matching table is detected, the query is blocked and the corresponding error is returned (if it is AcraCensor’s blacklist at work) or allowed (if it matches a record in the whitelist of AcraCensor).

There is a restriction for query types: only SELECT and INSERT types are allowed, other query types are not supported.

Rules

Rules work in the following manner: when a query addressing the database is checked, each rule (if any detected) is parsed alongside the incoming query. Parsing extracts the where node, tables, and columns from the body of the rule. If the rule and the query have matching nodes, the tables are checked. If matches for the tables are detected, columns are checked. When separate columns match, a decision to block (through blacklist) or to allow (through whitelist) the query is made.

Query type restriction applies: only the SELECT type is allowed. The rest of the queries are still not supported.

Patterns

Patterns work in the following manner: when a query addressing the database is checked, each pattern (if any detected) is parsed alongside the incoming query. Parsing extracts the WHERE nodes, tables, columns, and values from the body of the rule. If the rule and the query have matching nodes, the tables are checked. If matches for tables are detected, columns are checked. When separate columns match, a decision to block (through blacklist) or to allow (through whitelist) the query is made.

Most patterns work only for SELECT queries.

Supported patterns

This is a work-in-progress area, the list is a subject of fixes and updates.

VALUE pattern

%%VALUE%% represents literal value (string, binary, number, boolean), it's supported in the following cases: after WHERE, IN, ORDER BY, GROUP BY, BETWEEN.

  • WHERE ... VALUE
pattern
SELECT a FROM t WHERE ID = %%VALUE%%
matching queries
SELECT a FROM t WHERE ID = '123'
SELECT a FROM t WHERE ID = 35
non-matching queries
SELECT a, b FROM t WHERE ID = '123'
SELECT a FROM anothertable WHERE ID = '123
SELECT a FROM t WHERE ID > '123
  • BETWEEN VALUE and VALUE
pattern
SELECT a FROM t WHERE param BETWEEN %%VALUE%% and %%VALUE%%
matching queries
SELECT a FROM t WHERE param BETWEEN 1 and 3
SELECT a FROM t WHERE param BETWEEN "qwerty" and NULL
non-matching queries
SELECT a, b FROM t WHERE param BETWEEN 1 and 3
SELECT a FROM anothertable WHERE param BETWEEN 1 and 3
  • IN (VALUE, VALUE)
pattern
SELECT 1 FROM t WHERE b='qwe' and v IN (%%VALUE%%, %%VALUE%%, %%VALUE%%)
matching queries
SELECT 1 FROM t WHERE b='qwe' and v IN (1, 2, 3)
non-matching queries
SELECT 1 FROM t WHERE b='qwe' and v IN (1, 2)
SELECT 1 FROM t WHERE b='qwe' and v IN (1, 2, 3, 4, 5)
LIST_OF_VALUES pattern

%%LIST_OF_VALUES%% represents several values one by one.

pattern
SELECT 1 from t WHERE b='qwe' and v IN (%%LIST_OF_VALUES%%)
matching queries
SELECT 1 from t WHERE b='qwe' and v IN (1, 2, 3)
SELECT 1 from t WHERE b='qwe' and v IN (1, 'qwe', True, NULL, FALSE)
non-matching queries
SELECT 1 from anothertable WHERE b='qwe' and v IN (1, 2)
SUBQUERY pattern

%%SUBQUERY%% represents subquery expression starting from SELECT.

pattern
SELECT 1 from t WHERE a=%%SUBQUERY%%
matching queries
SELECT 1 from t WHERE a=(SELECT column1, column2 from table1 WHERE a=1)
SELECT 1 from t WHERE a=(SELECT column1, column2 from table1 WHERE a=1 union SELECT column1, column2 from table2)
non-matching queries
SELECT 1 from t WHERE a=2
SELECT 1 from anothertable WHERE a=(SELECT column1, column2 from table1 WHERE a=1)
WHERE pattern

%%WHERE%% represents one or more expressions after WHERE statement. This pattern works for SELECT/UPDATE/DELETE queries.

pattern
SELECT users FROM company %%WHERE%%
matching queries
SELECT users FROM company WHERE title = 'engineer'
SELECT users FROM company WHERE AGE IN ( 25, 27 )
SELECT users FROM company WHERE NAME LIKE 'Pa%'
SELECT users FROM company WHERE A=(SELECT age FROM company WHERE salary > 65000 limit 1) and B=(SELECT age FROM company123 WHERE salary > 1000 limit 1)
non-matching queries
SELECT users, cats FROM company WHERE a = 'someValue'
SELECT age FROM company4 WHERE age IS NULL
SELECT users FROM company4 INNER JOIN (SELECT age FROM company WHERE id = 1) AS t ON t.id=another_table.id WHERE AGE NOT IN (25, 27)
COLUMN pattern

%%COLUMN%% represents a column expression.

  • SELECT COLUMN
pattern
SELECT %%COLUMN%%, %%COLUMN%% FROM company
matching queries
SELECT users, cats FROM company
SELECT a, b FROM t WHERE ID = '123'
non-matching queries
SELECT users FROM company
SELECT users, cats, chameleons FROM company
SELECT users, cats, chameleons FROM company
SELECT users, cats FROM zoo
  • ORDER BY COLUMN
pattern
SELECT 1 FROM t ORDER BY %%COLUMN%%
matching queries
SELECT 1 FROM t ORDER BY age
SELECT 1 FROM t ORDER BY (case when f1 then 1 when f1 is null then 2 else 3 end)
non-matching queries
SELECT 1 FROM anothertable ORDER BY age
SELECT 1 FROM anothertable ORDER BY age DESC
SELECT pattern

%%SELECT%% represents a whole SELECT expression.

pattern
%%SELECT%%
matching queries
SELECT users, cats FROM company
SELECT dogs, chameleons FROM company
SELECT SUM(Salary) FROM Employee WHERE Emp_Age < 30
non-matching queries
INSERT INTO SalesStaff3 (StaffID, FullNameTbl) VALUES (X, M);
CREATE DATABASE stories
DROP TABLE users

This is a constantly updating list of useful configuration examples. Please see the examples we are using for integration testing in tests/acra-censor_configs and AcraCensor's unit tests. We will also appreciate your questions and Pull Requests.

Unparseable queries

Sometimes AcraCensor can't parse some particular SQL queries. It's potentially dangerous to send these queries to the database so AcraCensor blocks "unparseable" queries by default.

However, setting the configuration flag ignore_parse_error to true will push AcraCensor to ignore "unparseable" queries and send them to the database. Otherwise, all such queries should be added to the QueryIgnore Queries manually.

By default ignore_parse_error is false (check configs/acra-censor.example.yaml).

QueryCapture

The QueryCapture logger is a part of AcraCensor, it registers all the queries that pass through AcraServer and marks them as restricted when necessary (by default, all the queries are marked as "allowed"). The specially marked queries can then be passed to the existing processors (the whitelist and the blacklist) and you can set your own query management policy. In other words, you can log all unique queries, and then use this log as an input to whitelist/blacklist configurations.

To activate QueryCapture logger, you need to create AcraCensor configuration file (see the example configs/acra-censor.example.yaml), define query_capture as handler, and set path to the log file in a similar manner (AcraCensor will create a log file on its own)).

handler: query_capture
filepath: <path_to_file>

Don't forget to enable AcraCensor: set path to acra-censor.yaml in the AcraServer configuration file.

Exact values from query_capture log is being replaced for security purposes:

{"raw_query":"commit","_blacklisted_by_web_config":false}
{"raw_query":"update pg_settings set setting = :replaced1 where name = :replaced2","_blacklisted_by_web_config":false}
{"raw_query":"select data from test where id = :replaced1","_blacklisted_by_web_config":false}

AcraWebConfig

AcraWebConfig is a lightweight HTTP web server for managing AcraServer's certain configuration options.

Note: Acra components can be configured via config files, the main goal of AcraWebConfig is to provide the alternative, more simple, means for configuration.

The interface of AcraWebConfig consists of the following elements:

  • AcraServer Settings — managing AcraServer's settings.
  • AcraCensor (Coming soon) — Firewall/Intrusion detection settings.
  • Zones (Coming soon) — managing Zone keys.

AcraWebConfig Setup

go get github.com/cossacklabs/acra/cmd/acra_configui
  • AcraWebConfig uses HTTP API requests to get data from AcraServer and to change its settings. Using CLI-parameters (--destination_host, --destination_port), you can point it to AcraServer.
  • By default, AcraWebConfig accepts requests at localhost:8000. Use --incoming_connection_host and --incoming_connection_port to change this default preset.

You can create an SSH-tunnel to get access to the remote port:

ssh -L8000:localhost:8000 <acra-server>
  • To provide additional security, AcraWebConfig uses basic authentication. Its control option is --auth_mode. Possible values are:
  • auth_on — basic authentication is on (default).
  • auth_off_local — basic authentication is on for non-local requests (out of the range of 127.0.0.1/localhost).
  • auth_off — basic authentication is turned off.

Users/passwords are stored in an encrypted file and are managed by acra-authmanager utility:

go get github.com/cossacklabs/acra/cmd/acra-authmanager

To add user/password:

cd $GOPATH
$GOPATH/bin/acra-authmanager --set --user=<user> --password=<password>

To remove user/password:

cd $GOPATH
$GOPATH/bin/acra-authmanager --remove --user=<user>

Encrypted user/password-storage is stored in config/auth.keys, use --file to change this.

Note: We encrypt auth the storage with .acrakeys/auth_key that is auto-generated at the first run of acra-authmanager.

AcraWebConfig Usage

Open the AcraWebConfig HTTP endpoint in your browser.

In AcraServer settings you can save settings — AcraServer will be gracefully restarted while applying new options via a config file. AcraWebConfig rewrites config file with new values.

Note: If you want to use AcraWebConfig's AcraServer settings you should avoid using command line options for AcraServer as they have higher priority than the config file.

Logging in Acra

Logging configuration

Starting with version 0.77.0, Acra supports three different logging formats: plaintext, CEF, JSON.

Acra logs are compatible with various external log analysing tools (like ELK stack) and SIEM systems.

Logging mode and verbosity level can be configured for AcraServer, AcraConnector, AcraTranslator and AcraWebConfig in the corresponding yaml files or passed as CLI parameter.

Possible meanings for logging_format are: plaintext or cef or json (case independent). Verbosity level is enabled by changing v to true.

To see an example of logging, check out the default configuration of AcraServer in acra-server.yaml.

Logging output

The default output is stderr. Examples of output:

plaintext:

Nothing special, bonus is color-formatted output when connected to terminal.

INFO[2018-04-10T20:03:59+03:00] Starting service                             
INFO[2018-04-10T20:03:59+03:00] Validating service configuration             
INFO[2018-04-10T20:03:59+03:00] Initialising keystore                        
INFO[2018-04-10T20:03:59+03:00] Selecting transport: use Secure Session transport wrapper 
INFO[2018-04-10T20:03:59+03:00] Start listening to connections. Current PID: 66579 
INFO[2018-04-10T20:03:59+03:00] Start listening API: unix:///tmp/acra_api_unix_socket_10004 
INFO[2018-04-10T20:03:59+03:00] Start listening connection: unix:///tmp/unix_socket_10003 
INFO[2018-04-10T20:03:59+03:00] Got new connection to acra-server:
INFO[2018-04-10T20:03:59+03:00] Handle new connection                        
ERRO[2018-04-10T20:03:59+03:00] Can't wrap connection from acra-connector          code=538 error=EOF

JSON:

JSON log contains many useful fields like: product and versions, which define Acra components; timestamp in RFC3339 and unixTime to be compatible with a variety of log analysis tools.
The log levels could be debug/info/warning/error/panic.
Acra logs both error code and message if an error occurs (code and error fields respectively).

{"level":"info","msg":"Validating service configuration","product":"acra-server","timestamp":"2018-04-10T20:06:31+03:00","unixTime":"1523379991.311","version":"0.76"}
{"level":"info","msg":"Initialising keystore","product":"acra-server","timestamp":"2018-04-10T20:06:31+03:00","unixTime":"1523379991.311","version":"0.76"}
{"level":"info","msg":"Selecting transport: use Secure Session transport wrapper","product":"acra-server","timestamp":"2018-04-10T20:06:31+03:00","unixTime":"1523379991.311","version":"0.76"}
{"level":"info","msg":"Start listening to connections. Current PID: 67309","product":"acra-server","timestamp":"2018-04-10T20:06:31+03:00","unixTime":"1523379991.311","version":"0.76"}
{"level":"info","msg":"Start listening API: unix:///tmp/acra_api_unix_socket_10004","product":"acra-server","timestamp":"2018-04-10T20:06:31+03:00","unixTime":"1523379991.312","version":"0.76"}
{"level":"info","msg":"Start listening connection: unix:///tmp/unix_socket_10003","product":"acra-server","timestamp":"2018-04-10T20:06:31+03:00","unixTime":"1523379991.312","version":"0.76"}
{"level":"info","msg":"Got new connection to acra-server: ","product":"acra-server","timestamp":"2018-04-10T20:06:31+03:00","unixTime":"1523379991.386","version":"0.76"}
{"level":"info","msg":"Handle new connection","product":"acra-server","timestamp":"2018-04-10T20:06:31+03:00","unixTime":"1523379991.386","version":"0.76"}
{"code":538,"error":"EOF","level":"error","msg":"Can't wrap connection from acra-connector","product":"acra-server","timestamp":"2018-04-10T20:06:31+03:00","unixTime":"1523379991.387","version":"0.76"}

CEF:

CEF logger is compliant with the CEF protocol revision 16 from July 22, 2010, except the fact that Acra is using custom Dictionary Extension fields (like unixTime) because we found them to be very useful for log analysis.
Pay attention that severity levels range from 0 (debug level) to 10 (panic level), see severity mapping function in logging/cef_formatter.

CEF:0|cossacklabs|acra-server|0.76|100|Validating service configuration|1|unixTime=1523380315.046
CEF:0|cossacklabs|acra-server|0.76|100|Initialising keystore|1|unixTime=1523380315.046
CEF:0|cossacklabs|acra-server|0.76|100|Selecting transport: use Secure Session transport wrapper|1|unixTime=1523380315.046
CEF:0|cossacklabs|acra-server|0.76|100|Start listening to connections. Current PID: 68099|1|unixTime=1523380315.046
CEF:0|cossacklabs|acra-server|0.76|100|Start listening connection: unix:///tmp/unix_socket_10003|1|unixTime=1523380315.047
CEF:0|cossacklabs|acra-server|0.76|100|Start listening API: unix:///tmp/acra_api_unix_socket_10004|1|unixTime=1523380315.047
CEF:0|cossacklabs|acra-server|0.76|100|Got new connection to acra-server:|1|unixTime=1523380315.133
CEF:0|cossacklabs|acra-server|0.76|100|Handle new connection|1|unixTime=1523380315.134
CEF:0|cossacklabs|acra-server|0.76|538|Can't wrap connection from acra-connector|6|error=EOF unixTime=1523380315.134
CEF:0|cossacklabs|acra-connector|0.76|100|Got new connection to acra-connector:|1|unixTime=1523380315.167
CEF:0|cossacklabs|acra-server|0.76|100|Got new connection to acra-server:|1|unixTime=1523380315.168

Logging error codes

Most errors logged by Acra have their own error code. You can find the table of error codes in logging/event_codes.go.

We plan to use unique event codes for every system event, not only for errors, but these updates are not ready yet. Currently, the default event code is 100 for all events (except errors).

Implementing a new logging format

If you think that Acra is missing some handy logging formats, feel free to open an Issue and describe why you think it's important. Alternatively, you can add the required log formatter (logging folder is a good place to start) and open a Pull Request.
We are grateful for contributions.

Metrics in Acra

Starting with version 0.83.0, Acra supports tracking and exporting of the basic metrics of AcraServer, AcraConnector, and AcraTranslator to Prometheus.

Metrics configuration

If incoming_connection_prometheus_metrics_string is set, Acra will generate specific metrics (time of connection life, time of processing requests, AcraStruct decryption counters) and push them to Prometheus. This parameter can be configured for AcraServer, AcraConnector, and AcraTranslator in the corresponding yaml files or passed as a CLI parameter.

To see an example of collecting and displaying metrics, check out the Acra Engineering Demo that illustrates integration of Acra-protected infrastructure with Prometheus and Grafana dashboards.

Tracing in Acra

Starting with version 0.84.0, Acra supports tracing with OpenCensus. AcraServer, AcraConnector and AcraTranslator track every request from client application to the database and back. Each client request has a unique traceID, that helps measure how much time it needs to perform a certain data processing functions (ie. checking the requests via AcraCensor, encrypting data, decrypting AcraStructs, etc.).

Tracing configuration

If trace_log_enable is set, Acra component will produce, collect, and log traces.

Traces can be exported to Jaeger: set up parameters for Jaeger endpoint and authentication (trace_jaeger_enable, jaeger_agent_endpoint, jaeger_basic_auth_password, jaeger_basic_auth_username, jaeger_collector_endpoint).

Tracing-related parameters can be configured for AcraServer, AcraConnector, and AcraTranslator in the corresponding yaml files or passed as CLI parameter.

Acra Engineering Demo has examples of using and exporting traces.

Protecting data using Acra

Protecting data in SQL databases using AcraWriter and AcraServer

Acra Server: simplified architecture

This is what the process of encryption and decryption of the data in a database looks like:

  • Your application encrypts some data through AcraWriter by generating an AcraStruct using Acra storage public key and then updates the database. AcraStructs generated by AcraWriter can't be decrypted by it — only the Acra's server side has the right keys for decryption.
  • To retrieve the decrypted data, your application talks to AcraServer. It is a server-side service that works as database proxy: it sits transparently between your application and the database, listens silently to all the traffic that's coming to and from the database.
  • AcraServer monitors the incoming SQL requests and blocks the unwanted ones using the built-in configurable firewall AcraCensor. AcraServer only sends allowed requests to the database. Certain configurations for AcraServer can be adjusted remotely using AcraWebConfig web server.
  • Upon receiving the database response, AcraServer tries to detect the AcraStructs, decrypts them, and returns the decrypted data to the application.
  • AcraConnector is a client-side daemon responsible for providing encrypted and authenticated connection between the application and AcraServer. AcraConnector runs under a separate user/in a separate container and acts as middleware. AcraConnector accepts connections from the application, adds an extra transport encryption layer using TLS or Themis Secure Session, sends the data to AcraServer, receives the result, and sends it back to the application.

Protecting data in any file storage using AcraWriter and AcraTranslator

Acra Translator: simplified architecture

In some use cases, the application can store encrypted data as separate blobs (files that are not in a database, i.e. in a S3 bucket, local file storage, etc.). In this case, you can use AcraTranslator — a lightweight server that receives AcraStructs and returns the decrypted data.

This is what the process of encryption and decryption data using AcraTranslator looks like:

  • Your application encrypts some data using AcraWriter, generating an AcraStruct using Acra storage public key and puts the data into any file storage. AcraStructs generated by AcraWriter can't be decrypted by it — only the Acra's server side has the right keys for decrypting it.
  • To decrypt an AcraStruct, your application sends it to AcraTranslator as a binary blob via HTTP or gRPC API. AcraTranslator doesn’t care about the source of the data, it is responsible for holding all the secrets required for data decryption and for actually decrypting the data.
  • AcraTranslator decrypts AcraStructs and returns the decrypted data to the application.
  • To avoid sending the plaintext via an unsecured channel, AcraTranslator requires the use of AcraConnector, a client-side daemon responsible for providing encrypted and authenticated connection between the application and AcraServer. AcraConnector runs under a separate user/in a separate container and acts as a middleware. It accepts connections from the application, adds transport encryption layer using TLS or Themis Secure Session, sends data to AcraServer, receives the result, and sends it back to the application.

AcraTranslator and AcraServer are fully independent server-side components and can be used together or separately depending on your infrastructure.

When you don't need Acra

There are many cases when certain security model is useless, futile or even downright risky. This page is a growing collection of such cases, at least the ones we can think of.

When you don't need encryption

  • When you don't have anything to protect. However, even leaking e-mail+password might be a risk for your users - if not in your site, then somewhere else (it is useful to keep in mind that people frequently reuse passwords).

When Acra won't help

  • When you trust your existing protection methods and consider them sufficient.
  • Especially if they are long-proven by industry and exactly fit your use case.
  • When you can't afford yourself performance penalties Acra imposes (but then you're not really using languages Acra supports for development anyway, do you?).

Contacts and assistance

Contributing to Acra

Although Acra is an in-house-developed product by Cossack Labs Limited, it is still open-source, Apache 2-licensed software. This means you can hack it any way you want and contribute things back if you'd like to. As a software company, we focus on implementing features that are important to our products but would gladly spend some time on making Acra useful for everybody.

We highly encourage you to:

  • Report bugs and request features via GitHub Issues.
  • Report a bug and fix it with a patch via our GitHub Pull request (after creating a corresponding issue and leaving a link to the pull there).
  • Add something new to Acra. There is a certain design scheme according to which we'd like to keep developing Acra. If your contributions fall along with it, we'd be glad to accept some fundamental additions. It's better to discuss the subject using email before taking action (see below).

Every commit that goes into the master branch is audited and reviewed by somebody from Cossack Labs, so don't be surprised if it takes a bit long.

If you'd like to participate in the core development more deeply, get in touch.

Getting in touch

  • Requests/bugfixes/queries should go through GitHub Issues.
  • To get in touch with the developers, use this email at your own discretion :). Make sure you've tried reaching out through GitHub Issues first, before writing a direct email.
  • To talk to the business wing of Cossack Labs Limited, drop us an email.

Implementation assistance

If you would like to receive a one-time customized assistance in implementing Acra for the needs of your app/service, consider applying for our DataGuardian Assistance Program.