Acra documentation

Start here

Welcome to the official documentation for Acra — database security suite for cryptographic protection, intrusion detection, and greater security of distributed apps.

We also recommend checking out the Advanced and Tutorials & How Tos documentation sections.

If you want to skip ahead to practice, you can:

This documentation is for the open source version of Acra (Acra Community Edition or Acra CE), which is free to use. Please let us know in the Issues or via email if you stumble upon a bug, see a possible enhancement, or have a comment on security design.

There are also Pro and Enterprise versions of Acra available. Those versions provide better performance, redundancy/load balancing, come pre-configured with crypto-primitives of your choice (FIPS, GOST), integrate with key/secret management tools in your stack, and have plenty of utils and tools for your Ops and SREs to operate Acra conveniently – deployment automation, scaling, monitoring, and logging. Talk to us to get the full feature lists and a quote.

What is Acra

Acra – database security suite with protection for sensitive and personal data.

Acra provides selective encryption, multi-layered access control, database leakage prevention, and intrusion detection capabilities in a convenient, developer-friendly package. Acra was specifically designed for web and mobile apps with centralised data storage, including with distributed, microservice-rich applications.

Perfect Acra-compatible applications Typical industries
Web and mobile apps that store data in a centralised database or object storage
  • Healthcare
  • Finance
  • E-commerce
  • Critical infrastructures
  • Apps with > 1000 users
IoT apps that collect telemetry and process data in cloud
High-load data processing apps

Acra gives you tools for encrypting the data on the application's side into special cryptographic containers, storing them in the database or file storage, and then decrypting them in a secure compartmented area (separate virtual machine/container).

Cryptographic design of Acra ensures that no secret (password, key, etc.) leaked from the application or database will be sufficient for decryption of the protected data chunks that originate from it. Acra minimises the leakage scope, detects unauthorised behaviour, and prevents the leakage, informing operators of the incident underway.

Major security features of Acra

  • Cryptographic protection of data
  • during storage and transmission
  • Selective encryption
  • protect only the sensitive data to balance good security and performance
  • Key management tools
  • built-in tools for key distribution, key rotation, and compartmentalisation
  • Trust compartmentalisation
  • datastore and application components can be compromised, yet the data stays protected
  • Prevention of SQL injections
  • through a built-in SQL firewall
  • Intrusion detection system
  • early warning about suspicious behaviour
  • Searchable encryption ᵉ
  • available for Acra Enterprise users
  • Pseudonymisation
  • coming in the (nearest) future releases
  • Cryptographically protected audit log
  • Acra delivers different layers of defense for different parts and stages of the data lifecycle. This is what defence in depth is – an independent set of security controls aimed at mitigating multiple risks in case of an attacker crossing the outer perimeter.

    Acra was built with specific user experiences in mind:

    • Quick and straightforward integration of security instrumentation.
    • Easy to try: you can experience the full might of Acra without committing to its installation using Docker containers (you can also request access to Acra Live Demo to play around with Acra without installing a thing).
    • Compatibility with encryption-demanding compliances: Acra can run on certified crypto-libraries (FIPS, GOST).
    • Cryptographic protection of data: to compromise an Acra-powered app, the attacker will need to compromise a separate compartmented server, AcraServer - more specifically - its key storage and database; until AcraServer is compromised, the data is safe.
    • Cryptography is hidden under the hood: you're safe from the risk of selecting a wrong key length or algorithm padding.
    • Secure default settings to get you going.
    • Intrusion detection to give you an early warning about suspicious behaviour.
    • SQL injections prevention through a built-in SQL firewall.

    Acra is developer- and dev-ops-friendly and can be easily configured and automated using a configuration automation environment.

  • Secure default settings
  • your infrastructure is secure from the start without additional configuring
  • Cryptography is hidden
    under the hood
  • no risk of selecting the wrong key length or algorithm padding
  • Automation-friendly
  • easy to configure and automate
  • Quick infrastructure integration
  • via binary packages or Docker images
  • Easy client-side integration
  • client-side encryption libraries support ~11 languages
  • Code-less client-side integration
  • server-side encryption in AcraServer's Transparent proxy mode
  • Logging, metrics, tracing
  • throughout all Acra components;
    compatible with ELK stack, Prometheus, Jaeger
  • No vendor lock
  • rollback utilities to decrypt database into plaintext
  • Demos, examples, simulators
  • numerous web-based and Docker-based example projects available
  • Managed solution available
  • we can setup and manage Acra for you

    What Acra does

    Acra prevents plaintext data leakage from a database because the data is encrypted using context-dependent keys before it gets to the database. The database server has no means to decrypt the data and has no access to decryption keys. The data is encrypted with keys that are unique for each client application (or even data entity), making sure that even if one decryption key is leaked, only a minimum and amount of data can be decrypted as a result. AcraServer is the only entity responsible for data decryption and during the decryption, it combines the key from pieces – using the keys stored in key storage (i.e. Amazon KMS) and context from application.

    Acra prevents data leakage from application middleware because no keys accessible to middleware would be enough to decrypt the data and client applications can only encrypt the data. The data is transferred between the micro-services in encrypted form and it is put into the database in the end (encrypted). Cracking and tampering with the client application won't lead to data leakage.

    Acra is able to limit SQL injection and insider risk using SQL filtering and poison records. AcraCensor is a separate customisable SQL firewall module for AcraServer that checks every incoming request. AcraCensor will let the allowed queries through to the database and return an error on forbidden ones. Posion records are records specifically created in such a way that they wouldn't be queried by a user under normal circumstances, but will be included in the outputs of SELECT * requests. If the database response contains poison records, AcraServer will detect them and inform you of untypical behaviour.

    What's inside Acra

    Acra consists of a client-side library and several server-side applications. Server-side Acra components should run as separate services/servers.

    Client-side:

    Server-side:

    • AcraServer — a separate daemon that runs in an isolated environment (separate virtual machine or physical server). AcraServer is responsible for holding all the secrets required to decrypt the data and for actually decrypting this data.

    • AcraCensor — a firewall-like component that sits inside AcraServer and checks every SQL query to the database.

    • Acra also has AcraWebConfig — a lightweight HTTP web server for managing AcraServer's certain configuration options.

    • AcraTranslator – a lightweight daemon that runs in an isolated environment (a separate virtual machine or physical server), receives AcraStructs via HTTP or gRPC API, and returns the decrypted data. AcraTranslator doesn’t care about the source of the data, it is responsible for holding all the secrets required for data decryption and for actually decrypting the data. You can use AcraTranslator if you work with encrypted data stored as separate blobs (files, KV storages, NoSQL databases).

    Check out Protecting data using Acra to gain a better understanding of the way data protection processes work in Acra.

    Platforms and databases

    With the exception of AcraWriter, which is a library that developers integrate into the client-side application, all Acra components are distributed as binaries.

    Acra’s binaries are built for CentOS 7, Debian Stretch, Debian Jessie, Ubuntu Bionic, Artful, Xenial, Trusty.

    Acra currently supports PostgreSQL v9.4 - v11 and MySQL 5.7+.

    Design and special features

    Acra is a database protection suite with special features:

    • multi-layer access compartmentation to secrets for better security and early-stage intrusion detection;
    • special architecture for better developer usability and security of the in-app data;
    • single point of trust for minimal attack surface, inspecton of traffic that touches attack surface, and consistent risk management.

    One of the key design goals for Acra was to ensure that engineering behind the security guarantees is compatible with modern software development and reliability engineering practices, i.e.:

    • with the typical division between application and database,
    • with an arbitrary number of application hosts, consumer microservices, and database nodes,
    • high level of abstraction in the database usage patterns in the app code.

    Security design goals are rather simple (the guarantees are further explained and explored in Security design):

    • No amount of data and/or secrets leaked from AcraConnector and database server could be sufficient for decryption of the data stolen from the database. This should be achieved through cryptography rather than through general system design.
    • Sometimes even a well-behaved and trusted client (which is supposed to receive decrypted responses from AcraServer) might become compromised. However, any alteration of the app's data consumption in an unauthorised manner should be noticed, reported and prevented:
      • Acra provides a number of alarm triggers to detect potentially suspicious behaviour and requests from the application. Those triggers are configurable and allow creating unique sets of detection rules specific to the workflow and the app.
      • Acra provides a number of panic scripts, which either shut down the decryption, shut down the database connection, or provide other threat isolation methods.
      • These features are controlled programmatically and are built for modern automated environments.

    In some Acra's use cases, the application can store encrypted data as separate blobs (files that are not in a database - i.e. in the S3 bucket, local file storage, etc.). In this case, you can use AcraTranslator — a lightweight server that receives AcraStructs and returns the decrypted data.

    Acra is a continuously developing security tool. And as any proper security tool, it requires enormous human efforts for validation of the methods, code, and finding possible infrastructural weaknesses. We're constantly enhancing and improving Acra as we go. This is done to ensure that the provided security benefits are not rendered useless through implementation problems or increased complexity.

    Cryptography in Acra

    Acra relies on Themis cryptographic library, which implements high-level cryptosystems based on the best available open-source implementations of the most reliable ciphers.

    Themis' cryptographic services, used via Go wrapper in AcraServer and AcraConnector, and via corresponding language wrappers in AcraWriters, are high-level compositions of carefully chosen, well-known cryptographic primitives. Acra does not contain any self-made cryptographic primitives or obscure ciphers. To deliver its unique guarantees, Acra relies on the combination of well-known ciphers and a smart key management scheme.

    Currently, you can build Themis with using OpenSSL, LibreSSL, and Google's BoringSSL (a number of experimental build methods for LibSodium, BearSSL, and even CommonCrypto are available, too). They provide better security, use the implementations that have passed industry-specific compliance audits, and/or fit the hardware acceleration on your platform.

    The enterprise version of Acra can run on the certified crypto-libraries of your choice (i.e. FIPS, GOST), drop us an email to get a quote.

    Default crypto-primitive source OpenSSL
    Supported crypto-primitive sources ᵉ BoringSSL, LibreSSL, FIPS-compliant, GOST-compliant, HSM
    Storage encryption AES-256-GCM + ECDH
    Transport encryption TLS v1.2+ / Themis Secure Session
    KMS integration ᵉ Amazon KMS, Google Cloud Platform KMS, Hashicorp Vault, Keywhiz

    ᵉ — available in the Enterprise version of Acra only. Drop us an email to get a full list of features and a quote.

    How Acra helps with GDPR

    Acra can help reach better compliance with GDPR for your product(s) as it covers some of the demands of the articles 25,32,33, and 34. Read more in "Acra and GDPR compliance".

    Live Demo — try Acra without coding

    The fastest way to start out with Acra is to try it with Docker, but there are even easier ways to get acquainted with Acra and its capabilities.

    You try Acra with Acra Live Demo — an online playground that shows how typical web app (Python, PostgreSQL) is protected by Acra. There you can add encrypted data, try to run malicious SQL queries, and play with intrusion detection. Request the access to the demo by following the instructions under the link — we’ll need your email address to generate a database environment especially for you. And don’t worry, no unsolicited emails from us — we hate spam as much as you do.

    You can also try Acra and its capabilities with just one command, using specially prepared sets of Docker-based example projects that we call “demos”.

    Acra Live Demo (try Acra in one click)

    Acra Live Demo is a web-based demo that demonstrates the work of Acra (data protection, intrusion detection, rollback, and a number of other capabilities) in a typical web-infrastructure (deployed on our servers for your convenience).

    Acra Live Demo infrastructure contains: Django-based application, PostgreSQL database, AcraServer with AcraCensor, log monitor. Sensitive data is encrypted in a Django application, stored in a database, and decrypted through Acra.

    From the users' perspective, the website's work is unchanged. However, the data is securely protected so that even hacking the web application won't lead to data leakage.

    The available actions include:

    Requirements: Chrome, Firefox, or Safari browser.

    Note: We create a separate playground for each user, that's why we ask for your email; you'll receive an invitation email.

    Request an invitation to access to Acra Live Demo.

    Example Projects — see Acra in action

    You can see Acra and its features in a live infrastructure with just one command using specially prepared sets of Docker-based examples that we call "demos" (because they demo-nstrate the powerful capabilities of Acra).
    With just one command, you can try:

    Acra Engineering Demo (whole infrastructure in one command)

    Acra Engineering Demo illustrates the integration of Acra data protection suite into existing applications: Django-based and Ruby on Rails based web applications. We took well-known applications and added an encryption layer to them to illustrate how easily it could be done.

    Data protection is completely transparent for the users and requires minimal changes in the infrastructure.

    Developers and Ops friendly:

    • run a single command to deploy the application, database, Acra's components, logs, and dashboards;
    • read the code changes and see how little it takes to integrate encryption into the client application;
    • learn how Acra works by reading logs, monitoring metrics in Prometheus, and watching Grafana dashboards;
    • inspect Docker-compose files, architecture schemes, database tables, and much more.

    Requirements: Linux or macOS terminal.

    Run the Acra Engineering Demo!.

    Otherwise — proceed to Getting started to install and try Acra in your own infrastructure.

    AcraCensor Demo (SQL firewall in action)

    AcraCensor Demo is an example project that illustrates how to use AcraCensor as SQL firewall to prevent SQL injections. The target application is a well-known vulnerable web application OWASP Mutillidae 2.

    The demo project has a Docker compose file that runs the following web infrastructure:

    • OWASP Mutillidae web application,
    • Acra encryption suite.

    Acra works as a proxy between web and database. AcraCensor inspects every SQL query that runs from the web application to the database, and back.

    Run the AcraCensor Demo!.

    Otherwise — proceed to Getting started to install and try Acra in your own infrastructure.

    Acra Load Balancer Demo (High-availability infrastructure)

    Acra Load Balancer Demo is an example of high-availability infrastructure for Acra: Python app, HAProxy, PostgreSQL. It illustrates some of the many possible variants of building high availability and balanced infrastructures based on Acra data protection suite components, PostgreSQL, and Python application protected by Acra. In these examples, we used HAProxy – one of the most popular high availability balancers today.

    These stands were created only for demonstration purposes and structure of examples was intentionally simplified, HAProxy configuration is optimized NOT for performance, but for clarity of tests.

    This demo contains two examples:

    One Acra Server, two databases Two Acra Servers, two databases

    Requirements for Acra Load Balancer Demo: Linux or macOS terminal.
    Instructions for Acra Load Balancer Demo are in the repository's README.

    Run the Acra Load Balancer Demo!.

    What's next

    You can also play around with other pre-built applications protected by Acraor try AcraCensor SQL firewall example or try Acra Live Demo.

    Otherwise — proceed to Getting started to install and try Acra in your own infrastructure.

    Getting started

    Installing Acra is fairly simple:

    Requirements

    • A client application written in Ruby, Python, PHP, Node.js, or GoLang, which runs on a Linux machine.
    • The application talks to PostgreSQL or MySQL database via some language-specific framework (you can use SSL or non-SSL mode between AcraServer and AcraConnector for Acra; in non-SSL mode Secure Session will be used).
    • Themis and client application language wrapper are installed on your machine (in most cases it will be installed automatically during AcraWriter installation process).
    • Also, if you're installing Acra manually from the GitHub repository, you need to have Themis' dependencies and libssl-dev package installed, and libcrypto.so needs to be available in $PATH.

    Setup process

    • Generate keypair, which will identify this AcraConnector using acra-keymaker utility. AcraServer will recognise this AcraConnector instance by the (previously) specified <client_id>.
    • Deploy AcraConnector:
    • Add another system user on the machine that runs your application (for AcraConnector).
    • Deploy AcraConnector binaries to run under that user.
    • Distribute the keys: put the keys generated by acra-keymakerclient_id, client_id_server.pub, and client_id_storage.pub – into the corresponding .acrakeys folder.
    • Run AcraConnector as a service from under this user.
    • Point the database functions in your applications to talk to localhost:9494 (default port) or any other port you've chosen.
    • Deploy AcraServer to a separate machine:
    • Place the keys generated by acra-keymaker client_id.pub, client_storage, and client_id_server to the .acrakeys folder.
    • Copy AcraServer public key / Zone key to the .acrakeys folder belonging to:
      • AcraConnector's user,
      • Any user that runs AcraWriter code.
    • Integrate AcraWriter:
    • Make client_id_storage.pub accessible to AcraWriter code.
    • Integrate AcraWriter closely following the examples and using either AcraConnector (if available on this machine) or direct database connection.
    • Test everything:
    • Connect AcraConnector to AcraServer, send a regular request to your database through AcraConnector.
      Result: If you see the answer, AcraConnector and AcraWriter are able to connect and forward signals back and forth. It means that the network and the keys are fine.
    • Upon integrating AcraWriter into your code, try generating an AcraStruct from some payload.
      Result: If you succeed in running AcraWriter code, Themis library is installed properly and some of the keys are located in the expected place.
    • Write a row with AcraStruct into the database, either directly or through AcraConnector. Request this row through AcraConnector.
      Result: If you see decrypted payload in the response, the scheme works properly.
    • Thoroughly read the documentation.
    • Use Acra!
    • P.S. Feel free to raise an Issue or contact us at info@cossacklabs.com.

    Quickstart

    Note: Please read the documentation before starting out with Acra! There are some fundamental concepts that we highly advise you to understand before you proceed. Pay special attention to the Architecture.

    Also, remember that:

    • AcraServer and Acra's dependencies need to stay on a separate server / virtual machine.
    • AcraConnector can run in a container or under separate user on the same machine as the main application.

    Choose your weapon!;) There are different ways to start out with Acra. You can either:

    Using Acra with Docker

    Note: This way of using Acra is for testing purposes only and should never be used for real-life production environments.

    For a really quick start, we recommend trying Acra with Docker first. Using only two commands you will get all Acra components and database up and running, with a secure transport layer between them. We prepared several typical infrastructures to play with.

    The detailed instructions for installing and operating Acra with Docker can be found in the Trying Acra with Docker documentation page. Bear in mind that this way of using Acra is for testing purposes only and should never be used for real-life production environments.

    If you want to try Acra in your real-life setting, please use the installation instructions below and either install Acra from our official Cossack Labs repository or from the GitHub Acra repository.

    Installing Acra from the Cossack Labs repository

    Debian / Ubuntu

    1. Import the public key used by Cossack Labs to sign packages:

    wget -qO - https://pkgs.cossacklabs.com/gpg | sudo apt-key add -
    

    Note: If you wish to validate the key fingerprint, it is: 29CF C579 AD90 8838 3E37 A8FA CE53 BCCA C8FF FACB.

    2. You may need to install the apt-transport-https package before proceeding:

    sudo apt-get install apt-transport-https
    

    3. Add the Cossack Labs repository to your sources.list.
    You should add a line that specifies your OS name and the release name:

    deb https://pkgs.cossacklabs.com/stable/$OS $RELEASE main
    
    • $OS should be debian or ubuntu.
    • $RELEASE should be one of Debian or Ubuntu release names. You can determine this by running lsb_release -cs, if you have lsb_release installed.

    We currently build packages for the following OSs and RELEASE combinations:

    • CentOS 7,
    • Debian "Jessie" (Debian 8),
    • Debian "Stretch" (Debian 9),
    • Ubuntu Trusty Tahr (Ubuntu 14.04),
    • Ubuntu Xenial Xerus (Ubuntu 16.04),
    • Ubuntu Artful Aardvark (Ubuntu 17.10),
    • Ubuntu Bionic Beaver (Ubuntu 18.04).

    For example, if you are running Debian 9 "Stretch", run:

    echo "deb https://pkgs.cossacklabs.com/stable/debian stretch main" | \
      sudo tee /etc/apt/sources.list.d/cossacklabs.list
    

    4. Reload local package database:

    sudo apt-get update
    

    5. Install the package

    sudo apt-get install acra
    

    CentOS / RHEL / OEL

    Note: We only build RPM packages for x86_64.

    1. Import the public key used by Cossack Labs to sign packages:

    sudo rpm --import https://pkgs.cossacklabs.com/gpg
    

    Note: If you wish to validate the key fingerprint, it is: 29CF C579 AD90 8838 3E37 A8FA CE53 BCCA C8FF FACB.

    2. Create a Yum repository file for Cossack Labs package repository:

    wget -qO - https://pkgs.cossacklabs.com/stable/centos/cossacklabs.repo | \
      sudo tee /etc/yum.repos.d/cossacklabs.repo
    

    3. Install the package:

    sudo yum install acra
    

    That's it! You've successfully installed Acra from the Cossack Labs repository.

    Installing from GitHub

    These are the instruction for installation of AcraServer from the Cossack Labs' GitHub repository for Acra. You will need the same set of dependencies for each component.

    Installing the dependencies

    Install the dependencies for Acra:

    sudo apt-get install git golang libssl-dev make build-essential
    

    Set up your $GOPATH to some place where you will store the code.

    Install Themis

    git clone https://github.com/cossacklabs/themis.git
    cd themis
    make
    sudo make install
    

    Build the key generator and generate the keys

    go get github.com/cossacklabs/acra/cmd/acra-keymaker
    

    Then generate the keys and distribute them across the infrastructure.

    Remember to generate ACRA_MASTER_KEY and assign it to the environmental variable!

    Set up the environment for AcraServer

    On a separate machine, create a user for AcraServer:

    sudo useradd -m acra-server
    sudo su acra-server
    cd ~/
    export GOPATH=`pwd`
    

    Build AcraServer

    go get github.com/cossacklabs/acra/cmd/acra-server
    

    Place someid.pub, someid_storage and someid_server keys to .acrakeys directory for AcraServer. Now you can finally launch the AcraServer.

    Launching AcraServer

    Running AcraServer is easy, just point it to the database:

    $GOPATH/bin/acra-server --db_host=127.0.0.1
    

    If you see an error message "master key is empty", it means that you haven't generated ACRA_MASTER_KEY, please return to the Key Generation step.

    You can complement the command above with --db_port=5432 -v to adjust the listener port and add logs to get going quickly. For all the available CLI parameters, please refer to the corresponding section in How AcraServer works.

    You can also run with the options from config. You can copy the example config ($REPO_DIR/configs/acra-server.conf or from GOPATH $GOPATH/src/github.com/cossacklabs/acra/configs/acra-server.conf) or generate it yourself:

    $GOPATH/bin/acra-server --dump_config --config_file=acra-server.conf
    

    Also run:

    $GOPATH/bin/acra-server --config_file=acra-server.conf
    
    Logging
    $GOPATH/bin/acra-server --db_host=127.0.0.1 -v
    
    Explicit server_id
    $GOPATH/bin/acra-server --db_host=127.0.0.1 --server_id=some_name
    

    It is used as an identifier for Secure Session between AcraConnector and AcraServer and can also be used for checking the correctness of AcraConnector's setup.

    Listen on a custom port:
    $GOPATH/bin/acra-server --db_host=127.0.0.1 --incoming_connection_port=3000
    

    AcraServer listens to port 9393 by default.

    Switch PostgreSQL driver to escape_bytea (hex mode by default)
    $GOPATH/bin/acra-server --db_host=127.0.0.1 --pgsql_escape_bytea
    

    Before PostgreSQL 9, only escape_bytea mode was present. Now the hex mode is present by default during install.

    Additional features:

    AcraConnector

    Create a user for AcraConnector:

    sudo useradd -m acra-connector
    sudo su acra-connector
    cd ~/
    export GOPATH=`pwd`
    

    Build AcraConnector

    go get github.com/cossacklabs/acra/cmd/acra-connector
    

    Put someid and someid_server.pub keys into .acrakeys directory for AcraConnector.

    Run AcraConnector

    $GOPATH/bin/acra-connector --acraserver_connection_host=127.0.0.1 --client_id=someid -v
    

    If you see error message similar to "Configuration error: AcraConnector private key .acrakeys/someid doesn't exists", it means that you haven't generated keys or keys are place in a wrong folder, please return to the Key Generation step.

    AcraConnector is now listening on the localhost port 9494.

    Write an app with AcraWriter

    Read the introductory info about AcraWriter here, or read one of our tutorials: Django project, Rails project.

    Create a user for the app:

    sudo useradd -m application
    sudo su application
    cd ~/
    export GOPATH=`pwd`
    

    Copy AcraServer's public key with suffix _storage for the app. Write your own or adapt our examples to your settings.

    go get github.com/cossacklabs/acra/examples...
    nano src/github.com/cossacklabs/acra/examples/golang/src/example/examples.go
    or
    nano src/github.com/cossacklabs/acra/examples/ruby/example.rb
    or
    nano src/github.com/cossacklabs/acra/examples/nodejs/examples_with_zone.js
    

    Congratulations! Acra is ready.

    Architecture and data flow

    Acra components


    Acra-based application architecture using Themis SecureSession transport encryption.

    • AcraWriter — a client-side library that integrates into the app's workflow either through ORM or directly and provides the means for encryption of the sensitive data by generating AcraStructs (Go, Python, Ruby, PHP, Node.js).

    • AcraConnector — a client-side daemon that runs under a separate user / in a separate container, and which acts as a database listener that redirects all the queries to AcraServer and, upon receiving the results, feeds it back into the app. AcraConnector is an optional component, required in systems that are using extra transport encryption layer via Themis SecureSession.

    • AcraServer — a separate daemon that runs in an isolated environment (separate virtual machine or physical server). AcraServer is responsible for holding all the secrets required to decrypt the data and for actually decrypting this data.

    • AcraCensor – is a firewall-like component that sits inside AcraServer, and checks every SQL query to the database.

    • AcraWebConfig – a lightweight HTTP web server for managing AcraServer's certain configuration options.

    Using Themis SecureSession in Acra

    Data flow: reading data

    Acra's design values simplicity as much as security. Reading the data should not require much tweaking — just point the database address to AcraConnector and you're good to go:

    1. The application sends database request to AcraConnector (which pretends to be a database listener) using the standard PostgreSQL/MySQL protocol.
    2. AcraConnector sends that request to AcraServer using Secure Session (socket protection protocol).
    3. AcraServer sends a request to the database using the regular PostgreSQL/MySQL protocol and receives an answer. If this request is an SQL query, AcraServer checks with AcraCensor and blocks or allows the query to pass through depending on the AcraCensor settings.
    4. If AcraServer detects the presence of AcraStruct while parsing the answer, AcraServer will attempt to decrypt it and replace AcraStruct with the resulting plaintext in the answer, adjusting the size of the answer. If decryption fails, AcraServer just forwards the answer as is (in encrypted form).
    5. AcraServer returns the data to AcraConnector (with Secure Session), which in turn returns it to the application as if it had come from the database itself.

    Data flow: writing data

    Writing data requires:

    • Encrypting your sensitive piece of data with create_acrastruct, ORM, or a pre-created statement handler somewhere in your code. (Django example, Ruby on Rails example).
    • Actually writing the data:
    • Send the data via AcraConnector connection through AcraServer if you need a single entry point into your database.
    • Send the data directly into the database if you'd like to decouple writes from reads and have a lower quantity of sensitive secrets in this part of the code.

    The beauty of Acra's design is that you don't need AcraServer and all its components to write the data. When dealing with microservice-oriented architecture, you can push writes directly to the database where it's convenient or do it through AcraServer where it's present:

    Data flow with Zones: reading the data

    During the request, the application has to include Zone Id explicitly somewhere in the 'select' SQL query before the data that is to be decrypted.

    For example, table1 with sensitive data is in the data column and Zone Id is equal to client_id (stored in the client column). To make AcraServer decrypt everything properly, you need SQL statement like: SELECT client, data FROM table1;. You can also store the identifier elsewhere, separately from the data, and include it in a request statement, i.e.: SELECT '00fv8gIV41'::bytea, data from table1;, where '00fv8gIV41' is zone/client ID.

    Upon detecting ZoneID, AcraServer will use a corresponding private key to decrypt the next detected AcraStruct. AcraServer will ignore the AcraStruct if no Zone Id is detected before the arrival of AcraStruct.

    Data flow with Zones: writing data

    • Add a zone through Acra (either via console utility or using HTTP API through AcraConnector);
    • Encrypt your sensitive piece of data with create_acrastruct somewhere in the code, ORM, or in the prepared statement handler using zone key + Id as an argument (Django example, Ruby on Rails example);
    • Write the result to the database (similarly to a default zone).

    Using TLS/SSL in Acra

    Data flow: reading data

    If you rely on TLS/SSL in your system design and you are not using Zones, you might omit AcraConnector at all. Just point the database address to the AcraServer:

    1. The application sends database request to AcraServer (which pretends to be a database listener) using the standard PostgreSQL/MySQL protocol over SSL.
    2. AcraServer sends a request to the database using the regular PostgreSQL/MySQL protocol over SSL and receives an answer. If this request is an SQL query, AcraServer checks with AcraCensor and blocks or allows the query to pass through depending on the AcraCensor settings.
    3. If AcraServer detects the presence of AcraStruct while parsing the answer, AcraServer will attempt to decrypt it and replace AcraStruct with the resulting plaintext in the answer, adjusting the size of the answer. If decryption fails, AcraServer just forwards the answer as is (in encrypted form).
    4. AcraServer returns the data to the application as if it had come from the database itself.

    Data flow: writing data

    Writing data is the same as in the description above:

    • Encrypting your sensitive piece of data with create_acrastruct, ORM, or a pre-created statement handler somewhere in your code. (Django example, Ruby-on-Rails example).
    • Actually writing the data:
    • Send the data via AcraServer if you need a single entry point into your database.
    • Send the data directly into the database if you'd like to decouple writes from reads and have a lower quantity of sensitive secrets in this part of the code.

    The beauty of Acra's design is that you don't need AcraServer and all its components to write the data. When dealing with microservice-oriented architecture, you can push writes directly to the database where it's convenient, or do it through AcraServer where it's present:

    Zones

    Zones are the way to cryptographically compartmentalise records in an already-encrypted environment. Zones rely on different private keys on the server side.

    The idea behind Zones is very simple (yet quite specific to some use-cases): when we store sensitive data, it's frequently related to users / companies / some other binding entities. These entities could be described through some real-world identifiers, or (preferably) random identifiers, which have no computable relationship to the protected data.

    Acra uses this identifier to also identify, which key to use for decryption of a corresponding AcraStruct.

    How Zones work

    When analysing the database output stream, AcraServer searches for certain strings called Zone Ids (also ZoneId and zoneid/zone_id in code). They let Acra know that within this record a private key corresponding to the Zone Id should be used for the actual decryption of AcraStructs. It will only work if the user explicitly formats the output for Zone Id to precede the AcraStruct.

    Default zone

    If you extend the Zone concept to all Acra's behaviours, you'll see that a zoneless mode is actually using one default zone, which doesn't require a Zone Id (zone_id).

    Running AcraServer in Zone mode

    For AcraServer to be actually able to decrypt something, your AcraStructs-generating code must be written with Zones in mind.

    $GOPATH/bin/acra-server --db_host=127.0.0.1 --zonemode_enable
    

    More information about Zones

    • Data flow: Simplified data flow for Zone-based encryption/decryption processes.
    • Tuning Acra: How Zones affect certain tuning strategies for increased performance / security.
    • Using Zones on client: A practical How-To.

    AcraStruct

    Understanding AcraStruct

    AcraStruct is a container format. Before generating each AcraStruct, AcraWriter generates a keypair of throwaway keys that are used in the encryption process and then get zeroed (turned into zeros) in the memory once the process is over.

    AcraStruct = Begin_Tag + Throwaway_Public_Key + Encrypted_Random_Key + Data_Length + Encrypted_Data

    • Begin_Tag[8] — 8 bytes, header tag (can be changed);
    • Throwaway_Public_Key[45] — temporary public key generated by AcraWriter;
    • Encrypted_Random_Key[84] — encrypted Random Key by using SMessage (see next);
    • Data_Length[8] — length of the Encrypted data (see next);
    • Encrypted_Data[Data_Length] — payload encrypted with Random Key.

    Generating AcraStruct

    AcraWriter is used to generate AcraStruct, but the generation process is quite simple and can be implemented in any custom writer:

    • AcraWriter generates a keypair of throwaway keys using Themis EC key generator:
      Throwaway_Keypair = (Throwaway_Public_Key, Throwaway_Private_Key).
    • Generates Random Symmetric Key (RK), 32 bytes long.
    • Encrypts RK using Secure Message with Throwaway_Private_Key and Acra_Public_Key(or Zone key – see Zones):
      Encrypted_Random_Key = SMessage(RK, Throwaway_Private_Key, Acra_Public_Key).
    • Encrypts the payload with Secure Cell in Seal mode:
      Encrypted_Data = SCell(RK, payload).
    • Erases/fills with zeros memory area with the RK.
    • Calculates the encrypted payload length and transforms it to little endian 8 bytes long (Data_Length).
    • Connects attributes together as described in the original formula.
    • Erases/fills with zeros memory area with the Throwaway_Keypair and original payload.

    You can see how AcraStruct is generated in acra-writer.go, acrawriter.py, acrawriter.rb or acrawriter.php.

    Decrypting AcraStruct

    AcraServer, upon receiving and detecting valid AcraStruct, is able to:

    • Extract Throwaway Public Key (TPK).
    • Decrypt asymmetric envelope with TPK and Acra's Private Key (or Zone key).
    • Extract Random Key (RK) for Secure Cell container out of a decrypted envelope;
    • Decrypt Secure Cell, extract payload;
    • Reconstruct database answer in such a way that AcraStruct is replaced by decrypted data.

    Check implementation in decryptor/base/utils.go.

    Zone Ids

    Although not directly related to AcraStructs, Zone Ids are used for matching Zones to keys. The current format is 8 bytes "begin tag" + 16 symbols a-zA-Z.

    Storage Models

    The storage model modes used in Acra are WholeCell and InjectedCell.

    In WholeCell mode, AcraStruct represents a complete piece of data (i.e. database cell, a file, or some data transmitted into AcraTranslator). In this mode it is expected that the encrypted data will look something like:

    1. <AcraStruct>,
    2. <AcraStruct>,
    3. <AcraStruct>.

    In InjectedCell mode, AcraStruct is stored inside some piece of data. i.e. inside some file or in a database cell with a file inside, with AcraStruct as a piece of that file, not the whole file. In this mode, the encrypted data will look something like this:

    1. <Some AcraStruct data, some other AcraStruc data>,
    2. <AcraSctruct>,
    3. <File containing AcraStruct alongside other data>.

    The main difference between these modes lise in performance. In the WholeCell mode, AcraStructs are simply decrypted. In InjectedCell mode, AcraServer needs to find AcraStructs inside some other data element first and then decrypt them, which will obviously take longer. The process of searching for the necessary piece of data takes place as the data is going through Acra. Acra will look for AcraStructs in every piece of data in InjectedCell mode.

    Which mode should you choose?

    Let’s consider an example where we’re storing an email in a database and we’d like to encrypt it, “wrapping” it into an AcraStruct. We’d get a table:

    Email Column 2 Column 3
    Column2Value Column3Value
    Column2Value Column3Value

    In this case, an AcraStruct takes up a whole cell and we are trying to decrypt it as is, without searching for anything. But things can be different. In InjectedCell mode a binary MsgPack or protobuf could be stored in a table, and it is possible that in those data entities only one field is encrypted.

    In this case, such data entity wouldn’t be a single AcraStruct - it would be a data entity that contains an AcraStruct or several AcraStructs. This means that in the InjectedCell mode we stop assuming that the database cells can only contain complete AcraStructs. AcraStructs can be inside some other pieces of data and that’s where we’re starting to look for them. This consequently slows down the processing speed. However, not every task needs the InjectedCell mode, which is why one can switch between modes, depending on what you’re encrypting and how you’re storing it.

    By default, the WholeCell mode is active (--acrastruct_wholecell_enable). You can switch between the modes using startup parameters:

    --acrastruct_injectedcell_enable
        Acrastruct may be injected into any place of data cell
    --acrastruct_wholecell_enable
        Acrastruct will stored in whole data cell (default true)
    

    Please also see the CLI reference.

    Key management

    There are three types of keys used in Acra:

    • Transport keypair (see Secure Session) for protection of transport between AcraServer and AcraConnector, or AcraTranslator and AcraConnector. The two parties should exchange public keys, keeping the private keys to themselves.

    • Storage keypair: AcraWriter uses the public key for data encryption, AcraServer/AcraTranslator use private storage key for data decryption.

    • Authentication storage key for encryption/decryption credentials of AcraWebConfig users.

    Storage keys can be represented by either:

    • One keypair for each unique client_id(default Zone mode), which is used for encryption by AcraWriter and decryption by AcraServer/AcraTranslator (so, in fact, there can be many such keypairs, one for each client_id).
    • A set of Zone keys (multiple Zone mode). Each Zone represents a unique user or type of users and has corresponding encryption and decryption keys. Using Zones complicates an unwanted decryption: the attacker needs not only to get the decryption key, but to use a correct Zone Id as well.

    Key names and locations

    Each Acra component has to have its own key folder (by default it is .acrakeys) that contains a set of keys necessary for the correct functioning of the components:

    • AcraConnector needs to have its own transport private key and other party transport public key to establish a Secure Session connection. You should put transport public key of AcraServer or AcraTranslator into AcraConnector's key folder, depending on the connection type.

    • AcraServer should have:

      • AcraConnector's transport public key and AcraServer's own transport private key. This is necessary for accepting connections via Secure Session from the clients.
      • Storage private key(s) for decrypting AcraStructs that are generated by AcraWriter.
      • If AcraServer is running in Transparent proxy mode, it should possess the storage public key(s) for encrypting values from SQL queries to AcraStructs.
    • AcraTranslator should have:

      • AcraConnector's transport public key and AcraTranslator's own transport private key. This is necessary for accepting connections via Secure Session from the clients.
      • Storage private key(s) for decrypting AcraStructs generated by AcraWriter.
    • AcraWriter should have the storage public key(s). They are necessary for encrypting data into AcraStructs in such a way that would only be read by AcraServer/AcraTranslator.

    • If you're using AcraWebConfig HTTP server to configure the AcraServer remotely, the users' credentials are stored encrypted by authentication storage key and are decrypted on AcraServer upon logging in the user. AcraServer needs to have an authentication storage key.

    Keys table

    Purpose Private key Stays on Public key Put to
    Transport AcraConnector clientid AcraConnector clientid.pub -> AcraServer
    -> AcraTranslator
    Transport AcraServer clientid_server AcraServer clientid_server.pub -> AcraConnector
    Transport AcraTranslator clientid_translator AcraTranslator clientid_translator.pub -> AcraConnector
    AcraStruct encryption clientid_storage.pub -> AcraWriter
    -> AcraServer in Transparent proxy mode
    AcraStruct decryption clientid_storage AcraServer/
    AcraTranslator
    AcraStruct encryption with Zones zoneid_zone.pub -> AcraWriter
    -> AcraServer in Transparent proxy mode
    AcraStruct decryption with Zones zoneid_zone AcraServer/
    AcraTranslator
    Authentication storage key
    for AcraWebConfig's users
    auth_key AcraServer

    Generating all the Acra keys in one go

    We described many keys here and the private keys are (obviously) stored in an encrypted form. The main Acra key is ACRA_MASTER_KEY, which is used for decryption of private keys for every Acra service. Keep an eye on this key!

    1. Generating ACRA_MASTER_KEY

    Use acra-keymaker to generate master key into master.key file and assign it into the environment variable on the corresponding server:

    go install ./cmd/acra-keymaker
    $GOPATH/bin/acra-keymaker --generate_master_key=master.key
    export ACRA_MASTER_KEY=`cat master.key | base64`
    

    2. Generating transport and encryption keys

    After this, the master key will be used by acra-keymaker for encryption of private keys before those are written to the disk.

    Now, let's generate keypairs for Acra services:

    $GOPATH/bin/acra-keymaker --client_id=someid
    

    Carry out these operations on the machine running AcraServer to make sure that the private keys for Acra never leak outside from the server.

    The generator will generate and place 8 keys into the .acrakeys directory (you can change this with --keys_output_dir argument):

    .acrakeys/someid
    .acrakeys/someid.pub
    .acrakeys/someid_server
    .acrakeys/someid_server.pub
    .acrakeys/someid_storage
    .acrakeys/someid_storage.pub
    .acrakeys/someid_translator
    .acrakeys/someid_translator.pub
    

    Note: For acra to work properly, the key folders have to have proper permissions as set by acra-keymaker:

    • folder 700.
    • private keys 600.

    3. Key exchange

    3.1 Exchange transport keys between AcraConnector and AcraServer
    Component should contain keys named like these
    AcraConnector transport public key of AcraServer
    transport private key of AcraConnector
    .acrakeys/someid_server.pub
    .acrakeys/someid
    AcraServer transport public key of AcraConnector
    transport private key of AcraServer
    .acrakeys/someid.pub
    .acrakeys/someid_server
    3.2 Exchange transport keys between AcraConnector and AcraTranslator
    Component should contain keys named like these
    AcraConnector transport public key of AcraTranslator
    transport private key of AcraConnector
    .acrakeys/someid_translator.pub
    .acrakeys/someid
    AcraTranslator transport public key of AcraConnector
    transport private key of AcraTranslator
    .acrakeys/someid.pub
    .acrakeys/someid_translator
    3.3 Exchange AcraStruct encryption/decryption keys between AcraWriter and AcraServer/AcraTranslator
    Component should contain key named like this
    AcraWriter storage public key .acrakeys/someid_storage.pub
    AcraServer storage private key .acrakeys/someid_storage
    AcraTranslator storage private key .acrakeys/someid_storage
    3.4 Exchange AcraStruct Zone keys between AcraWriter and AcraServer/AcraTranslator

    Generating Zone keys is different from generating usual AcraStruct encryption keys. You should run acra-addzone on AcraServer to generate zone:

    go install ./cmd/acra-addzone
    $GOPATH/bin/acra-addzone
    

    When you run it, you should receive a JSON object that looks something like this:

    {"id":"DDDDDDDDQHpbUSOgYTzqCktp","public_key":"VUVDMgAAAC3yMBGsAmK/wBXZkL8iBv/C+7hqoQtSZpYoi4fZYMafkJbWe2dL"}
    

    By decoding a base64-wrapped public key, you get a binary Zone key which should be provided (placed into) to AcraWriter for generating AcraStructs. Zone keys are placed into the key folder and are named zoneid_zone for private key and zoneid_zone.pub for public key.

    Component should contain key named like this
    AcraWriter zone public key .acrakeys/zoneid_zone.pub
    AcraServer zone private key .acrakeys/zoneid_zone
    AcraTranslator zone private key .acrakeys/zoneid_zone

    Read more about using Zones in the AcraWriter guide.

    (Re)Generating only some of the keys

    If you want to update or re-generate only some particular keys, it can be easily done via CLI parameters for acra-keymaker:

    • --generate_acraconnector_keys – generates AcraConnector transport keypair;
    • --generate_acraserver_keys – generates AcraServer transport keypair;
    • --generate_acratranslator_keys – generates AcraTranslator transport keypair;
    • --generate_acrawriter_keys – generates data storage keypair to be split between AcraWriter and AcraServer/AcraTranslator;
    • --generate_acrawebconfig_keys – generates a symmetric key for encrypting credentials of AcraWebconfig' users;
    • --generate_master_key – generates new ACRA_MASTER_KEY, all private keys should be re-generated after changing master key.

    Client side: AcraConnector and AcraWriter

    AcraConnector and AcraWriter

    AcraConnector is a (separate) service running alongside your application — it pretends to be a database listener, relays all the requests to AcraServer, receives the responses, and returns them to the app, just like a normal database listener would do.

    To talk to AcraServer, you'll need to run AcraConnector on the same host as your application, in a separate container or as a separate user. You'll also need to route database requests to its address.

    To talk to AcraTranslator you'll need to do the same: run AcraConnector on the same host as your application, in a separate container or as a separate user, and use its URL as destination URL from your application.

    Why do we need a special piece of software to talk to your other piece of software?

    Acra needs a trusted agent on the application's side to protect the sensitive decrypted responses, to provide basic channel authentication, and to enforce certain behaviours.

    AcraConnector acts as a local proxy that receives requests from the app and returns decrypted answers. AcraConnector provides an encrypted and authenticated connection to AcraServer, which, in turn, fetches the response to the request from the database and decrypts all the data. AcraServer then returns the data to AcraConnector via a secure connection.

    AcraConnector works in a similar fashion with AcraTranslator, redirecting AcraStructs from application to AcraTranslator and delivering the decrypted response back.

    This enforces maximum secrecy, employing authentication that is easy to manage (pre-shared private keys), and requires minimal intervention into your code for successful implementation!

    Getting started with AcraConnector

    Method 1A. Launch AcraConnector using Docker (the fastest way to try AcraConnector)

    Note: Using Docker is recommended for testing purposes only. Please don't rely on Docker in real-life production settings.
    Note: The following examples focus on using AcraConnector and AcraWriter with PostgreSQL, but Acra also supports MySQL.

    Clone the Acra repository, build images, and start Docker compose with PostgreSQL, AcraServer, AcraConnector, and Secure Session between them:

    git clone https://github.com/cossacklabs/acra.git
    make docker
    docker-compose -f docker-compose.pgsql-nossl-server-ssession-connector.yml up
    

    Okay, now the basic infrastructure is all set up, all components are connected, and keys are distributed into the appropriate folder.

    Now, proceed to step launching AcraConnector. Or install AcraConnector manually:

    Method 1B. Manual launch

    Note: Skip this if you've used the Docker method described above.

    • Install dependencies: Themis cryptographic library:
    git clone https://github.com/cossacklabs/themis.git
    cd themis
    make
    sudo make install
    
    • Install AcraConnector:
    go get github.com/cossacklabs/acra/cmd/acra-connector
    

    Note: Hereinafter all the commands starting with 'go' are meant to be executed from the 'acra' folder (the folder with the repository code).

    • Install key generation utility:
    go get github.com/cossacklabs/acra/cmd/acra-keymaker
    
    • Use acra-keymaker to generate master key into master.key file and assign it into the environment variable like this:
    $GOPATH/bin/acra-keymaker --generate_master_key=master.key
    export ACRA_MASTER_KEY=`cat master.key | base64`
    

    Read more about the different types of keys used in Acra in the Key Management section of the documentation.

    • Generate the "client" proxy keypair:
    $GOPATH/bin/acra-keymaker --client_id=client_name --generate_acraconnector_keys
    

    The name of the key should be longer than 5 characters. It is also used as an identifier for the Secure Session connection between AcraConnector and AcraServer.

    The generated keypair client_name and client_name.pub will appear in .acrakeys (or anywhere you ask with --keys_output_dir=/path/to/dir argument).

    • Exchange public keys:

    • You should put public key client_name.pub into the corresponding folder (.acrakeys) on AcraServer.

    • You should put AcraServer's public key (called client_name_server.pub) to AcraConnector's key folder (.acrakeys or anything you chose in --keys_output_dir).

    Now, proceed to launching AcraConnector.

    Launching AcraConnector

    By default, AcraConnector is ready to talk to AcraServer. You need to use a one-line command:

    $GOPATH/bin/acra-connector --client_id=client_name --acraserver_connection_host=acra.server.host
    

    To point AcraConnector to AcraTranslator, configure an appropriate connection host/port/string and mode:

    $GOPATH/bin/acra-connector --client_id=client_name --acratranslator_connection_host=acra.translator.host --mode=acratranslator
    

    For security reasons, consider configuring your firewall to allow the connections only from legit AcraConnector IPs. If an attacker compromises your client application and AcraConnector, filtering IP addresses might prevent DoS.

    AcraConnector CLI reference

    The following parameters can be used for configuring AcraConnector from the command line or from the configuration file (see below). Both ways can be used (and are, in fact, used by most users). The typical flow is to:

    • run AcraConnector with CLI parameters while testing/debugging, then
    • save them to a file,
    • and the run production AcraConnector instances from the configuration file.
    --acraserver_api_connection_port
      Port of Acra HTTP API (default 9090)
    --acraserver_api_connection_string
      Connection string to Acra's API like tcp://x.x.x.x:yyyy or unix:///path/to/socket
    --acraserver_connection_host
      IP or domain to AcraServer daemon
    --acraserver_connection_port
      Port of AcraServer daemon (default 9393)
    --acraserver_connection_string
      Connection string to AcraServer like tcp://x.x.x.x:yyyy or unix:///path/to/socket
    --acraserver_securesession_id
      Expected id from AcraServer for Secure Session (default "acra_server")
    --acraserver_tls_transport_enable
      Use tls to encrypt transport between AcraServer and AcraConnector/client
    --acraserver_transport_encryption_disable
      Enable this flag to omit AcraConnector and connect the client app to AcraServer directly using raw transport (tcp/unix socket). For security reasons, at the very least please use TLS encryption (over tcp socket) between AcraServer and the client app.
    acraserver_transport_encryption_disable: false
    --acratranslator_connection_host
      IP or domain to AcraTranslator daemon (default "0.0.0.0")
    --acratranslator_connection_port
      Port of AcraTranslator daemon (default 9696)
    --acratranslator_connection_string
      Connection string to AcraTranslator like grpc://0.0.0.0:9696 or http://0.0.0.0:9595
    --acratranslator_securesession_id
      Expected id from AcraTranslator for Secure Session (default "acra_translator")
    --client_id
      Client ID
    --config_file
      path to config
    -d  Log everything to stderr
    --dump_config
      dump config
    --generate_markdown_args_table
      Generate with yaml config markdown text file with descriptions of all args
    --http_api_enable
      Enable connection to AcraServer via HTTP API
    --incoming_connection_api_port
      Port for AcraConnector HTTP API (default 9191)
    --incoming_connection_api_string
      Connection string like tcp://x.x.x.x:yyyy or unix:///path/to/socket (default "tcp://127.0.0.1:9191/")
    --incoming_connection_port
      Port to AcraConnector (default 9494)
    --incoming_connection_prometheus_metrics_string
      URL (tcp://host:port) which will be used to expose Prometheus metrics (use <URL>/metrics address to pull metrics)
    --incoming_connection_string
      Connection string like tcp://x.x.x.x:yyyy or unix:///path/to/socket (default "tcp://127.0.0.1:9494/")
    --jaeger_agent_endpoint
      Jaeger agent endpoint that will be used to export trace data (default "127.0.0.1:6831")
    --jaeger_basic_auth_password
      Password used for basic auth (optional) to jaeger
    --jaeger_basic_auth_username
      Username used for basic auth (optional) to jaeger
    --jaeger_collector_endpoint
      Jaeger endpoint that will be used to export trace data (default "127.0.0.1:14268")
    --keys_dir
      Folder from which will be loaded keys (default ".acrakeys")
    --logging_format
      Logging format: plaintext, json or CEF (default "plaintext")
    --mode
      Expected mode of connection. Possible values are: AcraServer or AcraTranslator. Corresponded connection host/port/string/session_id will be used. (default "AcraServer")
    --tls_acraserver_sni
      Expected Server Name (SNI) from AcraServer
    --tls_auth
      Set authentication mode that will be used in TLS connection with AcraServer/AcraTranslator. Values in range 0-4 that set auth type (https://golang.org/pkg/crypto/tls/#ClientAuthType). Default is tls.RequireAndVerifyClientCert (default 4)
    --tls_ca
      Path to root certificate which will be used with system root certificates to validate AcraServer's certificate
    --tls_cert
      Path to certificate
    --tls_key
      Path to private key that will be used in TLS handshake with AcraServer
    --tracing_jaeger_enable
      Export trace data to jaeger
    --tracing_log_enable
      Export trace data to log
    --user_check_disable
      Disable checking that connections from app running from another user
    -v  Log to stderr all INFO, WARNING and ERROR logs
    

    Changing configuration options for AcraConnector

    Reminder: Before changing the configuration file, don't forget to backup your current file.

    You can copy the original example config from GitHub, from the downloaded repository $REPO_DIR/configs/acra-connector.yaml, or from GOPATH $GOPATH/src/github.com/cossacklabs/acra/configs/acra-connector.yaml.

    Configuring Client ID
    --client_id
    

    ClientID is used to identify this AcraConnector instance (and client application it's connected to) in front of AcraServer. AcraConnector uses ClientID to read the appropriate transport encryption keys. If the ClientID is wrong, AcraConnector won't launch and will print an error message.

    Configuring incoming connections
    --incoming_connection_port
    --incoming_connection_string
    

    You can change the port on which AcraConnector will be listening to receive the data from AcraServer/AcraTranslator:

    For example, setting the listening port to 5432:

    $GOPATH/bin/acra-connector --acraserver_connection_host=acra.server.host --client_id=client_name --incoming_connection_port=5432
    
    Configuring connections to AcraServer's HTTP API
    --http_api_enable
    

    AcraServer supports two modes: binary data and HTTP API commands.

    Start AcraConnector with HTTP API port open to accept API requests for AcraWebConfig and new Zones' requests by sending a request to http://127.0.0.1:9191/getNewZone. With the former, you will receive a json file {"id": "zoneid", "public_key": "base 64 encoded zone public key"} in the response.

    $GOPATH/bin/acra-connector --client_id=client_name --acraserver_connection_host=acra.server.host --http_api_enable
    

    Incoming connection API port:

    --incoming_connection_api_port
    --incoming_connection_api_string
    

    You can change the port for listening to AcraServer responses made through HTTP API:

    $GOPATH/bin/acra-connector --client_id=client_name --acraserver_connection_host=acra.server.host --http_api_enable --incoming_connection_api_port=12345
    

    If you're running AcraConnector from the same machine that contains your app's code (and remember - you must run AcraConnector from a different user!), you can connect to locahost:5432 from your app to have a normal database experience.

    Configuring connections to AcraServer
    --acraserver_connection_string
    --acraserver_connection_port
    --acraserver_api_connection_string
    --acraserver_api_connection_port
    

    You might specify custom binary data and API connection strings (or a host/port pair) to connect to AcraServer:

    $GOPATH/bin/acra-connector --client_id=client_name --acraserver_connection_host=127.0.0.1 --acraserver_connection_port=10000
    

    Secure transport between AcraConnector and AcraServer:

    --acraserver_securesession_id
    --acraserver_tls_transport_enable
    --tls_auth
    --tls_ca
    --tls_cert
    --tls_key
    

    Set SecureSessionId to use Themis Secure Session between AcraConnector and AcraServer.

    Alternatively, you can use TLS: set acraserver_tls_transport_enable to true to enable using TLS between AcraConnector and AcraServer.

    tls_auth defines an authentication mode that will be used in TLS connection; possible values are taken from TLS configuration in Go: from 0 (don't request, don't check certificate) to 4 (request and verify the client certificate).

    AcraConnector requires the path to keys and certificates: tls_ca is the path to root certificate; tls_cert path to the TLS server certificate and tls_key is the path to the TLS server key.

    --acraserver_transport_encryption_disable
    

    Disable SecureSession/TLS encryption between AcraConnector and AcraServer.

    Configuring connections to the database
    --tls_acraserver_sni
    

    If the client application is connecting to the database with SSL enabled (i.e. sslmode values either allowed or required), you should define tls_acraserver_sni as the database hostname that AcraConnector checks during TLS handshake on the connection attempt.

    Configuring connections to AcraTranslator
    --mode=acratranslator
    --acratranslator_connection_host
    --acratranslator_connection_port
    --acratranslator_connection_string
    --acratranslator_securesession_id
    

    To connect to AcraTranslator, you should specify the exact mode and provide the connection parameters:

    $GOPATH/bin/acra-connector --client_id=client_name --mode=acratranslator --acratranslator_connection_host=127.0.0.1 --acratranslator_connection_port=10000
    
    Other important configuration options:
    --keys_dir
    

    Sets a custom key directory location. By default, AcraConnector is looking for keys in .acrakeys, but you can specify another location.

    --disable_user_check
    

    Turns off the test that checks if AcraConnector has started from under a user different from the one running the app. This option was only added to simplify your tests!

    $GOPATH/bin/acra-connector --client_id=client_name --acraserver_connection_host=127.0.0.1 --disable_user_check
    
    Logging, tracing, metrics

    To change the logging format to plaintext, CEF, or json, use:

    --logging_format      
    

    Read more about analysing logs here.

    --incoming_connection_prometheus_metrics_string
    

    Configure the URL to be used to expose Prometheus metrics (use /metrics address to pull metrics).

    --tracing_enable
    --tracing_jaeger_enable
    --tracing_log_enable
    

    Enable the collecting of tracing by setting tracing_enable to true. You can print traces to log or export to Jaeger (see below).

    --jaeger_agent_endpoint
    --jaeger_basic_auth_password
    --jaeger_basic_auth_username
    --jaeger_collector_endpoint
    

    Setup Jaeger and Jaeger agent endpoints and credentials to Jaeger.

    See real-world applications where Acra's configuration is heavily changed in Acra example projects.

    AcraWriter

    After you have configured AcraConnector, your application can keep using your database handling code as before — all the extra work will now be taken care of by Acra's components.

    AcraWriter is a library for your code that can be used anywhere within your app whenever you need to encrypt sensitive records.

    Under the hood, it is basically the Themis library generating AcraStructs with the keys you've provided to AcraWriter.

    To start protecting your data, pick a place in your code where you would like to integrate AcraWriter (or any self-made AcraStruct encrypter).

    You can use our libraries for Golang, Python, Ruby, PHP, Node.js.

    Or you can take a look at our examples: Golang, Python, Ruby, PHP, Node.js.

    If you plan to use Acra with:

    • PostgreSQL — use BYTEA binary type;

    • MySQL — use mysql binary type.

    Acra example usage scenario

    You can encrypt the sensitive data by generating AcraStructs with AcraWriter anywhere across your app. Send INSERTS/UPDATES to the database either through AcraConnector or directly via a separate connection.

    You can decrypt AcraStructs by sending database requests to AcraConnector and receive responses that went through AcraServer.

    Acra does not dictate a way of storing database requests in the database. If your code is a monolith running on one server as one service, it might make sense to point the writes into the same connections as reads, i.e. AcraConnector.

    However, if your code is a huge family of microservices where some of them just write data and some just read it, it is essential to be able to do it the way you want it. You can write AcraStructs generated by AcraWriter directly into the database so that you don't need to carry all the AcraConnector keys with every piece of code on every machine.

    For example, we're aware of a setup where AcraStructs are sent down the Kafka stream and emerge in the database at some further point in the future.

    Client-side without Zones

    This is an instruction for PostgreSQL.

    Writes

    When you need to store some sensitive data:

    • before doing INSERT/UPDATE on data, you should use AcraWriter libraries specific to your language, encrypting sensitive data into AcraStruct with AcraServer's public key with suffix _storage (some_client_id_storage.pub) that is different from the keypair used by AcraConnector for communicating with AcraServer.
    • then you can proceed with doing INSERT/UPDATE, either through AcraConnector or directly.

    Reads

    Read requests are the same as the regular reads, but their remote address is AcraConnector instead of your database listener. A typical PostgreSQL connection address would look like this: postgresql://user:password@127.0.0.1:9494/db_name.

    Note for PostgreSQL: In the current layout, you aren't required to use SSL when trying to connect to the database. Transport protection for sensitive (decrypted) data is provided between AcraServer and AcraConnector via Secure Session. However, you can setup using SSL connection, too.

    Client-side with Zones

    Description of the client-side with Zones for PostgreSQL.

    Zone-based encryption is the best way to cryptographically compartmentalise the data that comes from different sources, following the user's choice. Cryptographically, the Zones are a mechanism for modifying the AcraServer key according to the Zone Id and for mapping Zone's private keys during the decryption. In theory, you could use a new key for every record. However, it would introduce an unwanted overhead on AcraServer's performance.

    Writing

    To write AcraStruct with Zone keys, you need to:

    • generate the Zone keys on AcraServer,
    • acquire Zone key and Zone Id from AcraServer,
    • encrypt data using this key as AcraServer key in a standard AcraStruct process.

    There are two ways you can generate the keys:

    • First, make sure that you already generated ACRA_MASTER_KEY and out it into an environmental variable, or generate it like this:
    export ACRA_MASTER_KEY=$(echo -n "My_Very_Long_Key_Phrase_ge_32_chars" | base64)
    

    Read more about different types of keys used in Acra on Key Management page.

    • Then build AcraAddZone utility in AcraServer from Acra repository:
    go build acra-addzone
    ./acra-addzone
    

    When you run it, you should get JSON object that looks something like this:

    {"id":"DDDDDDDDQHpbUSOgYTzqCktp","public_key":"VUVDMgAAAC3yMBGsAmK/wBXZkL8iBv/C+7hqoQtSZpYoi4fZYMafkJbWe2dL"}
    

    By decoding base64-wrapped public key, you get the binary Zone key, which can be used for generating AcraStructs.

    • AcraConnector locally provides HTTP API to add Zones and return public keys on 127.0.0.1:9191/getNewZone. You can change the port via --incoming_connection_api_port argument.

    Both approaches provide identical results.

    Reading

    Reading with Zones takes place just as it does without Zones. However, to point AcraServer to the exact Zone key, it has to use, you need to structure your query in such a way that AcraStruct is preceded by the corresponding Zone Id in its answer. For example, you may want to store the Zone Id in the preceding column and execute the query in the following way:

    SELECT id, lastname, zone, data, datetime FROM table;
    

    where zone contains Zone Id and data stands for AcraStruct.

    If you keep the Zone Id in some other place (i.e. configuration file, other tables, or other database storage), you need to explicitly state it in the select, for example:

    SELECT id, lastname, 'DDDDDDDDQHpbUSOgYTzqCktp'::bytea, data, datetime FROM table;
    

    Note: It's worth mentioning that ::bytea conversion is necessary for PostgreSQL to work properly.

    Building AcraWriter for your language

    Acra uses Themis for performing cryptographic computations. AcraWriter is basically Themis + high level code that generates AcraStructs from input data and public key.

    There are many languages / architectures supported by Themis, and eventually, most of them will support Acra, too.

    Common steps for all languages

    Install dependencies

    sudo apt install git libssl-dev make build-essential
    

    Install Themis

    git clone https://github.com/cossacklabs/themis.git
    cd themis
    make
    sudo make install
    cd -
    

    Get Acra sources

    git clone https://github.com/cossacklabs/acra.git
    cd acra
    

    Building AcraWriter for Python

    Before building AcraWriter for Python, make sure you've installed Themis as system library!

    Installation

    sudo apt install python-pip
    pip install acrawriter
    

    Testing

    from pythemis.skeygen import GenerateKeyPair, KEY_PAIR_TYPE
    from acrawriter import create_acrastruct
    
    keypair = GenerateKeyPair(KEY_PAIR_TYPE.EC)
    create_acrastruct(b'some data', keypair.export_public_key())
    

    Output

    '""""""""UEC2\x00\x00\x00-\xc2\xda\xbc\x03\x03`/\xc5\x8d.|DuX\xc4\x1c\xe5\x11\xd2>.\xc7.\xf3\xe5\xa5\xec\x83\xdbC\xd9\xdb/\x85K\x19\xe5 \'\x04&T\x00\x00\x00\x00\x01\x01@\x0c\x00\x00\x00\x10\x00\x00\x00 \x00\x00\x00\xb1o\xde\xbf\xcc\xa1\xe6\x13\x00MT)\x98\x95@Z?\xd1\xea\xa2n\xa1\xf7\x93\x9dD\xf4Gr\n?8J\x94\x1c$\x86\x91\xdc\x05^\x1d\xe0>\x18ru\xa4A\x04\xf9\x98X]\x13\x9c$\x97~\x145\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01@\x0c\x00\x00\x00\x10\x00\x00\x00\t\x00\x00\x00D\xe7\xafw\x82\xb3\xff"q3\x94\x17X\x14+\xe6e\x07\xecR>\xc9\x9d\xd4\xa8"T\xf5(\xcf:\x83\xe8\x82\x02\xcf\x16'
    

    Examples

    With Zones and without Zones.

    A real-world example project that protects Django web application and a set of small Python command-line applications is available in Acra example projects.

    Building AcraWriter for Ruby

    Before building AcraWriter for Ruby, make sure you've installed Themis as system library!

    Installation

    gem install acrawriter
    

    Test

    gem install --user-install minitest rake
    rake test
    

    Output

    Finished in 0.003829s, 261.1501 runs/s, 0.0000 assertions/s.
    
    1 runs, 0 assertions, 0 failures, 0 errors, 0 skips
    

    AcraWriter supports ActiveDirectory

    gem 'activerecord_acrawriter'
    

    Examples

    With Zones and without Zones.

    A real-world example project that protects Ruby on Rail application is available in Acra example projects.

    Building AcraWriter for Nodejs

    Before building AcraWriter for NodeJS, make sure you've installed Themis as system library!

    Installation

    sudo apt install node npm
    sudo apt install nodejs-legacy
    sudo npm install -g npm
    cd wrappers/nodejs
    npm install nan
    npm install
    

    Testing

    node test.js
    

    Output

    work
    

    Examples

    With Zones and without Zones.

    Building AcraWriter for PHP

    Before building AcraWriter for PHP, make sure you've installed Themis as system library!

    Installation

    sudo apt install php5 php5-dev
    cd themis
    sudo make phpthemis_install
    # extension=phpthemis.so
    sudo nano /etc/php5/cli/php.ini
    # check that extension is available
    php --ri phpthemis
    cd -
    

    Testing

    php acra/wrappers/php/test.php
    

    Output

    work
    

    Now use acra/wrappers/php/acrawriter.php in your project. In future acrawriter will contain composer.json and will be added to packagist.

    Examples

    With Zones and without Zones

    Building AcraWriter for Go

    Before building AcraWriter for Go, make sure you've installed Themis as system library!

    Installation

    sudo apt install golang
    # leave your GOPATH or set some
    export GOPATH=$HOME/work
    go get github.com/cossacklabs/acra/acra-writer
    

    Testing

    go test github.com/cossacklabs/acra/acra-writer
    

    Output

    ok      github.com/cossacklabs/acra/acra-writer 0.112s
    

    Examples

    With Zones and without Zones

    Building AcraWriter for C++

    Before building AcraWriter for C++, make sure you've installed Themis as system library!

    Dependencies

    Additionally, install ThemisPP (Themis C++ wrapper) as system library from Themis source folder:

    cd themis
    make themispp_install
    cd -
    

    Installation

    Grab acrawriter.hpp source file and add it to your project. Link themis and crypto libs.

    Example of CMake file:

    project(my_project)
    set(CMAKE_CXX_STANDARD 14)
    
    //path to acrawriter.hpp file
    set(ACRAWRITER ../../wrappers/cpp)
    include_directories(${ACRAWRITER})
    
    set(
        SOURCE_FILES
        ${ACRAWRITER}/acrawriter.hpp
        main.cpp
        )
    
    add_executable(my_project ${SOURCE_FILES})
    target_link_libraries(my_project themis crypto)
    

    C++ examples and tests

    See CLion project example in examples/cpp for generating AcraStruct with and without Zones, and tests.

    Building AcraWriter for iOS

    Installation via CocoaPods

    Update your Podfile with:

    pod 'acrawriter'
    

    AcraWriter for iOS supports bitcode and has Themis (and OpenSSL) as dependencies.

    iOS examples and tests

    Check out the iOS project example using Objective-C in examples/objc for generation of an AcraStruct with and without Zones and decrypting them using AcraTranslator via HTTP API.

    Check out another iOS project example using Swift in examples/swift for generation of an AcraStruct with and without Zones.

    Building AcraWriter for Android

    Installation

    Installation via maven: AcraWriter is placed into our bintray.

    First, update your build.gradle file with URL to our maven repository:

    repositories {
            // whatever you have here, add maven
            maven { url "https://dl.bintray.com/cossacklabs/maven/" }
    }
    

    Then link acrawriter from app/build.gradle file:

    dependencies {
         // ....
        implementation 'com.cossacklabs.com:acrawriter:1.0.1'
    }
    

    Android examples and tests

    See Android project example in examples/android_java for generating AcraStruct with and without Zones, and decrypting them using AcraTranslator via HTTP API.

    Server side: AcraServer

    How AcraServer works

    AcraServer is the most important part of Acra — it is the proxy server responsible for decrypting all the database responses and forwarding them back to clients.

    One AcraServer instance can only work with one RDBMS at the time.

    AcraServer supports scaling and load-balancing — take a look at our HAProxy-based example project.

    This is what a typical Acra-protected application flow looks like:

    • Upon the first AcraConnector connection, AcraServer does not automatically connect to the database and sits in the waiting mode instead. The type of the database is defined by setting postgresql_enable or mysql_enable fields to true. By default, AcraServer is trying to connect to PostgreSQL.
    • If the client application is connecting to the database where SSL is enabled (sslmode values either allowed or required), you should define tls_db_sni field for AcraServer. It indicates the database hostname that AcraServer checks during the TLS handshake on the connection attempt.
    • When the first AcraConnector connection arrives, AcraServer initialises secure communication.
      • By default, the secure communication is based on Secure Session. During the initialisation stage, AcraConnector sends its Id within the Secure Session protocol, whereas AcraServer tries to find the public key <id>.pub and private key <id>_server, which should be in .acrakeys or wherever the keys_dir parameter points to. If such keys are found, AcraServer is able to establish the connection.
      • Another option is to use TLS connection (it can be enabled by setting acraconnector_tls_transport_enable flag to true). AcraServer requires the path to keys and certificates: tls_ca is the path to root certificate; tls_cert is the path to TLS server certificate and tls_key is the path to TLS server key. Bear in mind that you need explicitly set the client_id because for the time being AcraServer can only process zoneless connections when using a single client_id.
    • After a successful initialisation of the session, AcraServer creates a database connection via --db_host/--db_port parameters and starts forwarding all the requests coming from AcraConnector into the database.
    • Upon receiving the database answer, AcraServer attempts to decrypt zone_id and AcraStruct. If Zones are disabled, the stream only gets parsed for AcraStruct.
      • If AcraStruct was not detected, the response of the database will be passed 'as is'.
      • If AcraStruct was recognised (with or without a Zone), AcraServer will attempt to unpack the AcraStruct and to decrypt the payload. After that, AcraServer will replace the AcraStruct with the decrypted payload, change the packet's length, and return the answer to the application via AcraConnector. Acra allows users to combine several Zones and Zone-dependent AcraStructs into one query — the only rule is that for each AcraStruct an attempt at decryption will be made (using the Zone key that corresponds to the latest preceding matching Zone_id).
      • If AcraServer detects a poison record within the AcraStruct's decryption stream, AcraServer will either shut down the decryption, run an alarm script, or both, depending on the pre-set parameters (poison_run_script_file, poison_shutdown_enable).
    • Every incoming request to AcraServer is passed through AcraCensor (Acra's firewall). AcraCensor will pass the allowed queries and return an error on the forbidden ones. The rules are configured and stored in a yaml file.
    • The cycle loops and repeats ∞.

    Transparent proxy mode

    Transparent encryption proxy mode allows you to configure AcraServer to encrypt records in specific database columns without altering the application code.

    Basically, AcraServer here performs AcraWriter's duties. This mode is useful for large distributed applications, where updating the source code of each client app separately is complicated.

    • Imagine that the client application wants to update some data in the database. The application logic doesn't need to change because there's no need to integrate AcraWriter library into the application code. The application sends SQL requests through AcraConnector and AcraServer to the database, as described above.

    • When the first AcraConnector connection arrives, AcraServer initialises secure communication with AcraConnector via TLS or Secure Session. This step is no different from what you already know. All the data pieces coming from the application are transmitted via a secure channel.

    • After initialising the connection, AcraConnector sends queries from the application, AcraServer parses each query and encrypts the desired values into AcraStructs. To know which values to encrypt, AcraServer uses a configuration file (known as encryptor_config_file) that describes which columns in which tables should be encrypted. AcraServer uses AcraStruct encryption key just as AcraWriter would.

    • After encrypting the values into AcraStructs, AcraServer passes the modified queries to the database and the database response – back to the client application via AcraConnector.

    • When the client application wants to read the data, it sends a read query to the database (via AcraConnector and AcraServer). Upon retrieving the database response, AcraServer tries to detect AcraStructs, decrypts them, and returns the decrypted data to the application. Note that the client application should be ready to receive the decrypted data in the binary format because AcraServer doesn't know the nature of data (original data can be binary or string or int), it can't encode it into the original data format.

    Transparent mode has the same features as the "usual" mode – including SQL firewall, intrusion detection based on poison records, Zone support, etc.

    To enable this mode, you need to create a separate encryptor configuration file (acra-encryptor.yaml) that describes which columns to encrypt and provide a path to it in the AcraServer configuration file (or via CLI params --encryptor_config_file=acra-encryptor.yaml). Please read about the format of the encryptor configuration file below.

    Getting started with AcraServer

    1a. Launch using Docker (the fastest way to try AcraServer)

    Note: Using Docker is recommended for testing purposes only. Please don't rely on Docker in real-life production settings.
    Note: The following examples focus on using AcraServer, AcraConnector and AcraWriter with PostgreSQL, but Acra also supports MySQL. If you need to support KV databases (NoSQL, MongoDB), jump to AcraTranslator.

    Clone the Acra repository, build images, and start Docker compose with PostgreSQL, AcraServer, AcraConnector, and Secure Session between them:

    git clone https://github.com/cossacklabs/acra.git
    make docker
    docker-compose -f docker-compose.pgsql-nossl-server-ssession-connector.yml up
    

    Okay, now that the basic infrastructure is all set up, all components are connected, and keys are placed in the appropriate folder you can go to step 2 – launch AcraServer.

    1b. Manual launch

    Note: Skip this if you've used the Docker method described above.

    • Install the dependencies: Themis cryptographic library:
    git clone https://github.com/cossacklabs/themis.git
    cd themis
    make
    sudo make install
    
    • Install AcraServer:
    go get github.com/cossacklabs/acra/cmd/acra-server
    

    Note: All the commands starting with 'go' are meant to be executed from the 'acra' folder (the folder with the repository code).

    • Install the key generation utility:
    go get github.com/cossacklabs/acra/cmd/acra-keymaker
    
    • Use acra-keymaker to generate the master key into the master.key file and assign it into the environment variable like this:
    $GOPATH/bin/acra-keymaker --generate_master_key=master.key
    export ACRA_MASTER_KEY=`cat master.key | base64`
    

    Read more about the different types of keys used in Acra on the Key Management page.

    • Generate transport keys and storage keys:
    $GOPATH/bin/acra-keymaker --client_id=client_name --generate_acraserver_keys
    $GOPATH/bin/acra-keymaker --client_id=client_name --generate_acrawriter_keys
    

    The client_id of the key should be longer than 5 characters. It is used as an identifier for the Secure Session connection between AcraConnector and AcraServer.

    The generated transport keypair client_name_server and client_name_server.pub will appear in .acrakeys (or anywhere you ask with --keys_output_dir=/path/to/dir argument). It's used to establish the Secure Session connection between AcraConnector and AcraServer.

    The keypair for AcraWriter has two keys: public (which is used on AcraWriter's side to encrypt the data into AcraStruct) and private (used by AcraServer to decrypt AcraStructs). Their names are: client_storage and client_storage.pub.

    • Exchange the public keys:

    • You should take the public key of AcraConnector client_name.pub and put it into the corresponding folder (.acrakeys) on AcraServer.

    • You should put AcraServer's transport public key (called client_name_server.pub) into AcraConnector's key folder (.acrakeys or anything you chose in --keys_output_dir).
    • You should embed AcraServer's storage public key (called client_storage.pub) into AcraWriter client library.

    AcraServer reads the decryption keys from the key folder and stores them in the memory in an encrypted form. It uses LRU cache to increase the performance by keeping only the actively used keys in the memory. The size of the LRU cache can be configured depending on your server's load.

    2. Launching AcraServer

    Launching AcraServer is a one-line command that takes many parameters :).

    By default, AcraServer is ready to talk with AcraConnector and PostgreSQL database:

    $GOPATH/bin/acra-server --client_id=client_name --incoming_connection_host=acra.connector.host --incoming_connection_port=9393 --db_host=postgresql.db.host --db_port=5432
    

    AcraServer is just one part of the whole, you should read about the AcraConnector setup to bring it all together.

    AcraServer CLI reference

    The following parameters can be used for configuring AcraServer from the command line or from the configuration file (see below). Both ways can be used. The typical flow is to:

    • run AcraServer with CLI parameters while testing/debugging, then
    • save them to a file,
    • and run еру production AcraServer instances from the configuration file.
    --acracensor_config_file
        Path to AcraCensor configuration file
    --acraconnector_tls_transport_enable
        Use tls to encrypt transport between AcraServer and AcraConnector/client
    --acraconnector_transport_encryption_disable
        Use raw transport (tcp/unix socket) between AcraServer and AcraConnector/client (don't use this flag if you not connect to database with SSL/TLS
    --acrastruct_injectedcell_enable
        Acrastruct may be injected into any place of data cell
    --acrastruct_wholecell_enable
        Acrastruct will stored in whole data cell (default true)
    --auth_keys
        Path to basic auth passwords. To add user, use: `./acra-authmanager --set --user <user> --pwd <pwd>` (default "configs/auth.keys")
    --client_id
        Expected client ID of AcraConnector in mode without encryption
    --config_file
        path to config
    -d  Log everything to stderr
    --db_host
        Host to db
    --db_port
        Port to db (default 5432)
    -ds
        Turn on http debug server
    --dump_config
        dump config
    --encryptor_config_file
        Path to Encryptor configuration file
    --generate_markdown_args_table
        Generate with yaml config markdown text file with descriptions of all args
    --http_api_enable
        Enable HTTP API
    --incoming_connection_api_port
        Port for AcraServer for HTTP API (default 9090)
    --incoming_connection_api_string
        Connection string for api like tcp://x.x.x.x:yyyy or unix:///path/to/socket (default "tcp://0.0.0.0:9090/")
    --incoming_connection_close_timeout
        Time that AcraServer will wait (in seconds) on restart before closing all connections (default 10)
    --incoming_connection_host
        Host for AcraServer (default "0.0.0.0")
    --incoming_connection_port
        Port for AcraServer (default 9393)
    --incoming_connection_prometheus_metrics_string
        URL (tcp://host:port) which will be used to expose Prometheus metrics (<URL>/metrics address to pull metrics)
    --incoming_connection_string
        Connection string like tcp://x.x.x.x:yyyy or unix:///path/to/socket (default "tcp://0.0.0.0:9393/")
    --jaeger_agent_endpoint
        Jaeger agent endpoint that will be used to export trace data (default "127.0.0.1:6831")
    --jaeger_basic_auth_password
        Password used for basic auth (optional) to jaeger
    --jaeger_basic_auth_username
        Username used for basic auth (optional) to jaeger
    --jaeger_collector_endpoint
        Jaeger endpoint that will be used to export trace data (default "127.0.0.1:14268")
    --keys_dir
        Folder from which will be loaded keys (default ".acrakeys")
    --keystore_cache_size
        Maximum number of keys stored in in-memory LRU cache in encrypted form. 0 - no limits, -1 - turn off cache
    --logging_format
        Logging format: plaintext, json or CEF (default "plaintext")
    --mysql_enable
        Handle MySQL connections
    --pgsql_escape_bytea
        Escape format for Postgresql bytea data
    --pgsql_hex_bytea
        Hex format for Postgresql bytea data (default)
    --poison_detect_enable
        Turn on poison record detection, if server shutdown is disabled, AcraServer logs the poison record detection and returns decrypted data (default true)
    --poison_run_script_file
        On detecting poison record: log about poison record detection, execute script, return decrypted data
    --poison_shutdown_enable
        On detecting poison record: log about poison record detection, stop and shutdown
    --postgresql_enable
        Handle Postgresql connections (default true)
    --securesession_id
        Id that will be sent in secure session (default "acra_server")
    --tls_auth
        Set authentication mode that will be used in TLS connection with AcraConnector. Values in range 0-4 that set auth type (https://golang.org/pkg/crypto/tls/#ClientAuthType). Default is tls.RequireAndVerifyClientCert
    --tls_ca
        Path to root certificate which will be used with system root certificates to validate Postgresql's and AcraConnector's certificate
    --tls_cert
        Path to tls certificate
    --tls_db_sni
        Expected Server Name (SNI) from database
    --tls_key
        Path to private key that will be used in AcraServer's TLS handshake with AcraConnector as server's key and database as client's key
    --tracing_enable
        Enable tracing
    --tracing_jaeger_enable
        Export trace data to jaeger
    --tracing_log_enable
        Export trace data to log
    -v  Log to stderr all INFO, WARNING and ERROR logs
    --zonemode_enable
        Turn on zone mode
    

    AcraServer configuration files

    All these parameters are available in a yaml-file which you can feed to the system through --config_file CLI argument.

    You can copy the original example config from GitHub, or from downloaded repository $REPO_DIR/configs/acra-server.yaml or from GOPATH $GOPATH/src/github.com/cossacklabs/acra/configs/acra-server.yaml.

    Before changing the configuration file, don't forget to backup your current file. Let us repeat this again - backup your current file ;).

    AcraCensor SQL firewall
    --acracensor_config_file
    

    Enable AcraCensor SQL firewall by providing the path to its configuration file. If no configuration file is provided, SQL firewall is disabled.

    Transparent mode proxy
    --encryptor_config_file
    

    Enable AcraServer encryption in Transparent mode by providing a path to its configuration file. If no configuration file is provided, Transparent proxy mode stays disabled.

    Configuring connections with AcraConnector via Themis Secure Session

    By default, AcraServer and AcraConnector use Themis Secure Session as the transport encryption protocol.

    --securesession_id
    

    Customise the SecureSessionId of AcraServer (default value is "acra-server") so AcraConnector to be able to distinguish different AcraServers.

    See the configuration examples in the Docker folder, look for examples marked as "nossl" and without "connector"., like docker-compose.mysql-ssl-server-ssl.yml.

    Configuring connections with AcraConnector via TLS

    AcraServer supports TLS for connecting with AcraConnector and database:

    --acraconnector_tls_transport_enable
    --tls_auth
    --tls_ca
    --tls_cert
    --tls_key
    --client_id
    

    Setup TLS by enabling acraconnector_tls_transport_enable. Provide the TLS authentication mode (how to check TLS certificates), CA, and AcraServer's certificate.

    tls_auth defines an authentication mode that will be used in TLS connection, possible values are taken from TLS configuration in Go: from 0 (don't request, don't check certificate) to 4 (request and verify client certificate).

    tls_ca is the path to the root certificate, which will be used with system root certificates to validate AcraConnector's, client's, and AcraServer's certificate. tls_cert path to the TLS server certificate and tls_key is the path to the private key that will be used in the TLS handshake with AcraConnector as server's key and the database as the client's key. Provide clientID so AcraConnector knows which AcraServer it's talking to.

    It's possible to mix Themis Secure Session and TLS: imagine that the client applications expects to use TLS when connecting to the database, but AcraConnector uses Themis Secure Session to connect to AcraServer. It means that AcraServer supports two connection types simultaneously: it validates the client TLS certificate and uses own TLS certificates to connect to the database (TLS); at the same time it uses Themis Secure Session to connect to AcraConnector.

    Having difficulties with this one? No worries, proxying secure sockets is complicated, we sometimes struggle with it, too - you can file us an Issue and we'll do our best to explain how to do it (and update the docs eventually).

    See the configuration examples in the Docker folder, look for examples marked as "ssl" and "connector", like docker-compose.mysql-ssl-server-ssl-connector.yml.

    Configuring TLS connections with client application directly, omitting AcraConnector

    In some infrastructures, you might want to omit AcraConnector and use a direct TLS connection between the client app and AcraServer.

    In this case, you can disable acraconnector_transport_encryption_disable, setup client_id, and provide TLS settings for AcraServer.

    --acraconnector_transport_encryption_disable
    --client_id
    --tls_auth
    --tls_ca
    --tls_cert
    --tls_key
    

    See the configuration examples in docker folder, look for examples marked as "ssl" and without "connector", like docker-compose.mysql-ssl-server-ssl.yml.

    Configuring plaintext connections with client application directly, omitting AcraConnector

    During the debugging process, you might want to disable any encryption used between the client application and AcraServer. You can remove AcraConnector from the infrastructure. If encryption is disabled, client_id should be provided as an expected client ID of the client application (otherwise, how would AcraServer know which client app it is talking to?).

    --acraconnector_transport_encryption_disable
    --client_id
    

    Please be careful, this mode doesn't have transport encryption so the data is leaving the client application in plaintext. Do not use this for production.

    Configuring incoming connections settings

    Incoming connections port and host:

    --incoming_connection_host
    --incoming_connection_port
    --incoming_connection_string
    

    You can change the host (default '0.0.0.0') and the port (default '9393') on which AcraServer will be listening to receive the data from AcraConnector. Instead of using host and port parameters, you can use only a string parameter (and pass the connection string in a format "tcp://host:port/"), but if you setup all the three of them, the values from the host and port will have a higher priority.

    Incoming connection timeout:

    --incoming_connection_close_timeout
    

    Configure the waiting time for AcraServer(in seconds) on restart before closing all connections (the default value is 10).

    Configuring connections via HTTP API
    --http_api_enable
    

    AcraServer supports two modes: binary data and HTTP API commands. By using HTTP API, any client can get a new Zone or reset AcraServer's config. HTTP API is used in AcraWebConfig – a visual interface that allows updating AcraServer's settings.

    If http_api_enable is disabled, AcraWebConfig (or any other HTTP API client) can't connect to AcraServer via HTTP API.

    Incoming connection API port:

    --incoming_connection_api_port
    --incoming_connection_api_string
    

    You can change the port that AcraServer listens on for HTTP API.

    Who can connect to AcraServer via AcraWebConfig:

    --auth_keys
    

    Provide the path to file with user passwords in Basic Auth format. To add a user, use: ./acra-authmanager --set --user <user> --pwd <pwd>. The default file path is configs/auth.keys. Read more in AcraWebConfig setup.

    Configuring connections to the database
    --db_host
    --db_port
    

    Database host and port that AcraServer connects to.

    MySQL or PostgreSQL:

    --mysql_enable
    --postgresql_enable
    

    PostgreSQL data format:

    --pgsql_escape_bytea
    --pgsql_hex_bytea
    

    The default format is hex format for Postgresql bytea data, but escape format can also be used.

    Secure the transport between the database and AcraServer:

    --tls_ca
    --tls_cert
    --tls_db_sni
    --tls_key
    

    AcraServer can use TLS connection to connect to the database. AcraServer requires the path to the keys and certificates: tls_ca is the path to the root certificate, which will be used with the system root certificates to validate the database's and AcraServer's certificate. tls_cert path to the TLS server certificate and tls_key is the path to the private key that will be used in TLS handshake with AcraConnector as server's key and database as client's key. tls_db_sni is the expected server name (SNI) from the database.

    AcraStruct format configuration
    --acrastruct_injectedcell_enable
    --acrastruct_wholecell_enable
    

    AcraServer supports two different ways of placing AcraStruct inside a database cell. WholeCell mode means that the whole database cell data is used by AcraStruct. InjectedCell mode means that the database cell has some data in it and AcraStruct is "hidden" somewhere in that cell. AcraServer can work only with one mode after launch (so please select the mode carefully). The default mode is the "WholeCell" mode.

    --zonemode_enable
    

    Turn on Zone mode, which will trigger AcraServer to use Zone keys. Upon detecting ZoneID in database response, AcraServer will use a corresponding private key to decrypt the next detected AcraStruct.

    Key management
    --keys_dir
    

    Sets a custom key directory location. By default, AcraConnector is looking for keys in .acrakeys, but you can specify another location.

    --keystore_cache_size
    

    The maximum number of keys that will be stored in the in-memory LRU cache in an encrypted form before adding a new key will remove the last one. Set 0 to set an indefinite quantity, set -1 to disable LRU cache.

    Intrusion detection system (poison records)
    --poison_detect_enable
    

    Enable detection of poison records (default is true). Upon detection of a poison record in the database response, AcraServer logs a warning and returns the decrypted data.

    AcraServer can perform one of the following actions upon detecting poison records:

    --poison_run_script_file
    --poison_shutdown_enable
    

    It can either log a warning, execute a script file (provided by path), return decrypted data or log warning, stop and shutdown completely.

    Logging, tracing, metrics

    To change the logging format to plaintext, CEF, or json, use:

    --logging_format      
    

    Read more about analysing logs here.

    --incoming_connection_prometheus_metrics_string
    

    Configure the URL (tcp://host:port) to be used to expose Prometheus metrics (use /metrics address to pull metrics).

    --tracing_enable
    --tracing_jaeger_enable
    --tracing_log_enable
    

    Enable the of collecting tracing by setting tracing_enable to true. You can print the traces to log or export them to Jaeger (see below).

    --jaeger_agent_endpoint
    --jaeger_basic_auth_password
    --jaeger_basic_auth_username
    --jaeger_collector_endpoint
    

    Setup Jaeger and Jaeger agent endpoints and credentials to Jaeger.

    See the real-world applications where Acra's configuration is heavily changed in Acra example projects.

    AcraServer configuration in Transparent proxy mode

    In the Transparent proxy mode, AcraServer parses each query from AcraConnector (the client application) to the database and encrypts the specified values into AcraStructs using AcraStruct encryption key or Zone encryption key.

    To know which values to encrypt, AcraServer uses a configuration file (known as encryptor_config_file) that describes which columns in which tables should be encrypted. An example encryptor configuration file is available in GitHub or in the downloaded repository $REPO_DIR/configs/acra-encryptor.example.yaml, or from GOPATH $GOPATH/src/github.com/cossacklabs/acra/configs/acra-encryptor.example.yaml.

    Take a look at the real applications that use AcraServer in Transparent proxy mode with custom encryptor configuration in Acra example projects repository.

    It describes the database scheme, each table, plaintext columns, and to-be-encrypted columns.

    schemas:
    - table: <table name>
      columns:
      - <plaintext column name>
      - <plaintext column name>
      encrypted:
      - column: <encrypted column name>
        zone_id: <zoneID value means to use public key of specific Zone>
      - column: <encrypted column name>
        client_id: <clientID value means to use public key of specific client>
      - column: <encrypted column name>
        <no specific clientID or ZoneID means to use clientID value from AcraConnector transport request>
    

    For each table, you can specify an unlimited amount of plaintext columns, the values of which AcraServer will pass as is. If INSERT or UPDATE SQL query from client application contains values from encrypted columns, AcraServer will take the corresponding public key (either the specific client's public key or the specific Zone's public key, or will select the clientID based on the request from AcraConnector) to encrypt the values into AcraStruct.

    Then AcraServer passes the modified queries to the database and the database response back to the client application via AcraConnector.

    AcraCensor — Acra’s firewall

    AcraCensor is a separate SQL firewall module for AcraServer that checks every incoming request. AcraCensor logs SQL requests directed to the database, allows and denies those requests according to certain rules (configured as allowlist/denylist), or ignores them and logs an exception (this might be useful if some requests can’t be parsed by AcraCensor). AcraCensor can be configured for your exact use case, allowing to setup an allowlist and a denylist for tables, query patterns, and exact queries. AcraCensor's goal is to protect the database from SQL injections and suspicious SQL queries.

    AcraCensor supports SQL database types — MySQL and PostgreSQL (and their flavours i.e. MariaDB). It is built on top of xwb1989/sqlparser, which we had extended significantly.

    AcraCensor is compatible with SIEM systems, with logs that can be used for alerts' configuration and anomaly detection.

    To demonstrate the process of configuring AcraCensor for SQL injection prevention in OWASP Mutillidae 2 example app, we created a Docker-based demo project. Read engineering details about how we built AcraCensor in our blog.

    Launching AcraCensor

    AcraCensor is a built-in component of AcraServer, so it starts running when AcraServer does if the path to its configuration file is provided. You can provide a path for AcraCensor's configuration file using --acracensor_config_file parameter in acra-server.yaml configuration.

    You can find the basic example configuration file configs/acra-censor.example.yaml of AcraCensor in our GitHub repository. More real-world examples are available in the AcraCensor demo project. You can use these examples as a template for your own, using the database of your choice.

    Starting AcraCensor using exact configuration file:

    $GOPATH/bin/acra-server --acracensor_config_file=configs/acra-censor.example.yaml --client_id=client_name --incoming_connection_host=acra.connector.host --db_host=postgresql.db.host 
    

    Refer to the AcraServer documentation if you have problems with passing configuration parameters.

    Configuring AcraCensor rules

    The configuration file describes how AcraCensor should process the incoming SQL queries – log, pass, block, ignore. Each action is described by the corresponding handler. Their position in the file represents their priority of processing.

    We suggest the following structure of handlers for better security: put query_capture first to log every query into the censor.log, then put query_ignore to ignore some database-specific control queries that occur when the database is starting and when the database drives check the connections, then put the deny handler to block all the unwanted queries, then put the allow handler to allow some specific "normal" queries.

    New configuration file format allows configuring the allow and deny handlers separately or simultaneously.

    ignore_parse_error: false
    version: 0.85.0
    parse_errors_log: unparsed_queries.log
    handlers:
      - handler: query_capture
        filepath: censor.log
      - handler: query_ignore
        queries:
          # mysql queries AcraCensor will ignore
          - ROLLBACK
          - SET AUTOCOMMIT = 0
          - SHOW VARIABLES LIKE 'sql_mode'
          - show collation where `Charset` = 'utf8' and `Collation` = 'utf8_bin'
      - handler: deny
        queries:
          - select data_raw from plaintext_table
        tables:
          - acrarollback_output
        patterns:
          - "SELECT password from users %%WHERE%%"
          - "SELECT * from users where type=private"
      - handler: allow
        tables:
          - users
          - accounts
          - blogs_table
        patterns:
          - "%%INSERT%%"
          - "SELECT * from users where type=public"
    

    Next, see the explanation for each handler below, in order of importance.

    Configuration versioning

    Starting wirh Acra 0.85.0 release in March 2019, we've introduced a new more flexible configuration format for AcraCensor rules.

    AcraServer v0.85+ will work with AcraCensor's configuration file versioned 0.85 and above. AcraServer doesn't support configuration files that have an old format, all users should migrate to the new format.

    The migration procedure consists of the following steps:

    • rename blacklist handler to deny handler
    • rename whitelist handler to allow handler
    • add version line on the top of the file "version: 0.85.0"

    These are all required steps to make your previous configuration file compatible with new AcraServer/AcraCensor versions.

    How is the new configuration is more flexible?

    • you can use the deny, allow handlers multiple times, but the order is important: AcraCensor processes handlers one-by-one, top to bottom;
    • added a new allowall handler. If the allowall handler is the final statement of a configuration file, it means that all the queries that were not processed by the previous handlers are allowed;
    • added new denyall handler. If denyall handler is the final statement of a configuration file, it means that all the queries that were not processed by previous handlers are denied;
    • added new patterns for pattern matching, see below.

    Allowlist and denylist

    There are two basic handler types for AcraCensor:

    • The allow handler, which allows specific queries or queries that match specific patterns and access to specific tables and blocks everything else. If the allowall handler is a final statement of a configuration file, it means that all the queries that were not processed by previous handlers are allowed.

    • The deny handler, which blocks specific queries or queries that match specific patterns and access to the specific tables and allows everything else. If the denyall handler is a final statement of a configuration file, it means that all the queries that were not processed by previous handlers are allowed.

    For each handler, there are settings that regulate:

    • Queries — exact match character-by-character of the incoming SQL query and configured one;
    • Tables — match queries for specified tables;
    • Patterns — match queries by specific SQL statement types.

    You can configure the allowlist and the denylist separately or simultaneously. The order of priority for the lists is defined by their order in the configuration file. Priority of processing for each list is the following: queries, followed by tables, followed by patterns.

    Queries

    Queries work in the following manner: the queries addressed to the database are alternately checked (lines are compared) for the queries matching the ones that the firewall needs to block/allow. If there is a match, the query addressing the database is blocked and a corresponding error is returned (if it's AcraCensor's deny handler at work) or the query is allowed through (if it matches a record in the allow handler of AcraCensor).

    There are no restrictions for the query types (SELECT, INSERT, UNION, etc.).

    Tables

    Tables work in the following manner: the names of the tables are alternately extracted from the queries that address the database and each table is compared to the list of tables from the configuration file. If a matching table is detected, the query is blocked and the corresponding error is returned (if it is AcraCensor’s deny handler at work) or allowed (if it matches a record in the allow handler of AcraCensor).

    There is a restriction for query types: only SELECT and INSERT types are allowed, other query types are not supported.

    Patterns

    Patterns work in the following manner: when a query addressing the database is checked, each pattern (if any is detected) is parsed alongside the incoming query. The parsing process extracts the SQL statement type (SELECT, INSERT, UPDATE etc), WHERE and UNION nodes, tables, columns, and values from the body of the query. If the pattern and the query have matching nodes, then tables are checked. If matches for tables are detected, then columns are checked. When separate columns match, a decision to block (through denylist) or to allow (through allowlist) the query is made.

    Supported patterns

    This is a work-in-progress area, the list is a subject of fixes and updates.

    VALUE pattern

    %%VALUE%% represents literal value (string, binary, number, boolean), it's supported in the following cases: after WHERE, IN, ORDER BY, GROUP BY, BETWEEN.

    • WHERE ... VALUE
    pattern
    SELECT a FROM t WHERE ID = %%VALUE%%
    matching queries
    SELECT a FROM t WHERE ID = '123'
    SELECT a FROM t WHERE ID = 35
    non-matching queries
    SELECT a, b FROM t WHERE ID = '123'
    SELECT a FROM anothertable WHERE ID = '123
    SELECT a FROM t WHERE ID > '123
    • BETWEEN VALUE and VALUE
    pattern
    SELECT a FROM t WHERE param BETWEEN %%VALUE%% and %%VALUE%%
    matching queries
    SELECT a FROM t WHERE param BETWEEN 1 and 3
    SELECT a FROM t WHERE param BETWEEN "qwerty" and NULL
    non-matching queries
    SELECT a, b FROM t WHERE param BETWEEN 1 and 3
    SELECT a FROM anothertable WHERE param BETWEEN 1 and 3
    • IN (VALUE, VALUE)
    pattern
    SELECT 1 FROM t WHERE b='qwe' and v IN (%%VALUE%%, %%VALUE%%, %%VALUE%%)
    matching queries
    SELECT 1 FROM t WHERE b='qwe' and v IN (1, 2, 3)
    non-matching queries
    SELECT 1 FROM t WHERE b='qwe' and v IN (1, 2)
    SELECT 1 FROM t WHERE b='qwe' and v IN (1, 2, 3, 4, 5)
    LIST_OF_VALUES pattern

    %%LIST_OF_VALUES%% represents several values one by one.

    pattern
    SELECT 1 from t WHERE b='qwe' and v IN (%%LIST_OF_VALUES%%)
    matching queries
    SELECT 1 from t WHERE b='qwe' and v IN (1, 2, 3)
    SELECT 1 from t WHERE b='qwe' and v IN (1, 'qwe', True, NULL, FALSE)
    non-matching queries
    SELECT 1 from anothertable WHERE b='qwe' and v IN (1, 2)
    SUBQUERY pattern

    %%SUBQUERY%% represents subquery expression starting from SELECT.

    pattern
    SELECT 1 from t WHERE a=%%SUBQUERY%%
    matching queries
    SELECT 1 from t WHERE a=(SELECT column1, column2 from table1 WHERE a=1)
    SELECT 1 from t WHERE a=(SELECT column1, column2 from table1 WHERE a=1 union SELECT column1, column2 from table2)
    non-matching queries
    SELECT 1 from t WHERE a=2
    SELECT 1 from anothertable WHERE a=(SELECT column1, column2 from table1 WHERE a=1)
    WHERE pattern

    %%WHERE%% represents one or more expressions after WHERE statement. This pattern works for SELECT/UPDATE/DELETE queries.

    pattern
    SELECT users FROM company %%WHERE%%
    matching queries
    SELECT users FROM company WHERE title = 'engineer'
    SELECT users FROM company WHERE AGE IN ( 25, 27 )
    SELECT users FROM company WHERE NAME LIKE 'Pa%'
    SELECT users FROM company WHERE A=(SELECT age FROM company WHERE salary > 65000 limit 1) and B=(SELECT age FROM company123 WHERE salary > 1000 limit 1)
    non-matching queries
    SELECT users, cats FROM company WHERE a = 'someValue'
    SELECT age FROM company4 WHERE age IS NULL
    SELECT users FROM company4 INNER JOIN (SELECT age FROM company WHERE id = 1) AS t ON t.id=another_table.id WHERE AGE NOT IN (25, 27)
    COLUMN pattern

    %%COLUMN%% represents a column expression.

    • SELECT COLUMN
    pattern
    SELECT %%COLUMN%%, %%COLUMN%% FROM company
    matching queries
    SELECT users, cats FROM company
    SELECT a, b FROM t WHERE ID = '123'
    non-matching queries
    SELECT users FROM company
    SELECT users, cats, chameleons FROM company
    SELECT users, cats, chameleons FROM company
    SELECT users, cats FROM zoo
    • ORDER BY COLUMN
    pattern
    SELECT 1 FROM t ORDER BY %%COLUMN%%
    matching queries
    SELECT 1 FROM t ORDER BY age
    SELECT 1 FROM t ORDER BY (case when f1 then 1 when f1 is null then 2 else 3 end)
    non-matching queries
    SELECT 1 FROM anothertable ORDER BY age
    SELECT 1 FROM anothertable ORDER BY age DESC
    SELECT pattern

    %%SELECT%% represents a whole SELECT expression.

    pattern
    %%SELECT%%
    matching queries
    SELECT users, cats FROM company
    SELECT dogs, chameleons FROM company
    SELECT SUM(Salary) FROM Employee WHERE Emp_Age < 30
    non-matching queries
    any non-SELECT query
    INSERT pattern

    %%INSERT%% represents a whole INSERT expression.

    pattern
    %%INSERT%%
    matching queries
    INSERT INTO Customers (CustomerName, ContactName) VALUES ('Cardinal', 'Tom B. Erichsen')
    INSERT INTO dbo.Points (PointValue) VALUES ('1,99');
    non-matching queries
    any non-INSERT query
    UPDATE pattern

    %%UPDATE%% represents a whole UPDATE expression.

    pattern
    %%UPDATE%%
    matching queries
    UPDATE t SET a=1 WHERE ID = 1
    UPDATE Customers SET ContactName = 'Alfred Schmidt', City = 'Frankfurt' WHERE CustomerID = 1
    non-matching queries
    any non-UPDATE query
    DELETE pattern

    %%DELETE%% represents a whole DELETE expression.

    pattern
    %%DELETE%%
    matching queries
    DELETE FROM t WHERE removed = TRUE
    DELETE FROM Customers WHERE CustomerName = 'Alfreds Futterkiste'
    non-matching queries
    any non-DELETE query
    BEGIN pattern

    %%BEGIN%% represents a whole BEGIN expression.

    pattern
    %%BEGIN%%
    matching queries
    BEGIN
    non-matching queries
    any non-BEGIN query
    COMMIT pattern

    %%COMMIT%% represents a whole COMMIT expression.

    pattern
    %%COMMIT%%
    matching queries
    COMMIT
    non-matching queries
    any non-COMMIT query
    ROLLBACK pattern

    %%ROLLBACK%% represents a whole ROLLBACK expression.

    pattern
    %%ROLLBACK%%
    matching queries
    ROLLBACK
    non-matching queries
    any non-ROLLBACK query

    This is a constantly updating list of useful configuration examples. Please see the examples we are using for integration testing in tests/acra-censor_configs and AcraCensor's unit tests. We will also appreciate your questions and Pull Requests.

    Prepared statements

    The root of the SQL injection problem is that code and data get mixed up and can be mistaken for each other. SQL prepared statements help to separate code execution from the data itself.

    AcraCensor has limited support of filtering prepared statements. AcraCensor doesn't apply allow/deny rules for SQL requests with prepared statements. However, it's possible to use query_capture/query_ignore to filter the whole statement when necessary (through comparing the incoming SQL char-by-char with the SQL from the config file).

    Ignoring specific queries

    It might be useful not to apply allowlist/denylist rules to some exact queries, for example, the SQL queries on starting up the database. If the query is ignored, AcraCensor won't apply any rules to it (so basically AcraCensor will pass this query to the database).

    - handler: query_ignore
      queries:
        # mysql queries AcraCensor will ignore
        - ROLLBACK
        - SET AUTOCOMMIT = 0
        - SHOW VARIABLES LIKE 'sql_mode'
        - show collation where `Charset` = 'utf8' and `Collation` = 'utf8_bin'
    

    To ignore some queries, place them to the query_ignore handler. AcraCensor compares each input query with queries from the query_ignore list. If there is a match, the query is allowed through and no allow/deny handlers are applied.

    We suggest looking into tests/acra-censor_configs to see which PostgreSQL and MySQL queries we've added to the ignore list for running integration tests.

    Unparseable queries

    Sometimes AcraCensor can't parse some particular SQL queries. It's potentially dangerous to send such queries to the database as they might contain SQL injections. So for security reasons, AcraCensor blocks "unparseable" queries by default.

    However, some of these queries might be totally legit, so users should have a way to allow them. To remedy this, we've added a special ignore_parse_error field into the configuration.

    ignore_parse_error: false
    

    By default, ignore_parse_error is false in AcraCensor (check configs/acra-censor.example.yaml). Setting it to true will push AcraCensor to ignore "unparseable" queries and send them to the database — allowing developers to configure AcraCensor rules during the development and integration stages.

    In a production environment, security-oriented users keep ignore_parse_error disabled, but add such “unparseable but legit” queries to the query_ignore handler manually, resulting in a well-defined and secure firewall configuration. The users encountering parsing errors can send us their GitHub Issue or PR and we'll look into it and try to help.

    Logging and masking queries

    Continuous logging into syslog

    It’s very important that AcraCensor doesn’t log sensitive data. That’s why the first thing AcraCensor does is masking the values in SQL queries’ logs. All values from SQL queries are edited and masked to replaced keyword.

    This is what the logs from AcraCensor look like:

    select * from accounts where cid = :replaced1
    insert into hitlog(hostname, ip, browser, referer, `date`) values (:replaced1, :replaced2, :replaced3, :replaced4, now())
    insert into accounts(username, password, mysignature) values (:replaced1, :replaced2, :replaced3)
    

    Logging errors (unparseable queries)

    You can have a separate log file that contains unparseable queries for the debugging and configuration purposes. Sometimes AcraCensor can't parse all of the incoming queries and it is useful to have a separate log for them. Provide a path to this log file into parse_errors_log:

    parse_errors_log: unparsed_queries.log
    

    Be careful, these queries are logged in plaintext, without masking.

    Logging unique queries to the file

    You might need to have a separate log file that contains unique queries that AcraCensor processes or even multiple log files – for each stage of processing.

    The query_capture logger registers and logs the queries that pass through this handler to a file. Depending on the position of this handler in the configuration file (closer to the top, closer to the bottom), it will log more or fewer queries. For example, if you place this handler below the "deny" handler, you won't see the queries blocked on the previous step.

    You can use this handler to log unique queries, and then use this log to improve your allow/deny configurations. AcraCensor won't log queries that can't be parsed (use parse_errors_log for this).

    To activate this logger, define query_capture as a handler, and set the path to the log file (AcraCensor will create a log file if it was not created before).

    handler: query_capture
    filepath: <path_to_file>
    

    This is how query_capture.log content looks like (real values are being masked):

    {"raw_query":"commit","_blacklisted_by_web_config":false}
    {"raw_query":"update pg_settings set setting = :replaced1 where name = :replaced2","_blacklisted_by_web_config":false}
    {"raw_query":"select data from test where id = :replaced1","_blacklisted_by_web_config":false}
    

    Security-wise configurations

    Despite the fact that AcraCensor can be configured in many ways, we suggest the following structure of handlers for better security:

    • Put query_capture first to log every query into the censor_log.
    • Next, put query_ignore to ignore some database-specific control queries that occur when the database is starting and when the database drives check the connections. Following the secure by default technique, put the allow handler to allow listed queries. Finally, put the denyall handler. In this case, all queries that are not on the allowlist will be blocked. There is a less secure (but still useable) approach when the deny handler is set for blocking all the unwanted queries and then the allowall handler is put to allow all the other queries.

    Example configuration for “Allow — Denyall” rule:

    parse_errors_log: unparsed_queries.log
    handlers:
      - handler: query_capture
        filepath: censor.log
      - handler: query_ignore
        queries:
          - ROLLBACK
          - COMMIT
          - BEGIN
      - handler: deny
        patterns:
          - SELECT * FROM users
          - SELECT %%COLUMN%% FROM users %%WHERE%%
          - DROP table users
      - handler: allow
        tables:
          - users
          - accounts
          - blogs_table
        patterns:
          - "%%INSERT%%"
      - handler: denyall
    

    AcraWebConfig

    AcraWebConfig is a lightweight HTTP web server for managing AcraServer's certain configuration options.

    Note: Acra components can be configured via config files, the main goal of AcraWebConfig is to provide the alternative, more simple, means for configuration.

    The interface of AcraWebConfig consists of the following elements:

    • AcraServer Settings — managing AcraServer's settings.
    • AcraCensor (Coming soon) — Firewall/Intrusion detection settings.
    • Zones (Coming soon) — managing Zone keys.

    AcraWebConfig Setup

    go get github.com/cossacklabs/acra/cmd/acra_configui
    
    • AcraWebConfig uses HTTP API requests to get data from AcraServer and to change its settings. Using CLI-parameters (--destination_host, --destination_port), you can point it to AcraServer.
    • By default, AcraWebConfig accepts requests at localhost:8000. Use --incoming_connection_host and --incoming_connection_port to change this default preset.

    You can create an SSH-tunnel to get access to the remote port:

    ssh -L8000:localhost:8000 <acra-server>
    
    • To provide additional security, AcraWebConfig uses basic authentication. Its control option is --auth_mode. Possible values are:
    • auth_on — basic authentication is on (default).
    • auth_off_local — basic authentication is on for non-local requests (out of the range of 127.0.0.1/localhost).
    • auth_off — basic authentication is turned off.

    Users/passwords are stored in an encrypted file and are managed by acra-authmanager utility:

    go get github.com/cossacklabs/acra/cmd/acra-authmanager

    To add user/password:

    cd $GOPATH
    $GOPATH/bin/acra-authmanager --set --user=<user> --password=<password>
    

    To remove user/password:

    cd $GOPATH
    $GOPATH/bin/acra-authmanager --remove --user=<user>
    

    Encrypted user/password-storage is stored in config/auth.keys, use --file to change this.

    Note: We encrypt auth the storage with .acrakeys/auth_key that is auto-generated at the first run of acra-authmanager.

    AcraWebConfig Usage

    Open the AcraWebConfig HTTP endpoint in your browser.

    In AcraServer settings you can save settings — AcraServer will be gracefully restarted while applying new options via a config file. AcraWebConfig rewrites config file with new values.

    Note: If you want to use AcraWebConfig's AcraServer settings you should avoid using command line options for AcraServer as they have higher priority than the config file.

    Logging in Acra

    Logging configuration

    Starting with version 0.77.0, Acra supports three different logging formats: plaintext, CEF, JSON.

    Acra logs are compatible with various external log analysing tools (like ELK stack) and SIEM systems.

    Logging mode and verbosity level can be configured for AcraServer, AcraConnector, AcraTranslator and AcraWebConfig in the corresponding yaml files or passed as CLI parameter.

    Possible meanings for logging_format are: plaintext or cef or json (case independent). Verbosity level is enabled by changing v to true.

    To see an example of logging, check out the default configuration of AcraServer in acra-server.yaml.

    Logging output

    The default output is stderr. Examples of output:

    plaintext:

    Nothing special, bonus is color-formatted output when connected to terminal.

    INFO[2018-04-10T20:03:59+03:00] Starting service                             
    INFO[2018-04-10T20:03:59+03:00] Validating service configuration             
    INFO[2018-04-10T20:03:59+03:00] Initialising keystore                        
    INFO[2018-04-10T20:03:59+03:00] Selecting transport: use Secure Session transport wrapper 
    INFO[2018-04-10T20:03:59+03:00] Start listening to connections. Current PID: 66579 
    INFO[2018-04-10T20:03:59+03:00] Start listening API: unix:///tmp/acra_api_unix_socket_10004 
    INFO[2018-04-10T20:03:59+03:00] Start listening connection: unix:///tmp/unix_socket_10003 
    INFO[2018-04-10T20:03:59+03:00] Got new connection to acra-server:
    INFO[2018-04-10T20:03:59+03:00] Handle new connection                        
    ERRO[2018-04-10T20:03:59+03:00] Can't wrap connection from acra-connector          code=538 error=EOF
    

    JSON:

    JSON log contains many useful fields like: product and versions, which define Acra components; timestamp in RFC3339 and unixTime to be compatible with a variety of log analysis tools.
    The log levels could be debug/info/warning/error/panic.
    Acra logs both error code and message if an error occurs (code and error fields respectively).

    {"level":"info","msg":"Validating service configuration","product":"acra-server","timestamp":"2018-04-10T20:06:31+03:00","unixTime":"1523379991.311","version":"0.76"}
    {"level":"info","msg":"Initialising keystore","product":"acra-server","timestamp":"2018-04-10T20:06:31+03:00","unixTime":"1523379991.311","version":"0.76"}
    {"level":"info","msg":"Selecting transport: use Secure Session transport wrapper","product":"acra-server","timestamp":"2018-04-10T20:06:31+03:00","unixTime":"1523379991.311","version":"0.76"}
    {"level":"info","msg":"Start listening to connections. Current PID: 67309","product":"acra-server","timestamp":"2018-04-10T20:06:31+03:00","unixTime":"1523379991.311","version":"0.76"}
    {"level":"info","msg":"Start listening API: unix:///tmp/acra_api_unix_socket_10004","product":"acra-server","timestamp":"2018-04-10T20:06:31+03:00","unixTime":"1523379991.312","version":"0.76"}
    {"level":"info","msg":"Start listening connection: unix:///tmp/unix_socket_10003","product":"acra-server","timestamp":"2018-04-10T20:06:31+03:00","unixTime":"1523379991.312","version":"0.76"}
    {"level":"info","msg":"Got new connection to acra-server: ","product":"acra-server","timestamp":"2018-04-10T20:06:31+03:00","unixTime":"1523379991.386","version":"0.76"}
    {"level":"info","msg":"Handle new connection","product":"acra-server","timestamp":"2018-04-10T20:06:31+03:00","unixTime":"1523379991.386","version":"0.76"}
    {"code":538,"error":"EOF","level":"error","msg":"Can't wrap connection from acra-connector","product":"acra-server","timestamp":"2018-04-10T20:06:31+03:00","unixTime":"1523379991.387","version":"0.76"}
    

    CEF:

    CEF logger is compliant with the CEF protocol revision 16 from July 22, 2010, except the fact that Acra is using custom Dictionary Extension fields (like unixTime) because we found them to be very useful for log analysis.
    Pay attention that severity levels range from 0 (debug level) to 10 (panic level), see severity mapping function in logging/cef_formatter.

    CEF:0|cossacklabs|acra-server|0.76|100|Validating service configuration|1|unixTime=1523380315.046
    CEF:0|cossacklabs|acra-server|0.76|100|Initialising keystore|1|unixTime=1523380315.046
    CEF:0|cossacklabs|acra-server|0.76|100|Selecting transport: use Secure Session transport wrapper|1|unixTime=1523380315.046
    CEF:0|cossacklabs|acra-server|0.76|100|Start listening to connections. Current PID: 68099|1|unixTime=1523380315.046
    CEF:0|cossacklabs|acra-server|0.76|100|Start listening connection: unix:///tmp/unix_socket_10003|1|unixTime=1523380315.047
    CEF:0|cossacklabs|acra-server|0.76|100|Start listening API: unix:///tmp/acra_api_unix_socket_10004|1|unixTime=1523380315.047
    CEF:0|cossacklabs|acra-server|0.76|100|Got new connection to acra-server:|1|unixTime=1523380315.133
    CEF:0|cossacklabs|acra-server|0.76|100|Handle new connection|1|unixTime=1523380315.134
    CEF:0|cossacklabs|acra-server|0.76|538|Can't wrap connection from acra-connector|6|error=EOF unixTime=1523380315.134
    CEF:0|cossacklabs|acra-connector|0.76|100|Got new connection to acra-connector:|1|unixTime=1523380315.167
    CEF:0|cossacklabs|acra-server|0.76|100|Got new connection to acra-server:|1|unixTime=1523380315.168
    

    Logging error codes

    Most errors logged by Acra have their own error code. You can find the table of error codes in logging/event_codes.go.

    We plan to use unique event codes for every system event, not only for errors, but these updates are not ready yet. Currently, the default event code is 100 for all events (except errors).

    Implementing a new logging format

    If you think that Acra is missing some handy logging formats, feel free to open an Issue and describe why you think it's important. Alternatively, you can add the required log formatter (logging folder is a good place to start) and open a Pull Request.
    We are grateful for contributions.

    Metrics in Acra

    Starting with version 0.83.0, Acra supports tracking and exporting of the basic metrics of AcraServer, AcraConnector, and AcraTranslator to Prometheus.

    Metrics configuration

    If incoming_connection_prometheus_metrics_string is set, Acra will generate specific metrics (time of connection life, time of processing requests, AcraStruct decryption counters) and push them to Prometheus. This parameter can be configured for AcraServer, AcraConnector, and AcraTranslator in the corresponding yaml files or passed as a CLI parameter.

    To see an example of collecting and displaying metrics, check out the Acra Engineering Demo that illustrates integration of Acra-protected infrastructure with Prometheus and Grafana dashboards.

    Tracing in Acra

    Starting with version 0.84.0, Acra supports tracing with OpenCensus. AcraServer, AcraConnector and AcraTranslator track every request from client application to the database and back. Each client request has a unique traceID, that helps measure how much time it needs to perform a certain data processing functions (ie. checking the requests via AcraCensor, encrypting data, decrypting AcraStructs, etc.).

    Tracing configuration

    If trace_log_enable is set, Acra component will produce, collect, and log traces.

    Traces can be exported to Jaeger: set up parameters for Jaeger endpoint and authentication (trace_jaeger_enable, jaeger_agent_endpoint, jaeger_basic_auth_password, jaeger_basic_auth_username, jaeger_collector_endpoint).

    Tracing-related parameters can be configured for AcraServer, AcraConnector, and AcraTranslator in the corresponding yaml files or passed as CLI parameter.

    Acra Engineering Demo has examples of using and exporting traces.

    Protecting data using Acra

    Protecting data in SQL databases using AcraWriter and AcraServer

    Acra Server: full architecture

    Scheme explanation: On the left - typical infrastructure with AcraConnector and AcraServer, data is encrypted on client application using AcraWriter. On the right - server-side encryption using AcraServer in Transparent proxy mode.

    Integrating client-side encryption using AcraWriter

    This is what the process of encryption and decryption of the data in a database looks like:

    • Your application encrypts some data through AcraWriter by generating an AcraStruct using Acra storage public key and then updates the database. AcraStructs generated by AcraWriter can't be decrypted by it — only the Acra's server side has the right keys for decryption.
    • To retrieve the decrypted data, your application talks to AcraServer. It is a server-side service that works as database proxy: it sits transparently between your application and the database, listens silently to all the traffic that's coming to and from the database.
    • AcraServer monitors the incoming SQL requests and blocks the unwanted ones using the built-in configurable firewall AcraCensor. AcraServer only sends allowed requests to the database. Certain configurations for AcraServer can be adjusted remotely using AcraWebConfig web server.
    • Upon receiving the database response, AcraServer tries to detect the AcraStructs, decrypts them, and returns the decrypted data to the application.
    • AcraConnector is a client-side daemon responsible for providing encrypted and authenticated connection between the application and AcraServer. AcraConnector runs under a separate user/in a separate container and acts as middleware. AcraConnector accepts connections from the application, adds an extra transport encryption layer using TLS or Themis Secure Session, sends the data to AcraServer, receives the result, and sends it back to the application.

    Protecting data in any file storage using AcraWriter and AcraTranslator

    Acra Translator: simplified architecture

    In some use cases, the application can store encrypted data as separate blobs (files that are not in a database, i.e. in a S3 bucket, local file storage, etc.). In this case, you can use AcraTranslator — a lightweight server that receives AcraStructs and returns the decrypted data.

    This is what the process of encryption and decryption data using AcraTranslator looks like:

    • Your application encrypts some data using AcraWriter, generating an AcraStruct using Acra storage public key and puts the data into any file storage. AcraStructs generated by AcraWriter can't be decrypted by it — only the Acra's server side has the right keys for decrypting it.
    • To decrypt an AcraStruct, your application sends it to AcraTranslator as a binary blob via HTTP or gRPC API. AcraTranslator doesn’t care about the source of the data, it is responsible for holding all the secrets required for data decryption and for actually decrypting the data.
    • AcraTranslator decrypts AcraStructs and returns the decrypted data to the application.
    • To avoid sending the plaintext via an unsecured channel, AcraTranslator requires the use of AcraConnector, a client-side daemon responsible for providing encrypted and authenticated connection between the application and AcraServer. AcraConnector runs under a separate user/in a separate container and acts as a middleware. It accepts connections from the application, adds transport encryption layer using TLS or Themis Secure Session, sends data to AcraServer, receives the result, and sends it back to the application.

    AcraTranslator and AcraServer are fully independent server-side components and can be used together or separately depending on your infrastructure.

    When you don't need Acra

    There are many cases when certain security model is useless, futile or even downright risky. This page is a growing collection of such cases, at least the ones we can think of.

    When you don't need encryption

    • When you don't have anything to protect. However, even leaking e-mail+password might be a risk for your users - if not in your site, then somewhere else (it is useful to keep in mind that people frequently reuse passwords).

    When Acra won't help

    • When you trust your existing protection methods and consider them sufficient.
    • Especially if they are long-proven by industry and exactly fit your use case.
    • When you can't afford yourself performance penalties Acra imposes (but then you're not really using languages Acra supports for development anyway, do you?).

    Contacts and assistance

    Contributing to Acra

    Although Acra is an in-house-developed product by Cossack Labs Limited, it is still open-source, Apache 2-licensed software. This means you can hack it any way you want and contribute things back if you'd like to. As a software company, we focus on implementing features that are important to our products but would gladly spend some time on making Acra useful for everybody.

    We highly encourage you to:

    • Report bugs and request features via GitHub Issues.
    • Report a bug and fix it with a patch via our GitHub Pull request (after creating a corresponding issue and leaving a link to the pull there).
    • Add something new to Acra. There is a certain design scheme according to which we'd like to keep developing Acra. If your contributions fall along with it, we'd be glad to accept some fundamental additions. It's better to discuss the subject using email before taking action (see below).

    Every commit that goes into the master branch is audited and reviewed by somebody from Cossack Labs, so don't be surprised if it takes a bit long.

    If you'd like to participate in the core development more deeply, get in touch.

    Getting in touch

    • Requests/bugfixes/queries should go through GitHub Issues.
    • To get in touch with the developers, use this email at your own discretion :). Make sure you've tried reaching out through GitHub Issues first, before writing a direct email.
    • To talk to the business wing of Cossack Labs Limited, drop us an email.

    Implementation assistance

    If you would like to receive a one-time customized assistance in implementing Acra for the needs of your app/service, consider applying for our DataGuardian Assistance Program.