Hermes-core tutorials

C tutorial for Hermes-core

In this tutorial, we are going to launch storage entities for data, public and encryption keys, and will save/delete/edit the data with the help of a Hermes-core console app, as well as grant/revoke access to the data for other users. All this will be carried out cryptographically.

Launching the storage entities

The infrastructure of Hermes-core is divided into 3 parts (you can read more about each entity here and in the scientific paper on Hermes): - Data store (data storage entity), - Keystore (storage entity for keys with the help of which the data is encrypted), - Credential store (storage entity for the users' public keys with the help of which the authentification and authorization take place).

Besides, we provide keypair generator to generate keys for the users.

Installing the necessary libraries

Let's install the libraries and utilities that we're going to need.

For Debian the command is:

sudo apt-get update && sudo apt-get install build-essential libssl-dev git 

We need build-essential for building binary libraries and libssl as backend for Themis.

If you're using another OS please refer to the Installation guide.

Let's download and install Themis into your system:

git clone https://github.com/cossacklabs/themis
cd themis
make && sudo make install
cd ..

Now you should download and install Hermes-core:

git clone https://github.com/cossacklabs/hermes-core
cd hermes-core
make && sudo make install

Building all the services necessary for the Hermes-core infrastructure

The next step is building keypair generator and stores: Keystore, Credential store, Data store.

You can find each component in docs/examples/c folder, and run make for each of them separately.

However, we recommend building them all at once:

make examples

Creating folder structure for the services

With the following command, we are creating a folder structure in which the services we are creating will store the data:

# create folder structure
mkdir -p db/credential_store
mkdir -p db/key_store
mkdir -p db/data_store

Register service's keys in the Credential store

The service examples have hardcoded private/public keys. To simplify the first run, we placed the public keys into the repository close to examples. Those files have filenames that are base64-encoded names of services that are declared in docs/examples/c/mid_hermes/common/config.h and have binary content of public keys.

cp docs/examples/c/service_keys/* db/credential_store/

Generating user keys

User 1

./docs/examples/c/key_gen/key_pair_gen user1.priv db/credential_store/$(echo -n user1 | base64)

Here we are using our key generator and command it to save a private key in a folder with the name user1.priv, and put the public key into the folder that will be assigned to/available to Credential store — db/credential_store. The filename will be a user identifier in base64. In our case, it is going to be user1.

Note: It is important that the filename is in the correct base64 format as consequently it will be used as the user identifier.

User 2

Let's create another user, this time with user2 identifier:

./docs/examples/c/key_gen/key_pair_gen user2.priv db/credential_store/$(echo -n user2 | base64)

Launching services

Now we need to launch all the necessary services that we have built.

This should be done in separate console tabs:

Credential store

./docs/examples/c/mid_hermes/credential_store_service/cs

Keystore

./docs/examples/c/mid_hermes/key_store_service/ks

Data store

./docs/examples/c/mid_hermes/data_store_service/ds

Now all the parts of Hermes-core are working and we can start using the infrastructure — add data, grant/revoke access, edit, etc.

Installing Client

To install the C client for Hermes-core, do the following:

make mid_hermes_example_client

On MacOS you might need to install argp:

brew install argp-standalone

Our Client app is ready to be used. Now, this is where the fun begins.

Creating the first document

Let's add some data — for example, let's create a file with simple content:

echo "some content" > testfile
Folder structure

This is an example folder structure — you probably have something similar right now. Docs/examples/c contains the core components of Hermes-core. The C client is located in docs/examples/c/mid_hermes/client folder. The database folder db contains encrypted data and access keys / tokens.

hermes-core/
+-- docs/examples/
|   +-- c/
|       +-- key_gen/
|       L-- mid_hermes/
|           +-- client
|           +-- credential_store_service/
|           +-- data_store_service/
|           L-- key_store_service/
L-- db/
|   +-- credential_store/
|   +-- data_store/
|   L-- key_store/
+-- somefile
+-- user1.priv
L-- user2.priv

Adding data

Let's add the first document into database:

./docs/examples/c/mid_hermes/client/client add_block user1 $(cat user1.priv | base64) "somefile" "first-file"

The first 4 parameters for this console client always follow this order:

  1. The command add_block adds block with to Hermes system.

  2. User's identifier (user1 in the example above).

  3. User's private key in the base64 format (cat user1.priv | base64 in the example above).

  4. The path the file that acts as a block on which all the (Hermes) operations are performed (somefile in the example above).

Some commands also have a 5th parameter which accepts the metadata information (first-file in the example above).

Find the list of all the other commands supported by the example using help command:

docs/examples/c/mid_hermes/client/client --help
usage: client <command> <user id> <base64 encoded user private key> <name of file for proceed> <meta> <for user>.
           <command>                         - executes the command to be performed by the client, see below;
           <user id>                         - user identifier (user needs to be registered in Credential store);
           <base64 encoded user private key> - base64 encoded private key of the user;
           <name of file to be processed>    - filename of the file to be processed (file name is used as block ID in Hermes);
           <meta>                            - some data associated with a file that is stored in the database in plaintext;
           <for user>                        - identifier of the user for which permissions are provided/revoked from (this information is needed by some commands).

commands:
           add_block - add <name of file to be processed> block with <meta> to Hermes system
           read_block - read <name of file to be processed> block with <meta> from Hermes system
           update_block - update <name of file to be processed> block with <meta> in Hermes system
           delete_block - delete <name of file to be processed> block from Hermes system
           rotate - rotate <name of file to be processed> block from Hermes system
           grant_read - grant read access for <for user> to <name of file to be processed> block in Hermes system
           grant_update - grant update access for <for user> to <name of file to be processed> block in Hermes system
           revoke_read - deny read access for <for user> to <name of file to be processed> block in Hermes system
           revoke_update - deny update access for <for user> to <name of file to be processed> block in Hermes system

Reading data

Let's try to read this data by typing the following:

./docs/examples/c/mid_hermes/client/client read_block user1 $(cat user1.priv | base64) "somefile"

# output
new content
first-file

The command looks the same, it's only that add_block was switched for read_block and we don't have to set the 5th metadata parameter. In all the following commands we'll most often see the change in the command type and addition/deletion of the 5-th parameter that contains metadata.

So now we're seeing the content and metadata that we'd previously passed. All this data is stored in the folder that was created earlier — ls db/data_store which will contain a folder

So now the content and the meta data we transferred before is displayed for us. All this data is stored as files in the previously created folder ls db/data_store which will contain a folder with the name that matches the filename of the added file in base64 format — c29tZWZpbGU=. This ls db/data_store/c29tZWZpbGU= folder will contain 3 files:

data
mac
meta

To make sure that your data is truly encrypted, you can display it using:

cat db/data_store/c29tZWZpbGU=/data

The output will be unprintable because the encrypted data is in binary state and you will not find the traces of the initial data in it. Currently, only the person (user) who had added the data can make can make changes to it. But let's change the initial file and update it in Hermes-core:

echo "some new content" > somefile

And let's take another look at what is stored in Hermes-core now. The content is not changed, because we haven't pushed new somefile to data store.

./docs/examples/c/mid_hermes/client/client read_block user1 $(cat user1.priv | base64) "somefile"

# output
new content
first-file

Updating the data

Let's now try updating the data while using the identifier (ID) and the key that belong to a different user — user2:

./docs/examples/c/mid_hermes/client/client update_block user2 $(cat user2.priv | base64) "somefile" "first-file"

# output
error: block adding error

As a result, we get an error. But now let's perform the update on behalf of the previous user — user1 (who was the entity that added the data in the first place):

./docs/examples/c/mid_hermes/client/client update_block user1 $(cat user1.priv | base64) "somefile" "first-file"

Let's now output (display) the data to make sure it was indeed updated:

./docs/examples/c/mid_hermes/client/client read_block user1 $(cat user1.priv | base64) "somefile"

# output
some new content
first-file

And let's make sure that user2 has no access to the data:

./docs/examples/c/mid_hermes/client/client read_block user2 $(cat user2.priv | base64) "somefile"

# output
error: block getting error

As we can see — user2 indeed has no access to the data.

Granting READ permissions

But let's now grant READ permission to user2:

./docs/examples/c/mid_hermes/client/client grant_read user1 $(cat user1.priv | base64) "somefile" "user2"

Here the identifier of user2 was set as the 5th argument.

So let's try and read the file using the key belonging to user2:

./docs/examples/c/mid_hermes/client/client read_block user2 $(cat user2.priv | base64) "somefile"

# output
some new content
first-file

Making sure that user2 didn't also receive the UPDATE permissions alongside with READ permissions:

./docs/examples/c/mid_hermes/client/client update_block user2 $(cat user2.priv | base64) "somefile" "first-file"

# output
error: block adding error

Ok, we got an error so everything is functioning as intended.

Granting UPDATE permissions

Now, let's grant the permission to UPDATE to user2:

./docs/examples/c/mid_hermes/client/client grant_update user1 $(cat user1.priv | base64) "somefile" "user2"

Let's update the file and read it:

echo "user 2 data" > somefile
./docs/examples/c/mid_hermes/client/client update_block user2 $(cat user2.priv | base64) "somefile" "first-file"
./docs/examples/c/mid_hermes/client/client read_block user2 $(cat user2.priv | base64) "somefile"

# output
user 2 data
first-file

Adding one more user and distributing permissions further

Let's now add user3 just as we did before with user2, to be able to grant permissions from user2 to user3 (using the credentials of user2):

./docs/examples/c/key_gen/key_pair_gen user3.priv db/credential_store/`echo -n user3 | base64`

Granting access/permissions from user2 to user3:

./docs/examples/c/mid_hermes/client/client grant_read user2 $(cat user2.priv | base64) "somefile" "user3"

Let's try to read the data now as the user3:

./docs/examples/c/mid_hermes/client/client read_block user3 $(cat user3.priv | base64) "somefile"

# output
user 2 data
first-file

Making sure that user3 has no permission to perform UPDATE on the data:

./docs/examples/c/mid_hermes/client/client update_block user3 $(cat user3.priv | base64) "somefile" "first-file"

# output
error: block adding error

ROTATING data and keys

At this step, let's perform data and key(s) rotation as user2. This rotation means re-encrypting the data using new keys. To make sure that the key and data rotation indeed takes place (because the data is binary and it would be hard to tell one batch of encrypted data from another), we need to calculate the hash-sum of the encrypted file before and after rotation and check if they match (they shouldn't). If after rotation we read the data again and get the same decrypted data output, it means that the rotation was successful.

Let's calculate and save the hashsum of the ecnrypted file (which is located in db/data_store/dGVzdGZpbGU=/data) into a temporary file:

sha256sum db/data_store/dGVzdGZpbGU=/data > /tmp/1.sha

Perform rotation:

./docs/examples/c/mid_hermes/client/client rotate user1 $(cat user1.priv | base64) "somefile"

and calculate the hashsum again:

sha256sum db/data_store/dGVzdGZpbGU=/data > /tmp/2.sha

Using the diff comand, we'll see that the previous and the new hashsums are different:

diff /tmp/1.md5 /tmp/2.md5 
1c1
< 9baad30131be8b6beb4453b0ea3aeef1  db/data_store/dGVzdGZpbGU=/data
---
> c5425afdaf906e4eeec3a70923018e34  db/data_store/dGVzdGZpbGU=/data

If the diff command provides some text output, it means that the files (hashsums) we are comparing are different. Otherwise, there would be no text output. So the key and data rotation (its re-encryption using new encryption keys) was successful.

To make sure that the data stays the same, let's read the data (as user3) again:

docs/examples/python/hermes_client.py --id user3 --config=docs/examples/python/config.json --private_key user3.priv --doc testfile --read

The resulting data is unchanged. Rotation of keys and data worked as intended.

Revoking permissions/access

To revoke the read permissions from user3 (as user2), do the following:

./docs/examples/c/mid_hermes/client/client revoke_read user2 $(cat user2.priv | base64) "somefile" "user3"

Attempting to read the data now as user3 will predictably lead to an error:

./docs/examples/c/mid_hermes/client/client read_block user3 $(cat user3.priv | base64) "somefile"

# output
error: block getting error

Let's now also revoke the UPDATE permissions from the initial data owner — user1 (acting as user2):

./docs/examples/c/mid_hermes/client/client revoke_update user2 $(cat user2.priv | base64) "somefile" "user1"

Checking if the revocation of rights was successful:

./docs/examples/c/mid_hermes/client/client update_block user1 $(cat user1.priv | base64) "somefile" "first-file"

# output
error: block adding error

The attempt to edit (update) the data by user1 ended with an error, which means that the revocation of rights was successful.

Let's now revoke all rights to the data from user1:

./docs/examples/c/mid_hermes/client/client revoke_read user2 $(cat user2.priv | base64) "somefile" "user1"

Now it is only the user2 who has any rights to the data:

./docs/examples/c/mid_hermes/client/client read_block user2 $(cat user2.priv | base64) "somefile"

# output
user 2 data
first-file

Deleting the record

Let's elegantly finish this transfer of rights to the data by deleting the record altogether (acting as user2):

./docs/examples/c/mid_hermes/client/client delete_block user2 $(cat user2.priv | base64) "somefile"

And now, let's make sure that the data is indeed gone — if it is, we'll see an error while trying to read it as user2 (or any other user from this tutorial, for that matter):

./docs/examples/c/mid_hermes/client/client read_block user2 $(cat user2.priv | base64) "somefile"

# output
error: block getting error

The block getting error confirms that it's now impossible to get the block as it had been deleted. So the data is gone, deleted by user2 who gained all the permissions.

Summary

As you can see, launching the storage entities for data, public and encryption keys for Hermes-core is easy, and the process of granting/revoking permissions (access) to the data between authorised users in Hermes-core is very straightforward and convenient.

We hope that this tutorial was fun and informative and that you now have gained enough understanding of how the things work with Hermes-core — and that now you’ll try using it or build a Hermes-based app of your own.

Go tutorial for Hermes-core

In this tutorial, we are going to launch storage entities for data, public and encryption keys, and will save/delete/edit the data with the help of a Hermes-core console app, as well as grant/revoke access to the data for other users. All this will be carried out cryptographically.

Launching the storage entities

The infrastructure of Hermes-core is divided into 3 parts (you can read more about each entity here and in the scientific paper on Hermes): - Data store (data storage entity), - Keystore (storage entity for keys with the help of which the data is encrypted), - Credential store (storage entity for the users' public keys with the help of which the authentification and authorization take place).

Besides, we provide keypair generator to generate keys for the users.

Installing the necessary libraries

Let's install the libraries and utilities that we're going to need.

For Debian the command is:

sudo apt-get update && sudo apt-get install build-essential libssl-dev git 

We need build-essential for building binary libraries and libssl as backend for Themis.

If you're using another OS please refer to the Installation guide.

Let's download and install Themis into your system:

git clone https://github.com/cossacklabs/themis
cd themis
make && sudo make install
cd ..

Now you should download and install Hermes-core:

git clone https://github.com/cossacklabs/hermes-core
cd hermes-core
make && sudo make install

Building all the services necessary for the Hermes-core infrastructure

The next step is building keypair generator and stores: Keystore, Credential store, Data store.

You can find each component in docs/examples/c folder, and run make for each of them separately.

However, we recommend building them all at once:

make examples

Creating folder structure for the services

With the following command, we are creating a folder structure in which the services we are creating will store the data:

# create folder structure
mkdir -p db/credential_store
mkdir -p db/key_store
mkdir -p db/data_store

Register service's keys in the Credential store

The service examples have hardcoded private/public keys. To simplify the first run, we placed the public keys into the repository close to examples. Those files have filenames that are base64-encoded names of services that are declared in docs/examples/c/mid_hermes/common/config.h and have binary content of public keys.

cp docs/examples/c/service_keys/* db/credential_store/

Generating user keys

User 1

./docs/examples/c/key_gen/key_pair_gen user1.priv db/credential_store/$(echo -n user1 | base64)

Here we are using our key generator and command it to save a private key in a folder with the name user1.priv, and put the public key into the folder that will be assigned to/available to Credential store — db/credential_store. The filename will be a user identifier in base64. In our case, it is going to be user1.

Note: It is important that the filename is in the correct base64 format as consequently it will be used as the user identifier.

User 2

Let's create another user, this time with user2 identifier:

./docs/examples/c/key_gen/key_pair_gen user2.priv db/credential_store/$(echo -n user2 | base64)

Launching services

Now we need to launch all the necessary services that we have built.

This should be done in separate console tabs:

Credential store

./docs/examples/c/mid_hermes/credential_store_service/cs

Keystore

./docs/examples/c/mid_hermes/key_store_service/ks

Data store

./docs/examples/c/mid_hermes/data_store_service/ds

Now all the parts of Hermes-core are working and we can start using the infrastructure — add data, grant/revoke access, edit, etc.

Installing Client

Since we've installed Golang in the beginning of this tutorial (as it is one of Themis' dependencies), our client app is ready to be used.

Install hermes_client dependencies.

go get -u github.com/cossacklabs/hermes-core/docs/examples/go/...

Now, this is where the fun begins.

Creating the first document

Let's add some data — for example, let's create a file with simple content:

echo "some content" > testfile
Folder structure

This is an example folder structure — you probably have something similar right now. Docs/examples/c contains core components of Hermes-core. The Go client is located in docs/examples/go folder. The database folder db contains encrypted data and access keys / tokens.

hermes-core/
+-- docs/examples/
|   +-- c/
|   ¦   +-- key_gen/
|   ¦   L-- mid_hermes/
|   ¦       +-- credential_store_service/
|   ¦       +-- data_store_service/
|   ¦       L-- key_store_service/
|   L-- go/
|       L-- hermes_client.go
L-- db/
|   +-- credential_store/
|   +-- data_store/
|   L-- key_store/
+-- testfile
+-- user1.priv
L-- user2.priv

Adding data

Let's add the first document into database:

go run docs/examples/go/hermes_client.go -command=add_block -doc testfile -id=user1 -private_key=$(cat user1.priv | base64) -meta="some meta data" -config=docs/examples/go/config.conf

# output
success
  • -command=add_block — is the command that is executed. This parameter also accepts the following command types: add_block, read_block, update_block, delete_block, grant_read_access, grant_update_access, revoke_read_access, revoke_update_access.

  • doc testfile — path to the file that will be read and added to Hermes-core.

  • id=user1 — identifier of the user on behalf of which the command will be executed.
  • -private_key=cat user1.priv | base64 — passing the private key of the user on behalf of which the command is executed. The key is in base64 format.
  • -meta="some meta data" — metadata for this block. This parameter is only used when something is added or changed in the data block.
  • -config=docs/examples/go/config.conf — config file with connection settings and public keys that will be used to establish Secure Session with Credential/Key/Data stores with predefined values for this example.

Reading data

Let's try to read this data by typing the following:

go run docs/examples/go/hermes_client.go -command=read_block -doc testfile -id=user1 -private_key=$(cat user1.priv | base64) -meta="some meta data" -config=docs/examples/go/config.conf

# output
some content
some meta data
success

The command remains the same, only add_block was changed to read_block and this time we don't have to set the meta parameter. In all the following commands the changes will most often touch the change of command type and the --meta parameter will be added/deleted.

So now the content and the meta data we transferred before is displayed for us. All this data is stored as files in the previously created folder ls db/data_store which will contain a folder with the name that matches the filename of the added file in base64 format — dGVzdGZpbGU=. This ls db/data_store/dGVzdGZpbGU= folder will contain 3 files:

data
mac
meta

To make sure that your data is truly encrypted, you can display it using:

cat db/data_store/dGVzdGZpbGU=/data

The output will be unprintable because the encrypted data is in binary state and you will not find the traces of the initial data in it. Currently, only the person (user) who had added the data can make can make changes to it. But let's change the initial file and update it in Hermes-core:

echo "some new content" > testfile

And let's take another look at what is stored in Hermes-core now. The content is not changed, because we haven't pushed new testfile to data store.

go run docs/examples/go/hermes_client.go -command=read_block -doc testfile -id=user1 -private_key=$(cat user1.priv | base64) -meta="some meta data" -config=docs/examples/go/config.conf

# output
some content
some meta data
success

Updating the data

Let's now try updating the data while using the identifier (ID) and the key that belong to a different user — user2:

go run docs/examples/go/hermes_client.go -command=update_block -doc testfile -id=user2 -private_key=$(cat user2.priv | base64) -meta "some user2 meta" -config=docs/examples/go/config.conf

# output
panic: MidHermes update block error

As a result, we get an error. But now let's perform the update on behalf of the previous user — user1 (who was the entity that added the data in the first place):

go run docs/examples/go/hermes_client.go -command=update_block -doc testfile -id=user1 -private_key=$(cat user1.priv | base64) -meta="some new meta data" -config=docs/examples/go/config.conf

# output
success

Let's now output (display) the data to make sure it was indeed updated:

go run docs/examples/go/hermes_client.go -command=read_block -doc testfile -id=user1 -private_key=$(cat user1.priv | base64) -config=docs/examples/go/config.conf  

# output
some new content
some new meta data
success

And let's make sure that user2 has no access to the data:

go run docs/examples/go/hermes_client.go -command=read_block -doc testfile -id=user2 -private_key=$(cat user2.priv | base64) -config=docs/examples/go/config.conf

# output
panic: MidHermes read block error

Granting READ permission

As we can see — user2 indeed has no access to the data. But let's now grant READ permission to user2:

go run docs/examples/go/hermes_client.go -command=grant_read_access -doc testfile -id=user1 -private_key=$(cat user1.priv | base64) -for_user=user2 -config=docs/examples/go/config.conf

# output
success

Here we added a new argument -for_user=user to indicate to which user we are granting the access (READ permission in this instance). So let's try and read the file using the key belonging to user2:

go run docs/examples/go/hermes_client.go -command=read_block -doc testfile -id=user2 -private_key=$(cat user2.priv | base64) -config=docs/examples/go/config.conf

# output
some new content
some new meta data
success

Making sure that user2 didn't also receive the UPDATE permissions alongside with READ permissions:

go run docs/examples/go/hermes_client.go -command=update_block -doc testfile -id=user2 -private_key=$(cat user2.priv | base64) -meta "some user2 meta" -config=docs/examples/go/config.conf

# output
panic: MidHermes update block error

Ok, we got an error so everything is functioning as intended.

Granting UPDATE permissions

Now, let's grant the permission to UPDATE to user2:

go run docs/examples/go/hermes_client.go -command=grant_update_access -doc testfile -id=user1 -private_key=$(cat user1.priv | base64) -for_user=user2 -config=docs/examples/go/config.conf

# output
success

Let's update the file and read it:

echo "user 2 data" > testfile
go run docs/examples/go/hermes_client.go -command=update_block -doc testfile -id=user2 -private_key=$(cat user2.priv | base64) -meta "some user2 meta" -config=docs/examples/go/config.conf

# output
success
go run docs/examples/go/hermes_client.go -command=read_block -doc testfile -id=user2 -private_key=$(cat user2.priv | base64) -config=docs/examples/go/config.conf

# output
user 2 data
some user2 meta
success

Adding one more user and distributing permissions further

Let's now add user3 just as we did before with user2, to be able to grant permissions from user2 to user3(using the credentials of user2):

./docs/examples/c/key_gen/key_pair_gen user3.priv db/credential_store/$(echo -n user3 | base64)

Granting access/permissions from user2 to user3:

go run docs/examples/go/hermes_client.go -command=grant_read_access -doc testfile -id=user2 -private_key=$(cat user2.priv | base64) -for_user=user3 -config=docs/examples/go/config.conf

# output
success

Let's try to read the data now as the user3:

go run docs/examples/go/hermes_client.go -command=read_block -doc testfile -id=user3 -private_key=$(cat user3.priv | base64) -config=docs/examples/go/config.conf

# output
user 2 data
some user2 meta
success

Making sure that user3 has no permission to perform UPDATE on the data just yet:

go run docs/examples/go/hermes_client.go -command=update_block -doc testfile -id=user3 -private_key=$(cat user3.priv | base64) -meta "some user2 meta" -config=docs/examples/go/config.conf

# output
panic: MidHermes update block error

ROTATING data and keys

At this step, let's perform data and key(s) rotation as user2. This rotation means re-encrypting the data using new keys. To make sure that the key and data rotation indeed takes place (because the data is binary and it would be hard to tell one batch of encrypted data from another), we need to calculate the hash-sum of the encrypted file before and after rotation and check if they match (they shouldn't). If after rotation we read the data again and get the same decrypted data output, it means that the rotation was successful.

Let's calculate and save the hashsum of the ecnrypted file (which is located in db/data_store/dGVzdGZpbGU=/data) into a temporary file:

sha256sum db/data_store/dGVzdGZpbGU=/data > /tmp/1.sha

Perform rotation:

go run docs/examples/go/hermes_client.go -command=rotate_block -doc testfile -id=user1 -private_key=$(cat user1.priv | base64) -config=docs/examples/go/config.conf

# output

success

and calculate the hashsum again:

sha256sum db/data_store/dGVzdGZpbGU=/data > /tmp/2.sha

Using the diff comand, we'll see that the previous and the new hashsums are different:

diff /tmp/1.md5 /tmp/2.md5 
1c1
< 9baad30131be8b6beb4453b0ea3aeef1  db/data_store/dGVzdGZpbGU=/data
---
> c5425afdaf906e4eeec3a70923018e34  db/data_store/dGVzdGZpbGU=/data

If the diff command provides some text output, it means that the files (hashsums) we are comparing are different. Otherwise, there would be no text output. So the key and data rotation (its re-encryption using new encryption keys) was successful.

To make sure that the data stays the same, let's read the data (as user3) again:

docs/examples/python/hermes_client.py --id user3 --config=docs/examples/python/config.json --private_key user3.priv --doc testfile --read

The resulting data is unchanged. Rotation of keys and data worked as intended.

Revoking permissions/access

To revoke the read permissions from user3 (as user2), do the following:

go run docs/examples/go/hermes_client.go -command=revoke_read_access -doc testfile -id=user2 -private_key=$(cat user2.priv | base64) -for_user=user3 -config=docs/examples/go/config.conf

# output
success

Attempting to read the data now as user3 will predictably lead to an error:

go run docs/examples/go/hermes_client.go -command=read_block -doc testfile -id=user3 -private_key=$(cat user3.priv | base64) -config=docs/examples/go/config.conf

# output
panic: MidHermes read block error

Let's now also revoke the UPDATE permissions from the initial data owner — user1 (acting as user2):

go run docs/examples/go/hermes_client.go -command=revoke_update_access -doc testfile -id=user2 -private_key=$(cat user2.priv | base64) -for_user=user1 -config=docs/examples/go/config.conf

# output
success

Checking if the revocation of rights was successful:

go run docs/examples/go/hermes_client.go -command=update_block -doc testfile -id=user1 -private_key=$(cat user1.priv | base64) -meta "some user2 meta" -config=docs/examples/go/config.conf

# output
panic: MidHermes update block error

The attempt to edit (update) the data by user1 ended with an error, which means that the revocation of rights was successful.

Let's now revoke all rights to the data from user1:

go run docs/examples/go/hermes_client.go -command=revoke_read_access -doc testfile -id=user2 -private_key=$(cat user2.priv | base64) -for_user=user1 -config=docs/examples/go/config.conf

# output
success

Now it is only the user2 who has any rights to the data:

go run docs/examples/go/hermes_client.go -command=read_block -doc testfile -id=user2 -private_key=$(cat user2.priv | base64) -config=docs/examples/go/config.conf

# output
user 2 data
some user2 meta
success

Deleting the record

Let's elegantly finish this transfer of rights to the data by deleting the record altogether(acting as user2):

go run docs/examples/go/hermes_client.go -command=delete_block -doc testfile -id=user2 -private_key=$(cat user2.priv | base64) -config=docs/examples/go/config.conf

# output
success

And now, let's make sure that the data is indeed gone — if it is, we'll see an error while trying to read it as user2 (or any other user from this tutorial, for that matter):

go run docs/examples/go/hermes_client.go -command=read_block -doc testfile -id=user2 -private_key=$(cat user2.priv | base64) -config=docs/examples/go/config.conf

# output
panic: MidHermes read block error

The read block error confirms that it's now impossible to get the block as it had been deleted. So the data is gone, deleted by user2 who gained all the permissions.

Summary

As you can see, launching the storage entities for data, public and encryption keys for Hermes-core is easy, and the process of granting/revoking permissions (access) to the data between authorised users in Hermes-core is very straightforward and convenient.

We hope that this tutorial was fun and informative and that you now have gained enough understanding of how the things work with Hermes-core — and that now you’ll try using it or build a Hermes-based app of your own.

Python tutorial for Hermes-core

In this tutorial, we are going to launch storage entities for data, public and encryption keys, and will save/delete/edit the data with the help of a Hermes-core console app, as well as grant/revoke access to the data for other users. All this will be carried out cryptographically.

Launching the storage entities

The infrastructure of Hermes-core is divided into 3 parts (you can read more about each entity here and in the scientific paper on Hermes): - Data store (data storage entity), - Keystore (storage entity for keys with the help of which the data is encrypted), - Credential store (storage entity for the users' public keys with the help of which the authentification and authorization take place).

Besides, we provide keypair generator to generate keys for the users.

Installing the necessary libraries

Let's install the libraries and utilities that we're going to need.

For Debian the command is:

sudo apt-get update && sudo apt-get install build-essential libssl-dev git python3-dev

We need build-essential for building binary libraries and libssl as backend for Themis, and python-dev for our client written in Python.

If you're using another OS please refer to the Installation guide.

Let's download and install Themis into your system:

git clone https://github.com/cossacklabs/themis
cd themis
make && sudo make install
cd ..

Now you should download and install Hermes-core:

git clone https://github.com/cossacklabs/hermes-core
cd hermes-core
make && sudo make install

Building all the services necessary for the Hermes-core infrastructure

The next step is building keypair generator and stores: Keystore, Credential store, Data store.

You can find each component in docs/examples/c folder, and run make for each of them separately.

However, we recommend building them all at once:

make examples

Creating folder structure for the services

With the following command, we are creating a folder structure in which the services we are creating will store the data:

# create folder structure
mkdir -p db/credential_store
mkdir -p db/key_store
mkdir -p db/data_store

Register service's keys in the Credential store

The service examples have hardcoded private/public keys. To simplify the first run, we placed the public keys into the repository close to examples. Those files have filenames that are base64-encoded names of services that are declared in docs/examples/c/mid_hermes/common/config.h and have binary content of public keys.

cp docs/examples/c/service_keys/* db/credential_store/

Generating user keys

User 1

./docs/examples/c/key_gen/key_pair_gen user1.priv db/credential_store/$(echo -n user1 | base64)

Here we are using our key generator and command it to save a private key in a folder with the name user1.priv, and put the public key into the folder that will be assigned to/available to Credential store — db/credential_store. The filename will be a user identifier in base64. In our case, it is going to be user1.

Note: It is important that the filename is in the correct base64 format as consequently it will be used as the user identifier.

User 2

Let's create another user, this time with user2 identifier:

./docs/examples/c/key_gen/key_pair_gen user2.priv db/credential_store/$(echo -n user2 | base64)

Launching services

Now we need to launch all the necessary services that we have built.

This should be done in separate console tabs:

Credential store

./docs/examples/c/mid_hermes/credential_store_service/cs

Keystore

./docs/examples/c/mid_hermes/key_store_service/ks

Data store

./docs/examples/c/mid_hermes/data_store_service/ds

Now all the parts of Hermes-core are working and we can start using the infrastructure — add data, grant/revoke access, edit, etc.

Installing Client

The Python client for Hermes-core supports both Python 2 and Python 3 (Python 2.7.x and Python 3.3+). We recommend using Python 3.3, but you can use the version you want.

To install the Python client for Hermes-core, do the following:

sudo apt install python3 virtualenv
virtualenv --python=python3 hermes_env
source hermes_env/bin/activate
cd pyhermes
python3 setup.py install
cd -

Our Client app is ready to be used. Now, this is where the fun begins.

Creating the first document

Let's add some data — for example, let's create a file with simple content:

echo "some content" > testfile
Folder structure

This is an example folder structure — you probably have something similar right now. Docs/examples/c contains the core components of Hermes-core. The Python client is located in docs/examples/python folder. The database folder db contains encrypted data and access keys / tokens.

hermes-core/
+-- docs/examples/
|   +-- c/
|   ¦   +-- key_gen/
|   ¦   L-- mid_hermes/
|   ¦       +-- credential_store_service/
|   ¦       +-- data_store_service/
|   ¦       L-- key_store_service/
|   L-- python/
|       L-- hermes_client.py
L-- db/
|   +-- credential_store/
|   +-- data_store/
|   L-- key_store/
+-- testfile
+-- user1.priv
L-- user2.priv

Adding data

Let's add the first document into database:

python docs/examples/python/hermes_client.py --id user1 --config=docs/examples/python/config.json --private_key user1.priv --doc testfile --add --meta first-file

# output:
added <testfile> with meta <first-file>
done
  • --id user1 — specifies the identifier of the user from which we are executing an operation. This ID will tell Hermes-core which public key to use in its operations.
  • --private_key ../user1.priv — here we're specifying our private key — the user1.priv key, convert it to base64 format and pass as an argument for --private_key.
  • --doc testfile — indicates where to read our file from.
  • --add — indicates that it is a command for adding data.
  • --meta first-file — additional metadata that might be used by the current implementation of the Data store.
  • --config=docs/examples/python/config.json — config file with connection settings and public keys that will be used to establish Secure Session with Credential/Key/Data stores with predefined values for this example.

Reading data

Let's try to read this data by typing the following:

python docs/examples/python/hermes_client.py --id user1 --config=docs/examples/python/config.json --private_key user1.priv --doc testfile --read

# output:
('some content\n', 'first-file')
done

The command remains the same, only --add was changed to --read and this time we don't have to set the --meta. In all the following commands, the changes will most often be about changing the command type and the --meta parameter will be added/deleted.

So now the content and the meta data we transferred before is displayed for us. All this data is stored as files in the previously created folder ls db/data_store which will contain a folder with the name that matches the filename of the added file in base64 format — dGVzdGZpbGU=. This ls db/data_store/dGVzdGZpbGU= folder will contain 3 files:

data
mac
meta

To make sure that your data is truly encrypted, you can display it using:

cat db/data_store/dGVzdGZpbGU=/data

The output will be unprintable because the encrypted data is in binary state and you will not find the traces of the initial data in it. Currently, only the person (user) who had added the data can make can make changes to it.

But let's change the initial file and update it in Hermes-core:

echo "some new content" > testfile

And let's take another look at what is stored in Hermes-core now. The content is not changed, because we haven't pushed new testfile to data store.

python docs/examples/python/hermes_client.py --id user1 --config=docs/examples/python/config.json --private_key user1.priv --doc testfile --read

# output:
('some content\n', 'first-file')

Updating the data

Let's now try updating the data while using the identifier (ID) and the key that belong to a different user — user2:

python docs/examples/python/hermes_client.py --id user2 --config=docs/examples/python/config.json --private_key user2.priv --doc testfile --update --meta first-file

# output:
Traceback (most recent call last):
  File "hermes_client.py", line 110, in <module>
    mid_hermes.updBlock(args.doc_file_name.encode(), block, args.meta.encode())
hermes.error: MidHermes.delBlock error

As a result, we get an error. But now let's perform the update on behalf of the previous user — user1 (who was the entity that added the data in the first place):

python docs/examples/python/hermes_client.py --id user1 --config=docs/examples/python/config.json --private_key user1.priv --doc testfile --update --meta first-file

# output:
updated <testfile> with meta <first-file>
done

Let's now output (display) the data to make sure it was indeed updated:

python docs/examples/python/hermes_client.py --id user1 --config=docs/examples/python/config.json --private_key user1.priv --doc testfile --read

# output:
('some new content\n', 'first-file')

And let's make sure that user2 has no access to the data:

python docs/examples/python/hermes_client.py --id user2 --config=docs/examples/python/config.json --private_key user2.priv --doc testfile --read

# output:
Traceback (most recent call last):
  File "hermes_client.py", line 110, in <module>
    print(mid_hermes.getBlock(args.doc_file_name.encode()))
hermes.error: MidHermes.getBlock error

As we can see — user2 indeed has no access to the data.

Granting READ permissions

Now let's now grant READ permission to user2:

python docs/examples/python/hermes_client.py --id user1 --config=docs/examples/python/config.json --private_key user1.priv --doc testfile --grant_read --for_user=user2

# output:
granted read access <testfile> for user <user2>
done

Here we added a new argument --for_user=user2 to indicate, which user we are granting the access to (READ permission in this case).

So let's try and read the file using the key belonging to user2:

python docs/examples/python/hermes_client.py --id user2 --config=docs/examples/python/config.json --private_key user2.priv --doc testfile --read

# output:
('some new content\n', 'first-file')

Making sure that user2 didn't also receive the UPDATE permissions alongside with READ permissions:

python docs/examples/python/hermes_client.py --id user2 --config=docs/examples/python/config.json --private_key user2.priv --doc testfile --update --meta first-file

# output:
Traceback (most recent call last):
  File "hermes_client.py", line 110, in <module>
    mid_hermes.updBlock(args.doc_file_name.encode(), block, args.meta.encode())
hermes.error: MidHermes.delBlock error

Ok, we got an error so everything is functioning as intended.

Granting UPDATE permissions

Now, let's grant the permission to UPDATE to user2:

python docs/examples/python/hermes_client.py --id user1 --config=docs/examples/python/config.json --private_key user1.priv --doc testfile --grant_update --for_user=user2

# output:
granted update access <testfile> for user <user2>
done

Note: only user1 that already has UPDATE permission can grant UPDATE permission to user2.

Let's update the file and read it:

echo "user 2 data" > testfile
python docs/examples/python/hermes_client.py --id user2 --config=docs/examples/python/config.json --private_key user2.priv --doc testfile --update --meta first-file

# output:
updated <testfile> with meta <first-file>
done
python docs/examples/python/hermes_client.py --id user2 --config=docs/examples/python/config.json --private_key user2.priv --doc testfile --read

# output:
('user 2 data\n', 'first-file')
done

Adding one more user and distributing permissions further

Let's now add user3 just as we did before with user2, to be able to grant permissions from user2 to user3 (using the credentials of user2):

Generating keypair for user3:

./docs/examples/c/key_gen/key_pair_gen user3.priv db/credential_store/$(echo -n user3 | base64)

Granting access/permissions from user2 to user3:

python docs/examples/python/hermes_client.py --id user2 --config=docs/examples/python/config.json --private_key user2.priv --doc testfile --grant_read --for_user=user3

# output:
granted read access <testfile> for user <user3>
done

Let's try to read the data now as the user3:

python docs/examples/python/hermes_client.py --id user3 --config=docs/examples/python/config.json --private_key user3.priv --doc testfile --read

# output:
('user 2 data\n', 'first-file')
done

Making sure that user3 has no permission to perform UPDATE on the data:

python docs/examples/python/hermes_client.py --id user3 --config=docs/examples/python/config.json --private_key user3.priv --doc testfile --update --meta first-file

# output:
Traceback (most recent call last):
  File "hermes_client.py", line 110, in <module>
    mid_hermes.updBlock(args.doc_file_name.encode(), block, args.meta.encode())
hermes.error: MidHermes.delBlock error

ROTATING data and keys

At this step, let's perform data and key(s) rotation as user2. This rotation means re-encrypting the data using new keys. To make sure that the key and data rotation indeed takes place (because the data is binary and it would be hard to tell one batch of encrypted data from another), we need to calculate the hash-sum of the encrypted file before and after rotation and check if they match (they shouldn't). If after rotation we read the data again and get the same decrypted data output, it means that the rotation was successful.

Let's calculate and save the hashsum of the ecnrypted file (which is located in db/data_store/dGVzdGZpbGU=/data) into a temporary file:

sha256sum db/data_store/dGVzdGZpbGU=/data > /tmp/1.sha

Perform rotation:

python docs/examples/python/hermes_client.py --id user2 --config=docs/examples/python/config.json --private_key user2.priv --doc testfile --rotate

# output:
rotated <testfile>
done

and calculate the hashsum again:

sha256sum db/data_store/dGVzdGZpbGU=/data > /tmp/2.sha

Using the diff comand, we'll see that the previous and the new hashsums are different:

diff /tmp/1.md5 /tmp/2.md5 
1c1
< 9baad30131be8b6beb4453b0ea3aeef1  db/data_store/dGVzdGZpbGU=/data
---
> c5425afdaf906e4eeec3a70923018e34  db/data_store/dGVzdGZpbGU=/data

If the diff command provides some text output, it means that the files (hashsums) we are comparing are different. Otherwise, there would be no text output. So the key and data rotation (its re-encryption using new encryption keys) was successful.

To make sure that the data stays the same, let's read the data (as user3) again:

docs/examples/python/hermes_client.py --id user3 --config=docs/examples/python/config.json --private_key user3.priv --doc testfile --read

The resulting data is unchanged. Rotation of keys and data worked as intended.

Revoking permissions/access

To revoke the read permissions from user3 (as user2), do the following:

python docs/examples/python/hermes_client.py --id user2 --config=docs/examples/python/config.json --private_key user2.priv --doc testfile --revoke_read --for_user=user3

# output:
revoked read access <testfile> for user <user3>
done

Attempting to read the data now as user3 will predictably lead to an error:

python docs/examples/python/hermes_client.py --id user3 --config=docs/examples/python/config.json --private_key user3.priv --doc testfile --read

# output:
Traceback (most recent call last):
  File "hermes_client.py", line 110, in <module>
    print(mid_hermes.getBlock(args.doc_file_name.encode()))
hermes.error: MidHermes.getBlock error

Let's now have some fun and revoke access to the data from both user3 and user1: revoking access from user3

python docs/examples/python/hermes_client.py --id user2 --config=docs/examples/python/config.json --private_key user2.priv --doc testfile --revoke_read --for_user=user3

# output:
revoked read access <testfile> for user <user3>
done

revoking access from user1, the initial data owner

python docs/examples/python/hermes_client.py --id user2 --config=docs/examples/python/config.json --private_key user2.priv --doc testfile --revoke_read --for_user=user1

# output:
revoked read access <testfile> for user <user1>
done

Now only the user2 has the right to READ/UPDATE/etc. the data:

python docs/examples/python/hermes_client.py --id user2 --config=docs/examples/python/config.json --private_key user2.priv --doc testfile --read

# output:
('user 2 data\n', 'first-file')
done

Deleting the record

Let's elegantly finish this transfer of rights to the data by deleting the record altogether (acting as user2):

python docs/examples/python/hermes_client.py --id user2 --config=docs/examples/python/config.json --private_key user2.priv --doc testfile --delete

# output:
deleted <testfile>
done

And now, let's make sure that the data is gone indeed — if it is, we'll see an error while trying to read it as user2 (or any other user from this tutorial, for that matter):

python docs/examples/python/hermes_client.py --id user2 --config=docs/examples/python/config.json --private_key user2.priv --doc testfile --read

# output:
Traceback (most recent call last):
  File "hermes_client.py", line 110, in <module>
    print(mid_hermes.getBlock(args.doc_file_name.encode()))
hermes.error: MidHermes.getBlock error    

The MidHermes.getBlock error confirms that it's now impossible to get the block as it had been deleted. So the data is gone, deleted by user2 who gained all the permissions.

Summary

As you can see, launching the storage entities for data, public and encryption keys for Hermes-core is easy, and the process of granting/revoking permissions (access) to the data between authorised users in Hermes-core is very straightforward and convenient.

We hope that this tutorial was fun and informative and that you now have gained enough understanding of how the things work with Hermes-core — and that now you’ll try using it or build a Hermes-based app of your own.

Similar tutorials are also available for the C language and Golang.

Create a Hermes-based client-server app

Intro

Hermes-based application architecture requires 4 main system components:

  1. Credential Store: a trusted source of user/service public credentials.
  2. Data Store: a physical or logical unit for storing system data.
  3. Keystore: a physical or logical unit for storing data access control keys.

These three components are considered to be Server side.

  1. Client: an active entity in the architecture that produces or consumes the data.

All the system components need to be physically distributed to different servers for security reasons (Keystore and Data Store can theoretically be placed together). Client and Server (as well as the Server components) need to have some form of communication between them (transport layer).

Note: A quick reminder that Hermes-core currently includes examples with tutorials in C, Python, and Go - you may choose to familiarise yourself with those first before proceeding.

Architecture and design

As an abstract framework, Hermes-core doesn't include any communication and storage components, only interfaces. Communication and storage entities (Data store, Credential store, Keystore) must be implemented before using Hermes-core.

Communication (Transport)

Transport is a means of providing connection and communication between the Server and the Client, as well as between the separate Server components. There is only one requirement towards the communication between the components of Hermes-core — security.

For this reason, Hermes-core has a built-in wrapper that creates Themis' Secure Session communication channel under the abstract transport that needs to be implemented by the user. Such transport can be created using any available mechanism and it must be able to implement the following interface (include/hermes/rpc/transport.h):

typedef uint32_t(*hm_rpc_transport_send_t)(void *transport, const uint8_t *buffer, const size_t buffer_length);

typedef uint32_t(*hm_rpc_transport_recv_t)(void *transport, uint8_t *buffer, size_t buffer_length);

typedef uint32_t(*hm_rpc_transport_get_remote_id_t)(void *transport, uint8_t **id, size_t *id_length);

typedef struct hm_rpc_transport_type {
    hm_rpc_transport_send_t send;
    hm_rpc_transport_recv_t recv;
    hm_rpc_transport_get_remote_id_t get_remote_id;
    void *user_data;
} hm_rpc_transport_t;

There is already implemented Secure Session transport interface in include/hermes/secure_transport/transport.h.

typedef struct secure_transport_type {
    // transport that will be wrapped
    hm_rpc_transport_t* user_transport;
    // secure session for this connection
    secure_session_t* session;
    secure_session_user_callbacks_t* session_callback;

} secure_transport_t;

uint32_t destroy_secure_transport(secure_transport_t** transport_);
uint32_t destroy_rpc_secure_transport(hm_rpc_transport_t** transport_);
hm_rpc_transport_t* create_secure_transport(
        const uint8_t *user_id, size_t user_id_length,
        const uint8_t *private_key, size_t private_key_length,
        const uint8_t *public_key, size_t public_key_length,
        const uint8_t *public_key_id, size_t public_key_id_length,
        hm_rpc_transport_t* user_transport,
        bool is_server);

hm_rpc_transport_t* create_secure_transport_with_callback(
        const uint8_t *user_id, size_t user_id_length,
        const uint8_t *private_key, size_t private_key_length,
        secure_session_user_callbacks_t* callback,
        hm_rpc_transport_t* user_transport,
        bool is_server);
````


A simple TCP/IP socket transport implementation can be found in the following examples:   
* [`docs/examples/c/mid_hermes/common/transport.h`](https://github.com/cossacklabs/hermes-core/blob/master/docs/examples/c/mid_hermes/common/transport.h)     
* [`docs/examples/c/mid_hermes/common/transport.c`](https://github.com/cossacklabs/hermes-core/blob/master/docs/examples/c/mid_hermes/common/transport.c)

Here are the examples of the way to wrap a simple transport into Secure Session:
* [`docs/examples/c/mid_hermes/client/hermes_client.c`](https://github.com/cossacklabs/hermes-core/blob/master/docs/examples/c/mid_hermes/client/hermes_client.c)
* [`docs/examples/c/mid_hermes/credential_store_service/main.c`](https://github.com/cossacklabs/hermes-core/blob/master/docs/examples/c/mid_hermes/credential_store_service/main.c)
* [`docs/examples/c/mid_hermes/key_store_service/main.c`](https://github.com/cossacklabs/hermes-core/blob/master/docs/examples/c/mid_hermes/key_store_service/main.c)
* [`docs/examples/c/mid_hermes/data_store_service/main.c`](https://github.com/cossacklabs/hermes-core/blob/master/docs/examples/c/mid_hermes/data_store_service/main.c)

To check out the transport interfaces for Python and Go, see the corresponding examples:
- Python examples: [docs/examples/python/hermes_client.py](https://github.com/cossacklabs/hermes-core/blob/master/docs/examples/python/hermes_client.py),
- Golang examples: [docs/examples/go/hermes_client.go](https://github.com/cossacklabs/hermes-core/blob/master/docs/examples/go/hermes_client.go).

Alternatively, you may choose to implement transport using your own preferred means (i.e. TLS or an unencrypted connection).

##### Data store
Hermes-core doesn't have special requirements for the Data store database. Such database needs to be able to implement the following interface ([`include/hermes/data_store/db.h`](https://github.com/cossacklabs/hermes-core/blob/master/include/hermes/data_store/db.h)):
```c
typedef uint32_t(*hm_ds_db_insert_block)(
        void* db, const uint8_t* block, const size_t block_length,
        const uint8_t* meta, const size_t meta_length,
        const uint8_t* mac, const size_t mac_length,
        uint8_t** id, size_t* id_length);
typedef uint32_t(*hm_ds_db_insert_block_with_id)(
        void* db, const uint8_t* id, const size_t id_length,
        const uint8_t* block, const size_t block_length,
        const uint8_t* meta, const size_t meta_length,
        const uint8_t* mac, const size_t mac_length);
typedef uint32_t(*hm_ds_db_get_block)(
        void* db, const uint8_t* id, const size_t id_length,
        uint8_t** block, size_t*  block_length,
        uint8_t** meta, size_t*  meta_length);
typedef uint32_t(*hm_ds_db_update_block)(
        void* db, const uint8_t* id, const size_t id_length,
        const uint8_t* block, const size_t block_length,
        const uint8_t* meta, const size_t meta_length,
        const uint8_t* mac, const size_t mac_length,
        const uint8_t* old_mac, const size_t old_mac_length);
typedef uint32_t(*hm_ds_db_rem_block)(
        void* db, const uint8_t* id, const size_t id_length,
        const uint8_t* old_mac, const size_t old_mac_length);

typedef struct hm_ds_db_type{
    void* user_data;
    //insert block to data store. Will return id of added block
    hm_ds_db_insert_block insert_block; 
    //insert block to data store with predefined id. if block with provided id is already present in data store error will return
    hm_ds_db_insert_block_with_id insert_block_with_id;
    //read block from data store
    hm_ds_db_get_block get_block;
    //update block in data store 
    hm_ds_db_update_block update_block;
    //delete block from data store
    hm_ds_db_rem_block rem_block;
}hm_ds_db_t;

A simple filesystem-based data storage implementation can be found in the following examples: docs/examples/c/mid_hermes/data_store_service/db.h docs/examples/c/mid_hermes/data_store_service/db.c

Сredential store

Hermes-core doesn't have special requirements towards the Credential store database. The database needs to be able to implement the following interface (include/hermes/credential_store/db.h):

typedef uint32_t(*hm_cs_db_get_pub_by_id_t)(void *db, const uint8_t *id, const size_t id_length, uint8_t **key, size_t *key_length);

typedef struct hm_cs_db_type {
    void *user_data;
    // get public key by provided user id
    hm_cs_db_get_pub_by_id_t get_pub;
} hm_cs_db_t;

A simple filesystem-based Credential store implementation can be found in the following examples: docs/examples/c/mid_hermes/credential_storage_service/db.h docs/examples/c/mid_hermes/credential_storage_service/db.c

Keystore

Hermes-core doesn't have special requirements towards the Keystore database. The database needs to be able to implement the following interface (include/hermes/key_store/db.h):

typedef uint32_t(*hm_ks_db_set_token)(
        void* db, const uint8_t* block_id, const size_t block_id_length,
        const uint8_t* user_id, const size_t user_id_length,
        const uint8_t* owner_id, const size_t owner_id_length,
        const uint8_t* read_token, const size_t read_token_length);
typedef uint32_t(*hm_ks_db_get_token)(
        void* db, const uint8_t* block_id, const size_t block_id_length,
        const uint8_t* user_id, const size_t user_id_length,
        uint8_t** write_token, size_t* write_token_id_length,
        uint8_t** owner_id, size_t* owner_id_length);
typedef uint32_t(*hm_ks_db_del_token)(
        void* db, const uint8_t* block_id, const size_t block_id_length,
        const uint8_t* user_id, const size_t user_id_length,
        const uint8_t* owner_id, const size_t owner_id_length);
typedef uint32_t(*hm_ks_db_get_indexed_rights)(
        void* db, const uint8_t* block_id, const size_t block_id_length,
        const size_t index, uint8_t** user_id,
        size_t* user_id_length, uint32_t* rights_mask);

typedef struct hm_ks_db_type{
    void* user_data;
    // set read token
    hm_ks_db_set_token set_rtoken;
    // set update token
    hm_ks_db_set_token set_wtoken;
    // get read token
    hm_ks_db_get_token get_rtoken;
    // get update token
    hm_ks_db_get_token get_wtoken;
    // return user id and user rights mask ("r" or "w") with shift by index 
    hm_ks_db_get_indexed_rights get_indexed_rights;
    // delete read token
    hm_ks_db_del_token del_rtoken;
    // delete update token
    hm_ks_db_del_token del_wtoken;
}hm_ks_db_t;

A simple filesystem-based Keystorage implementation can be found in the following examples:
docs/examples/c/mid_hermes/key_storage_service/db.h docs/examples/c/mid_hermes/key_storage_service/db.c

Step by step

After implementing all the necessary interfaces, the necessary Hermes-core components can be created using the following method:

1. Credential store, Data store, Keystore

Use an appropriate service helper: include/hermes/credential_store/service.h or include/hermes/data_store/service.h or include/hermes/key_store/service.h, i.e.:

hm_credential_store_service_t* service = hm_credential_store_service_create(transport, db);

The service is a helper object, which launches an infinite loop after the start method has been called to receive a command, execute, send the result, repeat, etc. However, start is a blocking method, so for a more efficient implementation, each service needs to be created in a separate thread.

The default Data/Credential/Keystore thread implementation may look like this docs/examples/c/mid_hermes/data_store_service/main.c:

void* data_store(void* arg){
    // create transport with credential store to use him as secure session callback for retrieving public keys
    hm_rpc_transport_t* raw_credential_store_transport = server_connect(CREDENTIAL_STORE_IP, CREDENTIAL_STORE_PORT);
    if (!raw_credential_store_transport){
        perror("can't connect to credential store\n");
        return (void*)FAIL;
    }
    hm_rpc_transport_t* credential_store_transport = create_secure_transport(
            data_store_id, strlen((char*)data_store_id), data_store_private_key, sizeof(data_store_private_key),
            credential_store_pk, sizeof(credential_store_pk),
            credential_store_id, strlen((char*)credential_store_id), raw_credential_store_transport, false);
    if(!credential_store_transport){
        perror("can't initialize secure transport to credential store\n");
        transport_destroy(&raw_credential_store_transport);
        perror("can't create connection to credential store\n");
        return (void*)FAIL;
    }

    // create secure transport with new client
    hm_rpc_transport_t* client_transport=transport_create((int)(intptr_t)arg);
    if(!client_transport){
        perror("client transport creation error ...\n");
        return (void*)FAIL;
    }

    secure_session_user_callbacks_t* session_callback = get_session_callback_with_remote_credential_store(
            credential_store_transport);
    hm_rpc_transport_t* secure_client_transport = create_secure_transport_with_callback(
            data_store_id, strlen((char*)data_store_id),data_store_private_key, sizeof(data_store_private_key),
            session_callback, client_transport, true);

  hm_ds_db_t* db=db_create();
  if(!db){
    transport_destroy(&client_transport);
    perror("can't create data store\n");
    return (void*)FAIL;
  }
  hm_data_store_service_t* service=hm_data_store_service_create(secure_client_transport, db);
  if(!service){
    transport_destroy(&client_transport);
    perror("service creation error ...\n");
    return (void*)FAIL;
  }
  fprintf(stderr, "service started ...\n");
  hm_data_store_service_start(service);
  fprintf(stderr, "service stoped ...\n");
  hm_data_store_service_destroy(&service);
  transport_destroy(&client_transport);
  return NULL;
}

Data store and Keystore use Credential store to receive the public keys of clients that connect to them. So for processing the clients' requests, they use Credential store as a callback in Secure Session to receive the public key(s). This means that a connection to Credential Store must be created:

hm_rpc_transport_t* raw_credential_store_transport = server_connect(CREDENTIAL_STORE_IP, CREDENTIAL_STORE_PORT);
if (!raw_credential_store_transport){
    perror("can't connect to credential store\n");
    return (void*)FAIL;
}

Wrap the connection in Secure Session and indicate that it is the client's connection through the last false parameter:

hm_rpc_transport_t* credential_store_transport = create_secure_transport(
        data_store_id, strlen((char*)data_store_id), data_store_private_key, sizeof(data_store_private_key),
        credential_store_pk, sizeof(credential_store_pk),
        credential_store_id, strlen((char*)credential_store_id), raw_credential_store_transport, false);

Then create a callback for Secure Session, which will use Credential store for receiving the public keys:

secure_session_user_callbacks_t* session_callback = get_session_callback_with_remote_credential_store(
            credential_store_transport);

Wrap the client connection into the corresponding transport interface (see an example here - docs/examples/c/mid_hermes/common/transport.c):

// create secure transport with new client
hm_rpc_transport_t* client_transport=transport_create((int)(intptr_t)arg);

Wrap the connection into Secure Session:

hm_rpc_transport_t* secure_client_transport = create_secure_transport_with_callback(
        data_store_id, strlen((char*)data_store_id),data_store_private_key, sizeof(data_store_private_key),
        session_callback, client_transport, true);

What comes next is the initialization of the service with the transport being passed to it.

This procedure is also applicable for Keystore and Data store that use Credential Store in Secure Session. However, this process will be slightly different for Credential Store because it will have to use itself for the authentication of users:

secure_session_user_callbacks_t* session_callback = get_session_callback_with_local_credential_store(db);

hm_rpc_transport_t* secure_transport = create_secure_transport_with_callback(
      credential_store_id, strlen((char*)credential_store_id),
      credential_store_private_key, sizeof(credential_store_private_key),
      session_callback, client_transport, true);

Here a callback is explicitly created with a function for receiving the public key, which will use the Credential store's own mechanics (the 'db' object). The implementation of the functions for receiving the public key can be found here - src/secure_transport/session_callback.c. The rest of the process for initializing the service is similar to that described for Data store and Credential store.

2. Client

The element of Hermes-core that is the most important for the creation of Client is mid_hermes_t with the following interface (include/hermes/mid_hermes/mid_hermes.h):

typedef struct mid_hermes_type mid_hermes_t;

mid_hermes_t *mid_hermes_create(
        const uint8_t *user_id, const size_t user_id_length,
        const uint8_t *private_key, const size_t private_key_length,
        hm_rpc_transport_t *key_store_transport,
        hm_rpc_transport_t *data_store_transport,
        hm_rpc_transport_t *credential_store_transport);

hermes_status_t mid_hermes_destroy(mid_hermes_t **mh);

hermes_status_t mid_hermes_create_block(
        mid_hermes_t *mid_hermes,
        uint8_t **id, size_t *id_length,
        const uint8_t *block, const size_t block_length,
        const uint8_t *meta, const size_t meta_length);

hermes_status_t mid_hermes_read_block(
        mid_hermes_t *mid_hermes,
        const uint8_t *block_id, const size_t block_id_length,
        uint8_t **block, size_t *block_length,
        uint8_t **meta, size_t *meta_length);

hermes_status_t mid_hermes_update_block(
        mid_hermes_t *mid_hermes,
        const uint8_t *block_id, const size_t block_id_length,
        const uint8_t *block, const size_t block_length,
        const uint8_t *meta, const size_t meta_length);

hermes_status_t mid_hermes_delete_block(
        mid_hermes_t *mid_hermes, const uint8_t *block_id, const size_t block_id_length);

hermes_status_t mid_hermes_rotate_block(
        mid_hermes_t *mid_hermes, const uint8_t *block_id, const size_t block_id_length);

hermes_status_t mid_hermes_grant_read_access(
        mid_hermes_t *mid_hermes,
        const uint8_t *block_id, const size_t bloc_id_length,
        const uint8_t *user_id, const size_t user_id_length);

hermes_status_t mid_hermes_grant_update_access(
        mid_hermes_t *mid_hermes,
        const uint8_t *block_id, const size_t bloc_id_length,
        const uint8_t *user_id, const size_t user_id_length);

hermes_status_t mid_hermes_deny_read_access(
        mid_hermes_t *mid_hermes,
        const uint8_t *block_id, const size_t bloc_id_length,
        const uint8_t *user_id, const size_t user_id_length);

hermes_status_t mid_hermes_deny_update_access(
        mid_hermes_t *mid_hermes,
        const uint8_t *block_id, const size_t bloc_id_length,
        const uint8_t *user_id, const size_t user_id_length);

To construct the mid_hermes_t instance function mid_hermes_create, in addition to the user id and the user private key, three transports instances to Credential store, Data store, and Keystore are needed respectively.

You can check client implementation in docs/examples/c/mid_hermes/client/hermes_client.c.

First, you need to connect to all the services of Credential store, Data store, and Keystore, and create a Secure Session between them using the Hermes wrapper (unless you want to use non-secure unencrypted open transports like TCP/UDP/Websocket or want to implement your own means of supporting encryption like TLS/SSL).

Create a connection to Credential store:

transports_container_t container = {NULL, NULL, NULL, NULL, NULL, NULL};
container.raw_credential_store_transport = server_connect(CREDENTIAL_STORE_IP, CREDENTIAL_STORE_PORT);

Here calling the function server_connect(CREDENTIAL_STORE_IP, CREDENTIAL_STORE_PORT) must return the implementation of the transport interface defined in include/hermes/rpc/transport.h.

A simple TCP/IP socket transport implementation can be found in the following examples: docs/examples/c/mid_hermes/common/transport.h docs/examples/c/mid_hermes/common/transport.c

Transport needs to be wrapped into Secure Session by calling the create_secure_transport function from include/hermes/secure_transport/transport.h and passing the user's id, user's public key that will be used for establishing the session, ID of the service we're connecting to (in this case it is Credential store), the service's public key, and the transport that's being wrapped.

The type of connection that needs to be established must also be indicated here - either the server type (then the last parameter will be true) or the client type (the last parameter will be false).

The necessity to indicate the connection type is due to the fact that the session is always initialized by the Client who needs to send a request for establishing a session. You can read more here.

container.credential_store_transport = create_secure_transport(
        user_id, user_id_length, 
        user_private_key, user_private_key_length, 
        credential_store_pk, sizeof(credential_store_pk),
        credential_store_id, strlen((char*)credential_store_id), 
        container.raw_credential_store_transport, 
        false);

The connection with Data store and Keystore is created in a similar manner 1, 2.

Now the mid_hermes object can be created, requests to the API will be sent through it. When creating the mid_hermes object, the following parameters need to be passed - user_id and its private_key, as well as the 3 connections to the services, created earlier:

mh = mid_hermes_create(
    user_id, user_id_length,
    user_private_key, user_private_key_length,
    container.key_store_transport,
    container.data_store_transport,
    container.credential_store_transport)

After a successful creation of mid_hermes_t instance, each instance of the interface method can be called. Each mid_hermes_t interface method represents one of the Hermes operations:

1. CREATE block
hermes_status_t mid_hermes_create_block(
        mid_hermes_t *mid_hermes,
        uint8_t **id, size_t *id_length,
        const uint8_t *block, const size_t block_length,
        const uint8_t *meta, const size_t meta_length);
2. READ block
hermes_status_t mid_hermes_read_block(
        mid_hermes_t *mid_hermes,
        const uint8_t *block_id, const size_t block_id_length,
        uint8_t **block, size_t *block_length,
        uint8_t **meta, size_t *meta_length);
3. UPDATE block
hermes_status_t mid_hermes_update_block(
        mid_hermes_t *mid_hermes,
        const uint8_t *block_id, const size_t block_id_length,
        const uint8_t *block, const size_t block_length,
        const uint8_t *meta, const size_t meta_length);
4. DELETE block
hermes_status_t mid_hermes_delete_block(
        mid_hermes_t *mid_hermes, 
        const uint8_t *block_id, const size_t block_id_length);
5. ROTATE block
hermes_status_t mid_hermes_rotate_block(
        mid_hermes_t *mid_hermes, 
        const uint8_t *block_id, const size_t block_id_length);
6. GRANT read access
hermes_status_t mid_hermes_grant_read_access(
        mid_hermes_t *mid_hermes,
        const uint8_t *block_id, const size_t bloc_id_length,
        const uint8_t *user_id, const size_t user_id_length);
7. GRANT update access
hermes_status_t mid_hermes_grant_update_access(
        mid_hermes_t *mid_hermes,
        const uint8_t *block_id, const size_t bloc_id_length,
        const uint8_t *user_id, const size_t user_id_length);
8. DENY read access
hermes_status_t mid_hermes_deny_read_access(
        mid_hermes_t *mid_hermes,
        const uint8_t *block_id, const size_t bloc_id_length,
        const uint8_t *user_id, const size_t user_id_length);
9. DENY update access
hermes_status_t mid_hermes_deny_update_access(
        mid_hermes_t *mid_hermes,
        const uint8_t *block_id, const size_t bloc_id_length,
        const uint8_t *user_id, const size_t user_id_length);