Java Keytool, TLS, and Zookeeper Security

Introduction

I recently had to deal with setting up TLS on ZooKeeper to troubleshoot some issues. It is then I realized how complicated the current PKI toolchain is. In this short article, I’m going to walk you through the basics of how PKI works and how these tools work with each other. By the end of this article you should be able to:

1. Understand how truststore and keystore works
2. Setup required keystore and truststore for TLS enabled applications
3. Test this setup with ZooKeeper

Chain of trust in secured communication

The fundamental idea behind public-key security is that you can verify information signed with a private key with the corresponding public key. That information is captured in what is usually referred to as certificates.

For example, when you are connecting to google.com over HTTPS, the communication is considered as “secure” because it uses encryption. The encryption keys are sent over in a container called “certificates.” Putting this in simple terms, to ensure the website you are visiting is actually from Google, not some third-party hijacking it, your browser contains a set of authoritative sources who can vouch for Google’s behalf and validate that those encryption certificates are authentic. Those authoritative sources are referred to as certificate authorities (CA).

As you can see, in this example, GTS Root R1 is pre-installed on your computer and is considered fully trusted. Then it digitally signed GTS CA 1C3, which you trust now. Then finally it signed the certificate that Google used to communicate with you. This is what’s called a “chain of trust”. And similarly, any TLS encrypted application has those 2 parts: trusted CA and the actual encryption key used for communication.

Java TLS security: Keystore and Truststore

In the Java world, the trusted CAs are stored in “Truststore” and the encryption keys are stored in “Keystore.” When a client receives an encryption key (via certificate) it checks the certificate’s authenticity using the list of trusted CAs in the Truststore. The server needs both the Keystore and Truststore to establish its authenticity.

What is keytool?

Keytool is a command that’s shipped with Java. You should be able to invoke keytool from the terminal if you have JDK installed.

How to generate keys for the local development environment

Step 1: Generate a certificate that identifies the server

$keytool -genkeypair -alias mycert -keyalg RSA -keysize 2048 -dname "cn=localhost" -keypass keypassword -keystore keystore.jks -storepass keystorepassword

What this command does:

1. Alias: An unique identifier for your certificate. This can be anything, just remember what you choose.
2. Keyalg: Key algorithm–the algorithm to use for the key. RSA is the standard. The other option is DES but I don’t recommend it.
3. Keysize: 2048 is usually good enough.
4. dname: This allows you to specify “distinguished name.” The distinguished name is just a fancy way of referring to the collection of things that identifies you. In this case, we care about CN (common name) within dname. CN is simply a fully qualified domain name. The example I provided above tells the certificate that it can validate things from the domain of localhost.
5. keypass: oftentimes a certificate’s private key needs to be encrypted, thus you’d like to specify “keypassword” or something similar.
6. keystore: Which file to store the certificate in
7. storepass: The key store file also has a password to protect it.

Overall, this command will create a certificate that can be used to verify “localhost” and store it into a keystore.jks. But this is not enough, because what if someone hijacks the connection between you and the client and fakes this certificate? This is where we’ll need to get the certificate to be signed by someone else – a certificate authority. But given this is a development environment, we can just force the application to trust this certificate.

Step 2: Build more trust

keytool -exportcert -alias mycert -keystore keystore.jks -file thiscert.cer -rfc

It will prompt you for keystore password, which you’ll enter “keystorepassword” (without the quotes). This command exports your certificate from keystore.jks into thiscert.cer file. Remember, keystore can actually contain more than one certificate, but usually only one certificate. The next part you want to import “thiscert.cert” into something called a truststore.jks. When you give “truststore” to your application, your application will trust all the certificates within.

keytool -importcert -alias mycert -file thiscert.cer -keystore truststore.jks -storepass truststorepassword

In this case, what you’ve done is to build a store of “trusted certificates.”

Step 3: Run your application

Most java applications will require you to specify both truststore and keystore.

For example, ZooKeeper requires the following parameters to be set:

ssl.trustStore.location= truststore.jks
ssl.trustStore.password=“truststorepassword”
ssl.keyStore.location= keystore.jks
ssl.keyStore.password=“keystorepassword”

Interestingly, hostname validation only validates the hostname and not the port number. This means multiple services on the same host can share the same certificate.

Example

In order to test this setup, you can use the docker-compose.yaml below to set up your local mini ZooKeeper cluster.

version: '2'

services:
  zookeeper1:
    image: 'bitnami/zookeeper:latest'
    ports:
      - '3881:3181'
      - '2888'
      - '3888'
    volumes:
      - ./zk01:/bitnami/zookeeper
      - ./keystore.jks:/bitnami/zookeeper/certs/keystore.jks:ro
      - ./truststore.jks:/bitnami/zookeeper/certs/truststore.jks:ro
    environment:
      - ZOO_SERVER_ID=1
      - ZOO_SERVERS=0.0.0.0:2888:3888,zookeeper2:2888:3888,zookeeper3:2888:3888
      - ZOO_ENABLE_AUTH=no
      - ALLOW_ANONYMOUS_LOGIN=yes
      - ZOO_TLS_CLIENT_ENABLE=yes
      - ZOO_TLS_CLIENT_KEYSTORE_FILE=/bitnami/zookeeper/certs/keystore.jks
      - ZOO_TLS_CLIENT_KEYSTORE_PASSWORD=keystorepassword
      - ZOO_TLS_CLIENT_TRUSTSTORE_FILE=/bitnami/zookeeper/certs/truststore.jks
      - ZOO_TLS_CLIENT_TRUSTSTORE_PASSWORD=truststorepassword
      - ZOO_TLS_CLIENT_AUTH=none

  zookeeper2:
    image: 'bitnami/zookeeper:latest'
    ports:
      - '3882:3181'
      - '2888'
      - '3888'
    volumes:
      - ./zk02:/bitnami/zookeeper
      - ./keystore.jks:/bitnami/zookeeper/certs/keystore.jks:ro
      - ./truststore.jks:/bitnami/zookeeper/certs/truststore.jks:ro
    environment:
      - ZOO_SERVER_ID=2
      - ZOO_SERVERS=zookeeper1:2888:3888,0.0.0.0:2888:3888,zookeeper3:2888:3888
      - ZOO_ENABLE_AUTH=no
      - ALLOW_ANONYMOUS_LOGIN=yes
      - ZOO_TLS_CLIENT_ENABLE=yes
      - ZOO_TLS_CLIENT_KEYSTORE_FILE=/bitnami/zookeeper/certs/keystore.jks
      - ZOO_TLS_CLIENT_KEYSTORE_PASSWORD=keystorepassword
      - ZOO_TLS_CLIENT_TRUSTSTORE_FILE=/bitnami/zookeeper/certs/truststore.jks
      - ZOO_TLS_CLIENT_TRUSTSTORE_PASSWORD=truststorepassword
      - ZOO_TLS_CLIENT_AUTH=none
  zookeeper3:
    image: 'bitnami/zookeeper:latest'
    ports:
      - '3883:3181'
      - '2888'
      - '3888'
    volumes:
      - ./zk03:/bitnami/zookeeper
      - ./keystore.jks:/bitnami/zookeeper/certs/keystore.jks:ro
      - ./truststore.jks:/bitnami/zookeeper/certs/truststore.jks:ro
    environment:
      - ZOO_SERVER_ID=3
      - ZOO_SERVERS=zookeeper1:2888:3888,zookeeper2:2888:3888,0.0.0.0:2888:3888
      - ZOO_ENABLE_AUTH=no
      - ALLOW_ANONYMOUS_LOGIN=yes
      - ZOO_TLS_CLIENT_ENABLE=yes
      - ZOO_TLS_CLIENT_KEYSTORE_FILE=/bitnami/zookeeper/certs/keystore.jks
      - ZOO_TLS_CLIENT_KEYSTORE_PASSWORD=keystorepassword
      - ZOO_TLS_CLIENT_TRUSTSTORE_FILE=/bitnami/zookeeper/certs/truststore.jks
      - ZOO_TLS_CLIENT_TRUSTSTORE_PASSWORD=truststorepassword
      - ZOO_TLS_CLIENT_AUTH=none