Configure a Vault server to hold your archive keystore

This article applies to CrashPlan Enterprise and MSPs.png

Overview

External keystores mean that even in the CrashPlan cloud environment, you can maintain copies of your CrashPlan encryption keys that secure your backed-up data.

While the keys reside securely on your users' devices for use in encrypting and decrypting files, you can store a copy of the encryption keys separately from the CrashPlan cloud, in a Vault keystore system. Vault is a third-party application specifically built to secure secrets. This article describes how to configure a private, self-administered Vault server to store copies of your CrashPlan encryption keys.

Once you have your Vault server running and tested as described in this article, move copies of your CrashPlan keys into Vault by following the instructions for migrating keys to a new keystore.

 Let CrashPlan manage your keys

Instead of managing your encryption keys in Vault, CrashPlan can manage your keys for you. See How CrashPlan handles your encryption keys for file backup for details. For more information, contact your Customer Success Manager (CSM).

Considerations

 Vault is not a CrashPlan product

Our technical support engineers can assist you with migrating your keystore to your private, self-administered Vault. Crashplan technical support cannot, however, provide assistance with Vault-specific tasks, such as upgrade, installation, configuration, networking, and exporting certificates. For assistance with Vault, consult the Vault documentation.

Vault versions

To ensure secure operation, install the latest Vault version from the Vault downloads page. Previous versions are available from the Vault releases page

Vault uses two SSL certificates

A Vault server connecting with the CrashPlan cloud uses two CA-signed SSL certificates:

  • Your Vault domain certificate secures your Vault server's domain (for example, vault.example.com). It provides encryption for all communications between Vault and the CrashPlan cloud. It's the same process at work in most HTTPS connections between clients and servers.
  • Your Vault user/administrator certificate authenticates the user of your Vault server who administers your CrashPlan cloud key storage. Your Vault server uses this certificate to authenticate and authorize requests from your CrashPlan cloud organization.

Before you begin

Install a Vault server

Install a Vault server at a location available to the CrashPlan cloud. For system requirements, see the Vault Reference Architecture guide. 

  1. Set up a machine or virtual machine with network access that allows TLS/HTTPS communication with the CrashPlan cloud.
  2. Plan storage capacity of roughly 1 KB for each of your CrashPlan users.
  3. Download a Vault package for that machine's operating system.
  4. Install and configure a Vault server as described by the Vault documentation
  5. Connect your Vault server to whatever storage backend suits your purposes.

Back up Vault

To guard against loss of your encryption keys, make sure that you back up your Vault. For Vault's backup instructions, see the Vault Disaster Recovery Replication guide.

 Back up your Vault!

When you create a private keystore, create a scheduled process to back it up.
Losing a self-administered private keystore is catastrophic. A wide range of CrashPlan functions fail. You cannot create new users. You cannot replace a lost or damaged device from backup. You cannot restore data via the console. The only sure way to protect your data is to restart all user backups from scratch.

Configure the Vault server

The following steps describe configuring a Vault server on a Linux operating system.

 Sample commands

The commands below are samples only. CrashPlan does not guarantee that these commands are suitable for your environment.

Step 1: Configure your Vault to work with CrashPlan

To allow the CrashPlan cloud access to your encryption keys, configure your Vault server. For more information about configuring Vault, see Vault's documentation

Change the values in the examples below as needed for your environment.

  1. Set environment variables:
    1. Set Vault_ADDR to the Vault external DNS, including the protocol (https) and port mapping (8200):

      export VAULT_ADDR="https://vault.example.com:8200"
    2. Set Vault_DNS to the Vault external DNS name:

      export VAULT_DNS="vault.example.com"
  2. Enable the secrets engine.
    1. Enable the secrets engine with the kv option run in KV Version 1 mode:

      vault secrets enable -path=secret kv
    2. Enable certificate authentication:

      vault auth enable cert
    3. Tune Vault to issue short-lived authentication tokens for TLS connections:

      vault secrets tune -default-lease-ttl=60s auth/cert
    4. Mount the Vault public key infrastructure (PKI):

      vault secrets enable pki
    5. Tune Vault to set the life span of the SSL certificates it generates;

      vault secrets tune -max-lease-ttl=87600h pki
  3. Generate certificates.
    1. Generate and configure CA_cert.crt:

      vault write -field=certificate pki/root/generate/internal common_name="crashplan.com" ttl=87600h > CA_cert.crt vault write pki/config/urls issuing_certificates="http://${VAULT_DNS}/v1/pki/ca" crl_distribution_points="http://${VAULT_DNS}:8200/v1/pki/crl"
    2. Generate intermediate certificates:

      vault secrets enable -path=pki_int pki vault secrets tune -max-lease-ttl=43800h pki_int vault write -format=json pki_int/intermediate/generate/internal common_name="crashplan.com Intermediate Authority" ttl="43800h" | jq -r '.data.csr' > pki_intermediate.csr vault write -format=json pki/root/sign-intermediate csr=@pki_intermediate.csr format=pem_bundle ttl="43800h" | jq -r '.data.certificate' > intermediate.cert.pem vault write pki_int/intermediate/set-signed certificate=@intermediate.cert.pem
    3. Create a role to generate credentials:

      vault write pki_int/roles/cpRole allow_any_name="true" allow_subdomains=true client_flag=true
    4. Issue the crashplan.com administration certificate and parse it:

      vault write pki_int/issue/cpRole common_name="crashplan.com" ttl="8760h" -format=json > certs.json cat certs.json | jq -r '.data.ca_chain[0]' > ca_chain.pem cat certs.json | jq -r '.data.certificate' > certificate.pem cat certs.json | jq -r '.data.private_key' > private_key.pem
       

       Set TTL as needed

      A setting ttl="8760h" sets the "time to live" for the crashplan.com administration certificate to one year. Update this setting according to your company's best practices for certificate management.

      Before this certificate reaches its expiration date as set by the TTL setting, you must make a new certificate to maintain uninterrupted connection to CrashPlan. See Create a new administration certificate below.

  4. Create policies.
    1. Create crashplanPolicyFile.hcl with the following policy entries:

      path "pki/issue/*" {capabilities = ["create","read", "update", "delete", "list"]} path "sys/policy/*" {capabilities = ["create", "read", "update", "delete", "list"]} path "auth/cert/certs/*" {capabilities = ["create", "read", "update", "delete", "list"]}
    2. Write the policy and authorize the certificate:

      vault policy write crashplanPolicy crashplanPolicyFile.hcl vault write auth/cert/certs/crashplanUser display_name="crashplanUser" policies=crashplanPolicy certificate=@certificate.pem ttl=60
  5. Create a PKCS12 key and certificate file for import to the CrashPlan cloud:

    export LC_CTYPE=C openssl_pw=$(head /dev/urandom | tr -dc A-Za-z0-9 | head -c 16 ; echo '') echo "openssl_pw: ${openssl_pw}" >> secrets.env openssl pkcs12 -export -out crashplan.p12 -inkey private_key.pem -in certificate.pem -password pass:${openssl_pw}
  6. Clean up files:

    rm ./CA_cert.crt rm ./pki_intermediate.csr rm ./intermediate.cert.pem rm ./certs.json rm ./certificate.pem rm ./private_key.pem rm ./ca_chain.pem rm ./crashplanPolicyFile unset openssl_pw

Step 2: Open your firewall to the CrashPlan cloud

Open your firewall to our CrashPlan IP addresses

Implement firewall rules to open an inbound port at your Vault server that allows requests from the CrashPlan cloud. This allows the CrashPlan cloud to access keys in your Vault to encrypt and decrypt user backup data.

Step 3: Configure the CrashPlan cloud to work with your Vault

The crashplan.p12 file created at the end of Step 1 is a PKCS12 certificate (also called a PFX or P12 file) that identifies your CrashPlan cloud organization to your Vault server.

Provide the file and its password to the CrashPlan cloud by following the instructions for migrating keys to a new keystore.

The maximum file size is 5 MB.

Create a new administration certificate

The preceding steps describe how to configure your Vault installation to work with CrashPlan for the first time, and demonstrate how to set the Time To Live (TTL) for the original administration certificate (crashplan.com certificate). Before the original certificate expires per the TTL setting, you must create a new certificate to replace the original certificate to maintain an uninterrupted connection to CrashPlan.

Following are sample steps to create the new administration certificate. Adapt the steps as needed to fit your environment.

  1. Issue a new crashplan.com administration certificate and split the result into separate files:


    vault write pki_int/issue/cpRole common_name="crashplan.com" ttl="8760h" -format=json > certs.json cat certs.json | jq -r '.data.ca_chain[0]' > ca_chain.pem cat certs.json | jq -r '.data.certificate' > certificate.pem cat certs.json | jq -r '.data.private_key' > private_key.pem
  2. Update the system certificate in the Vault certificate store:

    vault write auth/cert/certs/crashplanUser display_name="crashplanUser" policies=crashplanPolicy certificate=@certificate.pem ttl=60
  3. Generate a PFX/P12 file from the issued certificate:

    export LC_CTYPE=C openssl_pw=$(head /dev/urandom | tr -dc A-Za-z0-9 | head -c 16 ; echo '') echo "openssl_pw: ${openssl_pw}" >> secrets.env openssl pkcs12 -export -out crashplan.p12 -inkey private_key.pem -in certificate.pem -password pass:${openssl_pw}
  4. Clean up stray files and the environment variable:

    rm ./certs.json rm ./certificate.pem rm ./private_key.pem rm ./ca_chain.pem unset openssl_pw
  5. Update the certificate in the CrashPlan cloud

Vault best practices

Following are best practices for configuring Vault. For more information, see Vault's Production Hardening guide.

Use a tested and trusted backend

Vault is compatible with different kinds of storage backends. Choose a storage backend that fits your production needs. Consul is a backend developed by HashiCorp that can function as both a backend for storing data and also service discovery in a multi-cluster setup of Vault. DynamoDB is also widely-used because it supports high availability mode while providing persistent data for Vault.

Implement a disaster recovery strategy

In a Vault disaster recovery (DR) strategy, you configure your deployment to replicate data across datacenters. This setup provides performance improvements in addition to disaster recovery. Enable both DR primary replication and secondary replication. Disaster recovery is only available for Vault Enterprise and Vault Enterprise Modules but not Vault Open Source (compare Vault packages).

Run in high availability mode

High availability mode (HA) requires multiple Vault nodes deployed as members of the same cluster. This ensures that when the active node goes offline, the secondary node can continue to respond to API calls. Certain storage backends do not support an HA deployment, so it is important to ensure this is considered when choosing a backend. Some storage backends that support HA include Consul, DynamoDBEtcd, FoundationDB, and Zookeeper.

Do not run in development mode

You can run Vault in development mode with the following command:  vault server -dev

Never use this command to launch Vault in production environments. Launch Vault from a script like init.d or systemd as found in a Linux distribution (distro). You should also create a Vault user and group, and preferably a volume mount that only Vault can read and write to.

Use a network load balancer

Use a network load balancer compatible with TLS to distribute traffic across a cluster of Vault servers. Load balancing will improve system responsiveness and increase availability of Vault data. You can perform network load balancing with Consul using a number of different tools.

Always use TLS

Vault should always be used with Transport Layer Security (TLS) in production. Most high availability production environments would normally have a load balancer or reverse proxy route traffic to the Vault nodes. It is important to ensure that this traffic is not terminated on these devices. There should end-to-end TLS communication from CrashPlan authorities to the load balancer to Vault nodes.

Run as a single tenant

Run Vault as a single tenant on a machine. This reduces the risk of compromising the Vault service and exposing the keys because there is no other process on the machine that can be compromised.

Configure firewall rules

Vault listens on ports 8200 and 8201. In a production deployment you should use a local firewall to control all traffic to Vault. In cloud provider environments, use a security group or firewall rule to achieve the same objective.

Enable auditing 

Enable auditing to provide a record of Vault operations. You can send audit logs to a different destination or other system processes like syslog.

Consider your upgrade procedure

If you use automated deployments for upgrading applications, it is important to consider how you upgrade Vault and your storage backends. Choose a backend that allows for persistent data during automated deployment upgrades. DynamoDB and Google Cloud Spanner both ensure persistency during a rolling upgrade.

Was this article helpful?
0 out of 0 found this helpful

Articles in this section

See more