Over the last few cycles the keystone community has been actively preparing to switch the default token format. As of the Newton release, the default is keystone’s original UUID token provider and ever since Kilo we introduced Fernet in hopes of replacing it. The Fernet token format has several advantages over other formats. The most significant of them is that they are completely non-persistent in nature. A presentation on the approach, benefits, and management of Fernet was done during the Newton Summit in Austin, Texas.
If the Fernet format is already available, why does it need to be the default? The keystone development community strives to provide a production-ready configuration out-of-the-box. Of the token providers available in keystone, Fernet is the most production friendly. The Ocata release will be the first release where Fernet is keystone’s default token format.
What you need to know before you upgrade to Ocata
Do you know if you specify a token format in your configuration or do you rely on the default from keystone? If you explicitly set a format in your keystone configuration, even if it is just the default, you’re one step ahead of the game. This is a transparent change and it buys you the peace of mind of knowing a changing default won’t unexpectedly affect your deployment. It also lets you make the switch to Fernet when you’re ready to.
Making the switch
Whether you’re making the switch to Fernet during your Ocata upgrade, or at a later date, a couple steps are required before running any code. The Fernet token provider builds a token by encrypting information about the token using a secure set of keys (all the of this is thoroughly explained in the presentation linked above). Setting up the encryption keys is a mandatory prerequisite to starting any keystone services configured to issue Fernet tokens. Failure to set up a key repository will leave keystone crippled and unable to issue or validate tokens. The setup consists of at least two steps, possibly three if you’re deployment contains multiply keystone nodes.
Create a key repository
The first step requires you to create a key repository and bootstrap it with encryption keys. This sounds scarier than it really is. We’ve added a utility into keystone’s management CLI to assist you with this process called `keystone-manage fernet_setup`. The command will populate a key repository with the minimum number of keys required to start issuing and validating tokens. For clarity, the key repository is simply a directory on disk that is readable and writeable by the user running the keystone process. By default, the key repository is located in `/etc/keystone/fernet-keys/`.
Distribute the key repository
If your deployment has multiple keystone nodes that are expected to validate tokens issued by one another, then you have another step before upgrading. As previously stated, the Fernet format builds tokens by encrypting information. Different encryption keys will yield different tokens, even if the information being encrypted is the same. Therefore, if two different keystone nodes have different key sets, they will not be able to validate tokens issued by each other because they can’t decrypt something without the key that was used to encrypt it. To ensure each node has the same set of keys, the key repository created by the `keystone-manage fernet_setup` command must be distributed to all keystone nodes in the cluster. This can be done in a variety of ways and keystone doesn’t force or implement a specific distribution mechanism. It can be done using `cron` and `rsync`. It can be orchestrated through ansible. It can even be manual. We suggest using whatever makes sense for your deployments configuration management system. Regardless of how the keys are rotated, all keys must have the same contents and name on each node and live within that node’s key repository location.
Update the configuration
Now that all the necessary prerequisite work is done, it’s time to set Fernet as your new token format. You can do this by setting `keystone.conf [token] provider = fernet`. As stated above, it is good to be explicit when defining your token format. So, even though Fernet will be the default in Ocata, it is good practice to set it anyway.
After the upgrade
Once you’ve configured your deployment to issue Fernet tokens, all tokens issued by the previous token provider will be invalid. This will require users to re-authenticate after the upgrade.
After you’ve taken the jump into the world of Fernet, you’ll have to be mindful of rotating your encryption keys. Remember a Fernet token is essentially the ciphertext of a bunch of information about that token. If only one key is used to create every token, with enough time, resources, and tokens, it is possible to reverse engineer a key. By periodically rotating new keys into the repository, we can minimize the risk of a key being compromised. This can be done using another command built into keystone’s management CLI called `keystone-manage fernet_rotate`. It’s imperative to distribute the new key repository to each node in the cluster after every rotation. This makes it so every keystone node has the ability to validate tokens issued from other nodes in the cluster. Depending on your deployments exposure and security requirements, it is recommended that you build a rotation plan to fit your deployments security needs. Examples of key rotation and distribution in a clustered deployment can be found in the previously mentioned presentation.
What if I don’t want to switch token formats?
That’s totally fine. Just be sure to explicitly set your token provider before upgrading (i.e. `keystone.conf [token] provider = uuid`). The same is true for deployments using a custom token provider.
To summarize, we can take away two points. The first is to make sure we’re explicit about specifying the token format, even it if is just the default. The second is to make sure we setup and distribute the key repository before issuing Fernet tokens and after every key rotation. Keeping both of those in mind before and during your upgrade should make it smooth and uneventful.
Feel free to come find us in #openstack-keystone if you have any unanswered questions about Fernet. Good luck!