Monday, December 11, 2017

EJBCA 6.10.1: Performance and Rethinking Certificate Transparency

Here is the official release companion post for the release of EJBCA 6.10.1, where I'm going to go into a bit of detail about the new features to check out in the latest release. First and foremost, let's talk about the two easy ones:


Some of our truly high volume customers (you know who you are) discovered a performance degradation in EJBCA when trying to upgrade to EJBCA 6.9 and later a few months back. Sadly this was nothing we discovered ourselves as we normally don't test at such high volumes, and neither do most of our customers. Testing and diagnostics showed that this degradation has been gradual over the course of a couple of years.

With EJBCA 6.10.1 we have put a ton of effort into profiling and optimizing, and while we're not yet fully back to previous levels we're at least within parity, and we still have some improvements to make. Most of you don't produce the volumes of transactions where you'd notice the difference, but for those of you that would EJBCA 6.10.1 can perform double the throughput in comparison to EJBCA 6.10. 

In order to avoid similar degradations in the future we've also added a performance testing project to our CI environment with weekly monitoring. 

Custom Certificate Extensions for CV Certificates 

Another new feature introduced in EJBCA 6.10.1 (for Enterprise customers only) is the addition of custom certificate extensions for CV certificates as well. Setting them up is done as usual under System Configuration Custom Certificate Extensions

Improvements to Certificate Transparency Logs 

We've actually redefined quite a bit how to set up CT logging in anticipation to the new Certificate Transparency logging requirements which are going to be coming into effect in Chrome in April of 2018. In short, the new requirements can be summarized as the following:
  1. Writing to an n number of CT logs is not sufficient, but at least one of these logs must be one of the Google CT logs. 
  2. The minimum number of logs to be written to should be defined by the validity time of the certificate. 
  3. Performance requirements by CAs require that writing to n logs in the quickest manner possible takes precedence over log order. 
We made a small improvement back in EJBCA 6.10.0 in which we introduced the concept of "mandatory" logs in order to solve the first requirement. 
The Mandatory requirement introduced in EJBCA 6.10
We ourselves felt that it wasn't going to be sufficient in April, so we revisited the concept and redesigned it a bit, coming up with the following. Firstly, we redefined the mandatory/non-mandatory-status as freeform labels instead:
Labels redone in EJBCA 6.10.1
What you see above is the results of the first screenshot upgraded ot 6.10.1 Instead of the Mandatory-checkbox all of the logs have been sorted in under a label named Mandatory. You may notice that one of the logs wasn't marked as mandatory originally, and this is a conscious choice where we've chosen to view all of Google's logs as mandatory as well. Any non-mandatory non-Google logs will be sorted under Unlabeled. Upgraded logs can then be relabeled and new/existing labels can be applied to new logs that are added:
The contract that these labels infer is that at issuance time, at least one log from each label will be written to (satisfying requirement 1), and given that constraint the other logs will be written to in parallell and the first n SCTs received in reply will be used in the certificate, optimizing issuance time and satisfying requirement 3. 

You may also have noticed the table in the second to last screenshot:

The values from this table are from Google's own specification, but are configurable to allow for future changes. The purpose of this table is to be able to set the number n of CT logs to write to at issuance time based on the validity of the certificate being signed. As a reminder, setting up the minimum/maximum amount of logs to write to in previous versions of EJBCA was done in the following manner in the certificate profiles:

In EJBCA 6.10.1 setting up Certificate Transparency in the certificate profiles looks like the following:
Naturally, you'll notice that individual logs have instead been replaced by labels. The min/max setup from previous versions still remains, but we've added the By Validity-option, which instead sources the sought value from the previously show table att issuance time.

So that's it for now for us. We've been working on EJBCA 6.11 in parallell to this release, so you can expect to see it come out quite shortly. Any feedback on our CT implementation would be greatly appreciated, as we still have plenty of time to amend it before April.

Mike Agrenius Kushner
Product Owner EJBCA

Friday, December 1, 2017

The Definitive EJBCA Upgrade Guide

So, we've heard that a lot of you have been having trouble with upgrades, so we've made an effort to both explain and make the process way easier.

Yeah, we know. You can stop laughing now. Really. We've done better this time, we promise.


The official steps for upgrading any EJBCA installation are:

If running EJBCA < 4.0.16 on JDK6 or earlier:
  1. Upgrade to EJBCA 4.0.16
  2. Run ant upgrade from the console
  3. Run ant post-upgrade from the console
  4. Continue below
If running EJBCA >= 4.0.16 but < 5.0.12 on JDK6 or earlier:
  1. Upgrade to EJBCA
  2. Run ant upgrade from the console
  3. Run ant post-upgrade from the console
  4. Upgrade to JDK7 or JDK8 
  5. Upgrade application server to a JEE6 supporting server
  6. Deploy the latest version of EJBCA 
  7. Run ant upgrade from the console
  8. Run post-upgrade from the UI
If running EJBCA >= 5.0.12 but < 6.4.0:
      1. Upgrade to JDK7 or JDK8 (if required)
      2. Upgrade application server to a JEE6 supporting server (if required) 
      3. Upgrade to latest version of EJBCA 
      4. Run ant upgrade from the console
      5. Run post-upgrade from the UI
      If running EJBCA >= 6.4.0:
      1. Upgrade to latest version of EJBCA 
      2. Run post-upgrade from the UI


      A typical upgrade path:

      1. EJBCA 4.0.16 (on JDK6, JBoss 5.1.0.GA) 
      2. EJBCA (on JDK6, JBoss 5.1.0.GA) 
      3. EJBCA (on JDK8, WildFly 10.1) 
      4. EJBCA 6.x


            The background to writing this guide both stems from the understandable confusion in regards to upgrading EJBCA and many of our users experiencing problems when upgrading decade old installations. Thus there are some concepts we'd like to go through and explain:

            The Intermediate Release: EJBCA

            Description: During EJBCA 6.8.0 we refactored the roles and access rules massively, which lead to an upgrade break when upgrading from versions of EJBCA prior to 5.0 (though upgrading via EJBCA 5.0 was still possible). As we realized that solving this issue while preserving 100% uptime requirements (see below) was impossible, as well as due to the technology jump (see the next section) and bugs that we discovered while testing upgrading from ancient installations, we created EJBCA in order to handle all the intermediate steps. As of today EJBCA is published and available in the Community Edition on SourceForge, and in the download area for customers.  

            Technology Jump - JDK6 → JDK7

            When: EJBCA 6.4.0

            Description: All good things must come to an end, as must support for legacy runtime versions. As much as we value not having to put our customers through unnecessary hoops by forcing them to upgrade underlying technology such as the JDK, at some point we have to drop support due for several reasons: being held back by not being able to use modern developments, because other dependent systems like Application Servers drop support as well and because the JDKs themselves come to the end of their service lives and will no longer receive support from the vendor. 

            Technology Jump - JEE5 → JEE6

            When: EJBCA 6.4.0

            In EJBCA 6.4.0 we decided to move on to JDK7, which means that it can no longer be deployed to application servers based on JDK6 such as JBoss versions 4 and 5. The latest version that can still run under JDK6 is EJBCA For an upgrade path this means that you can continue running on your old JBoss 5.1.0.GA server (JEE5) up to, and including, the EJBCA intermediate release. At this stage you must upgrade JDK and the application server, suggestedly to JDK8 and JBoss EAP 7 or WildFly 10.

            100% Uptime during Upgrade

            When: EJBCA 4.0

            Description: While this may be familiar to many of you, EJBCA has ever since version 4.0 supported full uptime during upgrades for clustered installations. What this means is that we pledge that a clustered installation can continue to sign certificates, issue CRLs and answer OCSP queries during the upgrade process with no noticeable downtime for the end user. 

            This is why the upgrade process you may be familiar with is split up into two steps: upgrade and post-upgrade. In short, upgrade performs whatever steps may be required for the first node to be upgraded to be able to function once it comes online again, while post-upgrade performs whatever steps that remain (such as clean up) that can only be performed once all nodes are running the latest code. 

            Automatic Upgrade

            When: EJBCA 6.4.0

            Description: Stunningly, prior to EJBCA 6.4.0 we hadn't actually thought of tracking the database version internally, thus requiring our user to manually enter this value. From EJBCA 6.4.0 and later we do in fact track this, doing away with the need to run the upgrade command entirely. Instead, it'll be automatically run from the first node running the upgraded code. 

            post-upgrade from Console

            When: EJBCA 6.8.0

            Description: In a similar vein, as more and more of our customers run EJBCA on the PrimeKey Appliance and thus don't have access to the command line. As of EJBCA 6.8.0 it's been possible to perform post-upgrades from the UI. When a post-upgrade is required, the System Upgrade option will appear in the menu:
            Choosing it will bring you to a screen used to perform the post-upgrade action:


            With this blog post and our latest round of QA, we hope that we've solved all existing upgrade issues, and that we can make running the latest version of EJBCA as easy and manageable as possible.

            Tomas Gustavsson

            Mike Agrenius Kushner
            Product Owner EJBCA