The Definitive EJBCA Upgrade Guide

So, we've heard that a lot of you have been having trouble with upgrades, so we've made an effort to both explain and make the process way easier.

Yeah, we know. You can stop laughing now. Really. We've done better this time, we promise.

tl;dr:

The official steps for upgrading any EJBCA installation are:

If running EJBCA < 4.0.16 on JDK6 or earlier:

1. Upgrade to EJBCA 4.0.16
2. Run ant upgrade from the console
3. Run ant post-upgrade from the console
4. Continue below

If running EJBCA >= 4.0.16 but < 5.0.12 on JDK6 or earlier:

1. Upgrade to EJBCA 6.3.2.6
2. Run ant upgrade from the console
3. Run ant post-upgrade from the console
4. Upgrade to JDK7 or JDK8 
5. Upgrade application server to a JEE6 supporting server
6. Deploy the latest version of EJBCA 
7. Run ant upgrade from the console
8. Run post-upgrade from the UI

If running EJBCA >= 5.0.12 but < 6.4.0:

1. Upgrade to JDK7 or JDK8 (if required)
2. Upgrade application server to a JEE6 supporting server (if required) 
3. Upgrade to latest version of EJBCA 
4. Run ant upgrade from the console
5. Run post-upgrade from the UI

If running EJBCA >= 6.4.0:

1. Upgrade to latest version of EJBCA 
2. Run post-upgrade from the UI

Example:

A typical upgrade path:

1. EJBCA 4.0.16 (on JDK6, JBoss 5.1.0.GA) 
2. EJBCA 6.3.2.6 (on JDK6, JBoss 5.1.0.GA) 
3. EJBCA 6.3.2.6 (on JDK8, WildFly 10.1) 
4. EJBCA 6.x

Concepts

The background to writing this guide both stems from the understandable confusion in regards to upgrading EJBCA and many of our users experiencing problems when upgrading decade old installations. Thus there are some concepts we'd like to go through and explain:

The Intermediate Release: EJBCA 6.3.2.6

Description: During EJBCA 6.8.0 we refactored the roles and access rules massively, which lead to an upgrade break when upgrading from versions of EJBCA prior to 5.0 (though upgrading via EJBCA 5.0 was still possible). As we realized that solving this issue while preserving 100% uptime requirements (see below) was impossible, as well as due to the technology jump (see the next section) and bugs that we discovered while testing upgrading from ancient installations, we created EJBCA 6.3.2.6 in order to handle all the intermediate steps. As of today EJBCA 6.3.2.6 is published and available in the Community Edition on SourceForge, and in the download area for customers.  

Technology Jump - JDK6 → JDK7

When: EJBCA 6.4.0

Description: All good things must come to an end, as must support for legacy runtime versions. As much as we value not having to put our customers through unnecessary hoops by forcing them to upgrade underlying technology such as the JDK, at some point we have to drop support due for several reasons: being held back by not being able to use modern developments, because other dependent systems like Application Servers drop support as well and because the JDKs themselves come to the end of their service lives and will no longer receive support from the vendor. 

 

Technology Jump - JEE5 → JEE6

When: EJBCA 6.4.0

In EJBCA 6.4.0 we decided to move on to JDK7, which means that it can no longer be deployed to application servers based on JDK6 such as JBoss versions 4 and 5. The latest version that can still run under JDK6 is EJBCA 6.3.2.6. For an upgrade path this means that you can continue running on your old JBoss 5.1.0.GA server (JEE5) up to, and including, the EJBCA 6.3.2.6 intermediate release. At this stage you must upgrade JDK and the application server, suggestedly to JDK8 and JBoss EAP 7 or WildFly 10.

100% Uptime during Upgrade

When: EJBCA 4.0

Description: While this may be familiar to many of you, EJBCA has ever since version 4.0 supported full uptime during upgrades for clustered installations. What this means is that we pledge that a clustered installation can continue to sign certificates, issue CRLs and answer OCSP queries during the upgrade process with no noticeable downtime for the end user. 

This is why the upgrade process you may be familiar with is split up into two steps: upgrade and post-upgrade. In short, upgrade performs whatever steps may be required for the first node to be upgraded to be able to function once it comes online again, while post-upgrade performs whatever steps that remain (such as clean up) that can only be performed once all nodes are running the latest code. 

Automatic Upgrade

When: EJBCA 6.4.0

Description: Stunningly, prior to EJBCA 6.4.0 we hadn't actually thought of tracking the database version internally, thus requiring our user to manually enter this value. From EJBCA 6.4.0 and later we do in fact track this, doing away with the need to run the upgrade command entirely. Instead, it'll be automatically run from the first node running the upgraded code. 

post-upgrade from Console

When: EJBCA 6.8.0

Description: In a similar vein, as more and more of our customers run EJBCA on the PrimeKey Appliance and thus don't have access to the command line. As of EJBCA 6.8.0 it's been possible to perform post-upgrades from the UI. When a post-upgrade is required, the System Upgrade option will appear in the menu:

Choosing it will bring you to a screen used to perform the post-upgrade action:

Conclusion

With this blog post and our latest round of QA, we hope that we've solved all existing upgrade issues, and that we can make running the latest version of EJBCA as easy and manageable as possible.

Cheers!
Tomas Gustavsson
CTO

Mike Agrenius Kushner
Product Owner EJBCA

EJBCA Development - Moving towards Continuous Delivery (finally...)

So a slightly more informal post from me, but I'd like to talk about a few of the changes we've been making in our development process here in the EJBCA team, and how they affect you as our customers.
I officially took on the role as Product Owner of EJBCA a bit less than a year ago without it really existing beforehand. How we got to that point is mostly historical and tied to our roots as an open source project. Tomas, EJBCAs founder and PrimeKey's current CTO was (and still is) EJBCA's face to the world, and with a small and tight development team around him responsibility for features, product cycles and roadmap was mostly ad-hoc, and this is where I came in nigh eight years ago as a developer.
In the time that has passed since then we've grown quite a bit and our user base has grown even more; as we mature from being a scrappy little FOSS project to what will hopefully be seen as a solid and well built software suite that can contend with the best of them.

Changes are coming, some which you all may notice directly and others that hopefully will be felt by us being quicker to adapt, better att keeping our deadlines and delivering better quality on the first try. One of the changes which has been silently in place for a while, but which I feel brave enough to advertise now is that we've moved towards continuous delivery:

A snapshot of our public repository. 

Since a while back the EJBCA team has been running on three week sprints, and with some tinkering we've finally gotten to the point where we can reliably produce a deliverable at the end of each sprint. Pictured above is the first Alpha of EJBCA 6.11.0, which we released at the end of the sprint on Wednesday. On Wednesday in three weeks it'll be joined by the next Alpha, and so forth until the release.
These Alpha releases are available for download for all Enterprise customers, the purpose of which is primarily for you guys to be able to evaluate and give feedback on ongoing development. In the future I'll also to try figure out a good way of showcasing the contents of each Alpha, while also making sure that there is some form of VM available for those of you who don't have a testing environment ready to deploy to.

EJBCA 6.10.0.1 Patch Release

Just a quick note, we just released a patch release of EJBCA 6.10.0. In it we've fixed a couple of corner cases for CAA, as well a library used in the CMP Proxy which we had missed renaming in our configuration files.

Presenting EJBCA 6.10: Customized RA Layouts and CMP Keypair Generation

Happy halloween to all, we the Plucky Khobolds of PKI have been toiling away at another release.

Customized RA Layouts

Speaking of costumes and dressing up, EJBCA 6.10 introduces an extremely neat feature to the RA web: not only the ability to upload custom stylesheets and logos on the CA web to be used in the RA, and not only setting these per role, but having these transmitted to a remote RA over the Peers protocol. This means that the look-and-feel of an RA placed in an entirely different country than the CA can be modified CA-side without even  requiring a restart of the RA, and it can be done for multiple users depending on their role.

Adding a custom style is trivial, just go to System Configuration and click on the Custom RA Styles-tab. From there simply upload an archive containing a modified copy of the RA's stylesheets and/or a custom logo, and then give it a name.

Thereafter you may simply go to the Administrator Roles-screen, where there now is a new column to set a custom stylesheet for each role if one wishes.

ROCA

On the theme of scares and frights, we're sure that nobody missed the ROCA vulnerability that was made public this month, as written about here. While EJBCA has never used Infineon libraries for key generation (and to the best of our knowledge, none of our supported HSM vendors do either), we've still been capable of signing weak keys submitted from other sources. Fortunately since we introduced the RSA Key Validator back in EJBCA 6.9, adding a ROCA check there as well was trivial. For those of you running or planning on running RSA Key validation, we strongly recommend activating checking for ROCA weak keys.

Central Keypair Generation over CMP

On the CMP side we've added the concept of Central Key Generation which allows for a request for a keypair generated CA side to be transmitted and returned over CMP.

Other Fixes

Certificate Transparency has been given the ability to specify, apart from the minimum number of required logs, which logs which are considered mandatory to write to - this in anticipation of new requirements from Chrome coming in 2018. We've also kept working on our CAA validator, hammering out various corner cases and parallelising DNS lookups for certificates containing multiple DNSNames.

From an upgrade perspective we're happy to see many legacy installations (EJBCA 4.0 and older) beginning to upgrade towards more modern versions of EJBCA, and have received some bug reports specific to older deployments which we've fixed in this release. Currently we support upgrading directly from EJBCA 5.0.16 or later. EJBCA 6.10 introduces no database changes, so upgrading from 6.9.x doesn't involve any automatic or manual upgrade steps.

Introducing the EJBCA RA, Part 3: Architecture of an RA Proxy

This blog post will describe features in EJBCA Enterprise which will not be released to the Community Edition. If you're interested in what you read about here and would like to learn more, please mail sales@primekey.com 

Hello there and welcome back to the next feature of our RA series, which is going to discuss a bit about the RA, what led to it and who needs it. I know it's very much belayed, but we've had our hands full with our regular releases =)

The What? 

So for those of you reading who are either running EJBCA in small scale or internally, the new EJBCA RA interface is just a fancy new frontend for EJBCA. We know that many of you are running your own RA services through our SOAP interface, others via CMP or SCEP. 

For those of you running EJBCA Community, that is where it will end once we release the RA into a community release. Those of you running EJBCA Enterprise, or who might have plans on doing so, this is where you perk your ears.

The interface is actually just the tip of the iceberg, because what hides behind it is actually a fully armed and operational battle station RA Proxy.

The Why? 

Many of you already running large scale CA installations will be more than aware of this issue, but I'll say it plainly: exposing your CA server to the internet is a bad thing. You're suddenly vulnerable to every security vulnerability, zero-day and attack vector known to mankind, and a CA is a more than inviting target. 

Yet, most CAs (barring root CAs) need some form of exposure, if not to just publish CRLs and answer OCSP requests. For this reason CAs will be placed behind a firewall allowing only outbound communication, through which CRLs can be published to servers outside the firewall and to VAs placed in the DMZ.

When using EJBCA as a VA, those of you who have been in the game long enough know that we used to support a VA publisher (now considered legacy) that published directly to the VA database (using an extra port opened in the firewall), which has since then been replaced with EJBCA Peers (an Enterprise feature) which allow the CA to communicate to the VA running TLS, with the protocol only making use of outbound communication (with replies) in order to fulfill to security requirements. 

Legacy Features 

RA functionality is on the other hand more difficult, since the communication is by its very nature inbound. Up until EJBCA 6.6.0, PrimeKey offered some solutions for an RA wishing to communicate with a CA sitting behind a one-way firewall: 

• The ExternalRA - known to some of our enterprise customers, the old EJBCA ExternalRA (also known as ExtRA) is an asynchronous proxy designed to run in the DMZ. The ExternalRA doesn't provide more than a very rudimentary RA interface, but instead intercepts SOAP messages and stores them in a message database. By opening an outbound database port in the firewall (much as was done for the legacy VA), the CA can poll the ExternalRA's database and thus handle registration requests. 
• The SCEP and CMP Proxies - while similar to the ExternalRA solution (and sharing some of its codebase) the SCEP and CMP proxies fill a similar role, intercepting inbound communications and storing the payload in the database for future polling. 

The legacy stack, using legacy PrimeKey icons. 

All in all, these solutions are not entirely without warrant. While slow due to their asynchronous nature and extremely limited in terms of communications feature sets, they do not only provide RAs with a means to communicate with a secure CA but also mitigate some DOS attacks by vetting inbound communications before the CA needs to handle them. 

The How?

As I mentioned before, back in EJBCA 6.3.0 we introduced the EJBCA Peers protocol, in essence a method for two EJBCA instances to be able to communicate with each other over TLS, but in practice a secure alternative to direct database publishing for VAs (see this old blogpost). EJBCA finally saw the long awaited expansion of the Peers protocol to also encompass RA communications, and in doing so introduced new demands. 

Our Initial Implementation

While designing the protocol to communicate between CA and VA, one constraint that we introduced was generality: there should in essence be no difference between a CA installation of EJBCA compared to a VA installation. We want our customers to be able to follow a standard installation procedure up until the point where it's time to designate the purpose of the actual instance, and this counts doubly when using a PrimeKey appliance. Between a CA and VA this is relatively simple: an instance is always a CA, and can also act as a VA for an externally imported CA. Communication is always initiated by the CA. 

Introducing RAs into the Mix 

Adding RAs implies another few layers of complexity. Communication goes inherently upstream, breaking the outbound-traffic-only constraint on the CA security environment. Thinking ahead, we realized that proxies can exist in several layers (communications being routed through several instances of EJBCA before reaching their final installation), with any installation acting as CA and RA proxy simultaneously. As developers we direly want to avoid reinventing our own wheel, so we direly want to avoid maintaining one interface for designated RAs and another for RA tasks on the CA. 

We solved this in EJBCA 6.6.0 by adding another layer of abstraction on top of RA calls, the RaMasterApi. The API checks the source of the message, availability of peers upstream to itself and the message type and from that decides on whether to process the message locally or pass it on. 

The beauty if this design is that it's essentially stateless. A new EJBCA instance born to this world can simply be set up to accept incoming peer messages, and that's basically all the configuration required to make it into an RA. Adding an outgoing peer to this instance and configuring it for RA communications would in no way detract from its capacity to act as RA, but it would also relay messages from downstream up to the CA.

Even for the RA, all communications are still initiated by the CA, specifically by the CA leaving long hanging message thread in a pool for the RA to use as needed.

Lastly, the CA is also protected by a rights system, in which any credentials used to identify on the RA are nestled within the credentials set on the CA for the RA itself. In other words, if an RA has been limited to a certain subset of RA operations, and a certain subset of CAs, even logging in with a superadmin certificate on the RA won't allow any more privileges than those allowed to the RA. It means that there are essentially watertight compartments between different RA instances, allowing the CA to serve several different entities with RA services without risk of leaking information between them.

So, in our next and final instalment I'll be showing a quick tutorial on how to set up an RA to speak to a VA and demonstrate some of the concepts I've spoken about in this post. Until then!

Cheers!
Mike Agrenius Kushner
Product Owner EJBCA

Signing weak RSA Keys? Not on our watch!

I'm sure very few of you have missed the rather crippling flaw found in a widely used code library to generate RSA keys which has been named the Return of Coppersmith's Attack, or ROCA for short.

Fortunately none of PrimeKey's products uses these libraries, nor do any of our supported HSM vendors to the best of our knowledge, so no key pairs produced by EJBCA should be affected by this flaw.

Source: https://crocs.fi.muni.cz/_detail/public/papers/roca_impact.png?id=public%3Apapers%3Arsa_ccs17

Nonetheless, EJBCA does run the risk of signing such keys as part of a Certificate Signing Request. As fears of similar flaws have been lifted to us before, in EJBCA 6.9.0 released in late August of this year we introduced the concept of Validators, among them the RSA Key Validator.

For EJBCA 6.10, slated to be released on the 1st of November, we've added functionality to the RSA Key Validator to reject keys affected by the ROCA flaw.

All you need to do after upgrading to EJBCA 6.10.0 or later is to check this box in your validator, and you're set to go!

Cheers!
Mike Agrenius Kushner,
Product Owner EJBCA

Master's Thesis Paper on Post-Quantum Algorithms for Digital Signing in PKI Available Now

Mikael Sjöberg has been with us here at PrimeKey this spring exploring different algorithms suitable for use in PKI even after quantum computers becomes a reality.

As Mikael's mentor I am very pleased with his work and that he choose to do the study at PrimeKey.

The final version of the master's thesis has now been approved.  Read the abstract below or download the full version PDF.

Cheers,
Markus Kilås
Product Owner SignServer

Post-quantum algorithms for digital
signing in Public Key Infrastructures

MIKAEL SJÖBERG

Abstract

One emerging threat to Public Key Infrastructures is the possible development of large-scale quantum computers, which would be able to break the public-key cryptosystems used today. Several possibly post-quantum secure cryptographic algorithms have been proposed but so far they have not been used in many practical settings. The purpose of this thesis was to find post-quantum digital signature algorithms that might be suitable for use in Public Key Infrastructures today.

To answer the research question, an extensive literature study was conducted where relevant algorithms were surveyed. Algorithms with high-grade implementations in different cryptographic libraries were benchmarked for performance. Hash-based XMSS and SPHINCS, multivariate-based Rainbow and lattice-based BLISS-B were benchmarked and the results showed that BLISS-B offered the best performance, on par with RSA and ECDSA. All the algorithms did however have relatively large signature sizes and/or key sizes.

Support for post-quantum digital signature algorithms in Public Key Infrastructure products could easily be achieved since many algorithms are implemented in cryptographic libraries. The algorithms that could be recommended for use today were SPHINCS for high-security applications and possibly BLISS-B for lower security applications requiring higher efficiency. The biggest obstacles to widespread deployment of post-quantum algorithms was deemed to be lack of standardisation and either inefficient operations compared to classical algorithms, uncertain security levels, or both.

Full report (PDF)