Keep track of certificate issuance using Graylog (and pretty dashboards)

Running an EJBCA based PKI can be a very boring task, usually everything just works. One complaint that we get is that it just works so stably, that the operations staff forgets what to be done when something happens.

Apart from being boring to run, a question that i asked now and then is to get reports or displays of issued certificates, with different flavor sand slices of reporting. As EJBCA is open, you can find all needed information in the database, or in the audit logs. The question is of course how to aggregate information across cluster nodes, and how to process it in a nice way.

One answer to this is to use a central log system, which can process logs from all nodes in a cluster, or even from different segments, like Issuing CAs, OCSP responders, RAs etc.

Using the Open Source Log Collection and Analysis tool Graylog, we can do all this. For those familiar with other similar tools such as Splunk, it works quite similarly, and EJBCA users currently successfully use both Splunk and Graylog in production. Some buzzwords for this are Log Aggregation, Central Log Analysis, SIEM, etc.

So what can it look like?

Example EJBCA Dashbord on Graylog

A Little Background on Logs

An EJBCA PKI system has the following types of logs:

• Security Audit Log: Used for PKI auditors to audit important security PKI events that the system performs.
• System Log: Used to monitor daily operations in the system, debug and track down errors etc.
• Transaction Log: Used for accounting of specific functions, mainly validation (OCSP).

The Security Audit Log constrains greatly what it logs (defined by EJBCA's Security Target), and does not log any other events. Events pertinent to log are ones such as "Certificate issued", "Certificate Profile edited", "Administrator accessed resource", etc. One of the most important aspects to consider is that the Security Audit log does not log things that do not happen. Things that do not happen are for example invalid requests that the system rejects, because the PKI system did not perform any important auditable event.

The System Log, on the other hand, logs all events that are interesting to monitor, such as rejecting invalid requests, reading profiles etc.

Full information of EJBCA logging can be found in the documentation.

Integrating with Graylog

The easiest integration, that I found, was to simply send logs using Syslog from EJBCA to Graylog. Assuming you have EJBCA running, and Graylog running (I used the AWS AMI to get up and running in no-time) it is easy to start sending logs to Graylog.

Configuring Syslog Sending and Receiving

I used Syslog-TCP, which is not enabled by default in Graylog, nor in JBoss/WildFly (where EJBCA runs).

Enabling Syslog TCP Input in Graylog

To enable using syslog TCP input in Graylog, do the following:

1. Go to the Graylog Web Console and select System > Input.
2. Select Syslog TCP in the Select Input list menu and click Launch new input.

Configuring EJBCA Logging

On the EJBCA server, configure JBoss/WildFly to send messages to Graylog with syslog TCP. This is done by adding the following section in the logging subsystem in the JBoss/WildFly standalone.xml:

<custom-handler name="SYSLOGTCP" class="org.jboss.logmanager.handlers.SyslogHandler" module="org.jboss.logmanager">
    <level name="INFO"/>
    <encoding value="ISO-8859-1"/>
    <formatter>
        <pattern-formatter pattern="%-5p [%c] (%t) %s%E%n"/>
    </formatter>
    <properties>
      <property name="appName" value="WildFly"/>
      <property name="facility" value="LOCAL_USE_5"/>
      <property name="serverHostname" value="ec2-52-72-41-146.compute-1.amazonaws.com"/>
      <property name="hostname" value="-"/>
      <property name="port" value="514"/>
      <property name="syslogType" value="RFC5424"/>
      <property name="protocol" value="TCP"/>
      <property name="messageDelimiter" value="-"/>
      <property name="useMessageDelimiter" value="true"/>
    </properties>
</custom-handler>

You also need to configure the root-logger to use the new handler, this will start sending the same logs to syslog. Add the new handler by modifying the root-loggers section in standalone.xml:

<root-logger>
    <level name="INFO"/>
    <handlers>
        <handler name="CONSOLE"/>
        <handler name="FILE"/>
        <handler name="SYSLOGTCP"/>
    </handlers>
</root-logger>

Send some log items by performing an action in the EJBCA Admin UI, for example saving a certificate profile.

Configuring Graylog

Configuring Graylog involves a few tasks:

• Creating Exctractors
• Making some Searches
• Adding searches to a Dashboard

Create Graylog Extractors

Graylog Extractors are used to extract fields, that can be used in queries in Graylog, from the log stream. You can create Extractors for your log input under System > Input in the Graylog Web Console.
You can analyze log items and create the extractors you need. In our example, the following extractors are created:

• RADN: An administrators DN who issued a certificate, for example 'CN=RA Admin,O=PrimeKey,C=SE'
• Extractor type: Split & Index
• Split by: ;
• Target index: 6
• EVENT: The event that happened, for example 'CERT_CREATION'
• Extractor type: Split & Index
• Split by: ;
• Target index: 2
• CERTPROFILE: the certificate profile a certificate was issued for, for example 'certprofile=346136222'
• Extractor type: Split & Index
• Split by: ;
• Target index: 11

You can edit and create extractors of many different types, the above are simple examples.

After the extractors have been created, go to EJBCA and make some actions, to log something you want to visualize. Such as running a stresstest to issue a bunch of certificate, by different RAs.

Add and Visualize Searches on a Dashboard

The following provides examples of search result information that you can add to your dashboard.

Examples of search result information to visualize:

• CERT_CREATION last day
  • Search in the last 1 day
  • EVENT:CERT_CREATION
  • Click Add count to dashboard
• CERT_REVOKED last day
  • Search in the last 1 day
  • EVENT:CERT_REVOKED
  • Click Add count to dashboard
• Certs issued by SuperAdmin all time
  • Search in all messages
  • EVENT:CERT_CREATION AND RADN:CN=SuperAdmin
  • Click Add count to dashboard
• Certs per RA
  • Search in all messages
  • EVENT:CERT_CREATION
  • Select RADN and click to expand, click Quick values, and then Add to dashboard when you see the graph.
• Certs issued per day
  • Search in the last 30 days
  • EVENT:CERT_CREATION
  • Select Day in the Histogram and then click Add to dashboard
• Exceptions last week
• Search in the last 7 days
• Exception
• Select Hour in the Histogram and then click Add to dashboard

You can now go to your dashboard and rearrange the widgets using Unlock/Edit.

Cheers,

Tomas Gustavsson

CTO