Problem description

Sometimes in cloud environments (AWS, Azure) and occasionally also in non-cloud environments, the Apache Tomcat embedded in a Denodo installation is not fast enough starting up when the Denodo Virtual DataPort server is started causing the Denodo web applications not to be deployed.

This is because Tomcat 7+ heavily relies on the SecureRandom class to provide random values for its session ids and in other places. Depending on JRE used, if the entropy source that is used to initialize SecureRandom is short of entropy this can cause delays during startup. This could happen for instance in cloud environments where servers will be virtualized and created from the same templates. The lack of any external peripherals which produce unpredictable inputs (like keyboard, mouse, soundchips, etc) may lead to a low entropy. Using a blocking entropy source will guarantee a sufficient level of entropy but - as the name suggests - will block in case of low entropy until a sufficient level is reached.

The default (blocking) entropy source used by Java under Linux is /dev/random. This can be swapped out for the non-blocking version /dev/urandom by setting the Java system property:

However, if you are running in a highly sensitive environment this might be of concern to you since reading from /dev/urandom will not block waiting for more entropy but if there is not sufficient entropy in the entropy pool, the returned values are theoretically vulnerable to a cryptographic attack.

Problem detection

This issue is detected when Tomcat takes too much time to start, resulting in Denodo failing to find a deployer of the configured webapps.

Virtual DataPort logs (vdp.log):

The vdp.log will show errors similar to the ones below, but note that these logs are logged at the DEBUG and WARN levels, so this will only be visible increasing the log level of your Virtual DataPort server.

Tomcat log (catalina.log):

Solution for Linux systems

The Random Number Generator or RNG is a mechanism in charge of producing pseudo-random numbers that are then used for generating SSH keys, random PIDs for processes, TCP sequence numbers, and UUIDs for example. To get good pseudo-random numbers, you need some entropy (randomness collected by an operating system). It is interesting to note that entropy doesn’t come from a single source in the Linux kernel but from several ones.

The steps described below have been tested with Red Hat Linux Enterprise 7.7 64-bit. For other Linux distributions the commands used might be different but the basic steps needed will be the same.

Install the Random Number Generator Tool

Start the RNGD service

Make the RNGD service permanent

Check the RNGD service status

When running this command in a virtual machine or a server without DNRG/TPM the following output saying that the service is not active will show up:

This is the expected behavior as the rngd service does not start when no hardware entropy source is available and thus there is simply nothing to mix.

To run the rngd service or if your application is blocked when asking for pseudo random numbers through the /dev/random device, there is a pseudo solution: you can configure the rngd service to get its entropy from the /dev/urandom device and feed the /dev/random device.

WARNING: This is a pseudo solution because you fill in the /dev/random device with data from the /dev/urandom device: the quality of the pseudo random numbers delivered by the /dev/random device is lower. This can be a problem if these numbers produce SSH keys for example.

Configuring the RNGD service without a hardware entropy source (only if needed)

To be able to run the rngd service without hardware entropy source:

Create a copy of the rngd.service unit file:

Edit the /etc/systemd/system/rngd.service file and replace the line starting with ExecStart with:

Reload the Systemd configuration:

Restart the rngd service:

Check the new service status:

Check the solution is working

The /dev/random pool can be queried with the commands below. When running into this Tomcat startup issue, the values returned will be about a few hundreds. However, after implementing the proposed solution, the output should be about a few thousand, which will allow a timely server start.

Important Warning regarding this solution

Seeding /dev/random with data derived from /dev/urandom plays a trick on the system, the entropy_avail reported will increase, but the real entropy is actually decreasing. A software-only random number generator like rngd is not a proper substitute for a good hardware random number generator. Do not use rngd in this fashion unless you understand and accept this difference.

Alternative Solution for Non System Administrators Users

Since the previous solution requires being a system administrator, in case you cannot implement that solution, to get the embedded Apache Tomcat and web applications running it is possible to change the entropy source of your Denodo Platform and embedded Tomcat server from the default (blocking) entropy source (/dev/random) to the non-blocking version (/dev/urandom):

Change entropy source of the Denodo Platform

1. Open <DENODO_HOME>/conf/vdp/VDBConfiguration.properties
2. Append to the “java.env.DENODO_OPTS_STARTS” property the value: -Djava.security.egd=file\:/dev/./urandom
3. Execute the following script, <DENODO_HOME>/bin/regenerateFiles.sh, to regenerate the launcher scripts.

Change entropy source of the embedded Apache Tomcat

1. Open the <DENODO_HOME>/resources/apache-tomcat/bin/setenv.sh file.
2. Add the following line at the end:

CATALINA_OPTS="$CATALINA_OPTS -Djava.security.egd=file:/dev/./urandom"

3. Execute the following command:

chmod a+x $DENODO_HOME/resources/apache-tomcat/bin/setenv.sh

Restart the servers after making sure they have been previously stopped.

WARNING: if you are running in a highly sensitive environment this might be of concern to you since reading from /dev/urandom will not block waiting for more entropy but if there is not sufficient entropy in the entropy pool, the returned values are theoretically vulnerable to a cryptographic attack

References

Fixing Tomcat startup performance on cloud servers

Using the random number generator

Presentation of Random Number Generator