How to improve Apache Tomcat startup times

Applies to: Denodo 8.0 , Denodo 7.0
Last modified on: 24 Oct 2019
Tags: Administration Tomcat Web container

Download document

You can translate the document:

Problem description

Sometimes in cloud environments (AWS, Azure) and occasionally also in non-cloud environments, the Apache Tomcat embedded in a Denodo installation is not fast enough starting up when the Denodo Virtual DataPort server is started causing the Denodo web applications not to be deployed.

This is because Tomcat 7+ heavily relies on the SecureRandom class to provide random values for its session ids and in other places. Depending on JRE used, if the entropy source that is used to initialize SecureRandom is short of entropy this can cause delays during startup. This could happen for instance in cloud environments where servers will be virtualized and created from the same templates. The lack of any external peripherals which produce unpredictable inputs (like keyboard, mouse, soundchips, etc) may lead to a low entropy. Using a blocking entropy source will guarantee a sufficient level of entropy but - as the name suggests - will block in case of low entropy until a sufficient level is reached.

The default (blocking) entropy source used by Java under Linux is /dev/random. This can be swapped out for the non-blocking version /dev/urandom by setting the Java system property:

-Djava.security.egd=file:/dev/./urandom

However, if you are running in a highly sensitive environment this might be of concern to you since reading from /dev/urandom will not block waiting for more entropy but if there is not sufficient entropy in the entropy pool, the returned values are theoretically vulnerable to a cryptographic attack.

Problem detection

This issue is detected when Tomcat takes too much time to start, resulting in Denodo failing to find a deployer of the configured webapps.

Virtual DataPort logs (vdp.log):

The vdp.log will show errors similar to the ones below, but note that these logs are logged at the DEBUG and WARN levels, so this will only be visible increasing the log level of your Virtual DataPort server.

...

135274 [Thread-2] DEBUG 2019-10-03T22:33:42.552 com.denodo.tomcat.manager.TomcatManager [] - Failed to obtain deployer (30)  

135277 [Thread-2] DEBUG 2019-10-03T22:33:42.555 com.denodo.tomcat.manager.TomcatManager [] - deploy: Deploying Web application '/denodo-restfulws'  

135289 [Thread-2] WARN  2019-10-03T22:33:42.567 com.denodo.tomcat.manager.TomcatManager [] - TomcatManager.deploy[/denodo-restfulws]  

javax.management.InstanceNotFoundException: DenodoPlatform-7.0:type=Deployer,host=localhost

...

Tomcat log (catalina.log):

81257 [localhost-startStop-1] INFO  2017-04-19T15:56:12.906 org.apache.catalina.util.SessionIdGeneratorBase  - Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [75,603] milliseconds.  

81300 [main] INFO  2017-04-19T15:56:12.949 org.apache.catalina.startup.Catalina  - Server startup in 80996 ms

Solution for Linux systems

The Random Number Generator or RNG is a mechanism in charge of producing pseudo-random numbers that are then used for generating SSH keys, random PIDs for processes, TCP sequence numbers, and UUIDs for example. To get good pseudo-random numbers, you need some entropy (randomness collected by an operating system). It is interesting to note that entropy doesn’t come from a single source in the Linux kernel but from several ones.

The steps described below have been tested with Red Hat Linux Enterprise 7.7 64-bit. For other Linux distributions the commands used might be different but the basic steps needed will be the same.

Install the Random Number Generator Tool

# yum install -y rng-tools

Start the RNGD service

# systemctl start rngd

Make the RNGD service permanent

# systemctl enable rngd

Check the RNGD service status

# systemctl status rngd -l

When running this command in a virtual machine or a server without DNRG/TPM the following output saying that the service is not active will show up:

# systemctl status rngd -l

rngd.service - Hardware RNG Entropy Gatherer Daemon

Loaded: loaded (/usr/lib/systemd/system/rngd.service; enabled)

Active: failed (Result: exit-code) since Wed 2015-09-30 19:51:35 CEST; 12min ago

Process: 588 ExecStart=/sbin/rngd -f (code=exited, status=1/FAILURE)

Main PID: 588 (code=exited, status=1/FAILURE)

CGroup: /system.slice/rngd.service

Sep 30 19:51:35 vm.example.com systemd[1]: Started Hardware RNG Entropy Gatherer Daemon.

Sep 30 19:51:35 vm.example.com rngd[588]: Unable to open file: /dev/tpm0

Sep 30 19:51:35 vm.example.com rngd[588]: can't open any entropy source

Sep 30 19:51:35 vm.example.com rngd[588]: Maybe RNG device modules are not loaded

Sep 30 19:51:35 vm.example.com systemd[1]: rngd.service: main process exited, code=exited, status=1/FAILURE

Sep 30 19:51:35 vm.example.com systemd[1]: Unit rngd.service entered failed state.

This is the expected behavior as the rngd service does not start when no hardware entropy source is available and thus there is simply nothing to mix.

To run the rngd service or if your application is blocked when asking for pseudo random numbers through the /dev/random device, there is a pseudo solution: you can configure the rngd service to get its entropy from the /dev/urandom device and feed the /dev/random device.

WARNING: This is a pseudo solution because you fill in the /dev/random device with data from the /dev/urandom device: the quality of the pseudo random numbers delivered by the /dev/random device is lower. This can be a problem if these numbers produce SSH keys for example.

Configuring the RNGD service without a hardware entropy source (only if needed)

To be able to run the rngd service without hardware entropy source:

Create a copy of the rngd.service unit file:

# cp /usr/lib/systemd/system/rngd.service /etc/systemd/system

Edit the /etc/systemd/system/rngd.service file and replace the line starting with ExecStart with:

ExecStart=/sbin/rngd -f -r /dev/urandom

Reload the Systemd configuration:

# systemctl daemon-reload

Restart the rngd service:

# systemctl restart rngd

Check the new service status:

# systemctl status rngd

rngd.service - Hardware RNG Entropy Gatherer Daemon

   Loaded: loaded (/etc/systemd/system/rngd.service; enabled)

   Active: active (running) since Wed 2015-09-30 19:56:57 CEST; 6s ago

 Main PID: 24972 (rngd)

   CGroup: /system.slice/rngd.service

           └─24972 /sbin/rngd -f -r /dev/urandom

Sep 30 19:56:57 vm.example.com systemd[1]: Starting Hardware RNG Entrop....

Sep 30 19:56:57 vm.example.com systemd[1]: Started Hardware RNG Entropy....

Hint: Some lines were ellipsized, use -l to show in full.

Check the solution is working

The /dev/random pool can be queried with the commands below. When running into this Tomcat startup issue, the values returned will be about a few hundreds. However, after implementing the proposed solution, the output should be about a few thousand, which will allow a timely server start.

# cat /proc/sys/kernel/random/entropy_avail

# cat /proc/sys/kernel/random/poolsize

Important Warning regarding this solution

Seeding /dev/random with data derived from /dev/urandom plays a trick on the system, the entropy_avail reported will increase, but the real entropy is actually decreasing. A software-only random number generator like rngd is not a proper substitute for a good hardware random number generator. Do not use rngd in this fashion unless you understand and accept this difference.

Alternative Solution for Non System Administrators Users

Since the previous solution requires being a system administrator, in case you cannot implement that solution, to get the embedded Apache Tomcat and web applications running it is possible to change the entropy source of your Denodo Platform and embedded Tomcat server from the default (blocking) entropy source (/dev/random) to the non-blocking version (/dev/urandom):

Change entropy source of the Denodo Platform

  1. Open <DENODO_HOME>/conf/vdp/VDBConfiguration.properties
  2. Append to the java.env.DENODO_OPTS_STARTS property the value: -Djava.security.egd=file\:/dev/./urandom
  3. Execute the following script, <DENODO_HOME>/bin/regenerateFiles.sh, to regenerate the launcher scripts.

Change entropy source of the embedded Apache Tomcat

  1. Open the <DENODO_HOME>/resources/apache-tomcat/bin/setenv.sh file.
  2. Add the following line at the end:

CATALINA_OPTS="$CATALINA_OPTS -Djava.security.egd=file:/dev/./urandom"

  1. Execute the following command:

chmod a+x $DENODO_HOME/resources/apache-tomcat/bin/setenv.sh

Restart the servers after making sure they have been previously stopped.

WARNING: if you are running in a highly sensitive environment this might be of concern to you since reading from /dev/urandom will not block waiting for more entropy but if there is not sufficient entropy in the entropy pool, the returned values are theoretically vulnerable to a cryptographic attack

References

Fixing Tomcat startup performance on cloud servers

Using the random number generator

Presentation of Random Number Generator

Questions

Ask a question
You must sign in to ask a question. If you do not have an account, you can register here

Featured content

DENODO TRAINING

Ready for more? Great! We offer a comprehensive set of training courses, taught by our technical instructors in small, private groups for getting a full, in-depth guided training in the usage of the Denodo Platform. Check out our training courses.

Training