Denodo Platform Container 7.0 QuickStart Guide
You can translate the document:
Running Denodo Platform in a Docker container
Pre-requisites
- A compatible OS upon which Docker CE/EE version 18.03+ can be installed.
- Denodo Platform Container image in tar format.
- A valid Denodo Platform license, either standalone or managed by an existing Denodo Solution Manager.
Installation of Docker
Follow the instructions corresponding to the Operating System that you are using from the official Docker website. For example, the instructions for Ubuntu are available here. Once Docker is installed, you can verify its version by executing the following command from a terminal:
$ docker version
Client:
Version: 18.03.1-ce
API version: 1.37
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:12:48 2018
OS/Arch: windows/amd64
Experimental: false
Orchestrator: swarm
Server:
Engine:
Version: 18.03.1-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:22:38 2018
OS/Arch: linux/amd64
Experimental: true
Loading the Denodo Platform image
Once you have downloaded the denodo-container-7.0-xxxxxxxx.zip file you need to uncompress it to get the Denodo Platform container image in tar format that you can load in Docker with the docker load command.
For example, you can load the denodo-platform-latest.tar file by executing the following command:
$ docker load -i denodo-platform-latest.tar
Loaded image: denodo-platform:7.0-latest
Running the Denodo Platform image
The Denodo Platform container image can be used either with Standalone Denodo Licenses (like the Denodo Express License or an Evaluation License) or with licenses managed by a Denodo Solution Manager.
Standalone Licenses
If you are using a Standalone Denodo License (like the Denodo Express License or an Evaluation License) you have to download the license file and place it in a location in the host OS. You can then run the image by simply executing the following command:
$ docker run -d -h <hostname> -p 9999:9999 -p 9997:9997 -p 9996:9996 -p 9090:9090 -v <path>:/opt/denodo/conf/denodo.lic --rm --name denodo-vdpserver denodo-platform:7.0-latest ./denodo-container-start.sh --vdpserver
where:
-d
Optional. Run container in background and print container ID.
-h <hostname>
Sets the internal name for the new container. You need to use the hostname value that you will use to connect to the VDP Server running in this container, and that hostname must be resolved and reachable from client applications. You can use localhost to simplify connections from the host only.
-p 9999:9999 -p 9997:9997 -p 9996:9996 -p 9090:9090
Publishes the container ports. The following default ports used by the VDP server are published to the host: 9999, 9997, 9996, 9090. If you want to use a different configuration, read the “Changing Denodo ports” section below.
-v <path>:/opt/denodo/conf/denodo.lic
Specifies the Standalone Denodo License file to use by the Denodo Server, where:
<path> - Path for the license file in the host OS, for example: /home/denodo/denodo.lic
/opt/denodo/conf/denodo.lic - Path for the license file in the container (Do not change this)
--rm
Optional. This flag tells Docker to automatically clean up the container and remove the file system when the container exits.
--name denodo-vdpserver
Optional. Name of the container to reference it from your host system.
Licenses Managed by a Denodo Solution Manager
If you are using a Denodo Solution Manager to manage your licenses, you just need to create a new server in the Solution Manager that will correspond to the VDP Server running in the container.
When registering the VDP Server in the Solution Manager admin tool, it is important that the host field matches the hostname that you will assign to the container and the port field points to 9999 (the default port that the VDP Server uses inside the container).
You need to create a Solution Manager configuration file in a location in the host OS. This configuration file must detail the host and port values for the License Manager. The host needs to be resolved by the container to the IP address where the License Manager is running and the port specifies where the License Manager is listening (by default this port is 10091). The container needs to have connectivity to that IP address and port number.
If your Solution Manager is running in a host with name solution-manager with the default port configuration, the corresponding SolutionManager.properties file is:
com.denodo.license.host=solution-manager
com.denodo.license.port=10091
Once the configuration file has been created you can run VDP server in a container with the following command:
$ docker run -d -h <hostname> -p 9999:9999 -p 9997:9997 -p 9996:9996 -p 9090:9090 -v <path>:/opt/denodo/conf/SolutionManager.properties --rm --name denodo-vdpserver denodo-platform:7.0-latest ./denodo-container-start.sh --vdpserver
where:
-d
Optional. Run container in background and print container ID.
-h <hostname>
Specifies the hostname for the container. This name has to match the one used in the Denodo Solution Manager definition for this VDP Server, and that hostname must be resolved and reachable from client applications.
-p 9999:9999 -p 9997:9997 -p 9996:9996 -p 9090:9090
Publishes the container ports. The following default ports used by the VDP server are published to the host: 9999, 9997, 9996, 9090. If you want to use a different configuration, read the “Changing Denodo ports” section below.
-v <path>:/opt/denodo/conf/SolutionManager.properties
Specifies the Solution Manager configuration file to use by the Denodo Server, where:
<path> - Path for the Solution Manager configuration file in the host OS, for example: /home/denodo/SolutionManager.properties
/opt/denodo/conf/SolutionManager.properties - Path for the SolutionManager.properties file in the container (Do not change this)
--rm
Optional. This flag tells Docker to automatically clean up the container and remove the file system when the container exits.
--name denodo-vdpserver
Optional. Name of the container to reference it from your host system.
Loading the Solution Manager image
Once you have downloaded the solution-manager-container-8.0-xxxxxxxx.zip file you need to uncompress it to get the Denodo Platform container image in tar format that you can load in Docker with the docker load command.
For example, you can load the solution-manager-latest.tar file by executing the following command:
$ docker load -i solution-manager-latest.tar
Loaded image: solution-manager:7.0-latest
Running the Solution Manager image
The Solution Manager container image requires a Solution Manager License deployed locally, so you have to download the license file and place it in a location in the host OS. You can then run the image by simply executing the following command:
$ docker run -d -h <hostname> -p 10090:10090 -p 10091:10091 -p 19090:19090 -v <path>:/opt/denodo/conf/denodo.lic --rm --name solution-manager solution-manager:7.0-latest ./denodo-container-start.sh --vdpserver --lmserver --smserver --smadmin
where:
-d
Optional. Run container in background and print container ID.
-h <hostname>
Sets the internal name for the new container. You need to use the hostname value that you will use to connect to the VDP Server running in this container, and that hostname must be resolved and reachable from client applications. You can use localhost to simplify connections from the host only.
-p 10090:10090 -p 10091:10091 -p 19090:19090
Publishes the container ports. The following default ports used by Solution Manager are published to the host: 10090, 10091, 19090. If you want to use a different configuration, read the “Changing Denodo ports” section below.
-v <path>:/opt/denodo/conf/denodo.lic
Specifies the Standalone Denodo License file to use by the Denodo Server, where:
<path> - Path for the license file in the host OS, for example: /home/denodo/denodo.lic
/opt/denodo/conf/denodo.lic - Path for the license file in the container (Do not change this)
--rm
Optional. This flag tells Docker to automatically clean up the container and remove the file system when the container exits.
--name solution-manager
Optional. Name of the container to reference it from your host system.
Appendix
Stopping containers
It is advisable to stop the containers gracefully by using the docker stop command instead of the docker kill command. This ensures that running services are stopped and acquired resources freed before stopping the container. This is of particular importance when VDP Server uses a license managed by a Denodo Solution Manager since the license will get locked during a grace period unless the VDP Server releases the license during the proper shutdown. The command to stop the container gracefully is:
$ docker stop denodo-vdpserver
Where denodo-vdpserver is the name that you assigned to the container.
Loading metadata
Once the container is running you can import metadata in it in two ways:
- copying a VQL export file by using the docker cp command and execute the import.sh scripts located in the $DENODO_HOME/bin directory with the docker exec command, or
- importing the VQL export file as you would normally do using the VDP Administration Tool’s “File > Import” option.
Example of using the command line for loading metadata:
$ docker cp metadata.vql denodo-vdpserver:/opt/denodo
$ docker exec denodo-vdpserver ./bin/import.sh --singleuser -f metadata.vql -h localhost/admin?admin@admin
Once you have imported the VQL you can save the container state to persist the changes to the container.
Saving the container state
If you would like to persist the metadata changes in the container you can use Docker volumes or you can create new Docker images by committing the changes using the docker commit command.
You can issue the following command to commit:
$ docker commit denodo-vdpserver [repository:tag]
If your container does not have a name then you need first to get the container id by running docker ps and use it instead of the name. If you want to overwrite the original image you can use the same repository:tag which is used to run the container. If you do not provide a repository:tag value, an unnamed image will be created.
Connecting with the VDP Administration Tool
It is possible to connect to a VDP Server running in a container with a VDP Administration Tool running in the host or in a separate computer, but you must ensure that the computer where you want to run the VDP Administration Tool is able to resolve the name that you assigned to the Docker container (either via DNS or hosts file).
Using the following default connection settings:
Login: admin
Password: admin
Server URI: //<hostname>:9999/admin
Where 9999 represents the published (external) port of the VDP Server container.
Starting additional tools
The Denodo container image comes pre-configured to allow you to start the Denodo Data Catalog and/or the Denodo Diagnostic & Monitoring Tool in the same container where you are running the VDP Server.
If you want to start any of these extra tools use the command:
$ docker run -d -h <hostname> -p 9999:9999 -p 9997:9997 -p 9996:9996 -p 9090:9090 -v <path>:/opt/denodo/conf/SolutionManager.properties --rm --name denodo-vdpserver denodo-platform:7.0-latest ./denodo-container-start.sh --vdpserver --datacatalog --dmt
where:
--datacatalog
Optional. Starts the Denodo Data Catalog web application.
--dmt
Optional. Starts the Denodo Diagnostic & Monitoring Tool.
You also can run these two additional tools with a Standalone Denodo License.
These additional applications require a local VDP Server to run, so you can not start them in a separate container without also including a VDP Server.
Changing Denodo ports
The Docker commands documented previously where publishing the same ports in the host than the ports used by Denodo Platform inside the container. This was defined with the docker arguments:
-p 9999:9999 -p 9997:9997 -p 9996:9996 -p 9090:9090
These default ports have the following use in Denodo Platform:
9999 - RMI Registry (JDBC, VDP Admin Tool...)
9997 - RMI Factory (JDBC, VDP Admin Tool...)
9996 - ODBC
9090 - Web Container
These default ports have the following use in Solution Manager:
10090 - Solution Manager
10091 - License Manager
19090 - Web Administration
When running several Denodo Platform containers in the same host these published ports must be different for each container. You can change any of these ports replacing the previous docker argument with the following one:
-e FACTORY_PORT=<factory_port> -p <factory_port>:<factory_port> -p <registry_port>:9999 -p <odbc_port>:9996 -p <web_port>:9090
Where:
<factory_port>
Represents the host port number that will be used as the RMI factory port required by the JDBC driver. This same port will be used in the host and also inside the container.
<registry_port>, <odbc_port>, <web_port>
Represents the host port number that maps to the respective internal ports: 9999, 9996, 9090. The ports used inside the container for these purposes will keep the default configuration.
Changing Denodo settings
You can change any Denodo settings in the container image by following standard Denodo Administration procedures. You can open an interactive shell in a denodo container by executing the command:
$ docker exec -it denodo-vdpserver bash
After completing the changes and exiting the interactive bash session you can save the container state as instructed before.
It’s typical to configure JVM memory options to suit your needs. It is recommended that you read the Denodo Admin and Development Best Practices article section JVM Configuration before changing these values.
Restricting the number of cores
If you want to limit the number of processors used by the Denodo container, you can use the --cpuset-cpus Docker option, which restricts the cores where the docker container is allowed to run. Remember that the range of CPUs set by this parameter cannot surpass the maximum number of CPUs assigned to the Docker daemon.
The following command restricts the execution of VDP Server to the cores from 0 to 3.
$ docker run --cpuset-cpus 0-3 -d -h <hostname> -p 9999:9999 -p 9997:9997 -p 9996:9996 -p 9090:9090 -v <path>:/opt/denodo/conf/SolutionManager.properties --rm --name denodo-vdpserver denodo-platform:7.0-latest ./denodo-container-start.sh --vdpserver
Configuring the network
You can assign the container specific IP addresses either from an existing Docker network or by creating a new Docker network. To create a new network the following command can be used:
$ docker network create --subnet=172.19.0.0/16 denodonet
The container can then be assigned an IP from this new network by using the following command:
$ docker run --net denodonet --ip 172.19.0.3 -d -h <hostname> -p 9999:9999 -p 9997:9997 -p 9996:9996 -p 9090:9090 -v <path>:/opt/denodo/conf/SolutionManager.properties --rm --name denodo-vdpserver denodo-platform:7.0-latest ./denodo-container-start.sh --vdpserver
Running both Denodo Platform and Solution Manager
You can run both the Denodo Platform and the Solution Manager as containers in the same of different computers, but you need to consider that each container needs to be able to connect to the other container, and this means to be able to resolve their hostnames to an IP address that they can reach.
You can achieve this by means of an external DNS resolver and assigning appropriate hostnames to the containers, or using the hosts file editing capabilities of Docker to use custom hostnames.
In order to configure the hosts files you will need to obtain the public IP of the computer or computers where the containers are running first. You can get the local IP address with the Linux command hostname -I or the Windows command ipconfig | findstr IPv4 and we will refer to them as the <denodo-public-ip> and <solman-public-ip> although they will be the same value if both containers are running on the same computer.
You need to start the Solution Manager with the command:
$ docker run -d --name <solman-container-name> -h <solman-hostname> --add-host <denodo-hostname>:<denodo-public-ip> -p 10090:10090 -p 10091:10091 -p 19090:19090 -v <path-to-solution-manager-license>:/opt/denodo/conf/denodo.lic solution-manager:7.0-latest ./denodo-container-start.sh --lmserver --smserver --smadmin
Then you will need to connect to the Solution Manager web administration http://localhost:19090/solution-manager-web-tool/ to register a new Denodo Platform server and ensure that you use the same <denodo-hostname> value for the server that you will use later on when starting the actual server. Please notice that the complete startup of the Solution Manager will take a bit, and that due to the ephemeral nature of containers, once the Solution Manager container is removed all its data will be deleted. Check this KB article to learn more about Data Persistence in Containers
After the server is registered you can start your Denodo Platform instance. In order to configure the Denodo Platform with Licenses Managed by a Denodo Solution Manager you will need a custom SolutionManager.properties file with the content:
com.denodo.license.host=<solman-hostname>
com.denodo.license.port=10091
Then the command to start the Denodo Platform would be:
$ docker run -d --name <denodo-container-name> -h <denodo-hostname> --add-host <solman-hostname>:<solman-public-ip> -p 9999:9999 -p 9997:9997 -p 9996:9996 -p 9090:9090 -v <path-to-solution-manager-properties>:/opt/denodo/conf/SolutionManager.properties denodo-platform:7.0-latest ./denodo-container-start.sh --vqlserver
You can check if the Denodo Platform server is getting the license by looking at the License Usages section of the Solution Manager.