The goal of this guide is to provide a quick guide of how to use the Automated Mode of the Solution Manager to deploy the Denodo Platform.
The Solution Manager 8.0 can automate the deployment of the Denodo Platform on Amazon AWS. That is, it automates:
The main benefit is that you can instantiate and manage your entire AWS deployment without having to create and configure custom AWS elements and without SSH connections to each individual server to configure various settings.
To automate the installation and launch of the Denodo Platform servers, the Solution Manager requires a service account for AWS provided by your organization. That way, the Solution Manager automates the deployment of the Denodo Platform with the benefit that all the infrastructure is still under the control of your organization and no one outside your organization can access it.
In the version 8.0 GA, the Solution Manager automates this on Amazon AWS. The update 8.0u20220126 includes this automated mode for Microsoft Azure.
This guide explains how to:
When you create a cluster in automated mode, the Solution Manager creates an AWS EC2 instance for each component of the Denodo Platform: one or more instances for Virtual DataPort, one or more instances for Data Catalog and one for Scheduler.
We recommend being familiar with these AWS concepts.
To be able to use the Automated Cloud Mode with AWS you need to create a service account on your AWS account. The goal is to obtain the Access key ID and a Secret access key of this user account.
Usually, the administrators of Denodo will request these to their AWS administrator. These credentials need the right privileges to be able to create instances on AWS.
Go to the next section if you already have these credentials.
If you cannot get them from the administrator, try these steps. Your account on the AWS console may not have the privileges to complete them.
Check Configuring Automated Mode for more information.
Prerequisite: this AWS account must have at least one Amazon Virtual Private Cloud (VPC) created (see documentation of AWS - Getting started with Amazon VPC).
In AWS, an IAM policy defines the permissions of the user accounts associated with this policy.
First, you need to create an IAM policy that will allow the service account associated with the Solution Manager to invoke all the necessary operations of the AWS API.
Follow these steps to do this:
To learn more about the privileges of this policy, read Creating an IAM policy. It also explains what privileges are optional and which ones are mandatory.
Solution Manager can launch instances with a specific instance profile role for logging or update purposes. The permissions corresponding to the policy attached to the instance profile role are defined in the following sections:
Once you create the IAM policy, follow these steps to create the service account:
At this point, you will obtain the Access key ID and the Secret access key of this service account. You will use this later in this document, during the initial set-up of the Denodo Solution Manager.
Important: keep these values secure like a password.
You need to create a security group to allow connections between your computer and the users’ computers, and the EC2 instance in which you will install the Solution Manager.
This section explains how to do this. See more about security groups in the documentation of AWS - Security groups for your VPC.
Repeat this, for each of the ports of the table below.
Values to Enter in the Column Source
Custom TCP - 10090
CIDR of your VPC (e.g. 220.127.116.11/16)
Custom TCP - 10091
CIDR of your VPC (e.g. 18.104.22.168/16)
Custom TCP - 19090
- My IP and
- CIDR of your VPC (e.g. 22.214.171.124/16)
Custom TCP - 19443
- My IP and
- CIDR of your VPC (e.g. 126.96.36.199/16)
Note: by selecting My IP you are allowing inbound connections from your local computer (i.e. your public IPv4 address) to the Solution Manager.
Entering the CIDR block of your Amazon Virtual Private Cloud (VPC) is necessary to allow incoming connections from any instance deployed in your VPC. That way, the Denodo Platform servers will be able to connect to the Solution Manager.
These ports are required for you to have access to the Solution Manager.
After creating the service account for the Solution Manager, follow these steps to launch the EC2 instance where you will install the Solution Manager.
To do this, follow these steps:
You can use any of the supported operating systems (see documentation). In this example, we use Amazon Linux 2.
In the next section, you will use this key pair to connect with SSH to this instance.
After launching the instance, wait for the EC2 instance to be created. It takes a couple of minutes.
This section explains how to install the Solution Manager on your AWS instance.
Note: This example assumes that you are going to install the Solution Manager in the path /opt/denodo/denodo-solutionmanager-8.0.
To obtain the EC2 hostname of this instance, go to Instances and click the instance you just created. On the panel at the bottom of the script, copy the Public DNS (IPv4).
sudo mkdir /opt/denodo
sudo chown --recursive ec2-user:ec2-user /opt/denodo/
chmod +x installer_cli.sh
This creates the folder to install the Solution Manager and launches the installer of the Solution Manager.
During the installation, do this:
Note: there is a visualization problem when choosing the “express” mode: the summary displayed during the “express” installation says Virtual DataPort will use the ports 9999, 9998, 9997, 9996 and 9995. In reality, it is going to use 19999, 19998, 19997, 19996 and 19995. The same for the web container, it says it will listen to connections on the port 9090 but actually, it will listen in 19090.
These commands initialize all the necessary services.
After installing the Solution Manager, you have to set it up. To do this, follow these steps:
Follow these steps to provide the required configuration for the automated mode.
User name: admin
Important: obtain these access keys. Otherwise, you will have to create your own Denodo Platform AMI (this process is not trivial) to be able to create a cluster of Denodo servers in automated mode.
Later, when you create clusters of Denodo servers in automated mode, by default, the Solution Manager will create the AWS instances based on these AMIs (in the configuration of the cluster, you can specify different AMIs).
Even though you select the same AMI, when the Solution Manager launches an instance based on this AMI, it instructs the AMI to start Virtual DataPort or Scheduler or Data Catalog.
To create a new environment in automated mode, follow these steps. Note that in Solution Manager version 8.0, you can still register servers as you did in version 7.0.
This list is obtained from your AWS account. To make the administration easier, select the same key pair of this EC2 instance.
For evaluation purposes, set this to 1. To use more instances you will have to set-up the feature to store the configuration of Scheduler on an external database so they all share the same configuration.
Note: each instance runs one and only one Denodo service. That is, there will be one EC2 instance for Virtual DataPort, one for Data Catalog and one for Scheduler.
The Solution Manager begins creating the EC2 instances based on the AMI(s) you indicated.
It usually takes 5 to 10 minutes to create the EC 2 instances and launch the Denodo servers.
If after a few minutes of refreshing, none of the nodes show up here, it means that that service has not started.
After this, click on the Denodo logo on the top left to go back to My Applications. Now you should see the links to access Data Catalog, Design Studio, Diagnostic & Monitoring Tool.
Solution Manager 8.0 is capable of integrating with Identity Providers (IdP) that support Kerberos, OAuth, SAML or OpenID. With this, the users will be able to use single sign-on. That is, they will only have to enter the credentials once on the identity provider (IdP) or never in the case of Kerberos. After that, they will be able to log in to Solution Manager and from there, to Design Studio (the new web interface for Virtual DataPort), Data Catalog and Scheduler, without entering their username and password.
The privileges system of the Solution Manager is equivalent to the one of Virtual DataPort. That is, the Solution Manager obtains the roles assigned to users. Then, the actions they can do depend on the privileges assigned to these roles in the Solution Manager.
To enable single sign-on, follow these steps:
Find more details about this process in the section Authenticating with Single Sign-On of the Solution Manager Administration Guide.
After doing this, you need to create the necessary roles in the Solution Manager and in the Virtual DataPort you are going to connect to:
After doing this, log out of the Solution Manager. You will now see a button Single Sign-On. If you click on it, you will automatically log in to the Solution Manager, if you already logged in to your Identity Provider.
The Solution Manager and the Denodo servers run on the AWS account of your organization. That is why, to work in automated cloud mode you have to provide your own AWS Client ID and client secret.
The Solution Manager uses the AWS SDK for Java provided by AWS. Using this SDK, the Solution Manager launches new instances based on the AMI indicated when creating the cluster. To do this, it uses your AWS credentials.
When the Solution Manager launches an EC2 instance, it uses a feature of AWS called Specify Instance User Data at Launch that is meant to pass information to EC2 instances. In the case of the Solution Manager, it passes this information:
In the AWS console, you can see the data that the Solution Manager passed to each instance it created; go to the Images page, right-click on an instance > Instance settings > User data. You will see a JSON document.
The AMI for Denodo is configured to execute the script aws_init_config.py (in /opt/denodo/denodo-platform-8.0/tools/cloud/aws/) when the instance starts. This script reads the “user data” passed to this instance and among other things:
See more information about this mechanism in the documentation of AWS (Running Commands on Your Linux Instance at Launch).
When you create a cluster of Denodo servers in automated mode, for each component (Virtual DataPort, Scheduler and Data Catalog), you can choose to either use an existing AWS security group or for the Solution Manager to create the security group.
For evaluation purposes, it is easier to select Create a security group because the Solution Manager will automatically create one security group for each component and each one of these groups will allow inbound traffic only to the ports of that component. However, in a “real world scenario”, you should check with the AWS team of your organization to determine the best option depending on the architecture of the network.
Alternatively, you can edit the security group you created for the Solution Manager and allow inbound connections to the following ports
The incoming connections must be allowed from the computer of the users of Denodo and the client applications that connect to Denodo.
See more about this in the documentation of AWS - Security groups for your VPC.
On Linux, an application cannot listen to connections in a port under 1024, unless the application is launched by the user account root.
If you prefer that your users do not have to specify a port in the URL when connecting from their browser to a Denodo service, execute the following to redirect the incoming requests to the ports 80 and 443 to the port 19090 and 19943 (the default ports for incoming HTTP and HTTPs connections of the Solution Manager).
sudo iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 19090
sudo iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 19943
sudo iptables -t nat -I OUTPUT -p tcp -o lo --dport 443 -j REDIRECT --to-ports 19443
Then, in the configuration of the security group of the instance, add the ports HTTP and HTTPS to the list of allowed income connections.
Another option is to configure port forwarding from ports 80 and 443 to ports 19090
and 19943 using the balancer itself, not through routing tables.