• User Manuals /
  • Denodo 8.0 Automated Cloud Mode for AWS Quick Start Guide

Denodo 8.0 Automated Cloud Mode for AWS Quick Start Guide

Last modified on: 06 Sep 2022

Download original document


You can translate the document:

Introduction

The goal of this guide is to provide a quick guide of how to use the Automated Mode of the Solution Manager to deploy the Denodo Platform.

The Solution Manager 8.0 can automate the deployment of the Denodo Platform on Amazon AWS. That is, it automates:

  • Creation and management of clusters: choose the type of EC2 instance, number of EC2 instances in the cluster, etc.

  • Creation of load balancers and Auto Scaling groups.

  • Installation and launch of the Denodo servers.

  • Installation of Denodo updates on clusters with several servers, without downtime.

The main benefit is that you can instantiate and manage your entire AWS deployment without having to create and configure custom AWS elements and without SSH connections to each individual server to configure various settings.

To automate the installation and launch of the Denodo Platform servers, the Solution Manager requires a service account for AWS provided by your organization. That way, the Solution Manager automates the deployment of the Denodo Platform with the benefit that all the infrastructure is still under the control of your organization and no one outside your organization can access it.

In the version 8.0 GA, the Solution Manager automates this on Amazon AWS. The update 8.0u20220126 includes this automated mode for Microsoft Azure.

This guide explains how to:

  1. Create a service account on AWS for the Solution Manager.

  1. Launch an AWS EC2 instance to install the Solution Manager.

  1. Install the Solution Manager

  1. Deploy a cluster of Denodo Platform servers.

When you create a cluster in automated mode, the Solution Manager creates an AWS EC2 instance for each component of the Denodo Platform: one or more instances for Virtual DataPort, one or more instances for Data Catalog and one for Scheduler.

We recommend being familiar with these AWS concepts.

Configuring Your AWS Account to Use the Automated Mode

To be able to use the Automated Cloud Mode with AWS you need to create a service account on your AWS account. The goal is to obtain the Access key ID and a Secret access key of this user account.

Usually, the administrators of Denodo will request these to their AWS administrator. These credentials need the right privileges to be able to create instances on AWS.

Go to the next section if you already have these credentials.

If you cannot get them from the administrator, try these steps. Your account on the AWS console may not have the privileges to complete them.

Check Configuring Automated Mode for more information.

Prerequisite: this AWS account must have at least one Amazon Virtual Private Cloud (VPC) created (see documentation of AWS - Getting started with Amazon VPC).

Creating an IAM Policy

In AWS, an IAM policy defines the permissions of the user accounts associated with this policy.

First, you need to create an IAM policy that will allow the service account associated with the Solution Manager to invoke all the necessary operations of the AWS API.

Follow these steps to do this:

  1. Log into the AWS Management Console.

  1. Go to IAM.

  1. In the left panel, expand Access management and click on Policies.

  1. Click on Create policy.

  1. Download this file to your computer: Denodo_Solution_Manager_8_0_IAM_PolicyPermissions.json 
    This is the IAM policy.

  1. Click the tab JSON, clear the content of the text box and paste the content of the file you just downloaded. Then, click on Review policy. If the AWS console shows any warning message regarding the JSON, ignore it and continue with the policy creation.

  1. Enter the name denodo_80_solution_manager (you can use another name if you prefer) and a description of the new policy. Then, click on Create policy.

To learn more about the privileges of this policy, read Creating an IAM policy. It also explains what privileges are optional and which ones are mandatory.

Solution Manager can launch instances with a specific instance profile role for logging or update purposes. The permissions corresponding to the policy attached to the instance profile role are defined in the following sections:

Creating the Service Account

Once you create the IAM policy, follow these steps to create the service account:

  1. In the left panel, expand Access management and click Users. Then, click on Add user.

  1. Enter User name and select Programmatic access.

  1. Click Next permissions.

  1. Select Attach existing policies directly and in the table, search for the policy denodo_80_solution_manager. Then, select the check box next to the policy.

  1. Click Next: Tags. Then enter tags if you want and click on Next: Review and then, Create user.

At this point, you will obtain the Access key ID and the Secret access key of this service account. You will use this later in this document, during the initial set-up of the Denodo Solution Manager.

Important: keep these values secure like a password.

Creating a Security Group

You need to create a security group to allow connections between your computer and the users’ computers, and the EC2 instance in which you will install the Solution Manager.

This section explains how to do this. See more about security groups in the documentation of AWS - Security groups for your VPC.

  1. On the left panel of the AWS console, expand NETWORK & SECURITY and click Security Groups.

  1. Click Create security group.

  1. Enter a Security group name and a Description.

    Copy the name of the security group, you will need it later.

  1. In the drop-down VPC, select the VPC in which to create the security group. You need to select one.

  1. Write down the CIDR block of the VPC you have selected. In the screenshot below, that would be 111.31.0.0/16.

  1. Below Inbound rules, click Add rule and in the drop-down Type, select SSH. In the drop-down Source select My IP.

Repeat this, for each of the ports of the table below.

        

Ports

Values to Enter in the Column Source

SSH

My IP

Custom TCP - 10090

CIDR of your VPC (e.g. 111.31.0.0/16)

Custom TCP - 10091

CIDR of your VPC (e.g. 111.31.0.0/16)

Custom TCP - 19090

- My IP and

- CIDR of your VPC (e.g. 111.31.0.0/16)

Custom TCP - 19443

- My IP and

- CIDR of your VPC (e.g. 111.31.0.0/16)

Note: by selecting My IP you are allowing inbound connections from your local computer (i.e. your public IPv4 address) to the Solution Manager.

Entering the CIDR block of your Amazon Virtual Private Cloud (VPC) is necessary to allow incoming connections from any instance deployed in your VPC. That way, the Denodo Platform servers will be able to connect to the Solution Manager.

These ports are required for you to have access to the Solution Manager.

Launching an EC2 Instance for the Solution Manager

After creating the service account for the Solution Manager, follow these steps to launch the EC2 instance where you will install the Solution Manager.

To do this, follow these steps:

  1. Log into the AWS Management Console of your organization and then, go to the EC2 Management Console.

  1. Go to the section Instances and click on Launch instance.

  1. In the row of Amazon Linux 2 AMI, select 64-bit (x86) and click on Select.

You can use any of the supported operating systems (see documentation). In this example, we use Amazon Linux 2.

  1. Choose an instance type:

  1. For a production environment: m4.xlarge or m5.xlarge.

  1. For evaluations, consider this:

  1. Pick one with at least 8 gigabytes of memory

  1. You can use a smaller type.

  1. See more about this in the documentation (Hardware Requirements).

  1. Click on Configure Instance Details. In this page, in Network, select the VPC in which you created the security group.

  1. Click on Next: Add Storage. In this page, enter this:

  1. Size: we recommend: 10 gigabytes.

  1. About the rest of the options. their default options are correct for the majority of use cases.

  1. See more about this in the documentation (Hardware Requirements).

  1. Click Next: Add Tags and then, Next: Configure Security Group.

    In this page, select     Select an existing security group and select the security group you have just created.

  1. Click Launch. In this page, you have the option to create a new key pair for this instance or use an existing one.

In the next section, you will use this key pair to connect with SSH to this instance.

Installing the Solution Manager

After launching the instance, wait for the EC2 instance to be created. It takes a couple of minutes.

This section explains how to install the Solution Manager on your AWS instance.

Note: This example assumes that you are going to install the Solution Manager in the path /opt/denodo/denodo-solutionmanager-8.0.

  1. Connect to your EC2 instance (some tips to connect to your Linux instance on AWS).

To obtain the EC2 hostname of this instance, go to Instances and click the instance you just created. On the panel at the bottom of the script, copy the Public DNS (IPv4).

  1. Upload these files:

  1. Installer of the Denodo Solution Manager.

  1. The license file of Denodo. That is, the Solution Manager license (the name of the license file contains “-SOL-”)

  1. Execute this:

unzip denodo-install-solutionmanager-8.0-linux64.zip

sudo mkdir /opt/denodo

sudo chown --recursive ec2-user:ec2-user /opt/denodo/

unset DISPLAY

cd denodo-install-solutionmanager-8.0

chmod +x installer_cli.sh

./installer_cli.sh install

This creates the folder to install the Solution Manager and launches the installer of the Solution Manager.

During the installation, do this:

  • Choose express setup. The default settings are valid for almost all situations.

Note: there is a visualization problem when choosing the “express” mode: the summary displayed during the “express” installation says Virtual DataPort will use the ports 9999, 9998, 9997, 9996 and 9995. In reality, it is going to use 19999, 19998, 19997, 19996 and 19995. The same for the web container, it says it will listen to connections on the port 9090 but actually, it will listen in 19090.

  • Installation path: /opt/denodo/denodo-solutionmanager-8.0

  • License file: enter the path to the license file so the installer automatically stores it in the Solution Manager.

  1. Start the Solution Manager components (run the following):

cd /opt/denodo/denodo-solutionmanager-8.0/bin

./licensemanager_startup.sh

./solutionmanager_startup.sh

./solutionmanagerwebtool_startup.sh

These commands initialize all the necessary services.

Basic Set-Up of the Solution Manager

After installing the Solution Manager, you have to set it up. To do this, follow these steps:

Configure Global AWS Configuration

Follow these steps to provide the required configuration for the automated mode.

  1. Go to http://<EC2 hostname>:19090/solution-manager-web-tool

        

  1. Log into the Solution Manager. The default credentials are:

User name: admin

Password: admin

  1. Click Configuration > Automated mode.

  1. In the tab General > AWS of this page, enter this:

  • Access key ID and Secret access key. These are the credentials of the AWS account you created on the section Creating the Service Account above.

  • Denodo Support Access Keys: enter these credentials (you can obtain them from the Denodo Support Site or request them to the Denodo Support Team).

Important: obtain these access keys. Otherwise, you will have to create your own Denodo Platform AMI (this process is not trivial) to be able to create a cluster of Denodo servers in automated mode.

  • These access keys are necessary for two things:

  1. Obtain the identifier of the Amazon Machine Images (AMI) generated by Denodo. These AMIs include the Denodo Platform pre-installed and are configured to be managed with the Solution Manager.

  1. Download the updates from the Denodo Support site.

  • For now, leave these fields empty: Profile for S3 bucket and S3 base location URL.

  • Profile for S3 bucket Server configuration

  • Logging & Patches configuration

  1. In the section Server Access, leave the boxes empty. The documentation explains what these options are for.

  1. In the section Default Region & AMIs, enter the following:

  • Default region: when you create a cluster of Denodo servers in automated mode, by default the AWS instances will be created in this region.

  • For Virtual DataPort AMI, select Provided by Denodo and select the first option available.

    Do the same for     Data Catalog AMI and Scheduler AMI.

Later, when you create clusters of Denodo servers in automated mode, by default, the Solution Manager will create the AWS instances based on these AMIs (in the configuration of the cluster, you can specify different AMIs).

Even though you select the same AMI, when the Solution Manager launches an instance based on this AMI, it instructs the AMI to start Virtual DataPort or Scheduler or Data Catalog.

Create an Environment in Automated Mode

To create a new environment in automated mode, follow these steps. Note that in Solution Manager version 8.0, you can still register servers as you did in version 7.0.

  1. Create a new environment: click on Environments > New environment. Then, in Environment type, select Automated Cloud Mode (AWS) and click Create environment. In this form, enter this:

  • Provide Name and Description.

  • Select a License scenario.

  • Leave empty the fields Access key ID and Secret access key to use the same values you provided in the global AWS settings. You can enter different credentials if you want to use different credentials to create the EC2 instances of this cluster.

  • For AWS region and VPC, select one of the values available. The default  values are the region and VPC of the AWS instance where you installed the Solution Manager.

  1. Create a new cluster. In the form to create a new cluster, enter this:

  • Tab General

  • Provide Name and Description.

  • KeyPair: this is the key-pair file you will need to connect with SSH to the EC2 instances that will be created.

This list is obtained from your AWS account. To make the administration easier, select the same key pair of this EC2 instance.

  • Subnet: select one of the available options. If you do not know what to select, select Use existing subnet.

  • In the group Advanced Options, leave all the fields empty for now. If S3 logging is enabled, the Denodo servers of this cluster will store its log information on an S3 bucket (you have to create the bucket).

  • Tab VDP Instance

  • Number of instances: 1.

    For evaluation purposes, set this to 1. To use more instances you will have to set-up the feature     Storing the Metadata on an External Database so all the Virtual DataPort servers of this cluster share the same metadata.

  • Select an Instance type. For example, m4.xlarge. For more details about the instance type, see the hardware requirements.

  • Load Balancing & Auto Scaling:

  • Enable Internet Facing Load Balancer.

    Disable this if there is a VPN between the computer of all the users that connect to Virtual DataPort and the AWS instances of your organization. Otherwise, enable it.

  • Leave Launch instances in Auto Scaling Group to NO since this cluster is only meant for evaluations.

  • In the panel EBS Storage, in Volume size, enter 8 or larger. This is the minimum size required to run the AMI of Denodo Platform 8.0. The other options are valid for an evaluation.

  • In Advanced options:

  • In AMI, if you enter your Denodo Support Access Keys on the page Configuration > Automated mode > AWS, you will be able to select Provided by Denodo and select the option Denodo Platform 8.0.

    If you did not enter your     Denodo Support Access Keys, we suggest you do it now. Otherwise, you will have to create your own Denodo Platform AMI.

  • Tab Data Catalog Instances:

  • Number of instances: 1.

    For evaluation purposes, set this to 1. To use more instances  you will have to set-up the feature to store the configuration of Data Catalog on an external database so they all share the same configuration.

  • Regarding the other options of this component, enter the same options as for Virtual DataPort.

  • Tab Scheduler Instances:

  • Number of instances: 1.


For evaluation purposes, set this to 1. To use more instances you will have to set-up the feature to store the configuration of Scheduler on an external database so they all share the same configuration.

  • Regarding the other options of this panel, enter the same options as for Virtual DataPort.

Note: each instance runs one and only one Denodo service. That is, there will be one EC2 instance for Virtual DataPort, one for Data Catalog and one for Scheduler.

  1. Click Save.

The Solution Manager begins creating the EC2 instances based on the AMI(s) you indicated.

  1. Click Environments > Overview. In the table you will see the status of the environment.

It usually takes 5 to 10 minutes to create the EC 2 instances and launch the Denodo servers.

  1. Right-click Solution Manager > Refresh Catalog (on the left side).

If after a few minutes of refreshing, none of the nodes show up here, it means that that service has not started.

After this, click on the Denodo logo on the top left to go back to My Applications. Now you should see the links to access Data Catalog, Design Studio, Diagnostic & Monitoring Tool.

Enable Single Sign-On

Solution Manager 8.0 is capable of integrating with Identity Providers (IdP) that support Kerberos, OAuth, SAML or OpenID. With this, the users will be able to use single sign-on. That is, they will only have to enter the credentials once on the identity provider (IdP) or never in the case of Kerberos. After that, they will be able to log in to Solution Manager and from there, to Design Studio (the new web interface for Virtual DataPort), Data Catalog and Scheduler, without entering their username and password.

The privileges system of the Solution Manager is equivalent to the one of Virtual DataPort. That is, the Solution Manager obtains the roles assigned to users. Then, the actions they can do depend on the privileges assigned to these roles in the Solution Manager.

To enable single sign-on, follow these steps:

  1. In your Identity Provider register a new application for the Solution Manager.

  1. Click Configuration > Authentication.

  1. In this dialog, expand the panel Single Sign-On Configuration and toggle Enabled to Yes.

  1. Select the Authentication method

  1. Enter the connection details. The administrator of the Identity Provider of your organization will be able to provide them. They depend on the authentication method selected.

Find more details about this process in the section Authenticating with Single Sign-On of the Solution Manager Administration Guide.

After doing this, you need to create the necessary roles in the Solution Manager and in the Virtual DataPort you are going to connect to:

  • To create roles in the Solution Manager, click on the menu Configuration > Role management. In this dialog, click New and then, enter the name of the role.

    For example, if in your Identity Provider the users that will connect to the Solution Manager have the role “denodo_administrators”, you need to create this role.

    At this point, you can either grant a global privilege to this new role or grant more fine-grained privileges to this role. To grant fine-grained privileges, click on     Configuration > Permissions. In this dialog, you assign privileges to roles over specific environments. Click the “key” icon next to the environment to assign privileges to roles over that environment.

    See more about this in the section     Authorization of the Solution Manager Administration Guide.

  • Create the same roles in the Virtual DataPort of the cluster (not in the Virtual DataPort of Solution Manager). To do this, connect to Virtual DataPort using the Administration Tool (not the Design Studio) and create the roles that will be imported.

After doing this, log out of the Solution Manager. You will now see a button Single Sign-On. If you click on it, you will automatically log in to the Solution Manager, if you already logged in to your Identity Provider.

Support

To obtain support, post your question on the Q&A page of the Denodo Community Site. The Denodo Team and data virtualization professionals and enthusiasts will assist you.

Please check more information about Denodo for AWS Support and Denodo Maintenance and Support Service Guide for further information.

FAQ

Does Denodo Technologies Access My Denodo Instances When Running on Automated Cloud Mode?

No.

The Solution Manager and the Denodo servers run on the AWS account of your organization. That is why, to work in automated cloud mode you have to provide your own AWS Client ID and client secret.

How does the Solution Manager Create New Denodo servers?

The Solution Manager uses the AWS SDK for Java provided by AWS. Using this SDK, the Solution Manager launches new instances based on the AMI indicated when creating the cluster. To do this, it uses your AWS credentials.

When the Solution Manager launches an EC2 instance, it uses a feature of AWS called Specify Instance User Data at Launch that is meant to pass information to EC2 instances. In the case of the Solution Manager, it passes this information:

  1. The host and port of the Solution Manager and License Manager

  1. What modules of Denodo have to be started

  1. If TLS is enabled on the Solution Manager

In the AWS console, you can see the data that the Solution Manager passed to each instance it created; go to the Images page, right-click on an instance > Instance settings > User data. You will see a JSON document.

The AMI for Denodo is configured to execute the script aws_init_config.py (in /opt/denodo/denodo-platform-8.0/tools/cloud/aws/) when the instance starts. This script reads the “user data” passed to this instance and among other things:

  • Updates some parameters in the configuration file of Virtual DataPort (e.g. the property com.denodo.vdb.vdbinterface.server.VDBManagerImpl.registryURL of VDBConfiguration.properties).

  • Launches the required module: Virtual DataPort or Data Catalog or Scheduler.

See more information about this mechanism in the documentation of AWS (Running Commands on Your Linux Instance at Launch). 

I Want to Use my Security Groups Instead of the Solution Manager Creating Them

When you create a cluster of Denodo servers in automated mode, for each component (Virtual DataPort, Scheduler and Data Catalog), you can choose to either use an existing AWS security group or for the Solution Manager to create the security group.

For evaluation purposes, it is easier to select Create a security group because the Solution Manager will automatically create one security group for each component and each one of these groups will allow inbound traffic only to the ports of that component. However, in a “real world scenario”, you should check with the AWS team of your organization to determine the best option depending on the architecture of the network.

Alternatively, you can edit the security group you created for the Solution Manager and allow inbound connections to the following ports

  • 9999, 9997, 9996, 9995, 9090 and 9443 (the ports of Virtual DataPort)

  • 8000, 7998 (the ports of Scheduler)

  • 9090, 9443 (the ports of Data Catalog)

  • Incoming connections to the ports 10091, 19090 and 19443 (the ports of Solution Manager) from Virtual DataPort, Scheduler and Data Catalog.

The incoming connections must be allowed from the computer of the users of Denodo and the client applications that connect to Denodo.

See more about this in the documentation of AWS - Security groups for your VPC.

I Want the Web Applications of Denodo to use the Default HTTP Ports (80 and 443)

On Linux, an application cannot listen to connections in a port under 1024, unless the application is launched by the user account root.

If you prefer that your users do not have to specify a port in the URL when connecting from their browser to a Denodo service, execute the following to redirect the incoming requests to the ports 80 and 443 to the port 19090 and 19943 (the default ports for incoming HTTP and HTTPs connections of the Solution Manager).

sudo iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 19090

sudo iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 19943

sudo iptables -t nat -I OUTPUT -p tcp -o lo --dport 443 -j REDIRECT --to-ports 19443

Then, in the configuration of the security group of the instance, add the ports HTTP and HTTPS to the list of allowed income connections.

Another option is to configure port forwarding from ports 80 and 443 to ports 19090

and 19943 using the balancer itself, not through routing tables.