Denodo 8.0 Automated Cloud Mode for Azure Quick Start Guide

Last modified on: 29 Mar 2023

Download original document

Introduction

The goal of this guide is to provide a quick guide of how to use the Automated Mode of the Solution Manager to deploy the Denodo Platform.

The Solution Manager 8.0 can automate the deployment of the Denodo Platform on Microsoft Azure. That is, it automates:

• Creation and management of clusters: choose the type and number of virtual machines in the cluster, its disk type, etc.

• Creation of load balancers and Scale Sets.

• Installation and launch of the Denodo servers.

• Installation of Denodo updates on clusters with several servers, without downtime.

The main benefit is that you can instantiate and manage your entire Azure deployment without having to create and configure custom Azure elements and without RDP connections to each individual server to configure various settings.

To automate the installation and launch of the Denodo Platform servers, the Solution Manager requires Azure credentials provided by your organization. That way, the Solution Manager automates the deployment of the Denodo Platform with the benefit that all the infrastructure is still under the control of your organization and no one outside your organization can access it.

The Automated Cloud mode support for Azure is available since 8.0u20220126 so this only applies for this update or a more recent one.

This guide explains how to:

1. Obtain Azure credentials for the Solution Manager.

2. Launch an Azure Virtual Machine to install the Solution Manager.

3. Install the Solution Manager

4. Deploy a cluster of Denodo Platform servers.

When you create a cluster in automated mode for Azure, the Solution Manager creates Azure Virtual Machines for each component of the Denodo Platform: one or more instances for Virtual DataPort, one or more instances for Data Catalog and one or more instances for Scheduler.

We recommend being familiar with these concepts.

Configuring Azure Credentials

To be able to use Denodo in Automated Cloud Mode with Azure you need to obtain the needed credentials. The Azure credentials used in the Solution Manager are made up of the following fields: subscription ID, client ID, tenant ID and client secret. The subscription ID is obtained at the subscription detail form, the rest of the fields are obtained registering an application.

Usually, the administrators of Denodo will request these to their Azure administrator. These credentials need the right privileges to be able to create the desired resources on Azure.

Go to the next section if you already have these credentials.

If you cannot get them from the administrator, try these steps. Your account on the Azure portal may not have the enough privileges to complete them.

Check Configuring Automated Mode for more information.

Prerequisite: this Azure account must have at least one Virtual Network created (check the Azure documentation for further details).

Register an Application

To obtain the Azure credentials, an App registration is needed, where you can obtain the clientID, tenant ID and client secret. To register an application follow these steps:

1. Log into the Azure portal.

2. Go to App registrations and click on New registration.

3. Input the desired name.

4. Choose the supported account types. Usually the Single tenant is the option to choose when the use is internal to your organization.

5. Click on Register.

6. Once on the App Registration detail form, you can obtain the client ID and tenant ID.

7. To generate a client secret, you can click on Add a certificate or secret or click on the option Certificates & secrets. Once there click on New client secret. You have to specify a name and an expiration date.

It is important to have in mind that client secret values cannot be viewed, except for immediately after creation. Be sure to save the secret when created before leaving the page.

At this point, you should have all the fields used as Azure credentials: subscription ID, client ID, tenant ID and client secret.

Creating a Custom Role

In Azure, a Role Definition is a collection of permissions that you can use to grant access to Azure Resources. It lists the actions that can be performed, such as read, write, and delete. A Role Definition is typically just called a role, and it can be high-level, like owner, or specific, like virtual machine reader. Please check the Azure documentation for further details.

Before using the Automated Cloud Mode for Azure, you have to define a Custom role that will allow the App registration associated with the Solution Manager to invoke all the necessary operations of the Azure API. Once the Custom role is created, it has to be assigned to the App registration. Please check the Azure documentation for further details.

Follow these steps to do create the Custom role:

1. Log into the Azure portal.

2. Go to the Subscription detail page.

3. Go to Access control (IAM).

4. Click on Add custom role.

5. Download this file to your computer Denodo_Solution_Manager_8_0_Azure_Permissions.json. It defines the needed permissions.

6. Select to Start from JSON and use the file you just downloaded. The custom role name is denodo_80_solution_manager (you can use another name if you prefer).

7. Select the subscription or resource group to add as an assignable scope for the custom role, it depends on your needs. You can only choose from the scopes that you have access to.

8. Click on Review + create, if there are no warnings or validation errors, click on Create to create the custom role.

To learn more about the needed custom role, read Creating an Azure Custom role. It also explains what permissions are optional and which ones are mandatory.

Once the App registration and the Custom role are created, go to the subscription or resource group selected as assignable scope and assign the role following the next steps:

1. Go to Access Control (IAM).

2. Click on Add role assignment.

3. On Role, select the created custom role.

4. On Members, select the created App registration.

5. Confirm on Review + assign the desired scope and click on Review + create to add the role assignment.

Creating a Network Security Group

You need to create a Network Security Group to allow connections between your computer and the users’ computers, and the Virtual Machines in which you will install the Solution Manager.

This section explains how to do this. Please check the Azure documentation for further details.

1. Log into the Azure portal.

2. Go to Network security groups and click on Create.

3. Enter the Subscription, Resource Group and Region where the Network security group will be created.

4. Enter a name.

5. Go to Review + create, if there are no warnings or validation errors, click on Create to create the Network security group.

6. Once created, go to its detail page and click on Inbound security rules. Create the rules detailed below:

1. Allow inbound connections from your local computer to the Solution Manager:

Source: your public IPv4 address

Destination: VirtualNetwork.

Destination port ranges: 3389,19090,19443

Protocol: Any

Launching a Virtual Machine for the Solution Manager

Follow these steps to launch a Virtual Machine where you will install the Solution Manager.

1. Log into the Azure portal.

2. Click on Create a resource.

3. Select the Compute category and click on Create Virtual Machine.

4. Enter the Subscription, Resource Group and Region where the Virtual Machine will be created.

5. Enter a name.

6. Select a Windows Image supported (see documentation). In this example, we use Windows Server 2022.

7. Choose a Size. In this example, we use Standard_B4ms.

8. Enter an Administrator account.

9. Select None Public inbound ports, we will be using the created network security group.

10. On Networking:

1. Select your Virtual Network.

2. Ensure to use a public ip to be able to connect to the Virtual Machine.

3. On NIC network security group select Advanced and select the created Network Security Group.

11. Go to Review + create, check the mandatory fields, if there are no warnings or validation errors, click on Create to create the Virtual Machine.

Installing the Solution Manager on your Virtual Machine

After launching the virtual machine, wait until it finishes its creation. It takes several minutes.

This section explains how to install the Solution Manager on your virtual machine.

1. Connect to the virtual machine using RDP. You can obtain the rdp file from the created virtual machine detail page on the Azure portal, clicking on Connect > RDP.

2. Upload from your local machine or download from the internet in the virtual machine these files:

1. Installer of the Denodo Solution Manager.

2. Last update for the Solution Manager. The Automated Cloud mode support for Azure is available since 8.0u20220126 so this only applies for this update or a more recent one.

3. The license file of Denodo. That is, the Solution Manager license (the name of the license file contains “-SOL-”)

3. Install the Solution Manager (see documentation).

4. Install the latest update (see documentation).

5. Start the Solution Manager components (see documentation).

6. You also have to create an inbound port rule in the Windows Defender Firewall with Advanced Security for the same ports that were open in the Azure Network Security Group: 10090, 10091, 19090 and 19443. These are the default ports used by the Solution Manager modules. Check the Windows documentation for further details.

Basic Set-Up of the Solution Manager

After installing the Solution Manager, you have to set it up. To do this, follow these steps:

Configure Global Cloud Configuration

Follow these steps to provide the required configuration for the automated mode. You can access the web tool from your local computer because the Network security group has your public ip configured.

1. Go to http://<VM public ip>:19090/solution-manager-web-tool

        

2. Log into the Solution Manager. The default credentials are:

User name: admin

Password: admin

3. Click Configuration > Automated mode.

4. In the tab General of this page, enter this:

1. Azure

• Subscription ID, Client ID, Tenant ID and Secret. These are the Azure Credentials needed that you created on the section Configuring Azure credentials above.

2. Denodo Support

• Access key ID and Secret access key. These are the Denodo Support Access Keys that you can obtain from the Denodo Support Site or request to the Denodo Support Team). Please check the Denodo Support Site Quick Start Guide for further details.

These access keys are necessary for two things:

• Obtain the identifier of the Images generated by Denodo. These images include the Denodo Platform pre-installed and are configured to be managed with the Solution Manager.

• Download updates from the Denodo Support site.

Important: obtain these access keys. Otherwise, you will have to create your own Denodo Platform Image (this process is not trivial) to be able to create a cluster of Denodo servers in automated mode. It is also necessary to allow connectivity from the machine where the Solution Manager is installed to the Denodo Support Site. It is mandatory in order to use the images provided by Denodo and download updates.

5. In the section Server Access, leave the boxes empty. The documentation explains what these options are for.

6. In the section Default Region & Images > Azure, enter the following:

• Default region: when you create a cluster of Denodo servers in automated mode, by default the Azure virtual machines will be created in this region.
• Resource group: when you create a new Automated Cloud Mode (Azure) environment, by default the selected Resource Group will be the one selected here.
• For VDP image, select Provided by Denodo and select the version with the same update as the Solution Manager.

Do the same for Data Catalog image and Scheduler image.

Later, when you create clusters of Denodo servers in automated mode for Azure, by default, the Solution Manager will create the virtual machines based on these images (in the configuration of the cluster, you can specify different values).

Even though you select the same image, when the Solution Manager launches a virtual machine based on this image, it instructs to start Virtual DataPort or Scheduler or Data Catalog.

Create an Environment in Automated Mode

To create a new environment in automated mode, follow these steps. Note that in Solution Manager version 8.0, you can still register servers as you did in version 7.0.

1. Create a new environment: click on Environments > New environment. Then, in Environment type, select Automated Cloud Mode (Azure) and click Create environment. In this form, enter this:

• Provide Name and Description.

• Select a License scenario.

• Leave empty the Azure credentials, fields Subscription id, Client id, Tenant id and Secret, to use the same values you provided in the global Automated mode configuration. You can enter different credentials if you want to use different credentials to create the virtual machines of this cluster.

• For the Azure region and Resource group fields, select one of the values available. By default, the selected values are the ones you provided in the global Automated mode configuration.

• For the Virtual network, select your virtual network, by default the one used in the virtual machine you installed the Solution Manager.

2. Create a new cluster. In the form to create a new cluster, enter this:

• Tab General

• Provide Name and Description.

• Provide an Administrator Username and Password: this is the user and password you will need to connect with RDP to the virtual machines that will be created.

• Subnet: select one of the available options. If you do not know what to select, select Use existing subnet.

• Tab VDP Instance

• Number of instances: 1.
  
  For evaluation purposes, set this to 1. To use more instances you will have to set-up the feature Storing the Metadata on an External Database so all the Virtual DataPort servers of this cluster share the same metadata.

• Select a  Virtual Machine Size. For example, Standard_B4ms. For more details about the instance type, see the hardware requirements.

• Load Balancing & Auto Scaling:

• Enable Internet Facing Load Balancer.
  
  Disable this if there is a VPN between the computer of all the users that connect to Virtual DataPort and the virtual machines of your organization. Otherwise, enable it.

• Leave Launch instances in a Scale Set to NO since this cluster is only meant for evaluations.

• In the panel Disk, use Type Standard SSD and Size, enter 128 GiB(E10) or larger. This is the minimum size required to run the Image of Denodo Platform 8.0.

• In Advanced options:

• In Image, if you enter your Denodo Support Access Keys on the page Configuration > Automated mode, you will be able to select Provided by Denodo.
  
  If you did not enter your Denodo Support Access Keys, we suggest you do it now. Otherwise, you will have to create your own Denodo Platform Image.

Important: in order to use the image provided by Denodo, it is mandatory to accept the terms, otherwise an error will be shown trying to create the cluster. They can be accepted in the Azure Cloud Shell through the next command: PS> az vm image terms accept --publisher denodo --offer denodo-8_0-vm-auto --subscription <your subscription id> --plan denodo-8_0-vdp-auto-win-<update>2200 The plan depends on the image selected. For example: with version 8.0u20220126 the value for the plan parameter will be denodo-8_0-vdp-auto-win-202201262200; with version 8.0u20220815 the value for the plan parameter will be denodo-8_0-vdp-auto-win-202208152200 Since Denodo Solution Manager 8.0 update 20230301, the terms are automatically accepted when creating an automated Azure cluster for the subscription configured in the automated mode. Note that the next permissions are required to do so: Microsoft.MarketplaceOrdering/offertypes/publishers/offers/plans/agreements/read Microsoft.MarketplaceOrdering/offertypes/publishers/offers/plans/agreements/write Check the Denodo documentation.

• Select Create security group. If you select Use existing security group, read this section below I Want to Use my Network Security Groups Instead of the Solution Manager Creating Them.

• Tab Data Catalog Instances:

• Number of instances: 1.
  
  For evaluation purposes, set this to 1. To use more instances  you will have to set-up the feature to store the configuration of Data Catalog on an external database so they all share the same configuration.

• Regarding the other options of this component, enter the same options as for Virtual DataPort.

• Tab Scheduler Instances:

• Number of instances: 1.

For evaluation purposes, set this to 1. To use more instances you will have to set-up the feature to store the configuration of Scheduler on an external database so they all share the same configuration.

• Regarding the other options of this panel, enter the same options as for Virtual DataPort.

Note: each instance runs one and only one Denodo service. That is, there will be one virtual machine for Virtual DataPort, one for Data Catalog and one for Scheduler.

3. Click Save.

The Solution Manager begins creating the instances based on the Image(s) you indicated.

4. Click Environments > Overview. In the table you will see the status of the environment.

It usually takes 5 to 15 minutes to create the instances and launch the Denodo servers.

5. Right-click Solution Manager > Refresh Catalog (on the left side).

If after a few minutes of refreshing, none of the nodes show up here, it means that that service has not started.

After this, click on the Denodo logo on the top left to go back to My Applications. Now you should see the links to access Data Catalog, Design Studio, Diagnostic & Monitoring Tool and Scheduler.

Enable Single Sign-On

Solution Manager 8.0 is capable of integrating with Identity Providers (IdP) that support Kerberos, OAuth, SAML or OpenID. With this, the users will be able to use single sign-on. That is, they will only have to enter the credentials once on the identity provider (IdP) or never in the case of Kerberos. After that, they will be able to log in to Solution Manager and from there, to Design Studio (the new web interface for Virtual DataPort), Data Catalog and Scheduler, without entering their username and password.

The privileges system of the Solution Manager is equivalent to the one of Virtual DataPort. That is, the Solution Manager obtains the roles assigned to users. Then, the actions they can do depend on the privileges assigned to these roles in the Solution Manager.

To enable single sign-on, follow these steps:

1. In your Identity Provider register a new application for the Solution Manager.

2. Click Configuration > Authentication.

3. In this dialog, expand the panel Single Sign-On Configuration and toggle Enabled to Yes.

4. Select the Authentication method

5. Enter the connection details. The administrator of the Identity Provider of your organization will be able to provide them. They depend on the authentication method selected.

Find more details about this process in the section Authenticating with Single Sign-On of the Solution Manager Administration Guide.

After doing this, you need to create the necessary roles in the Solution Manager and in the Virtual DataPort you are going to connect to:

• To create roles in the Solution Manager, click on the menu Configuration > Role management. In this dialog, click New and then, enter the name of the role.
  
  For example, if in your Identity Provider the users that will connect to the Solution Manager have the role “denodo_administrators”, you need to create this role.
  
  At this point, you can either grant a global privilege to this new role or grant more fine-grained privileges to this role. To grant fine-grained privileges, click on Configuration > Permissions. In this dialog, you assign privileges to roles over specific environments. Click the “key” icon next to the environment to assign privileges to roles over that environment.
  
  See more about this in the section Authorization of the Solution Manager Administration Guide.

• Create the same roles in the Virtual DataPort of the cluster (not in the Virtual DataPort of Solution Manager). To do this, connect to Virtual DataPort using the Administration Tool (not the Design Studio) and create the roles that will be imported.

After doing this, log out of the Solution Manager. You will now see a button Single Sign-On. If you click on it, you will automatically log in to the Solution Manager, if you already logged in to your Identity Provider.

Support

To obtain support, post your question on the Q&A page of the Denodo Community Site. The Denodo Team and data virtualization professionals and enthusiasts will assist you.

Please check more information about Denodo for Azure Support and Denodo Maintenance and Support Service Guide for further information.

FAQ

Does Denodo Technologies Access My Denodo Instances When Running on Automated Cloud Mode for Azure?

No.

The Solution Manager and the Denodo servers run on the Azure subscription of your organization. That is why, to work in automated cloud mode you have to provide your own Azure credentials.

How does the Solution Manager Create New Denodo Servers?

The Solution Manager uses the Azure SDK for Java provided by Azure. Using this SDK, the Solution Manager launches new instances based on the image indicated when creating the cluster. To do this, it uses your Azure credentials.

When the Solution Manager launches an instance, it uses a feature of Azure called User Data for Azure Virtual Machine that is meant to pass information to virtual machine instances. In the case of the Solution Manager, it passes this information:

1. The host and port of the Solution Manager and License Manager

2. What modules of Denodo have to be started

3. If TLS is enabled on the Solution Manager

The Image for Denodo is configured to execute the script azure_init_config.py (in <Denodo_platform_installation_path>/tools/cloud/azure/) when the instance starts. This script reads the “user data” passed to this instance and among other things:

• Updates some parameters in the configuration file of Virtual DataPort (e.g. the property com.denodo.vdb.vdbinterface.server.VDBManagerImpl.registryURL of VDBConfiguration.properties).

• Launches the required module: Virtual DataPort or Data Catalog or Scheduler.

I Want to Use my Network Security Groups Instead of the Solution Manager Creating Them

When you create a cluster of Denodo servers in automated mode, for each component (Virtual DataPort, Scheduler and Data Catalog), you can choose to either use an existing Network Security Group or for the Solution Manager to create a new one.

For evaluation purposes, it is easier to select Create a security group because the Solution Manager will automatically create one security group for each component and each one of these groups will allow inbound traffic only to the ports of that component. However, in a “real world scenario”, you should check with the Azure team of your organization to determine the best option depending on the architecture of the network.

Alternatively, you can edit the security group you created for the Solution Manager and allow inbound connections to the following ports

• 9995-9999, 9090 and 9443 (the ports of Virtual DataPort)

• 8000, 7998 (the ports of Scheduler)

• 9090, 9443 (the ports of Data Catalog)

The incoming connections must be allowed from the computer of the users of Denodo and the client applications that connect to Denodo.

Please check the Azure documentation for further details.