AUTOMATED LIFECYCLE MANAGEMENT

Hello!! Welcome to this tutorial about the new Denodo Solution Manager so let's start with a brief introduction about it.

Solution Manager offers automated lifecycle management solutions to customers by providing a new level of control to increasing complex implementations that are now common in modern organizations. It supports multiple environments from development through production, multiple clusters comprising of individual servers within those clusters and tools to simplify DevOps tasks with continuous delivery of agile development lifecycle.

Regardless of the number of installations and individual developer implementations, all Denodo Platform servers will be provisioned a license from a single License manager component of Solution Manager. This makes the administrator's job easier to manage the licenses that are assigned to various servers within the organization from a centralized management console by integrating with organization's version control system.

The recommended Archiecture of a Solution Manager in an organization is as follows:

Recommended architecture for a Denodo deployment in an organization

This tutorial will help you to understand the following concepts of Denodo Solution Manager:

  • Managing Denodo 8.0 licenses.
  • Handling Metadata Promotion between Environments.
  • Monitoring Denodo Platform Servers.

An IT consulting firm, a customer of Denodo, provides solutioning services to one of their esteemed customers with Denodo Platform from its older versions. They maintained several environments, namely, Development, Test and Production to handle their license files and code promotion activities in Denodo Platform.

Due to frequent and complex deployments across environments, they faced multiple challenges during the code promotions between environments.

Now, they have upgraded to Denodo 8.0 with newly introduced features in Solution Manager component that offers integrated monitoring tool to monitor any server in the organization, Automated Deployments, Single point of access, etc. Read the document New Features of the Solution Manager 8.0 to get an overview about the new features.

Based on that scenario, in the next chapter, we will see how to configure Solution Manager for Denodo Platform 8.0 licenses management.

License Management

When it comes to License Management in Solution Manager, it is very important to understand the purpose of the Denodo global license that was newly introduced in Denodo Platform 7.0 to minimize the cumbersome activity of installing new licenses on multiple machines of the organization's Denodo infrastructure.

A Denodo global license file is a multi-license file that defines all the different license scenarios, each of which has its individual features and restrictions. For example, the Production environment with the features of the production environment (number of servers, maximum number of cpus, etc.), other environments like QA/Staging and Development with their own restrictions and so on.

The Denodo License Manager server is a centralized location from where the rest of Denodo Platform 8.0 servers could obtain their license and renew it periodically. Denodo License Manager will have the global license installed and grants the license to Denodo servers as per its scenario assigned on the environment.

Now, let us go through the steps involved in installing a global license.

Denodo License Installation

A Denodo global license can be installed in the following two methods:

  • During the installation of Solution Manager: While installing Solution Manager, you can set up the license by browsing through the license file path available in the local machine.

  • After the installation of Solution Manager: the license can be installed after logging into the Solution Manager Administration Tool by following the below steps:
  1. Login to Solution Manager Administration Tool. The home page appears as below:

  1. From the Menu bar, Navigate to Licenses > Install License tab. A Pop-up window appears so that you can drag and drop the license file or browse from your local machine.

Now that we have installed the global license, environments, clusters and servers could be created in the Solution Manager Administration Tool.

Environments configuration

The first step for defining your Denodo infrastructure is to create your environments (Production, Testing, Development...) from the Menu bar Click on Environments and select the option New Environment.

Creation of Environments

In our Use case scenario, we need to create a license for three different Environments (Development, Test and Production). Now let us follow the below steps to create the Development Environment within the Solution Manager and you can follow the same steps to create other environments too.

  • To create a New Environment and select a License scenario, Click on the Environment and click on New Environment.

  • A new popup window appears where you could choose the Environment type. Solution Manager can work in two modes:

  1. Standard: This is the intended mode to work with on premises cluster, you have to manually add the cluster resources.
  2. Automated: All the resources are managed by the Solution Manager, you only have to set the desired capacities.

For this tutorial, Choose the Standard mode and click on Create Environment

  • A new popup window appears where you could name the Environment and select the appropriate License scenario to which environment you want to assign the license (For example, Development) and save the environment.

  • Save the environment.

The next step will be to create a cluster because each environment could be composed by a group of Denodo servers including Virtual DataPort, ITPilot and Scheduler servers.

Creation of Clusters

A cluster is a group of Denodo servers which belongs to the same environment. All the Virtual DataPort servers in the same cluster share the same metadata. This means that, before promoting changes from another environment, you have to define, at the environment level, the properties required to execute a deployment on the Virtual DataPort servers

Let us follow the below steps to create a new cluster for the development environment just created.

  1. Click on the newly created environment name and then select New Cluster.
  2. Provide the name and description. Leave the Cluster enabled option with the default settings.

  1. Save the Cluster.

Now, you can see the cluster ‘Development' in the navigator !!!

Creation of Servers

In a Cluster, we could have different type of Denodo servers (Virtual DataPort, Scheduler, ITPilot, etc.).

For Example, let us create a Virtual Data Port server within our cluster by following the steps below:

  1. Click on the created Cluster and select New Server

  1. Provide a name and description of the server.
  2. Provide Hostname (or IP address) and Port details
  3. Select the type of server you want to create (Virtual DataPort, Scheduler, ITPilot Browser Pool or ITPilot Verification).
  4. Provide the Authentication details (mainly User and Password).
  5. Save the server!

You will now be able to see the server added under the cluster, as shown in the screenshot below:

At this point you have the Solution Manager configued! Well done! See next chapters to know how to connect your Denodo servers to your Solution Manager.

Connecting to your License Manager from a Denodo Platform 8.0 installation

In the previous sections we have learn how to configure our brand new Solution Manager. At this point, an additional step is required: we need to connect our Denodo Platform 8.0 installations to that Solution Manager.

You could follow the below steps:

  1. Open the Denodo Platform 8.0 Control Center.
  2. Click on Configure option.

  1. Under LICENSE CONFIGURATION, select the option Connect to a License Manager.

  1. Provide Solution Manager Hostname or (IP Address) and Port number (by default, 10091).
  2. And click on the Apply button.
  3. Finally, you have to restart the Denodo Platform 8.0 servers and tools.

We have now our Denodo Platform servers configured to get the License from the Solution Manager!

You can test it, for example, launching again the Denodo Platform 8.0 Control Center. Now, it will connect to the Solution Manager for getting the license used by this environment in order to validate the Denodo servers allowed to start:

In the next section, let us learn how to promote the code (deploying your virtual databases, data sources, views, Scheduler jobs, etc.) from one environment to other using Solution Manager.

OVERVIEW: Promotion between Denodo Environments

In this section of the tutorial we are going to see another useful feature included in Denodo Solution Manager, the promotion of changes between all your Denodo environments.

A promotion migrates all Denodo objects that need to be part of the migration in the server's metadata from one environment to the other. Also, it takes into account all the tasks that you should perform to migrate changes to the target environment. All the changes to be promoted are represented by what we call a revision.

Let's see a diagram of the promotion process:

This tutorial will show you a how to perform the most basic tasks of a promotion:

  • How to Create Revision in Solution Manager (selecting the elements to promote).
  • How to Validate a Revision
  • And finally, How to Deploying a Revision into other Denodo environment.

Next we will learn how to create revisions!

How to Create Revision

A revision is a list of Virtual DataPort catalog elements (data sources, views, etc.) and Scheduler jobs. The Solution Manager Administration Tool is able to connect to the source Denodo environment, so users can select those elements for creating the revision. Let's see how.

The below steps have to be followed to create a revision:

  1. Click on the source environment (let's assume Development in this case, which includes a Virtual DataPort server).
  2. Select New Revision

  1. Provide the details of the new revision in the General section of the revision window (check the documentation for more information):

  1. Change to Revision Elements in order to select the required elements to be promoted. In our example, the tree of elements created in our Development server will be displayed.

  1. Select the element/s you want to include in the revision. If the element has some dependencies, they will be visible in the Dependencies tree panel. You can also select the dependencies of the elements for the promotion that you also want to include in the revision.
  2. Click on the Save button on the top right corner to save the revision.

Done! You have created your first revision!

Finally, find below the list of elements which can be part of a revision in Denodo Solution Manager:

  • Virtual DataPort databases (data sources, views, data services, etc.)
  • Associations
  • Scheduler tasks (only "VDP Cache" and "VDP Indexer" jobs)
  • Users (default users are ignored)
  • Roles (default roles are ignored)

Once the revision is created, the next step is to validate it. Let's see how to do that.

How to Validate a Revision

Once the revision has been created in Solution Manager, before promoting it to another environment, it must be validated!

The validation consists in a full check that all the required properties are defined in the target environment. For example, a revision could depend on a set of properties which includes data source configurations, users passwords, etc.

So, for a successful deployment, the target environment has to include every property used by the promoted VQL. In our example lets suppose we are promoting the revision created from our

Development environment to a

Test environment.

For checking that all the required properties are defined in the target environment, users must click on the validation icon in the revisions section and select the targer environment (

Test in our example):

The first time you validate a revision including new data sources you will get an error because the configuration of those data sources is not present in the target environment. You will be able to create these new properties from that dialog:

After filling the values of the new properties into the target environment, the revision will be automatically validated again. This time it is a successful validation!!

The Revisions table also indicates that the revision has been validated successfully in the target environment (Test in our case):

Up next we will see how to Deploying a Revision.

How to Deploying a Revision

Now let's learn how to deploy the revision validated in the previous section of the tutorial by following the steps below:

  1. In the Revisions table, select the validated revision and deploy it by clicking on the symbol :

  1. Select the target environment (in our case, Test) and click on Deploy button:

  1. That's all! The revision has been deployed! For seeing the historical information of deployments you can go to the Deployments section:

  1. The successful deployments will be marked as and the failed one will be marked as .

  1. Of course, you can check the errors in order to get the reason for the failure by clicking on the status icon of the failed deployment and navigating to Output > Execution Errors:

Congratulations! You have deployed your first revision into a different Denodo environment. Now you have two environments, one for developing and another one for testing a specific version of your virtual model.

Up Next, it's time to see another useful feature of Solution Manager: Monitoring all your environments.

OVERVIEW: Denodo Platform Monitoring

Denodo Solution Manager provides administrators with a web tool called Diagnostics and Monitoring Tool (also known as DMT) to monitor all the Virtual DataPort servers from a single place in real-time in order to detect malfunctions at the earliest.

At the same time, Solution Manager also achieve historical monitoring by enabling Denodo Monitor in specific servers at any moment. Denodo Monitor will generate several historical logs that can be loaded later into the Diagnostics and Monitoring Tool to view the graphical representation of the server performance metrics.

Now, let us go through the two types of monitoring (live and historical), one by one. Below you have direct links to each section:

  • Real-time (live) monitoring.
  • Historical monitoring.

We will see how to start learning about Live monitoring!

Live Monitoring

Using the Diagnostic and Monitoring Tool of Solution Manager, you can configure multiple servers across environments and monitor them in real time (live).

  1. First Step: Launch web application

  • In the Diagnostic & Monitoring Tool login page, you need to provide the valid login credentials of the tool (by default admin / admin) and click on the Login button.
  • After that the home page of the tool will appear:

  1. Configuration: Creating Environments

In the web tool, you could notice that the servers registered in the Solution Manager are already available which is beacuse Diagnostic & Monitoring Tool is now integrated with Solution Manager.

Starting from Denodo Platform 8.0, Denodo Security Token are distributed with Solution Manager by default. If you want to monitor the servers which are integrated in DMT then it is necessary to configure Denodo Security Token in the external Denodo Platform installations. Refer to the Configure for Denodo Platform section of the Denodo 8.0 User Manuals for more information.

For this tutorial, we can create new Environment manually and add the Virtual DataPort Servers.

You can create Environments in order to group several Denodo servers belonging to the same environment. In our example, there's only one server per environment but in a real scenario could be several servers. You can create a new Environment by using two methods:

  • From the Menu Bar > Options > New Environment.
  • From the Navigator, Right-click on the Home button and then Select New Environment.

For creating the environment, you only have to provide the Name of the environment. In our case, Development:

  1. Configuration: Connecting to Denodo servers

You can add the server within the Development environment by following below steps:

  • Right click on the created Environment and click on New Server.
  • Set the Name of the server.
  • ProvideHost name (or IP address) and set the Port number of Virtual DataPort.
  • Provide the User and Password (by default, admin / admin).
  • Set the Database name.
  • And click on the Create button.

On double-clicking the server, you can view some graphs displaying the status of the Virtual DataPort server. This graphs are updated every few seconds:

Check the Monitoring section of the Denodo 8.0 User Manuals for more information about all the graphs you will have available in Diagnostic and Monitoring Tool!

Now it's time to learn how to analyze historical information.

Historical Monitoring

Until now, we have seen how to monitor the Denodo servrs in real time using the Diagnostic and Monitoring Tool. By default, that tool only shows information of the last 30 minutes. What happen if we need to analyze the Denodo servers during a specif time frame some days before because final users reported errors at that time?

There are two main steps involved in achieving the historical monitoring:

  • First, the most important thing is to have enabled Denodo Monitor.
  • Logs generated by Denodo Monitor could be imported into the Diagnostics and Monitoring Tool at any time to view the graphical representation of the server performance metrics.

What is Denodo Monitor?

Denodo Monitor is a command-line tool which runs indefinitely in order to gather information about the servers periodically. That information is saved as tab-separated log files.

Denodo Solution Manager provides the option to start/stop Denodo Monitor, so let us go through:

  1. How to start/stop Denodo Monitor.
  2. How to load Denodo Monitor log files into Diagnostic and Monitoring Tool.
A. How to start/stop Denodo Monitor.
(I) How to start Denodo Monitor

We can start the Denodo Monitor from the Solution Manager web tool by following the below steps:

  1. Click on the environment (in our example, Development).
  2. Click on Start Monitor option from the drop down:

  1. A popup window appears where you have to click the YES button to start monitoring on the particular environment.
(II). How to Check the List of Environments being Monitorized

You can check the list of monitoring environments by selecting the option Environments > Active Monitors from the header menu:

(III). How to stop Denodo Monitor

For stopping Denodo monitor, you only have to select the check-box of the respective environment and click on the Stop Monitors button.

Up next, we will learn how to load the Denodo Monitor log files into Diagnostic and Monitoring Tool.

B. Load Denodo Monitor log files into Diagnostic & Monitoring Tool

Previously in this tutorial you have learn how to start Denodo Monitor for storing the information of your environments into several log files. For analyzing the logs, for example, for diagnosing something which happened in the past, you could check the log files manually (reading the files) but, the Diagnostic and Monitoring Tool includes a very useful feature: it allows users to upload these Denodo Monitor log files to have a graphical presentation of them.

Let's see how:

  1. Right click on the created server in Diagnostic & Monitoring Tool and select Load Diagnostic Info ... option:

  1. A pop up appears where you need to set the Diagnostic name and the log file directory:

  1. Once the Create button is pressed, all logs within the configured directory will be loaded into the tool in a graphical manner:

With the help of this tutorial, we have understood the importance of Solution Manager and its advantages with respect to its following functionalities:

  • How to manage licenses, that are assigned to various servers within the organization from a centralized management console by integrating with organization's version control system.
  • How to manage promotion of Denodo elements between environments.
  • How to achieve Live and Historical Monitoring using Diagnostics and Monitoring Tool and Denodo Monitor.

Congratulations! You have finished the Denodo Solution Manager tutorial. Take a look at the official reference manuals of Denodo Platform 8.0 and play with your own use cases.