You can translate the document:

Lookout

Expert trails guide Denodo users through all the relevant materials related to a specific topic, including official docs, KB articles, training, Professional Services offering, and more. The main goal is to give users a single place with references to all the information that they need to become a Denodo expert on any specific topic.

Installing the Denodo Platform and its modules is the first step before starting Data Virtualization. This expert trail provides all the necessary steps to install and maintain the Denodo Platform and its modules including managing updates, deployments, and backup and recovery processes.

The Hike

Stage 1: Installing the Solution Manager

The Denodo setup has two parts:

  • Solution Manager: Denodo Solution Manager is a tool which can be used to manage Denodo Platform Licenses, manage large Denodo Platform deployments and take into account requirements of high availability and load balancing.
  • Denodo Platform: The Denodo Platform is a global solution for the real-time integration of heterogeneous and dispersed, structured and unstructured data sources.

Before doing the installation of the Solution Manager and Denodo Platform, it is required to define the servers that are going to be included in the organization’s infrastructure. Information about the process of this is listed in the Expert Trail: Deployment Topology.

The first step before installing the Solution Manager is to check out the supported operating systems which includes Hardware and Software requirements.


After reviewing the installation requirements, the software can be downloaded from Denodo Support Site. More detailed information is explained in the Knowledge Base article: Denodo Solution Manager: First Steps.

  • Installing Denodo platform and Solution Manager can be done via two ways: Graphical installation and Headless installation.
  • After installing the software, it will be necessary to check if the relevant ports needed by the servers to run are open through the firewall for communication. For example, Licence Manager by default runs on port 10091 and is needed by Denodo platform servers (Virtual DataPort server, Scheduler server etc.) to obtain the license. The document Default ports used by Solution Manager modules provides more information. 
  • Once the software has been installed and the relevant ports have been opened, it is necessary to install the license file. The section Install the license of Solution Manager Installation Guide explains the steps to install the global license file.
  • For complete information on Solution Manager installation components, please read the Knowledge base article Solution Manager Installation Checklist.

Stage 2: Installing the Denodo Platform

After installing the Solution Manager which acts as a single point of contact for licenses and managing the Denodo servers, it is time to install the Denodo Platform. After reviewing the Hardware and Software requirements the software can be downloaded from the Denodo Support site for the intended operating system.

  • After installing the Denodo platform, it is necessary to make sure that the ports needed by the Denodo platform servers are available and open through the firewall. The document Ports needed by the VDP server explains more about the same.
  • To start the Denodo servers, they need to request the license from the license manager server (which is a part of Solution Manager installation). For this, it is required to:

Stage 3: Installing the Latest Update

Once the Solution Manager and Denodo platform is installed, it is important to install the latest update release of both the installations to take advantage of latest features in the product and also to get rid of the latest bug fixes and security vulnerabilities. Updates can be downloaded from the Denodo Support Site navigating to Downloads -> Updates section. Updates can be installed in two ways:

If you are using AWS or Azure Automated mode, you can use the Solution Manager to install the latest updates. For more information, please follow the section Install an update in a cluster section of Solution Manager Guide.

Note: All Denodo Platform and Solution Manager servers should be stopped before installing an update. In other words, all Denodo related Java processes should be stopped completely before you install an update on the Denodo Platform.  

Stage 4: Deployment Process

Within the Solution Manager, a promotion takes under consideration all the tasks that you simply ought to perform to migrate changes within the servers’ metadata from one environment to another. These changes are spoken to by a modification, which could be a collection of Virtual DataPort components and Scheduler jobs to migrate. The Solution Manager Admin Tool helps you make Promotions. You can select the elements you need to promote by creating revisions from particular Virtual DataPort and Scheduler servers and also Data Catalog servers. For a detailed explanation, please follow the Expert Trail: Development Lifecycle.

Based on the environment, deployments can be of two types:

Stage 5: Upgrade Process

The process of upgrading to a new major version of the Denodo Platform involves installing the new version in a separate computer - or in a different directory of the same computer - and transferring the metadata and settings from the previous version to the new installation. The same applies to the Solution Manager.

Although the Denodo Platform is backward compatible, switching back between the major versions is not supported. So, once the Denodo Platform metadata is transferred to the new version, rolling back the metadata to the previous one is not allowed.

The upgrade process might slightly differ based on the version you are upgrading from. For a detailed explanation and steps involved in upgrading the Denodo platform, please refer to the Denodo Platform Upgrade Guide.

Stage 6: Backup and Recovery Policies

Backup of the metadata of Virtual DataPort and other servers is important if any adverse situation arises when you need to restore the servers to its previous state, which even includes restoring the servers to a previous update release. Periodic backups of Denodo platform installation directories can be helpful and easy to restore the servers to a previous state, if needed. Apart from that, there are other ways to export the metadata from Virtual DataPort, Scheduler and Data Catalog server.

In a Denodo production installation, it is recommended to have a backup server that will be connected to the Load Balancer when some of the main servers are not working for an unexpected reason. For those scenarios, if it’s allowed by your license, you can configure Hot Backups or Cold Backup servers in order to maintain its metadata in sync with the Production servers so they can quickly replace the Production ones if they fail.

Similar to the Virtual DataPort module, other modules of the Denodo Platform should also be backed up.

  • Scheduler Server: Scheduler supports import server configuration, projects, plugins and JDBC adapters from a file containing the metadata of a server in a zip file. For more information, please follow the section Import/Export backups section of Scheduler Admin Guide.
  • Data Catalog: You can also export and import the Data Catalog’s configuration and metadata (categories, tags, saved queries, etc.) as a collection of JSON files packed in a zip file. For more information, please follow the section Import and export Data Catalog metadata of Data Catalog Guide.
  • Solution Manager: Similar to Scheduler server and Data Catalog, solution manager server also provides an option to backup the metadata. For more information, please refer to the Import and Export section of Solution Manager Admin Guide.

Stage 7: Monitoring the Denodo Servers

The Expert Trail: Monitoring provides  information on monitoring.

Stage 8: Resource Management

In an environment with multiple concurrent user sessions that run queries, not all user sessions have the same importance. You may want to give more priority to queries executed by transactional applications, which need an immediate response, than to queries executed to generate a daily report.

The Resource Manager allows you to classify sessions into groups based on the attributes of the session, and to allocate resources to those groups in a way that optimizes resource utilization for your application environment.

The Resource Manager introduces two main concepts:

  • Plan: It is a set of restrictions assigned to a user session. For example: limit the number of concurrent queries the same user can run, limit the maximum number of rows of the queries’ results, etc.
  • Rule: It assigns a “plan” to the session of a user if a certain condition is met. This condition can use any of the attributes of the session. I.e. user name, access interface used by the application (JDBC, ODBC, etc.), IP address, etc. Please follow the section Available fields to evaluate a rule of VDP Admin Guide.

Exploration

Fill up your backpack with additional gear:

Installing the Solution Manager

Official Documentation

KB Articles

Additional Resources

Installing the Denodo Platform

Official Documentation

KB Articles

Update Installation

Official Documentation

KB Articles

Additional Resources

Deployment Process

Official Documentation

Additional Resources

Upgrade Process

Official Documentation

Backup and Recovery Policies

KB Articles

Resource Management

Official Documentation

Additional Resources

Guided Routes

Denodo Training Courses

Denodo training courses provide expert data virtualization training for data professionals, including administrators, architects, and developers.

If you are interested in “Operations Expert Trail” you should enroll the following course/s:

  • Denodo Platform Installation: This course provides in-depth knowledge of all activities required to install Denodo Platform 8.0. Highlights of these activities are: Installing and configuring the Denodo Platform and Updating best practices. 
  • Denodo Operations Management: This course covers the usual operations performed by a Denodo Administrator like configuration, creation of virtual databases, user management and backups.

Technical Advisory Sessions

Denodo Customers with active subscriptions have access to request Meet a Technical Advisory sessions.

These are the sessions available related to Operations.

Hardware & Software Requirements

Denodo Platform & Solution Manager Architecture Overview

Overview of the Denodo Platform components architecture. Describe all the different Denodo Platform components (VDP Server, Scheduler, Solution Manager, Data Catalog, etc.):

- Mission.

- How they interact with each other.

- How to interact with client applications and sources.

- How they fit in your organization.

Hardware & Software Requirements for Denodo Platform Components

Describe the Hardware and Software Requirements for the Denodo Platform: both the main installation and the Solution Manager installation. Assisting in:

- Selecting the appropriate hardware resources and software for your organization (physical, VM, container).

- Selecting cache, metadata storage, and VCS system.

Backup & Disaster Recovery

Backup & Recovery Policy

- Assist in defining a Disaster Recovery (DR) strategy.

- Define Failover Detection policy.

- Backup policies (metadata, configuration files, drivers, etc.).

- Configure Cold Backup server.

Install Update/Hotfix Procedure

Assist to define a policy and procedure to apply an update/hotfix to ensure correct rollout and rollback.

Services Management

Assist in defining Denodo Platform services management (startup/shutdown/restart):

- Configure as Windows Service.

- Configure as Linux Service.

- Create Startup/Shutdown scripts.

Professional Services

Denodo Professional Services can help you at the start or any part of your query performance trail. You can find information about the Denodo Professional Services offering in:

Professional Services for Data Virtualization | Denodo

  • Operations Quick Start

If you are a Denodo customer, you can reach out to your Customer Success Manager for details about any Guided Route that you need.

Big Hike Prep Check

Let’s see if you are ready to start your big trail. Take this 4-question questionnaire to check your readiness for an enjoyable hike.

Read the questions below, think about the solution and check if you got them right by looking at the solution. Have you become an expert?

  1. While upgrading the Denodo platform, can you use the same cache database ?

Click here to check if you got it right

It is possible to use the same catalog/schema for the cache database that you are already using because Cache schema/Catalog is compatible in Denodo 7.0 and 8.0; however, it is not possible to use the same cache schema simultaneously in both the versions. You have to shut down either of them while using the other if you are using the same schema for both the versions. If you are upgrading from Denodo 6.0 to Denodo 8.0, you cannot use the existing schema and have to create a new one for Denodo 8.0.

For more information, please follow the section selecting a catalog/schema of the cache of Denodo Upgrade Guide.

  1. You have multiple nodes in a cluster of standard environments. You want to minimize the downtime while deployment. How can you do that?

Click here to check if you got it right

Yes, with a Solution Manager you can minimize the downtime. There are two types of deployments according to whether the Solution Manager interacts with the load balancer or not.

  • Simple deployment mode, the Solution Manager only updates the metadata of the servers, one by one, without any interaction with the load balancer.
  • In Without service interruption deployment mode, the Solution Manager disables Virtual DataPort servers or clusters in the load balancer and enables them again when the deployment finishes.

When the deployment is configured as Without service interruption, you can choose among different strategies to perform the promotion. The actual options available depend on the topology of your infrastructure, which is shown by the parameter Number of clusters installed. This parameter shows the number of enabled clusters that have at least one enabled Virtual DataPort server in the environment

For more information, Standard mode deployment configuration section of the Solution Manager Guide.

  1. You want to automate the backup process in case the deployment does not work. Is it possible to do that in Denodo?

Click here to check if you got it right

Yes, you can do that in Denodo. Configuring your Solution Manager with Version Control integration will allow you to Enable Automated Backups as part of the deployment process.  With this feature, Solution Manager will automatically push a full backup of both your VDP and Scheduler metadata to a VCS repository.  With this, you can always retrieve a prior state backup from VCS and import it to a new or recovered server, or even use it to power a new Deployment to re-sync a whole Environment.
  1. Can the VDP nodes of the same environment share a metadata database so that the metadata synchronization becomes easy?

Click here to check if you got it right

Yes, the server can be configured to use an external database that is shared between the nodes of the same environment. The benefits are:

  1. Automatic replication: if the Virtual DataPort servers of a cluster share the same external database, you only need to perform a change in one of the servers and it will be propagated to the others, automatically. Otherwise, you have to do the same change on each server of the cluster.
  2. Backups: if the administrator of this external database is doing periodic backups of this database, the metadata of Virtual DataPort will also be backed up. This is beneficial even if you do not have a cluster.

This feature is meant to be used in environments where changes in elements are not done in a controlled way. That is, users create and modify views, web services… at any moment. These environments usually are development environments or Sandbox environments. These environments usually have the same views as the production servers but the users (e.g. data scientists) create additional views to perform analysis.

For more information, please checkout the section Storing the metadata in the external database of VDP Admin Guide.

Questions

Ask a question

You must sign in to ask a question. If you do not have an account, you can register here