Introduction

Big corporations with high availability and load balancing requirements tend to build a large infrastructure to manage their increasing amount of data. Such infrastructure may consist of several geographically distributed clusters. Each one of these clusters is made up of several Virtual DataPort servers that process the queries, possibly a local cache database and a Scheduler server that periodically executes jobs to keep the cache updated.

Moreover, every new version of the servers’ metadata has its own lifecycle. It is created in a development environment and promoted to several intermediate environments (testing, staging…) until it finally reaches production. Each one of these environments may try to replicate the final infrastructure, but on a different scale.

Scenarios like that face their own particular problems of administration. Simple tasks like installing a new license, upgrading a server or deploying a new version of the metadata, may become much more complex when they have to be performed on many machines at once.

Architecture

The Solution Manager is comprised of these components that work together to help administrators manage the Denodo Platform servers:

  1. License Manager. A centralized point from where administrators manage all the Denodo licenses of the organization.

  2. Solution Manager server. Its main feature are:

    1. For administrators that want to deploy the Denodo Platform on AWS, it installs Denodo servers on AWS automatically. That is, without having to create and configure AWS EC2 instances manually, load balancers, etc., and without having to connect to each individual instance to configure various settings.

    2. For developers, to promote changes from development to the other environments. For example, to propagate new views or changes to views of Virtual DataPort from the development server to the testing servers and then, to production.

  3. Solution Manager Administration Tool. Web tool to manage the Solution Manager and the License Manager.

  4. Virtual DataPort

  5. Diagnostic & Monitoring Tool

  6. Scheduler Administration Tool. Web tool to manage the Scheduler servers of all the installations of the Denodo Platform.

  7. Design Studio. Web tool is aimed at developers that create data sources, base views, derived views, publish web services, etc. Its layout is similar to the Administration Tool of Virtual DataPort but is a web tool. It can connect to all the Virtual DataPort servers of your organization.

This diagram pictures the recommended architecture for a Denodo deployment in an organization.

  • This architecture defines three environments: development, testing and production. If your organization has multiple working groups and each group uses its own set of Denodo servers, define these three environments for each group but we recommend using a single License Manager for the entire organization to manage the licenses.

  • Solution Manager includes a Virtual DataPort server, which is used for two things only:

    1. Authenticating the users of the Diagnostic & Monitoring Tool and the administration tool of the Solution Manager. All the configuration is done from the Solution Manager Administration Tool but underneath, it uses Virtual DataPort.

    2. The Diagnostic & Monitoring Tool creates a set of views in this Virtual DataPort to analyze the diagnostic files.

    It cannot be used as another regular Denodo instance.

  • When possible, use the Design Studio, the Scheduler Administration Tool and the Diagnostic & Monitoring Tool included with the Solution Manager. The installer of the Denodo Platform also includes these tools but if you open them from the Administration Tool of the Solution Manager, you will benefit from single sign-on, a single point of access to all the Denodo tools, etc.

  • We do not recommend installing the Solution Manager in the same host as other Denodo instances for performance reasons: the Diagnostic & Monitoring Tool executes complex queries to analyze the log files and this could impact the performance of the other Denodo servers running in the same host.

Role of the License Manager

  • When a Denodo component starts (for example, a Virtual DataPort server), it requests permission to run to the License Manager. If approved, the server starts and periodically, it requests permission to keep running. The benefit of this approach is that you do not need a license file for each server and there is no limitation by IP address. The licenses only keep a limitation of the number of Denodo servers you can run simultaneously, in addition to certain limitations depending on the environment.

  • Only server components request a license, not client applications. I.e. the Virtual DataPort administration tool does not connect to the License Manager because it does not need a license.

Role of the Solution Manager Regarding Promotion of Elements

The usual development workflow with Denodo has these steps:

  1. A developer creates/modifies elements on the Virtual DataPort and Scheduler servers of the development environment.

  2. The developer creates a revision using the Solution Manager web tool. To do this, the Solution Manager connects to the development server and the user selects the elements to include in the revision (data sources, views, Scheduler jobs…).

  3. The administrator reviews the revision and if it is correct, deploys it to the Denodo servers of the testing environment.

  4. The developer retests the changes in the testing environment.

  5. The administrator deploys this revision to the production environment.

The section Promotions explains this workflow in more detail.