Lookout

Expert trails guide Denodo users through all the relevant materials related to a specific topic, including official docs, KB articles, training, Professional Services offerings, and more. The main goal is to give users a single place with references to all the information that they need to become a Denodo expert on any specific topic.

Monitoring capabilities are crucial for any application deployment from smallest to largest. The advanced monitoring capabilities and tools included within the Denodo Platform allow every Denodo user to have an overall idea of their infrastructure and Denodo services performance.

The Hike

Stage 1: Health Monitoring

Before the monitorization of any server, the first step should be to check if that server is alive.

To help the customer with this task, Denodo provides several scripts, URLs, and methods aimed at each one of the Denodo Platform and Solution Manager components that help you to determine if those components are alive.

Further details on these utilities can be found in Health Monitoring.

Stage 2: Query Monitoring

The Denodo Platform supports three different features that allow customers to monitor the queries that a Virtual DataPort server receives:

Real-Time

• Query Monitor

The Denodo Query Monitor feature allows you to monitor in real-time the queries that are being executed in the VDP server where this feature has been launched from.

You can use the Query Monitor feature from both the Denodo Admin tool and the Design Studio web tool. The Query Monitor lists all the statements that Virtual DataPort is currently processing: SELECT, CALL, CREATE VIEW, ALTER DATASOURCE JDBC, CREATE USER, etc.

The Query Monitor is available to all users but the queries a user sees depend on the privileges granted to the user. This tool provides and helps you to analyze the Execution Trace of the queries being executed and that have not finished yet containing important information (Execution Trace Information) about the execution plan that is being followed by the Virtual DataPort server to resolve the query.

Historical

• VDP Logging System

All Denodo components use the Apache Log4j 2 library to log their activity. In particular, for a VDP server, information about the queries executed can be logged by enabling both the requests and queries logs. The steps to do this are described in How to log the requests issued on a VDP server.

In the Log column details article, you can find further details on the columns used to store the data in the log files.

Log information can be stored on Amazon AWS S3 in case of Denodo deployments that run on AWS. That allows it to keep those logs when a server is switched off and will no longer be used. The Knowledge Base article How to store Denodo logs in Amazon S3 explains how to set this up.

Besides the requests and queries logs, a VDP server can also store the execution traces of the queries that have been executed. You could also choose to store the execution trace of all the queries or only the queries that failed; and to store them in plain text, on a file for the Trace Viewer to review them, or both.

Note that if you store the execution trace of all the queries (not just the ones that failed), you need to consider the amount of disk space required.

Further details on the Denodo Platform logging system can be found in Configuring the Logging System.

Stage 3: Virtual DataPort Server Monitoring

Denodo divides the VDP server Monitoring into two main sections: real-time monitoring and historical data diagnosis. Besides, for each section, Denodo provides several features as you can see in the following subsections:

Real-Time

• JMX

It is possible to access the monitoring information in real-time of a particular VDP server using the Java Management eXtensions (JMX) standard. You can access the information provided by a VDP server through the JMX interface from any third-party JMX client (such as JavaTM VisualVM ) or the Diagnostic & Monitoring tool included within the Denodo Platform installation.

Note that before starting to monitor a VDP server via JMX, you may need to perform the following actions:

• Set the Host Name of Virtual DataPort
• Enable Authentication on the Monitoring Interface

Further information on the JMX interface can be found Monitoring with a Java Management Extensions (JMX) Agent.

• Diagnostic & Monitoring tool

The Diagnostic & Monitoring tool (DMT) is a web application that runs in the embedded web container and that tackles two different and complementary scenarios: Monitoring (Real-Time) and Diagnostic (Historical).

In this section, the focus is on real-time Monitoring, connecting to the JMX interface of particular VDP servers.

The DMT can handle all this information and makes it easier for a person to understand by displaying a graphical report composed of different charts sorted into sections. You can find further details about its functioning in the General Architecture.

More information about the DMT is available in the Diagnostic & Monitoring Tool Guide.

Historical

• VDP Logging System

As already introduced in “Stage 2”, VDP servers use the Apache Log4j 2 library to log their activity.

By default, VDP servers stores information related to its functioning in the following log file:  <DENODO_HOME>/logs/vdp/vdp.log.

Further details on the Denodo Platform logging system are described in Configuring the Logging System.

• Denodo Monitor

In order to monitor the Virtual DataPort server, Denodo provides two flavors of the same component: the Denodo Monitor embedded in the Solution Manager installation and the Denodo Monitor embedded in the Denodo Platform installation.

In order to monitor a VDP server functioning, you can use the Denodo Monitor logs that stores detailed information from several for a VDP server such as local data belonging to the host where the VDP is running, information about the data sources being used by VDP, connections, sessions, thread usage, cache, etc

NOTE: It is recommended to centralize the monitorization of your entire Denodo infrastructure using the Monitor feature of the Solution Manager instead of using the Denodo Monitor tool embedded in any Denodo Platform server.

Stage 4: Denodo Reporting Tools

Denodo provides the following web tools to help users analyze the information stored in the logs gathered by the Denodo Monitor tool and understand the Virtual DataPort server behavior relaying on the graphs provided as the result of the log analysis.

Diagnostic & Monitoring Tool

As already introduced in this document, the Diagnostic & Monitoring tool (DMT) is a web application that runs in the embedded web container and that tackles two different and complementary scenarios: Monitoring (Real-Time) and Diagnostic (Historical).

In this section, the focus is on the Diagnostic feature that helps customers to diagnose past problems by analyzing historical data (log files) generated by Virtual DataPort servers and the Denodo Monitor.

The Diagnostic feature allows customers to review the state of Virtual DataPort servers in the past by digesting VDP server logs and Denodo Monitor logs, and finally displaying a graphical report with different charts sorted in sections.

More information about the Diagnostic & Monitoring tool can be found in the “Diagnostic & Monitoring tool (Real-Time)” section explained in Stage 3 and the Diagnostic & Monitoring Tool Guide.

Denodo Dashboard

The Denodo Dashboard is a DenodoConnect component lying on top of a customized version of Apache Superset called Superset for Denodo. Its purpose is to visualize and analyze the state of Denodo installations mainly through data logged by the Denodo Monitor.

It includes a predefined set of dashboards ready to be imported and also helps create new dashboards for tailored and specific requirements including FinOps.

To work with the Denodo Dashboard, you will have to run or deploy the latest version of the Superset for Denodo. Then proceed with the necessary configuration steps to enable the Denodo Monitor to log its data to a Database rather than files. You can follow the Denodo Dashboard - User Manual for more detailed and thorough steps.

Exploration

Fill up your backpack with additional gear:

Guided Routes

Denodo Training Courses

Denodo training courses provide expert data virtualization training for data professionals, including administrators, architects, and developers.

If you are interested in Monitoring you should enroll in the following course:

• Denodo Monitoring and Auditing: This course explains how to monitor your Denodo cluster with the out-of-the-box tools shipped with Denodo Platform 9.

Success Services

Denodo Customer Success Services can help you at the start or any part of your Self-Service Analytics trail. You can find information about the Denodo Success Services offering in:

Success Services

Advisory Sessions

Denodo Customers with active subscriptions have access to request Meet a Technical Advisory sessions.

These are the sessions available related to Monitoring.

Success Accelerators

In addition to Advisory sessions, Success Services includes Success Accelerators that can help you.

In particular, you may be interested in the following module:

• Operations Quick Start

Additional other related modules could be:

• Capacity Assessment

If you are a Denodo customer, you can reach out to your Customer Success Manager for details about any Guided Route that you need.

Big Hike Prep Check

Let’s see if you are ready to start your big trail. Take this 5-question questionnaire to check your readiness for an enjoyable hike.

Read the questions below, think about the solution and check if you got them right by looking at the solution. Have you become an expert?

1. What should a developer review to understand how a query works?

Click here to check if you got it right

2. Would it be possible to store the execution trace only for the queries that fail?

Click here to check if you got it right

3. Would it be possible to generate load in the development environment to capture the behavior of the system before promoting the code to production?

Click here to check if you got it right

4. Can the Denodo Dashboard tool connect to the Denodo Monitor logs of the Solution Manager on a different server?

Click here to check if you got it right

5. Can email alerts be set up for resource management?  

Click here to check if you got it right