Expert Trail: Monitoring

Applies to: Denodo 8.0
Last modified on: 14 Feb 2022
Tags: Denodo Administrator Infrastructure Setup Maintenance

Download document

You can translate the document:

Lookout

Expert trails guide Denodo users through all the relevant materials related to a specific topic, including official doc, KB articles, training, Professional Services offering, and more. The main goal is to give users a single place with references to all the information that they need to become a Denodo expert on any specific topic.

Monitoring capabilities are crucial for any application deployment from smallest to largest. The advanced monitoring capabilities and tools included within the Denodo Platform allow every Denodo user to have an overall idea of their infrastructure and Denodo services performance.

The Hike

Stage 1: Health Monitoring

Prior to the monitorization of any server, the first step should be to check if that server is alive.

To help the customer with this task, Denodo provides several scripts, URLs, and methods aimed at each one of the Denodo Platform and Solution Manager components that help you to determine if those components are alive.

Further details on these utilities can be found in Health Monitoring.

Stage 2: Query Monitoring

The Denodo Platform supports three different features that allow customers to monitor the queries that a Virtual DataPort server receives:

Real-Time

  • Query Monitor

The Denodo Query Monitor feature allows you to monitor in real-time the queries that are being executed in the VDP server where this feature has been launched from.

You can use the Query Monitor feature from both the Denodo Admin tool and the Design Studio web tool. The Query Monitor lists all the statements that Virtual DataPort is currently processing: SELECT, CALL, CREATE VIEW, ALTER DATASOURCE JDBC, CREATE USER, etc.

The Query Monitor is available to all users but the queries a user sees depend on the privileges granted to the user. This tool provides and helps you to analyze the Execution Trace of the queries being executed and that have not finished yet containing important information (Execution Trace Information) about the execution plan that is being followed by the Virtual DataPort server to resolve the query.

Historical

  • VDP Logging System

All Denodo components use the Apache Log4j 2 library to log their activity. In particular, for a VDP server, information about the queries executed can be logged by enabling both the requests and queries logs. The steps to do this are described in How to log the requests issued on a VDP server.

In the Log column details article, you can find further details on the columns used to store the data in the log files.

Log information can be stored on Amazon AWS S3 in case of Denodo deployments that run on AWS. That allows it to keep those logs when a server is switched off and will no longer be used. The Knowledge Base article How to store Denodo logs in Amazon S3 explains how to set this up.

Besides the requests and queries logs, a VDP server can also store the execution traces of the queries that have been executed. You could also choose to store the execution trace of all the queries or only the queries that failed; and to store them in plain text, on a file for the Trace Viewer to review them, or both.

Note that if you store the execution trace of all the queries (not just the ones that failed), you need to consider the amount of disk space required.

Further details on the Denodo Platform logging system can be found in Configuring the Logging System.

Stage 3: Virtual DataPort Server Monitoring

Denodo divides the VDP server Monitoring into two main sections: real-time monitoring and historical data diagnosis. Besides, for each section, Denodo provides several features as you can see in the following subsections:

Real-Time

  • JMX

It is possible to access the monitoring information in real-time of a particular VDP server using the Java Management eXtensions (JMX) standard. You can access the information provided by a VDP server through the JMX interface from any third-party JMX client (such as JavaTM VisualVM ) or the Diagnostic & Monitoring tool included within the Denodo Platform installation.

Note that before starting to monitor a VDP server via JMX, you may need to perform the following actions:

Further information on the JMX interface can be found Monitoring with a Java Management Extensions (JMX) Agent.

  • Diagnostic & Monitoring tool

The Diagnostic & Monitoring tool (DMT) is a web application that runs in the embedded web container and that tackles two different and complementary scenarios: Monitoring (Real-Time) and Diagnostic (Historical).

In this section, the focus is on real-time Monitoring, connecting to the JMX interface of particular VDP servers.

The DMT can handle all this information and makes it easier for a person to understand by displaying a graphical report composed of different charts sorted into sections. You can find further details about its functioning in the General Architecture.

More information about the DMT is available in the Diagnostic & Monitoring Tool Guide.

Historical

  • VDP Logging System

As already introduced in “Stage 2”, VDP servers use the Apache Log4j 2 library to log their activity.

By default, VDP servers stores information related to its functioning in the following log file:  <DENODO_HOME>/logs/vdp/vdp.log.

Further details on the Denodo Platform logging system are described in Configuring the Logging System.

  • Denodo Monitor

In order to monitor the Virtual DataPort server, Denodo provides two flavors of the same component: the Denodo Monitor embedded in the Solution Manager installation and the Denodo Monitor embedded in the Denodo Platform installation.

In order to monitor a VDP server functioning, you can use the Denodo Monitor logs that stores detailed information from several for a VDP server such as local data belonging to the host where the VDP is running, information about the data sources being used by VDP, connections, sessions, thread usage, cache, etc

NOTE: It is recommended to centralize the monitorization of your entire Denodo infrastructure using the Monitor feature of the Solution Manager instead of using the Denodo Monitor tool embedded in any Denodo Platform server.

Stage 4: Denodo Reporting Tools

Denodo provides the following web tools to help users to analyze the information stored in the logs gathered by the Denodo Monitor tool and understand the Virtual DataPort server behavior relaying on the graphs provided as the result of the log analysis.

Diagnostic & Monitoring Tool

As already introduced in this document, the Diagnostic & Monitoring tool (DMT) is a web application that runs in the embedded web container and that tackles two different and complementary scenarios: Monitoring (Real-Time) and Diagnostic (Historical).

In this section, the focus is on the Diagnostic feature that helps customers to diagnose past problems by analyzing historical data (log files) generated by Virtual DataPort servers and the Denodo Monitor.

The Diagnostic feature allows customers to review the state of Virtual DataPort servers in the past by digesting VDP server logs and Denodo Monitor logs, and finally displaying a graphical report with different charts sorted in sections.

More information about the Diagnostic & Monitoring tool can be found in the “Diagnostic & Monitoring tool (Real-Time)” section explained in Stage 3 and the Diagnostic & Monitoring Tool Guide.

Denodo Monitor Reports

The Denodo Monitor Reports tool is also a web application however, in this case, it is not shipped within the Denodo Platform or Solution Manager installation. In fact, this tool is part of the DenodoConnects Components.

This web tool helps customers to analyze the logs gathered by the Denodo Monitor tool to

display the information using graphs sorted in several sections.

The reports shown by the Denodo Monitor Reports tool are complementary to the ones shown by the Diagnostic & Monitoring tool therefore, it is worthy to use both tools when you need to monitor your Denodo servers.

Exploration

Fill up your backpack with additional gear:

Execution Trace and Query Monitor

Official Documentation

Denodo Platform Logs

Official Documentation

KB Articles

Denodo Monitor

Official Documentation

KB Articles

Additional Resources

JMX

Official Documentation

Diagnostics and Monitoring Tool

Official Documentation

Webinars

Additional Resources

Third-Party Tools for Monitoring

Official Documentation

KB Articles

Webinars

Health check

Official Documentation

KB Articles

Alerts

KB Articles

Guided Routes

Denodo Training Courses

Denodo training courses provide expert data virtualization training for data professionals, including administrators, architects, and developers.

If you are interested in Monitoring you should enroll in the following course:

  • Denodo Performance Monitoring: This course explains how to monitor your Denodo cluster with the out-of-the-box tools shipped with Denodo Platform 8.0.

Technical Advisory Sessions

Denodo Customers with active subscriptions have access to request Meet a Technical Advisory sessions.

These are the sessions available related to Monitoring.

Platform Administration

Services Management

Assist in defining Denodo Platform services management (startup/shutdown/restart):

- Configure as Windows Service.

- Configure as Linux Service.

- Create Startup/Shutdown scripts.

Cache: Configuration & Administration Best Practices

- Configure the cache data source/s using a supported database.

- Configure Bulk Load for the cache.

- Set up Cache Maintenance Task.

- Monitoring the cache.

- Smart Caching: Summaries.

Security of Denodo components: Protocols

Security Architecture

Understand the Denodo Platform Security Architecture. Overview and advice to adapt it to your needs:

- Authentication and Authorization model.

- Secure connections: between Denodo Platform components, from clients (Northbound), and to data sources (Southbound). Pass-through credentials.

- Secure passwords of Denodo Platform components (Denodo Monitor, SSL, scripts, etc.)

- Integration with Vault Security Solutions (Protegrity, HPVoltage)

- Define Admin privileges, passwords, and access

- Data in motion (SSL/TLS)

System Stability

Monitoring & Diagnostics: Overview & Best Practices

- Review the different components and resources to monitor the Denodo Platform and how to set them up efficiently.

- Advice on how to interpret the monitoring information, determine the action needed (system problems vs. design), and decide what information to retain and archive.

Monitor Logs Practice Configuration

- Centralized management of logs generated by the different servers in the environment.

- Support to log to disc (or database) the activity of the system for later analysis.

Professional Services

Denodo Professional Services can help you at the start or any part of your query performance trail. You can find information about the Denodo Professional Services offering in:

Professional Services for Data Virtualization | Denodo

In particular, you may be interested in the following module:

  • Operations Quick Start

Additional other related modules could be:

  • Capacity Assessment

If you are a Denodo customer, you can reach out to your Customer Success Manager for details about any Guided Route that you need.

Big Hike Prep Check

Let’s see if you are ready to start your big trail. Take this 5-question questionnaire to check your readiness for an enjoyable hike.

Read the questions below, think about the solution and check if you got them right by looking at the solution. Have you become an expert?

  1. What should a developer review to understand how a query works?

Click here to check if you got it right

When a query has slow performance, it is needed to review the Execution Trace (or Query Monitor if still being executed) to check where the time is being consumed.

  • Swapping, Memory limit reached and Effective time are some of the significant properties when identifying a bottleneck.
  • Information about access to a data source contains Execution time, Response time, SQL sentence and Connection time that are also relevant in the study of bottlenecks.
  1. Would it be possible to store the execution trace only for the queries that fail?

Click here to check if you got it right

Storing the execution trace only for failed queries is possible and allows to gather detailed information on those queries and the cause of the failure while keeping the space used under control.

  1. Would it be possible to generate load in the development environment to capture the behavior of the system before promoting the code to production?

Click here to check if you got it right

This is possible and highly recommended. Either in the development or some testing environment tests should be done to replicate the production load and volume of data. In order to do so, JMeter can be used for instance as described in Denodo Load Testing with Apache JMeter while you monitor using the Denodo Monitor.

  1. Can the Denodo Monitor Reports tool connect to the Denodo Monitor logs of the Solution Manager in a different server?

Click here to check if you got it right

Yes, the only considerations will be that if they are in different servers the VDP Server under the Denodo Monitor Reports will need to have access to the Denodo Monitor information. That can be done:

  • Making the files available using NFS or Samba if the logs are in files.
  • Changing the Denodo Monitor to log to an external database.

This video shows how to store in an external database: Storing Denodo Monitor logs in an external Database

  1. Can email alerts be set up for resource management?

Click here to check if you got it right

The tools of the Denodo Platform do not provide features for email notifications. It is possible to use third party tools to do this. More information can be found in the following KB articles:

Questions

Ask a question
You must sign in to ask a question. If you do not have an account, you can register here

Featured content

DENODO TRAINING

Ready for more? Great! We offer a comprehensive set of training courses, taught by our technical instructors in small, private groups for getting a full, in-depth guided training in the usage of the Denodo Platform. Check out our training courses.

Training