Virtual DataPort Monitors

There are five monitors specific to Virtual DataPort servers:

  • Virtual DataPort Queries Monitor. It logs all the statements processed by Virtual DataPort. It is specific to Virtual DataPort and cannot be used to monitor other Denodo servers.

    The output is logged to denodo-monitor/logs/vdp-queries.log.

    It collects information about regular queries (i.e. SELECT and CALL) and DDL statements (i.e. CREATE TABLE, CREATE VIEW, DROP TABLE, etc.), CREATE USER, CONNECT, CLOSE, etc.

  • Virtual DataPort Query Data Sources Monitor. While Virtual DataPort Queries Monitor collects general information about a query, Virtual DataPort Query Data Sources Monitor provides details about the access to the different data sources including the number of rows read from the data source, the response time, and the cloud provider (if any) where the data source is located. This information can then be used for both performance and financial monitoring.

    • Performance monitoring: It allows to identify bottlenecks as it is possible to check what data sources are taking more time and identify operations that were not pushed to a data source among other things.

    • FinOps monitoring: Since update 8.0u20230914, it is possible to specify the cloud provider and region hosting the data sources (available for JDBC, custom, JSON, XML and delimited files), the Denodo servers and the clients accessing Denodo. This information is included in this monitor along with other data that can be relevant to make cost estimations on the cloud usage. For instance, the estimated number of bytes read in the data source or the number of rows transferred and the estimated size of each row. This allows to answer queries like what is the estimated amount of data transferred from a specific cloud provider or who are the teams causing more data traffic from a certain cloud provider and region.

  • Virtual DataPort Cache Monitor. It logs the activity of the processes that load the Virtual DataPort cache.

    The output is logged to denodo-monitor/logs/vdp-loadcacheprocesses.log.

  • Virtual DataPort Connections Monitor. It logs information about opened and closed connections, user account that opened/closed the connection, the interface it used (JDBC, ODBC, web service, etc.), from which IP, etc.

    The output is logged to denodo-monitor/logs/vdp-connections.log.

  • Virtual DataPort Data Sources Monitor. Every number of seconds (determined by the interval property), it logs information about all the data sources of the Virtual DataPort server. In addition, it can ping periodically some data sources and register the result.

    The output is logged to denodo-monitor/logs/vdp-datasources.log.

  • Virtual DataPort Query Blocks Monitor. When enabling this monitor, also enable the “Queries Monitor”. This monitor logs details of the statements processed by Virtual DataPort. The Summary Recommendations Tool uses this information combined with the information of the Virtual DataPort Queries Monitor.

    The output is logged to denodo-monitor/logs/vdp-queryblocks.log.

Logging Headers

Every time the Denodo Monitor starts monitoring or has to acquire a new connection to the server, the following header is included in the logs generated by the Server and Virtual DataPort monitors:

Logging started at: 2020-02-03T15:29:17.411. Denodo Monitor 7.0 started at: 2020-02-03T15:29:16.925. VDP Server 7.0 update 20190903 started at: 2020-02-03T14:36:24.260

This header provides timestamp information about:

  • Logging. Instant when the monitor started logging. It is different among monitors due to their different polling intervals.

  • Denodo Monitor. Instant when the Denodo Monitor started monitoring. It is the same value for all the monitors.

  • Virtual DataPort Server. Instant when Virtual DataPort Server was launched.

When the logs contain several logging headers, you can detect important events by comparing their values:

  • If Virtual DataPort Server start time changes, then Virtual DataPort Server was restarted.

  • If Denodo Monitor start time changes, then Denodo Monitor was restarted.

  • If both dates remain the same, then the connection with the server was lost.

Add feedback