Introduction

Denodo provides real time monitoring of all data service components in an integrated fashion, including Denodo platform components themselves and participating data sources. By analyzing parameters such as query response times, memory usage and CPU usage, etc, Denodo administrators can proactively rectify any potential data flow bottlenecks and determine when servers are at capacity.

All monitoring information is deliverable via the Denodo Dashboard and via SNMP and JMX standards. Denodo can therefore interoperate with most leading Systems Management packages.

Monitoring information can be saved for further analysis of how performance evolves over time. For these analyses, Denodo exposes server-side parameters (e.g., number of connections, memory and CPU usage, number of threads, data source usage, cache), along with statistics on query performance (execution time, number of rows, queries by type of CRUD operation invoked, use of cache, running status, etc.) and status indicators for each underlying data source.

Administrators can combine intelligence gathered from all of these monitoring sources to analyze the optimum performance of the Denodo server(s) and the data sources being virtualized as well.

Most relevant monitoring information

• General information on the server: memory usage, CPU usage, number of active and inactive connections …
• Configuration and accesses to the server cache
• Accesses to each data source
• Accesses to the views and stored procedures
• Transactions executed by the Denodo server
• Logon attempts

In the following sections the MBeans exposing this monitoring information are explained in detail.

General Information on JVM

java.lang:type=Memory: management interface for the memory system of the Java virtual machine.

• HeapMemoryUsage: current memory usage of the heap that is used for object allocation.
• NonHeapMemoryUsage.current memory usage of non-heap memory that is used by the Java virtual machine.

com.denodo.vdb.management.mbeans:type=MemoryManagementInfo: total memory occupied by the server, and memory peaks.

• TotalMemory: estimate of the total memory used by the Server.
• MaxMemory: estimate of the memory peak reached by the Server since it was launched.

java.lang:type=Threading: management interface for the thread system of the Java virtual machine.

• ThreadCount: Current number of live threads including both daemon and non-daemon threads
• ThreadCpuTime(long id):  total CPU time for a thread of the specified ID in nanoseconds.

java.lang:type=Runtime:  management interface for the runtime system of the Java virtual machine.

• Uptime: uptime of the Java virtual machine in milliseconds.

java.lang:type=OperatingSystem: management interface for the operating system on which the Java virtual machine is running.

• ProcessCpuLoad:  "recent cpu usage" for the Java Virtual Machine process. If the Java Virtual Machine recent CPU usage is not available, the method returns a negative value.
• AvailableProcessors: number of processors available to the Java virtual machine.

java.lang:type=ClassLoading: management interface for the class loading system of the Java virtual machine.

• LoadedClassCount: number of classes that are currently loaded in the Java virtual machine.

General Information on the server

com.denodo.vdb.management.mbeans:type=VDBServerManagementInfo: number of active connections and requests, instance name and instance role (master/slave).

• ActiveConnections: number of clients connected to the Server.
• TotalConnectionFailures: number of failed connections. For example, clients that tried to connect to the Server with invalid credentials.
• ActiveRequests: number of queries that the Server is currently executing.
• WaitingRequests: number of requests that are waiting to be executed. These requests are waiting because the Server has surpassed the “Maximum concurrent requests” threshold.

General Information on cache

com.denodo.vdb.management.mbeans:type=Cache,databaseName=<dbname>,subtype=DatabaseCache: cache information of a virtual database.

• DatabaseName: name of the database.
• DBURI: URI of the database used as cache.
• DriverClassName: class of the JDBC driver used to connect to the database that stores the cached data.

• CacheStatus: status of the cache: ON or OFF.
• ActiveRefreshCacheProcesses: number of processes that are currently updating the cache of the views of the database.
• MaxRefreshCacheProcesses (editable value): maximum number of concurrent processes that the Server may start to update the cache of the views of this database.
• LastCacheAccessStatus: indicates whether the last access to the cache database was successful (true) or not (false).

General Information on data sources

com.denodo.vdb.management.mbeans:type=DataSourceManagementInfo, name=GlobalDataSourceManagementInfo: number of data sources and active requests.

• TotalDataSourceCount: number of data sources.
• TotalActiveRequestCount: number of current requests to any data source.

For every type of data source there is an attribute with the number of data sources and the number of active requests to that type of source.

Each virtual database has also its own MBean for its own data sources:

com.denodo.vdb.management.mbeans:type=DataSourceManagementInfo, databasename=<VDP database name>,aux=LocalDataSourceManagementInfo

Besides, each data source has its own MBean:

com.denodo.vdb.management.mbeans:type=DataSourceManagementInfo, dataSourceType=<dstype>,dataSourceName=<dsname>

• ActiveRequests. Number of active requests to the data source.
• MaxActive, NumActive and NumIdle (only for JDBC and ODBC sources). Provide information about the pool of connections with the database:

• MaxActive: maximum number of connections in the pool.
• NumActive: number of connections established with the source that are being used to execute a query.
• NumIdle: number of idle connections established with the source.

Requests

com.denodo.vdb.management.mbeans:type= RequestsManagementInfo,databaseName=<dbname>: provides information on DML statements requests (SELECT, INSERT, UPDATE, DELETE, etc).  

• ActiveRequests. Number of active requests.

• MaxRequests. Maximum number of requests exported at the same time in the MBean. Each request appears as an attribute of the form Request<i>.
• Request<i> (compound value). Contains information about the request sent to this database.

General Auditing

• DatabaseName. Name of the database on which the statement is executed.
• WaitingTime. Number of milliseconds the query was waiting in the queue of queries before the Server began executing it.
• StartTime. Instant at which the Server received the query.
• EndTime. Moment at which the statement finished its execution.
• State. This attribute stores the state of the top node of the execution plan and its value can be OK, ERROR or PROCESSING.
• Completed. true if the query finished correctly, false otherwise.
• Cache. true if the query has accessed the cache during its execution, false otherwise.
• Swap. true if the server swapped the intermediate results to disk during the execution of the query, false otherwise.
• Queued. true if this query has been queued because the limit of maximum number of concurrent queries has been reached, false otherwise.

Detailed Auditing

• UserAgent. Name of the application that performed the request.
• AccessInterface. Type of client that performed the request.
• UserName. ID of the user running the statement

• VQLQuery. VQL code for the statement.
• RequestType. This indicates the type of statement.
• Elements. Name of the views and/or stored procedures referenced in the query.

Cache Load Processes

com.denodo.vdb.management.mbeans:type=Cache,databaseName=<dbname>,subtype=DatabaseCache: cache information of a virtual database.

• RefreshCacheProcess<i> (compound value): contains information about the process #i that was started to update the cache of a view.

• Identifier: unique identifier of the refresh cache process.
• DatabaseName: the name of the database.
• CacheStatus: mode of the cache that is being refreshed. Its value can be:

• OFF
• PARTIAL
• PARTIAL EXACT
• PARTIAL PRELOAD
• PARTIAL EXACT PRELOAD
• FULL.

• Exception: if there was a problem while inserting data into the cache, this attribute contains the cause of the problem.
• NumOfInsertedRows: number of rows that have been inserted in the cache so far.
• NumOfReceivedRows: number of rows returned by the query and that have to be inserted in the cache.
• SqlViewName: name of the table in the cache data source where the cached data is stored.
• StartDataStorageTime and EndDataStorageTime: instant at which the Server started/finished storing the data in the cache.
• TTLInCache: time to live (in seconds) of the data stored by this “refresh cache process”. When this time is reached, the contents inserted by this process will be invalid.
• VDPConditionList: list of conditions in the query.
• ViewName: name of the view.

This MBean provides the operation getActiveRefreshCacheProcesses( <maxActiveCacheProcesses:integer>) that returns information about the processes that are currently inserting data in the cache of views of this database.

View cache

com.denodo.vdb.management.mbeans:type=Cache,databaseName=<dbname>,subtype=ViewCache,viewName=<viewname>: cache information of a view: status, last access, last refresh, current TTL, etc.

• DatabaseName: name of the database that this view belongs to.
• ViewName: name of the view.
• CacheStatus: status of the view’s cache.
• LastAccess: last access to the view’s cache.
• LastRefresh: last time the content of this view’s cache was updated.
• RefreshFailuresCount: number of failures when writing new tuples in the view cache.
• TTLInCache: time to live (in seconds) of the data in the view’s cache. When this time is reached, the contents of the cache are considered invalid.
• BatchSizeInCache: the Server inserts rows into the cache, in batches. This attribute controls the number of rows inserted, per batch.

Most relevant JMX notifications

Denodo server generates different types of JMX notifications whenever:

• A login attempt is made.
• A DDL statement (Catalog Access) affecting the metadata of a virtual database is executed. e.g. a new view is created, a view definition is modified, the privileges of a user are changed, etc.
• A DML statement is executed. E.g., SELECT, INSERT, etc.
• A transaction starts or ends.

Login and session

The com.denodo.vdb.management.mbeans:type=VDBServerManagementInfo MBean provides the following notifications:

• loginOk: when a client logs in correctly into the Virtual DataPort server.
• loginFailure: when a client fails to connect. For example, because the password provided is not correct or the user does not have enough privileges to connect to the database.
• logout: when a client logs out.
• openSession: when a client opens a session.
• closeSession: when a client closes a session.

All of these notifications provide this information:

• AccessInterface
• ClientIP
• IntermediateClientIP
• ConnectionId
• ConnectionStartTime
• ConnectionEndTime
• SessionId
• SessionStartTime
• SessionEndTime
• Login
• DatabaseName
• JMSQueueName
• WebServiceName
• UserAgent

DDL statements

com.denodo.vdb.management.mbeans.CatalogManagementInfo,databaseName=<dbname>: provides information about the Data Definition Language (DDL) statements processed by the Server.

On subscribing to this MBean, notifications are sent every time a DDL query is executed  (e.g., a new view is created, a view definition is modified, the privileges of a user are changed, etc.), or a transaction starts/ends with information on the statement/transaction.

DML statements

com.denodo.vdb.management.mbeans:type= RequestsManagementInfo,databaseName=<dbname>: provides information on DML statements requests (SELECT, INSERT, UPDATE, DELETE, etc).

On subscribing, notifications are sent with detailed query information. Every time a DML statement is run on the database, two notifications are sent: one, indicating the beginning of the request and the other indicating the end.

Transactions

com.denodo.vdb.management.mbeans:type= TransactionsManagementInfo ,databaseName=<dbname>: provides information about the transactions executed in a database of the Server.

It is possible to subscribe to the events of the TransactionsManagementInfo MBean. In that case, every time a transaction starts or ends on the specified database, a notification is received.