Monitoring

Note

Only global administrators, monitor administrators (users with the role monitor_admin) and users with the privilege Monitor for this environment can execute monitoring operations. More information is available in the Authorization section.

Note

Before using this feature, configure the installations of the Denodo Platform to communicate with the Denodo Security Token Server.

With the Solution Manager you can launch the Denodo Monitor to collect the execution logs from a single Virtual DataPort server, or from all the servers of a cluster or environment.

To monitor an environment, a cluster or a server, click on that element and then, click Start Monitor. The elements that are being monitored are marked with an “eye” icon next to the name of the element.

Start Denodo Monitor for an environment

Start Denodo Monitor for an environment

To stop the Denodo Monitor on an environment, cluster or server, click on that element and click Stop Monitor.

Take the following into account:

  • If a server is being monitored, you cannot begin monitoring its cluster or environment.

  • If a cluster is being monitored, you cannot begin monitoring its environment or one of its servers.

  • If an environment is being monitored, you cannot begin monitoring one of its clusters or servers.

  • The list of servers of a monitored cluster or environment can change while the monitoring is running. For instance, the user can add or remove a server, or an environment in automated cloud mode can adjust the number of servers due to the autoscaling. The Denodo Monitor is aware of changes in the catalog of the Solution Manager and will automatically update the list of monitored servers accordingly.

  • If a monitored cluster in automated cloud mode is stopped, the Denodo Monitor will be suspended. It will resume as soon as the cluster is started again.

  • If a monitored environment, cluster or server is removed, the Denodo Monitor will be stopped. This is specially relevant when you are monitoring a server that belongs to an environment in automated cloud mode. On that scenario, operations like restart the cluster, perform a deployment, install an update, etc. rely on removing the current servers and adding new ones.


To see the list of elements that are being monitored, click the menu Monitoring > Active Monitoring.

Active monitoring table

Active monitoring table

Each row of this table is an element that is being monitored. If the element is a cluster or an environment, the Denodo Monitor will connect to all the servers of that cluster or environment.

To stop monitoring a server, click stop-btn or select several rows and click Stop Monitors.


By default, when you shut down the Solution Manager, the Denodo Monitor keeps running. Leaving the Denodo Monitor running permanently is useful to keep a history of the activity of the Denodo servers. For example, to log all the queries that all the Virtual DataPort servers of the organization run.

Nevertheless, you can configure that when you shut down the Solution Manager, it automatically shuts down the Denodo Monitor. To do this, follow these steps:

  1. Stop the Solution Manager.

  2. Edit the file <SOLUTION_MANAGER_HOME>/conf/solution-manager/SMConfigurationParameters.properties.

  3. Set this property to true:

    com.denodo.solutionmanagerserver.monitoring.stopRunningMonitorsOnShutdown
    
  4. Start the Solution Manager.

After this change, the next time you shut down the Solution Manager, the Denodo Monitor will shut down too. When you start the Solution Manager again, you will have to start manually the monitors you started before the restart.


The Denodo Monitor generates the log files in the folder <SOLUTION_MANAGER_HOME>/resources/solution-manager-monitor/work:

  • Log files of environments: <EnvironmentName>/logs

  • Log files of clusters: <EnvironmentName>/<ClusterName>/logs

  • Log files of servers: <EnvironmentName>/<ClusterName>/<ServerName>/logs

For example, if you monitor the “Production” environment, the logs are in the folder <SOLUTION_MANAGER_HOME>/resources/solution-manager-monitor/work/Production/logs.

Note

To store these log files on a different directory, change the value of the property com.denodo.solutionmanagerserver.monitoring.workDir in the file <SOLUTION_MANAGER_HOME>/conf/solution-manager/SMConfigurationParameters.properties. Restart the Solution Manager server to apply the changes.

Monitoring Configuration

There are two configuration files for the Denodo Monitor of the Solution Manager, they are in the folder <SOLUTION_MANAGER_HOME>/conf/solution-manager/denodo-monitor:

  1. ConfigurationParametersGeneral.template: general configuration template file.

  2. ConfigurationParametersServer.template: configuration template file with specific server properties (ping timeout, monitors, list of data sources to ping…).

Solution Manager generates a configuration file for each monitoring execution based on these template files. The generated file is called ConfigurationParameters.properties and it is stored in the conf folder, in the working folder of the monitored element.

JDBC Logging Configuration

You can configure the Denodo Monitor to store the information generated by the Queries monitor and/or the Cache monitor in a database. When you enable this feature, Denodo Monitor will keep storing this information in the log files.

To enable this feature, follow these steps:

  1. If you never monitored any Denodo server, cluster or environment, start monitoring one server. This will initialize the configuration of this module.

  2. Copy the JDBC driver of the database (i.e. its jar file(s)) to the folder <SOLUTION_MANAGER_HOME>/resources/solution-manager-monitor/denodo-monitor/lib.

  3. Go to the folder <SOLUTION_MANAGER_HOME>/resources/solution-manager-monitor/denodo-monitor/sql and pick the script of the database where you want the logs to be stored.

  4. Execute this SQL script on the database. This script creates the tables REQUEST_NOTIFICATION and CACHE_NOTIFICATION.

  5. Edit the file <SOLUTION_MANAGER_HOME>/conf/solution-manager/denodo-monitor/ConfigurationParametersGeneral.template:

  6. To store in the database the information of the “Queries Monitor”, do this:

    1. Set the property vdpqueries.jdbcagent.enable to true

    2. Look for the group of properties of the database you are going to use and uncomment its properties. For example, if you want to use MySQL, look for “JDBC Agent Parameters: MySQL” and uncomment the nine properties below. Change the following properties (do not modify the others):

      1. vdpqueries.jdbcagent.url: set to the URL to the database.

      2. vdpqueries.jdbcagent.user: set to the user name that will connect to the database.

      3. vdpqueries.jdbcagent.password set to the password of the user name that will connect to the database.

        To enter the password encrypted, open a command line and run this:

        For Windows:

        cd <SOLUTION_MANAGER_HOME>\resources\solution-manager-monitor\denodo-monitor\bin
        encrypt_password "<password of the database>"
        

        For Linux:

        cd <SOLUTION_MANAGER_HOME>/resources/solution-manager-monitor/denodo-monitor/bin
        ./encrypt_password.sh "<password of the database>"
        

        Copy the result of this program to the property vdpqueries.jdbcagent.password and set vdpqueries.jdbcagent.password.encrypted to true.

  7. To store in the database the information of the “Cache Monitor”:

    1. Set the property vdploadcacheprocesses.jdbcagent.enable to true.

    2. Repeat the same steps above, but for the properties whose name start with vdploadcacheprocesses.

Kerberos Configuration

The JDBC connections can be configured to use Kerberos credentials: user and pass or user and keytab options are supported. To use a Kerberized JDBC connection establish the following properties:

  • [vdpqueries|vdploadcacheprocesses].jdbcagent.useKerberos: if true, the information generated by the queries monitor and/or the cache monitor will attempt to create a JDBC kerberized connection.

  • [vdpqueries|vdploadcacheprocesses].jdbcagent.krbUser: Kerberos user.

  • [vdpqueries|vdploadcacheprocesses].jdbcagent.krbPassword and [vdpqueries|vdploadcacheprocesses].jdbcagent.krbPassword.encrypted: Kerberos password.

  • [vdpqueries|vdploadcacheprocesses].jdbcagent.krbKeyTab: Kerberos keytab path.

  • [vdpqueries|vdploadcacheprocesses].jdbcagent.krbConfigFile: path to the krb5.ini file in case it is not in the default paths.

  • [vdpqueries|vdploadcacheprocesses].jdbcagent.krbDriverOptions: custom properties to be used by the driver when stablishing a JDBC kerberized connection. If present, it will ignore natively supported connection properties. It must follow the format: prop1=value1;prop2=value2;…

Cloud Storing Configuration

By default, the Denodo Monitor stores the log files on the local file system. You can configure it to store these log files in the cloud (in AWS S3) as well.

To do this, first you have to create an S3 bucket and obtain the AWS region of this bucket, the name of the bucket and the path within this bucket in which you want to store the log files.

To enable this feature, edit the file <SOLUTION_MANAGER_HOME>/conf/solution-manager/denodo-monitor/ConfigurationParametersGeneral.template and do these changes:

  • Uncomment the property monitors.cloudStorage.provider and change its value to AWS_S3.

  • Uncomment these properties and set its value to the key and secret respectively of your S3 account:

    monitors.cloudStorage.s3.credentials.awsKey
    monitors.cloudStorage.s3.credentials.awsSecret
    
  • Uncomment these properties and set its value

    monitors.cloudStorage.s3.destination.region
    monitors.cloudStorage.s3.destination.bucket
    monitors.cloudStorage.s3.destination.path
    

    These properties represent respectively, the region of the AWS bucket, the name of the bucket and the path within the bucket to store the log files

  • To store the logs compressed in AWS, uncomment the property monitors.cloudStorage.compression.enabled and change its value to true.

  • The property monitors.cloudStorage.buffer.size controls the maximum number of log events that the Denodo Monitor will hold in memory, before storing them in S3. The default value is correct for the majority of scenarios. To change it, uncomment the property.

  • The property monitors.cloudStorage.buffer.age controls the maximum amount of time in minutes that Denodo Monitor will hold a log event in memory, before storing it in S3. The default value is correct for the majority of scenarios. To change it, uncomment the property.

To “Uncomment a property” remove the character # at the beginning of the line.

The default values for these properties are correct for the majority of scenarios.