Introduction

Prometheus and Grafana are widely used open-source tools for monitoring and observability.

Prometheus focuses on collecting and storing metrics in a time-series database, using its own query language (PromQL) and supporting alerts through Alertmanager.

Grafana, on the other hand, provides powerful visualization capabilities, allowing users to build dashboards and analyze data from Prometheus and other sources.

In this article we will detail the process to install Prometheus and Grafana to track the Denodo Lakehouse Accelerator (formerly known as Denodo Embedded MPP)  cluster health, resource usage, and query performance, helping administrators ensure stability, optimize workloads, and quickly identify issues across the distributed environment.

Prerequisites

• A working configuration of the Denodo Lakehouse Accelerator.
• Kubernetes node resources to deploy Prometheus and Grafana, a separated  Node Group/ Node Pool is recommended to avoid potential performance and networking issues with the Denodo Lakehouse Accelerator nodes.

Denodo Lakehouse Accelerator configuration

First, it is necessary to enable the collection of metrics from the underlying Presto engine. Presto exposes its internal performance data, such as memory usage, query execution statistics, and JVM information, through Java Management Extensions (JMX). To make these metrics accessible, a Prometheus JMX agent is included in the Presto coordinator and worker nodes.. This agent acts as an intermediary, publishing the metrics in a format that Prometheus can scrape. Once the JMX exporter is deployed and properly configured, Prometheus can begin gathering these metrics, which will later be visualized and analyzed through Grafana dashboards.

JMX Configuration

The JMX agent requires to be configured specifying the metrics it will collect. For this, the values.yaml file in the Denodo Lakehouse Accelerator chart has a section for its activation and configuration.

metrics:   prometheus:     enabled: true     port: 8081     config:       rules:         - pattern: 'java.lang<type=Memory><(.*)>'         - pattern: 'java.lang<type=GarbageCollector, name=(.*)><(.*)>'         - pattern: 'java.lang<type=Threading><(.*)>'         - pattern: 'java.lang<type=OperatingSystem><(.*)>'         - pattern: 'com.facebook.presto.execution<name=(.*)><(.*)>'           name: "presto_execution_$2"           labels:             query: "$1"           type: GAUGE         - pattern: 'com.facebook.presto.execution<type=QueryManager><>(.*)'           name: "presto_execution_querymanager_$1"           type: GAUGE         - pattern: 'com.facebook.presto.sql<name=(.*)><(.*)>'           name: "presto_sql_$2"           labels:             query: "$1"           type: GAUGE         - pattern: ".*"           action: "drop"

In this section, you can enable monitoring, configure the port on which the JMX agent will be deployed, and configure the rules that will apply.
You can find the details of the parameters at the JMX agent documentation page.

Velox Configuration

If using the Velox version of the Lakehouse Accelerator you have to enable the previous JMX configuration for the Coordinator.

Additionally, the following property must be set for the workers in the values.yaml file.

worker:   additionalConfig: [     runtime-metrics-collection-enabled=true   ]

Note: When using the Velox version of the Lakehouse Accelerator, avoid using “-” in external catalog names, as those names are used in the generated metrics, and it is unsupported.

               

Once everything is set up we can run the helm upgrade command to update the configuration of our Denodo Lakehouse Accelerator Cluster deployment:

helm upgrade prestocluster <prestocluster-folder-path> -n <prestocluster-namespace>

Installing Prometheus

You can install Prometheus using the helm charts available at the prometheus-community github repository.

1. First, you can create a namespace to deploy the resources separately.
   

kubectl create namespace <your-namespace>

Note: These components should run on a node that is isolated from the Denodo Lakehouse Accelerator to avoid potential performance or networking conflicts.

2. Then you have to add the helm repository using the following command:
   

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update

3. Afterwards, you can download the values.yaml file from the repo and adjust it to the needs of your deployment with the following command:

helm show values prometheus-community/prometheus > prometheus-values.yaml

For a basic deployment in AWS you can use this Prometheus prometheus-embedded-mpp-values.yaml or this prometheus-embedded-mpp-prestissimo-values.yaml for a Velox based deployment .

This values.yaml file creates the following resources:

• A private load balancer that allows you to connect to the Prometheus page within your VPC, otherwise it is only accessible within the Kubernetes Cluster by default.
  
• One volume of type gp2 to persist the Prometheus server data.
  
• One volume of type gp2 to persist the alert manager data.

• Sets the default retention period of the metrics to 5 days (can be changed at “server.retention”).
  
• The Prometheus Server, Alert Manager and Kube State Metrics at the nodegroup you specify (replace <your-nodegroup> with the name of your nodegroup).
  
• Scrape configs that automatically scrape the running Presto coordinator and workers on 2 different jobs using their labels.
  
• 7 alert rules.

4. Finally, after creating your values.yaml file you can deploy your Prometheus server with the following command:
   

helm install prometheus prometheus-community/prometheus -f values.yaml -n <your-namespace>

Replace <your-namespace> with the namespace you created at the first step, if it is not specified, it will be deployed at the default namespace.

Accessing Prometheus Web UI

You can access the Prometheus web UI from your network using the IP of the load balancer created with the previous values.yaml file.

Using that ip (172.17.5.137 in the example) we can access the web UI at port 80 through our private network.

There you can check the connection with the pods at the Status > Target health tab.

Configuring Alerts

While monitoring the Denodo Lakehouse Accelerator environment provides valuable insights into its performance and health, monitoring alone is not enough. To ensure a timely response to potential issues, it is essential to configure alerts that automatically notify operators when abnormal behavior or critical conditions occur. Proper alerting enables proactive detection of problems, such as high resource consumption, query failures, or memory shortages, allowing the team to act quickly and maintain the stability and efficiency of the Denodo Lakehouse Accelerator environment.

Alerts can be configured at the Prometheus values.yaml file under the serverFiles.alerting_rules.yml section. The values.yaml specified previously includes 7 rules, as shown below:

• Coordinator Down: This alert triggers when one minute has passed without a running coordinator.
  
• Too Few Worker Pods: This alert triggers when 3 minutes have passed and the number of running workers is below the expected one.
  
• HPA at MaxCapacity: This alert triggers when the HPA in your cluster has reached its max capacity.
  
• Too many container restarts: This alert triggers when a container in your deployment has restarted more than 5 times in the last 15 minutes.
  
• Queries killed OOM: This alert triggers when the Cluster kills queries due to out of memory.
  
• High CPU Usage: This alert triggers when a worker or the coordinator reaches the 90% CPU usage.
  
• High Memory Usage: This alert triggers when a worker or the coordinator reaches the 90% Memory usage.

You can also see the alerts configured at the Prometheus Server page under the “Alerts” tab.

Configuring Receivers

You can configure the notification receivers at the alertmanager.global.config section of the values.yaml file. Prometheus offers a wide range of receiver options, such as email, Microsoft Teams or Jira, you can check the complete list and how to configure them at the Prometheus receiver configuration documentation.

Installing Grafana

You can install Grafana using the helm charts available at the Grafana github repository.

1. Deploy Grafana in the same namespace you created for Prometheus. This is necessary so that the Prometheus datasource can be configured when the Grafana pod initializes.
   
2. Then you have to add the helm repository using the following command:
   

helm repo add grafana https://grafana-community.github.io/helm-charts helm repo update

3. Afterwards, you can download the values.yaml file from the repo and adjust it to the needs of your deployment with the following command:
   

helm show values grafana/grafana > grafana-values.yaml

For a basic deployment in AWS with pre-configured dashboards you can use the grafana-values.yaml or the grafana-prestissimo-values.yaml for a Velox based deployment.

This values.yaml file creates the following resources:

• A private load balancer that allows you to connect to the Grafana monitoring page within your VPC, otherwise it is only accessible within the Kubernetes Cluster by default.
  
• One volume of type gp2 to persist the Grafana data.
  
• The Grafana pods at the nodegroup you specify (change the <your-nodegroup> for the name of your nodegroup).
  
• A configmap with a dashboard with label embedded-mpp-dashboard.

4. Finally, after creating your values.yaml file you can deploy your Grafana pods with the following command:
   

helm install grafana grafana/grafana -n <your-namespace> -f values.yaml

Changing <your-namespace> for the namespace you created at the first step, if it is not specified, it will be deployed at the default namespace.

5. Once the deployment is done you will get a command like this to get the admin password to access the Grafana Web UI.
   

kubectl get secret --namespace <your-namespace> grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Accessing Grafana Web UI

Similar to Prometheus, you can access the Grafana web UI from your network using the IP of the load balancer created with the values yaml of the deployment.

Using that ip (172.17.6.12 in the example) we can access the web UI at port 80 through our private network.

At the Dashboard tab you will be able to see the pre-created Denodo Lakehouse Accelerator Dashboard.

You can also see a summary of the alerts created at the Prometheus Server under the “Alerting > Alert rules” tab.

Denodo Lakehouse Accelerator Dashboard

The Denodo Lakehouse Accelerator Dashboard is composed of 3 tabs: Denodo Lakehouse Accelerator Overview, Coordinator Resources and Workers Resources.

Denodo Lakehouse Accelerator Overview

This tab contains the following graphs:

• Running Queries: Number of queries running.
• Completed Queries (rate): Number of queries completed per minute.
• Failed Queries (rate): Number of failed queries per minute.
• Failed Requests (rate): Failed coordinator http requests per minute.
• Killed Queries Due to Out of Memory: Number of queries killed for lack of memory.
• Workers Available: Number of workers running.
• Queued Queries: Number of queued queries.

Coordinator Resources

This tab contains the following graphs:

• Coordinator CPU Usage %: Percentage of CPU used by the coordinator running.
• Coordinator Heap Available Memory (Gib): Available heap memory for each running coordinator in Gib.
• Coordinator Heap Memory Usage (Gib): Current used heap memory for each running coordinator in Gib.
• Coordinator Non Heap Available Memory (Gib): Available non heap memory for each running coordinator in Gib.

Workers Resources

This tab contains the following graphs:

• Workers CPU Usage %: Percentage of CPU used by the workers running.
• Workers Heap Available Memory (Gib): Available heap memory for each running worker in Gib.
• Workers Heap Memory Usage (Gib): Current used heap memory for each running worker in Gib.
• Workers Non Heap Available Memory (Gib): Available non heap memory for each running worker in Gib.

Denodo Lakehouse Accelerator Velox Dashboard

The Velox based dashboards use the same base dashboards as the Java version with some modifications and additions.

Workers Resources

This tab contains the following graphs:

• Workers CPU Usage %: Percentage of CPU used by the workers running.
• Workers Available Memory (Gib): Available memory for the arbitrator to allocate for each running worker in Gib.
• Workers Allocated Memory (Gib): Current allocated memory to each running worker in Gib.

Cache

This tab contains the following graphs:

• Cache Hits (rate): Number of cache hits per second in the last 5 minutes interval.
• Current SSD Cached Entries: Current number of entries on the  SSD Cache.
• SSD Cache Writes (rate): Number of writes to the ssd cache per second in the last 5 minutes interval.
• SSD Cache Reads (rate): Number of reads to the ssd cache per second in the last 5 minutes interval.

Creating New Dashboards

When deploying Grafana with the preconfigured values.yaml provided in this article, a PersistentVolumeClaim is created to preserve your dashboard changes. This allows you to modify existing dashboards or create new ones through the Grafana UI, with those changes retained persistently.

You can refer to the Grafana documentation to learn how to create and modify dashboards using the Grafana UI.

Conclusion

Integrating Prometheus and Grafana with the Denodo Lakehouse Accelerator provides a unified approach to monitoring system health, performance, and resource utilization. This observability framework enhances operational efficiency through real-time insights, proactive alerting with Prometheus Alertmanager, and intuitive visualization via the Grafana interface, enabling teams to maintain stability, optimize workloads, and ensure the reliability of the Denodo Lakehouse Accelerator environment.

References

Denodo Lakehouse Accelerator Observability with OpenTelemetry