Introduction

This document aims to demonstrate a sample implementation of OpenTelemetry observability for the Denodo Platform using the OpenTelemetry Java Agent.

You can see the architecture this document will follow below:

The OpenTelemetry Java Agent automatically adds monitoring code to your running Java application by modifying its bytecode at startup, acting like an invisible sensor system that tracks performance, errors, and behavior without you having to change any source code. We will use this Agent because Denodo is built with Java.

NOTE: you can set up the Java Agent with any of the Denodo Platform components like the Scheduler Server, Solution Manager Server, and so on. In this article we will set up the Java Agent for the Virtual DataPort Server.

The OpenTelemetry Collector is a central service that receives, processes, and forwards monitoring data (traces, metrics, and logs) from your applications to your monitoring systems like Prometheus, Loki, and Tempo, acting as a reliable middleman that can handle data transformation and buffering.

There are 3 main signal types in OpenTelemetry:

Traces

Record how a request moves through your system step by step, measuring how long each part takes and showing which services and operations were involved in handling it.

In this document we will connect the OpenTelemetry Collector to Grafana Tempo in order to store and query the traces.

IMPORTANT NOTE: Support for trace collection will be added in Denodo 9.2.

Metrics

Numbers that show how your system is running right now, like how many requests per second it handles, how much memory it uses, and how high your CPU usage is. We will use Prometheus as our metrics database.

Logs

Detailed text records that show exactly what your application is doing at each moment, including what went wrong during errors, with extra data attached to help find and fix problems. We will use Loki to store and help us search through these logs.

Finally, we will set up Grafana as our visualization frontend.

Alternative Backends

While this guide demonstrates the setup of Tempo (traces), Prometheus (metrics), and Loki (logs) as backend systems, OpenTelemetry allows for various alternative backends. The OpenTelemetry Collector can be configured to export data to many different observability systems, such as:

• Traces: Jaeger, Zipkin, AWS X-Ray…
• Metrics: Victoria Metrics, InfluxDB, Datadog…
• Logs: Promtail, Fluentd, Elastic Logstash…

The configuration provided in this guide can be adapted to work with these alternatives by modifying the collector's exporter configuration.

Setting up the OpenTelemetry Java Agent

To configure the Java OpenTelemetry agent to run along with our VDP server, we will include the following Java properties in the java.env.DENODO_OPTS_START property in the <DENODO_HOME>/conf/VDBConfiguration.properties file:

• -Dotel.service.name\=<arbitrary_service_name>

• Defines the name of your service in the observability system

• -Dotel.service.version\=<arbitrary_version_value>

• Specifies the version of your service, e.g. 1.0

• -Dotel.deployment.environment\=<deployment_environment>

• Indicates the deployment environment (development, production, etc.) which helps in filtering telemetry data by environment

• -javaagent:<path/to/opentelemetry-javaagent.jar>

• Specifies the path to the OpenTelemetry Java agent

• -Dotel.exporter.otlp.endpoint\=<endpoint_for_sending_data>

• Defines the endpoint where the OpenTelemetry data will be sent e.g. http://collector:5555

• -Dotel.logs.exporter\=otlp

• Configures the logs to be exported using the OTLP (OpenTelemetry Protocol) format, that is the standardized data format/protocol OpenTelemetry uses
• Note that you do not have to explicitly set the exporter for metrics and traces when using OTLP

• -Dotel.exporter.otlp.logs.endpoint\=<endpoint_for_sending_data>

• Defines the endpoint where the OpenTelemetry logs will be sent

• -Dotel.jmx.config\=<path/to/JMX/config/file/DenodoRules.yaml>

• contains rules for collecting specific JMX metrics from your Denodo VDP instance. Continue reading to see an example.

Sample configuration of the Java agent:

-Dotel.service.name\=DockerVDP -Dotel.service.version\=1.0 -Dotel.deployment.environment\=development -javaagent:/opt/tmp/observability/javaagent/opentelemetry-javaagent.jar -Dotel.exporter.otlp.endpoint\=http://collector:5555 -Dotel.logs.exporter\=otlp -Dotel.exporter.otlp.logs.endpoint\=http://collector:5555 -Dotel.jmx.config\=/opt/tmp/observability/javaagent/DenodoRules.yaml

JMX is  a Java technology that provides tools for monitoring and managing Java applications. In Denodo it is used to expose key metrics such as the number of connections or the memory usage that you can collect and monitor.

This YAML configuration file maps Denodo's JMX metrics (exposed through MBeans) to OpenTelemetry metrics, specifying which server attributes to monitor and how they should be represented in your observability system.

MBeans (Management Beans) are Java objects that provide a standardized way to monitor Java applications by exposing their metrics and operational data through a common interface.

Each mapping defines how a specific JMX attribute (like active connections or memory usage) should be collected and labeled in your monitoring platform.

Below you will find an example YAML file and an explanation of its content:

rules:   - bean: com.denodo.vdb.management.mbeans:type=DataSourceManagementInfo,name=GlobalDataSourceManagementInfo     mapping:       TotalActiveRequestsCount:         metric: globaldatasourcemanagementinfo.totalactiverequestscount         type: updowncounter         desc: The total number of active requests         unit: "1"

bean: The full name of the MBean in Denodo

• Format: package:type=Name,name=SpecificName
• Example: com.denodo.vdb.management.mbeans:type=DataSourceManagementInfo,name=GlobalDataSourceManagementInfo
• This MBean monitors global data source activity across Denodo

mapping: Under this, you list the attributes of the MBean you want to collect. Each name must match the exact MBean attribute name, for instance:

• TotalActiveRequestsCount: Tracks all active requests.
• DFActiveRequestsCount: Tracks DataFrame requests.
• JDBCActiveRequestsCount: Tracks JDBC data source connection requests.
• JSONActiveRequestsCount: Tracks JSON data source requests.

For each attribute, you define:

• metric: The name given in your monitoring system (e.g., globaldatasourcemanagementinfo.totalactiverequestscount).
• type: Set to updowncounter because request counts can increase or decrease.

• Other types:

• gauge: A value that can go up and down (like current connections).
• counter: A value that only increases (like total failures).

• desc: A description of what is being measured.
• unit: Set to "1" because we are counting individual requests.

For more details see

• Available Denodo MBeans: Monitoring Denodo servers with JMX
• OpenTelemetry JMX mapping configuration: JMX Metric Insight

Quickstart Docker Compose

In this section we will provide and explain a quickstart Docker Compose template you can use to implement the architecture above.

• OpenTelemtetry Javaagent Setup

• Prepare the javaagent folder structure:

```

/path/to/OpenTelemetry/javaagent/folder/

├── opentelemetry-javaagent.jar

└── DenodoRules.yaml                        # Add this folder to the javaagent folder

 ```

• Docker-compose.yaml

• Setup the following file structure:

                project_root/

├── docker-compose.yml

├── collector/

│ └── collector-config-local.yaml           # OTLP config & routing

├── prometheus/

│ └── prometheus.yaml                  # Metrics scraping config

├── tempo/

│ ├── tempo.yaml                 # Traces storage config

│ └── tempo-data/                # Volume for trace data

├── loki/

│ ├── loki-config.yml                 # Log aggregation config

│ └── loki-data/                         # Volume for log data

• This Docker Compose sets up Denodo with a complete observability pipeline using OpenTelemetry, where the Collector receives and routes Denodo's telemetry data to Tempo (traces), Prometheus (metrics), and Loki (logs) for monitoring and troubleshooting.

• Denodo Server:

• Runs Denodo with OpenTelemetry Java Agent instrumentation.
• Specifies ports and port mapping.
• Requires a license file mount and a Denodo container image.

• OpenTelemetry Collector:

• Ports: 5555 (OTLP in), 6666 (Prometheus metrics).
• Depends on Tempo, Prometheus and Loki being healthy.

• Temp:

• Ports: 3200 (HTTP API), 4317 (OTLP) .
• Local storage in ./tempo/tempo-data.
• Includes health check configuration.

• Prometheus:

• Port: 9091 (remapped from 9090 since 9090 is in use by Denodo).
• Includes health check configuration.

• Loki:

• Port: 3100 for log ingestion.
• Local storage in ./loki/loki-data.
• Includes health check configuration.

        

services:   vdpserver_opentelemetry:     image: <denodo_container_image>         container_name: vdpserver_telemetry     hostname: vdpserver_telemetry     networks:       - denodo_opentelemetry     ports:       - "9995-9999:9995-9999"       - "9090:9090"       - "9443:9443"     volumes:       - "<path/to/dendo/license.lic>:/opt/denodo/conf/denodo.lic"       - "<path/to/OpenTelemetry/javaagent/folder>:/opt/tmp/observability/javaagent"     environment:       DENODO_REGISTRY_HOST: "vdpserver"       DENODO_WEBCONTAINER_STARTUP: "true"     # Required for OpenTelemetry Agent to work       DENODO_JVM_OPTS: "-Dotel.service.name=DockerVDP -Dotel.service.version=1.0 -Dotel.deployment.environment=development -javaagent:/opt/tmp/observability/javaagent/opentelemetry-javaagent.jar -Dotel.exporter.otlp.endpoint=http://collector:5555 -Dotel.logs.exporter=otlp -Dotel.exporter.otlp.logs.endpoint=http://collector:5555 -Dotel.jmx.config=/opt/tmp/observability/javaagent/DenodoRules.yaml       DENODO_DEBUG: true     command: ["--vqlserver","--designstudio", "--datacatalog"]     collector:     #image: otel/opentelemetry-collector:latest     image: otel/opentelemetry-collector-contrib:0.97.0     container_name: collector     hostname: collector     networks:       - denodo_opentelemetry     restart: always     depends_on:       tempo:         condition: service_healthy       prometheus:         condition: service_healthy       loki:         condition: service_healthy     command: ["--config=/etc/collector-config.yaml"]     volumes:       - ./collector/collector-config-local.yaml:/etc/collector-config.yaml     ports:       - "5555:5555"       - "6666:6666"   tempo:     image: grafana/tempo:latest     command: ["-config.file=/etc/tempo.yaml" ]     container_name: tempo     hostname: tempo     networks:       - denodo_opentelemetry     restart: always     volumes:       - ./tempo/tempo.yaml:/etc/tempo.yaml       - ./tempo/tempo-data:/tmp2     environment:       TMPDIR: /tmp2     ports:       - "3200:3200"       - "4317:4317"     healthcheck:       interval: 5s       retries: 10       test: wget --no-verbose --tries=1 --spider http://localhost:3200/status || exit 1   prometheus:     image: prom/prometheus:v2.39.2     container_name: prometheus     hostname: prometheus     networks:       - denodo_opentelemetry      restart: always     command:       - --config.file=/etc/prometheus.yaml       - --web.enable-remote-write-receiver       - --enable-feature=exemplar-storage     volumes:       - ./prometheus/prometheus.yaml:/etc/prometheus.yaml     ports:       - "9091:9090"    #frorwarding 9091 to 9090, since 9090 is already used be denodo     healthcheck:       interval: 5s       retries: 10       test: wget --no-verbose --tries=1 --spider http://localhost:9090/status || exit 1 # ##################################################################################################################   loki:     image: grafana/loki:latest     container_name: loki     hostname: loki     networks:       - denodo_opentelemetry     restart: always     #restart: unless-stopped     volumes:        - ./loki/loki-config.yml:/mnt/config/loki-config.yml        - ./loki/loki-data:/loki     ports:        - 3100:3100     command:         - '-config.file=/mnt/config/loki-config.yml'     healthcheck: test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:3100/ready || exit 1"]       interval: 10s       timeout: 5s       retries: 5   grafana:     container_name: grafana-demo     hostname: grafana-demo     networks:       - denodo_opentelemetry      image: grafana/grafana     volumes:       - ./grafana/grafana-datasources.yml:/etc/grafana/provisioning/datasources/datasources.yml     restart: always     ports:       - "3000:3000" networks:     denodo_opentelemetry:         name: denodo_opentelemetry         driver: bridge

Configuration Files Setup

• Collector Configuration (./collector/collector-config-local.yaml):

• OTLP receiver on port 5555 for incoming telemetry.
• Routes data to appropriate backends:

• Traces → Tempo
• Metrics → Prometheus
• Logs → Loki

• Uses batch processing with 1 second timeout and 1024 batch size.
• Separate pipelines for metrics, traces, and logs.

receivers:   otlp:     protocols:       grpc:         endpoint: 0.0.0.0:5555 processors:   batch:     timeout: 1s     send_batch_size: 1024 exporters:   prometheus:     endpoint: collector:6666     namespace: denodo   otlp/tempo:     endpoint: tempo:4317     tls:       insecure: true   loki:     endpoint: http://loki:3100/loki/api/v1/push   otlp/jaeger:     endpoint: "http://jaeger:4317"     tls:       insecure: true service:   pipelines:     metrics:       receivers: [otlp]       processors: [batch]       exporters: [prometheus]     traces:       receivers: [otlp]       processors: [batch]       exporters: [otlp/jaeger,otlp/tempo]     logs:       receivers: [otlp]       exporters: [loki]

•  Tempo Configuration (./tempo/tempo.yaml):

• Configures Tempo trace storage.
• Defines retention settings.
• Sets up receivers and search capabilities.
  

stream_over_http_enabled: true server:   http_listen_port: 3200   log_level: info query_frontend:   search:     duration_slo: 5s     throughput_bytes_slo: 1.073741824e+09   trace_by_id:     duration_slo: 5s distributor:   receivers:                           # this configuration will listen on all ports and protocols that tempo is capable of.     jaeger:                            # the receives all come from the OpenTelemetry collector.  more configuration information can       protocols:                       # be found there: https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver         thrift_http:                   #         grpc:                          # for a production deployment you should only enable the receivers you need!         thrift_binary:         thrift_compact:     zipkin:     otlp:       protocols:         http:         grpc:     opencensus: ingester:   max_block_duration: 5m               # cut the headblock when this much time passes. this is being set for demo purposes and should probably be left alone normally compactor:   compaction:     block_retention: 1h                # overall Tempo trace retention. set for demo purposes metrics_generator:   registry:     external_labels:       source: tempo       cluster: docker-compose   storage:     path: /tmp/tempo/generator/wal     remote_write:       - url: http://prometheus:9090/api/v1/write         send_exemplars: true storage:   trace:     backend: local                     # backend configuration to use     wal:       path: /tmp/tempo/wal             # where to store the the wal locally     local:       path: /tmp/tempo/blocks overrides:   defaults:     metrics_generator:       processors: [service-graphs, span-metrics] # enables metrics generator

• Prometheus Configuration (./prometheus/prometheus.yaml):

• Scrapes metrics from collector:6666 every 5 seconds.
• Matches the collector's Prometheus exporter endpoint.
• Global evaluation interval: 15 seconds.

global:   scrape_interval: 15s   evaluation_interval: 15s scrape_configs:   - job_name: 'collector'     scrape_interval: 5s     static_configs:       - targets: [ 'collector:6666' ]

• Loki Configuration (./loki/loki-config.yml):

• Receives logs on port 3100 via /loki/api/v1/push.
• Uses filesystem storage with TSDB schema.
• 100MB embedded cache.
• Auth disabled for development setup.

auth_enabled: false server:   http_listen_port: 3100   grpc_listen_port: 9096 common:   instance_addr: 127.0.0.1   path_prefix: /tmp/loki   storage:     filesystem:       chunks_directory: /tmp/loki/chunks       rules_directory: /tmp/loki/rules   replication_factor: 1   ring:     kvstore:       store: inmemory query_range:   results_cache:     cache:       embedded_cache:         enabled: true         max_size_mb: 100 schema_config:   configs:     - from: 2020-10-24       store: tsdb       #changed from boltdb-shipper to the newer tsdb       object_store: filesystem       schema: v13       #changed from v11 to v13       index:         prefix: index_         period: 24h ruler:   alertmanager_url: http://localhost:9093 # By default, Loki will send anonymous, but uniquely-identifiable usage and configuration # analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/ # # Statistics help us better understand how Loki is used, and they show us performance # levels for most users. This helps us prioritize features and documentation. # For more information on what's sent, look at # https://github.com/grafana/loki/blob/main/pkg/usagestats/stats.go # Refer to the buildReport method to see what goes into a report. # # If you would like to disable reporting, uncomment the following lines: #analytics: #  reporting_enabled: false

• Grafana (./grafana/):

• Contains the data source configuration to connect to the observability backends.
• Note: You can also manually adjust the data source configuration in the Grafana web tool.

apiVersion: 1 datasources:   - name: Prometheus     type: prometheus     uid: prometheus     access: proxy     orgId: 1     url: http://prometheus:9090     basicAuth: false     isDefault: false     version: 1     editable: true     jsonData:       httpMethod: GET       exemplarTraceIdDestinations:         - datasourceUid: tempo           name: trace_id   - name: Loki     type: loki     uid: loki     access: proxy     orgId: 1     url: http://loki:3100     basicAuth: false     isDefault: false     version: 1     editable: true   - name: Tempo     type: tempo     access: proxy     orgId: 1     url: http://tempo:3200     basicAuth: false     isDefault: true     version: 1     editable: true     apiVersion: 1     uid: tempo     jsonData:       httpMethod: GET       serviceMap:         datasourceUid: prometheus       tracesToLogsV2:         datasourceUid: loki         spanStartTimeShift: '-1h'         spanEndTimeShift: '1h'         filterByTraceID: true         filterBySpanID: true         tags: [ { key: 'service.name', value: 'job' } ]

Starting the services

From your project root directory you can now use the docker compose up command to start all the previously described services.

Connecting to Grafana

You can now connect to Grafana using your Grafana host address and the (default) port 3000 e.g. localhost:3000 where you will be prompted to login. You can use the default admin:admin credentials initially and you will be prompted to change the password.

Under Home > Connections > Data sources you can see our data sources:

NOTE: In order for the Denodo server to be monitored by tools using the JMX protocol, the VDP host name has to be set Set the Host Name of Denodo Servers.

References

Monitoring Denodo servers with JMX

JMX Metric Insight