SKIP TO MAIN CONTENT
Sign In
Register
What's New in Denodo Platform
Expert Trails
Materialization (pka Cache)
Datasource Connectivity
Deployment Topology
Development Lifecycle
Modeling and Metadata Organization
Monitoring
Operations
Query Performance Optimization
Security
Considerations for Designing a Self-Service Analytics Initiative
Enterprise Data Services with APIs
Resources
User Manuals
Knowledge Base
Videos
Tutorials
Browse Content Library
Drivers
Data Management Blog
Technical Newsletter
Try Denodo
Denodo Platform - Developer Tier
Test Drives
Agile BI and Analytics on AWS
Data Catalog and Data Science on AWS
Learn
Training Courses
Learning Paths
Course Catalog
Your Progress
Workshops
Certification Exams
Academic Program
Overview
University Challenge
Collaborate
Q&A Forum
All Questions
Ask a Question
Search by tag
Denodo Sharing
User Groups
Support
Support Site
Contact Us
USER MANUALS
Denodo Platform 9.4
Denodo 9.5
Denodo 9.4
Denodo 9.3
Denodo 9.2
Denodo 9.1
Denodo 9.0
English
日本語
Denodo Platform Installation Guide
Legal Notes
Third-Party Libraries Distributed with the Denodo Platform 9.4.0
Third-Party Libraries Distributed with the Denodo Platform 9.3.0
Third-Party Libraries Distributed with the Denodo Platform 9.2.0
Third-Party Libraries Distributed with the Denodo Platform 9.1.0
Third-Party Java Libraries Distributed with the Denodo Platform 9.0.0
Third-Party Non-Java Libraries Distributed with the Denodo Platform 9.0.0
Preface
Scope
Who Should Use This Document
Summary of Contents
Introduction
Pre-Installation Tasks
Hardware Requirements
Minimum Requirements to Run the Denodo Platform On Premise
Minimum Requirements to Run the Denodo Platform on Amazon AWS
Minimum Requirements to Run the Denodo Platform on Microsoft Azure
Minimum Hardware Requirements when Installing each Module Separately
Considerations Regarding the Disk Space Requirements of the Denodo Platform
Running the Denodo Platform on Docker
Requirements for the Virtual DataPort Administration Tool
Software Requirements
Supported Operating Systems
Supported Browsers
Configuring the Server for the Denodo Platform
Select a User Account to Install the Denodo Platform
Grant Privileges to This User Account
Check the PATH Environment Variable (Windows)
Check that the Required Ports Are Free
Download the Installer
Using the Graphical Installation Wizard
Settings you can Modify when you Select the Custom Installation
Common to All Modules
Virtual DataPort
Scheduler Server
Scheduler Index Server
Embedded Web Container
Using the Command Line Installer
Unattended Installation of the Denodo Platform
Modifying the Denodo Platform Installer to Include the Latest Update
Post-Installation Tasks
Replicate the Encryption Key Across All the Installations
If You Already Started One of the Components of the Denodo Platform / Solution Manager
Install the Latest Update
Register the Denodo Servers in the License Manager
Enable SSL/TLS in the Denodo Platform
Obtaining and Installing an SSL/TLS Certificate
Denodo SSL/TLS Configurator Script
Enabling SSL/TLS in Denodo Platform Servers
Enabling HTTPS in the Embedded Apache Tomcat
Enabling SSL/TLS in the Administration Tool and Others
Enabling SSL/TLS for External Clients
SSL/TLS Versions Supported by the Denodo Platform Servers
Change the Default Passwords
Virtual DataPort
Design Studio
Data Marketplace
Scheduler
Set the Host Name of Denodo Servers
Virtual DataPort
Scheduler Server
Scheduler Index Server
Configure Windows Services
Upgrading From a Previous Version and Other Tasks
Installing Updates, Bugfixes and Extra-Features
Installing an Update Graphically
Installing an Update From the Command Line
Troubleshooting the Updates Installer
Enable Logging in the Installer
Skip Updating the Java Runtime Environment
Using feature “Credentials vault” with CyberArk
Denodo Platform Control Center
Launching the Denodo Platform Control Center
Starting Platform Servers and Tools
Denodo Platform Configuration
Configuring the Connection to the License Manager
Configuration of the JVM Parameters From the Command Line
Control Center Help
Uninstalling the Denodo Platform
Appendix
Supported Java Development Kits (JDK)
Troubleshooting the Denodo Platform Installer
Denodo Platform Upgrade Guide
Preface
Introduction to Upgrading the Denodo Platform
Major Steps in the Upgrade Process
Running Both Versions in the Same Computer
Upgrading From Denodo Platform 7.0 or Earlier
Upgrading From Denodo Platform 7.0, 6.0 and 5.5
Upgrading From Denodo Platform 5.0 or Earlier
Preparing to Upgrade
Common to All Modules
Kerberos
Virtual DataPort
Virtual DataPort: Select a Catalog/Schema for the Cache Engine
Virtual DataPort: Create a Repository for Version Control
Data Marketplace: Create a Catalog/Schema on the External Database
Scheduler: Create a Catalog/Schema on the External Database
Solution Manager
Solution Manager: Create a Git Repository or Branch for Version Control
Solution Manager: Create a Catalog/Schema on the External Database
Pre-Upgrade Tasks for the New Installation
Common to All Modules
Copy the Encryption Key
Install the Latest Update
Enable SSL
Configure Windows Services
Kerberos: Copy Krb5 File
Virtual DataPort
Virtual DataPort: Enable the External Database
Virtual DataPort: JDBC Data Sources that Use the Bulk Data Load API
Virtual DataPort: Enable the Cache Engine
Virtual DataPort: Install Third-Party Connectors
Virtual DataPort: Other Third-Party Jar Files
Virtual DataPort: Version Control System
Virtual DataPort: Delete Unused Objects
Virtual DataPort: Delete Aracne and Google Search Data Sources
Virtual DataPort - Published Web Services: Customize Logo
Virtual DataPort - Denodo Monitor: Configure to Store the Information in a Database
Data Marketplace: Enable the External Database
Scheduler: Enable the External Database
Solution Manager
Solution Manager: Enable the External Database
Solution Manager: High Availability
Solution Manager: Enable Git Support
Solution Manager: Configure Monitoring JDBC Appenders
Upgrading From Denodo 6.0: Export the Settings and Metadata of the Current Installation
Transforming Views Automatically to Use the New Datetime Data Types
Configuration Properties of Virtual DataPort for Upgrades: “exportMigrationCompatibility” and “exportOnlyResources”
Alternatives for When You Cannot Install the Latest Update of Denodo 7.0
Upgrading From Denodo 7.0: Export the Settings and Metadata of the Current Installation
Configuration Properties of Virtual DataPort for Upgrades: “exportMigrationCompatibility” and “exportOnlyResources”
Alternatives for When You Cannot Install the Latest Update of Denodo 7.0
Upgrading From Denodo 8.0: Export the Settings and Metadata of the Current Installation
Configuration Properties of Virtual DataPort for Upgrades: “exportMigrationCompatibility” and “exportOnlyResources”
Alternatives for When You Cannot Install the Update 20240306 or Later of Denodo 8.0
Review the Virtual DataPort Metadata
Import Settings and Metadata in to the New Installation
Common Errors when Importing the Metadata of Virtual DataPort to the New Version
Post-Upgrade Tasks for the New Installation
Common to All Components
Configure the Connection to the Denodo Security Token Server
Review Memory Allocation In Production Servers
Virtual DataPort
Increase the Memory Allocated to the Tomcat of Denodo Platform
Design Studio
Virtual DataPort Administration Tool
Embedded MPP
Data Marketplace
Scheduler
Scheduler Index
Solution Manager
External Applications: Upgrade the JDBC and ODBC Drivers
Test the New Denodo Platform Installation
Backward Compatibility
Changes in Denodo Platform 9
Features Removed From Denodo Platform 9
Changes that Apply to All the Modules of the Denodo Platform and Solution Manager
Virtual DataPort
Design Studio
Scheduler
Solution Manager
Diagnostic & Monitoring Tool
User Manuals
Changes in Denodo Platform 8.0 GA
Changes Common to All the Modules of Denodo 8.0 GA
Changes in Virtual DataPort 8.0 GA
Changes in Solution Manager 8.0 GA
Features Removed From Aracne 8.0 GA
Features Removed From Virtual DataPort 8.0 GA
Features Removed From Scheduler 8.0 GA
Features Removed From Solution Manager 8.0 GA
Changes in Denodo Platform 7.0
Changes Common to All the Modules of Denodo 7.0
Changes in Virtual DataPort 7.0
Changes in the Privileges System in Virtual DataPort 7.0
Features Removed From Virtual DataPort 7.0
Changes Introduced in Aracne 7.0
Changes in Denodo Platform 6.0
Changes Common to All the Modules of Denodo 6.0
Changes in Virtual DataPort 6.0
Changes in Scheduler 6.0
Changes in the Embedded Web Container of Denodo 6.0
Changes in Denodo Platform 5.5
Changes in Virtual DataPort 5.5
Features Removed in Denodo 5.5
Changes in Denodo Platform 5.0
Changes in Virtual DataPort 5.0
Features Deprecated in Denodo Platform
Virtual DataPort
Administration Tool
Data Type “Date”
Materialization (Cache)
VQL Syntax
Data Sources
Denodo Stored Procedures
Role JMXAdmin
Denodo Stored Procedures API: getNumOfAffectedRows Method
Denodo Custom Wrappers API: Deprecated Methods
Denodo JDBC Driver
Denodo Web Services
Version Control Systems (VCS)
Script Export: Deprecated Parameters
Launching Several Virtual DataPort Instances of the Same Installation
Resource Manager
Privilege FILE
Data Marketplace
Roles “selfserviceadmin”, “selfserviceexporter”
Scheduler
Scheduler
Private Scheduler REST API
Solution Manager
REST API: Authentication Using Session
Denodo Platform New Features Guide
Preface
Audience
What Is New in Denodo 9.4
What Is New in Denodo 9.3
What Is New in Denodo 9.2
What Is New in Denodo 9.1
Denodo 9 Highlights
New Features of the Denodo Platform 9
Denodo Platform Administration Guide
Preface
Audience
Adding AWS RDS Certificate to JRE cacerts
Audit Trail
Audit Information
How to Configure Audit in Scheduler
How to Configure Audit in Data Marketplace
How to Configure Audit in Solution Manager
Masking Sensitive Information
Default Ports Used by the Denodo Platform Modules
Denodo Assistant Features
Denodo Assistant Features That Use Large Language Models
Denodo Assistant View Features
Denodo Assistant for VQL Shell
Features
Data Sent
Privileges Required to Use These Features
Other Denodo Assistant Features
Denodo Platform - Subscription Bundles
Denodo Enterprise Plus
AI & Denodo Assistant FeaturePack
Semantics FeaturePack
Embedded Massive Parallel Processing (MPP) FeaturePack
Denodo Security Token Configuration File
Common Settings
Token Signing Credentials
Authentication Modules
SAML 2.0 Delegation
OAuth 2.0 Delegation
OpenID Connect Delegation
Enable Authentication in the Monitoring Interface of the Web Container
Enabling Kerberos Authentication Without Joining a Kerberos Realm
Health Monitoring
How to Debug Kerberos in Web Applications
Virtual DataPort Server and the Web Administration Tools Are on the Same Installation
Virtual DataPort Server and the Web Administration Tools Are not on the Same Installation
Increasing the Maximum Simultaneous Requests
Installation Encryption Key
Configure Keystore Password Loader
Direct Keystore Password Configuration
File Keystore Password Configuration
Environment Variable Keystore Password Configuration
AWS Keystore Password Configuration
Azure Keystore Password Configuration
Google Cloud Keystore Password Configuration
Keystore Management
Generating a New Keystore with an Encryption Key and its Configuration
Update the Keystore Password
Reset the Encryption Key
Language of Data Marketplace, Design Studio and Scheduler
Changing the Language of the Applications
Creating Your Own Language Files
Considerations When Translating the Language Files
How to Refresh Custom Messages After Installing an Update
Password Policies in the Denodo Platform and Solution Manager
Policies that Are Disabled by Default in Virtual DataPort and Solution Manager
Password Expiration
Lock Account After Many Attempts to Log in with the Wrong Password
When All the Administrators are Locked Out of Virtual DataPort or Solution Manager
Important Notes
Prevent Load Balancers From Closing Connections of Data Marketplace and Design Studio
Enabling this Feature in Data Marketplace
Enabling this Feature in Design Studio
Providing a Krb5 File for Kerberos Authentication
Restrictions of Denodo Express
Reverse Proxy Settings (Design Studio, Data Marketplace, Scheduler)
Reverse Proxy Configuration
Server Process Identifiers
Session Configuration (Data Marketplace, Design Studio, Diagnostic & Monitoring Tool, Scheduler and Solution Manager Administration Tool)
Idle Session Timeout
Maximum Session Timeout
Single-User Concurrent Sessions Control
Setting-up Kerberos Authentication
Configuring the Kerberos Authentication Server (e.g. Active Directory)
Creating a User in the Active Directory
Declaring a Service Principal Name (SPN)
Enable Kerberos Delegation for this User Account
Generating a Keytab File
Distribute the Keytab and Enable Kerberos Authentication on Denodo Servers
Setting-up Kerberos Authentication in Scheduler
Scheduler and Virtual DataPort Are in the Same Machine
Scheduler and Virtual DataPort Are in Different Machines
Setting-up Kerberos Authentication in the Data Marketplace
The Data Marketplace and the Virtual DataPort Server Are in Different Machines
The Data Marketplace and the Virtual DataPort Server Are in the Same Machine
Transparent Metadata Encryption
Virtual DataPort Administration Guide
Preface
Scope
Who Should Use This Document
Summary of Contents
Introduction
General Architecture
Physical Layer
Logical Layer
Data Module: Cache
User Layer
Starting Virtual DataPort
Starting the Virtual DataPort Server
Starting Design Studio
Main Areas of Design Studio
Design Studio Preferences
Using the Design Studio VQL Shell
Design Studio Query Monitor
Design Studio Catalog Search
Design Studio Invalidate Cache
Design Studio Trace Viewer
Launching the Administration Tool
Main Areas of the Administration Tool
Tool Preferences
Using the VQL Shell
Query Monitor
Catalog Search
Invalidate Cache
Trace Viewer
Creating Data Sources and Base Views
Database Sources (SQL)
Preparing the Connection to the Database
Importing JDBC and ODBC Sources
ODBC Drivers on Linux
Database Specific Information
Connecting to a Database (JDBC) with Kerberos Authentication
Connecting to a JDBC Source with OAuth Authentication
Creating Base Views From a JDBC Data Source
Importing Graphically Stored Procedures From a Database
Creating Base Views From SQL Queries
SOAP Web Service Sources
Importing SOAP Web Service Sources
Creating Base Views From a SOAP Web Service
Multidimensional Database Sources
Creating a Base View Over a Multidimensional Data Source, Graphically
Creating a Base View Over a SAP Bex Query (only for SAP BW)
Creating a Base View Over a Multidimensional Data Source with an MDX Query
Multidimensional Data Sources to Oracle Essbase
Multidimensional Data Sources Push Down Capabilities
Path Types in Virtual DataPort
Local Path
Path From Variable
HTTP Path
FTP / SFTP / FTPS Path
HDFS Path
S3 Path
Azure Blob Storage and ADLS Gen2 (ABFS)
Hadoop-Compatible Storage
Compressed or Encrypted Data Sources
Custom Input Filters
Paths and Other Values with Interpolation Variables
XML Sources
Direct
From OpenAPI 3 Document
JSON Sources
Direct
From OpenAPI 3 Document
Delimited File Sources
Examples of How to Define a Tuple Pattern
Paths Using Date Ranges
Excel Sources
Size Limit When Processing Excel Files
Reading an Excel File with Multiple File Entries
Use Temporary Files to Load Excel Files
Fields Detected as Text During Introspection with Another Format
LDAP Sources
BAPI Sources
Salesforce Sources
Registering Denodo as a Connected Application
Creating a Salesforce Data Source
Creating Base Views From a Salesforce Data Source
Creating Base Views From SOQL Queries
Binary Fields
Limitations of Salesforce
Additional Settings
Bulk Mode
MongoDB and DocumentDB Sources
Creating a MongoDB Data Source
Creating Base Views From a MongoDB Data Source
MongoDB Aggregation Pipelines
Delegation Capabilities and Limitations
Views When some Field has Different Data Types in the Same Collection
Differences between a MongoDB Data Source and a DocumentDB Data Source
Object Storage Data in Open Table Formats (Parquet, Iceberg and Delta Lake)
Object Storage Routes vs External Catalogs
Create base views from data in an object storage considering schema evolution
Manage Views Created From Object Storage
How to Connect to a Kerberized HDFS From the Denodo Lakehouse Accelerator Data Source
Creating Base Views for
Parquet
,
Iceberg
, and
Delta Lake
Datasets
Create base views from Parquet tables using
Hive-style partitioning
Accessing data in Delta Lake format
OData 2 Sources
Create OData v2 Data Sources
Create OData v2 Base Views
Connecting to the SAP OData API
Examples
Known Limitations
Troubleshooting
References
Appendix
OData 4 Sources
Create OData v4 Data Sources
Create OData v4 Base Views
Examples
Known Limitations
Dynamics Endpoint Limitations
Connecting to the SAP OData API
References
Custom Sources
Using Interpolation Variables as Custom Sources’ Input Parameters
Software as a Service Wizards
Adobe Analytics
Dynamics 365 Business Central
Dynamics 365 Customer Engagement
Google Analytics 4
Google Analytics Migration
Marketo
Marketo Bulk Extract
ServiceNow
SharePoint
SharePoint For OData
SharePoint REST API
Viewing the Schema of a Base View
View the VQL of an Element
Used By…
Source Type Properties
Data Source Configuration Properties
Source Refresh
Working with Blob Fields of Base Views
Creating Derived Views
Creating Union Views
Creating Extended Union Views
Creating Standard Union Views
Creating Partitioned Union Views
Creating Join Views
Join Conditions with Similarity Operators
Creating Selection Views
Creating Conditions with the Compound Values Editor
Parameters of Derived Views
Creating Flatten Views
Creating Intersection Views
Creating Minus Views
Creating Interface Views
Viewing the Schema of a Derived View
Tree View
Obtaining This Information Programmatically
Data Lineage
Advanced Configuration of Views
Vendor-specific capabilities for schema evolution
Configuring the Materialization of a View (pka Cache)
Search Methods, Configuration Properties and Internationalization
Indexes of Views
Statistics of the View
Defining the Data Movements of the View
Materialized Tables
Temporary Tables
Querying Views
Execution Trace of a Statement
Editing / Replacing a View
Modifying Base Views
Modifying Derived Views
Views Affected by Modifications
Stored Procedures
Importing Java Stored Procedures
Executing Stored Procedures
Use of Stored Procedures in Creating Views
Publication of Web Services
Publishing SOAP Web Services
Operations Tab
Settings Tab (SOAP)
Advanced Tab (SOAP)
Publishing REST Web Services
Resources Tab
Settings Tab (REST)
Advanced Tab (REST)
XSL Transformations
Web Services Authentication
HTTP Basic
HTTP Basic with VDP
HTTP Digest
HTTP SPNEGO (Kerberos)
OAuth 2.0 and OpenID
SAML 2.0
Impersonating a User
How Web Services Query the Virtual DataPort Server
Connection From the Web Services to the Server
Web Service Container Status Table
Invoking Denodo Web Services
Invoking Web Services with SAML Authentication
Types Conversion Table for REST / SOAP Published Web Services
Privileges Required
RESTful Architecture
Primary Keys of Views
Associations
Creating an Association
Multiplicity of Associations
Referential Integrity in Associations
Role Preconditions
Why You Should Define Associations Between Views
RESTful Web Service
Representations
Input Parameters of the RESTful Web Service
IDU Requests
Requests with Input Compound Values
Status Codes
Obtaining the Number of Rows of a Result Set
Security and Access Privileges
Configuring the RESTful Web Service
Denodo OData 4.0 Service
Serving Metadata
Querying Data: The Basics
Navigating Associations
Advanced Querying
Pagination
Configuration
Entity Collection Streaming
OpenAPI Specification
Debug Option
Privilege Requirements
Limitations
Features
Denodo GraphQL Service
Configuration
Privilege Requirements
Schema
Introspection
Basic Querying
GraphQL Names
Advanced Querying
Authentication
Resource Limitations
Limitations
Features
JMS Listeners
Creating JMS Listeners
Privileges Required
Advanced JMS Configuration
Acknowledge Mode
Acknowledge On Query Finish and Serialized Processing
Enabling Advanced Logging for JMS Listeners
Kafka Listeners
Creating Kafka Listeners
Privileges Required
Advanced Kafka Configuration
Remote Tables
Managing Remote Tables
Creating Remote Tables
Editing Remote Tables
Refresh Data
Dropping Remote Tables
Customizing the Table Creation Template
Create Parquet or Iceberg tables on an Object Storage like HDFS, S3 or ADLS
Remote Tables vs Other Mechanisms for Persisting Data
Server Configuration
Configuring the Cache
Limiting the Number of Concurrent Requests
Credentials Vault
Enabling Access to the Credentials Vault
Supported Credentials Vaults
How a Data Source Obtains the Credentials From the Vault
Specify a Pattern as Secret Name, Parameterized by the Login
Configuring the Default Internationalization
Default Configuration of HTTP Proxy
Identifiers Charset
Restricted Mode
Unicode Mode
Configuring the Memory Usage and Swapping Policy
Maximum Size of a Query
Swapping Settings
Automatic Out Of Memory recovery
Privileges
Indirect Access Control
The FILE Privilege
Roles Privilege
Global Settings of REST Web Services
Enable/Disable the OpenAPI Endpoints
Accessing the REST Web Services Through a Reverse Proxy
Changing the HTTP Header “Content-type”
Changing the HTTP Header “WWW-Authenticate”
Obfuscating that the REST Web Services are Published by the Denodo Platform
Queries Optimization
Server Authentication
LDAP Authentication
Kerberos Authentication
SAML Authentication
OAuth Authentication
Denodo Security Token Authentication
Server Connectivity
Configuring Runtime Parameters for Stored Procedures
Threads Pool
Storing the Metadata on an External Database
Replication Process
Databases Supported
Considerations About This Feature
How to Configure Virtual DataPort to Store the Metadata on an External Database
Denodo Assistant
LLM Configuration
General
Vector Database Integration
Recommendations and Limitations for the Denodo Assistant Vector Database
Memory Management
Streaming Vs Non-Streaming Operators
Swapping Parameters
Limit the Maximum Amount of Memory of a Query
Edge Cases in Streaming Operation
Cache Load Processes
Resource Manager
Defining a Plan
Defining a Rule
Logging Resource Manager Actions
Exporting and Importing the Server Metadata
Exporting the Server Metadata
Exporting and Importing Elements Across Different Environments
Exporting Environment-Dependent and Independent Elements to Different Files
Export to a File with Properties
Importing Metadata into a Server
Importing Extensions
Importing a JDBC Driver
Importing a JDBC Driver for a Database that is not Listed
Importing Other Types of Elements
Using a Script to Upload Extensions
Cache Module
Materialization Types
Query Results Type
Full Mode
Incremental Mode
Recommended Parameters for Queries that Load the Cache
Caching Very Large Data Sets
Caching the Result of Queries that Fail
Cache Maintenance Task
Maintaining the Underlying Database
Specific Information about Cache Databases
Amazon Athena
Azure SQL Data Warehouse
Google BigQuery
Kudu
Microsoft Fabric Data Warehouse
PrestoDB/Trino
Denodo Lakehouse accelerator
Spark
Yellowbrick
Databases Using a Distributed Object Storage
Generic Support for Other Databases
Table Creation Templates Management
Creating or Editing the Table Creation Template On the Data Sources
Configure the Cache Table Creation Templates
Customize the Cache Management Tables Template
Creating or Editing the Table Creation Template On Views
Template Parameters
Internal Parameters List
Materialization Decision Guide: Comparison Matrix
Version Control Systems Integration
VCS Configuration
Virtual DataPort Server Configuration
Environment Management
Database Configuration
Importing an Existing Database From a VCS Server
Workspaces
VCS Integration Features
Integration with the Server Explorer
VCS Operations for GIT
Dependencies Management
VCS Settings of the Administration Tool
Scenarios and Recommended Uses
Picking a VCS Workflow
Centralized Workflow with Shared Databases
Distributed Workflow
Workflow Based on Workspaces and Branches
Centralized Workflow with Private Databases
Best Practices to Manage Global Elements
Promoting Changes From Development to Testing and Production
Using VCS Environments
Best Practices When Using the Integration with a VCS
Tagging and Branching Releases
Recommendations for the Testing Environment
Databases, Users and Access Rights in Virtual DataPort
Databases in Virtual DataPort
User and Access Rights in Virtual DataPort
Users
Roles
Types of Access Rights
Privileges at Database Level
Fine-Grained Privileges at View Level
Column Privileges
Row Restrictions
Global Security Policies
Tags Management
Global Security Policy
Custom View Policies
Dynamic Restrictions Policy
Modeling Fine-Grained Role Restrictions on Multiple Roles
Privileges are Cumulative, not Exclusionary
Data Available to a User with Roles R1,R2 is Equals to the Data Available to R1 + the Data Available to R2 in General but NOT Always
The execution engine only considers the privileges in roles with execution privilege on the query views
Limit Indirect Access to Views Shared with Other Development Teams
Limit Indirect Access to Shared Views Using Global Security Policies
Limit Indirect Access to Shared Views Using the Indirect Access Privilege
Administration of Databases, Users, Roles and Their Access Rights
Creating Databases
Configuring and Deleting Databases
Creating Users
Modifying and Deleting Users
Creating Roles
Privilege Auditing
Logging Details when Processing Privileges
Impersonation
Import Tags From External Catalog
Monitoring the Virtual DataPort Server
Monitoring with a Java Management Extensions (JMX) Agent
Using JavaTM VisualVM
General Information on the Server
Information on Data Sources
Information on the Cache
Information and Events on Catalog Access (DDL Statements)
Information and Events on the Running of Statements
Information and Events on Transactions
Setting the User Agent of an Application
Setting Cloud Infrastructure Properties
Denodo Monitor
Local Monitors
Server Monitors
Virtual DataPort Monitors
Configuring the Denodo Monitor
Launching the Denodo Monitor
FinOps
Cluster Architectures / Server Backup
Denodo Tools
How to Check If a Virtual DataPort Server Is Alive
Using the Ping Script
Alternatives to the Ping Script
Using the Import/Export Scripts for Backup And/or Replication
Export Script
Import Script
Data Lake Storage Management
CSV, JSON and XML File Formats
Apache Parquet File Format
Delta Lake Table Format
Iceberg Table Format
Time travel support
Troubleshooting problems accessing an object storage
Bulk Data Load
AlloyDB for PostgreSQL
Amazon Athena
Amazon Aurora MySQL
Amazon Aurora PostgreSQL
Amazon RDS for MySQL
Amazon RDS for PostgreSQL
Amazon Redshift
Amazon RDS for Microsoft SQL Server
Amazon RDS for Oracle
Apache Doris
Temporary File Generation on the Denodo Side
Loading Data into Apache Doris (LOAD DATA LOCAL INFILE)
ClickHouse
DB2
Databricks
Denodo Lakehouse Accelerator
Iceberg Tables
Configure the Denodo Server to Use the Denodo Lakehouse Accelerator as Cache Using Iceberg Tables
Create Iceberg Tables in the Denodo Lakehouse Accelerator
Bulk Data Loading into Iceberg Tables
Create Iceberg Tables in the Denodo Lakehouse Accelerator or Presto Data Source When the Catalog Name is not Iceberg
Troubleshooting
BigQuery
Hive
Huawei GaussDB
Impala
Data Load to Impala Using Built-in Implementation
Data Load to Impala Using External Hadoop Installation
Kudu
Steps Virtual DataPort Performs to Bulk Load Data into Kudu
MySQL
Oracle
Bulk data load with vectors
PostgreSQL
PrestoDB/Trino
Snowflake
Spark
Microsoft SQL Server and Azure Synapse SQL
Sybase ASE
SAP IQ / SYBASE IQ
Microsoft Fabric DataWarehouse
Teradata
Yellowbrick
Settings of the Generation of the Temporary Files
Generation of Temporary Files for a Hadoop-compatible Distributed Object Storage
Bulk Data Load on a Distributed Object Storage like HDFS, S3 or ADLS
Built-in Implementation
External Hadoop Installation
Optimizing Queries
Optimizing DF Data Sources
Optimizing Join Operations
Merge Join
Nested Join
Nested Parallel Join
Hash Join
Automatic Simplification of Queries
Removing Redundant Branches of Queries (Partitioned Unions)
Pushing Down GROUP BY Views Below JOIN Views
Pushing Down GROUP BY Views Below UNION Views
Selecting the Most Optimal Source When the Data Is Replicated in Several Sources
Cost-Based Optimization
Enabling the Cost-Based Optimization
Gathering the Statistics of Views
Tuning the Cost-Based Optimization Process
Current Limitations of the Cost-Based Optimization Process
When Should You Enable Automatic Cost-Based Optimization?
Data Movement
Data Movements From/To Netezza Databases
Examples of Data Movement
Known Limitations
Forcing the Data Movement of a View
Options of the CONTEXT Clause that Control a Data Movement
Customize the Data Movement Table Creation Command
Restrict Data Movement Optimizations From Sensitive Data Sources
Parallel Processing
Data movement
vs
MPP Movement
vs
Embedded MPP Acceleration
Embedded MPP Acceleration
Parallel Processing Using an External MPP
Use of an MPP Engine as Cache
Force Movement of Views to the MPP Engine Using the CONTEXT Clause
Window Functions
Execution of Window Functions
Smart Query Acceleration Using Summaries
Managing Summaries
Summary Promotions
Summary Rewrite Optimization
Automatic Summary Recommendations
Denodo Lakehouse Accelerator
Sizing Recommendations for the Denodo Lakehouse Accelerator
Sizing recommendations for each Denodo environment
Promoting the Denodo Lakehouse Accelerator
Denodo Lakehouse Accelerator Extended Execution Trace
MPP Route Plan
MPP Execution Plan
MPP Stage Plan
MPP Stage Plan Operation Nodes
Errors in the Extended Execution Trace
Monitoring the Denodo Lakehouse Accelerator
Denodo Lakehouse Accelerator Resource Management
Import Model
Overview
Data Modeling
RDFS and OWL Ontologies
Best Practices for Importing Models
Usage
Limitations
Tags Supported
Appendix
Avoiding SQL Injections
Backward Compatibility Between the Virtual DataPort Server and Its Clients
Configuring the Logging System
Obfuscation of Sensitive Information of Requests
Configuring the Network Interface Restriction in the Denodo Platform
Considerations When Configuring Data Sources with Pass-Through Credentials
Web Services
List of Data sources Supporting Authentication with Pass-Through
Data Sources That Support Merge Data
Using the Databricks API and DBFS for Bulk Loads
Mounting External File Systems on the DBFS
Change Password Script (Design Studio)
Enable the “Export” Endpoint of the Web Container
Execution Trace Information
Configure Export Key Script
Importing the Certificates of Data Sources (SSL/TLS Connections)
Installing the Connector for Oracle Essbase
Requirements of the Essbase 11 Adapter
Installing the JMS Connectors to Create JMS Listeners and Web Services with SOAP Over JMS
JMS Client Jars
Installing the SAP Java Connector
Installing the Connector
Testing the Connector on Windows
Testing the Connector on Linux
Grant Privileges in SAP for BAPI Data Sources
Grant Privileges in SAP BW for Multidimensional Data Sources
JMS Connection Details: JNDI Properties
Apache ActiveMQ 6.1.x
IBM WebSphere MQ 7.0
Progress SonicMQ 8.0
Launching Several Virtual DataPort Instances of the Same Installation
How to Configure the Secondary Instances
Deploying Web Services on This Configuration
Legacy ODBC Adapter
Enabling the Support for ODBC Sources
Enabling the Support for ODBC Sources When Using an External JRE
Creating an ODBC Data Source
Creating Base Views over an ODBC Data Source
Moved Permanently: Creating Multidimensional Base Views Over a Multidimensional Data Source
Mapping Multidimensional Data to a Relational Model
Creating Multidimensional Base Views Over a Multidimensional Data Source
Mapping the Result of MDX Queries
Moved Permanently: Mapping the Result of MDX Queries
OpenAPI 3 Data Sources FAQ
Resource Manager: Available Fields to Evaluate a Rule
Examples of Resource Manager Rules
Restriction of Number of Data Sources
Semantic Search Using Embeddings
Understanding Vector Embeddings
Configuration
Examples
Setting-up Features of Design Studio that Rely on Solution Manager
Prerequisites for Using these Features
Configuration Steps
Single User Mode
Explicit Single User Mode
Automatic Single User Mode
Supported JDBC Data Sources
Transforming Incoming/Outgoing Soap/Rest Messages with XSLT Stylesheets
Useful Tools to Debug Issues with Active Directory or Other LDAP Servers
Values of the Attribute “Access Interface”
Change Password Script (VDP)
Virtual DataPort Developer Guide
Preface
Scope
Who Should Use This Document
Summary of Contents
Introduction
Examples
Access Through JDBC
JDBC Driver Parameters
Connecting to Virtual DataPort Through a Load Balancer
Connecting to Virtual DataPort Using Kerberos Authentication
When the Client Application Does Not Belong to the Domain
Connecting to Virtual DataPort Using OAuth Authentication
Details of the JDBC Interface
Description of Views and Their Fields
Retrieving the Content Type of Blob Values
Working with Datetime Values with the Denodo JDBC Driver
Obtaining the Names of Elements Inside a Struct (Register)
Connecting From Applications That Filter Incoming Serialization Data
Queries with Datetime Literals
Using the VECTOR data type with the Denodo JDBC Driver
Get Auto-Generated Keys in IDU Statements
Enabling JDBC Logging
Access Through ODBC
Configuration of the ODBC Driver on Windows
Install the ODBC Driver on Windows
Set Up a DSN on Windows
Install Multiple Versions of the ODBC Driver on Windows
Configuration of the ODBC Driver in Linux and Other UNIX
Obtain the Appropriate ODBC Driver
Install UnixODBC
Register the Denodo ODBC Driver in UnixODBC
Register a Data Source (DSN) on UnixODBC
Compiling UnixODBC
Troubleshooting Issues
Creating a DSN-Less Connection
Use Kerberos Authentication
Use OAuth Authentication
Details of the ODBC Interface
How the ODBC Interface Reports the Datetime and Interval Data Types
Integration with Third-Party Applications
Supporting Queries with Brackets
Maximum Length of Text Values
Maximum Length of Error Messages
Disabling Transactions
Limit the Number of Incoming ODBC Connections
Backward Compatibility of the ODBC Driver
Access Through Arrow Flight-SQL
Access using Flight SQL
Access using Flight SQL JDBC driver
Access using Flight SQL Python driver
Current Restrictions
Type mapping: Virtual DataPort to Apache Arrow
Mapping: Virtual DataPort Types to Arrow Types
Access Through an ADO.NET Data Provider
Using Kerberos Authentication
Access Through OLE DB
Developing Extensions
Developing Custom Functions
Creating Custom Functions with Annotations
Creating Custom Functions Using Name Conventions
Compound Types
Custom Function Return Type
Getting Information About the Context of the Query
Dealing with Datetime and Interval Types
Developing Java Stored Procedures
Using Datetime Values in Denodo Stored Procedures
Required Libraries to Develop Stored Procedures
Developing VQL Stored Procedures
Developing Custom Wrappers
Required Libraries to Develop Custom Wrappers
Extending AbstractCustomWrapper
Overriding AbstractCustomWrapper
Dealing with Conditions
Dealing with the ORDER BY Clause
Configuring a Custom Wrapper
Updating the Custom Wrapper Plan
Dealing with Datetime and Interval Types
Developing Custom Input Filters
Required Libraries to Develop Custom Filters
Developing Custom Filters
Developing Custom Authenticators
Implementing Custom Authenticators
Configuring Custom Authenticators
Using Custom Authenticators
Diagnosing Custom Authenticators
Custom View Policies
Developing a Custom View Policy
Predefined View Policies
DYNAMIC_RESTRICTIONS_POLICY
Appendix
Output Schema of the LIST Command
Output Schema of the DESC Commands
Error Codes Returned by Virtual DataPort
Compile the Denodo ODBC Driver
Prerequisites for Compiling the ODBC Driver on Linux
Compile the ODBC Driver (Standard Method)
Compile the ODBC Driver to Obtain Forwardable Tickets
Troubleshooting the Compilation of the Denodo ODBC Driver
Virtual DataPort VQL Guide
Preface
Scope
Who Should Use This Document
Summary of Contents
Introduction
General Overview of Virtual DataPort
Creating or Defining Data
Defining Base Views
Defining Data Sources and Wrappers
Defining the Views of the Global Schema
Executing Statements
Language for Defining and Processing Data: VQL
Element Names (Identifiers)
Using Back Quotes
Object Descriptions
Data Types
Data Types for Dates, Timestamps and Intervals
Internationalization
Statements
Logical Operators
Comparison Operators
Comparing Literals
Three-valued Logic
Concatenation Operator
Functions for Conditions and Derived Attributes
Syntax Conventions
Syntax of Functions and Condition Values
Comments
Functions
Aggregation Functions
AVG
COUNT
FIRST
GROUP_CONCAT
LAST
LIST
MAX
MEDIAN
MIN
NEST
STDEV
STDEVP
STRING_AGG
SUM
VAR
VARP
Conversion Functions
ARRAY_TO_STRING
CAST
CREATETYPEFROMXML
REGISTER
Datetime Functions
ADDDAY
ADDHOUR
ADDMILLIS
ADDMINUTE
ADDMONTH
ADDSECOND
ADDWEEK
ADDYEAR
AT TIME ZONE
CONVERT_TIMEZONE
CURRENT_DATE
CURRENT_TIMESTAMP
EXTRACT
FIRSTDAYOFMONTH
FIRSTDAYOFWEEK
FORMATDATE
GETDAY
GETDAYOFWEEK
GETDAYOFYEAR
GETDAYSBETWEEN
GETHOUR
GETMILLISECOND
GETMINUTE
GETMONTH
GETMONTHSBETWEEN
GETQUARTER
GETSECOND
GETTIMEFROMMILLIS
GETTIMEINMILLIS
GETWEEK
GETWEEKSBETWEEN
GETYEAR
LASTDAYOFMONTH
LASTDAYOFWEEK
LOCALTIME
LOCALTIMESTAMP
MAX
MIN
NEXTWEEKDAY
NOW
PREVIOUSWEEKDAY
SUBTRACT
TO_DATE
TO_LOCALDATE
TO_TIME
TO_TIMESTAMP
TO_TIMESTAMPTZ
TRUNC
Embedding Functions
EMBED_AI
VECTOR_DISTANCE
VECTOR_COSINE_DISTANCE
VECTOR_L1_DISTANCE
VECTOR_L2_DISTANCE
VECTOR_NEGATIVE_INNER_PRODUCT
JSON Functions
AVRO_TO_JSON
COMPLEX_TYPE_TO_JSON
JSONPATH
JSON_TO_AVRO
JSON_TO_COMPLEX_TYPE
Large Language Model Functions
CLASSIFY_AI
ENRICH_AI
ENRICH_AI_BINARY
EXTRACT_AI
SENTIMENT_AI
SUMMARIZE_AI
TRANSLATE_AI
Numeric Functions
ABS
ACOS
ASIN
ATAN
ATAN2
CEIL
COS
COT
DEGREES
DIV
EXP
FLOOR
LN
LOG
MAX
MIN
MOD
MULT
PI
POWER
RADIANS
RAND
ROUND
SIGN
SIN
SQRT
SUBTRACT
SUM
TAN
TRUNC
Other Functions
COALESCE
CONTEXTUALSUMMARY
DECODE_BASE64
DECRYPT
DECRYPT_FIXED
ENCODE_BASE64
ENCRYPT
ENCRYPT_FIXED
GETVAR
GETCONTEXTVAR
GETSESSION
HASH
HASH_FUNCTION
MAP
NULLIF
PIVOTREGISTER
ROWNUM
UNPIVOTREGISTER
Spatial Functions
ST_AREA
ST_AREA_METERS
ST_BOUNDARY
ST_BUFFER
ST_BUFFER_METERS
ST_CENTROID
ST_CONTAINS
ST_CONVEXHULL
ST_CREATE_POINT
ST_CROSSES
ST_DIFFERENCE
ST_DIMENSION
ST_DISJOINT
ST_DISTANCE
ST_DISTANCE_METERS
ST_ENDPOINT
ST_ENVELOPE
ST_EQUALS
ST_EXTERIORRING
ST_GEOMETRYTYPE
ST_GEOM_TO_STRUCT
ST_GEOMETRYN
ST_INTERIORRINGN
ST_INTERSECTION
ST_INTERSECTS
ST_ISCLOSED
ST_ISEMPTY
ST_ISSIMPLE
ST_ISRING
ST_LENGTH
ST_LENGTH_METERS
ST_NUMGEOMETRIES
ST_NUMINTERIORRINGS
ST_NUMPOINTS
ST_OVERLAPS
ST_POINTN
ST_RELATE
ST_STARTPOINT
ST_SYMDIFFERENCE
ST_TOUCHES
ST_TRANSFORM
ST_UNION
ST_WITHIN
ST_WKBTOWKT
ST_WKTTOWKB
ST_X
ST_Y
Text Functions
ASCII
BASE64_TO_BASE10
BASE64_TO_HEX
CHAR
CONCAT
CONCATLIST
DELETESPACES
ENDWITH
HEX_TO_BASE64
INITCAP
INSTR
LEFT
LEFTPAD
LEN
LOWER
LTRIM
MAX
MIN
POSITION
PRINTF
PROPERCASE
REGEXP
REGEXP_COUNT
RIGHT
REMOVEACCENTS
REPEAT
REPLACE
REPLACEMAP
RIGHT
RIGHTPAD
RTRIM
SIMILARITY
SPLIT
STARTWITH
SUBSTRING / SUBSTR
TRIM
UPPER
Window Functions
APPROX_COUNT_DISTINCT
AVG
COUNT
CUME_DIST
DENSE_RANK
FIRST_VALUE
LAG
LAST_VALUE
LEAD
LISTAGG
MAX
MIN
NTILE
PERCENTILE_CONT
PERCENTILE_DISC
PERCENT_RANK
RANK
ROW_NUMBER
SUM
STDEV
XML Functions
XMLQUERY
XPATH
XSLT
Converting XML Data into Virtual DataPort Compound Types
Generating Wrappers and Data Sources
Creating Data Sources
JDBC Data Sources
ODBC Data Sources (Legacy)
Multidimensional Data Sources
Data Sources for SOAP Web Services
XML Data Sources
JSON Sources
DF Data Sources
LDAP Data Sources
BAPI Data Sources
Salesforce Data Sources
MongoDB Data Sources
Custom Data Sources
Data Source Configuration Properties
Specifying Paths in Virtual DataPort
Creating Wrappers
Execution Context and Interpolation Strings
Wrapper Metadata
JDBC Wrappers
Multidimensional Databases Wrappers
ODBC Wrappers
Web Services Wrappers
XML Wrappers
JSON Wrappers
DF Wrappers
LDAP Wrappers
BAPI Wrappers
Salesforce Wrappers
MongoDB Wrappers
CUSTOM Wrappers
Wrapper Configuration Properties
Valid Conversions Between Types in Wrappers and VDP Types
Native-type Conversions of a Wrapper to Java Types
Filters
Creating a Base View
Query Capabilities: Search Methods and Wrappers
Query Constraints
Assigning Wrappers to Search Methods
Example of How a Search Method Is Created
Modifying a Base View
Queries: SELECT Statement
FROM Clause
JOIN Operation
INTERSECT Operation
MINUS Operation
FLATTEN View (Flattening Data Structures)
Subqueries
WITH Clause
SELECT Clause
Derived Attributes
WHERE Clause
Conditions with Compound Values
Group BY Clause
Use of Aggregation Functions
HAVING Clause
UNION Clause
ORDER BY Clause
OFFSET, FETCH and LIMIT
CONTEXT Clause
TRACE Clause
CASE Expressions
CASE Clause Examples
Defining a Derived View
Modifying a Derived View
Defining an Interface View
Defining the Statistics of a View
Materialized Tables
Creating Materialized Tables
Inserting Data into Materialized Tables
Temporary Tables
Creating Temporary Tables
Privileges Required
Remote Tables
CREATE REMOTE TABLE Command
Summaries
CREATE SUMMARY VIEW Command
ALTER SUMMARY VIEW Command
Inserts, Updates and Deletes Over Views
INSERT Statement
UPDATE Statement
DELETE Statement
Return Modified Rows
Limitations
Use of WITH CHECK OPTION
Refreshing Views
REFRESH Statement
ALTER SESSION Statement
Transactions in Virtual DataPort
Stored Procedures
Importing a Stored Procedure
Invoking Denodo Stored Procedures
Predefined Stored Procedures
CACHE_CONTENT
CATALOG_ELEMENTS (deprecated)
CATALOG_FKS (deprecated)
CATALOG_METADATA_VIEWS
CATALOG_PERMISSIONS
CATALOG_PKS (deprecated)
CATALOG_VDP_METADATA_VIEWS
CATALOG_VIEWS (deprecated)
CHECK_METADATA
CLEAN_CACHE_DATABASE
COLUMN_DEPENDENCIES
COMPACT_CACHE
COMPACT_HADOOP_CACHE (deprecated)
COMPACT_METADATA_TABLES
COMPUTE_SOURCE_TABLE_STATS
CREATE_REMOTE_TABLE
CREATE_SCHEMA_ON_SOURCE
CREATE_TAGS_FROM_COLLIBRA
CREATE_TAGS_FROM_VIEW
DENODO_ASSISTANT_GENERATE_FIELDS_DESCRIPTION
DENODO_ASSISTANT_GENERATE_FIELDS_NAME
DENODO_ASSISTANT_GENERATE_VIEW_DESCRIPTION
DISCOVER_OBJECT_STORAGE_MPP_PROCEDURE
DROP_REMOTE_TABLE
DROP_SCHEMA_ON_SOURCE
DUAL
GENERATE_SMART_STATS_FOR_FIELDS (deprecated)
GENERATE_SMART_STATS_FOR_FIELDS_BY_TABLENAME
GENERATE_STATS (deprecated)
GENERATE_STATS_FOR_FIELDS (deprecated)
GENERATE_VQL_TO_CREATE_JDBC_BASE_VIEW
GENERATE_VQL_TO_CREATE_JDBC_BASE_VIEW_FROM_QUERY
GET_ACTIVE_LOGGERS
GET_ASSOCIATIONS
GET_AVAILABLE_STATS_MODES
GET_CACHE_COLUMNS
GET_CACHE_CONFIGURATION
GET_CACHE_LOAD_VIEW_DEPENDENCIES
GET_CACHE_TABLE
GET_CATALOG_EFFECTIVE_PERMISSIONS
GET_CATALOG_METADATA_WS
GET_DATABASES
GET_DELEGATED_SQLSENTENCE
GET_ELEMENTS
GET_EXPORTED_KEYS
GET_FOREIGN_KEYS
GET_GLOBAL_SECURITY_POLICIES_TAGS
GET_ICEBERG_VIEW_SNAPSHOTS
GET_JDBC_DATASOURCE_TABLES
GET_PARAMETER
GET_PRIMARY_KEYS
GET_PROCEDURE_COLUMNS
GET_PUBLIC_VIEW_DEPENDENCIES
GET_QUERY_EXECUTION_PLAN
GET_SERVER_CONNECTIVITY
GET_SESSIONS
GET_SOURCE_CHANGES
GET_SOURCE_COLUMNS
GET_SOURCE_TABLE
GET_STATS_FOR_FIELDS
GET_TYPE_ATTRIBUTES
GET_USER_ACCOUNTS
GET_USER_DEFINED_TYPES
GET_VIEW_COLUMNS
GET_VIEW_INDEXES
GET_VIEW_STATISTICS
GET_VIEW_TAGS
GET_VIEWS
LIST_JDBC_DATASOURCE_TABLES (deprecated)
LOGCONTROLLER
MAINTAIN_METADATA_TABLES
MIGRATE_DATE_TYPES
PING_DATA_SOURCE
REFRESH_BASE_VIEW
REFRESH_EMBEDDED_MPP_TABLES_METADATA
REFRESH_SOURCE_TABLE_METADATA
REGISTER_EMBEDDED_MPP
REMOVE_ICEBERG_VIEW_ORPHAN_FILES
REMOVE_ICEBERG_VIEW_SNAPSHOTS
ROLLBACK_ICEBERG_VIEW_TO_SNAPSHOT
SET_ICEBERG_VIEW_CURRENT_SNAPSHOT
SOURCE_CHANGES (deprecated)
VALIDATE_MPP_LICENSE
VIEW_DEPENDENCIES
WAIT
WEBCONTAINER_ELEMENTS (deprecated)
WEBCONTAINER_ELEMENT_STATUS
WEBCONTAINER_META_INF
WRITELOGERROR
WRITELOGINFO
Tags
Defining Other Elements of the Catalog
Creating Folders
Defining a Data Type
Defining a Map
Defining JAR Extensions
Defining JMS Listeners
Defining Kafka Listeners
Creating Databases, Users, Roles and Access Privileges
Creating and Modifying Virtual DataPort Databases
Managing Users
Modifying and Deleting Users
Connecting to Another Database
Modifying the Privileges of a User
Managing User Roles
Ownership of Elements
Global Security Policies
Describing Catalog Elements
Exporting Metadata
Importing Metadata
Listing Elements in the Catalog
Removing Elements From the Catalog
Publication of Web Services
Creating SOAP Web Services
Modifying SOAP Web Services
Creating REST Web Services
Modifying REST Web Services
Connection Parameters
Web Services Authentication
Basic and Digest
SAML 2.0
VDP
Embedded Web Container Management
Deploying and Exporting SOAP and REST Web Services
RESTful Architecture
Associations
Navigational Queries
Help Command
Version Control Systems Integration Commands
Statements to Work with GIT
DVCSCOMMIT
DVCSPUSH
DVCSPULL
DVCSREVERT
VCSCONTENTS
VCSSHOW
REVISIONS
GET_REVISION
UNRESOLVED_ELEMENT
Commands for Environment Management
Other Version Control Commands
VCSDISCARD
Resource Manager
Managing the Plans of the Resource Manager
Managing the Rules of the Resource Manager
Advanced Characteristics
Management of Compound Values
Processing of Compound Types: Example
Changing Settings of Virtual DataPort and the Web Container
Using the Cache
Cache Invalidation
Cache Table Recreation
Displaying the Contents of the Cache
Cache Table Creation Templates
Test Table Creation Templates
Updating Cache Control Columns
Configuring Swapping Policies
Managing Internationalization Configurations
Replacing an I18n Map
Deleting an I18n Map
Execution Context of a Query and Interpolation Strings
Adding Variables to Selection Conditions (GETVAR, GETCONTEXTVAR and SETVAR)
Appendix
Date and Time Pattern Strings
Virtual DataPort API Javadoc
Denodo Lakehouse Accelerator Guide
Data Marketplace Guide
Preface
Scope
Who Should Use This Document
Summary of Contents
Introduction
Installation and Execution
Launching the Data Marketplace
Introducing the Data Marketplace
Search Data Assets or Query Their Content
Role and permission needed
Search
Metadata Search
Metadata Search in Index
Content Search
Denodo Assistant Chatbot
Accessing the Chatbot
Using the Search Bar
Using the Floating Button
Panel Controls
Interacting with the Assistant
Sending Questions and Managing Responses
Filtering Context
Follow-up Questions
Tools
Metadata
Data
Deep Query
Direct Answer
Browse
Databases
Domains and Categories
Tags
Databases
Summary of Databases
Requests of Databases
Associations of Databases
Views
Summary of Views
Schema of Views
Query of Views
Building a Query
Executing a Query
Taking Advantage of Associations
Data Preparation
Loading Data
Preparing Data
Executing the Query
Practical Scenario
Filters Definition
Transformation Definition
Assisted Query
Generating a Query
Executing the Generated Query
Other Options
Examples
Assisted Query Role
Log Assisted Query HTTP information
Associations of Views
Lineage of Views
Reference Lineage of Views
Requests of Views
Usage of Views
Data Profiling of Views
Search of Views
Web Services
Summary of Web Services
Schema of Web Services
Schema of SOAP Web Services
Schema of REST Web Services
Query of Web Services
Building a Query
Executing a Query
Taking Advantage of Associations
Requests of Web Services
Usage of Web Services
External Elements
Summary of External Elements
Reference Lineage of External Element
Requests of External Elements
Workflow Requests
My Requests
Request Management
My Notifications
Administrator Notifications
Email Templates
Queries
My Queries
Deploying Saved Queries
Sharing Saved Queries
Exporting and Importing Saved Queries
VQL Shell
User Settings
Language and Region Settings
Internationalize Results
Internationalize Field Descriptions
User Interface Language
Denodo Assistant Settings
My Requests
My Notifications
Admin Notifications
Accessibility Mode
Change theme
Authentication
Authentication with Login and Password
Single Sign-On with Kerberos
Single Sign-On with an Identity Provider
Local Authentication
Authorization
Privileges on the Data Marketplace
Marketplace Management
Administration
Collaboration
User
Request
External element
Denodo Assistant
Roles with Predefined Privileges
data_catalog_editor
data_catalog_classifier
data_catalog_manager
data_catalog_content_admin
Roles with Implicit Privileges
data_catalog_exporter
data_catalog_data_preparation
data_catalog_assisted_query
data_catalog_admin
Users with Implicit Privileges
Data Marketplace Local User
Denodo Global Administrator
Administration
Server Set-Up
Configure the Virtual DataPort Servers
Configure the Connection Settings to the Virtual DataPort Servers
Enable Single Sign-On with Kerberos
Use an External Database for the Data Marketplace
Configure the Permissions
Large Language Models (LLM) Configuration
Vector Database Configuration
Search Engines
Index Servers
Content Search
Metadata Search
Personalization
Elements
Information
Export
Connection
Usage Statistics
Data Preparation
Denodo Assistant
Theme
Language and Region
Accessibility
Requests
Configuration
External Tools
Register a Tableau External Tool Server
Register a Microsoft Power BI External Tool Server
Synchronization
Categories Management
Create Categories
Edit Categories
Remove Categories
Assign Categories
Tags Management
Create Tags
Edit Tags
Remove Tags
Assign Tags
Elements Management
Property Groups Management
Manage Property Groups
Manage the Custom Properties of a Property Group
Assign Property Groups
Import and Export Data Marketplace Metadata
Export Metadata
Import Metadata
Synchronize with Virtual DataPort
Appendix
Installing the Data Marketplace on a Cluster: Sharing the Same Settings Across All the Nodes
Setting Up the Common Database
Configure the Data Marketplace to Use the Common Database
REST API
HTTP Basic Authentication (stateless)
OAuth Token (stateless)
OAuth Token (Stateful)
CORS Configuration
Apache Lucene Search Syntax
Terms
Fields
Term Modifiers
Boolean Operators
Grouping
Field Grouping
Escaping Special Characters
Data Marketplace Scripts
Change Password Script
Configure Export Key Script
Export Metadata Script
Import Metadata Script
Change Database Settings Script
Set Up External Tool Environment
Set Up Power BI
Set Up Tableau
Diagnostic & Monitoring Tool Guide
Preface
Scope
Who Should Use This Document
Summary of Contents
Introduction
General Architecture
Installation and Execution
Launching the Diagnostic & Monitoring Tool of the Denodo Platform
Launching the Diagnostic & Monitoring Tool of the Denodo Solution Manager
Introducing the Diagnostic & Monitoring Tool
Authentication
Authorization
Creating Servers and Environments
Creating Servers
Creating Environments
Monitoring
Monitoring Servers
Monitoring - Resources
Monitoring - State
Monitoring - Sessions
Monitoring - Requests
Monitoring - Cache
Monitoring - Data Sources
Monitoring - Threads
Monitoring - Denodo Lakehouse Accelerator Cluster
Monitoring Environments
Overview
Creating Diagnostics and Diagnostic Intervals
Creating Diagnostics
Creating JDBC Diagnostics
Creating Diagnostic Intervals
Diagnosing
Diagnosing - Resources
Diagnosing - State
Diagnosing - Sessions
Diagnosing - Requests
Diagnosing - Cache
Diagnosing - Data Sources
Diagnosing - Threads
Diagnosing - Errors
Export and Import
Export
Import
Configuration
User Configuration
Configuration - General
Configuration - Overview
Configuration - Resources
Configuration - State
Configuration - Sessions
Configuration - Requests
Configuration - Cache
Configuration - Data Sources
Configuration - Threads
Configuration - Errors
Configuration - Denodo Lakehouse Accelerator
Server Configuration
Informative Message
Database Configuration
Scheduler Administration Guide
Preface
Scope
Who Should Use This Manual
Summary of Contents
Introduction
General Architecture
Installation and Execution
Administration
Authentication
Scheduler Server Local Authentication
Web Administration Tool Local Authentication
Web Configuration
Kerberos Configuration
Informative Message Configuration
Change Password
Community Resources Configuration
Language Configuration
Scheduler Server Configuration
Server Set-up
Management
Import/Export Backups
Index Server
Log Configuration
Creating and Scheduling Jobs
Jobs
Data Sources
Scheduler Index Data Sources
CSV Data Sources
Elasticsearch Sources
JDBC Data Sources
VDP Data Sources
AWS S3 Data Sources
Azure Data Lake Storage (ADLS) Data Sources
Google Cloud Storage Data Sources
Alibaba Cloud OSS Data Sources
Configuring New Jobs
General Structure of a Job
Individual Query Configuration Section
Simple Cache Management Configuration Section
DAG Cache Management Configuration Section
Data Loader Configuration Section
Data Indexer Configuration Section
Exporters Section
Retry Section
Handler Section
Time-based Job Scheduling Section (Triggers Section)
Reports Section
Dependencies Among Jobs
Developer API
Scheduler RMI Client API
Scheduler REST Client API
Public
Private (Deprecated)
Extensions (Plugins)
Exporters
Handlers
Appendix
JDBC Drivers
Use of Configure Scheduler Export Key Script
Use of the Import/Export Scripts for Backup (Scheduler)
Export
Import
Use of the Import/Export Scripts for Backup (Scheduler Index)
Export
Import
Use of the Ping Script (Scheduler)
Use of the Ping Script (Scheduler Index)
Use of the Install Plugins/Drivers Scripts
Install Plugins
Install Drivers
ChangePassword Script (Scheduler)
ChangePassword Script (Scheduler Web Administration Tool)
ChangePassword Script (Scheduler Index)
How to Manually Configure the Metadata Database
Scheduler API Javadoc
Solution Manager Installation Guide
Preface
Who Should Use This Document
Pre-Installation Tasks
Hardware Requirements
Minimum Requirements to Run the Solution Manager On Premise
Minimum Requirements to Run the Solution Manager on Amazon AWS
Recommendations to Run the Solution Manager on Microsoft Azure
Software Requirements
Supported Operating Systems
Supported Browsers
Other Pre-installation Tasks
Select a User Account to Install the Solution Manager
Check that the Required Ports Are Free
Grant Privileges to This User Account
Check the PATH Environment Variable on Windows
Download the Installer
Using the Graphical Installation Wizard
Settings you can Modify when you Select the Custom Installation
Common to All Modules
Virtual DataPort
Solution Manager
Using the Command Line Installer
Unattended Installation of the Solution Manager
Modifying the Solution Manager Installer to Include the Latest Update
Unattended Installation of the Solution Manager
Post-Installation Tasks
Replicate the Encryption Key Across All the Installations
If You Already Started One of the Components of the Denodo Platform / Solution Manager
Install the Latest Update
Install the License
Change the Default Passwords
Solution Manager Server
Design Studio
Enable SSL/TLS in the Solution Manager
Obtaining and Installing an SSL Certificate
Enabling SSL/TLS in Solution Manager Servers
Configure Windows Services
High Availability
Step #1: Configuring the Solution Manager Installations for High Availability
Step #2: Configuring the Load Balancer for High Availability
Step #3: Configuring Access Through the Load Balancer
JVM Memory Configuration
Denodo Platform Control Center
Starting Platform Servers and Tools
Installing Updates, Bugfixes and Extra Features
Virtual Machine and Web Container Configuration
Configuration of the JVM Parameters From the Command Line
Control Center Help
Uninstalling the Solution Manager
Appendix
Configuring the Network Interface Restriction in the Solution Manager
Default Ports Used by the Solution Manager Modules
Transparent Metadata Encryption in the Solution Manager
Providing a Krb5 File for Kerberos Authentication
Session Configuration (Solution Manager Administration Tool, Design Studio, Scheduler, Diagnostic & Monitoring Tool)
Troubleshooting the Solution Manager Installer
Solution Manager Administration Guide
Preface
Scope
Who Should Use This Document
Summary of Contents
Introduction
Architecture
Starting the Solution Manager
Introducing the Solution Manager Administration Tool
License Management
Install a License
Check Global License Information
Assign Licenses
Assign a License to a Server
Check the License Information for an Environment
How Licenses Work
How a Server Resolves Its License
Customize Hostname and Port in License Request
How Denodo Servers Behave on License Responses
License Usages Table
Configuration
Setting-Up an External Database
Changing the Password of the External Database
Kerberos Support
VCS Configuration
Enabling Support for VCS Git Backups Bigger Than 100 Megabytes
Setting-Up the Informative Message
Denodo Support
Global Monitoring Configuration
Cloud Storage Appenders
JDBC Appenders
Authentication and Authorization
User Management
Authenticating with LDAP
Enabling the LDAP Roles Cache in Solution Manager
Authenticating with Single Sign-On
SAML Configuration
OAuth Configuration
OpenID Configuration
Kerberos Configuration
Authorization
Global Privileges
Privileges Granted to a Role For an Environment
Role Management
Import Roles From an LDAP Server
Denodo Security Token
Architecture
Configure the Denodo Platform Components to Reach the Security Token Server
Security Considerations
External Identity Provider Considerations
Configure the Connection to the Security Token Server From the Denodo Platform
Configure HTTP Timeouts and Limits for the Request of Public Keys
Load Balancer Considerations
Authentication Credentials
Regenerate Autogenerated Credentials
Single Sign-On Sequence
Environments
Clusters
Working Modes
Differences Wizard
Standard Mode
Creating Environments
Configuring Environments
Creating Standard Clusters
Configuring Clusters
Automated Cloud Mode
Configuring Mode
General
Server Access
Default Region & Images
Creating Environments
Automated Cloud Mode (AWS)
Automated Cloud Mode (Azure)
Configuring Environments
AWS Environment
Azure Environment
Creating Clusters
New AWS Cluster
New Azure Cluster
Configuring Clusters
AWS Cluster
Azure Cluster
Managing Clusters
View the Cluster Summary
View the Progress of the Last Action
Enable a Cluster
Disable a Cluster
View All Cluster Events
Start a Cluster
Stop a Cluster
Restart a Cluster
Recreate a Cluster From an Image
Recreate a Cluster From a Server
Install a Denodo Update in a Cluster
Monitor the Cluster with the Diagnostic & Monitoring Tool
Environment Configuration
Configuring Virtual DataPort Properties
Configuring Deployments
Standard Mode Environments Deployments Configuration
Cloud Environments Deployments Configuration
Data Marketplace Server Synchronization
Parallel Deployment Strategy
Configuring Deployment Scripts (Standard Mode)
Examples of Deployment Scripts
Configuring Environment Logging Level
Configuring Environment Monitoring
Clusters Configuration
Configuring Cluster Load Balancing Variables (Standard Mode)
Configuring Scheduler Properties
Configuring Cluster Logging Level
Servers
Creating Servers
Configuring Servers
Configuring Server Load Balancing Variables (Standard Mode)
Configuring Server Logging Level
Load Balancing Variables
Creating Load Balancing Variables
Assigning Values to Loading Balancing Variables
Promotions
Creating Revisions
General Options
Additional Options
Revision Candidate Elements
Create Revisions From VQL
Create Revisions From Compare Environments Wizard
Compare Environments Wizard
Revision Candidate Elements
Creating Revisions From Compare Environments Wizard
Revisions Table
Create a Revision
Edit a Revision
Edit a Revision From the Compare Environments Wizard
See the Information of a Revision
Remove Revisions
Filter Revisions in the Table
Download the Metadata of a Revision
Validate a Revision
Check the Validation Summary of a Revision
Copy Revision
Deploy Revisions
Check the Environments Where a Revision Was Deployed
Standard Mode Deployments
Deployment Preconditions
Deployment Paradigms
Cache Considerations
Cloud Deployments
Cloud Deployments Options
Cache Considerations
Considerations for Deployments
Cache
Summaries
Denodo Lakehouse Accelerator
Deployments Table
Cancel Deployments
Remove Deployments
Filter Deployments in the Table
Check the Progress Summary of a Deployment
See the Errors of a Deployment
See the Revisions of a Deployment
Download Backup Data for a Deployment
Deployment Progress
Monitoring
Monitoring Configuration
Monitoring Virtual DataPort Servers Running Behind a Load Balancer
Import and Export
Import From Solution Manager 8.0
Appendix
REST API
Get the List of Environments
Create an Environment
Get the List of Licenses Available
Update an Environment
Get Environment Permissions
Update Environment Permissions
Delete an Environment
Update AWS Global Credentials
Update AWS Environment Credentials
Update Global Monitoring Configuration
Update Azure Global Credentials
Update Azure Environment Credentials
Start Automated Mode Cluster
Stop Automated Mode Cluster
Restart Automated Mode Cluster
Recreate Automated Mode Cluster
Recreate From Servers Automated Mode Cluster
Install Denodo Update Automated Mode Cluster
Get the List of Virtual DataPort Properties Associated to an Environment
Create and Delete Environment Properties
Get the List of Clusters
Start Environment Monitoring
Get Environment Monitoring Status
Stop Environment Monitoring
Create a Cluster
Get a Cluster
Get Cluster Status
Update a Cluster
Delete a Cluster
Get the List of Scheduler Properties Associated to a Cluster
Create and Delete Cluster Properties
Get the List of Servers
Create a Server
Get a Server
Update a Server
Delete a Server
Export Solution Manager Catalog and Configuration
Import Solution Manager Catalog and Configuration
Get the List of Revisions
Validate Revisions
Create a Revision From a VQL File
Download the VQL of a Revision
Download the Scheduler ZIP of a Revision
Get the List of Deployments
Start a New Deployment From a List of Revisions
Get the Progress of a Deployment
List Roles
Create a role
Get a Role
Delete a Role
Update a Role
Free License Usage
Free License Usage by Server Id
Ping License Manager Server
Ping to Solution Manager Server
Get VCS configuration
Set VCS configuration
Best Practices: Deployment of Updates Across Your Organization
Updating Solution Manager with High Availability
Solution Manager Compatibility with the Denodo Platform Servers
Creating A Custom AWS AMI for the Denodo Platform
Creating A Custom Azure Image for the Denodo Platform
Suggested Reading Materials to Use the Automated Cloud Mode
Architecture of a Deployment of Solution Manager in AWS
Architecture of a Deployment of Solution Manager in Azure
Best Practices for Automated Mode Environments
Maximizing Uptime and High Availability
Infrastructure Deployment Considerations
Health Checks
Backup and Recovery
Restoring Backup
Maintenance
Automated Cloud Mode - Creating a Backup
Creating an IAM Policy
Main Features
Automated VPC Management
Automated Subnet Management
Automated Auto Scaling Group Management
Automated Security Group Management
Creating an Azure Custom Role
Main Features
Automated Resource Group Management
Automated Virtual Network Management
Automated Subnet Management
Automated Network Security Group Management
Minimizing Downtime Option
Automated Auto Scale Set Management
Automated Custom Azure DNS Records Management
Configuration Files of the Monitoring in Solution Manager
JDBC Logging Configuration
Cloud Storing Configuration
Agora Guide
Agora - Subscription Bundles
Differences between Agora and Self-Managed Instances
External Identity Provider
External Identity Provider General Settings
Identity Provider Configuration
Entra ID Configuration
Okta Configuration
SCIM
SCIM Provisioning Configuration
SCIM Configuration for Entra ID
SCIM Configuration for Okta
AWS
Available Regions
Log Configuration
Configuring the S3 Bucket in Agora Console
Permissions of your S3 bucket
Connectivity Guidelines
Default Scenario
Southbound Connections
Northbound Connections
Auto Scaling Guide
How to Deploy a Cluster with Auto Scaling
How to Identify the Name of the Auto Scaling Group
Example of Automatic Scaling Configuration
Execution Plane Architecture
How to Identify AWS Elements Provisioned by Agora
List of AWS Elements Created by Agora in the User’s Account
AWS Elements Created Deploying with Agora Production Template
Denodo AI SDK
Accessing the Tools
Updating the AI Tools Configuration
Azure
Available Regions
Log Configuration
Configuring the Storage Account in Agora Console
Permissions of your Azure Storage Account
Connectivity Guidelines
Azure Network Elements
Southbound Connections
Northbound Connections
Auto Scaling Guide
How to Deploy a Cluster with Virtual Machine Scale Set
How to Identify the Name of the Scale Set
Example of Automatic Scaling Configuration
Best Practices for Azure Auto Scaling
Execution Plane Architecture
How to Identify Azure Elements Provisioned by Agora
List of Azure Elements Created by Agora in the User’s Account
Azure Elements Created Deploying with Agora Production Template
DenodoConnect Components
Apache Zeppelin for Denodo - User Manual
Denodo AI SDK - User Manual
Denodo Azure AI Search Custom Wrapper - User Manual
Denodo Couchbase Custom Wrapper
Denodo Custom Wrapper for IBM Cognos TM1 - User Manual
Denodo Custom Wrapper for SAS - User Manual
Denodo Dashboard - User Manual
Denodo Dialect for SQLAlchemy - User Manual
Denodo Dialect for Spark SQL - User Manual
Denodo Distributed File System Custom Wrapper - User Manual
Denodo DynamoDB Custom Wrapper - User Manual
Denodo Email Exported Files Custom Handler - User Manual
Denodo FileSystem CustomWrapper - User Manual
Denodo Geo Service - User Manual
Denodo Google Sheets Custom Wrapper - User Manual
Denodo Google Sheets Tool - User Manual
Denodo Governance Bridge - Collibra Protect Integration - User Manual
Denodo Governance Bridge for Collibra - User Manual
Denodo Governance Bridge for IGC - User Manual
Denodo Governance Bridge for Microsoft Purview - User Manual
Denodo Governance Bridge for OpenLineage - User Manual
Denodo GraphQL Custom Wrapper - User Manual
Denodo HL7 FHIR Service - User Manual
Denodo Hibernate Dialect - User Manual
Denodo Kafka Custom Wrapper - User Manual
Denodo Log Custom Wrapper - User Manual
Denodo MCP Server - User Manual
Denodo MDX Service - User Manual
Denodo OData 2.0 Service - User Manual
Denodo Power BI Custom Connector - User Manual
Denodo RFCReadTable Custom Wrapper - User Manual
Denodo SFTP Exported Files Custom Handler - User Manual
Denodo SSH Custom Wrapper - User Manual
Denodo Tableau Exporter - User Manual
Denodo Testing Tool - User Manual
Denodo VQL Generation Stored Procedures - User Manual
Denodo VQL Templates for PPDM - Quick Guide
Generic XML Exporter - User Manual
Quick Start Guides
Denodo Enterprise
Denodo 9 Enterprise for AWS Quick Start Guide
Denodo 9 Enterprise for Azure Quick Start Guide
Denodo 9 Enterprise for GCP Quick Start Guide
Denodo Enterprise Plus
Denodo 9 Enterprise Plus for AWS Quick Start Guide
Denodo 9 Enterprise Plus for Azure Quick Start Guide
Denodo 9 Enterprise Plus for GCP Quick Start Guide
Denodo Professional
Denodo 9 Professional for GCP Quick Start Guide
Denodo Standard
Denodo 9 Standard for GCP Quick Start Guide
Denodo Platform
Denodo Platform 9 Container Registry Quick Start Guide
Denodo Platform 9 Helm Charts Quick Start Guide
Denodo Platform Container 9 QuickStart Guide
Denodo Platform for AWS 9 BYOL Quick Start Guide
Denodo Platform for AWS 9 CBP Quick Start Guide
Denodo Platform for GCP 9 BYOL Quick Start Guide
Agora Quick Start Guide
Agora on AWS Marketplace Free Trial Quick Start Guide
Agora on AWS Marketplace Quick Start Guide
Denodo 9 for Cloud Marketplaces Frequently Asked Questions (FAQ)
Denodo Developer 9 Quick Start Guide
Denodo Express 9 Quick Start Guide
Important:
To see the latest version of the document click here
User Manuals
/
Denodo Lakehouse Accelerator Guide
Denodo Lakehouse Accelerator Guide
¶
This page is available in the
latest version of the Denodo Lakehouse Accelerator Guide
.
Feedback:
Share:
Add feedback
