SKIP TO MAIN CONTENT
Sign In
Register
Denodo Platform 9.1
Expert Trails
Cache
Datasource Connectivity
Deployment Topology
Development Lifecycle
Modeling and Metadata Organization
Monitoring
Operations
Query Performance Optimization
Security
Considerations for Designing a Self-Service Analytics Initiative
Enterprise Data Services with APIs
Q&A
All Questions
Ask a Question
Search by tag
Resources
User Manuals
Knowledge Base
Videos
Tutorials
Browse Content Library
Drivers
Data Management Blog
Technical Newsletter
Test Drives
Agile BI and Analytics on AWS
Data Catalog and Data Science on AWS
Agile BI and Analytics on Azure
Data as a Service (DaaS) on GCP
Agile BI and Analytics on Alibaba
Denodo Express
Learn
Training Courses
Learning Paths
Course Catalog
Your Progress
Workshops
Certification Exams
Academic Program
Overview
University Challenge
User Groups
Support
Support Site
Contact Us
USER MANUALS
Denodo Platform
8.0
English
日本語
Denodo Platform Installation Guide
Legal Notes
Third-Party Java Libraries Distributed with the Denodo Platform 8.0u20240926
Third-Party Non-Java Libraries Distributed with the Denodo Platform 8.0u20240926
Third-Party Java Libraries Distributed with the Denodo Platform 8.0u20240306
Third-Party Non-Java Libraries Distributed with the Denodo Platform 8.0u20240306
Third-Party Java Libraries Distributed with the Denodo Platform 8.0u20230914
Third-Party Non-Java Libraries Distributed with the Denodo Platform 8.0u20230914
Third-Party Java Libraries Distributed with the Denodo Platform 8.0u20230301
Third-Party Non-Java Libraries Distributed with the Denodo Platform 8.0u20230301
Third-Party Java Libraries Distributed with the Denodo Platform 8.0u20220815
Third-Party Non-Java Libraries Distributed with the Denodo Platform 8.0u20220815
Preface
Scope
Who Should Use This Document
Summary of Contents
Introduction
Pre-Installation Tasks
Hardware Requirements
Minimum Requirements to Run the Denodo Platform On Premise
Minimum Requirements to Run the Denodo Platform on Amazon AWS
Minimum Requirements to Run the Denodo Platform on Microsoft Azure
Minimum Hardware Requirements when Installing each Module Separately
Considerations Regarding the Disk Space Requirements of the Denodo Platform
Running the Denodo Platform on Docker
Requirements for the Virtual DataPort Administration Tool
Software Requirements
Supported Operating Systems
Supported Browsers
Software Requirements for ITPilot
Configuring the Server for the Denodo Platform
Select a User Account to Install the Denodo Platform
Grant Privileges to This User Account
Check the PATH Environment Variable (Windows)
Check that the Required Ports Are Free
Close All the Browsers
Download the Installer
Using the Graphical Installation Wizard
Commons Settings
Components and Configuration of Virtual DataPort
Components and Configuration of ITPilot
Initial Configuration of the Browser Pool
ITPilot Wrapper Server
ITPilot Verification Server
Installation Settings of Scheduler
Scheduler Server
Scheduler Index Server
Embedded Web Container
Using the Command Line Installer
Unattended Installation of the Denodo Platform
Modifying the Denodo Platform Installer to Include the Latest Update
Post-Installation Tasks
Install the Latest Update
Set the Host Name of Virtual DataPort
Register the Denodo Servers in the License Manager
Enable SSL/TLS in the Denodo Platform
Obtaining and Installing an SSL/TLS Certificate
Denodo SSL/TLS Configurator Script
Enabling SSL/TLS in Denodo Platform Servers
Enabling HTTPS in the Embedded Apache Tomcat
Enabling SSL/TLS in the Administration Tool and Others
Enabling SSL/TLS for External Clients
SSL/TLS Versions Supported by the Denodo Platform Servers
Configure Windows Services
Post-Installation Tasks: Virtual DataPort
Change the Default Passwords
Setting-up Kerberos Authentication
Setting-up Kerberos Authentication in the Data Catalog
Installing the Connector for Oracle Essbase
Installing the SAP Java Connector
Installing the JMS Connectors to Create JMS Listeners and Web Services with SOAP Over JMS
Configuring the VCS Clients
Importing the Certificates of Data Sources (SSL/TLS Connections)
Preparing the Connection to Data Sources
Post-Installation Tasks: Scheduler
Change the Default Passwords
Setting-up Kerberos Authentication in Scheduler
Post-Installation Tasks: Web Container
Enable Authentication on the Monitoring Interface
Post-Installation Tasks: ITPilot
Checking that the Generation Environment Has Been Installed
Configuration of Microsoft Internet Explorer
Enabling Internet Explorer Sequence Generation Toolbar in Microsoft Windows Server 2008
Disabling Internet Explorer Enhanced Security Configuration in Microsoft Windows 2008
Disabling Internet Explorer Enhanced Security Configuration in Microsoft Windows Server 2012
Manual Configuration of Adobe Acrobat Professional
Automatic Verification Database
Launching the Wrapper Generator Tool in High DPI Displays
Upgrading from a Previous Version and Other Tasks
Installing Updates and Hotfixes
Installing an Update or Hotfix Graphically
Installing an Update or Hotfix from the Command Line
Troubleshooting the Updates Installer
Denodo Platform Control Center
Launching the Denodo Platform Control Center
Starting Platform Servers and Tools
Denodo Platform Configuration
Configuring the Connection to the License Manager
Configuration of the JVM Parameters from the Command Line
Control Center Help
Uninstalling the Denodo Platform
Appendix
Denodo Platform - Subscription Bundles
Denodo Enterprise Plus
Restrictions of Denodo Express
Default Ports Used by the Denodo Platform Modules
Health Monitoring
Enabling Kerberos Authentication Without Joining a Kerberos Realm
How to Debug Kerberos in Web Applications
Virtual DataPort Server and the Web Administration Tools Are on the Same Installation
Virtual DataPort Server and the Web Administration Tools Are not on the Same Installation
Providing a Krb5 File for Kerberos Authentication
Denodo Security Token Configuration File
Common Settings
Authentication Modules
Troubleshooting the Denodo Platform Installer
Transparent Metadata Encryption
Supported Java Development Kits (JDK)
Increasing the Maximum Simultaneous Requests
Language of the Web Administration Tools
Install a Language File for Data Catalog
Install a Language File for Design Studio
Install a Language File for the Scheduler Administration Tool
Data Catalog, Design Studio and Scheduler in Other Languages
How to Refresh Custom Messages After Installing an Update
Session Configuration (Design Studio, Data Catalog, Scheduler, Diagnostic & Monitoring Tool)
Maximum Session Timeout
Idle Session Timeout
Single-User Concurrent Sessions Control
Proxy Settings (Design Studio, Data Catalog, Scheduler)
Reverse Proxy Configuration
Password Policies in the Denodo Platform and Solution Manager
Policies that Are Disabled by Default in Virtual DataPort and Solution Manager
When All the Administrators are Locked Out of Virtual DataPort or Solution Manager
Important Notes
Prevent Load Balancers From Closing Connections of Data Catalog and Design Studio
Enabling this Feature
Audit Trail
Audit Information
Masking Sensitive Information
How to Configure Audit in Scheduler
How to Configure Audit in Solution Manager
Server Process Identifiers
Adding AWS RDS Certificate to JRE cacerts
Denodo Platform Upgrade Guide
Preface
Introduction to Upgrading the Denodo Platform
Major Steps in the Upgrade Process
Running Both Versions in the Same Computer
Upgrading from Denodo Platform 5.5 or Earlier
Upgrading from Denodo Platform 5.5
Upgrading from Denodo Platform 5.0 or Earlier
Preparing to Upgrade
Common to All Modules
Kerberos
Virtual DataPort
Virtual DataPort: Select a Catalog/Schema for the Cache Engine
Virtual DataPort: Create a Repository for Version Control
Data Catalog: Create a Catalog/Schema on the External Database
Scheduler: Create a Catalog/Schema on the External Database
Solution Manager
Solution Manager: Create a Git Repository or Branch for Version Control
Solution Manager: Create a Catalog/Schema on the External Database
Pre-Upgrade Tasks for the New Installation
Common to All Modules
Install the Latest Update
Enable SSL
Configure Windows Services
Copy Krb5 File
Virtual DataPort
Virtual DataPort: JDBC Data Sources that Use the Bulk Data Load API
Virtual DataPort: Enable the Cache Engine
Virtual DataPort: Install Third-Party Connectors
Virtual DataPort: Other Third-Party Jar Files
Virtual DataPort: Version Control System
Virtual DataPort: Delete Unused Objects
Virtual DataPort: Delete Aracne and Google Search Data Sources
Virtual DataPort - Published Web Services: Customize Logo
Data Catalog: Enable the External Database
Scheduler: Enable the External Database
Solution Manager
Solution Manager: Enable the External Database
Solution Manager: High Availability
Solution Manager: Enable Git Support
Upgrading from Denodo 6.0: Export the Settings and Metadata of the Current Installation
Transforming Views Automatically to Use the New Datetime Data Types
Configuration Properties of Virtual DataPort for Upgrades: “exportMigrationCompatibility” and “exportOnlyResources”
Alternatives for When You Cannot Install the Latest Update of Denodo 7.0
Upgrading from Denodo 7.0: Export the Settings and Metadata of the Current Installation
Configuration Properties of Virtual DataPort for Upgrades: “exportMigrationCompatibility” and “exportOnlyResources”
Alternatives for When You Cannot Install the Latest Update of Denodo 7.0
Review the Virtual DataPort Metadata
Import Settings and Metadata in to the New Installation
Common Errors when Importing the Metadata of Virtual DataPort 7.0 to 8.0
Post-Upgrade Tasks for the New Installation
Common to All Components
Configure the Connection to the Denodo Security Token Server
Virtual DataPort
Increase the Memory Allocated to the Tomcat of Denodo Platform
Virtual DataPort Administration Tool
Data Catalog
Data Catalog: Update the URL of Virtual DataPort
Data Catalog: Update the URL to the Index Server
Data Catalog: Synchronize Metadata
Data Catalog: Configure the Gathering of Statistics
Data Catalog: Kerberos
Scheduler
Scheduler Index
Scheduler Index: Copy the Index Files
Scheduler Index: Configure the Indexes
Solution Manager
Solution Manager: Active Monitors
Solution Manager: Kerberos Configuration
External Applications: Upgrade the JDBC and ODBC Drivers
Test the New Denodo Platform Installation
Backward Compatibility
Changes in Denodo Platform 8.0 GA
Changes Common to All the Modules of Denodo 8.0 GA
Changes in Virtual DataPort 8.0 GA
Changes in Solution Manager 8.0 GA
Features Removed from Aracne 8.0 GA
Features Removed from Virtual DataPort 8.0 GA
Features Removed from Scheduler 8.0 GA
Features Removed from Solution Manager 8.0 GA
Changes in Denodo Platform 7.0
Changes Common to All the Modules of Denodo 7.0
Changes in Virtual DataPort 7.0
Changes in the Privileges System in Virtual DataPort 7.0
Features Removed from Virtual DataPort 7.0
Changes Introduced in Aracne 7.0
Changes in Denodo Platform 6.0
Changes Common to All the Modules of Denodo 6.0
Changes in Virtual DataPort 6.0
Changes in Scheduler 6.0
Changes in the Embedded Web Container of Denodo 6.0
Changes in Denodo Platform 5.5
Changes in Virtual DataPort 5.5
Features Removed in Denodo 5.5
Changes in Denodo Platform 5.0
Changes in Virtual DataPort 5.0
Features Deprecated in Denodo Platform 8.0
Virtual DataPort
Data Type “Date”
VQL Syntax
Data Sources
Denodo Stored Procedures
Role JMXAdmin
Denodo Stored Procedures API: getNumOfAffectedRows Method
Denodo Custom Wrappers API: Deprecated Methods
Denodo JDBC Driver
Denodo Web Services
Widgets
Version Control Systems (VCS)
Script Export: Deprecated Parameters
Launching Several Virtual DataPort Instances of the Same Installation
Resource Manager
Data Catalog
Roles “selfserviceadmin”, “selfserviceexporter”
Elasticsearch 5.x and 6.x
ITPilot
Scheduler
Private Scheduler REST API
Elasticsearch 5.x and 6.x
Solution Manager
REST API: Authentication Using Session
Denodo4Eclipse Plugin
Denodo Platform New Features Guide
Preface
Audience
What Is New in Denodo 8.0u20240926
What Is New in Denodo 8.0u20240306
What Is New in Denodo 8.0u20230914
What Is New in Denodo 8.0u20230301
What Is New in Denodo 8.0u20220815
What Is New in Denodo 8.0u20220126
What Is New in Denodo 8.0u20210715
What Is New in Denodo 8.0u20210209
Denodo 8.0 GA Highlights
New Features of the Solution Manager 8.0 GA
Automated Deployments
Single Point of Access
Authentication and Authorization
New Authentication Protocols and Single Sign-On
Privileges System Extended
Unified User Interface to Manage Roles and Privileges
Diagnostic & Monitoring Tool of the Solution Manager
Diagnostic & Monitoring Tool Populates Servers Automatically
Diagnostic & Monitoring Tool Starts Automatically
New Features of Virtual DataPort 8.0 GA
Design Studio
General
Storing the Catalog on an External Database
Graphical Support to Upload Drivers and Other Libraries
Graphical Support to Upload Kerberos Files
Connections Only Use One Port
Managing Configuration Properties
Rename Elements
Execution Engine
Smart Query Acceleration: “Summary Views”
Data Sources
Hadoop Distribution is Not Required for Bulk Data Load
Authentication and Security
Global LDAP Authentication
Apply Changes to OAuth 2.0, SAML 2.0 and Kerberos Settings Without Restarting
Propagations of Column Privileges, Row Restrictions and Custom Policies
Denodo Stored Procedures
Procedure Create Remote Table Provides More Information
Administration Tool
Easier Access to OData and GraphQL Services
Warning When Deleting the Implementation of an Interface View
Database Is Optional in the Server URI
Northbound Connections
New Denodo GraphQL Service
New Features of Scheduler 8.0 GA
User Interface
New Features Common to All Modules of 8.0 GA
Java Runtime Environment
New Script to Enable SSL/TLS on the Denodo Platform
New Supported Operating System
All Components Are Installed as Windows Services
Web Container of Denodo
New Version of the Web Container: Apache Tomcat 9.0
Only Allow TLS 1.2 and 1.3
Virtual DataPort Administration Guide
Preface
Scope
Who Should Use This Document
Summary of Contents
Introduction
General Architecture
Physical Layer
Logical Layer
Data Module: Cache
User Layer
Starting Virtual DataPort
Starting the Virtual DataPort Server
Launching the Administration Tool
Main Areas of the Administration Tool
Tool Preferences
Using the VQL Shell
Query Monitor
Catalog Search
Invalidate Cache
Trace Viewer
Starting the Design Studio
Main Areas of the Design Studio
Design Studio Preferences
Using the Design Studio VQL Shell
Design Studio Query Monitor
Design Studio Catalog Search
Design Studio Invalidate Cache
Design Studio Trace Viewer
Logging in to Design Studio
Creating Data Sources and Base Views
JDBC Sources
Importing JDBC Sources
Database Specific Information
Connecting to a JDBC Source with Kerberos Authentication
Connecting to a JDBC Source with OAuth Authentication
Creating Base Views from a JDBC Data Source
Importing Graphically Stored Procedures from a Database
Creating Base Views from SQL Queries
ODBC Sources
Creating Base Views from an ODBC Data Source
Enabling the Support for ODBC Sources When the Virtual DataPort Server Runs on Linux
Enabling the Support for ODBC Sources When Using an External JRE
SOAP Web Service Sources
Importing SOAP Web Service Sources
Creating Base Views from a SOAP Web Service
Multidimensional Database Sources
Creating a Base View Over a Multidimensional Data Source, Graphically
Creating a Base View Over a SAP Bex Query (only for SAP BW)
Creating a Base View Over a Multidimensional Data Source with an MDX Query
Multidimensional Data Sources to Oracle Essbase
Multidimensional Data Sources Push Down Capabilities
Path Types in Virtual DataPort
Local Path
Path From Variable
HTTP Path
Denodo Browser Path
FTP / SFTP / FTPS Path
HDFS Path
S3 Path
Hadoop-Compatible Storage
Compressed or Encrypted Data Sources
Custom Input Filters
Paths and Other Values with Interpolation Variables
XML Sources
Direct
From OpenAPI 3 Document
JSON Sources
Direct
From OpenAPI 3 Document
Delimited File Sources
Examples of How to Define a Tuple Pattern
Paths Using Date Ranges
Excel Sources
Size Limit When Processing Excel Files
Use Temporary Files to Load Excel Files
Reading an Excel File with Multiple File Entries
Fields Detected as Text During Introspection with Another Format
Web Sources (WWW)
LDAP Sources
BAPI Sources
Salesforce Sources
Registering Denodo as a Connected Application
Creating a Salesforce Data Source
Creating Base Views from a Salesforce Data Source
Creating Base Views from SOQL Queries
Binary Fields
Limitations of Salesforce
Additional Settings
Bulk Mode
Object Storage data in Parquet Format
How to Connect to a Kerberized HDFS from Embedded MPP Data Sources
From Object Storage vs From MPP Catalogs
Schema Evolution
Source Refresh on views from Object Storage
Manage Views Created from Parquet Files
Custom Sources
Using Interpolation Variables as Custom Sources’ Input Parameters
Viewing the Schema of a Base View
View the VQL of an Element
Used By…
Source Type Properties
Data Source Configuration Properties
Source Refresh
Working with Blob Fields of Base Views
Creating Derived Views
Creating Union Views
Creating Extended Union Views
Creating Standard Union Views
Creating Partitioned Union Views
Creating Join Views
Join Conditions with Similarity Operators
Creating Selection Views
Creating Conditions with the Compound Values Editor
Parameters of Derived Views
Creating Flatten Views
Creating Intersection Views
Creating Minus Views
Creating Interface Views
Viewing the Schema of a Derived View
Tree View
Data Lineage
Advanced Configuration of Views
Configuring the Cache of a View
Search Methods, Configuration Properties and Internationalization
Indexes of Views
Statistics of the View
Defining the Data Movements of the View
Materialized Tables
Temporary Tables
Querying Views
Execution Trace of a Statement
Editing / Replacing a View
Modifying Base Views
Modifying Derived Views
Views Affected by Modifications
Stored Procedures
Importing Java Stored Procedures
Executing Stored Procedures
Use of Stored Procedures in Creating Views
Publication of Web Services
Publishing SOAP Web Services
Operations Tab
Settings Tab (SOAP)
Advanced Tab (SOAP)
Publishing REST Web Services
Resources Tab
Settings Tab (REST)
Advanced Tab (REST)
XSLT Transformations
Web Services Authentication
HTTP Basic
HTTP Basic with VDP
HTTP Digest
HTTP SPNEGO (Kerberos)
OAuth 2.0 and OpenID
SAML 2.0
Impersonating a User
How Web Services Query the Virtual DataPort Server
Connection from the Web Services to the Server
Web Service Container Status Table
Invoking Denodo Web Services
Invoking Web Services with SAML Authentication
Types Conversion Table for REST / SOAP Published Web Services
Privileges Required
RESTful Architecture
Primary Keys of Views
Associations
Creating an Association
Multiplicity of Associations
Referential Integrity in Associations
Role Preconditions
Why You Should Define Associations Between Views
RESTful Web Service
Representations
Input Parameters of the RESTful Web Service
IDU Requests
Requests with Input Compound Values
Status Codes
Obtaining the Number of Rows of a Result Set
Security and Access Privileges
Configuring the RESTful Web Service
Denodo OData 4.0 Service
Serving Metadata
Querying Data: The Basics
Navigating Associations
Advanced Querying
Pagination
Configuration
Entity Collection Streaming
Debug Option
Privilege Requirements
Limitations
Features
Denodo GraphQL Service
Configuration
Privilege Requirements
Schema
Introspection
Basic Querying
GraphQL Names
Advanced Querying
Authentication
Resource Limitations
Limitations
Features
JMS Listeners
Creating JMS Listeners
Privileges Required
Advanced JMS Configuration
Acknowledge Mode
Acknowledge On Query Finish and Serialized Processing
Enabling Advanced Logging for JMS Listeners
Kafka Listeners
Creating Kafka Listeners
Privileges Required
Advanced Kafka Configuration
Remote Tables
Managing Remote Tables
Creating Remote Tables
Editing Remote Tables
Refresh Data
Dropping Remote Tables
Remote Tables vs Other Mechanisms for Persisting Data
Server Configuration
Configuring the Cache
Limiting the Number of Concurrent Requests
Credentials Vault
Enabling Access to the Credentials Vault
Supported Credentials Vaults
How A Data Source Obtains the Credentials from the Vault
Specify a Pattern as Secret Name, Parameterized by the Login
Configuring the Default Internationalization
Default Configuration of HTTP Proxy
Identifiers Charset
Restricted Mode
Unicode Mode
Configuring the Memory Usage and Swapping Policy
Memory Usage Settings
Swapping Settings
Maximum Size of a Query
Automatic Out Of Memory Recovery
Privileges
The FILE Privilege
Global Settings of REST Web Services
Enable/Disable the OpenAPI Endpoints
Accessing the REST Web Services Through a Reverse Proxy
Changing the HTTP Header “Content-type”
Changing the HTTP Header “WWW-Authenticate”
Obfuscating that the REST Web Services are Published by the Denodo Platform
Queries Optimization
Server Authentication
LDAP Authentication
Kerberos Authentication
SAML Authentication
OAuth Authentication
Denodo Security Token Authentication
Server Connectivity
Configuring Runtime Parameters for Stored Procedures
Threads Pool
Storing the Metadata on an External Database
Replication Process
Databases Supported
Considerations About This Feature
How to Configure Virtual DataPort to Store the Metadata on an External Database
Memory Management
Streaming Vs Non-Streaming Operators
Swapping Parameters
Limit the Maximum Amount of Memory of a Query
Edge Cases in Streaming Operation
Cache Load Processes
Resource Manager
Defining a Plan
Defining a Rule
Logging Resource Manager Actions
Exporting and Importing the Server Metadata
Exporting the Server Metadata
Exporting and Importing Elements Across Different Environments
Exporting Environment-Dependent and Independent Elements to Different Files
Export to a File with Properties
Importing Metadata into a Server
Importing Extensions
Importing a JDBC Driver
Importing a JDBC Driver for a Database that is not Listed
Importing Other Types of Elements
Using a Script to Upload Extensions
Cache Module
Cache Modes
Partial Mode
Full Mode
Incremental Mode
Recommended Parameters for Queries that Load the Cache
Caching Very Large Data Sets
Caching the Result of Queries that Fail
Cache Maintenance Task
Maintaining the Underlying Database
Specific Information about Cache Databases
Amazon Athena
Azure SQL Data Warehouse
Google BigQuery
Kudu
PrestoDB/Trino
Embedded Denodo MPP
Spark
Yellowbrick
Databases Using a Distributed Object Storage
Generic Support for Other Databases
Table Creation Templates Management
Creating or Editing the Table Creation Template On the Data Sources
Configure the Cache Table Creation Templates
Customize the Cache Management Tables Template
Creating or Editing the Table Creation Template On Views
Template Parameters
Internal Parameters List
Version Control Systems Integration
VCS Configuration
Virtual DataPort Server Configuration
Environment Management
Database Configuration
Importing an Existing Database from a VCS Server
VCS Integration Features
Supported Element Types
Integration with the Server Explorer
VCS Operations for Microsoft TFS and Subversion
VCS Operations for GIT
Delete Operations
Other Version Control Operations
Dependencies Management
VCS Settings of the Administration Tool
Scenarios and Recommended Uses
Picking a VCS Workflow
Centralized Workflow with Shared Databases
Distributed Workflow
Centralized Workflow with Private Databases
Best Practices to Manage Global Elements
Promoting Changes from Development to Testing and Production
Using VCS Environments
Best Practices When Using the Integration with a VCS
Tagging and Branching Releases
Recommendations for the Testing Environment
Uniqueness Detection
Enabling Uniqueness Detection
Examples
LS Optimization for Subversion
Activating the LS Optimization
Tested Environments
Databases, Users and Access Rights in Virtual DataPort
Databases in Virtual DataPort
User and Access Rights in Virtual DataPort
Users
Roles
Types of Access Rights
Privileges at Database Level
Fine-Grained Privileges at View Level
Column Privileges
Row Restrictions
Modeling Fine-Grained Role Restrictions on Multiple Roles
Privileges are Cumulative, not Exclusionary
Data Available to a User with Roles R1,R2 is Equals to the Data Available to R1 + the Data Available to R2 in General but NOT Always
The execution engine only considers the privileges in roles with execution privilege on the query views
Difference in Behavior since Denodo 8.0u20220126
Limit Indirect Access to Views Shared with Other Development Teams
Limit Indirect Access to Shared Views Using Global Security Policies
Limit Indirect Access to Shared Views Using the Indirect Access Privilege
Administration of Databases, Users, Roles and Their Access Rights
Creating Databases
Configuring and Deleting Databases
Creating Users
Modifying and Deleting Users
Creating Roles
Privilege Auditing
Logging Details when Processing Privileges
Impersonation
Global Security Policies
Tags Management
Removing Tags
Privileges Required to Manage and Assign/Unassign Tags
Global Security Policy
Global Security Policies Management
Creating a Global Security Policy
Custom Masking Expression with Wildcard Tag Name
Privileges
Import Tags from External Catalog
Monitoring the Virtual DataPort Server
Monitoring with a Java Management Extensions (JMX) Agent
Using JavaTM VisualVM
General Information on the Server
Information on Data Sources
Information on the Cache
Information and Events on Catalog Access (DDL Statements)
Information and Events on the Running of Statements
Information and Events on Transactions
Setting the User Agent of an Application
Setting Cloud Infrastructure Properties
Denodo Monitor
Local Monitors
Server Monitors
Virtual DataPort Monitors
Configuring the Denodo Monitor
Launching the Denodo Monitor
FinOps
Cluster Architectures / Server Backup
Denodo Tools
How to Check If a Virtual DataPort Server Is Alive
Using the Ping Script
Alternatives to the Ping Script
Using the Import/Export Scripts for Backup And/or Replication
Export Script
Import Script
Bulk Data Load
AlloyDB for PostgreSQL
Amazon Athena
Amazon Aurora MySQL
Amazon Aurora PostgreSQL
Amazon Redshift
DB2
Databricks
BigQuery
Hive
Impala
Data Load to Impala Using Built-in Implementation
Data Load to Impala Using External Hadoop Installation
Kudu
Steps Virtual DataPort Performs to Bulk Load Data into Kudu
MySQL
Oracle
PostgreSQL
PrestoDB/Trino
Denodo Embedded MPP
Snowflake
Spark
Microsoft SQL Server and Azure Synapse SQL
Teradata
Yellowbrick
Settings of the Generation of the Temporary Files
Generation of Temporary Files for a Hadoop-compatible Distributed Object Storage
Bulk Data Load on a Distributed Object Storage like HDFS or S3
Built-in Implementation
External Hadoop Installation
Optimizing Queries
Optimizing DF Data Sources
Optimizing Join Operations
Merge Join
Nested Join
Nested Parallel Join
Hash Join
Automatic Simplification of Queries
Removing Redundant Branches of Queries (Partitioned Unions)
Pushing Down GROUP BY Views Below JOIN Views
Pushing Down GROUP BY Views Below UNION Views
Selecting the Most Optimal Source When the Data Is Replicated in Several Sources
Cost-Based Optimization
Enabling the Cost-Based Optimization
Gathering the Statistics of Views
Tuning the Cost-Based Optimization Process
Current Limitations of the Cost-Based Optimization Process
Data Movement
Data Movements From/To Netezza Databases
Examples of Data Movement
Known Limitations
Options of the CONTEXT Clause that Control a Data Movement
Customize the Data Movement Table Creation Command
Parallel Processing
Embedded MPP Acceleration
Parallel Processing Using an External MPP
Use of an MPP Engine as Cache
Force Movement of Views to the MPP Engine Using the CONTEXT Clause
Smart Query Acceleration Using Summaries
Managing Summaries
Summary Promotions
Summary Rewrite Optimization
Automatic Summary Recommendations
Denodo Embedded MPP
Sizing Recommendations for the Denodo Embedded MPP
Promoting the Denodo Embedded MPP
Import Model
Overview
Data Modeling
RDFS and OWL Ontologies
Usage
Limitations
Tags Supported by the Import Model Tool
Appendix
Supported JDBC Data Sources
Data Sources That Support Merge Data
Using the Databricks API and DBFS for Bulk Loads
Mounting External File Systems on the DBFS
Backward Compatibility Between the Virtual DataPort Server and Its Clients
Mapping Multidimensional Data to a Relational Model
Creating Multidimensional Base Views Over a Multidimensional Data Source
Mapping the Result of MDX Queries
Considerations When Configuring Data Sources with Pass-Through Credentials
Web Services
Configuring the Logging System
Obfuscation of Sensitive Information of Requests
Configuring the Network Interface Restriction in the Denodo Platform
Installing the Denodo Solution for Microsoft SharePoint
Transforming Incoming/Outgoing Soap/Rest Messages with XSLT Stylesheets
JMS Connection Details: JNDI Properties
Apache ActiveMQ 5.6.0
IBM WebSphere MQ 7.0
Progress SonicMQ 8.0
Execution Trace Information
Useful Tools to Debug Issues with Active Directory or Other LDAP Servers
Single User Mode
Explicit Single User Mode
Automatic Single User Mode
Resource Manager: Available Fields to Evaluate a Rule
Examples of Resource Manager Rules
Values of the Attribute “Access Interface”
Avoiding SQL Injections
Enable the “Export” Endpoint of the Web Container
Publication of Views as Widgets
Publish a View as a Widget
Auxiliary Web Services
Export to JSR-168 or JSR-286 Portlet
Export as Microsoft Web Part
Deployment of a Microsoft Web Part
Customizing Events and Public Render Parameters of JSR-286 Portlets
Launching Several Virtual DataPort Instances of the Same Installation
How to Configure the Secondary Instances
Deploying Web Services on This Configuration
Change Password Script (Design Studio)
Restriction of Number of Data Sources
Configure Export Key Script
OpenAPI 3 Data Sources FAQ
Change Password Script (VDP)
Virtual DataPort Developer Guide
Preface
Scope
Who Should Use This Document
Summary of Contents
Introduction
Examples
Access Through JDBC
JDBC Driver Parameters
Connecting to Virtual DataPort Through a Load Balancer
Connecting to Virtual DataPort Using Kerberos Authentication
When the Client Application Does Not Belong to the Domain
Connecting to Virtual DataPort Using OAuth Authentication
Details of the JDBC Interface
Description of Views and Their Fields
Retrieving the Content Type of Blob Values
Working with Datetime Values with the Denodo JDBC Driver
Obtaining the Names of Elements Inside a Struct (Register)
Connecting from Applications that Filter Incoming Serialization Data
Queries with Datetime Literals
Get Auto-Generated Keys in IDU Statements
Enabling JDBC Logging
Access Through ODBC
Configuration of the ODBC Driver on Windows
Install the ODBC Driver on Windows
Set Up a DSN on Windows
Install Multiple Versions of the ODBC Driver on Windows
Configuration of the ODBC Driver in Linux and Other UNIX
Obtain the Appropriate ODBC Driver
Install UnixODBC
Register the Denodo ODBC Driver in UnixODBC
Register a Data Source (DSN) on UnixODBC
Compiling UnixODBC
Troubleshooting Issues
Creating a DSN-Less Connection
Use Kerberos Authentication
Use OAuth Authentication
Details of the ODBC Interface
How the ODBC Interface Reports the Datetime and Interval Data Types
Integration with Third-Party Applications
Supporting Queries with Brackets
Maximum Length of Text Values
Maximum Length of Error Messages
Disabling Transactions
Limit the Number of Incoming ODBC Connections
Backward Compatibility of the ODBC Driver
Access Through an ADO.NET Data Provider
Using Kerberos Authentication
Access Through OLE DB
Developing Extensions
Developing Custom Functions
Creating Custom Functions with Annotations
Creating Custom Functions Using Name Conventions
Compound Types
Custom Function Return Type
Getting Information About the Context of the Query
Dealing with Datetime and Interval Types
Developing Java Stored Procedures
Using Datetime Values in Denodo Stored Procedures
Required Libraries to Develop Stored Procedures
Developing VQL Stored Procedures
Developing Custom Wrappers
Required Libraries to Develop Custom Wrappers
Extending AbstractCustomWrapper
Overriding AbstractCustomWrapper
Dealing with Conditions
Dealing with the ORDER BY Clause
Configuring a Custom Wrapper
Updating the Custom Wrapper Plan
Dealing with Datetime and Interval Types
Developing Custom Input Filters
Required Libraries to Develop Custom Filters
Developing Custom Filters
Custom View Policies
Developing a Custom View Policy
Appendix
Output Schema of the LIST Command
Output Schema of the DESC Commands
Error Codes Returned by Virtual DataPort
Compile the Denodo ODBC Driver
Pre-requisites to Compile the ODBC Driver on Linux
Compile the ODBC Driver (Standard Method)
Compile the ODBC Driver to Obtain Forwardable Tickets
Troubleshooting the Compilation of the Denodo ODBC Driver
Virtual DataPort VQL Guide
Preface
Scope
Who Should Use This Document
Summary of Contents
Introduction
General Overview of Virtual DataPort
Creating or Defining Data
Defining Base Views
Defining Data Sources and Wrappers
Defining the Views of the Global Schema
Executing Statements
Language for Defining and Processing Data: VQL
Element Names (Identifiers)
Using Back Quotes
Object Descriptions
Data Types
Data Types for Dates, Timestamps and Intervals
Internationalization
Statements
Logical Operators
Comparison Operators
Comparing Literals
Three-valued Logic
Concatenation Operator
Functions for Conditions and Derived Attributes
Syntax Conventions
Syntax of Functions and Condition Values
Comments
Functions
Aggregation Functions
AVG
COUNT
FIRST
GROUP_CONCAT
LAST
LIST
MAX
MEDIAN
MIN
NEST
STDEV
STDEVP
SUM
VAR
VARP
Analytic Functions
AVG
COUNT
CUME_DIST
DENSE_RANK
FIRST_VALUE
LAG
LAST_VALUE
LEAD
LISTAGG
MAX
MIN
NTILE
PERCENTILE_CONT
PERCENTILE_DISC
PERCENT_RANK
RANK
ROW_NUMBER
SUM
STDEV
Workaround to Execute Analytic Functions
Conversion Functions
ARRAY_TO_STRING
CAST
CREATETYPEFROMXML
REGISTER
Datetime Functions
ADDDAY
ADDHOUR
ADDMINUTE
ADDMONTH
ADDSECOND
ADDWEEK
ADDYEAR
AT TIME ZONE
CONVERT_TIMEZONE
CURRENT_DATE
CURRENT_TIMESTAMP
EXTRACT
FIRSTDAYOFMONTH
FIRSTDAYOFWEEK
FORMATDATE
GETDAY
GETDAYOFWEEK
GETDAYOFYEAR
GETDAYSBETWEEN
GETHOUR
GETMILLISECOND
GETMINUTE
GETMONTH
GETMONTHSBETWEEN
GETQUARTER
GETSECOND
GETTIMEINMILLIS
GETWEEK
GETYEAR
LASTDAYOFMONTH
LASTDAYOFWEEK
LOCALTIME
LOCALTIMESTAMP
MAX
MIN
NEXTWEEKDAY
NOW
PREVIOUSWEEKDAY
SUBTRACT
TO_DATE
TO_LOCALDATE
TO_TIME
TO_TIMESTAMP
TO_TIMESTAMPTZ
TRUNC
JSON Functions
AVRO_TO_JSON
COMPLEX_TYPE_TO_JSON
JSONPATH
JSON_TO_AVRO
JSON_TO_COMPLEX_TYPE
Numeric Functions
ABS
ACOS
ASIN
ATAN
ATAN2
CEIL
COS
COT
DEGREES
DIV
EXP
FLOOR
LN
LOG
MAX
MIN
MOD
MULT
PI
POWER
RADIANS
RAND
ROUND
SIGN
SIN
SQRT
SUBTRACT
SUM
TAN
TRUNC
Other Functions
COALESCE
CONTEXTUALSUMMARY
GETVAR
GETSESSION
HASH
IS_PROJECTED_FIELD
MAP
NULLIF
ROWNUM
Text Functions
ASCII
CHAR
CONCAT
INSTR
LEN
LEFT
LOWER
LTRIM
MAX
MIN
POSITION
REGEXP
REGEXP_COUNT
REMOVEACCENTS
REPEAT
REPLACE
REPLACEMAP
RIGHT
RTRIM
SIMILARITY
SPLIT
SUBSTRING / SUBSTR
TEXTCONSTANT
TRIM
UPPER
XML Functions
XMLQUERY
XPATH
XSLT
Converting XML Data into Virtual DataPort Compound Types
Generating Wrappers and Data Sources
Creating Data Sources
JDBC Data Sources
ODBC Data Sources
Multidimensional Data Sources
Data Sources for SOAP Web Services
XML Data Sources
JSON Sources
DF Data Sources
LDAP Data Sources
BAPI Data Sources
Salesforce Data Sources
Custom Data Sources
Data Source Configuration Properties
Specifying Paths in Virtual DataPort
Creating Wrappers
Execution Context and Interpolation Strings
Wrapper Metadata
JDBC Wrappers
Multidimensional Databases Wrappers
ODBC Wrappers
WWW Wrappers
Web Services Wrappers
XML Wrappers
JSON Wrappers
DF Wrappers
LDAP Wrappers
BAPI Wrappers
Salesforce Wrappers
CUSTOM Wrappers
Wrapper Configuration Properties
Valid Conversions Between Types in Wrappers and VDP Types
Native-type Conversions of a Wrapper to Java Types
Filters
Creating a Base View
Query Capabilities: Search Methods and Wrappers
Query Constraints
Assigning Wrappers to Search Methods
Example of How a Search Method Is Created
Modifying a Base View
Queries: SELECT Statement
FROM Clause
JOIN Operation
INTERSECT Operation
MINUS Operation
FLATTEN View (Flattening Data Structures)
Subqueries
WITH Clause
SELECT Clause
Derived Attributes
WHERE Clause
Conditions with Compound Values
Group BY Clause
Use of Aggregation Functions
HAVING Clause
UNION Clause
ORDER BY Clause
OFFSET, FETCH and LIMIT
CONTEXT Clause
TRACE Clause
CASE Clause
Defining a Derived View
Modifying a Derived View
Defining an Interface View
Defining the Statistics of a View
Materialized Tables
Creating Materialized Tables
Inserting Data into Materialized Tables
Temporary Tables
Creating Temporary Tables
Privileges Required
Remote Tables
CREATE REMOTE TABLE Command
Summaries
CREATE SUMMARY VIEW Command
Inserts, Updates and Deletes Over Views
INSERT Statement
UPDATE Statement
DELETE Statement
Return Modified Rows
Limitations
Use of WITH CHECK OPTION
Refreshing Views
REFRESH Statement
Transactions in Virtual DataPort
Stored Procedures
Importing a Stored Procedure
Invoking Denodo Stored Procedures
Predefined Stored Procedures
CACHE_CONTENT
CATALOG_ELEMENTS
CATALOG_FKS
CATALOG_METADATA_VIEWS
CATALOG_PERMISSIONS
CATALOG_PKS
CATALOG_VDP_METADATA_VIEWS
CATALOG_VIEWS
CHECK_METADATA
CLEAN_CACHE_DATABASE
COLUMN_DEPENDENCIES
COMPACT_CACHE
COMPACT_HADOOP_CACHE
COMPACT_METADATA_TABLES
COMPUTE_SOURCE_TABLE_STATS
CREATE_REMOTE_TABLE
CREATE_SCHEMA_ON_SOURCE
CREATE_TAGS_FROM_COLLIBRA
CREATE_TAGS_FROM_VIEW
DISCOVER_OBJECT_STORAGE_MPP_TABLES
DROP_REMOTE_TABLE
DROP_SCHEMA_ON_SOURCE
DUAL
GENERATE_SMART_STATS_FOR_FIELDS
GENERATE_SMART_STATS_FOR_FIELDS_BY_TABLENAME
GENERATE_STATS
GENERATE_STATS_FOR_FIELDS
GENERATE_VQL_TO_CREATE_JDBC_BASE_VIEW
GET_ACTIVE_LOGGERS
GET_ASSOCIATIONS
GET_AVAILABLE_STATS_MODES
GET_CACHE_COLUMNS
GET_CACHE_CONFIGURATION
GET_CACHE_TABLE
GET_CATALOG_EFFECTIVE_PERMISSIONS
GET_CATALOG_METADATA_WS
GET_DATABASES
GET_DELEGATED_SQLSENTENCE
GET_ELEMENTS
GET_EXPORTED_KEYS
GET_FOREIGN_KEYS
GET_ICEBERG_VIEW_SNAPSHOTS
GET_JDBC_DATASOURCE_TABLES
GET_PARAMETER
GET_PRIMARY_KEYS
GET_PROCEDURE_COLUMNS
GET_PUBLIC_VIEW_DEPENDENCIES
GET_SERVER_CONNECTIVITY
GET_SESSIONS
GET_SOURCE_CHANGES
GET_SOURCE_COLUMNS
GET_SOURCE_TABLE
GET_STATS_FOR_FIELDS
GET_TYPE_ATTRIBUTES
GET_USER_ACCOUNTS
GET_USER_DEFINED_TYPES
GET_VIEW_COLUMNS
GET_VIEW_INDEXES
GET_VIEW_STATISTICS
GET_VIEW_TAGS
GET_VIEWS
LIST_JDBC_DATASOURCE_TABLES
LOGCONTROLLER
MAINTAIN_METADATA_TABLES
MIGRATE_DATE_TYPES
PING_DATA_SOURCE
REFRESH_EMBEDDED_MPP_TABLES_METADATA
REFRESH_SOURCE_TABLE_METADATA
ROLLBACK_ICEBERG_VIEW_TO_SNAPSHOT
SOURCE_CHANGES
VALIDATE_MPP_LICENSE
VIEW_DEPENDENCIES
WAIT
WEBCONTAINER_ELEMENTS
WEBCONTAINER_ELEMENT_STATUS
WEBCONTAINER_META_INF
WRITELOGERROR
WRITELOGINFO
Tags
Defining Other Elements of the Catalog
Creating Folders
Defining a Data Type
Defining a Map
Defining JAR Extensions
Defining JMS Listeners
Defining Kafka Listeners
Creating Databases, Users, Roles and Access Privileges
Creating and Modifying Virtual DataPort Databases
Managing Users
Modifying and Deleting Users
Connecting to Another Database
Modifying the Privileges of a User
Managing User Roles
Ownership of Elements
Global Security Policies
Describing Catalog Elements
Exporting Metadata
Importing Metadata
Listing Elements in the Catalog
Removing Elements from the Catalog
Publication of Web Services
Creating SOAP Web Services
Modifying SOAP Web Services
Creating REST Web Services
Modifying REST Web Services
Connection Parameters
Web Services Authentication
Basic and Digest
SAML 2.0
VDP
Embedded Web Container Management
Deploying and Exporting SOAP and REST Web Services
RESTful Architecture
Associations
Navigational Queries
Help Command
Version Control Systems Integration Commands
Statements to Work with Centralized Version Control Systems
VCSCOMMIT
VCSUPDATE
VCSCONTENTS
VCSSHOW
REVISIONS
Statements to Work with GIT
DVCSCOMMIT
DVCSPUSH
DVCSPULL
DVCSREVERT
Commands for Environment Management
Other Version Control Commands
VCSDISCARD
Resource Manager
Managing the Plans of the Resource Manager
Managing the Rules of the Resource Manager
Advanced Characteristics
Management of Compound Values
Processing of Compound Types: Example
Changing Settings of Virtual DataPort and the Web Container
Using the Cache
Cache Invalidation
Cache Table Recreation
Displaying the Contents of the Cache
Cache Table Creation Templates
Test Table Creation Templates
Configuring Swapping Policies
Managing Internationalization Configurations
Replacing an I18n Map
Deleting an I18n Map
Execution Context of a Query and Interpolation Strings
Adding Variables to Selection Conditions (GETVAR and SETVAR)
Appendix
Syntax of Condition Functions
CASE Clause Examples
Date and Time Pattern Strings
Publication of Widgets
Create New Widgets
Export a Widget
Deployment and Export of Auxiliary Web Services
Virtual DataPort API Javadoc
Embedded MPP Guide
Preface
Audience
Overview
Architecture
Kubernetes Deployment
Configuration
Container Image Registry
Container Image Registry Credentials
Private Container Image Registry
External Access to Embedded MPP
Ingress
SSL/TLS
Workers, CPU and Memory
Memory Settings for Queries
CPU and Memory management in Kubernetes
Denodo Server
Denodo with SSL/TLS Enabled
Import Certificate
Catalogs
Hive
Delta Lake
Iceberg
Embedded Hive Metastore
CPU and Memory management in Kubernetes
Embedded PostgreSQL
Persistent Volume
Backup
CPU and Memory management in Kubernetes
External Metastore
External Hive Metastore
Kerberos
AWS Glue Data Catalog
Query Performance Optimization
Object storage with SSL/TLS enabled
Scaling
Autoscaling
Manually
Resource Groups
Deployment
Object Storage
AWS S3
S3-compatible Storage
Azure Data Lake Gen 2
Azure Blob Storage
Google Cloud Storage
HDFS
Embedded MPP at Denodo
Bulk Data Load with the Denodo MPP
Configuration
Cost Based Optimization
Upgrade to a New Version
Cleanup
Troubleshooting
Troubleshooting Deployment
Troubleshooting Usage
Appendix
Other Uses of the Denodo Embedded MPP
Connect from Denodo
Bulk Data Load with the Denodo MPP
Metadata Discovery Tool
Cost Based Optimization
Hive Tables Syntax and Properties
Partition Tables
Delta Lake Tables Properties
Iceberg Tables Properties
Presto CLI
What Is New in Embedded MPP 8.0.20241007
How to Upgrade to this New Version
What Is New in Embedded MPP 8.0.20240505
How to Upgrade to this New Version
What Is New in Embedded MPP 8.0.20240307
Data Catalog Guide
Preface
Scope
Who Should Use This Document
Summary of Contents
Introduction
Installation and Execution
Launching the Data Catalog
Upgrading from 8.0GA or 8.0u20210209
Preparation Tasks Before Installing the Update
Migration Tasks After Installing the Update
How to Recover your Data from the Previous Data Catalog if You Have Already Installed the Update
Introducing the Data Catalog
Search
Metadata Search
Metadata Search in Index
Content Search
Browse
Databases
Categories
Tags
Databases
Summary of Databases
Requests of Databases
Views
Summary of Views
Schema of Views
Query of Views
Building a Query
Executing a Query
Taking Advantage of Associations
Assisted Query
Generating a Query
Executing the Generated Query
Other options
Assisted Query Permissions
Associations of Views
Lineage of Views
Requests of Views
Usage of Views
Data Profiling of Views
Search of Views
Web Services
Summary of Web Services
Schema of Web Services
Schema of SOAP Web Services
Schema of REST Web Services
Query of Web Services
Building a Query
Executing a Query
Taking Advantage of Associations
Requests of Web Services
Usage of Web Services
Workflow Requests
My Requests
Request Management
My Notifications
Administrator Notifications
Email Templates
Queries
My Queries
Deploying Saved Queries
Sharing Saved Queries
Exporting and Importing Saved Queries
VQL Shell
User Settings
Locale Configuration
My Requests
My Notifications
Admin Notifications
Accessibility Mode
Authentication
Authentication with Login and Password
Single Sign-On with Kerberos
Single Sign-On with an Identity Provider
Local Authentication
Authorization
Privileges on the Data Catalog
Catalog Management
Administration
Collaboration
User
Request
Roles with Predefined Privileges
data_catalog_editor
data_catalog_classifier
data_catalog_manager
data_catalog_content_admin
Roles with Implicit Privileges
data_catalog_exporter
data_catalog_admin
Users with Implicit Privileges
Data Catalog Local User
Denodo Global Administrator
Administration
Server Set-Up
Configure the Virtual DataPort Servers
Configure the Connection Settings to the Virtual DataPort Servers
Enable Single Sign-On with Kerberos
Use an External Database for the Data Catalog
Configure the Permissions
Configure the Assisted Queries
Search Engines
Index Servers
Content Search
Metadata Search
Personalization
Elements
Information
Export
Connection
Usage Statistics
Theme
Requests
Categories Management
Create Categories
Edit Categories
Remove Categories
Assign Categories
Tags Management
Create Tags
Edit Tags
Remove Tags
Assign Tags
Elements Management
Property Groups Management
Manage Property Groups
Manage the Custom Properties of a Property Group
Assign Property Groups
Import and Export Data Catalog Metadata
Export Metadata
Import Metadata
Synchronize with Virtual DataPort
Appendix
Installing the Data Catalog on a Cluster: Sharing the Same Settings Across All the Nodes
Setting Up the Common Database
Configure the Data Catalog to Use the Common Database
REST API
HTTP Basic Authentication (stateless)
OAuth Token (stateless)
OAuth Token (Stateful)
CORS Configuration
Apache Lucene Search Syntax
Terms
Fields
Term Modifiers
Boolean Operators
Grouping
Field Grouping
Escaping Special Characters
Data Catalog Scripts
Change Password Script
Configure Export Key Script
Encrypt Password Metadata DB Script
Export Metadata Script
Import Metadata Script
Change Database Settings Script
Data Catalog Guide (up to Update 20210209)
Preface
Scope
Who Should Use This Document
Summary of Contents
Introduction
Installation and Execution
Launching the Data Catalog
Authentication and Authorization
Administration
Servers Configuration
Configure the Virtual DataPort Servers
Configure the Connection Settings to the Virtual DataPort Servers
Use an External Database for the Data Catalog
Content Search Configuration
Kerberos Configuration
Personalization
Elements Shown to the Users
Browse Configuration
User Interface
Export Format Configuration
Database, View and Web Service Connection URIs
Usage Statistics
Catalog Management
Categories Configuration
Tags Configuration
Databases Configuration
Elements Configuration
Property Groups Configuration
Import and Export Data Catalog Metadata
Import and Export Data Catalog Metadata from the UI
Import and Export Data Catalog Metadata Using a Script
Configuration
Browse
Databases
Categories
Tags
Relationships
Views
Summary of Views
Query of Views
Associations of Views
Lineage of Views
Search of Views
Usage of Views
Web Services
Summary of Web Services
Query a Web Service
Saved Queries
Deploying Saved Queries
Copying Saved Queries
Search
Catalog Search
Content Search
Queries
Appendix
Installing the Data Catalog on a Cluster: Sharing the Same Settings Across All the Nodes
Setting Up the Common Database
Configure the Data Catalog to Use the Common Database
REST API
Synchronize Elements with the Server
Compute the Usage Statistics
Export Metadata
Import Metadata
Apache Lucene Search Syntax
Terms
Fields
Term Modifiers
Boolean Operators
Grouping
Field Grouping
Escaping Special Characters
Diagnostic & Monitoring Tool Guide
Preface
Scope
Who Should Use This Document
Summary of Contents
Introduction
General Architecture
Installation and Execution
Launching the Diagnostic & Monitoring Tool of the Denodo Platform
Launching the Diagnostic & Monitoring Tool of the Denodo Solution Manager
Introducing the Diagnostic & Monitoring Tool
Authentication
Authorization
Creating Servers and Environments
Creating Servers
Creating Environments
Monitoring
Monitoring Servers
Monitoring - Resources
Monitoring - State
Monitoring - Sessions
Monitoring - Requests
Monitoring - Cache
Monitoring - Data Sources
Monitoring - Threads
Monitoring Environments
Overview
Creating Diagnostics and Diagnostic Intervals
Creating Diagnostics
Creating Diagnostic Intervals
Diagnosing
Diagnosing - Resources
Diagnosing - State
Diagnosing - Sessions
Diagnosing - Requests
Diagnosing - Cache
Diagnosing - Data Sources
Diagnosing - Threads
Diagnosing - Errors
Export and Import
Export
Import
Configuration
User Configuration
Configuration - General
Configuration - Overview
Configuration - Resources
Configuration - State
Configuration - Sessions
Configuration - Requests
Configuration - Cache
Configuration - Data Sources
Configuration - Threads
Configuration - Errors
Server Configuration
Informative Message
Scheduler Administration Guide
Preface
Scope
Who Should Use This Manual
Summary of Contents
Introduction
General Architecture
Installation and Execution
Administration
Authentication
Scheduler Server Local Authentication
Web Administration Tool Local Authentication
Web Configuration
Kerberos Configuration
Informative Message Configuration
Change Password
Community Resources Configuration
Scheduler Server Configuration
Server Set-up
Management
Import/Export Backups
Index Server
Log Configuration
Creating and Scheduling Jobs
Jobs
Data Sources
Scheduler Index Data Sources
CSV Data Sources
Elasticsearch Sources
JDBC Data Sources
VDP Data Sources
Configuring New Jobs
General Structure of a Job
VDP Extraction Section
VDPCache Extraction Section
VDPDataLoad Extraction Section
VDPIndexer Extraction Section
Exporters Section
Retry Section
Handler Section
Time-based Job Scheduling Section (Triggers Section)
Reports Section
Dependencies Among Jobs
Developer API
Scheduler RMI Client API
Scheduler REST Client API
Public
Private (Deprecated)
Extensions (Plugins)
Exporters
Handlers
Appendix
JDBC Drivers
Use of Configure Scheduler Export Key Script
Use of the Import/Export Scripts for Backup (Scheduler)
Export
Import
Use of the Import/Export Scripts for Backup (Scheduler Index)
Export
Import
Use of the Ping Script (Scheduler)
Use of the Ping Script (Scheduler Index)
ChangePassword Script (Scheduler)
ChangePassword Script (Scheduler Web Administration Tool)
ChangePassword Script (Scheduler Index)
How to Manually Configure the Metadata Database
Upgrade/Migrate Lucene Indexes
Scheduler API Javadoc
Solution Manager Installation Guide
Preface
Who Should Use This Document
Pre-Installation Tasks
Hardware Requirements
Minimum Requirements to Run the Solution Manager On Premise
Minimum Requirements to Run the Solution Manager on Amazon AWS
Recommendations to Run the Denodo Platform on Microsoft Azure
Software Requirements
Supported Operating Systems
Supported Browsers
Other Pre-installation Tasks
Select a User Account to Install the Solution Manager
Check that the Required Ports Are Free
Grant Privileges to This User Account
Check the PATH Environment Variable on Windows
Download the Installer
Using the Graphical Installation Wizard
Commons Settings
Components and Configuration of Virtual DataPort
Components and Configuration of Solution Manager
Embedded Web Container
Using the Command Line Installer
Unattended Installation of the Solution Manager
Modifying the Solution Manager Installer to Include the Latest Update
Unattended Installation of the Solution Manager
Post-Installation Tasks
Install the Latest Update
Install the License
Change the Default Passwords
Solution Manager Server
Design Studio
Enable SSL/TLS in the Solution Manager
Obtaining and Installing an SSL Certificate
Enabling SSL/TLS in Solution Manager Servers
Configure Windows Services
Post-Installation Tasks: Web Container
Enable Authentication on the Monitoring Interface
High Availability
Step #1: Configuring the Solution Manager Installations for High Availability
Step #2: Configuring the Load Balancer for High Availability
Step #3: Configuring Access Through the Load Balancer
JVM Memory Configuration
Denodo Platform Control Center
Starting Platform Servers and Tools
Installing Updates and Hotfixes
Virtual Machine and Web Container Configuration
Configuration of the JVM Parameters from the Command Line
Control Center Help
Uninstalling the Solution Manager
Appendix
Default Ports Used by the Solution Manager Modules
Providing a Krb5 File for Kerberos Authentication
Troubleshooting the Solution Manager Installer
Transparent Metadata Encryption in the Solution Manager
Configuring the Network Interface Restriction in the Solution Manager
Session Configuration (Solution Manager Administration Tool, Design Studio, Scheduler, Diagnostic & Monitoring Tool)
Maximum Session Timeout
Idle Session Timeout
Single-User Concurrent Sessions Control
Solution Manager Administration Guide
Preface
Scope
Who Should Use This Document
Summary of Contents
Introduction
Architecture
Starting the Solution Manager
Introducing the Solution Manager Administration Tool
License Management
Install a License
Check Global License Information
Assign Licenses
Assign a License to a Server
Check the License Information for an Environment
How Licenses Work
How a Server Resolves Its License
Customize hostname and port in license request
How Denodo Servers Behave on License Responses
License Usages Table
Configuration
Setting-Up an External Database
Changing the Password of the External Database
VCS Configuration
Setting-Up the Informative Message
Global Monitoring Configuration
Cloud Storage Appenders
JDBC Appenders
Authentication and Authorization
User Management
Authenticating with LDAP
Enabling the LDAP Roles Cache in Solution Manager
Authenticating with Single Sign-On
SAML Configuration
OAuth Configuration
OpenID Configuration
Kerberos Configuration
Authorization
Global Privileges
Privileges Granted to a Role For an Environment
Role Management
Import Roles from an LDAP Server
Denodo Security Token
Architecture
Configure the Denodo Platform Components to Reach the Security Token Server
Security Considerations
External Identity Provider Considerations
Configure the Connection to the Security Token Server from the Denodo Platform
Configure HTTP Timeouts and Limits for the Request of Public Keys
Load Balancer Considerations
Authentication Credentials
Regenerate Autogenerated Credentials
Single Sign-On Sequence
Environments
Clusters
Working Modes
Differences Wizard
Standard Mode
Creating Environments
Configuring Environments
Creating Standard Clusters
Configuring Clusters
Automated Cloud Mode
Configuring Mode
General
Server Access
Default Region & Images
Creating Environments
Automated Cloud Mode (AWS)
Automated Cloud Mode (Azure)
Configuring Environments
AWS Environment
Azure Environment
Creating Clusters
New AWS Cluster
New Azure Cluster
Configuring Clusters
AWS Cluster
Azure Cluster
Managing Clusters
View the Cluster Summary
View the Progress of the Last Action
Enable a Cluster
Disable a Cluster
View All Cluster Events
Start a Cluster
Stop a Cluster
Restart a Cluster
Recreate a Cluster from an Image
Recreate a Cluster from a Server
Install a Denodo Update in a Cluster
Monitor the Cluster with the Diagnostic & Monitoring Tool
Environment Configuration
Configuring Virtual DataPort Properties
Configuring Deployments
Standard Mode Environments Deployments Configuration
Cloud Environments Deployments Configuration
Data Catalog Server Synchronization
Configuring Deployment Scripts (Standard Mode)
Examples of Deployment Scripts
Configuring Environment Logging Level
Configuring Environment Monitoring
Clusters Configuration
Configuring Cluster Load Balancing Variables (Standard Mode)
Configuring Scheduler Properties
Configuring Cluster Logging Level
Servers
Creating Servers
Configuring Servers
Configuring Server Load Balancing Variables (Standard Mode)
Configuring Server Logging Level
Load Balancing Variables
Creating Load Balancing Variables
Assigning Values to Loading Balancing Variables
Promotions
Creating Revisions
General Options
Additional Options
Revision Candidate Elements
Create Revisions from VQL
Create Revisions from Compare Environments Wizard
Compare Environments Wizard
Revision Candidate Elements
Creating Revisions from Compare Environments Wizard
Revisions Table
Create a Revision
Edit a Revision
Edit a Revision from the Compare Environments Wizard
See the Information of a Revision
Remove Revisions
Filter Revisions in the Table
Download the Metadata of a Revision
Validate a Revision
Check the Validation Summary of a Revision
Copy Revision
Deploy Revisions
Check the Environments Where a Revision Was Deployed
Standard Mode Deployments
Deployment Preconditions
Deployment Paradigms
Cache Considerations
Cloud Deployments
Cloud Deployments Options
Cache Considerations
Considerations for Deployments
Cache
Summaries
Denodo Embedded MPP
Deployments Table
Cancel Deployments
Remove Deployments
Filter Deployments in the Table
Check the Progress Summary of a Deployment
See the Errors of a Deployment
See the Revisions of a Deployment
Download Backup Data for a Deployment
Deployment Progress
Monitoring
Monitoring Configuration
Monitoring Virtual DataPort Servers Running Behind a Load Balancer
Import and Export
Appendix
REST API
Get the List of Environments
Create an Environment
Get the List of Licenses Available
Update an Environment
Delete an Environment
Update AWS Global Credentials
Update AWS Environment Credentials
Update Azure Global Credentials
Update Azure Environment Credentials
Start Automated Mode Cluster
Stop Automated Mode Cluster
Restart Automated Mode Cluster
Recreate Automated Mode Cluster
Recreate From Servers Automated Mode Cluster
Install Denodo Update Automated Mode Cluster
Get the List of Virtual DataPort Properties Associated to an Environment
Create and Delete Environment Properties
Get the List of Clusters
Start Environment Monitoring
Stop Environment Monitoring
Create a Cluster
Get a Cluster
Get Cluster Status
Update a Cluster
Delete a Cluster
Get the List of Scheduler Properties Associated to a Cluster
Create and Delete Cluster Properties
Get the List of Servers
Create a Server
Get a Server
Update a Server
Delete a Server
Export Solution Manager Catalog and Configuration
Import Solution Manager Catalog and Configuration
Get the List of Revisions
Create a Revision from a VQL File
Download the VQL of a Revision
Download the Scheduler ZIP of a Revision
Get the List of Deployments
Start a New Deployment from a List of Revisions
Get the Progress of a Deployment
Free License Usage
Free License Usage by Server Id
Ping License Manager Server
Ping to Solution Manager Server
Best Practices: Deployment of Updates Across Your Organization
Updating Solution Manager with High Availability
Solution Manager Compatibility with the Denodo Platform Servers
Creating A Custom AWS AMI for the Denodo Platform
Creating A Custom Azure Image for the Denodo Platform
Suggested Reading Materials to Use the Automated Cloud Mode
Architecture of a Deployment of Solution Manager in AWS
Architecture of a Deployment of Solution Manager in Azure
Best Practices for Automated Mode Environments
Maximizing Uptime and High Availability
Infrastructure Deployment Considerations
Health Checks
Backup and Recovery
Restoring Backup
Maintenance
Automated Cloud Mode - Creating a Backup
Creating an IAM Policy
Main Features
Automated VPC Management
Automated Subnet Management
Automated Auto Scaling Group Management
Automated Security Group Management
Creating an Azure Custom Role
Main Features
Automated Resource Group Management
Automated Virtual Network Management
Automated Subnet Management
Automated Network Security Group Management
Minimizing Downtime Option
Automated Auto Scale Set Management
Configuration Files of the Monitoring in Solution Manager
JDBC Logging Configuration
Cloud Storing Configuration
Promotion Mode Behavior between Denodo Updates
Denodo4Eclipse Plugin Guide
Preface
Scope
Who Should Use This Document
Summary of Contents
Installing Denodo4Eclipse Plugin
VDP/ITP Server Tutorial
Creating an Extension Project with a Custom Stored Procedure
Creating Debug Configurations for the VDP/ITP Server and the VDP Administration Tool
Starting the VDP/ITP Server in Debug Mode
Debugging the Stored Procedure
Deploying the Stored Procedure
Wrapper Generation Tool Tutorial
Creating an Extension Project with an ITPilot Custom Function
Creating a Debug Configuration for the Wrapper Generation Tool
Starting the Wrapper Generation Tool
Debugging the Custom Function
Deploying the Custom Function
Reference of the Dialogs of the Denodo4E Plugin
Create
The Denodo Extension Project Wizard
The Denodo Extension Wizard
Debug
Denodo Application Debug Configuration
Supported Denodo Applications
Support for Dependencies
Debug Remote Applications
Deploy
The Deploy Wizard
Supported Extension Types
ITPilot
ITPilot User Guide
Preface
Scope
Who Should Use This Document
Summary of Contents
Introduction
Denodo ITPilot Environments
Distribution of Environments
Distribution of the Generation Environment
Distribution of the Execution Environment
Distribution of the Verification Environment
Installation and Initial Configuration
Execution
Starting Up the Administration Tool
Starting Up the Browser Pool
Starting Up the Wrapper Server
Starting Up the Verification Server
Starting Up the Verification Server with a Graphical Tool
Starting Up the PDF Conversion Server
Web Administration Tool
Adding, Editing and Deleting Servers
Configuring the Browser Pool Server
Configuration of the Wrapper Server
Configuring the Verification Server
Verification Server Graphical Configuration and Monitoring Tool
An Overview of the Graphical Tool
Configuration of the Verification Environment
Monitoring the Wrappers Under Verification
Wrapper Execution Trace Logs
Enabling Wrapper Trace Logging
Log Categories and Effective Log Level
Appenders
ITPilot Generation Environment Guide
Preface
Scope
Who Should Use This Document
Summary of Contents
Introduction
Presentation
Development of Component-Based Wrappers
Installation and Execution
Introduction to the Tools
Generation Environment Tools - Part I
Presentation of the Example
Creating Projects and Wrappers
Components in ITPilot
Process Initialization
Web Browsing Automation
Configuration of the Extractor Component
Processing the Retrieved Results
Wrapper Generation, Tests and Exporting
Generation Environment Tools - Part II
Extracting Multipaginated Data
Access to Details Pages
Generation Tool Global Preferences
Tagsets and Scanners
Generating the Data Extraction Specifications Manually
Using the Sequence Debugger
Saving a Flow as a Custom Component
Wrapper Advanced Options: Specific Browser Pool and Locale
Migrating Wrappers Between Generation Environments: Import and Export
Managing Extensions
Generating Navigation Sequences
Description of the Navigation Sequences Generator Interface
Steps for Generating a Navigation Sequence
The SelectFrame Action
The Transpose Table Button and Action
The Cancel Navigation Button
Recording Sequences with Passwords
Properties of the Navigation Bar
XPath Generator Utility
Appendix A: ITPilot Operators
Logical Operators
Comparison Operators
Appendix B: ITPilot Functions
Arithmetic Functions
Text Processing Functions
Functions for Handling Null Values
Functions for Handling Lists
Date Processing Functions
Type Conversion Functions
Functions for URL Processing
Functions for Page Handling
Functions for Handling Compressed Files
Extraction Functions
Case Clause
Appendix C: Catalog of Components
Add Record To List
Condition
Create List
Create Persistent Browser
Diff
Execute JavaScript
Expression
Extractor
Extractor Sequence
Filter
Form Iterator
Get Page
INIT
Iterator
JDBC Extractor
Loop
Next Interval Iterator
Output
Record Constructor
Release Persistent Browser
Repeat
SaveFile
Script
Sequence
Appendix D: Constraints of the Simplified DOM
Appendix E: List of Reserved Words
ITPilot Developer Guide
Preface
Scope
Who Should Use This Document
Summary of Contents
Introduction
ITPilot Development API
Connecting to the Server
Obtaining Wrappers
Using Wrappers
Processing Query Results
Example of Use
Creating Custom ITPilot Functions
Naming Conventions and Annotations
Compound Types
Page Type
Custom Function Return Type
Example
ITPilot DEXTL Guide
Preface
Scope
Who Should Use This Manual
Summary of Contents
Introduction
An Overview of DEXTL
Basic Syntax
Fundamentals of Pattern Specification
Ambiguity of Patterns and Demarcation of the Search Space
Relations and Subrelations
Advanced Syntax
Extracting Tag Attributes
Tags with Evaluated Attributes
Voracious Separators
Dynamic Separators
Attributes with FIXED Value and Default Value
Format Tags
Hierarchy of Elements
Alternative Specifications
Scanners
Structure of a DEXTL Program
Escape Characters in DEXTL
DEXTL Quick Reference Sheets
Putting All Together
Browser View
HTML Code View
FORMAT TAGS VIEW
DEXTL VIEW
RESULTS VIEW
ITPilot NSEQL Guide
Preface
Scope
Who Should Use This Manual
Summary of Contents
Introduction
Basic Concepts of NSEQL
Navigation Sequences: Basic Syntax
NSEQL Commands: General Overview
Configuration of Page Waiting Times
Example
List of NSEQL Commands
Usual Parameters
Error Parameters
Navigation Commands
Commands for Processing Links
Commands for Selecting Forms
Commands for Executing Actions on Elements of a Form
Frame Selection Commands
Commands for Selecting Elements
Commands for Executing on Actions on Elements
Commands for Handling Windows
Commands for Handling Pop-Up Dialogs
Commands for Editing the Browser Properties
Informative Commands
Conversion Commands
Commands for Saving Files
Commands for Executing Code
Commands for Clearing Browser Data
Util Functions
EncodeSEQ
ITPilot API Javadoc
DenodoConnect Components
Apache Zeppelin for Denodo - User Manual
Denodo Cloud Cache Load Bypass Stored Procedure - User Manual
Denodo Custom Wrapper for IBM Cognos TM1 - User Manual
Denodo Custom Wrapper for SAS - User Manual
Denodo Dashboard - User Manual
Denodo Dialect for SQLAlchemy - User Manual
Denodo Dialect for Spark SQL - User Manual
Denodo Distributed File System Custom Wrapper - User Manual
Denodo Dynamics 365 Business Central Templates - Quick Use Guide
Denodo Dynamics 365 Customer Engagement Templates - Quick Use Guide
Denodo DynamoDB Custom Wrapper - User Manual
Denodo Email Exported Files Custom Handler - User Manual
Denodo Embedded MPP - User Manual
Denodo FileSystem CustomWrapper - User Manual
Denodo Geo Service - User Manual
Denodo Google Sheets Custom Wrapper - User Manual
Denodo Google Sheets Tool - User Manual
Denodo Governance Bridge for Collibra - User Manual
Denodo Governance Bridge for IGC - User Manual
Denodo Governance Bridge for Microsoft Purview - User Manual
Denodo GraphQL Custom Wrapper - User Manual
Denodo HBase CustomWrapper - User Manual
Denodo HL7 FHIR Service - User Manual
Denodo Hibernate Dialect - User Manual
Denodo Incremental Cache Load Stored Procedure - User Manual
Denodo Kafka Custom Wrapper - User Manual
Denodo Log Custom Wrapper - User Manual
Denodo MDX Service - User Manual
Denodo Model Bridge - User Manual
Denodo MongoDB Custom Wrapper - User Manual
Denodo OData 2.0 Service - User Manual
Denodo OData2 Custom Wrapper - User Manual
Denodo OData4 Custom Wrapper - User Manual
Denodo Power BI Custom Connector - User Manual
Denodo RFCReadTable Custom Wrapper - User Manual
Denodo SFTP Exported Files Custom Handler - User Manual
Denodo SSH Custom Wrapper - User Manual
Denodo ServiceNow Templates - Quick guide
Denodo Tableau Exporter - User Manual
Denodo Templates for Adobe Analytics - Quick Use Guide
Denodo Templates for Google Analytics - Quick Use Guide
Denodo Templates for Marketo - Quick Use Guide
Denodo Templates for SharePoint - Quick Use Guide
Denodo Templates for Twitter - Quick Use Guide
Denodo Testing Tool - User Manual
Denodo VQL Generation Stored Procedures - User Manual
Denodo XSLT Mapper - User Manual
Denodo XtraFuncs - User Manual
Generic XML Exporter - User Manual
Quick Start Guides
Denodo Automated
Denodo 8.0 Automated Cloud Mode for AWS Quick Start Guide
Denodo 8.0 Automated Cloud Mode for Azure Quick Start Guide
Denodo Enterprise
Denodo 8.0 Enterprise for AWS BYOL Quick Start Guide
Denodo 8.0 Enterprise for AWS Quick Start Guide
Denodo 8.0 Enterprise for AliCloud Quick Start Guide
Denodo 8.0 Enterprise for Azure BYOL Quick Start Guide
Denodo 8.0 Enterprise for Azure Quick Start Guide
Denodo 8.0 Enterprise for GCP BYOL Quick Start Guide
Denodo 8.0 Enterprise for GCP Quick Start Guide
Denodo Enterprise Plus
Denodo 8.0 Enterprise Plus for AWS Quick Start Guide
Denodo 8.0 Enterprise Plus for AliCloud Quick Start Guide
Denodo 8.0 Enterprise Plus for Azure Quick Start Guide
Denodo 8.0 Enterprise Plus for GCP Quick Start Guide
Denodo Professional
Denodo 8.0 Professional Frequently Asked Questions (FAQ)
Denodo 8.0 Professional for AWS BYOL Quick Start Guide
Denodo 8.0 Professional for AWS Quick Start Guide
Denodo 8.0 Professional for AliCloud Quick Start Guide
Denodo 8.0 Professional for Azure BYOL Quick Start Guide
Denodo 8.0 Professional for Azure Quick Start Guide
Denodo 8.0 Professional for GCP BYOL Quick Start Guide
Denodo 8.0 Professional for GCP Quick Start Guide
Denodo Standard
Denodo 8.0 Standard for AWS Quick Start Guide
Denodo 8.0 Standard for AliCloud Quick Start Guide
Denodo 8.0 Standard for Azure Quick Start Guide
Denodo 8.0 Standard for GCP Quick Start Guide
Denodo Platform
Denodo Platform Container 8.0 QuickStart Guide
Denodo Platform for AWS 8.0 BYOL Quick Start Guide
Denodo Platform for AliCloud 8.0 BYOL Quick Start Guide
Denodo Platform for Azure 8.0 BYOL Quick Start Guide
Denodo Platform for GCP 8.0 BYOL Quick Start Guide
Denodo Container Registry Quick Start Guide
Denodo Express 8.0 Quick Start Guide
Denodo Helm Charts Quick Start Guide
Denodo for Cloud Marketplaces 8.0 Frequently Asked Questions (FAQ)
Important:
To see the latest version of the document click here
User Manuals
/
Virtual DataPort VQL Guide
/
Defining Other Elements of the Catalog
Defining Other Elements of the Catalog
¶
The following subsections explain how to create other elements of the catalog:
Creating Folders
Defining a Data Type
Defining a Map
Defining JAR Extensions
Defining JMS Listeners
Defining Kafka Listeners
Feedback:
Share:
Add feedback
