USER MANUALS

What Is New in Denodo 7.0 Update 7.0u20190903

This page lists the main enhancements added in Denodo 7.0 update 20190903. For a full list of enhancements and bug fixes, check the RELEASE NOTES of the update, inside the zip file of the update or in the Support Site.


Virtual DataPort

  • This update includes several performance improvements that reduce the execution time, CPU usage and memory usage of several operations:

    • Calculate the lineage and the tree view

    • Obtain the dependencies of the views

    • Generate the VQL of the views

    • Update the configuration of a data source

    • Generating the SQL queries to executed on a database.

    • Optimizing the execution plan of a query.

    • …

In views with several hundreds of columns, these improvements provide a tenfold reduction in the execution time of these operations.

Note

This enhancement only applies to elements created after you install this update. To benefit from these enhancements for the existing elements, you need to re-import their VQL (e.g. export the VQL of the entire Server and import it again).

  • The query optimizer can now push down the operations GROUP BY and JOIN under an extended UNION (i.e. a UNION of views that have a different schema).

Virtual DataPort (Data Sources)

  • Added support for new databases: Cassandra, Amazon Athena and Elasticsearch 6.7 (Elasticsearch 6.5 was already supported). The appendix Supported JDBC Data Sources of the Administration Guide lists all the officially supported databases.

  • The data sources Salesforce, DF, JSON and XML now support mutual authentication with certificates. That is, using a private key for authentication.

  • Denodo can now gather statistics from the system tables of DB2 for z/OS (list of databases from which Denodo can gather the statistics of tables/views from their system tables).

Virtual DataPort (Cache Engine)

  • Data sources using pass-through credentials can now be used as cache database. This helps maximize the number of queries that the execution engine can execute in the same data source. Before using this feature, read the appendix Considerations When Configuring Data Sources with Pass-Through Credentials of the Virtual DataPort Administration Guide because enabling this option may not be adequate in all scenarios.

Virtual DataPort (Administration Tool)

  • Performance improvements in the communications between Virtual DataPort and the administration tool. These improvements are noticeable in WAN environments and slow connections. To enable this option, open the menu Tools > Admin Tool preferences (tab Connection) and select the check box Use WAN optimized communications. This option is not enabled by default because, when the tool and the server are on the same network, it does not provide any benefit and it could even slow down the connection slightly.

  • Views are opened faster because now, the tool loads some information of the views on demand. E.g. the tool only requests the information about the tree view if the user opens the tree view.

  • Users no longer need to modify the Windows registry to use single sign-on. With previous updates, the user has to modify the registry. To benefit from this feature, you need to update Virtual DataPort and the administration tool.

  • “Execute” dialog and VQL Shell: the options “Limit rows” and “Stop query when the limit is reached” have been renamed to “Display rows” and “Retrieve all rows” respectively to make clearer what these options do.

Virtual DataPort (JDBC Driver)

  • The JDBC driver can now be loaded from client applications that support Java 9, 10 or 11. The driver included with previous updates can only be loaded from applications that run with Java 8.

  • Users no longer need to modify the Windows registry to connect to Denodo with single sign-on. With previous updates, you have to modify the registry. To benefit from this feature, you need to do the following:

    1. Update Virtual DataPort and use the latest JDBC driver.

    2. Add the necessary libraries to the classpath of the client application. See more details in the section Connecting to Virtual DataPort Using Kerberos Authentication of the Developer Guide.

Virtual DataPort (Operations and Management)

  • Now, you can automatically store the execution trace of the queries that fail or all the queries.

  • In this update, Virtual DataPort and its administration tool, have changed the way they store the Kerberos debug information. This information is not printed in the console anymore, instead it is stored in the same file as other logging information (i.e. in the “vdp.log” and “vdp-admin.log” files). This facilitates gathering this information because you do not need to launch the applications differently and each Kerberos message has a timestamp.

    In addition to enabling the “Kerberos debug mode”, you have to edit the file <DENODO_HOME>/conf/vdp/log4j2.xml and add the following line right below the line <Loggers>:

    <Logger name="com.denodo.util.logging.JavaConsoleLogging" level="debug" />

    Restart Virtual DataPort to apply the changes to this file.

    You will find the Kerberos debug messages in <DENODO_HOME>/logs/vdp/vdp.log.

    For the administration tool, you have to edit the file <DENODO_HOME>/conf/vdp-admin/log4j2.xml and do the same change. Then, restart the administration tool to apply the change. You will find the Kerberos messages in <DENODO_HOME>/logs/vdp-admin/vdp-admin.log.

    Important

    This change affects any log messages that with previous updates, Virtual DataPort printed in the standard output. For example, if you add the system property -Djavax.net.debug=all to Virtual DataPort, the output will also be stored in the log file of the component where you added this property.

    Note

    This feature has been added only to Virtual DataPort and its administration tool, not the other modules.


Data Catalog

  • The Data Catalog has new privileges that allow for a more fine-grained control of what users can do:

Tasks

Roles Required in Virtual DataPort To Perform These Tasks

Assign categories, tags and custom properties groups to views and web services.

data_catalog_classifier

Edit views, web services and databases. Create, edit and delete tags, categories, custom properties groups and custom properties.

data_catalog_editor

Can do the same as a user with the roles “data_catalog_editor” and “data_catalog_classifier”.

data_catalog_manager

Configure the personalization options and content search.

data_catalog_content_admin

Can perform any action.

data_catalog_admin

See more details about these roles in the section Administration of the Data Catalog Guide.

  • Better response times of the dialogs “Element Management”, “Tags” and “Content Search Configuration”.

  • Performance improvements in the metadata search and in the metadata synchronization with Virtual DataPort. This is noticeable on servers with a lot of elements.

  • The option Compute usage statistics now updates the data about usage before calculating the statistics.

  • You can now export and import the metadata of the Data Catalog (categories, tags, settings, etc.) programmatically. You can use a script or use the REST API of the Data Catalog. The goal is to facilitate the promotion of the metadata across environments.

  • Added support to store the metadata of the Data Catalog in the PostgreSQL edition of Amazon Aurora.


Enhancements that Apply to Several Components


Solution Manager

  • The REST API of the Solution Manager has new operations that allow you to register and deregister environments, clusters and servers. Before, you had to do it manually from the administration tool.

    With these operations, you can use the auto scaling capabilities of your cloud provider to start and stop Denodo servers when necessary, and register and deregister them on the Solution Manager. The article Configuring Auto Scaling of Denodo in AWS of the Knowledge Base explains how to do this.

    For example, if you have eight Denodo servers on your production environment, you can set a rule to start an additional server when the CPU usage of these servers is higher than 80% for a certain period of time.

  • Deploying revisions that include data sources is easier. When a revision includes properties that are not defined in the target environment (e.g. user/password of a new data source), the user can now choose between using the values of the source environment or entering new values.

  • You no longer have to initialize the Git repository if it is not initialized yet. If necessary, the Solution Manager will initialize it. With previous updates, you have to initialize the Git repository manually before enabling this feature. Otherwise, the stages “backup generation”, “VDP rollback” and “Scheduler rollback” of the deployments will fail if they are enabled.

Add feedback