USER MANUALS

Upgrading from Denodo 6.0: Export the Settings and Metadata of the Current Installation

This section explains how to export the settings and metadata of each module of the Denodo Platform 6.0.

Important

Important

Before proceeding with the upgrade, install the latest update of the Denodo Platform 6.0. The only supported configuration for an upgrade is with the latest update. If you export from previous updates of 6.0, the settings and metadata may have a format that Denodo Platform 9 cannot import.

If you cannot install permanently the latest update on the current installation, follow these steps:

  1. Stop all the servers of the current installation.

  2. Make a copy of the folder of the current installation.

  3. Install the latest update of Denodo 6.0.

  4. Perform the steps explained in this section.

  5. Delete the folder of this installation and put back in place the folder copied in step ii.

If you do not use a module, you do not need to export its metadata and settings.


Switch to Java 8 Update 161

In the installation of Denodo 6.0, replace the Java Runtime Environment (JRE) included in the Denodo Platform with the JRE version 8 update 161. At the end of this process, you can restore the existing JRE.

Important

If you do not change the version of Java, the rest of this process will fail.

There are several ways of obtaining Java 8 update 161:

  1. If you have access to an installation of Denodo 7.0, this version of Denodo has the necessary version of the JRE for this (you cannot copy the JRE of Denodo 9 because 9 includes Java 17).

  2. Or, download OpenJDK 8 - HotSpot:

    The project AdoptOpenJDK provides reliable OpenJDK binaries for all the platforms.

  3. Or, if you have access to Oracle downloads, download the Java Runtime Environment (JRE) from the Java SE 8 Archive Downloads.

Both the AdoptOpenJDK and the Oracle JRE are supported.

Then, follow these steps:

  1. Rename the current directory <DENODO_HOME_6_0>/jre to jre_1.7.

  2. Place the new version of Java in the directory <DENODO_HOME_6_0>/jre. You have to end up with the file <DENODO_HOME_6_0>/jre/bin/java.

  3. Copy the file <DENODO_HOME_6_0>/jre_1.7/lib/security/cacerts to <DENODO_HOME_6_0>/jre/lib/security/cacerts (replace the file).

    This file is the TrustStore and is necessary so the SSL connections that Denodo opens still work.

  4. Execute this from the command line to make sure the JRE is in the expected path:

On Linux
<DENODO_HOME_6_0>/jre/bin/java -version
On Windows
<DENODO_HOME_6_0>\jre\bin\java.exe -version

You should see something like:

openjdk version "1.8.0_275"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_275-b01)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.275-b01, mixed mode)

or like this:

java version "1.8.0_162"
Java(TM) SE Runtime Environment (build 1.8.0_162-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode)

Switching to the JRE version 8 update 161 is necessary so the passwords of the users and the data sources are exported using the same encryption algorithm that Denodo 9 uses, which is stronger than the algorithm of Denodo 6.0.

If you cannot find the JRE version 8 update 161, you can use a newer update of Java 8 as long as it is the update 255 or lower. You cannot use neither Java 9 nor a newer version.

Virtual DataPort

When exporting the metadata and settings of Virtual DataPort, you are going to obtain two files:

  1. The metadata file. This file contains the CREATE commands for the data sources, views, web services, etc.; the commands to change the settings, etc.

  2. The resources file. This is a file that contains the “resources” of Virtual DataPort. That is:

    • The JDBC drivers used by data sources and that are not included in the Denodo Platform. For example, if you have a Teradata data source, which requires the Teradata driver, the VQL file will include the Teradata driver itself so you do not have to copy it to the new installation.

    • The DenodoConnect components and other extensions you imported using the menu File > Extension management of Design Studio.

    • If you enabled Kerberos, the keytab file and the krb5 file.

    • If you imported the SAP Java Connector (JcO), it will also be included within this file.

    • Other resources like these.

    This file includes these resources encoded in Base64.

    To upgrade to previous versions of Denodo, you had to manually copy the drivers that were not included with Denodo, to the new installation. By generating the resources file, the process of upgrading to Denodo 9 is easier.

Important

During this process, it is mandatory to set the configuration property “com.denodo.exportMigrationCompatibility80” to “true” in several Denodo components (read more about this at the end of this page). Without this property, the files obtained may not be compatible with Denodo 9. In the case of Virtual DataPort, some VQL commands will fail when executed in Virtual DataPort 9.

  1. Open the Administration Tool of Virtual DataPort 6.0 and log in with an administrator account.

  2. Execute this from the VQL Shell:

    SET 'com.denodo.exportMigrationCompatibility80' = 'true';
SET 'com.denodo.exportOnlyResourcesAndJars' = 'true';
SET 'com.denodo.vdb.catalog.exportMigrationCompatibility.migrateDateTypes' = 'true';

    Setting the property migrateDateTypes is not mandatory but recommended. The section Transforming Views Automatically to Use the New Datetime Data Types below explains this property.

  3. Restart the Virtual DataPort server to apply the change to the value of these properties.

  4. Connect to the computer where the Denodo Platform 6.0 is installed, and execute this from the command line:

    • On Windows:

      cd <DENODO_HOME_6_0>
cd bin
export.bat --login <user name> --password "<password>" --file resources_denodo.vql --server "localhost:9999" -P includeJars=yes -P replaceExistingElements=yes

    • On Linux:

      cd <DENODO_HOME_6_0>
cd bin
./export.sh --login <user name> --password "<password>" --file resources_denodo.vql --server "localhost:9999" -P includeJars=yes -P replaceExistingElements=yes

    In this command, replace <user name> and <password> with the credentials an administrator user.

    This script will generate the resources file (resources_denodo.vql) in the folder <DENODO_HOME_6_0/bin. Copy this file to your computer.

  5. Go back to the VQL Shell and execute this:

    -- You do NOT need to restart to apply this change.
SET 'com.denodo.exportOnlyResourcesAndJars' = NULL;
SET 'com.denodo.vdb.catalog.exportMigrationCompatibilityIncludeResources' = 'false';

  6. Execute this from the command line (note that the value of the parameter --file is different than in step #3):

    • On Windows:

      cd <DENODO_HOME_6_0>
cd bin
export.bat --login <user name> --password "<password>" --file metadata_denodo.vql --server "localhost:9999" -P cluster=yes -P includeStatistics=yes -P includeDeployments=yes -P replaceExistingElements=yes

    • On Linux:

      cd <DENODO_HOME_6_0>
cd bin
./export.sh --login <user name> --password "<password>" --file metadata_denodo.vql --server "localhost:9999" -P cluster=yes -P includeStatistics=yes -P includeDeployments=yes -P replaceExistingElements=yes

    This script will generate the metadata file (metadata_denodo.vql) in the folder <DENODO_HOME_6_0/bin. Copy this file to your computer.

  7. If, after executing the script “export”, you see this message:

    MIGRATION TO 8.0 - WARNINGS SUMMARY

    copy the messages that appear right below. You will need this information in the next step of the upgrade process.

    If you do not get this warning, this VQL file can be imported into Virtual DataPort 9 without modifying it.

  8. Execute this from the VQL Shell:

    SET 'com.denodo.exportMigrationCompatibility80' = NULL;
SET 'com.denodo.exportOnlyResourcesAndJars' = NULL;
SET 'com.denodo.vdb.catalog.exportMigrationCompatibilityIncludeResources' = NULL;
SET 'com.denodo.vdb.catalog.exportMigrationCompatibility.migrateDateTypes' = NULL;

  9. Restart the Virtual DataPort server to apply the change to the value of these properties.

Information Self-Service Tool

  1. Log in to the Information Self-Service Tool 6.0 with an administrator account.

  2. Export all the saved queries of all users from Information Self-Service Tool. The section Saved Queries (documentation of Denodo 6.0) provides more details about this step.

Scheduler and Scheduler Index

  1. Edit the file <DENODO_HOME_6_0>/conf/scheduler/ConfigurationParameters.properties and add the following line:

    com.denodo.exportMigrationCompatibility80=true

  2. Edit the file <DENODO_HOME_6_0>/conf/arn-index/ConfigurationParameters.properties and add the following line:

    com.denodo.exportMigrationCompatibility80=true

  3. Restart Scheduler and Scheduler Index.

  4. Log in to Scheduler 6.0 with an administrator account and export all. To do this, go to Configuration > Server configuration and then, click Export. In this page, do this:

    1. Select all projects

    2. Select all the options.

  5. Connect to the computer where the Denodo Platform 6.0 is installed, and execute this:

    • On Windows:

      cd <DENODO_HOME_6_0>
cd tools\arn-index

export.bat -h localhost -p 9000 -l <user name> -P <password> -f scheduler-index_upgrade.zip

    • On Linux:

      cd <DENODO_HOME_6_0>
cd tools/arn-index

./export.sh -h localhost -p 9000 -l <user name> -P <password> -f scheduler-index_upgrade.zip

    In this command, replace <user name> and <password> with the credentials of an administrator user.

    This script will generate the file scheduler-index_upgrade.zip (folder <DENODO_HOME>/bin). This file will contain the configuration and indexes of Scheduler Index. Copy this file to the computer where you installed Denodo Platform 9.

  6. Stop Scheduler and Scheduler Index.

  7. Remove the line

    com.denodo.exportMigrationCompatibility80=true

    from these files:

    • <DENODO_HOME_6_0>/conf/scheduler/ConfigurationParameters.properties

    • <DENODO_HOME_6_0>/conf/arn-index/ConfigurationParameters.properties.

  8. Start Scheduler and Scheduler Index.

Final Steps

Although the Denodo Platform 6.0 is compatible with Java 8, just to make sure that no customizations are lost, restore the Java Runtime Environment version 6 that you replaced at the beginning of this process. That is remove the current folder jre and rename the folder jre_1.7 to jre.

Transforming Views Automatically to Use the New Datetime Data Types

Starting the version 7.0, Denodo provides new types for date and timestamp values: localdate, time, timestamp and timestamptz. These new types provide several advantages over the existing date type (mainly, they behave as indicated in the ANSI SQL standard). However, date is still supported in Denodo 9, to keep backward compatibility.

If you set the configuration property com.denodo.vdb.catalog.exportMigrationCompatibility.migrateDateTypes before exporting the metadata, the export process will analyze the date fields to see if they can be exported to one of the new types:

  • For JDBC base views, this process looks at the field “sourceTypeName” of the statement CREATE WRAPPER JDBC of the base view, and the adapter of the data source of the base view. With this information, it will export the date fields with one of the new types.

  • For non-JDBC base views or derived views, it looks into the property “sourceTypeId” of the field of the CREATE TABLE statement:

    • If the property “sourceTypeId” is DATE, the field is exported as localdate, which is the name of the type in Denodo to represent date values without time (equivalent to type DATE of SQL).

    • If it is TIME, the field is exported as time.

    • If it is TIMESTAMP, the field is exported as date (deprecated). There is not enough information to know if the field is timestamp or timestamptz.

    • If the field does not have a source type, the field is exported as date (deprecated).

In the following cases, the fields of type date will remain with the same type because there is not enough information to select one of the new date types:

  • Non-JDBC base views whose fields do not have the source type id defined.

  • Interface views.

  • Fields of derived views that are an expression that is the result of applying the function TO_DATE.

  • View parameters.

Note

Setting the property “migrateDateTypes” does not modify the metadata of your installation. It only affects how the views are exported to the VQL file.

We suggest setting the property “migrateDateTypes” because, although it may introduce a few backward compatibility issues with existing clients, these can be quickly solved. In the long run, by using the new date types, you will avoid the problems introduced by the fact that the date type in Denodo represents a timestamp with time zone while in many cases the column in the underlying database has a type without time zone.

If you do not set the property “migrateDateTypes” before exporting the metadata, the fields of type date do not change. You can still import the VQL file into the Denodo 9 because the type date is deprecated but is still supported. Later, in Denodo 9, you can use the stored procedure MIGRATE_DATE_TYPES, which makes the same changes on the views as the property “migrateDateTypes”. The only difference is that the procedure has the option to process and transform the views of the databases you want, not all the databases.

Configuration Properties of Virtual DataPort for Upgrades: “exportMigrationCompatibility” and “exportOnlyResources”

During the process of exporting the settings and metadata of each Denodo component, you have to set the property “exportMigrationCompatibility” to “true”. Without this property, the files obtained may not be compatible with Denodo 9.

During the time the property “exportMigrationCompatibility” is “true”, consider this for Virtual DataPort:

  • You can continue running queries and creating/modifying/deleting views.

  • Do not perform any task that involve generating VQL statements. For example, do not create a revision with Solution Manager or export the VQL of any element. This is because some of the VQL statements generated when this property is “true”, are only compatible with version 9.

  • Do not change the global settings of Virtual DataPort.

  • Do not modify data sources.

In the process of exporting the metadata of Virtual DataPort, we also instruct the administrator to do this:

  1. Set the property “com.denodo.vdb.catalog.exportOnlyResources” to “true”.

  2. Execute the script “export.bat/sh” to obtain the “resources file”.

  3. Set this property to “false”.

  4. Execute the script “export.bat/sh” again, to obtain the “metadata file”.

We recommend generating two files (the resources file and the metadata file) so you can modify the metadata file easily. Generally, you will not need to modify the metadata file before importing it into Denodo 9. However, if you do need to modify it and this file also included the resources, many text editors would not be able to open it because the resources use a lot of space (consider that the resources file includes the DenodoConnect components you imported, the JDBC drivers, etc.).

In basic scenarios, you can create only one file that includes both the resources and the metadata. To do this, do not set the property “exportOnlyResources” and only execute the script “export” once.

Alternatives for When You Cannot Install the Latest Update of Denodo 7.0

If you cannot install permanently the latest update on the current installation, there are two alternatives but they are more cumbersome:

  1. If you configured any module of the Denodo Platform to store metadata on an external database, follow these steps:

    1. Do a temporary installation of Denodo 8.0 and install the latest update there (you do not need to set up an external database in this new installation).

    2. Export the metadata of 7.0 from the existing installation and import into this temporary installation.

    3. Do the steps of this page (this is to export the metadata from the temporary installation).

    4. Stop the servers of this temporary installation and delete the folder.

  2. If you did not configure any of the modules to store metadata on an external database, follow these steps:

    1. Stop all the servers of the current installation.

    2. Make a copy of the folder of the current installation.

    3. Install the latest update of Denodo 8.0.

    4. Perform the steps explained in this section.

    5. Delete the folder of this installation and put back in place the folder copied in step #2.

The same applies for the installation of Solution Manager.

Add feedback