Denodo Governance Bridge exports Virtual DataPort (VDP) elements, including the lineage, to IBM® InfoSphere® Information Governance Catalog (IGC) to support governance across Denodo databases.
To achieve this, Denodo Governance Bridge, using the Open IGC API, extends IGC by registering new types of assets from the Denodo catalog:
These Denodo assets:
Figure 1 Denodo Governance Bridge architecture
The Denodo Governance Tool bridges the gap between IBM IGC and Denodo VDP metadata repository.
It offers two execution modes:
See the Synchronize VDP with IGC: interactive mode section for a more detailed explanation.
See the Export VDP elements to XML: batch mode section for a more detailed explanation.
Denodo Governance Bridge requires Open IGC, the technology that allows to define new asset types within the Information Governance Catalog, with their own names, icons, custom properties and containment relationships.
Denodo Governance Bridge requires at least:
as these versions are the first that allow the possibility to assign more than one parent type to an asset in Open IGC.
More information about Open IGC is available at
http://www-01.ibm.com/support/docview.wss?uid=swg21699130
The Denodo Governance Bridge distribution consists of:
If you need to use a different Denodo driver version that the one that is distributed, you have to replace this jar by the Denodo driver of the proper version.
For installing the Denodo Governance Bridge, just download the .zip file and extract the tool into the desired folder.
To run it, you need Java 8 and the environment variables JAVA_HOME and PATH correctly configured.
After running the script denodo-governance-bridge.sh|bat in the /bin folder, point your browser to http://localhost:10099 to access the home page:
Figure 2 Denodo Governance Bridge home page
When SSL is enabled in the IGC server, Denodo Governance Bridge has to trust the public key of the IGC server. For this to happen, you need to import the certificate into the trust store of the Denodo Governance Bridge.
keytool -importcert -alias some_alias -file IGC_SERVER_PUBLIC_KEY.cer -keystore jre\lib\security\cacerts -storepass "changeit" -noprompt |
In the interactive mode, the Denodo Governance Bridge registers in IGC the Denodo asset types transparently, before the first synchronization between a VDP database and IGC.
In the batch mode, the user should register, manually, the Denodo asset types before submitting the XML files to IGC.
For this, go to https://<IGC-SERVER>/ibm/iis/igc-rest-explorer, click on bundle and choose the POST /bundles/ to register the Denodo assets: the file is located in the bundle-igc folder of the Denodo Governance Bridge distribution.
Figure 3 Register bundle in IGC Rest explorer
After the registration (manual or transparent), Denodo asset types are displayed in the IGC browser in the Denodo Models group:
Figure 4 Denodo type assets in IGC
First of all, the Denodo Governance Bridge needs the connection details to communicate with the Denodo VDP server:
Once the authentication is complete, the user could choose a database to synchronize with IGC among all the databases the user can connect to.
Take into account that for the elements involved in the synchronization process, the login user should have:
Figure 5 VDP server connection
Then, the process to synchronize a Denodo VDP model with IGC is a five-step process.
Step 1. Users select which database in the VDP server they are interested in.
Figure 6 Databases listing
Step 2. The tool inspects the database and let users choose the elements to export to IGC.
Figure 7 Database elements tree
Figure 8 Database elements selection
Step 3. The tool informs the users about the transitive dependencies for the selected elements. These dependencies will also be exported to IGC.
In the picture below we can see that the derived view web_logs_all_with_blokinfo depends of:
All these VDP elements will be present in IGC at the end of the process.
Figure 9 Transitive dependencies are also synchronized
Step 4. Users identify which IGC Database assets are connected to Denodo JDBC DataSources to complete their lineage path. This is an optional step.
For Denodo JDBC BaseViews and Columns it is possible to identify which other Database assets that have been formerly imported into the Information Server Repository, participate in the lineage report. These other Database assets are Host, Database, Database Schema, Database Table, View and Database Column.
Figure 10 IGC Database assets
To achieve this, the user can identify the Hosts and the Databases in IGC, that are related with each Denodo DataSource of type JDBC that will be exported, as the Denodo Governance Bridge cannot deduce this information from the JDBC URI of Denodo DataSources.
Figure 11 Map Denodo JDBC assets with IGC Database assets
For each Denodo JDBC DataSource that the user maps to IGC Host and Database assets, the Denodo Governance Bridge will export the lineage from its Denodo JDBC BaseViews to IGC Database Tables (or Views ) and from its Denodo Columns to IGC Database Columns--only for Denodo BaseViews that were not built using a SQL statement in Denodo VDP.
Important |
When the property enable.virtual.assets of the conf/application.properties file is false (the default value), the IGC Hosts and Databases entered in this step must refer to existing assets in IGC. In the same way, Database Tables , Views and Database Columns referred by Denodo JDBC BaseViews and Columns, must exist in IGC and belong to those Hosts/Databases; otherwise the IGC REST API will complain with the following error:
Instead, when the property enable.virtual.assets of the conf/application.properties file is true if some of the IGC Database assets referred by Denodo assets does not exist in IGC a virtual asset will be created in IGC. Virtual assets are only displayed in lineage reports and in the Usage Information section of the assets Details page. The icons for virtual assets are lighter-colored icons than the ones which are used for assets in the catalog. |
Step 5. Users give the details to connect to an IGC server, using Basic Authentication under the hood.
The registration of the Denodo bundle in IGC is done transparently by the Governance Bridge, before the first synchronization between a VDP database and IGC. Later updates of the Governance Bridge could need to modify the Denodo bundle; the Update Denodo bundle version in IGC? checkbox forces the update of the Denodo bundle in IGC. This checkbox should only be checked the first time a synchronization process is executed after installing a Governance Bridge update.
Figure 12 IGC server connection
In the end, the tool shows all the elements that were synchronized to the IGC, the selected and the dependent ones.
The flow of data that defines the lineage is not explicitly displayed, but it will also be loaded in the IGC server.
Figure 13 Synchronization complete
Let’s show the synchronization process result in the IGC side.
The image below shows the details of the web_logs_all_with_blockinfo derived view:
Figure 14 Derived view in IGC
And, these two images below display the data lineage for the same view, web_logs_all_with_blockinfo.
Figure 15 Derived view lineage in IGC
Figure 16 After drilling down on the derived view lineage
Data lineage view in IGC is similar to the Data lineage view available in the VDP Admin Tool. The following image shows the VDP data lineage for the web_logs_all_with_blockinfo derived view.
Figure 17 Data lineage view in VDP
The batch mode of the Denodo Governance Bridge offers a service endpoint that exports the assets and flows to XML, but it does not communicate with the IBM IGC server. This execution mode does not involve user interaction with its web interface.
This mode is useful in case you want to post-process the XMLs (e.g. enriching them with additional data in an ad-hoc process), before submitting them to the IBM IGC server.
For this batch mode you need that the Denodo Governance Bridge is up and running (script denodo-governance-bridge.sh|bat). Then, execute the script denodo-export-governance-bridge.sh|bat you will find in the bin directory of the distribution:
$ cd denodo-governance-bridge-<VERSION> $ bin/denodo-export-governance-bridge.sh conf/input.json |
This script uses curl for invoking the export service endpoint of the Denodo Governance Bridge. You can check if you have curl installed in your system using the command:
$ curl --version |
If curl is not there you can install it from https://curl.haxx.se/dlwiz/.
The denodo-export-governance-bridge.sh|bat script needs a JSON file with the configuration:
input.json sample file
{ "vdpServerUrl":"//host:port/db", "login":"userlogin", "password":"ENC(encrypted_password)", "outputFile":"dir/file.xml", "startDateISO8601":"2019-06-04T14:20:00+02:00", "endDateISO8601":"date in ISO 8601 format", "mappings": [ {"datasource": "denodo_datasource1", "host":"igc_host1", "database":"igc_database1"}, {"datasource": "denodo_datasource2", "host":"igc_host2", "database":"igc_database2"} ] } |
The parameters in the JSON file are:
The export process will generate two files:
E.g.: if the outputFile is C:/export/denodo_sakila.xml. The result files will be:
Note that the user account running the Denodo Governance Bridge tool needs write privileges in the output folder.
When providing a value for startDateISO8601 and endDateISO8601, the VDP elements created or modified between the two intervals will be exported.
If startDateISO8601 is not provided, all the VDP elements that were created or modified before endDateISO8601 will be exported.
If endDateISO8601 is not provided, all the VDP elements that were created or modified after startDateISO8601 will be exported.
If neither startDateISO8601 nor endDateISO8601 are provided, all the VDP elements of the database will be exported.
This field is mandatory if you want to complete the lineage path between Denodo JDBC DataSources and other IGC Database assets that have been formerly imported into the Information Server Repository. These other Database assets are Host , Database , Database Schema , Database Table , View and Database Column .
For each Denodo JDBC DataSource that the user maps to IGC Host and Database assets, the Denodo Governance Bridge will export the lineage from its Denodo JDBC BaseViews to IGC Database Tables (or Views ) and from its Denodo Columns to IGC Database Columns--only for Denodo BaseViews that were not built using a SQL statement in Denodo VDP.
The output of the script includes the HTTP response headers. You can check the HTTP status code to see if the export process was OK:
$ bin/denodo-export-governance-bridge.sh conf/input.json HTTP/1.1 200 Content-Type: text/plain;charset=UTF-8 Content-Length: 21 Date: Tue, 8 June 2019 15:44:24 GMT Successfully exported |
Or it failed:
$ bin/denodo-export-governance-bridge.sh conf/input.json HTTP/1.1 500 Content-Type: application/json;charset=UTF-8 Transfer-Encoding: chunked Date: Tue, 8 June 2019 15:58:35 GMT Connection: close {"timestamp":1559059115009,"status":500,"error":"Internal Server Error","exception":"org.springframework.jdbc.CannotGetJdbcConnectionException","message":"Could not get JDBC Connection; nested exception is java.sql.SQLException: authentication error: Database 'test' not found","path":"/export"} |
Before submitting the XMLs files to the IBM IGC server the Denodo assets should have been registered in IGC. See Register Denodo Asset Types in IGC section for a detailed explanation.
Assets should be uploaded before flows to IGC. To submit the assets, go to https://<IGC-SERVER>/ibm/iis/igc-rest-explorer, click on bundles, choose POST /bundles/assets and paste the content of the file generated by the exportation process that ends with the suffix: _assets_timestamp.
Figure 18 Upload assets in IGC Rest explorer
To submit the flows, go to https://<IGC-SERVER>/ibm/iis/igc-rest-explorer, click on flows, choose POST /flows/uploads and paste the content of the file generated by the exportation process that ends with the suffix: _flows_timestamp.
Figure 19 Upload flows in IGC Rest explorer
The Denodo Governance Bridge expects encrypted passwords in the input.json file, to appear surrounded by ENC(...). You can compute these values using the Jasypt CLI tools, and use the DENODO_EXPORT_ENCRYPTION_PASSWORD environment variable, or Java system property, to communicate the encryption password to the Denodo Governance Bridge.
This way, you can use encrypted passwords in the input.json file:
... "password":"ENC(s2FdirMK4QORq1HZ6tcTTQ==)" ... |
These are the steps for encrypting passwords:
Your command should look like this:
Take note of the output. Example output: zrass64ls4LIx5hdFoXXyA==.
Before Jasypt: "password":"admin" After Jasypt "password":"ENC(zrass64ls4LIx5hdFoXXyA==)" |
Successive exports of the same database (or folder) accomplished by the Denodo Governance Bridge do not delete views in IGC that no longer exist in Denodo VDP. In these cases the user should delete these views manually in IGC.
The Denodo Governance Bridge exclusively manages Denodo elements and, in case of Denodo JDBC elements, their relation with native IGC Database assets. It is not aware of the presence of other metadata elements for the Denodo data source, elements that could be present in IGC or not, and managed in ways that might not be recognizable by external tools such as the Denodo Governance Bridge.
If this limitation affects your scenario, see the Export VDP elements to XML: batch mode section for an alternative that could fit your needs.