Overview

Collibra offers the Collibra Data Intelligence Cloud which is the system of record for data. Collibra delivers an end-to-end, integrated Data Intelligence platform that’s purpose-built to automate data workflows and deliver trusted data insights to users.

In order to connect Collibra to Denodo Platform, Collibra has developed the Denodo to Collibra Integration application. You can download it from the Collibra Marketplace.

Registering Denodo as a data source in Collibra is also available using the Denodo JDBC driver. However, the Denodo to Collibra Integration is the recommended option to retrieve metadata from Denodo, transform, and upsert it to a Collibra Platform instance as assets and complex relations.

Denodo to Collibra Integration

The Denodo to Collibra Integration is a Spring Boot application that retrieves, maps and ingests metadata from Denodo Platform to Collibra.

Prerequisites to run the Denodo to Collibra Integration

• Java JDK 8 (Technical lineage does not work with the latest Java v16)
• Maven
• Spring Boot Integration Library (see the Adding the Collibra Integration Library section)

Configuring Collibra

As the Denodo to Collibra Integration user manual mentions, in order to use the application, the Collibra instance must be customised beforehand.

You need to create or review if it already exists:

• A domain (it must be part of a community) with the type Physical Data Dictionary. This is where the Denodo integration assets are upserted.

The  name does not necessarily have to be “Denodo Data Dictionary”. The domain name you choose, along with the community name, should be included in the integration configuration (see the Configuring the Denodo to Collibra Integration section).

Example:

• A scope. A scope in Collibra is a selection of communities and domains that you can refer to in an assignment for an asset type. You have to create a scope and it must be associated with the Community/Domain that you are going to use in your integration. In order to make an example, we will use the scope "Denodo Scope".

• Asset types to map Denodo elements:

• An Attribute Type, that is going to be used to model derived views:

• A Complex Relation Type used to include the dependency type attribute between views to make the Denodo views lineage available:

• Assignments. An assignment is a collection of components that is assigned to an asset type. You need to apply these assignments to a scope (scoped assignment). A scoped assignment only applies to assets that are located in a domain or community that belongs to the scope, therefore, these assignments should be applied in the created scope.

You should assign the custom scope, in our example Denodo Scope, to the asset types Database, BI Folder, Table and Column. After that, you should add the following characteristics in the custom scope in the corresponding types:

Configuring the Denodo to Collibra Integration

The package com.collibra.marketplace.denodo.util contains the Enum CustomConstants.java. This enum is used to define the custom asset, attribute, relation and complex relation types that are used by the integration. You should update the IDs with the values for these custom types. They are available in the Settings section of the Collibra instance. Note that you must add a column that is hidden by default.

Additionally, under the src/main/resources folder, you have the application.properties file that is used by the Denodo to Collibra Integration. It contains properties that are used to communicate with the Denodo and Collibra Platform instances and other internal application properties. Check their meaning in the Denodo to Collibra Integration documentation and set the appropriate values.

Adding the Collibra Integration Library

The Denodo to Collibra Integration requires the Spring Boot Integration Library to work. This Collibra Integration Library is a set of Java classes that facilitate access to Collibra APIs for the purposes of building custom API-based integrations around the Collibra platform. Prior to deploying this integration, ensure that it is installed in the Maven repository.

For more information, see the Collibra Integration Library documentation.

General view

Physical Data Dictionary and Assets

In the Dashboard you can access your domain.

The assets created after running the integration are available clicking on the domain name, Denodo DD in this example:

You can also see in the Catalog section the table type assets created. Clicking on the Data Dictionary section and filtering by domain name:

Table

When you open an asset with Table type you can see its information.

The Details section shows the table type (Base View or Derived View) and the URL of the view in the Denodo RESTful Web service:

Column

When you open an asset with Column type you can see its technical data type in the Summary and Data Profiling sections:

Lineage

In the Diagram section of a Table or a Column, you can see the lineage. Note that the lineage from the columns does not exist since Denodo Metadata and View Lineage Integration version 1.3.0.

Diagram example:

Note that the exploration to see the lineage data is element by element, but the Preview option, that opens the right side panel, is very helpful to get information about the assets and then, based on this, explore the ones you want to get more data on. For example, if you have this diagram:

As you can see in the image above, clicking on the Denodo View Relation, you can know that the source is actor, therefore, you can explore this asset if you want to see the complete Denodo View Relation.

References

Collibra Marketplace: Denodo to Collibra Integration

Collibra Marketplace: Spring Boot Integration Library