USER MANUALS

Unity Catalog

In case that you already have a Unity Catalog containing Iceberg or Delta Lake tables, (using Uniform) you want to access from the Denodo Embedded MPP, you can use the Unity Catalog as an external Metastore.

This is the recommended way to access Delta Lake tables through the Databricks Unity Catalog: by enabling the Iceberg reads via Uniform. The following limitations exist for all tables with Iceberg reads enabled:

  • Iceberg reads do not work on tables with deletion vectors enabled.

  • Delta tables with Iceberg reads enabled do not support VOID types.

  • Iceberg client support is read-only. Writes are not supported.

To use the Unity Catalog as an external Metastore you have to define a new catalog. The recommended way is using the presto.catalog property in values.yaml. This new catalog will be accessed from the From MPP Catalogs tab.

Create Unity Views From MPP Catalog

Create Unity Views From MPP Catalog

See below for an example of an Iceberg catalog connected to Unity named unity

  catalog:
    #unity: |-
    #  connector.name=iceberg
    #  iceberg.catalog.type=rest
    #  iceberg.rest.uri=https://adb-xxxxxxxx.azuredatabricks.net/api/2.1/unity-catalog/iceberg
    #  iceberg.rest.auth.type=OAUTH2
    #  iceberg.rest.auth.oauth2.token=xxxx
    #  iceberg.catalog.warehouse=external_catalog
    #  iceberg.hadoop.config.resources=/opt/presto-server/etc/catalog/core-site.xml
    #  hive.config.resources=/opt/presto-server/etc/catalog/core-site.xml
    #  hive.pushdown-filter-enabled=true
    #  hive.parquet-batch-read-optimization-enabled=true

You can also define new catalogs creating a properties file in presto/conf/catalog/, e.g., presto/conf/catalog/unity.properties. Although it is preferred to define new catalogs in values.yaml to facilitate version upgrades and management of environment-specific configurations.

Below there is an example of an Iceberg catalog to connect to Unity catalog:

  connector.name=iceberg
  iceberg.catalog.type=rest
  iceberg.rest.uri=https://adb-xxxxxxxx.azuredatabricks.net/api/2.1/unity-catalog/iceberg
  iceberg.rest.auth.type=OAUTH2
  iceberg.rest.auth.oauth2.token=xxxx
  iceberg.catalog.warehouse=external_catalog
  iceberg.hadoop.config.resources=/opt/presto-server/etc/catalog/core-site.xml
  hive.config.resources=/opt/presto-server/etc/catalog/core-site.xml
  hive.pushdown-filter-enabled=true
  hive.parquet-batch-read-optimization-enabled=true

Properties

Property Name

Description

iceberg.rest.uri

REST API endpoint URI (required). Example: https://adb-xxxx.azuredatabricks.net/api/2.1/unity-catalog/iceberg

iceberg.rest.auth.type

The authentication type to use. Available values are NONE or OAUTH2 (default: NONE). OAUTH2 requires either a credential or token.

iceberg.rest.auth.oauth2.token

The Databricks personal access tokens (PATs)

iceberg.catalog.warehouse

The name of the Unity Catalog you want to connect.

Note

Depending on the Object Storage location of the Iceberg tables Unity accesses, you may need to provide credentials for that storage to the Embedded MPP cluster. To do this, you need to provide them using a Kubernetes secret created before deploying the MPP cluster. e.g. If you are using Azure Data Lake Storage Gen2 credentials you need to provide the Azure credentials for the Shared Key authentication.

kubectl create secret generic mpp-credentials
--from-literal=METASTORE_DB_PASSWORD=hive
--from-literal=ABFS_STORAGE_KEY=abfsstoragekey

In this case it’s also necessary to set true the objectStorage.azure.sharedKey.enabled and add the Azure Data Lake Gen 2 Storage Account in the objectStorage.azure.sharedKey.account properties in values.yaml.

Also, you need to add the necessary properties in the prestocluster/presto/conf/core-site.xml:

<property>
 <name>fs.abfs.impl</name>
 <value>org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem</value>
</property>

<property>
 <name>fs.abfss.impl</name>
 <value>org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem</value>
</property>

Supported Operations

Operation

Unity Catalog

Read

Yes

Create/Insert

No

Update

No

Merge

No

Delete

No
Add feedback