Delta Lake¶

Delta Lake is an open table format that extends Parquet data files with a file-based transaction log. This provides ACID transactions and scalable metadata handling for your data lakes.

The Denodo Lakehouse Accelerator includes a predefined catalog named delta connected to the Embedded Hive Metastore, enabling you to access Delta Lake tables immediately.

Here are the default properties for the delta catalog:

connector.name=delta

# connection to the Embedded Hive Metastore
hive.metastore.uri=thrift://hive-metastore:9083
hive.metastore-timeout=20s

hive.config.resources=/opt/presto-server/etc/catalog/core-site.xml

hive.parquet-batch-read-optimization-enabled=true
hive.pushdown-filter-enabled=true

To query Delta Lake tables you have to manually register those tables in the Denodo Lakehouse Accelerator Metastore through the CREATE TABLE sentence. Since the schema and the data file list are located in the Delta Log at the table’s location, you need to provide a dummy column as the schema of the Delta Lake table, to avoid the no columns error in the Metastore:

Register a Delta Lake table in the Denodo Lakehouse Accelerator¶

CREATE TABLE delta.default.orders (
   dummy bigint
) WITH (
  external_location = 'abfs://<file_system>@<account_name>.dfs.core.windows.net/<path>/<file_name>',
  format = 'PARQUET'
);

The WITH clause of the CREATE TABLE can also be used to set other properties on the table. See Delta Lake Tables Properties.

Once the Delta Lake table is registered, you can use the embedded data source in Denodo to create a base view on top of the table using the From MPP Catalogs tab.

Explore Delta Lake tables¶

Query Delta Lakes tables directly¶

Another option is to query the table directly using the table location as the table name without registering it in the Metastore.

Query a Delta Lake table from MPP¶

SELECT * FROM
   delta."$path$"."abfs://<file_system>@<account_name>.dfs.core.windows.net/<path>/<file_name>";

Features¶

The Denodo Lakehouse Accelerator data source provides the following features when treating with Delta Lake tables:

Create base views over existing Hive tables managed by the Embedded or External Metastore via the From MPP Catalogs tab of the Denodo Lakehouse Accelerator data source.
Querying
- Delta protocol version (3, 7) is supported starting with Denodo Lakehouse Accelerator 20241007
Embedded MPP Acceleration

Delta Lake Table Features¶

The following table lists the Delta Lake table features supported by Lakehouse Accelerator:

Feature	Documentation
Column mapping	Delta Column mapping
Deletion vectors	What are deletion vectors?
TimestampNTZ	TimestampNTZType
Liquid clustering	Use liquid clustering for Delta tables
Type widening	Delta type widening
Variant	Variant Data Type
V2 Checkpoints	V2 Checkpoint Spec

Limitations¶

When working with Delta Lake tables through the Denodo Lakehouse Accelerator, be aware of the following current limitations:

No graphical exploration and view creation via the From object storage tab of the Denodo Lakehouse Accelerator data source.
No bulk data load
No full cache mode
No remote tables
No write operations

Supported Operation by Metastore Type¶

The following table summarizes the Delta Lake operations currently supported by the Denodo Lakehouse Accelerator for various Metastore types:

Operation	Hive Metastore	AWS Glue Data Catalog	Unity Catalog (*)
Read	Yes	Yes	Yes
Create/Insert	No	No	No
Update	No	No	No
Merge	No	No	No
Delete	No	No	No

(*) You can read both managed and external Databricks Delta Lake tables with the Denodo Lakehouse Accelerator connected to the Unity Catalog, by enabling Iceberg reads via Uniform in Databricks.

S3 Endpoint Configuration¶

To ensure high-performance data processing, it is strongly recommended to explicitly configure the S3 regional endpoint if your bucket is located outside the default us-east-1 region.

Otherwise, omitting the regional endpoint introduces significant latency when accessing buckets outside the default us-east-1 region. This is both because the bucket has to travel farther physically (e.g., us-west-2 to us-east-1) and because the first time the bucket is accessed, the SDK must make an initial call to AWS to determine the correct bucket region.

When configuring the delta catalog in your values.yaml file, you must set the appropriate endpoint property for the delta predefined catalog (delta.s3Endpoint), or the generic property: hive.s3.endpoint, when defining new delta catalogs under the catalog section.

Use the format: s3.<region>.amazonaws.com (e.g., s3.eu-west-1.amazonaws.com). For a full list of regional endpoints, refer to the AWS S3 Service Endpoints.

Predefined Delta Catalogs

For the predefined Delta catalog already enabled in your configuration, add the endpoint property directly under the delta connector block:

delta:
  s3Endpoint: "s3.us-west-2.amazonaws.com"

New Custom Delta Catalogs

When defining additional delta catalogs under the catalog section, ensure you use the hive.s3.endpoint property:

catalog:
  delta_custom: |-
    connector.name=delta
    hive.s3.endpoint=s3.ap-southeast-1.amazonaws.com
    .
    .
    .

USER MANUALS