Features¶

The Denodo Embedded MPP data source provides the following features when interacting with Hive tables from Parquet files:

• Graphical exploration and view creation: Explore Parquet datasets, create tables in the MPP and base views in Denodo via the From object storage tab of the Embedded MPP data source.

• External Metastore Integration: Create base views over existing Hive tables managed by an External Metastore (e.g., Hive Metastore) via the From MPP Catalogs tab of the Embedded MPP data source.

  Note: To achieve this, you have to create a new catalog in the Embedded MPP that connects to the External Metastore. This is because the predefined hive catalog is restricted and cannot be accessed from the From MPP Catalogs tab.

• Querying

• Embedded MPP Acceleration

• Bulk data load

• Caching: full cache mode

• Managing Remote Tables

Limitations¶

When working with Hive tables and Parquet files through the Denodo Embedded MPP, be aware of the following limitations:

• Inserts limitations: it is not possible to insert data using bulk load into views created from Parquet files using either the From Object Storage or From MPP Catalogs tabs of the Embedded MPP data source.

  • Allowed insert strategies: You can add single rows by executing the INSERT statement, or multiple rows using the direct load strategy.

  Note

  Understanding load strategies:

  • Bulk Load: This strategy efficiently moves large amounts of data when the source view is not in the Embedded MPP.

    It generates temporary Parquet files and then uploads them those files to the HDFS URI path configured in the Write settings section of the Embedded MPP data source.

  • Direct Load: This strategy loads data directly into the database by executing an INSERT INTO table <SELECT QUERY> statement. Denodo will automatically choose this strategy over bulk load when possible. This strategy is possible as long as Denodo can push the entire <SELECT QUERY> to the Embedded MPP. That is, the query is delegable to the Embedded MPP data source, and both the source and target views are in the Embedded MPP.

Supported Operation by Metastore Type¶

The following table summarizes the Hive operations currently supported by the Denodo Embedded MPP for various Metastore types:

(*) To support write operations, ensure that the Embedded MPP Hive catalog has the following property enabled:

hive.non-managed-table-writes-enabled=true

Be aware of the insert limitations that apply to Hive tables within the Embedded MPP.

Kerberos for HDFS¶

If the Parquet files are located in an HDFS cluster secured with Kerberos, you need to configure Kerberos-related properties in the hive catalog in values.yaml.

1. Configure Kerberos properties in values.yaml. Add the following properties to the presto.hive.additionalConfig section:

   presto:
     hive:
       additionalConfig: [
         hive.hdfs.authentication.type=KERBEROS,
         hive.hdfs.presto.principal=xxxx@REALM, # Replace xxxx@REALM with your principal
         hive.hdfs.presto.keytab=/opt/secrets/xxx.keytab # Replace xxx.keytab with the name of your keytab file
       ]
   

   This configuration ensures that the Embedded MPP connects to HDFS using the specified Kerberos principal, hive.hdfs.presto.principal, and its corresponding keytab file, hive.hdfs.presto.keytab.

2. Kerberos Files:

   • Place the keytab file in the presto/secrets folder within your Denodo Embedded MPP Helm chart.

   • Place the krb5.conf file in the presto/conf/catalog/ folder.

   • Add the following property to the values.yaml to inform Java about the krb5.conf location:

   presto:
     jvm:
       additionalJVMConfig: [
         -Djava.security.krb5.conf=/opt/presto-server/etc/catalog/krb5.conf
       ]
   

3. Enable HDFS Wire Encryption (if required):

   If the HDFS cluster has HDFS wire encryption enabled, you must add one more property to the presto.hive.additionalConfig section in values.yaml:

      hive.hdfs.wire-encryption.enabled=true
   

For more detailed information on Hive Security Configuration and Kerberos support, you can refer to Hive Security Configuration — Kerberos Support.