Azure Data Lake Gen 2¶

Before deploying the Denodo Lakehouse Accelerator (formerly known as Denodo Embedded MPP) on Azure Kubernetes Service check Denodo Lakehouse Accelerator Azure Checklist to make sure you have everything you need.

There are four options to deploy a Denodo Lakehouse Accelerator that will access Azure Data Lake Storage Gen2 datasets:

1. The recommended one: Provide Azure Workload Identity configuration directly in Presto and Metastore core-site.xml.

   Azure Workload Identities (formerly “Azure AD pod identity”) writes OAuth 2.0 tokens to a file only accessible from the executing pod (/var/run/secrets/azure/tokens/azure-identity-token). The issued credentials can be used to authenticate. The Azure Portal/CLI is used to create the service identity.

   You have to provide the following properties to the kubectl create secret command before the Lakehouse Accelerator is deployed:

   ENV Variable	Description
   AZURE_TENANT_ID	Optional MSI Tenant ID
   AZURE_CLIENT_ID	Optional Client ID
   AZURE_FEDERATED_TOKEN_FILE	Token file path

   kubectl create secret generic mpp-credentials \
   --from-literal=METASTORE_DB_PASSWORD=hive \
   --from-literal=AZURE_TENANT_ID=azuretenantid \
   --from-literal=AZURE_CLIENT_ID=azureclientid \
   --from-literal=AZURE_FEDERATED_TOKEN_FILE=azurefederatedtokenfile
   

   As this option cannot be fully configured via values.yaml, you must also add the following configuration properties directly into the Presto and Metastore core-site.xml files:

   <property>
     <name>fs.azure.account.auth.type</name>
     <value>OAuth</value>
   </property>
   <property>
     <name>fs.azure.account.oauth.provider.type</name>
     <value>org.apache.hadoop.fs.azurebfs.oauth2.WorkloadIdentityTokenProvider</value>
   </property>
   <property>
     <name>fs.azure.account.oauth2.msi.tenant</name>
     <value>${env.AZURE_TENANT_ID}</value>
   </property>
   <property>
     <name>fs.azure.account.oauth2.client.id</name>
     <value>${env.AZURE_CLIENT_ID}</value>
   </property>
   <property>
     <name>fs.azure.account.oauth2.token.file</name>
     <value>${env.AZURE_FEDERATED_TOKEN_FILE}</value>
   </property>
   

   Run helm install sentence

   helm install lakehouseaccelerator lakehouseaccelerator/
   

2. Provide MSI Tenant ID and Client ID in the kubectl create secret command.

   Used when the Denodo Lakehouse Accelerator will run in Azure Kubernetes Service and will access Data Lake Storage gen2 using Azure Managed Identities (formerly “Managed Service Identities”).

   You have to provide the following properties to the kubectl create secret command before the Lakehouse Accelerator is deployed:

   ENV Variable	Description
   AZURE_OAUTH_MSI_TENANT_ID	MSI Tenant ID
   AZURE_OAUTH_CLIENT_ID	Client ID

   kubectl create secret generic mpp-credentials
   --from-literal=METASTORE_DB_PASSWORD=hive
   --from-literal=AZURE_OAUTH_MSI_TENANT_ID=azureoauthmsitenantid
   --from-literal=AZURE_OAUTH_CLIENT_ID=azureoauthclientid
   

   It’s also necessary to set true the objectStorage.azure.managedIdentities.enabled property in values.yaml

   Run helm install sentence

   helm install lakehouseaccelerator lakehouseaccelerator/
   

3. Provide OAuth2 client credentials to the kubectl create secret command.

   You have to provide the following properties to the kubectl create secret command before the Lakehouse Accelerator is deployed:

   ENV Variable	Description
   AZURE_OAUTH_DIRECTORY_ID(*)	Microsoft Directory ID or Tenant ID
   AZURE_OAUTH_CLIENT_ID	Client ID
   AZURE_OAUTH_CLIENT_SECRET	Client Secret

   (*) Note that AZURE_OAUTH_DIRECTORY_ID is https://login.microsoftonline.com/${env.AZURE_OAUTH_DIRECTORY_ID}/oauth2/token

   kubectl create secret generic mpp-credentials
   --from-literal=METASTORE_DB_PASSWORD=hive
   --from-literal=AZURE_OAUTH_DIRECTORY_ID=azureoauthdirectoryid
   --from-literal=AZURE_OAUTH_CLIENT_ID=azureoauthclientid
   --from-literal=AZURE_OAUTH_CLIENT_SECRET=azureoauthclientsecret
   

   It’s also necessary to set true the objectStorage.azure.oauth2ClientCredentials.enabled property in values.yaml

   Run helm install sentence

   helm install lakehouseaccelerator lakehouseaccelerator/
   

4. Provide the Azure credentials for the Shared Key authentication method to the kubectl create secret command before the Lakehouse Accelerator is deployed:

   ENV Variable	Description
   ABFS_STORAGE_KEY	Azure Data Lake Gen 2 Storage Key

   kubectl create secret generic mpp-credentials
   --from-literal=METASTORE_DB_PASSWORD=hive
   --from-literal=ABFS_STORAGE_KEY=abfsstoragekey
   

   It’s also necessary to set true the objectStorage.azure.sharedKey.enabled and add the Azure Data Lake Gen 2 Storage Account in the objectStorage.azure.sharedKey.account properties in values.yaml.

   Run helm install sentence

   helm install lakehouseaccelerator lakehouseaccelerator/