Denodo Lakehouse Accelerator AWS Checklist
You can translate the document:Introduction
This document summarizes the steps from the Denodo Lakehouse Accelerator documentation in a checklist format so it is easy to follow.
This document assumes that you are on the latest version of the component.
NOTE: For an automated deployment check Deploying Denodo Lakehouse Accelerator in AWS Using CloudFormation.
AWS Architecture
Elements Needed
Configuration Checklist
|
Step |
Done |
Task |
Section 0 - Planning |
||
|
0A |
⬜ |
Architecture Decisions
By default the recommendation is to use N + 2 nodes:
Depending on the purpose of the Installation (production,POC, etc), these are the minimum recommended number of nodes:
Each node should have at least 128GB of memory and 16-32 cores. For example, in Amazon Web Services you can start with m6a.8xlarge or r5a.4xlarge nodes.
Decide the number of nodes and if autoscaling will be enabled.
For more information, check the Administration guide |
|
0B |
⬜ |
Select How Denodo Lakehouse Accelerator will be exposed to Denodo VDP
You need to select one of these options to determine how Denodo Lakehouse Accelerator will communicate to the Denodo VDP Instance:
These values are configured in values.yaml in section 5. |
|
0C |
⬜ |
Denodo Lakehouse Accelerator Data
In the case of AWS, this data will be commonly stored in S3 but other HDFS compatible storages can be used. Decide what will be your data storage.
Remember that the S3 data files won’t be deleted automatically if the cluster is deleted. |
|
0D |
⬜ |
Denodo Lakehouse Accelerator Metadata
Denodo Lakehouse Accelerator Metastore has the following options available:
Depending on the option selected you can adjust you can reduce the number of nodes to N + 1 (instead of N +2) as no node will be required for the Metastore.
Decide if you will use the default metastore or a different one. More information can be found in the Denodo Lakehouse Accelerator User Manual. |
|
0E |
|
Denodo VDP Considerations
If you have a cluster of Denodo servers it needs to be configured to store its metadata in an external database to take full advantage of the Denodo Lakehouse Accelerator functionalities. In case you only have one node (during a PoC for example) you need to set this property to false: SET 'queryOptimization.parallelProcessing.denodoConnector.enableUsingSharedMetadataOnly'='false'
In order to use the Denodo Lakehouse Accelerator features you need Enterprise Plus, in other case you may get errors similar to:
“Error: EmbeddedMPPMaxProcessors are limited to X but current number is unknown”
Before configuring this feature, please validate that your license allows it executing CALL VALIDATE_MPP_LICENSE() |
Section 1 - Configure EKS (AWS) |
||
|
1A |
⬜ |
Create IAM Roles
The following Roles need to be created in the AWS IAM configuration
Create role and name it “Denodo Lakehouse Accelerator Cluster Role” adding the following policies:
Create role and name it “Denodo Lakehouse Accelerator Nodes Role” adding the following policies:
|
|
1B |
⬜ |
Network Configuration
|
|
1C |
⬜ |
Create the nodes and assign the roles to the EKS Cluster
Using the information from Section 0 - Planning:
|
|
1D |
⬜ |
Cluster access authentication
When creating the EKS Cluster you are given 3 authentication modes to choose from:
Important: It is strongly recommended to use the EKS API authentication mode since the configMap mode is planned to be removed on future releases.
EKS API authentication method
When creating the EKS Cluster with EKS API authentication mode an IAM access entry for the user/role that created the Cluster will be automatically created with the AmazonEKSAdminPolicy (grants admin privileges), therefore the user that created the Cluster will have all permissions over it.
To add a new user/role to the kubernetes Cluster using the EKS API authentication method you should add an IAM access entry on the Access section on the EKS Cluster Console
The mandatory fields to create an access entry are:
You can also use a kubernetes role and rolebinding for authorization instead of one of the default policies, by specifying a group name when creating the IAM access entry.
ConfigMap authentication method
When creating the EKS Cluster with configMap authentication mode, the configMap will be automatically configured to map the user/role that created the Cluster with the kubernetes admin role, therefore having all permissions over the Cluster.
The steps to add a new IAM principal on the configMap are:
To add a new admin user you can use the already created role and rolebinding system:masters, therefore you would only need to edit the configMap file, this is done with the following command “kubectl edit -n kube-system configmap/aws-auth”.
Then you would have to add an entry on the MapRole or MapUser section as follows:
apiVersion: v1 data: mapRoles: | - groups: - system:masters rolearn: arn:aws:iam::111122223333:role/my-admin-role username: my-admin-role mapUsers: | - groups: - system:masters userarn: arn:aws:iam::111122223333:user/my-admin-user username: my-admin-user
Additionally, AWS has a pre-built yaml file to create a console-viewer role and role binding, you can create it in your cluster using the command “kubectl apply -f https://s3.us-west-2.amazonaws.com/amazon-eks/docs/eks-console-full-access.yaml” and then you can edit the configMap adding the user or role with the group named “eks-console-dashboard-full-access-group”. |
|
1E |
⬜ |
Autoscaling
If autoscaling is required, this guide explains the configuration steps for EKS |
Section 2 - AWS Environment Requirements |
||
|
2A |
⬜ |
HDFS Storage and credentials
Using the information from section 0 - Planning:
eks.amazonaws.com/role-arn: arn:aws:iam::<aws_account_id>:role/your-role:
The Denodo Lakehouse Accelerator will access S3, using the permissions configured in that IAM role. To know more about the requirements for using IRSA check: https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html
kubectl create secret generic mpp-credentials \ --from-literal=METASTORE_DB_PASSWORD=hive \ --from-literal=AWS_ACCESS_KEY_ID=awsaccesskeyid \ --from-literal=AWS_SECRET_ACCESS_KEY=awssecretaccesskey
Obtain your details from your preferred option as they will be used later. |
|
2B |
⬜ |
Denodo Platform running and connectivity
Obtain your Denodo VDP url and credentials. The Denodo VDP server needs access to the EKS Denodo Lakehouse Accelerator load balancer (Ingress). The following ports represent the default values:
Public Subnet of the Denodo Cluster / Instance
|
|
2C |
⬜ |
Persistent Storage (only if using Postgresql as discussed in Section 0 - Planning)
In the AWS EKS service, install the Driver "Amazon EBS CSI" under your EKS cluster. You can do this in the initial creation of the cluster or under the tab Add-ons when you select your cluster in EKS.
This is done for EKS to know how to provide that EBS disk when it asks for a default persistent volume. In other case the postgresql pod may stay in a “pending” state.
More details can be found in the Persistent Storage section here |
Section 3.1 - Container Registry - (Option 1) - Denodo Harbor |
||
|
3.1A |
⬜ |
Considerations
Denodo Harbor credentials expire every 6 months. While this option is suitable for testing and proof-of-concept (POC) purposes, please consider a private registry (see section 3.2) for production scenarios as a best practice.
|
|
3.1B |
⬜ |
Denodo Container Registry (Harbor) Credentials and firewall access
Provide firewall access to Denodo’s Registry in Harbor https://harbor.open.denodo.com/
Obtain your denodo_account_username and registry profile secret that you can find at https://harbor.open.denodo.com/
Open the “User Profile” menu and click on “Generate Secret”. Copy and store the cli secret as it will be used later. |
Section 3.2 - Container Registry - (Option 2) - ECR (skip this section if using Harbor) |
||
|
3.2A |
⬜ |
Configure AWS CLI
Configure AWS CLI using ‘aws configure’ and download Denodo Lakehouse Accelerator images from support.denodo.com where the CLI is configured |
|
3.2B |
⬜ |
Login to ECR with your Docker installation
aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <aws_account_id>.dkr.ecr.<region>.amazonaws.com |
|
3.2C |
⬜ |
Download Denodo Lakehouse Accelerator (if not using them from Harbor)
You can find the Denodo Lakehouse Accelerator in the Denodo Connects section of the Denodo Support Site. Unzip the file once it is downloaded. It contains the images that will be uploaded to ECR. |
|
3.2D |
⬜ |
Upload Denodo Lakehouse Accelerator images to ECR
Follow these steps to upload all images needed to ECR:
# Option 1: If Images downloaded locally. Load postgres, hive metastore, and presto depending on what is defined in section 0D.
docker load < prestocluster-presto-image-<version>.tar.gz docker load < prestocluster-postgresql-image-<version>.tar.gz docker load < prestocluster-hive-metastore-image-<version>.tar.gz
# Option 2: If Using Harbor, use docker pull instead of docker load. Load postgres, hive metastore, and presto depending on what is defined in section 0D.
docker pull harbor.open.denodo.com/denodo-connects-8.0/images/prestocluster-presto:<version> docker pull harbor.open.denodo.com/denodo-connects-8.0/images/prestocluster-postgresql:<version> docker pull harbor.open.denodo.com/denodo-connects-8.0/images/prestocluster-hive-metastore:<version>
# Tag postgres image and push to ECR docker tag prestocluster-postgresql:<version> <account_id>.dkr.ecr.<ecr_region>.amazonaws.com/prestocluster-postgresql:<version> docker push <account_id>.dkr.ecr.<ecr_region>.amazonaws.com/prestocluster-postgresql:<version>
# Tag hive metastore image and push to ECR docker tag prestocluster-hive-metastore:<version> <account_id>.dkr.ecr.<ecr_region>.amazonaws.com/prestocluster-hive-metastore:<version> docker push <account_id>.dkr.ecr.<ecr_region>.amazonaws.com/prestocluster-hive-metastore:<version>
# Tag presto image and push to ECR docker tag prestocluster-presto:<version> <account_id>.dkr.ecr.<ecr_region>.amazonaws.com/prestocluster-presto:<version> docker push <account_id>.dkr.ecr.<ecr_region>.amazonaws.com/prestocluster-presto:<version>
|
Section 4 - Deployment Requirements |
||
|
4A |
⬜ |
Download Denodo Lakehouse Accelerator
You can find the Denodo Lakehouse Accelerator in the Denodo Connects section of the Denodo Support Site. Unzip the file once it is downloaded. It contains the register.sh script, the helm chart and the other dependent artifacts that will be used in the next section. |
|
4B |
⬜ |
Use Linux or a compatible shell if in Windows
To run the register.sh script on Windows you need to have a Bash compatible shell such as Cygwin or Git Bash installed or use Windows Subsystem for Linux (WSL).
Cygwin Installation: Download it from https://www.cygwin.com/install.html and execute the setup
Important: we will use the “unzip” package, make sure to install it in the setup when selecting the packages |
|
4C |
⬜ |
Install kubectl client
The helm install command to apply the deployment, needs kubectl to be properly configured with the correct context and with the right credentials
Installation steps for Linux: curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
Installation steps for Cygwin: curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/windows/amd64/kubectl.exe" mkdir -p $HOME/bin mv kubectl.exe $HOME/bin/kubectl.exe echo 'export PATH=$HOME/bin:$PATH' >> ~/.bashrc echo alias kubectl="kubectl.exe" >> ~/.bashrc source ~/.bashrc
You can check if kubectl is correct using: kubectl version --client
For any special case, check the documentation here: https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/ |
|
4D |
⬜ |
Install Helm V3 for Kubernetes
Installation steps for Linux: curl -LO https://get.helm.sh/helm-v3.11.3-linux-amd64.tar.gz tar -zxvf helm-v3.11.3-linux-amd64.tar.gz sudo mv linux-amd64/helm /usr/local/bin/helm
Installation steps for Cygwin: curl -LO https://get.helm.sh/helm-v3.15.1-windows-amd64.zip unzip helm-v3.15.1-windows-amd64.zip mv windows-amd64/helm.exe $HOME/bin/helm.exe echo 'export PATH=$HOME/bin:$PATH' >> ~/.bashrc echo alias helm="helm.exe" >> ~/.bashrc source ~/.bashrc
You can check if kubectl is correct using: helm version
For any special case, documentation can be found here: https://helm.sh/docs/intro/install/ |
|
4E |
⬜ |
Install Java and configure JAVA_HOME and PATH
An installation of Java (17 recommended) is required. The JAVA_HOME and PATH environment variables should be properly configured.
E.g. export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 export PATH="$PATH:$JAVA_HOME/bin" |
|
4F |
⬜ |
Import the Denodo Lakehouse Accelerator certificate
You need to import a certificate into the Denodo VDP server trust store using: sudo ./jre/bin/keytool -importcert -alias presto-denodo -file /denodo-presto-k8scluster-x.x-xxxxxx/certs/certificate.crt -cacerts -storepass changeit
We provide a testing certificate inside /certs/certificate.crt that is meant FOR TESTING PURPOSES ONLY. This certificate accepts presto-denodo as the Denodo Lakehouse Accelerator hostname.
We recommend that you use a certificate issued by a CA in production. Follow the documentation for these steps. |
Section 5 - Configure prestocluster/values.yaml for the Denodo Lakehouse Accelerator configuration |
||
|
5A |
⬜ |
Add the repository credentials (Denodo’s Harbor or AWS registry)
If using Denodo Harbor, add the repository and pullSecret created above image.repository: "harbor.open.denodo.com/denodo-connects-8.0/images" pullcredentials.enabled=true pullcredentials.name="denodo-mpp-registry-secret" pullcredentials.registry="harbor.open.denodo.com" pullcredentials.username= pullcredentials.pwd=
If using AWS Container Registry (leave pullSecret empty): image.repository: “<account_id>.dkr.ecr.<ecr_region>.amazonaws.com” image.pullSecret: “” |
|
5B |
⬜ |
Connection Details to the Denodo Server
Using the information from Section 2C, configure your Denodo Server parameters in the denodoConnector section:
This step will create a denodoConnector.user and denodoConnector.password. This user will be used by Denodo Lakehouse Accelerator to connect to Denodo.
The credentials used to create that user are prompted when running register.sh |
|
5C |
⬜ |
Cloud Storage credentials (If using IAM Roles for Service Accounts)
Using the information from Section 2A, configure your credentials:
Configure the serviceAccount.annotations in the values.yaml, e.g:
The Denodo Lakehouse Accelerator will access S3, using the permissions configured in that IAM role. If there are different routes to access, the IAM Role requires access to all of them. |
|
5D |
⬜ |
Configure your number of workers, cpu and memory that each Presto Worker will use
The following parameters allow you to configure the number of Presto workers and the CPU and memory resources used by each, in your EKS cluster.
# -- Number of Presto workers in the cluster numWorkers: 4 # -- Number of cores assigned to each worker cpusPerNode: 16 # -- Total memory, in GB, assigned to each worker memoryPerNode: 128
As N+2 is the nodes recommendation, presto.numWorkers should be your total number of nodes in EKS - 2. E.g. if you have 3 nodes total, your presto.numWorkers should be 1
|
|
5E |
⬜ |
Configure the Service Section (if not using the default Load Balancer)
If you want to configure a different Service option than LoadBalancer, you can use the service.type value of values.yaml
service: # -- Service type: ClusterIP, NodePort or LoadBalancer type: LoadBalancer |
|
5F |
⬜ |
Configure an external Metastore (if not using the default Hive Metastore - Postgresql)
If using the default Hive Metastore - Postgres option check section 5.1 for recommendations and skip section 5.2.
If using an external Metastore, skip section 5.1 and check “5.2 Configure an external Metastore” instead. |
|
5G |
⬜ |
Adjust your components memory properties
It is important to adjust memory settings for query performance, finding a balance between maximum memory per query and the maximum number of concurrent queries that can be run in the Denodo Lakehouse Accelerator.
You can configure the memory settings in the following sections of your values.yaml. file.
presto: coordinator: additionalConfig: [ query.max-memory-per-node=xGB, query.max-total-memory-per-node=yGB, query.max-memory=zGB ] …
presto: worker: additionalConfig: [ query.max-memory-per-node=xGB, query.max-total-memory-per-node=yGB, query.max-memory=zGB ]
Find additional details in section “Memory” here |
|
5H |
⬜ |
Configure an Internal Load Balancer
If you want to configure an internal load balancer for your EKS cluster you will have to add a specific annotation to your values.yaml
Remember that you will have to ensure connectivity from the internal ELB network using NAT Gateways for private networks or VPC Endpoints for isolated networks
… presto: Service: … annotations: service.beta.kubernetes.io/aws-load-balancer-scheme: "internal" … … |
|
5I |
⬜ |
Select Object Storage authentication type
If access keys will be used as authentication type, set to true the objectStorage.securityCredentials.enabled property of your values.yaml:
objectStorage: aws: securityCredentials: enabled: false |
Section 5.1 (Optional) - Default Metastore |
||
|
5.1A |
⬜ |
How to maintain the metadata after redeploys
Follow these steps if you want to maintain the metadata available in the Metastore after a redeploy:
pvClaim: annotations: "helm.sh/resource-policy": keep
...
securityContext: # -- Force to run as a non-root user to ensure the least privilege runAsNonRoot: true # -- User ID for the container. Ignored on OpenShift. runAsUser: 1001 # -- Group ID for the pod volumes. Ignored on OpenShift fsGroup: 1001 fsGroupChangePolicy: OnRootMismatch
Note: For deployments across multiple Availability Zones (AZ), it's crucial to ensure that the PostgreSQL pod is always rescheduled within the same AZ as its associated Persistent Volume (PV). To achieve this, you should verify the allowedTopologies for the StorageClass and the AZ affinity settings for PostgreSQL (found in the prestocluster values.yaml file). In the example below (in bold) you can see how the pod is always assigned to the same AZ:
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: standard provisioner: kubernetes.io/example parameters: type: pd-standard volumeBindingMode: WaitForFirstConsumer allowedTopologies: - matchLabelExpressions: - key: topology.kubernetes.io/zone Values: - us-central-1b |
Section 5.2 (Optional) - Configure an external Metastore |
||
|
5.2A |
⬜ |
Configuring an existing Hive Metastore
With this option, Denodo won’t provision Postgresql or Hive Metastore and will connect to an existing external Metastore.
If kerberos is required you can follow these steps. |
|
5.2B |
⬜ |
Configuring Hive Metastore with a non-existing database different from Postgresql
In this scenario, Denodo will provision Hive Metastore but will use a database different from the default Postgresql. This option is used when there are policies restricting the type of RDBMS that can be installed or if there are issues with step 2.B.
|
|
5.2C |
⬜ |
Configuring Glue as Metastore
In case that you already have a AWS Glue Data Catalog containing table definitions you want to access from the Denodo Lakehouse Accelerator, you could use the AWS Glue Data Catalog as an external Metastore either as a replacement of the default Metastore or in addition.
|
|
5.2D |
⬜ |
Configuring multiple Metastore
With this option you will be able to connect to different metastore. This option is common in migration scenarios. For example, if planning on migrating from Athena to Denodo Lakehouse Accelerator.
More information can be found in the Denodo Lakehouse Accelerator User Manual. |
Section 6 - Deploying the server |
||
|
6A |
⬜ |
Deploy the Cluster with Helm
This command will deploy the Denodo Lakehouse Accelerator cluster using the helm chart downloaded.
helm install prestocluster ./prestocluster |
|
6B |
⬜ |
Check that the pods are in “Running” status
Execute the following command to check the status of each pod: kubectl get pods
The following pods should have STATUS=Running:
The pods use a Readiness Probe and Liveness Probe. Both probes are used to control the health of the cluster. If the Liveness probe fails, the container will be restarted, while if the Readiness probe fails, the container will stop serving traffic.
If some of the pods didn’t start correctly, please check the “Kubernetes debugging commands” section at the end of this document. |
|
6C |
⬜ |
Denodo VDP Hosts File (When using the Self Service Certificate)
When using the Self Service Certificate as described previously, the name of the DDenodo Lakehouse Accelerator Server needs to be presto-denodo. In order to do that, we need to edit the hosts file (e.g. in Linux /etc/hosts or in Windows WIN/system32/drivers/etc/hosts of the server where Denodo is installed. Add the following line:
Denodo_Lakehouse_Accelerator_IP presto-denodo
To obtain the value for Denodo_Lakehouse_Accelerator_IP you can execute kubectl get svc and obtain the external url for the LoadBalancer (This depends on step 0B)
|
|
6D |
⬜ |
Register the Denodo Lakehouse Accelerator Server in the Denodo VDP Server
Once all the pods are running, we need to create a datasource in our Denodo VDP Server pointing to Denodo Lakehouse Accelerator. To do that execute:
register.sh –register-user <denodo-user> –register-password <denodo-password>
To run register.sh on Windows you need to have a Bash compatible shell such as Cygwin or Git Bash installed or use Windows Subsystem for Linux (WSL). |
|
6E |
⬜ |
Check that there is connectivity between Denodo VDP and Denodo Lakehouse Accelerator
In Denodo, there should be a new vdb (by default admin_denodo_mpp) and a Denodo Lakehouse Accelerator data source (embedded_mpp). There are a couple of things that can be tested:
|
Section 7 - First steps after deployment |
||
|
7A |
⬜ |
Configure section “Use bulk data load APIs”
The information in this section will be used when inserting data in your Denodo Lakehouse Accelerator Cluster. Once the section is configured click on “Test Bulk Load” and validate that there are no errors |
|
7B |
⬜ |
Create a base view for a parquet file
Click on “Create Base View” to validate that you can introspect the parquet files (if any) available in your data storage routes |
Kubernetes debugging commands
The following commands are useful in case something fails:
- kubectl describe pod <pod name> (pod status information)
- kubectl logs <pod name> (pod logs)
- kubectl logs <pod name> --previous (logs of the last run of the pod before it crashed. Useful in case you want to figure out why the pod crashed in the first place.)
- kubectl exec -it <pod name> -- bash (interactive shell access to the pods)
- Interactive shell access to crashing pods, e.g. hive metastore:
- Edit prestocluster/templates/hive-metastore-template.yaml
- Replace:
command: ["/opt/run-hive-metastore.sh"]
by
command: [ "sleep" ]
args: [ "infinity" ]
- Redeploy the Denodo Lakehouse Accelerator
- Enter the hive metastore container to continue your debugging:
- kubectl exec -it <pod name> -- bash
- Since 8.0.20240306 you have to disable the livenessProbe in the values.yaml, otherwise the livenessProbe will kill the pod after a few seconds
- E.g. for the metastore: metastore.livenessreadiness.enabled: false
- Change Presto configuration without redeploying (since 8.0.20240306):
execute helm upgrade prestocluster prestocluster/
Troubleshooting
To troubleshoot 403 issues from Denodo Lakehouse Accelerator to S3 buckets, Denodo Lakehouse Accelerator visual introspection, bulk data load, etc. check the following documentation:
The information provided in the Denodo Knowledge Base is intended to assist our users in advanced uses of Denodo. Please note that the results from the application of processes and configurations detailed in these documents may vary depending on your specific environment. Use them at your own discretion.
For an official guide of supported features, please refer to the User Manuals. For questions on critical systems or complex environments we recommend you to contact your Denodo Customer Success Manager.
