AWS S3¶
Before deploying the Denodo Embedded MPP on Amazon EKS check Denodo Embedded MPP AWS Checklist to make sure you have everything you need.
There are two options to deploy a Denodo Embedded MPP that will access AWS S3 datasets:
The recommended one: provide no AWS S3 credentials to the
kubectl create secretcommand.Used when the Denodo Embedded MPP will run in Amazon EKS and will access S3 with one of these three methods:
EKS Pod Identities
To do this, you must set up the Kubernetes
serviceAccountname provided by your EKS administrator in thevalues.yaml, e.g:serviceAccount: create: true name: "pod-identity-service-account" annotations: {}
IAM Roles for Service Accounts
To do this you can associate an IAM role with the
serviceAccountthrough theserviceAccount.annotationsin thevalues.yaml, e.g:serviceAccount: create: true annotations: eks.amazonaws.com/role-arn: arn:aws:iam::<awsaccountid>:role/<role>
and the Denodo Embedded MPP will access S3, using the permissions configured in that IAM role.
IAM EC2 instance profile
Provide the AWS S3 access and secret key ID to the
kubectl create secretcommand:ENV Variable
Description
AWS_ACCESS_KEY_ID
AWS access key ID
AWS_SECRET_ACCESS_KEY
AWS secret key
kubectl create secret generic mpp-credentials --from-literal=METASTORE_DB_PASSWORD=hive --from-literal=AWS_ACCESS_KEY_ID=awsaccesskeyid --from-literal=AWS_SECRET_ACCESS_KEY=awssecretaccesskey
It’s also necessary to set true the
objectStorage.aws.securityCredentials.enabledproperty invalues.yamlRun
helm installsentencehelm install prestocluster prestocluster/
AWS Privileges for AWS S3¶
The AWS privileges required by the Denodo Embedded MPP when accessing the AWS S3 buckets are:
Reading from AWS S3:
s3:GetObjects3:ListBucket
Writing to AWS S3. Same as for reading and also:
s3:PutObjects3:DeleteObject
Important
AWS credentials provider.
The Denodo Embedded MPP is shipped with a credentials provider chain configured by default DenodoAWSCredentialsProviderChain.
This chain looks for AWS credentials in this order:
SimpleAWSCredentialsProvider: Loads credentials fromfs.s3a.access.keyandfs.s3a.secret.keyproperties in Hadoop configuration files.EnvironmentVariableCredentialsProvider: Loads credentials from environment variablesAWS_ACCESS_KEY_IDandAWS_SECRET_KEY.SystemPropertiesCredentialsProvider: Loads credentials from Java system propertiesaws.accessKeyIdandaws.secretKey.WebIdentityTokenCredentialsProvider: Loads Web Identity Token credentials from the environment or container.ProfileCredentialsProvider: Load credential profiles file at the default location~/.aws/credentials.EC2ContainerCredentialsProviderWrapper: Loads credentials from EC2, typically using theInstanceProfileCredentialsProvider.
If none of these providers fit your needs, you will need to change the credentials provider configured in:
presto/conf/catalog/core-site.xml:Replace the value of the
presto.s3.credentials-providerproperty with the AWS credentials provider of your choice.Include any other properties required by this credential provider.
hive-metastore/conf/core-site.xml:Replace the value of the
fs.s3a.aws.credentials.providerproperty with the AWS credentials provider of your choice.Include any other properties required by this credential provider.
