AWS S3¶
Before deploying the Denodo Embedded MPP on Amazon EKS check Denodo Embedded MPP AWS Checklist to make sure you have everything you need.
There are two options to deploy a Denodo Embedded MPP that will access AWS S3 datasets:
The recommended one: provide no AWS S3 credentials to the
kubectl create secret
command.Used when the Denodo Embedded MPP will run in Amazon EKS and will access S3 with one of these three methods:
EKS Pod Identities
To do this, you must set up the Kubernetes
serviceAccount
name provided by your EKS administrator in thevalues.yaml
, e.g:serviceAccount: create: true name: "pod-identity-service-account" annotations: {}
IAM Roles for Service Accounts
To do this you can associate an IAM role with the
serviceAccount
through theserviceAccount.annotations
in thevalues.yaml
, e.g:serviceAccount: create: true annotations: eks.amazonaws.com/role-arn: arn:aws:iam::<awsaccountid>:role/<role>
and the Denodo Embedded MPP will access S3, using the permissions configured in that IAM role.
IAM EC2 instance profile
Provide the AWS S3 access and secret key ID to the
kubectl create secret
command:ENV Variable
Description
AWS_ACCESS_KEY_ID
AWS access key ID
AWS_SECRET_ACCESS_KEY
AWS secret key
kubectl create secret generic mpp-credentials --from-literal=METASTORE_DB_PASSWORD=hive --from-literal=AWS_ACCESS_KEY_ID=awsaccesskeyid --from-literal=AWS_SECRET_ACCESS_KEY=awssecretaccesskey
It’s also necessary to set true the
objectStorage.aws.securityCredentials.enabled
property invalues.yaml
Run
helm install
sentencehelm install prestocluster prestocluster/
AWS Privileges for AWS S3¶
The AWS privileges required by the Denodo Embedded MPP when accessing the AWS S3 buckets are:
Reading from AWS S3:
s3:GetObject
s3:ListBucket
Writing to AWS S3. Same as for reading and also:
s3:PutObject
s3:DeleteObject
Important
AWS credentials provider.
The Denodo Embedded MPP is shipped with a credentials provider chain configured by default DenodoAWSCredentialsProviderChain
.
This chain looks for AWS credentials in this order:
SimpleAWSCredentialsProvider
: Loads credentials fromfs.s3a.access.key
andfs.s3a.secret.key
properties in Hadoop configuration files.EnvironmentVariableCredentialsProvider
: Loads credentials from environment variablesAWS_ACCESS_KEY_ID
andAWS_SECRET_KEY
.SystemPropertiesCredentialsProvider
: Loads credentials from Java system propertiesaws.accessKeyId
andaws.secretKey
.WebIdentityTokenCredentialsProvider
: Loads Web Identity Token credentials from the environment or container.ProfileCredentialsProvider
: Load credential profiles file at the default location~/.aws/credentials
.EC2ContainerCredentialsProviderWrapper
: Loads credentials from EC2, typically using theInstanceProfileCredentialsProvider
.
If none of these providers fit your needs, you will need to change the credentials provider configured in:
presto/conf/catalog/core-site.xml
:Replace the value of the
presto.s3.credentials-provider
property with the AWS credentials provider of your choice.Include any other properties required by this credential provider.
hive-metastore/conf/core-site.xml
:Replace the value of the
fs.s3a.aws.credentials.provider
property with the AWS credentials provider of your choice.Include any other properties required by this credential provider.