AWS S3¶
Before deploying the Denodo Embedded MPP on Amazon EKS check Denodo Embedded MPP AWS Checklist to make sure you have everything you need.
There are two options to deploy a Denodo Embedded MPP that will access AWS S3 datasets:
The recommended one: provide no AWS S3 credentials to the
cluster.sh deploy
command.cluster.sh deploy --credstore-password xxx
Used when the Denodo Embedded MPP will run in Amazon EKS and will access S3 with one of these three methods:
EKS Pod Identities
To do this, you must set up the Kubernetes
serviceAccount
name provided by your EKS administrator in thevalues.yaml
, e.g:serviceAccount: create: true name: "pod-identity-service-account" annotations: {}
IAM Roles for Service Accounts
To do this you can associate an IAM role with the
serviceAccount
through theserviceAccount.annotations
in thevalues.yaml
, e.g:serviceAccount: create: true annotations: eks.amazonaws.com/role-arn: arn:aws:iam::<awsaccountid>:role/<role>
and the Denodo Embedded MPP will access S3, using the permissions configured in that IAM role.
IAM EC2 instance profile
Provide the AWS S3 credentials to the
cluster.sh deploy
command:cluster.sh deploy --s3-access-key xxx --s3-secret-access yyy --credstore-password zzz
--s3-access-key
: the Access Key Id--s3-secret-access
: the Secret Access Key.If this secret is not specified in the command line,
cluster.sh deploy
will prompt for it, keeping secrets out of the bash history
AWS Privileges for AWS S3¶
The AWS privileges required by the Denodo Embedded MPP when accessing the AWS S3 buckets are:
Reading from AWS S3:
s3:GetObject
s3:ListBucket
Writing to AWS S3. Same as for reading and also:
s3:PutObject
s3:DeleteObject
Important
AWS credentials provider.
The Denodo Embedded MPP is shipped with a credentials provider chain configured by default DenodoAWSCredentialsProviderChain
.
This chain looks for AWS credentials in this order:
SimpleAWSCredentialsProvider
: Loads credentials fromfs.s3a.access.key
andfs.s3a.secret.key
properties in Hadoop configuration files.EnvironmentVariableCredentialsProvider
: Loads credentials from environment variablesAWS_ACCESS_KEY_ID
andAWS_SECRET_KEY
.SystemPropertiesCredentialsProvider
: Loads credentials from Java system propertiesaws.accessKeyId
andaws.secretKey
.WebIdentityTokenCredentialsProvider
: Loads Web Identity Token credentials from the environment or container.ProfileCredentialsProvider
: Load credential profiles file at the default location~/.aws/credentials
.EC2ContainerCredentialsProviderWrapper
: Loads credentials from EC2, typically using theInstanceProfileCredentialsProvider
.
If none of these providers fit your needs, you will need to change the credentials provider configured in:
presto/conf/catalog/core-site.xml
:Replace the value of the
presto.s3.credentials-provider
property with the AWS credentials provider of your choice.Include any other properties required by this credential provider.
hive-metastore/conf/core-site.xml
:Replace the value of the
fs.s3a.aws.credentials.provider
property with the AWS credentials provider of your choice.Include any other properties required by this credential provider.