USER MANUALS

AWS Glue Data Catalog

既存の AWS Glue Data Catalog に含まれるテーブル定義に Denodo 組み込み MPP からアクセスする場合、AWS Glue Data Catalog を外部メタストアとして使用できます。

それには、手動で新しいカタログを定義します。具体的には、 presto/conf/catalog/ に presto/conf/catalog/glue_hive.properties などのプロパティファイルを作成します。ファイル名の glue_hive がカタログ名になります。

Hive テーブル

AWS Glue Data Catalog から Parquet ファイルを読み込む Hive カタログ
connector.name=hive-hadoop2

hive.metastore=glue

# AWS region of the Glue Catalog
hive.metastore.glue.region=

# The ID of the Glue Catalog in which the metadata database resides
hive.metastore.glue.catalogid=

# Access Key and Secret Key for Glue
# Credentials and core-site.xml are not required when the MPP
# runs in EKS because it will use the IAM Roles for Service Accounts or the IAM EC2 instance profile,
# whichever is configured in EKS
hive.metastore.glue.aws-access-key=
hive.metastore.glue.aws-secret-key=
hive.config.resources=/opt/presto-server/etc/catalog/core-site.xml


# For Bulk Data load
hive.allow-drop-table=true
hive.non-managed-table-writes-enabled=true

hive.parquet.use-column-names=true

Delta Lake テーブル

AWS Glue Data Catalog から Delta Lake テーブルを読み込む Delta カタログ
connector.name=delta

hive.metastore=glue

# AWS region of the Glue Catalog
hive.metastore.glue.region=

# The ID of the Glue Catalog in which the metadata database resides
hive.metastore.glue.catalogid=

# Access Key and Secret Key for Glue
# Credentials and core-site.xml are not required when the MPP
# runs in EKS because it will use the IAM Roles for Service Accounts or the IAM EC2 instance profile,
# whichever is configured in EKS
hive.metastore.glue.aws-access-key=
hive.metastore.glue.aws-secret-key=
hive.config.resources=/opt/presto-server/etc/catalog/core-site.xml
#

hive.parquet.use-column-names=true

Iceberg テーブル

AWS Glue Data Catalog から Iceberg テーブルを読み込む Iceberg カタログ
connector.name=iceberg

iceberg.catalog.type=HIVE

hive.metastore=glue

# AWS region of the Glue Catalog
hive.metastore.glue.region=

# The ID of the Glue Catalog in which the metadata database resides
hive.metastore.glue.catalogid=

# Access Key and Secret Key for Glue
# Credentials and core-site.xml are not required when the MPP
# runs in EKS because it will use the IAM Roles for Service Accounts or the IAM EC2 instance profile,
# whichever is configured in EKS
hive.metastore.glue.aws-access-key=
hive.metastore.glue.aws-secret-key=
hive.config.resources=/opt/presto-server/etc/catalog/core-site.xml
#

hive.parquet.use-column-names=true

AWS Glue Data Catalog にアクセスするための AWS の権限

Denodo 組み込み MPP の IAM ロールが AWS Glue Data Catalog にアクセスするために必要な AWS の権限を以下に示します。

  • AWS Glue からの読み込み:

    • glue:GetDatabases

    • glue:GetDatabase

    • glue:GetTables

    • glue:GetTable

    • glue:GetPartitions

    • glue:GetPartition

    • glue:BatchGetPartition

  • AWS Glue への書き込み(読み込みと同じ権限に加えて以下の権限が必要):

    • glue:CreateTable

    • glue:DeleteTable

    • glue:UpdateTable

    • glue:BatchCreatePartition

    • glue:UpdatePartition

    • glue:DeletePartition

Add feedback