How to connect to the Amazon S3 REST API from Denodo

Applies to: Denodo 8.0 , Denodo 7.0
Last modified on: 19 Jan 2022
Tags: Amazon S3 Custom functions DF data sources XML data sources

Download document

You can translate the document:

Introduction

Amazon Simple Storage Service (Amazon S3), is a service that provides to developers and IT teams a secure, durable and highly-scalable object storage.

Amazon S3 stores data as objects within resources called "buckets”. It is possible to store as many objects as needed within a bucket. As well as write, read, and delete objects in a bucket.

There are several ways to integrate this service. The AWS Management Console provides a web-based interface for accessing and managing all available Amazon S3 resources. Programmatic access to Amazon S3 is enabled by the AWS SDKs and the AWS SDK Mobile. In addition, Amazon S3 provides a full REST API which will be the studied feature in this document.

In addition, HDFS Custom Wrappers available at the Support Site also support S3 routes, so they can be used to access directly data in S3. This is the recommended option to access Amazon S3, for more information read How to integrate Amazon S3 with Denodo Distributed File System Custom Wrapper.

How to connect to AWS S3 from Denodo?

Summary of options

  1. (Recommended) HDFS Custom Wrappers available at the Denodo Support Site to access delimited files, as well as non-standard files (Avro, Map files, Sequence files) stored in HDFS and AWS S3. This option allows access to delimited files (CSV), Avro, Map files, Sequence files and Parquet files. Any other file format stored in S3, such as JSON or XML can’t be accessed through these custom wrappers. Check How to integrate Amazon S3 with Denodo Distributed File System Custom Wrapper for more information.

  1. Through Amazon S3 REST API. This option allows access to file types not supported by the HDFS Custom Wrappers, such as JSON or XML. On the other hand, this method is more manual as Amazon S3 REST API has a proprietary authentication schema that must be provided in an Authentication Header

Authorization: AWS4-HMAC-SHA256 Credential={access-key}/{date}/{region}/s3/aws4_request,SignedHeaders=host;x-amz-date;{other-required-headers},Signature={signature}

This document describes the way to access file types not supported by the HDFS Custom Wrappers, such as JSON or XML, through the Amazon S3 REST API.

Amazon S3 REST API

In this section, we will introduce the Amazon S3 REST API and describe some of its most useful features.

Requests to Amazon S3 can be authenticated or anonymous. In case authenticated requests are used, credentials will be required so AWS can authenticate the requests. When making a REST API call, a signature has to be created using the user credentials and then included in the request. Being correctly authenticated is not enough to access an Amazon S3 resource, the user still needs to have enough permissions on the resource to access.

In the following sections we will describe some of the main requirements for authenticated requests and two of the most important authenticated Amazon S3 REST API methods: “ListObjectsV2” and “GetObject”.

For more information about the REST API, see the API Reference for Amazon S3.

Authenticated requests requirements.

Authentication requirements.

Authenticated requests in Amazon S3 need to send authentication information in order to be performed. For that purpose, the “HTTP Authorization header” is the most common method to authenticate Amazon S3 requests. This section introduces request authentication using the “HTTP Authorization header” method.

The authentication information sent in a request must include a “signature” that is calculated following a detailed process. In “AWS Signature version 4”, the basic signing requests process is composed of the following steps:

  1. Obtain a “string to sign”: it is necessary to compose a string concatenating a subset of elements from the request to be signed in the last step.
  2. Obtain a “signing key”: the “secret access key” provided by AWS is used to calculate the “signing key” that will be used to sign the string obtained in the previous step.
  3. Obtain the “signature”: it is the result of the signing process that signs the string obtained in step (a) using the key calculated in step (b).

The authentication method used in the examples, that will be described in this document, is the “Authorization Header” method which consists in adding an Authorization header to the request containing the calculated  “signature”. The Authentication header has the following form:

Authorization: AWS4-HMAC-SHA256 Credential={access-key}/{date}/{region}/s3/aws4_request,SignedHeaders=host;x-amz-date;{other-required-headers},Signature={signature}

The process in charge of getting the Authorization header is defined as follows:

  1. Authorization = “AWS4-HMAC-SHA256” + “ “ + “Credential=” + <AWSAccessKeyId> + “/” + <Date> + “/” + <aws-region> + “/s3/aws4_request,SignedHeaders=host;x-amz-date;x-amz-content-sha256,Signature=” + <Signature>
  2. Date = YYYYMMDD format.
  3. Signature = Hex(HMAC-SHA256(<SigningKey>, <StringToSing>))
  1. SigningKey = HMAC-SHA256(<DateRegionServiceKey>, “aws4_request”)
  1. DateRegionServiceKey = HMAC-SHA256(<DateRegionKey>, “s3”)
  2. DateRegionKey = HMAC-SHA256(<DateKey>, “<aws-region>”)
  3. DateKey = HMAC-SHA256(“AWS4” + “<AWSSecretAccessKey>”, <yyyymmdd>)
  1. StringToSing = “AWS4-HMAC-SHA256” + “\n” +

<TimeStamp> + “\n” +

<Scope> + “\n” +

Hex(SHA256Hash(<CanonicalRequest>))

  1. TimeStamp = current UTC time in ISO 8601 format.
  2. Scope = “<yyyymmdd>/<aws-region>/s3/aws4_request”
  3. CanonicalRequest = <HTTP Verb> + “\n” +

                        <Canonical URI> + “\n” +

                        <Canonical Query String> + “\n” +

                        <Canonical Headers> + “\n” +

                        <Signed Headers> + “\n” +

                        <Headed Payload>

  1. Headed Payload = Hex(SHA256Hash(<payload>))

To know more about authenticating requests in Amazon AWS and the “Authentication Header” method take a look at the following articles:

Timestamp requirement.

A valid timestamp (using either the HTTP Dateheader or the x-amz-date alternative) is mandatory for authenticated requests. The timestamp must be specified in the UTC time with ISO 8601 format, for example: “20211228T130942Z”.

Furthermore, the client timestamp included with an authenticated request must be within 15 minutes of the Amazon S3 system time when the request is received. If not, the request will fail with the RequestTimeTooSkewed error code. The intention of these restrictions is to limit the possibility that intercepted requests could be replayed by an adversary. For stronger protection against eavesdropping, use the HTTPS transport for authenticated requests.

When an “x-amz-date” header is present in a request, the system will ignore any “Date” header when computing the request signature.

Authenticated operations.

Operations on buckets.

List Objects (version 2)

This operation returns some or all (up to 1000) of the objects in a bucket. You can use the request parameters as selection criteria to return a subset of the objects in a bucket.

  • Request

https://BucketName.s3.amazonaws.com/?list-type=2[&...] HTTP/1.1

  • Syntax

GET /?list-type=2[&...] HTTP/1.1

        Host: BucketName.s3.amazonaws.com

        x-amz-content-sha256: <sha256-content>

x-amz-date: <timestamp>

Authorization: <authorization string>

  • Headers
  • Authorization: AWS4-HMAC-SHA256 Credential={access-key}/{date}/{region}/s3/aws4_request,SignedHeaders=host;x-amz-date;{other-required-headers},Signature={signature}
  • x-amz-content-sha256: is required for all AWS Signature Version 4 requests. It provides a hash of the request payload. If there is no payload, you must provide the hash of an empty string (e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855).
  • x-amz-date: the timeStamp is the current UTC time in ISO 8601 format (for example, 20130524T000000Z).
  • Request Parameters (some)
  • prefix: Limits the response to keys (file names) that begin with the specified prefix. The prefixes can be used to separate a bucket into different groupings of keys. A prefix can be used to make groups in the same way as a folder in a file system. This parameter is optional, if a prefix is not specified all the objects in the bucket will be returned.
  • max-keys: Sets the maximum number of keys returned in the response body. This parameter is optional. This parameter can be added to the request to retrieve fewer than the default 1000 keys. The response might contain fewer keys but will never contain more. If there are additional keys that satisfy the search criteria but were not returned because max-keys was exceeded, the response will contain <IsTruncated>true</IsTruncated>. To return the additional keys, the marker parameter can be used.
  • start-after: StartAfter is where you want Amazon S3 to start listing from. Amazon S3 starts listing after this specified key. StartAfter can be any key in the bucket.

  • Sample Request Using Request Parameters

https://quotes.s3.eu-central-1.amazonaws.com/?list-type=2&max-keys=40&prefix=j

This request returns up to 40 keys in the "quotes" bucket that start with "j".

GET /?list-type=2&max-keys=40&prefix=j HTTP/1.1

Host: quotes.s3.eu-central-1.amazonaws.com

Authorization: <authorization string>

x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

x-amz-date: 20211228T130942Z

  • Sample Response

HTTP/1.1 200 OK
x-amz-id-2: Y7t9a48FI9qfnZA/8qxJg3OTJGFzFceZP49JTsapvGHo0fIhypNkcjeOqb2+2h39oVdsncMZ3Mk=
x-amz-request-id: KEVHQF0WZWQ7231W

x-amz-bucket-region: eu-central-1

Date: Tue, 28 Dec 2021 13:10:02 GMT
Content-Type: application/xml
Server: AmazonS3

Body:

<?xml version="1.0" encoding="UTF-8"?>

<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">

    <Name>rest-api-test-eu</Name>

    <Prefix>j</Prefix>

    <KeyCount>2</KeyCount>

    <MaxKeys>40</MaxKeys>

    <IsTruncated>false</IsTruncated>

    <Contents>

        <Key>json/</Key>

        <LastModified>2021-04-28T12:12:17.000Z</LastModified>

        <ETag>&quot;d41d8cd98f00b204e9800998ecf8427e&quot;</ETag>

        <Size>0</Size>

        <StorageClass>STANDARD</StorageClass>

    </Contents>

    <Contents>

        <Key>json/university.json</Key>

        <LastModified>2021-04-28T12:13:06.000Z</LastModified>

        <ETag>&quot;9b6700dcc1a8fe704fcb534e2babdcc5&quot;</ETag>

        <Size>1615</Size>

        <StorageClass>STANDARD</StorageClass>

    </Contents>

</ListBucketResult>

Operations on objects.

GET Object

This implementation of the GET operation retrieves objects from Amazon S3. 

An Amazon S3 bucket does not have a directory hierarchy like the one in a typical file system. However, it is possible to create a logical hierarchy by using object key names that imply a folder structure. For example, instead of naming an object sample.jpg, it is possible to  name it as photos/2006/February/sample.jpg.

To get an object from such a logical hierarchy, specify the full key name for the object in the GET operation. For a virtual hosted-style request example, if the object is named  photos/2006/February/sample.jpg, the resource is referenced as /photos/2006/February/sample.jpg:

https://bucket.s3.amazonaws.com/photos/2006/February/sample.jpg

  • Request

https://BucketName.s3.amazonaws.com/ObjectName

  • Syntax

GET /ObjectName HTTP/1.1

                Host: BucketName.s3.amazonaws.com

                x-amz-content-sha256: <sha256-content>

x-amz-date: <timestamp>

Authorization: <authorization string>

  • Sample Request

https://quotes.s3.eu-central-1.amazonaws.com/json/university.json

The following request returns the object, json/university.json.

GET /json/university.json HTTP/1.1

Host: quotes.s3.eu-central-1.amazonaws.com

Authorization: <authorization string>

x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

x-amz-date: 20211228T173828Z

  • Sample Response

HTTP/1.1 200 OK
x-amz-id-2: PusbWJ8hRUgCeR5bHAEE3vP14aa+0bY6elnaGRc/Ou1k/ptl25E32LBWuI2HeBCynk90ruyoHss=
x-amz-request-id: KME5TYA3SJS53WDN
Date: Tue, 28 Dec 2021 17:38:51 GMT
Last-Modified: Wed, 28 Apr 2021 12:13:06 GMT
ETag: "9b6700dcc1a8fe704fcb534e2babdcc5"
Content-Length: 1615
Content-Type: application/json
Accept-ranges: bytes
Server: AmazonS3

Body:

{

    "University": [

        {

            "universityID": "univ-11",

            "universityName": "Boston Institute of Technology",

Getting the content of an object

Creating a parameterized data source to get the content of any object

  1. Select the type of data source needed depending on the type of file you want to recover from Amazon S3. In this example, a JSON data source will be used.

  1. Select the HTTP Client option as the Data Route parameter.

  1. Configure the “HTTP Client” data route to access an Amazon S3 stored object.
  1. HTTP Method: GET.
  2. URL:

https://bucketname.s3.amazonaws.com/bucketobject

In this example, we are going to parameterize the “bucketname” and “bucketobject” of the http request with the param_host and param_filepath input parameters, which will allow us to recover any object from any bucket.

https://@{param_host}@{param_filepath}

  1. Configure the “HTTP Headers”.

We need to create three headers, Authorization, “x-amz-content-sha256” , and “x-amz-date”, to recover objects from a bucket. We will parameterize Authorization and  “x-amz-date” meanwhile we will specify for “x-amz-content-sha256” the corresponding fixed value for requests with no payloads.

Click OK”, to confirm the HTTP Client configuration.

  1. At this point if you press the OK button again, it will show the configuration main window of the data source.

  1. Finally, after clicking on the “Save” button the data source is created showing the configuration set in the previous steps.

Creating a base view from the data source

  1. Click on the Create base view button.

At this moment, you have to set the values for the mandatory input parameters of the JSON data source configured in the previous step:

  • param_host: url of the host that contains the object to recover. It is composed of “bucket_name” + “.s3.amazonaws.com”.
  • param_filepath: name of the object to recover. It should start with the “/” character.
  • param_date: timestamp when the object is requested. It will be used to send as the “x-amz-date” header.
  • param_auth: value of the “Authorization” header that will be sent to the REST service. It contains the authorization signed string used by the REST service to authenticate and authorize the corresponding HTTP request.

To be able to set the param_auth value the process that calculates the AWS Authorization Request Header has to be implemented. To know more about this header you can take a look at the following guide: Signature Calculations for the Authorization Header: Transferring Payload in a Single Chunk (AWS Signature Version 4). An easy way to implement this authentication mechanism is to use a VDP custom function that could be invoked any time the view is queried to create the Authorization header that is needed for the authenticated requests.

Note: Remember that there is a limit of 15 minutes between the query’s authentication parameters being set up and the time that the request arrives to the Amazon S3 service.

  1. After that, the settings window of the new base view will be shown. Change the name of the view and modify the fields as needed and click on the OK button.

Creating a view to obtain the authentication parameters

After implementing the process that calculates the AWS Authorization Request Header using a custom function, for example with the “get_aws_auth_header” name, a view can be created to obtain the Authorization header running the following query on the VQL Shell:

Note that in our example the “get_aws_auth_header” custom function returns the values for the “Authorization” and “x-amz-date” headers within a record, so this would require the creation of two views: one returning the record and another flattening the record fields to directly access their values.

1st view returning the record with the values for the “Authorization” and “x-amz-date” headers:

 

CREATE OR REPLACE VIEW dv_get_aws_auth_header_register AS

SELECT

  get_aws_auth_header(host, region, filepath, access_key, access_secret, request_parameters) AS header,

  host,

  region,

  filepath,

  access_key,

  access_secret,

  request_parameters

 

FROM dual()

USING PARAMETERS ( host : text, region : text, filepath : text, access_key : text, access_secret : text, request_parameters : text);

2nd view flattening the record fields:

CREATE OR REPLACE VIEW dv_get_aws_auth_header_final FOLDER = '/views for signature creation' AS

SELECT

  dv_get_aws_auth_header_register.host AS host,

  dv_get_aws_auth_header_register.region AS region,

  dv_get_aws_auth_header_register.filepath AS filepath,

  dv_get_aws_auth_header_register.access_key AS access_key,

  dv_get_aws_auth_header_register.access_secret AS access_secret,

  dv_get_aws_auth_header_register.request_parameters AS request_parameters,

 

  (dv_get_aws_auth_header_register.header).authorization AS authorization,

  (dv_get_aws_auth_header_register.header).timestamp AS timestamp

 

FROM dv_get_aws_auth_header_register;

The following input parameters to the view will be needed for the view to be used as arguments of the custom function:

  • access_key: access key provided by Amazon S3.
  • secret_key: secret key provided by Amazon S3.
  • host: host where the file/object to retrieve is located.
  • filepath: filepath of the object to recover from the previously specified host.
  • region: region which the S3 bucket belongs to (e.g: eu-central-1).
  • request_parameters(optional): AWS S3 request parameters (e.g: ?list-type=2&max-keys=40&prefix=j)

The output of the 2nd view will consist of:

  • authorization: the result of the process that calculates the Authorization header. The syntax of this output will be AWS4-HMAC-SHA256 Credential={access-key}/{date}/{region}/s3/aws4_request,SignedHeaders=host;x-amz-date;{other-required-headers},Signature={signature}.
  • timestamp: date used to calculate the authorization header, in the UTC time with ISO 8601 format, for example: “20211228T130942Z”.

This output will be used to provide the values for the “Authorization” and “x-amz-date” headers of the data source respectively.

Creating a final view to get the content of an object

  1. Create a derived view joining the two previously created views: dv_get_aws_auth_header_final”, that gets the authorization data needed for the authenticated request, and bv_aws_s3_jsonthat gets the actual object from Amazon S3.

  1. Set the following join conditions:

  • dv_get_aws_auth_header_final.host = bv_aws_s3_json.param_host
  • dv_get_aws_auth_header_final.filepath = bv_aws_s3_json.param_filepath
  • dv_get_aws_auth_header_final.authorization = bv_aws_s3_json.param_auth
  • dv_get_aws_auth_header_final.timestamp = bv_aws_s3_json.param_date

  1. Set the name of the view and remove from the output the following redundant fields:

  • bv_aws_s3_json.param_host
  • Bv_aws_s3_json.param_filepath
  • bv_aws_s3_json.param_date

Obtaining an object from the bucket

Run the following query from the VQL Shell on the “dv_aws_s3_json” view providing values for the input parameters as where conditions, for instance:

SELECT * FROM dv_aws_s3_json

WHERE host = 'bucketname.s3.eu-central-1.amazonaws.com'

and filepath = '/json/university.json'

and region = 'eu-central-1'

and request_parameters = ''

and access_key = '#ACCESS_KEY#'

and access_secret = '#SECRET_KEY#';

Getting the list of object names in a bucket

Creating a parameterized data source to get the list of object names with a prefix

  1. Select the XML data source type to create a data source that recovers the list of filenames stored in a bucket.

  1. Select the HTTP Client option to set the Data Route parameter.

  1. Configure the “HTTP Client” data route to access any Amazon S3 bucket.
  1. HTTP Method: GET.
  2. URL:

https://bucketname.s3.amazonaws.com/?list-type=2&prefix=json/

In this case, to build the URL, the host and prefix will be parameters to recover the list of file names stored in any bucket.

https://@{param_host}/?list-type=2&prefix=@{param_prefix}

  1. Configure the “HTTP Headers”.

We need to create three headers, Authorization, “x-amz-content-sha256” , and “x-amz-date”, to recover objects from a bucket. We will parameterize Authorization and  “x-amz-date” meanwhile we will specify for “x-amz-content-sha256” the corresponding fixed value for requests with no payloads.

Click OK to confirm the HTTP Client configuration.

Creating a base view from the data source

  1. Click on the Create base view button.

To create the base view,  set the values for the mandatory input parameters of the delimited data source configured in the previous step:

  • param_host: url of the host to send the GET object names request. It is composed of “bucket_name” + “.s3.region_name.amazonaws.com”.
  • param_prefix: prefix used to filter the object names to recover. Limits the response to keys (file names) that begin with the specified prefix.
  • param_date: timestamp when the request is sent. It will be used to send as the “x-amz-date” header and as an input parameter to calculate the “param_auth” parameter.
  • param_auth: value of the Authorization header that will be sent to the REST service. It contains the authorization signed string used by the REST service to authenticate and authorize the corresponding http request.

The same view described before (dv_get_aws_auth_header_final”) can be used to set the param_auth and “param_date” values.

If you want to obtain the values for param_authand param_date when you are trying to get, for instance, the content for the following request https://denodotest.s3.eu-central-1.amazonaws.com/?list-type=2&prefix=json/, you have to set the following input parameters to the view that calculates the authentication header (dv_get_aws_auth_header_final):

  • access_key: access key provided by Amazon S3.
  • secret_key: secret key provided by Amazon S3.
  • host: ’denodotest.s3.eu-central-1.amazonaws.com’.
  • filepath: ’/’.
  • region: ‘eu-central-1’.
  • request_parameters: ‘list-type=2&prefix=json/’.

And then use the generated authorizationand timestamp resulted fields to fill the Create base view form.

  1. After providing the values the following window is displayed.

Select the Stream output at specified level option and the Contents array field in order to flatten the output of the view by the selected field and click OK to create the base view.

After creating this base view, the same steps indicated for the GET Object method can be used to create a final view: creating a view with the custom function to construct the authorization header and a join view between this view and the base view to query the data directly.

Creating a final view to list the object names in a bucket

  1. Create a derived view joining the two previously created views: dv_get_aws_auth_header_final”, that gets the authorization data needed for the authenticated request, and bv_aws_s3_listobjectsv2that lists the object names in a bucket.

        

  1. Set the following conditions:

  • dv_get_aws_auth_header_final.authorization = bv_aws_s3_listobjectsv2.param_auth
  • dv_get_aws_auth_header_final.host = bv_aws_s3_listobjectsv2.param_host
  • dv_get_aws_auth_header_final.timestamp = bv_aws_s3_listobjectsv2.param_date
  • dv_get_aws_auth_header_final.request_parameters = bv_aws_s3_listobjectsv2.param_prefix

  1. Since the dv_get_aws_auth_header_final.request_parameters contains all the parameters of this type of request (e.g: 'list-type=2&prefix=json/') and the bv_aws_s3_listobjectsv2.prefix only needs the value of the prefix parameter (e.g: ‘json/’), we need to edit the join condition between these two parameters to provide the corresponding values for the prefix parameter. To do so, please follow the below steps:
  1. Go to the “Join Conditions” tab.
  2. Click on the “Edit condition” button of the condition “dv_get_aws_auth_header_final.request_parameters = bv_aws_s3_listobjectsv2.prefix”. 
  3. Click on the “Complex condition” tab and specify the condition as follows:
  1. substr(dv_get_aws_auth_header_final.request_parameters, 20) = bv_aws_s3_listobjectsv2.param_prefix

  1. Set the name of the view (e.g: dv_aws_s3_listobjectsv2) and remove from the output the following redundant fields:

  • bv_aws_s3_listobjectsv2.param_host
  • bv_aws_s3_listobjectsv2.param_auth
  • bv_aws_s3_listobjectsv2.param_date
  • bv_aws_s3_listobjectsv2.param_prefix

Obtaining the list of object names in a bucket

Run the following query from the VQL Shell on the “bucket_list_dv” view providing values for the input parameters as where conditions, for instance:

SELECT * FROM dv_aws_s3_listobjectsv2

WHERE host = 'denodotest.s3.eu-central-1.amazonaws.com'

  And request_parameters = 'list-type=2&prefix=json/'

  and region = 'eu-central-1'

  and filepath = '/'

  and access_key = #ACCESS_KEY#

  and access_secret = #SECRET_KEY#

Getting the content of the objects listed in a bucket

Creating a final view to get the content of the objects listed in a bucket

  1. Create a derived view joining the two previously created views: dv_aws_s3_listobjectsv2”, that lists the object names in a bucket, and dv_aws_s3_jsonthat gets the actual object from Amazon S3.

  1. Set the following conditions:

  • dv_aws_s3_listobjectsv2.access_key = dv_aws_s3_json.access_key
  • dv_aws_s3_listobjectsv2.access_secret = dv_aws_s3_json.access_secret
  • dv_aws_s3_listobjectsv2.host = dv_aws_s3_json.host
  • dv_aws_s3_listobjectsv2.region = dv_aws_s3_json.region
  • dv_aws_s3_json.request_parameters = ‘’
  • ('/'||dv_aws_s3_listobjectsv2.key) = dv_aws_s3_json.filepath

Note: The “dv_aws_s3_json.request_parameters” has been set to empty string “” since no request parameters are needed to get the content of an object.

Note: The last condition adds a “/” to the dv_aws_s3_listobjectsv2.keysince the corresponding field dv_aws_s3_json.filepathmust start with the “/” character.

  1. Set the name of the view and remove from the output the following redundant fields:

  • dv_aws_s3_json.host
  • dv_aws_s3_json.region
  • dv_aws_s3_json.filepath
  • dv_aws_s3_json.access_key
  • dv_aws_s3_json.access_secret
  • dv_aws_s3_json.request_parameters
  • dv_aws_s3_json.authorization
  • dv_aws_s3_json.timestamp

Obtaining the content of the objects listed in a bucket

Run the following query from the VQL Shell on the “delimited_files_in_bucket” view providing values for the input parameters as where conditions, for instance:

SELECT * FROM json_files_in_bucket

WHERE access_key = #ACCESS_KEY#

  and access_secret = #SECRET_KEY#  

  and region = 'eu-central-1'

  and host = 'denodotest.s3.eu-central-1.amazonaws.com'

  and filepath = '/'

  and request_parameters = 'list-type=2&prefix=json/';

References

Denodo Distributed File System Custom Wrapper - User Manual

Questions

Ask a question
You must sign in to ask a question. If you do not have an account, you can register here

Featured content

DENODO TRAINING

Ready for more? Great! We offer a comprehensive set of training courses, taught by our technical instructors in small, private groups for getting a full, in-depth guided training in the usage of the Denodo Platform. Check out our training courses.

Training