You can translate the document:

Introduction

This document describes how to connect to Google Cloud Storage from Denodo Virtual DataPort.

Google Cloud Storage is Google based object storage solution for the cloud. Cloud storage is optimized for storing massive amounts of unstructured data, such as text or binary data.

Virtual DataPort can connect to Google Cloud storage in order to use it as a data source and to import information.

Prerequisites

In order to access files in Google Cloud Storage from Denodo, the following actions need to be performed in the Google Cloud Portal:

  1. Create oAuth Client ID credentials in the Google Cloud platform APIs & Services.
  2. Create a bucket and files in Google Cloud Storage.

Create the oAuth Client ID credentials

  • To create an oAuth Client credential, navigate to “APIs & Services” section from the Google Cloud Platform Portal and select “Credentials” Tab. Click on “+ Create Credentials -> oAuth Client ID” to open the “Create oAuth Client ID wizard” and provide the required details and click the “CREATE” button to define the Credentials inside the Google Cloud Platform.

  • Once the Application is created, it will display the “Your Client ID” and “Your Client Secret” as shown in the below image:

  • Once you close this pop up, the newly created Credentials will be displayed under the Auth2.0 Client IDs as shown in the below screenshot.

        

  • On selecting the application, you will see the Client ID, Client Secret and Creation Date of the Credentials. You also have the option to download the Credentials Information in JSON format by clicking the “Download JSON” option on top of the screen as shown below.

        

  • The downloaded JSON file has the following information that will be required to configure OAuth:
  • Client_id
  • Auth_uri
  • Token_uri
  • Client_secret

Create a Bucket and Files

  • After configuring the Credentials, click on the “Storage” section in the navigation pane. To create a new bucket, click on “Create Bucket” and follow the wizard to create the bucket in Google Cloud Storage.
  • Once the bucket is created, open the bucket and click on “UPLOAD FILES” to upload a file into this bucket.
  • Once the file is uploaded it will be displayed as shown below.

  • On selecting the file we would be able to see the information about these files as shown below. In order to access this file from the Denodo Platform we will use the REST API URL for this file. The REST API information of the Google Cloud services can be found in the Google Cloud Documentation.

Connecting to Google Cloud Storage files from Denodo

After completing the above steps, follow the steps given under this section to connect to Google Cloud Storage files from Denodo. These steps are applicable to other types of data sources, you will only need to select the data source that corresponds to the format of the file being accessed.

In this example we will explain how to connect from Google Cloud Storage to access Delimited Files. Since the addition of HDFS and S3 routes in Denodo, to access the Delimited File stored in Google Cloud Storage, follow the steps given under this section.

 

  • Launch the Web Design Studio Tool and then select the type of data source needed depending on the type of file which you want to access from Google Cloud storage by navigating to New > Data Source in the contextual menu. In this example, a “Delimited File” data source will be used.

  • Select the HDFS” option as the Data Route parameter.

  • In the Data route configuration add the “URI” as a gs route:

 gs://folder/<filepath>.

  • In addition, add the below properties to the Hadoop properties:

Name

Value

google.cloud.auth.service.account.enable

true

google.cloud.auth.service.account.json.keyfile

/PATH/TO/KEYFILE

  • Then, click on “Test Connection” and if the connection is successful, click on “Save”.

  • Once the data source is created, click on the “Create base view” option to create a base view to introspect source metadata available through the Data Source and provide the necessary parameters as follows,

  • File Name Pattern: <filename_*.csv> (Optional).
  • Separator: provide the column separator. The Default value is “,”.

  • Click “Save” to create the base view.
  • Now, the base view created on top of the Delimited File stored in Google Cloud Storage is ready for the execution and to be combined with the rest of the sources.

Note: The user who is accessing Google Cloud Storage must be an authenticated user.

References

Creating Client Credentials ID 

Creating and Managing Service Account

Create a Bucket

Scopes for Cloud Storage APIs

OAuth Authentication

Delimited File Sources

Questions

Ask a question

You must sign in to ask a question. If you do not have an account, you can register here