This document describes how to connect to Google Cloud Storage from Denodo Virtual DataPort.
Google Cloud Storage is Google based object storage solution for the cloud. Cloud storage is optimized for storing massive amounts of unstructured data, such as text or binary data.
Virtual DataPort can connect to Google Cloud storage in order to use it as a data source and to import information.
In order to access files in Google Cloud Storage from Denodo, the following actions need to be performed in the Google Cloud Portal:
- Create oAuth Client ID credentials in the Google Cloud platform APIs & Services.
- Create a bucket and files in Google Cloud Storage.
Create the oAuth Client ID credentials
- To create an oAuth Client credential, navigate to “APIs & Services” section from the Google Cloud Platform Portal and select “Credentials” Tab. Click on “+ Create Credentials -> oAuth Client ID” to open the “Create oAuth Client ID wizard” and provide the required details and click the “CREATE” button to define the Credentials inside the Google Cloud Platform.
- Once the Application is created, it will display the “Your Client ID” and “Your Client Secret” as shown in the below image:
- Once you close this pop up, the newly created Credentials will be displayed under the Auth2.0 Client IDs as shown in the below screenshot.
- On selecting the application, you will see the Client ID, Client Secret and Creation Date of the Credentials. You also have the option to download the Credentials Information in JSON format by clicking the “Download JSON” option on top of the screen as shown below.
- The downloaded JSON file has the following information that will be required to configure OAuth:
Create a Bucket and Files
- After configuring the Credentials, click on the “Storage” section in the navigation pane. To create a new bucket, click on “Create Bucket” and follow the wizard to create the bucket in Google Cloud Storage.
- Once the bucket is created, open the bucket and click on “UPLOAD FILES” to upload a file into this bucket.
- Once the file is uploaded it will be displayed as shown below.
- On selecting the file we would be able to see the information about these files as shown below. In order to access this file from the Denodo Platform we will use the REST API URL for this file. The REST API information of the Google Cloud services can be found in the Google Cloud Documentation.
Connecting to Google Cloud Storage files from Denodo
After completing the above steps, follow the steps given under this section to connect to Google Cloud Storage files from Denodo. These steps are applicable to other types of data sources, you will only need to select the data source that corresponds to the format of the file being accessed.
In this example we will explain how to connect from Google Cloud Storage to access Delimited Files. Since the addition of HDFS and S3 routes in Denodo, to access the Delimited File stored in Google Cloud Storage, follow the steps given under this section.
- Launch the Web Design Studio Tool and then select the type of data source needed depending on the type of file which you want to access from Google Cloud storage by navigating to New > Data Source in the contextual menu. In this example, a “Delimited File” data source will be used.
- Select the “HDFS” option as the “Data Route” parameter.
- In the Data route configuration add the “URI” as a gs route:
- In addition, add the below properties to the “Hadoop properties”:
- Then, click on “Test Connection” and if the connection is successful, click on “Save”.
- Once the data source is created, click on the “Create base view” option to create a base view to introspect source metadata available through the Data Source and provide the necessary parameters as follows,
- File Name Pattern: <filename_*.csv> (Optional).
- Separator: provide the column separator. The Default value is “,”.
- Click “Save” to create the base view.
- Now, the base view created on top of the Delimited File stored in Google Cloud Storage is ready for the execution and to be combined with the rest of the sources.
Note: The user who is accessing Google Cloud Storage must be an authenticated user.