You can translate the document:

Introduction

In this document we will explain how to integrate in Denodo the Google Custom Search API and how to configure its pagination. With this API, you can use RESTful requests to get either web search or image search results in JSON format.

Prerequisites

Before connecting to the API, first, you need to create a Programmable Search Engine. To create it, you can follow this tutorial: Creating a Programmable Search Engine. A Programmable Search Engine is a search engine where you can customize with different features your website, your blog, or a collection of websites.

Second, after creating the search engine you have to get an API key.

Once this is ready we can start using the API.

API Limits

It is important to know that the Custom Search JSON API provides 100 free searches per day for results. More intensive use of this API will come at a cost.

Integration in Denodo

Connecting to the API

Data source

The result of a search query to the Custom Search JSON API is a JSON object that includes three types of data: metadata describing the requested search, metadata describing the Programmable Search Engine and search results. So the first step to integrate it, is to create a JSON data source. To create the JSON data source, click on the Server Explorer and go to New > Data source > JSON.

The Tool will display the dialog to create the data source:

  1. Data route: HTTP client.
  2. Base URL:

https://www.googleapis.com/customsearch/v1?key=@key&cx=@cx&q=@word.

This URL has been obtained from the API documentation. To do a search, we need to specify the following parameters:

Pagination

In the data source we need to configure the pagination. We have to specify the parameter in the URL for page size (count), the parameter in the URL for next records (start), the index of first records, the offset for the next requests and a maximum number of requests.

The start parameter is calculated as: start=index of first records + (Offset for the next requests * N). For example: if index of first records=1, offset = 10 and N=2 (second request) then start = 21.

As the Google Search API will never return more than 100 results we have established the pagination parameters as in the screenshot below setting the maximum number of requests to 10.

Base view

Once the data source is created, we have to create a base view to query the search API:

  1. Create a base view from the data source.
  2. Enable “Use pagination of the data source”.

  1. We have to specify a tuple route (/JSONFile/items) to flatten the response.  This is the JSON route to access item details.

Final result

Derived view

From the base view created, we can create a derived view to select the output fields that we want to get in the final result:

  1. Click on the base view and go to New > Selection.
  2. Where Conditions: we have to introduce our api key and the id of the Programmable Search Engine as they will always be the same and we are not interested in exposing this information in the final view.

  1. Output: we check the boxes of the fields that we want to remove and after that, click on “Remove”.
  2. In this example, we have selected the fields shown in the screenshot.

Expected result

The following image shows the execution of the query: car. There, we can see the query entered and for each search result: the title, the link and the snippet. Also, we can see that 100 rows are returned: 10 requests to the API with 10 items returned per request.

References

Custom Google Search API

Customer Google Search API key

Creating a Programmable Search Engine

HTTP Path Pagination

Disclaimer
The information provided in the Denodo Knowledge Base is intended to assist our users in advanced uses of Denodo. Please note that the results from the application of processes and configurations detailed in these documents may vary depending on your specific environment. Use them at your own discretion.
For an official guide of supported features, please refer to the User Manuals. For questions on critical systems or complex environments we recommend you to contact your Denodo Customer Success Manager.

Questions

Ask a question

You must sign in to ask a question. If you do not have an account, you can register here