Denodo Templates for Twitter - Quick Use Guide

Last modified on: 07 May 2020

Overview

Twitter is a social networking site that relies on micro-blogging for communication. By typing short, 140 character messages (up to 280 since 2017) users can communicate with their followers and get information spread around. Since its inception, Twitter has grown and turned into a massive site used by mainstream media as well as average individuals to communicate about important issues, random issues, or just about what they’re doing at any time. It allows people to blog in short bursts, a practice also known as microblogging.

Twitter lets you make short posts about anything. These short posts are known as “tweets”, in reference to the chirping of birds, and appear chronologically on your Twitter profile page. In addition to posting, you can also follow others on Twitter to get their latest tweets in your Twitter feed. Conversely, if someone follows you, they get your tweets in their feeds.

Denodo Templates for Twitter

Importing the Denodo Templates for Twitter will grant you the access to the following functionalities, grouped by types:

Direct Messages

Direct Messages Events List

Lists

Lists List
Membership Lists
Ownership Lists

Tweets

Favorites List
Search Tweets
Home Timeline
Mentions Timeline
User Timeline
Retweets of Me

Users

Blocks List
Followers List
Friends List
Mutes Users List

Utils

Application Rate Limit Status

The templates are organized as described above. In each folder you’ll find the views provided for all this functionalities and two subfolders, one for the data sources and another one for the base views, which are more complex and will contain more info than the final views (and might be needed in some specific scenarios).

In the Configuration section you’ll find how to generate your own Denodo Templates for Twitter.

Requirements

The Denodo Templates for Twitter require:

When used with Denodo 6.0: At least the Denodo update 20181120.
When used with Denodo 7.0: At least the Denodo update 20190205.

Configuration

Get OAuth1 credentials

In order to get the parameters needed to authenticate to your Twitter account from Denodo, you need to follow these steps:

You must have a Developer Account in order to authenticate to the Twitter API. Navigate through https://developer.twitter.com/en/apps/ and select the Application that you wish to configure with Denodo by clicking in the Details button.

Now you have to go to the Keys and Tokens section. Those keys are what you need to get your templates working into Denodo with your Twitter App. Also, make sure that you have Read, write, and direct messages permissions, otherwise configure them in the Permissions tab.

Now, you have all the data needed to configure the .properties file.

Configure denodo_templates_twitter.properties file

Before importing the VQL templates in Denodo, you have to set the following parameters defined in the denodo_templates_twitter.properties file:

api_standard_version: the version you want to use of the Twitter API. By default, it’s set to v1.1.

client_id: Matches API key, obtained in the step 2 of the “Get OAuth1 credentials” section.

client_shared_secret: Matches API secret key, obtained in the step 2 of the “Get OAuth1 credentials” section.

access_token: Matches Access token, obtained in the step 2 of the “Get OAuth1 credentials” section.

access_token_secret: Matches Access token secret, obtained in the step 2 of the “Get OAuth1 credentials” section.

api_search_url: For the search functionality Twitter provides Standard, Enterprise and Premium APIs, so you can set this parameter using one of them. By default, this parameter is set to use Standard API.

twitter_templates_denodo.properties

Note: Make sure you leave an empty line at the end of the properties file.

Import the VQL file into Denodo

Once you have completed all the previous steps of this configuration section, you finally can load the denodo_templates_twitter.vql file into Denodo. You only have to import it, along with the .properties file, using the Import option of the VDP Administration Tool:

Refresh

After the import process finishes, refresh the Server Explorer by selecting File → Refresh and you will be able to see the data source and base views for accessing Denodo Templates for Twitter:

Denodo for Twitter views and folders structure

Query

Now, you can execute queries over the Twitter views that have been created.

Limitations

Twitter API Limits

The standard API rate limits are described in the table in this link. Note that endpoints not listed in the chart default to 15 requests per allotted user. All request windows are 15 minutes in length. These rate limits apply to the standard API endpoints only, does not apply to premium APIs. Note that some of the views included in the Denodo Templates for Twitter may require an amount of requests that might exceed the limits of your Twitter account under some circumstances.

Denodo 6 doesn’t support pagination in JSON data sources

To be able to get more than the results limited by the Twitter API for each view, we can use pagination, a feature included in Denodo 7 but not available in Denodo 6.

This way, we can get the amount of desired data as long as we don’t reach Twitter API limits.

Pagination in Denodo 7 and later

Although in Denodo 7 pagination can be done over JSON data sources, Twitter has set some limitations in their default API. For some of the endpoints pagination is not supported. You can see a more detailed explanation of the limitations in this link.

To configure pagination, you have to access the data source configuration and set up the following parameters in the Pagination section:

Parameter in URL for page size: count
Page size: <number> (Represents the number of elements to return per page.)
Token continuation

Parameter in URL for “next” token: cursor
Path to “next” token in response: next_cursor

Here are the limitations in the Standard API according to this pagination context:

Resource	Supports pagination	Max “Count” value
blocks/list	Yes	5000
mutes/users/list	Yes	5000
followers/list	Yes	200
friends/list	yes	200
lists/list	No	Unsupported
lists/memberships	Yes	1000
lists/ownerships	Yes	1000
search/tweets	No	100
statuses/retweets_of_me	No	100
statuses/home_timeline	No	200
statuses/mentions_timeline	No	200
statuses/user_timeline	No	200
favorites/list	No	200
direct_messages/events/list	Yes	50

Note that all the resources that support pagination are configured to use it, so in case you use an enterprise/premium API you'll be able to paginate your queries.