You can translate the question and the replies:

API error - 429

Hi Team, I am connecting to an API and started getting error 429 - too many requests. Is there a way where in i can put a wait before the next request? Or any other recommendations? Thanks!
user
03-01-2022 16:20:25 -0500
code

10 Answers

Hi, In general, the HTTP error 429 may occur if the client has sent more requests in a given amount of time. For your scenario, I understand that you are trying to connect to an API via Data sources(like JSON, etc.) in Denodo. In that case, at the moment, you could limit the maximum number of concurrent HTTP connections against a specific host, by executing the following global property. > SET 'com.denodo.parser.connection.http.maxConnectionsPerHost'=<number_of_connections>; Even after setting this property, in case if you receive any errors or need further assistance and if you have a valid support account, then you can raise a support case on Denodo Support Site so that our support team can help you. Hope this helps.!
Denodo Team
04-01-2022 07:52:31 -0500
code
Thanks. Can you please let me know where to set this property? Is there any specific denodo file?
user
04-01-2022 09:54:11 -0500
Hi, You could either use the **SET** command in the VQL shell of the Virtual DataPort Administration tool which will set the property in the VDBConfiguration.properties file or you could directly modify the <DENODO_HOME>/conf/vdp/VDBConfiguration.properties file and restart and/or regenerate files to apply the changes. For more information, you could refer to the section [ Changing Settings of Virtual DataPort and the Web Container](https://community.denodo.com/docs/html/browse/8.0/en/vdp/vql/advanced_characteristics/changing_settings/changing_settings#changing-settings-of-virtual-dataport-and-the-web-container) of the Virtual DataPort VQL Guide. Hope this helps!
Denodo Team
04-01-2022 23:49:46 -0500
code
I got this info on the API : The FRED API has a limit of 1000 items per request, with a rate limit of 120 requests per minute. So should i set the property as 120? How to control number of requests within a minute? SET 'com.denodo.parser.connection.http.maxConnectionsPerHost'='120'; Thanks,
user
10-01-2022 13:40:07 -0500
Also is there a way to add API Throttle?
user
10-01-2022 16:04:13 -0500
So i figured out whats going wrong. Let me try explainign it. There are 3 views which are UNIONed together (Month/Quarter/annual) data set. So during UNION all the queries run together and it might be causing the view to error out.All 3 independent views are running fine. Is there any recommendation on how to resolve this issue?
user
10-01-2022 19:27:31 -0500
Hi, About the property “com.denodo.parser.connection.http.maxConnectionsPerHost”, it can be used for limiting the number of concurrent connections to a specific host. Further, regarding the API throttle question, you could check the section [Limiting the Maximum number of Concurrent requests ](https://community.denodo.com/docs/html/browse/8.0/en/vdp/administration/server_configuration/limiting_the_number_of_concurrent_requests/limiting_the_number_of_concurrent_requests#limiting-the-number-of-concurrent-requests) from the Virtual DataPort Administration Guide which in general highlights about the requests processed in the Virtual DataPort server. Regarding the last question, I understand that you are able to execute the independent views and when you perform the UNION operation by using those three independent views, you get this error. If this is the case, I would suggest you to enable **Cache** for the three independent views, maybe as an example, use full cache for the three independent views and load the data into the cache, and while performing the UNION operation the server fetches the data from the cache table and does not hit the API. For more information, you could refer to the Knowledge Base Article [**Caching**](https://community.denodo.com/kb/en/view/document/Best%20Practices%20to%20Maximize%20Performance%20III:%20Caching?category=Best%20Practices#h.ycccegj3w8bn). In case if you receive any other errors related to the API execution via Virtual DataPort, you can raise a support case on Denodo Support Site so that our support team can help you. Hope this helps.!
Denodo Team
11-01-2022 07:45:57 -0500
code
Hi, Maximum number of concurrent requests -> is it the number of requests that are sent from the VDP? I need to set the API throttle. 120 limit is still beign reached as i have a need to send around 170 request and there is no way to limit 120 request per minute. Any suggestion how to limit it to 120 requests per minute? i tried setting as below : but its not working as well. SET 'com.denodo.parser.connection.http.maxConnectionsPerHost'='120' Thanks in advance!
user
27-01-2022 17:34:16 -0500
Hi, The ‘Maximum number of concurrent requests’ option is to limit the number of requests that the Virtual DataPort Server would accept concurrently. When that limit is reached, the new requests will be queued and executed according to their arrival order. Regarding the question about restricting / limiting the requests, I think in such scenarios, [Resource Manager](https://community.denodo.com/docs/html/browse/8.0/en/vdp/administration/resource_manager/resource_manager) can be useful where plans can be added to assign restrictions on query execution. But for your specific scenario, I would suggest you to raise a support case in the Denodo Support Site so that our support team can assist you and help you with the appropriate approach. Hope this helps!
Denodo Team
28-01-2022 04:40:39 -0500
code
May be it might help few others: There's an existing enhancement for this: the existing enhancement request: Add support to limit the number of requests that a data source with an HTTP route can send in a certain period of time. You can view the status of this enhancement at any time in the Support Site under "Support & Services" > "Enhancements" section and search for the enhancement ID "21101" or with the given enhancement title. Work around is to create another base view and add wait when calling the API: 1. CREATE VIEW bv_api_limit AS 2. SELECT * 3. FROM bv_Api 4. LEFT OUTER JOIN (SELECT 1 FROM wait(600));
user
16-02-2022 14:07:13 -0500
You must sign in to add an answer. If you do not have an account, you can register here