You can translate the question and the replies:

Query optimization for fixed length file sources

Hi, We have fixed lenth data source and it is mutli record type file for example the starting of the row will what type record it is like if start with 's' it is a student record it is start with 'm' it is marks record like that there will be mutiple type of rows available. As per requirement we need to show all the related data in one row. We create derieved view for each type of record like one for student and one for marks and each type of record has one common column by using that we are joined derived views to make final view but this final view is taking more time because there are 1millon+ rows are there in each source file and we have to read data from mutiple files. To get data from one view only it is taking more than 9 min. Please suggest any to optimize jion queries.
user
18-01-2021 05:02:31 -0500

1 Answer

Hi, Generally speaking, the Virtual DataPort Server maximizes the pushdown of the query to the underlying data source in order to maximize the performance. However, as flat files are not databases, the Denodo Platform allows you to utilize its cache engine for these purposes. The cache module allows you to store a local copy of the data retrieved from the data sources (the flat files) in a JDBC database which then again enhances the performance through the pushdown of queries (to that JDBC database). This way, I can retrieve data from flat files faster by making use of the cache engine. I would recommend you to review the documentation section [Cache Module](https://community.denodo.com/docs/html/browse/8.0/vdp/administration/cache_module/cache_module#cache-module) as well the Knowledge Base article [Best Practices to Maximize Performance III: Caching](https://community.denodo.com/kb/view/document/Best%20Practices%20to%20Maximize%20Performance%20III:%20Caching?category=Best+Practices) to learn more about it. Hope this helps!
Denodo Team
18-01-2021 10:18:33 -0500
You must sign in to add an answer. If you do not have an account, you can register here