You can translate the question and the replies:

Merging CSV Files With Delta Files Dumped Into A SFTP Server Daily.

Hey, I am using VDP v7.0 with the latest upated tools and server. I have a use case where we have an SFTP server acting as a data source for us, which has CSVs dumped into it by another system which creates reports for us. For our initial load, we will have around 80 files (with 1000 rows each), and then a daily delta dump, which will need to be merged on a daily basis. So this means we will have at least a file a day which will need to be merged with all the other data. Also, the delta may have updated data on rows that already exist, which would mean we have to update that row with the new information. All the files will have the same order of columns. Only the filenames will have datetime appended. Is this something we can do with Denodo? Or would we have to do the merge externally before bringing it into our system. Please let me know if you need any more information. Thanks
user
09-10-2018 13:40:36 -0400

1 Answer

Hello, Yes, you can use Denodo for what you are describing. So essentially bring in the data as data sources and create views off of them. Then, you can use [Derived views](https://community.denodo.com/docs/html/browse/6.0/vdp/administration/creating_views/creating_derived_views/creating_derived_views) such as Unions and Selections to not only join the data with the delta file daily, but add additional views on top of that join, so you can remove any duplicate rows and update the rows with the new information. Some of the many[ functions](https://community.denodo.com/docs/html/browse/7.0/vdp/vql/appendix/syntax_of_condition_functions/syntax_of_condition_functions) available in Denodo are [Arithmetic functions](https://community.denodo.com/docs/html/browse/7.0/vdp/vql/appendix/syntax_of_condition_functions/arithmetic_functions) (I am specifically thinking of the MIN/MAX function for what you are describing if any of the fields that need to be updated are dates). Hope this helps!
Denodo Team
10-10-2018 11:57:16 -0400
You must sign in to add an answer. If you do not have an account, you can register here