You can translate the question and the replies:

Dynamic Source File Configuration with SFTP

Hi I am experimenting with a prototype for my organization thru Denodo Express Virtual Data Port. Here is my problem: 1) Daily files are deposited into sftp via batch upload. 2) I have configured the data source in denodo to point to the sftp but it requires me to specify the .txt location, so its only picking up a single file. 3) How do i iterate through all the files in the sftp directory such that the virtualized data model is dynamically refreshed to pick up values from the last updated files? I am available for a chat to solve this and if required to share my screen as well. Pls let me know Rgds, Michelle
user
21-01-2020 02:38:04 -0500

1 Answer

Hi, Denodo accepts file patterns with regular expressions when configuring a path to a file. This allows the Denodo Platform to find all files with a certain pattern in a directory and import them. For example, when I want to access all the text files in a specified folder, I create a [Delimited File data source](https://community.denodo.com/docs/html/browse/latest/vdp/administration/creating_data_sources_and_base_views/delimited_file_sources/delimited_file_sources) and do not add a specific file to the file path. Instead, I add “(.*)\.txt” to the “File name pattern” field; Denodo will combine the data from each text file into a base view. Denodo also offers the [DateRange function](https://community.denodo.com/docs/html/browse/latest/vdp/administration/creating_data_sources_and_base_views/delimited_file_sources/delimited_file_sources#paths-using-date-ranges) for files ending in dates. Each execution of the base view will query all the files in the folder and add or remove rows in the base view according to the data from the files in the folder. Another method to select all files in the folder is to use the [Denodo FileSystem Custom Wrapper](https://community.denodo.com/docs/html/document/denodoconnects/7.0/Denodo%20FileSystem%20CustomWrapper%20-%20User%20Manual). The wrapper returns a view with the information about files in a specified parent folder. I use this to create a join from the FileSystem base view to a Delimited File source with ‘@FULLFILEPATH’ variable in the “File path” field connected to the “fullpath” field in the Filesystem base view. This will search the folder for each file and then extract the data as a Delimited Source. For more information, I’ve included links in the text to the Denodo User Manual for each of these topics. I hope this helps!
Denodo Team
22-01-2020 05:20:43 -0500
You must sign in to add an answer. If you do not have an account, you can register here