You can translate the question and the replies:

Reading delimited files from multiple unknown subfolders

Hi, I'm trying to read delimited files that are being saved to a partially known folder structure such as this: ![]( The root folder is known and will not change. The subfolder is partially known. E,g I want denodo to read data from within any subfolder that has the word 'folder' in it, (subfolder A & subfolderB). The third level folder (e.g 579718) is completely unknown, and it likely to change. I then want denodo to recursively look for any .txt file within those third level folders (and even any subfolders within them), regardless of what name the folder has. In short, it would be ideal if I could insert a regular expression into the "File Path" field within the delimited file source, as well as the file name pattern. ![]( I've found this to be possible when pulling data from S3 storage using the denodo-hdfs-customwrapper-7.0, but not on any local file system delimited source wrappers. I hope it makes sense what I'm trying to achieve. Thanks in advance!
27-10-2020 08:00:26 -0400

2 Answers

Hi, I think for your use case, the use of Interpolation variables in the File Path and pattern can help you achieve it. They are used in the Virtual DataPort to parameterize certain expressions used by the wrapper or the data source of a base view. You can refer to [Execution Context of a Query and Interpolation Strings]( section of the Virtual DataPort VQL Guide for more information. Hope this helps!
Denodo Team
30-10-2020 08:21:15 -0400
Thanks for the answer. Also got a response from the support team that seems to have worked pretty well. It uses a combination of the delimited file source and the custom wrapper, essentially using a column from the custom wrapper as the input to the delimited file wrapper through a join. Pasting here for reference. In order to read the files recursively from a folder, you can make use of the FileSystem Custom Wrapper to select a set of files in a folder (or folders, recursively). It helps to extend the native capabilities of Delimited File data sources which allow to read only one file or all files in one folder. • Create a Delimited File data source and choose Local and click on the Configure button. • In File Path, specify an interpolation variable "@{file_name}" instead of filename in file location path eg: "/opt/Denodo70/work/files/Data/@{filename}" and Click Ok. • Specify a default delimited file name for instance '1.csv' and save the data source. • Create a new base view (eg. bv_view1) and while executing the base view, provide the delimited file name say ('2.csv'). This will show the output of '2.csv' file. • Download the FileSystem Custom Wrapper which would read all the files from a directory and import it into Virtual DataPort Administration Tool. • Create a new Custom data source using File System Custom Wrapper with Class name 'com.denodo.connect.filesystem.ReadFileSystemConnector' and Create a base view (eg. bv_read_file_system). You could refer to the FileSystem Custom Wrapper - User Manual for more information and detailed steps. You could make the 'recursive' field in the bv_read_file_system view as obligatory (OBL) in options > search methods if all the delimited files are distributed in subfolders within the parent folder. Execute the view by providing ‘parent_folder’ value as '/opt/Denodo70/work/files/Data'. • Create a new derived view by joining the two views "bv_view1" and "bv_read_file_system" using the join condition on file path and navigate to Edit > Where Conditions and add a simple condition bv_read_file_system.parentfolder='/opt/Denodo70/work/files/Data'. Please let us know if you have any further questions.
03-11-2020 04:37:33 -0500
You must sign in to add an answer. If you do not have an account, you can register here