You can translate the question and the replies:

combine data from multiple sources with overlapping elements

Hello everyone, I'm currently working on a project where I need to combine data from multiple sources with overlapping elements. My goal is to create a common list by cross-referencing these data sources. I plan to use Denodo for this project. However, I don't want to interfere with the way each source owner manages their data source. I just want to set up a process that will attempt to create a common list by merging the data. I would appreciate any advice or insights on how to best approach this task, particularly in terms of optimizing the use of Denodo to merge the data. Thank you in advance for your help! I recently discover the DATA MESH concept (but I've just a first overview) I want use Denodo to - build list with merge data from these sources - build list of "differences" Just let know source owner about the discovered differences and let them correct. Does it seem to you that I have a good understanding of the concept of data mesh? Is there a specific term for the process of merging data sources? If I'm searching for documentation on this topic, what keywords should I use? Lastly, which specific features of Denodo might be helpful for my project? Currently, I am writing large views in VQL and comparing them to VQL functions that I am discovering. I appreciate any feedback or suggestions you can offer on these questions. Thank you! I apologize if my questions may seem unusual, but I often feel that I have a good intuitive approach to working with data and want to preserve a human touch in my approach. However, I am not always aware of established terminology or concepts that may already exist in the field. Therefore, any guidance or recommendations you can provide on this topic would be greatly appreciated. Thank you for your time and assistance!
user
24-04-2023 09:30:30 -0400
code

3 Answers

Hi, As per my understanding, you want to reconcile the data and metadata from different source systems to a common, standard and merged view. For example, you have the 2 data sources with table “*Customer*” with different schema/metadata and values, and then merge them into a reconciled view which contains the common and different schemas + the reconciled correct value if there’s any difference. I would call this “*data reconciliation from multiple sources*”, which you can search online for some guideline approaches. Denodo Platform can help you in building processes on the 2 common activities involved in data reconciliation, which I can summarise as the following: 1. **Reconciliation Analysis** - comparing the schema and data of different sources against the target reconciled view; determining which are the correct schema and data to use on the reconciled view; and informing the data sources if data and schema modification are required on the source side. 2. **Building of Reconciled Data** - building of the standardised, merged and reconciled view based on the reconciliation analysis done Data Mesh is not exactly the answer to your reconciliation project and use case. [Data Mesh](https://www.denodo.com/en/solutions/by-use-case/data-mesh-enabled-data-virtualization) is a modern decentralised data architecture that focuses on federated organisational structure, localised data management and governance. I suggest checking out [Logical Data Fabric](https://www.denodo.com/en/solutions/by-use-case/logical-data-fabric), which is focused more on creating a centralised logical layer for distributed architecture and technologies of your data sources. If you have a valid support user, I suggest opening a support ticket for further assistance and information on implementing the data reconciliation processes. Hope this helps!
Denodo Team
28-04-2023 02:12:51 -0400
code
I would like to read about Logical Data Fabric. As a customer of Denodo, we have a valid support user, but I am not a member of the data team. I investigate on my own and try to learn more about it. I was surprised when I discovered "Data Mesh" in the Denodo blog, as it aligns with my way of thinking. I am checking to see which part of my approach is covered by it. Also, I am trying to determine which parts of my approach are not covered by Data Mesh. Specifically, I am using Denodo to facilitate the identification of discrepancies between sources and to produce: A clean list that can be shared with everyone according to established rules by a working group that includes members from each source. A list of all discrepancies observed in each source compared to the clean list, which can be sent to the source owners to prompt action and obtain corrections. In practice, I go even further and aim for a finer granularity by identifying a responsible party for each row. Ultimately, my goal is to produce views * that enable source managers to import corrected values, * as well as views that facilitate reporting of the number of discrepancies by source, view, and owner, * and views importing of correction requests into a ticketing system.
user
28-04-2023 05:13:48 -0400
Hi, I can now see why you align the concepts of Data Mesh in your data reconciliation project due to the distributed implementation and ownership of the sources, and the centralised clean list. You could be going in the right direction. However “**Data Mesh**” is beyond what you’re trying to achieve at the moment: 1. It has to be organisation-wide to establish common and interoperable data standards, global security policies and governance which can be federated and enforced to domains/departments 2. Domains/Departments are responsible and autonomous for managing their distributed technology and infrastructure, and for managing their Data Products, which are trusted, good quality, and readily shareable to consumers Your current use case is more of a prerequisite or operational steps on how to build that trusted and good quality data products. I suggest watching this [webinar](https://www.denodo.com/en/webinar/enabling-data-mesh-architecture-and-data-sharing-culture-denodo) or download its slide deck to deep-dive on the concepts about Data Mesh. I suggest contacting your data team and asking them to open a support ticket on your behalf for further information on the data reconciliation processes. Your data team can provide you some information whether you could leverage on advisory sessions or other consulting services. Hope this helps!
Denodo Team
02-05-2023 23:32:55 -0400
code
You must sign in to add an answer. If you do not have an account, you can register here