Introduction

When was the last time that you onboarded to a new organization? How long did it take you to get ramped so that you could start contributing to that organization in a meaningful way? Did this process involve a lot of messaging, video calls or word of mouth “Did you know about this?”.

Imagine that the documentation at your organization is in a sweet spot where you just self-serve and answer your own questions and can start contributing as soon as possible.  You want those data consumers  to answer questions such as what does this data mean, how is it calculated, where is it coming from, or is it trustworthy?

The issue is that with traditional toolings,  preparing all those data artifacts is separated from the actual documentation of those artifacts.  In Denodo the development  and documentation are part of the same process, so that there is no lag between them.

The success of any consumer data marketplace starts with striking the right CORD:

• Connect - Having the ability to connect to different data sources.
• Own - Highlighting who is responsible for maintaining authenticity of the data
• Repurpose - Having the Ability to import the metadata from the underlying sources, and to share certain metadata with every view associated.
• Discipline - Ultimately it is thoroughly important to inculcate a discipline of curating and logging metadata with the appropriate naming conventions.

A good self-service environment ensures that the power of creating business friendly dashboards lies in the hands of the business users and not the IT. This plays a major role in enabling shorter time to market. It is crucial to make the tribal knowledge of these business users easily available to anyone in the organization.

This document highlights some of the best practices that can be followed by administrators of the Denodo Data Marketplace to kick start their activities.

The Denodo Data Marketplace provides a unified, self-service portal for enterprise data assets. The user experience elevates data access, making it exceptionally easy for business users to discover, explore, and request curated data products, similar to how they access services in modern applications. The goal is to accelerate time-to-insight by reducing reliance on IT for basic data fetching.

The Denodo Data Marketplace

The Denodo Data Marketplace is the enterprise component that enables self service data discovery and exploration. In its most basic form it is the exposition of the data products created in Denodo to facilitate the work of the users that interact with Denodo. These personas include Business Analysts, Data Developers, Data Scientists, report developers, analytics resources, DevOps developers and anybody who needs access to specific slices of corporate data.

Note that all the security that you have implemented in the Denodo core data  platform is automatically inherited and applied to all objects made available in this marketplace.

Consumption marketplaces such as Denodo’s have the following properties

• They offer Data  Assets/Products exposed for consumption.
• They are oriented towards data consumers, making discovery simple and intuitive.
• They have lightweight support for Data Quality and Governance, enhanced with an audit trail to track user and administrator actions.
• They have simplified support for approval workflows and role-based security for change control.
• It is connected to the delivery infrastructure so that it supports querying data and access to usage statistics, in Denodo 9, this is extended with a dedicated Usage tab that surfaces query counts and adoption metrics.
• They offer advanced search capabilities, including search-by-data to find values inside datasets, not just metadata.
• They enable self-service Data Preparation, allowing business users to interactively filter, join, and transform datasets using sample-based previews, then share the prepared results.
• They support workflow requests, enabling users to submit access requests, metadata change proposals, or data quality issue reports, which are routed to stewards for review and approval.

This document is organized as follows

1. We take a look at architectural and operational considerations.
2. We look at best practices from the point of view of  basic end user flows.
3. We look at augmented workflows: we introduce advanced features like collaboration, relationships, statistics, usage or indexing end user data.

Architectural Considerations

Denodo Self-Managed

This section only applies if you are managing Denodo yourself. This means that the Denodo embedded Tomcat application server will run the Data Marketplace as a web application.  The default installation will be to include all components of the platform server inside a physical machine, virtual machine or container. It is possible to place them in separate containers/machines

In general there will be a farm of Denodo servers, a possible deployment can be

As mentioned above, the Data Marketplace can be installed on the same machine as the Denodo Platform Virtual DataPort server  (VDP) or on separate machines, The general recommendation is to install it on a separate machine that is on the same subnet as the platform server. The advantage of this is the isolation of servers and the simplification of management in container-based architectures. Furthermore, if both VDP server and the Data Marketplace are installed in a cluster, by installing them on separate machines, it allows having a different number of nodes for each server and balancing accesses from the Data Marketplace to the VDP servers.

Metadata storage and deployments

The default installation of Denodo will leverage a local Apache Derby  metadata database. If you are using containers, always use an external metadata database. In general it is preferable to externalize the Data Marketplace metadata to one of the supported databases.

If you have multiple Data Marketplaces running then you will have multiple Apache Derby databases that will have to be synchronized. It is a best practice to externalize the Data Marketplace metadata database so that changes to this metadata are immediately reflected to all Data Marketplaces in the cluster.

In most deployments we will have at least two environments and  typically four: Development, Test, UAT and Production. The best practice from a Denodo perspective is to have at least two, one production and one non-production.

• The Denodo Virtual DataPort metadata (core metadata) is deployed with the Solution manager to the production VDP servers and then synchronized from the Data Marketplace. This includes virtual databases, views, REST web services, folders, security tags, categories etc.
• The Data Marketplace metadata (business metadata) can be defined in the lower environments and then moved to production, and it can also be defined in production as well.

• Business  metadata includes categories, tags that are not part of the security setup and  property groups.

• Descriptors for data assets (views, view columns and corresponding REST objects) may be migrated from lower environments or defined and managed in production.

High Availability and Disaster Recovery

As an application running on Tomcat, the Data Marketplace component will scale both up (increase compute and memory power) and out (increase the number of containers or servers) with a load balancer in front of these containers or servers. Depending on your configuration (co-located with VDP or in separate servers) you will scale accordingly.

From a disaster recovery perspective metadata needs to be backed up periodically, typically after each deployment to production (which involves a platform migration, Data Marketplace synchronization and also an import of the extended metadata into production)  and after changes are made to production in the Data Marketplace as well. Furthermore, periodic backups  are recommended depending on the frequency with which the metadata stored in the Data Marketplace changes (e.g. if changes are made directly in the production environment).

Sizing and Testing

For production environments, the reference is to start with a separate server machine with 4 cores and 16 GB of RAM. Depending on your implementation (i.e. depending on the amount of metadata and the volume of requests received) it may be necessary to increase its size.

Basic Data Flows for Self-service & Data Discovery

In this section we look at self service and data discovery basic data flows from a business user perspective.  You want consumers to have the best possible data discovery and exploration experience.  

The basic data flow consists of finding the appropriate data products/assets that you will need and then exploring them by looking at its definition, descriptors, custom groupings and collaboration messages. The end user will be able to see the data and verify its lineage.

The objective is to make it easy for the consumers to find the relevant information and be able to trust it. This is done through a combination of several factors. Some of them are designed and implemented in the Denodo Virtual DataPort server, not the Data Marketplace itself. Let us take a look at them

• Virtual DataPort Objects: Folder and folder organization, data sources, views, view descriptions, column names and their descriptions. web services names and corresponding descriptions and security Tags.

• For good practices on naming conventions see VDP Naming Conventions.

• Business Objects: Categories, Business Tags and Property (custom) groups defined in the Data Marketplace itself.

A user may start the process in different ways:

• Traditional Natural language based search:  Where the consumer enters a string and searches for views, descriptors, categories, etc. This is typically a first step for discovery search similar to our daily online search experience
• Category-based search: Categories classify views and web services into discrete groups of similar or related items according to some criteria like use cases or business areas. Categories are hierarchical in nature (category = finance, subcategory = general ledger, subcategory = Accounts Payable, subcategory = Accounts Receivable, etc.)

• The best practice is to categorize your data products by business function or use case.

• Extended discovery search: The consumer will use the traditional natural language based search accessible from either the home page or top left navigation, and then refine the search  leveraging categories,  tags and optionally databases and collaboration options.
• Tag search: Tags are labels on views and web services.  Depending on their objective they can be governance-based, security-based, version control based, etc. In general they complement category search. Tag examples can be “corporate”, “certified” or “trusted”.

Category search shown below

First you invoke the categories then select those categories to filter on them

Extended discovery search shown below. We searched Sales by Region and then filtered out to see only financial accounting category data products.

Note: Categories,  tags and property groups must be defined and associated with views and web services.

Note: Security tags are typically defined in the Denodo Virtual DataPort server and used in combination with user roles and attributes to define global security policies. Business tags are defined in Denodo Data Marketplace.

Views and web services will have descriptions on the views and the columns themselves.

When you drill down on the object you will be presented with several tabs. the ones we are concerned with in the basic data flow are:

• Summary: A summary of the view or web service. It will contain description of the data product, categories it belongs to and tagging. Optionally it will display custom properties, though these can in the additional properties tab or group specific tab
• Schema: Displays the view or web services fields with descriptions
• Query: You can query the data by selecting columns, filters and order by. It is a best practice to define Relationships between views in Denodo. These can be used to query data across multiple related views, as well as navigate the result set between views
• Data preparation: The Data preparation tab allows you to build queries by manipulating live data.
• Assisted Query: The assisted query tab allows you to generate a query explaining what you need in your own language.
• Data lineage: The data lineage tab displays a diagram with the current view and all the views it depends on. You can check the path that follows the data for each field in the schema, from the data sources to the view, traversing all the intermediate views.
• Reference lineage: The reference lineage tab displays a diagram with the current view and all the reports that use it. You can check the path that follows the data from the view to the report, traversing all the intermediate elements.
• Requests: The requests tab displays the requests created on this view.
• Search: In the same way as the search by content, the search pill allows you to check which tuples of the view match some query terms.
• Lineage: A diagram of the data sources and views used to build the current view. It also displays the source where the information of each field comes from.

Below, lineage shows the source for the discount field in the iv_marketing_promotions_normalized view.

Descriptors for the fields and the data products can be managed in different ways.

• In the Virtual DataPort, define descriptors in the data sources and import them to the base views, then make modifications as necessary in the derived views. Finally automatically synchronize them with the Data Marketplace.
• In an inventory catalog define descriptors and then import them into Virtual DataPort. Finally, automatically synchronize them with the Data Marketplace. Naturally the inventory catalog will have to read the view and view columns first from Virtual DataPort.
• Define descriptors in the Data Marketplace itself letting the business users manage the business definitions.

For security tags:

• Define the security tags in the Virtual DataPort server and then synchronize them with the Data Marketplace in the production environment
• Define the security tags and their mapping to Denodo data products. If an organization has an existing Data Governance tool in place, the security tags and their mappings can be imported to the Denodo Virtual DataPort.  After that, synchronize them with the Denodo Data Marketplace in production.

With categories, business tags (remember security tags are defined in Virtual DataPort) and property groups, it is possible to define them and their mappings to data products inside the Data Marketplace or in an external tool such as an inventory catalog and then import them to the Data Marketplace itself. The picture below shows the flow

Augmented Discovery flows

Augmented capabilities

The launchpad to discover new data products in the catalog will be the same. In this section we discuss those additional capabilities in the Data Marketplace that enhance the end user experience.  

The end capabilities we are concerned with are

• Viewing federated data: Includes the multiple methods for viewing data in the graphical user interface. These are VQL Shell, Queries, Assisted Data Query, Data Preparation and as an extension when these queries are saved in the Data Marketplace local catalog queries. All these methods are discussed in the next section
• Relationships: If you define relationships in the Virtual DataPort, these will be synchronized with the Data Marketplace and users will be able to create queries leveraging these relationships thus navigating across Denodo views.
• Usage:  Basic usage information on the data product, like most common queries, average execution time or average number of records returned to the client
• Data Profiling: Profiling captured in Virtual DataPort such as data completeness or data distribution.
• Collaboration: Endorsements, warnings and deprecations. Users can contribute to enriching the metadata with endorsements and warnings, administrators can deprecate data products without removing them from the graphical user interface.

• It is a best practice to first deprecate a data product in the Data Marketplace and offer a period of time for the transition to the new one before it is actually removed from the GUI. This prevents existing functionality such as dashboards and reports from braking.

• Request Management: Users can create requests addressing data quality, changes or just raise questions to a queue where they will be managed. Users can also open requests for data access.

• It is a best practice to have this type of capability available to your consumers. You can manage this workflow inside the data catalog or with an external tool. 

• Self-Service Data Preparation: Business users can interactively filter, join, and transform datasets using sample-based previews, then save and share the results.
• Personalized Recommendations: Denodo 9 provides AI-powered personalized dataset recommendations based on user activity and preferences, helping consumers quickly discover relevant data products.
• Smart Query Acceleration: The Denodo Assistant analyzes past queries and recommends optimizations for improved query performance and accelerated data retrieval.

Data consumption within the Data Marketplace

When it comes to querying data in Denodo there are four basic ways to do it

• VQL Shell: Advanced users with knowledge of Denodo VQL  will be able to write custom queries against Denodo.

• Designed for power SQL users.  Initial VQL can be generated from other Query methods and then modified in VQL Shell, or VQL can be created from scratch.

• Query: Discussed in the main flow section. Users can create queries leveraging relationships with filtering and ordering capabilities. It is intended for technical users

• Designed for technical users: Includes filtering and order by clauses

• Assisted Query: GenAI integration so that you can use natural language to generate Denodo VQL. It will allow you to write questions in natural language. This capability interfaces with GenAI to produce the actual VQL that will be executed against Denodo.

• The quality of this interface will depend on having good descriptions for the fields in the query to optimize the results of the engine.

• Data preparation: (new in Denodo 9): Consumers can query the data in a business friendly excel-like environment, review the data as they apply different transformations, calculations and filters, create new columns and graphically manipulate data.

• It is intended for business users not concerned with the technical details of SQL.

Additionally, it is possible to save these queries locally to the Denodo Data Marketplace for later retrieval.

From a best practice perspective:

• SQL Savvy users will Use the VQL Shell
• Technical users can use Queries tab with
• Business users will use Data Preparation
• All users can utilize the assisted query

Assisted query GenAI - Natural Language Query (NLP) integration example

Integrate Metadata with Third-party Tools

Denodo 9 enhances the integration of metadata with third-party tools, providing more robust and flexible options for data consumers and administrators:

Built-in Connectors for BI Tools (External Tool Servers)

For seamless integration with popular Business Intelligence (BI) platforms, Denodo supports the configuration of External Tool Servers, such as for Microsoft Power BI and Tableau. This capability enables the Denodo Data Marketplace to ingest metadata from the BI tools (e.g., reports, dashboards), and conversely, it allows for the synchronization of Denodo's business metadata (like descriptions and tags) back into the BI environment.

REST API for Custom Integration

The Denodo Data Marketplace offers a comprehensive REST API that allows external tools to programmatically retrieve, create, and update business metadata such as tags, categories, and custom properties. This is the most flexible method for connecting Denodo's business metadata layer with external data governance platforms, master data management (MDM) tools, or other custom applications for synchronization and federation.

Import and Export Metadata

Denodo facilitates the import and export of metadata between the Data Marketplace and external tools. This includes metadata such as databases, views, web services, categories, tags, custom properties, saved queries, and usage statistics. These operations can be performed using built-in scripts or the REST API, providing flexibility in metadata management.

ODBC and JDBC Connectivity

Denodo supports ODBC and JDBC connectivity, allowing external tools to connect directly to the Denodo Virtual DataPort Server. This enables tools that support these protocols to access and interact with data managed by Denodo, facilitating seamless data integration and analytics.

Security

From a Denodo best practice perspective:

• Encrypt the communication between the end users and the Denodo Data Marketplace and between the Denodo Data Marketplace and the Virtual Data Port Server (also between the VDP server and the data sources) leveraging TLS (SSL)
• Implement Single Sign On through the Denodo Solution Manager portal.
• Secure data access controls in the Virtual DataPort server. See detailed documentation leveraging tagging and global security policies.  

Also, there are visibility rules for functionality specific to the Data Marketplace. For example, who can manage categories, tags, property groups or elements; in other words anything related to business metadata management will be configured in  the Data Marketplace itself.

References

Data Marketplace Videos

Using Denodo Data Marketplace Tutorial

Data Marketplace Guide

Course Catalog | Denodo

Fueling Self-Service with a Data Marketplace Approach

Set Up External Tool Environment