This document shows an architecture pattern for logical data warehouses for analytical workloads.

Logical Data Warehouse

The Logical data warehouse pattern is a mature and widely accepted analytical architecture pattern. This pattern was originally identified by Garner which considers it right now as the best practice for Analytical architectures.

This pattern consists of deploying a semantic layer on top of a data warehouse. It doesn't substitute a data warehouse, rather it augments the capabilities of a data warehouse that's why Gartner also refers to it as data warehouse augmentation.

• Access to All data
  By adding this semantic layer users can expand the scope of analytics because it's possible to integrate data from any data source, without forcing the user to copy all data into the data warehouse. The semantic layer federates the access to all data sources.
• Not a virtual data warehouse:
  It's not a virtual warehouse in the sense that the user is not supposed to send analytical queries against operational systems through this layer, data for analytics is previously populated into the warehouse using ETL Pipelines It doesn't come as a replacement of the data warehouse, rather it adds additional capabilities to the data warehouse.
• More agile approach for building data products:

• Declarative approach: The user defines the required data combinations and transformations using zero-code graphical interfaces following a declarative approach, under the hood Denodo generates SQL. Compared to building pipelines, which are procedural in nature, and the user needs to define every single step it's a much more agile approach.
• High reusability: It fosters reusability because the user can build new data products on top of existing ones, something that is very difficult to accomplish when building pipelines. This approach greatly accelerates the generation of data products, with significant reductions in data engineering efforts of up to 80% compared to traditional approaches based on building data pipelines.
• Easy to accommodate changes for supporting new business requirements: The user can easily accommodate any change given that it is a metadata-based layer. It is therefore very easy to accommodate new business requirements. The user can quickly show results to the business users that can provide feedback, apply the changes, show the new results again to the business users, offering a very quick turnaround shortening the data product development life cycle.

• Less data replication:
  Within the semantic layer the user can build virtual data marts instead of physical data marts, minimizing the need for replication and the derived costs for maintaining and operating physical marts.
• Single Point for enforcing security and governance:
  It offers a single point for enforcing security and governance, policies can be defined and applied in this layer, rather than doing that source by data source offering significant advantages.
• Single Source of Truth for Consuming Applications:
  It offers a single source of truth for consuming applications This is very important in terms of governance, compared to semantic layers built within a BI tool, this approach allows any BI tool or enterprise application to consume the same model, guaranteeing data consistency across all tools.
• Analytical Workloads:
  They are typically read-only workloads with a low concurrency and having to deal with medium to large data volumes. Denodo writes back to analytical stores for query optimization purposes in some cases, for instance data can be transferred to another store for maximising query push down in the data movement optimization mechanism.
• Advanced Query Optimization for Analytics:
  Denodo is able to deal with very large data volumes which are common in analytical environments. The Denodo´s query optimizer generates the most efficient query execution strategy by decomposing the workload into partial workloads that are submitted for execution to the data sources, pushing down data processing to them with the goal of minimizing the data transit from the data sources to the intermediate semantic layer and its subsequent processing in this layer, which is typically the bottleneck in this type of architectures. The Denodo query optimizer includes the most advanced query rewriting techniques for this purpose.
• Protect data sources for overloading:
  This intermediate layer also helps to protect data sources from overloading, the user can define workload management policies, limiting the maximum query execution time, the maximum number of rows returned, forcing filter conditions in queries to avoid full scan of tables and caching data to reduce the workload and further accelerating query execution.

When to use this Architecture Pattern

This architectural pattern is commonly found in environments with a lot of heterogeneity where the data warehouse is still the central piece of the data management architecture, mostly used for very structured data that needs to be governed under stringent data governance rules.

An example could be a financial institution that has to deliver financial reports to regulatory bodies. It´s common to find in these environments a data lake for the processing of unstructured data, with lesser data governance rules, to help generate insights for the business using machine learning models (e.g. churn analysis, generation of individual offers, etc.). The Logical Data Warehouse pattern allows the combination of data from both the data warehouse and the data lake.

References

Logical Data Management Reference Architecture

Denodo Architectures: Data Lake and Lakehouse

Denodo Architectures: Operational Workloads