Enhancing Clinical Data Infrastructure for AI Research: A Strategic Guide for Pharma Professionals

As healthcare organizations transition toward data-driven paradigms, the volume and variety of datasets, ranging from electronic health records (EHRs) and genomic sequencing to wearable sensor streams and high-resolution radiology images, have grown exponentially.^1,2,3 The global volume of healthcare data was estimated to reach 2,314 exabytes by 2020, and this trajectory has only accelerated with the adoption of Internet of Things (IoT) devices and remote patient monitoring platforms.^1,2 The successful deployment of AI applications in clinical settings hinges critically on the underlying data management architecture. AI models require high-quality, bias-controlled, and representative training data to avoid the well-known “garbage in, garbage out” phenomenon. Furthermore, transparency through provenance tracking and versioning, standards-based interoperability for seamless data exchange, and the ability to handle large, multimodal datasets are non-negotiable prerequisites for any organization serious about clinical AI.¹

To evaluate the efficacy of data management solutions in this demanding context, industry standards increasingly rely on two complementary frameworks. The FAIR principles ensure that data is findable through unique persistent identifiers, accessible via standard communication protocols, interoperable through machine-readable formats and standardized vocabularies such as SNOMED-CT and LOINC, and reusable with clear licensing and domain-specific metadata.⁴ The 5 V’s of Big Data address the technical challenges of managing large-scale clinical datasets:

• Volume (the sheer size of EHR, genomic, and imaging data)
• Variety (the heterogeneity of structured, semi-structured, and unstructured formats)
• Velocity (the need for both batch and real-time processing)
• Veracity (data quality, integrity, and traceability)
• Value (the ability to extract actionable insights for research and patient care).

Adopting this combined FAIR-5V perspective enables healthcare institutions to build future-proof infrastructures that support both clinical operations and AI research.¹

The Three Pillars of Clinical Data Architecture

Clinical Data Warehouses: The Bastion of Governance

The Clinical Data Warehouse (cDWH) has been the established standard in healthcare data management for decades. Designed as a central data hub, it harmonizes data from various clinical sources, enforces interoperability standards, and organizes patient data into highly structured, tabular formats using a “schema-on-write” approach. A defining characteristic of cDWHs is their adherence to atomicity, consistency, isolation, and durability (ACID) properties, which ensures that every transaction is dependable, and data integrity is maintained even in the event of system failures. Atomicity guarantees all-or-nothing execution, consistency ensures data remains valid before and after transactions, isolation prevents concurrent transactions from interfering with each other, and durability ensures committed data persists permanently.¹

These properties make cDWHs exceptionally well-suited for environments requiring strict regulatory compliance, reliable structured analysis, and straightforward auditing.^1,5,6 They provide a stable foundation for training robust machine learning models on clean, well-cataloged tabular data, and are particularly valuable for hospital governance, long-term trend analyses, and business intelligence dashboards. However, cDWHs face significant challenges in the modern AI landscape. The rapid influx of unstructured data, such as radiology images, clinical notes, and waveform recordings, makes it increasingly difficult to maintain the rigid, fixed schema approach. Every new data element requires ETL (Extract, Transform, Load) reengineering, and scaling up to accommodate images, notes, or real-time feeds is both costly and slow. Furthermore, the batch-oriented processing model of cDWHs can delay the detection of acute clinical events, limiting their utility for time-critical medical decision-making.¹

Clinical Data Lakes (cDL): Flexibility and Scale

To address the limitations of cDWHs, the Clinical Data Lake (cDL) emerged as an architectural approach that enables the storage of vast amounts of raw data in its original format, whether structured EHR records, semi-structured HL7 or FHIR messages, or unstructured radiology images and clinical notes. Unlike cDWHs, cDLs utilize a “schema-on-read” approach, allowing organizations to ingest and store data without defining a schema upfront.^1,7,8 This provides exceptional scalability and flexibility, enabling multimodal patient views and supporting near real-time data processing through distributed file systems and big data frameworks such as Hadoop and Spark.¹

cDLs are highly cost-effective for storing petabytes of diverse data and are well-suited for exploratory research, machine learning prototyping, IoT and streaming data analysis, and future-proofing data assets for emerging analytic methods. However, this flexibility comes at a significant governance cost. Without disciplined metadata management and robust data governance policies, a cDL can quickly degrade into a “data swamp,” a repository where data becomes unfindable, unreliable, and ultimately unusable. The absence of built-in enforcement for data types or business rules means that imputation logic and provenance tracking must be manually recoded for every study, posing challenges for maintaining data quality and veracity across large-scale research programs.¹

Clinical Data Lakehouses (cDLH): The Hybrid Frontier

The Clinical Data Lakehouse (cDLH) represents a newer, more complex hybrid approach that seeks to combine the scalable storage capabilities of a cDL with the structured queries, performance optimizations, and ACID transaction guarantees of a cDWH. First conceptualized by Zaharia et al. (2021), the lakehouse architecture applies open data formats and comprehensive data management techniques to provide a unified platform where raw data can be stored cost-effectively and processed in structured formats as required. In the clinical context, cDLHs combine a lake-style raw data landing zone with warehouse-style Delta or ACID tables and a single metastore that tracks both processed and unprocessed assets.^1,9

cDLHs offer the most comprehensive capabilities among the three architectures, supporting both real-time data ingestion and structured querying within a single platform. They are ideal for large, research-intensive institutions that need to integrate classic tabular data with vast amounts of unstructured research data such as images and omics data, enabling both standard reporting and advanced AI applications on the same dataset. However, cDLHs are extraordinarily complex, often cloud-native, and require interdisciplinary expertise spanning traditional data warehousing principles, modern big data methods (distributed systems, streaming), and DevOps practices including Docker, Kubernetes, and identity management. The initial setup and ongoing maintenance demand significant resources, posing a particular challenge for organizations with limited technical capacity.^1,10

Comparative Evaluation Across the 5 Vs of Big Data

To systematically assess these architectures, this article evaluates them against the 5 V’s of Big Data using an illustrative scoring framework derived from the qualitative findings of Gebler et al. (2025).¹ As shown in the radar chart below, cDWHs score highly on Veracity, reflecting their strict schema enforcement and ACID compliance, but lag significantly in Variety (limited to structured formats) and Velocity (batch-oriented processing). Conversely, cDLs excel in Volume and Variety due to their schema-on-read flexibility and distributed storage, but struggle with Veracity owing to weaker governance controls. The cDLH architecture achieves consistently high scores across all five dimensions, reflecting its hybrid design that balances governance with flexibility, though at the cost of greater implementation complexity.¹

Navigating Complexity: Implementation, Maintenance, and Expertise

Beyond technical performance across the 5 V’s, healthcare executives and technical leaders must carefully weigh the operational trade-offs associated with each architecture. Gebler et al. identify four critical dimensions of complexity: implementation effort, maintenance and scalability, required technical expertise, and coordination complexity.¹

Implementation Effort: cDWHs require detailed data models, extensive ETL development, and a stable database system, typically following a fixed project plan with defined phases for schema design, ETL development, testing, and go-live. Although the initial effort is high, the resulting structure is clear and stable for consistent data management. cDLs benefit from a schema-on-read approach that reduces upfront modeling, but continuous maintenance and expertise in big data technologies such as Hadoop and cloud object storage are essential. cDLHs demand the orchestration of both schema-on-write and schema-on-read functionalities, complex ETL or transform pipelines, robust security concepts, and distributed compute environments, resulting in the highest initial integration and development effort among the three architectures.¹

Maintenance and Scalability: cDWHs require continuous maintenance for schema extensions, data source changes, and performance tuning, with vertical scaling becoming increasingly expensive as data volumes grow. The batch-oriented structure can become a bottleneck for real-time applications. cDLs offer cost-effective horizontal scaling through distributed file systems but require active monitoring, strict data governance, and continuous quality management to prevent data swamp degradation. cDLHs support dynamic scalability for both compute and storage but involve the most complex maintenance regime, as both the lake and warehouse components must be independently managed and synchronized.¹

Required Technical Expertise: cDWHs rely on well-established skills in relational databases, SQL, and ETL tools. cDLs demand expertise in big data frameworks (Hadoop, Spark), streaming tools (Kafka, Flume), and cloud storage, along with proficiency in analyzing and modeling unstructured data. cDLHs require the broadest range of interdisciplinary expertise, combining traditional data warehousing knowledge with big modern data techniques and cloud-native DevOps practices, a combination that poses a significant staffing challenge for many healthcare organizations.¹

Strategic Recommendations and Application Scenarios

The optimal data management architecture depends heavily on an organization’s specific needs, available resources, data landscape, and strategic goals. Drawing on the comparative analysis and real-world implementations documented by Gebler, et al., the following scenario-based recommendations are presented.

It is important to note that cost analysis and legacy system integration also play critical roles in architecture selection. While cDWHs may have high upfront costs due to extensive ETL development and schema design, they often offer lower long-term operational costs due to their stable nature.^1,11 cDLs offer cost-effective scalability for large datasets but require ongoing investment in metadata management and system monitoring. cDLHs promise a balance between structure and flexibility but require significant initial integration costs and ongoing maintenance. Healthcare organizations should also consider their existing IT infrastructure: cDWHs generally offer smoother integration with established relational databases and clinical information systems, while cDLs and cDLHs may require additional transformation layers and custom connectors.¹

Building Resilient, Future-Proof Data Ecosystems

As the healthcare sector continues to embrace AI and advanced analytics, the underlying clinical data infrastructure must evolve to meet increasingly demanding requirements. Clinical Data Warehouses provide the necessary stability, governance, and auditability for structured reporting and regulatory compliance, but they are increasingly constrained by the volume and variety of modern medical data.^1,5,12,13 Clinical Data Lakes offer the requisite scale and flexibility for exploratory research and machine learning innovation, but they introduce significant governance risks that can undermine data quality if not carefully managed. The Clinical Data Lakehouse emerges as the most comprehensive solution, bridging the gap between structured reliability and unstructured flexibility while supporting both real-time and batch analytics on a unified platform.

However, the high technical complexity and implementation costs of cDLHs mean they are currently best suited for large, research-intensive institutions with the resources and expertise to manage hybrid environments.^1,10,14,15 Future research and development should focus on reducing the complexity of lakehouse implementations, improving the integration of clinical standards such as HL7 FHIR, and developing more accessible tooling that lowers the barrier to entry for smaller organizations. Ultimately, healthcare organizations must align their architectural choices with their immediate analytical needs, long-term scalability goals, and available technical expertise to build resilient, future-proof data ecosystems that serve both clinical operations and the advancing frontier of AI-driven medical research.¹

Disclaimer: The views expressed in the article are those of the authors and not of the organizations they represent.

References

1. Gebler R, Reinecke I, Sedlmayr M, Goldammer M. Enhancing Clinical Data Infrastructure for AI Research: Comparative Evaluation of Data Management Architectures. J Med Internet Res. 2025;27: e74976. doi:10.2196/74976.
2. Shilo S, Rossman H, Segal E. Axes of a revolution: challenges and promises of big data in healthcare. Nat Med. 2020;26(1):29-38. doi:10.1038/s41591-019-0727-5.
3. Baloch L, Bazai SU, Marjan S, Aftab F, Aslam S, Neo T, et al. A review of big data trends and challenges in healthcare. Int J Technol. 2023;14(6):1320. doi:10.14716/ijtech. v14i6.6643.
4. Wilkinson MD, Dumontier M, Aalbersberg IJ, et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data. 2016; 3:160018. doi:10.1038/sdata.2016.18.
5. Pavlenko E, Strech D, Langhof H. Implementation of data access and use procedures in clinical data warehouses: a systematic review of literature and publicly available policies. BMC Med Inform Decis Mak. 2020;20(1):157. doi:10.1186/s12911-020-01177-z.
6. Sebaa A, Chikh F, Nouicer A, Tari A. Medical big data warehouse: architecture and system design, a case study: improving healthcare resources distribution. J Med Syst. 2018;42(4):59. doi:10.1007/s10916-018-0894-9.
7. El Aissi ME, Benjelloun S, Loukili Y, Lakhrissi Y, El Boushaki AE, Chougrad H, et al. Data lake versus data warehouse architecture: a comparative study. In: Proceedings of the 6th International Conference on Wireless Technologies, Embedded, and Intelligent Systems. 2020. doi:10.1007/978-981-33-6893-4_19.
8. Quix C, Hai R. Data Lake. In: Sakr S, Zomaya AY, editors. Encyclopedia of Big Data Technologies. Cham: Springer; 2018.
9. Zaharia M, Ghodsi A, Xin R, Armbrust M. Lakehouse: a new generation of open platforms that unify data warehousing and advanced analytics. In: Proceedings of the 11th Conference on Innovative Data Systems Research. 2021.
10. Harby AA, Zulkernine F. From data warehouse to Lakehouse: a comparative review. In: Proceedings of the 2022 IEEE International Conference on Big Data. 2022. doi:10.1109/BigData55660.2022.10020719.
11. Nambiar A, Mundra D. An overview of data warehouse and data lake in modern enterprise data management. Big Data Cogn Comput. 2022;6(4):132. doi:10.3390/bdcc6040132.
12. Bellazzi R. Big data and biomedical informatics: a challenging opportunity. Yearb Med Inform. 2014;9(1):8-13. doi:10.15265/IY-2014-0024.
13. Lee CH, Yoon HJ. Medical big data promises and challenges. Kidney Res Clin Pract. 2017;36(1):3-11. doi: 10.23876/j.krcp.2017.36.1.3.
14. Begoli E, Goethert I, Knight K. A Lakehouse architecture for the management and analysis of heterogeneous data for biomedical research and mega-biobanks. In: Proceedings of the 2021 IEEE International Conference on Big Data. 2021. doi:10.1109/BigData52589.2021.9671534.
15. Xiao Q, Zheng W, Mao C, Hou W, Lan H, Han D, et al. MHDML: construction of a medical Lakehouse for multi-source heterogeneous data. In: Proceedings of the 11th International Conference on Health Information Science. 2022. doi:10.1007/978-3-031-20627-6_12.