From Warehouse to Lakehouse: A Life Sciences Playbook for Regulated, AI-Ready Data Modernization

Life sciences organizations are caught in a familiar bind: the pressure to generate faster, richer insights keeps growing, while the regulatory and privacy obligations that govern how data is handled grow right alongside it. Traditional data warehouses—the structured, relational databases that have anchored enterprise reporting for decades—were built for a different era. They excel at curated, structured reporting, but they were not designed to absorb the sheer variety of data that modern drug development and manufacturing demand: genomics files, medical images, high-velocity sensor streams from production lines, real-world evidence from claims databases, and the free-text narratives that populate pharmacovigilance case reports.

A cloud data Lakehouse offers a practical way through this tension. The term refers to an architecture that combines the low-cost, flexible storage of a data lake with the governance, reliability, and query performance traditionally associated with a data warehouse. Rather than forcing organizations to choose one or the other, a Lakehouse sits in between—storing raw data in open, vendor-neutral file formats while layering on the controls and structure that regulated industries require. Crucially, it does not demand a wholesale replacement of existing systems. The most successful migrations in life sciences have been deliberate and phased, building trust incrementally across technology, organizational culture, and the regulatory environment.^1,2,3

The core technology choices in this transformation hinge on adopting open storage formats, guaranteeing transactional robustness, instituting disciplined metadata and lineage controls, and ensuring seamless interoperability with existing reporting environments and AI platforms. Equally critical are the organizational decisions: securing committed executive sponsorship, establishing shared cross-functional stewardship, and navigating the transition period when legacy and modern systems must operate in parallel. All of this must remain resilient under the industry's regulatory pressures: GxP expectations, HIPAA and GDPR privacy obligations, and 21 CFR Part 11 requirements for electronic records.^4,9,12

Why Lakehouse, Why Now?

The data landscape in life sciences has changed faster than most legacy architectures can accommodate. A single phase III clinical trial today might draw on electronic data capture (EDC) records, patient-reported outcomes, central laboratory feeds, wearable sensor streams, imaging data, and real-world data sourced from external partners, all simultaneously.¹¹ A biologics manufacturing network must continuously synthesize batch records, process analytical technology (PAT) signals, environmental monitoring readings, and laboratory information management system (LIMS) results to maintain quality and meet release timelines. A traditional warehouse can handle selected slices of this information, but only after labor-intensive extract-transform-load (ETL) processes—the pipelines that pull data from source systems, reshape it, and load it into the warehouse—have run their course. Those pipelines introduce delay, and in a world in which clinical teams need to act on site performance data the same day it is generated, delay is a strategic liability.

The Lakehouse paradigm addresses this directly (Table I). Raw data land once they are in scalable cloud object storage, preserving the original form. From there, curated layers apply standardization, quality controls, and access policies in a structured progression. By leveraging open table formats and transaction logs, organizations gain consistency, “time travel” (the ability to query data exactly as it appeared at a specific point in the past, which is directly relevant to audit and inspection readiness), and reproducible analytics. Compute power can be scaled up or down independently of the workload: a compliance dashboard has very different performance requirements from a genome-wide association study. The strategic prize is not cheaper infrastructure, though that matters. It is the creation of governed data products that can power faster trials, sharper safety surveillance, and more defensible regulatory submissions from a single, unified foundation.

Table I: A Life Sciences Reference Architecture^4,5

Abbreviations: AI/ML, artificial intelligence/machine learning; CDISC, clinical data interchange standards consortium; CRM, customer relationship management; CTMS, clinical trial management system; EDC, electronic data capture; EHR, electronic health record; RWE, real-world evidence; ERP, enterprise resource planning; eTMF, electronic trial master file; HL7, health level seven; FHIR, fast healthcare interoperability resources; LIMS, lab information management system; MES, manufacturing execution system; QMS, quality management system; SDTM/ADaM, study data tabulation model/analysis data model.
Source: Adapted from: Databricks. What Is the Medallion Lakehouse Architecture. Available at docs.databricks.com/aws/en/lakehouse/medallion. Accessed Aug 6, 2026.

What This Looks Like in Practice: A Lesson from Roche

Roche encountered a challenge that will feel familiar to many. The organization had built a clinical-trials data hub that housed more than 360 studies and over 1 billion documents.⁶ By any standard, it was a massive and valuable data estate. Yet, the platform was beginning to buckle under its own scale. Legacy silos prevented EDC data (i.e., structured case-report-form records) from non-CRF sources, such as laboratory outputs, imaging files, and pharmacokinetic datasets. Manual handoffs slowed the pace of new capabilities. Compliance and reporting demanded heavy manual effort. And as data volumes exceeded 50 terabytes, the underlying architecture simply could not keep up.

Roche's response was methodical rather than dramatic. The team migrated the platform to a cloud-native architecture on AWS, integrating tools to support real-time ingestion and harmonization of both structured and unstructured clinical data. Crucially, they did not attempt a single cutover. Instead, they ran a phased implementation: first validating the new environment at small scale for security and access controls, then deploying a full production-like environment for performance tuning, and finally executing a validated, GxP-compliant go-live with about 2 days of migration downtime. The outcomes were concrete: release cycles shortened from weeks to days, search and ingestion performance improved substantially, and the platform achieved end-to-end compliance traceability across all trials. Total cost of ownership fell through cloud auto-scaling and infrastructure optimization.⁶

The Roche example is instructive because the approach was disciplined. The team validated before they cut over. They ran parallel environments long enough to build confidence. They treated compliance as a design requirement, not a post-migration checklist item. That discipline is the thread that runs through every successful Lakehouse migration in regulated industries.

Migration Strategy: Phase the Work Around Risk and Value

A pragmatic migration begins by sorting existing warehouse workloads into three distinct groups (Table II). The first comprises stable, highly regulated reporting assets (e.g., batch disposition reports, submission-ready datasets, validated quality dashboards) that demand stringent validation before any change. These should migrate only after data lineage and reconciliation have been definitively proven in the new environment. The second group includes high-value analytical workloads that are currently bottlenecked by warehouse costs or latency: clinical enrollment forecasting, manufacturing process monitoring, safety signal detection. These are strong candidates for early migration because the business case is clear and the tolerance for some transitional complexity is higher. The third group encompasses exploratory workloads (omics analysis, natural language processing over medical literature, real-world evidence studies) in which the Lakehouse delivers immediate net-new value, because legacy warehouses were never designed to handle them at all.

Table II: Illustrative Prioritization of Life Sciences Workloads

Source: Adapted from: Kah M, Mui J, Ames M, et al. Migrating a Research Data Warehouse to a Public Cloud: Challenges and Opportunities. J Am Med Inform Assoc. 2022;29(4):592-600.

The phased sequence that emerges from this segmentation—assessment, pilot, parallel run, validated cutover, optimization—is not merely a project management preference. It is a risk management strategy. Running the old and new systems in parallel for a defined period allows quality and regulatory teams to compare outputs, identify discrepancies, and sign off on equivalence before the legacy system is retired. Every parallel run should have a strict exit plan tied to measurable criteria: reconciled key metrics, accepted performance benchmarks, security sign-offs, and hard retirement dates for the redundant pipelines. Indefinite duplication drains budgets and creates confusion about which system holds the authoritative version of a given dataset.

Technology Choices That Matter

In regulated environments, technology choices carry weight beyond performance benchmarks. They must also support auditability, long-term data accessibility, and the ability to demonstrate to an inspector exactly where a number came from and what transformations it passed through.

Open storage formats are the foundation. Parquet is a column-oriented file format that stores data in a highly compressed, efficient structure optimized for analytical queries; it has become the de facto standard for lakehouse storage. Unlike proprietary formats tied to a single vendor's platform, Parquet files can be read by a wide range of tools, which matters for long-term data accessibility and regulatory submissions.

On top of Parquet, open table formats such as Delta Lake, Apache Iceberg, and Apache Hudi add a crucial layer of reliability.¹⁷ Think of these as the equivalent of a database transaction log for a data lake: they ensure that data operations are atomic and consistent (properties collectively described as ACID [Atomicity, Consistency, Isolation, Durability] compliance), support schema evolution as data structures change over time, and enable the time-travel queries mentioned earlier. For a quality team that needs to reconstruct exactly what a dataset looked like on the day a batch was released, this capability is a compliance requirement.

The medallion architecture—a layered approach that organizes data into bronze (raw, unmodified), silver (standardized and quality-checked), and gold (curated, business-ready) tiers—provides the structural framework for applying these technologies in a governed way. Each layer should have explicit business ownership, defined quality thresholds, and documented permitted use cases. The bronze layer is not simply a staging area; it is the immutable record of what was received from source systems, and it may itself be subject to retention and access controls. The gold layer is not simply a reporting database; it is a governed data product with a defined steward, approved business definitions, and audit trails.

Comprehensive metadata and data catalogs are non-negotiable in this architecture. Whether a clinical data manager preparing a submission package or a manufacturing quality engineer investigating a deviation, every user must be able to determine where a dataset originated, what transformations it underwent, and whether it is approved for the decision at hand. Without this, the lakehouse becomes a sophisticated data swamp rather than a trusted analytical foundation.

Performance engineering deserves deliberate attention as well. Clinical dashboards require predictable, rapid response times, whereas complex omics pipelines demand massive, burstable compute. Partitioning—dividing large datasets into smaller, manageable chunks based on meaningful attributes, such as study identifier, country, or manufacturing batch—should reflect actual query patterns rather than storage convenience alone. Data observability tools that continuously monitor for data freshness, volume anomalies, schema drift, and referential integrity are essential for catching problems before they affect downstream decisions.¹⁸

Governance, Validation, and Compliance

For life sciences, governance is not an afterthought appended to the end of a project. It is the core of the migration strategy itself, and it must be designed in from the beginning.

Every data domain requires a clearly accountable owner and steward—a designated individual or team responsible for ensuring the quality, accuracy, and proper use of the data within that domain. Governance policies must draw firm boundaries between exploratory research environments, non-GxP analytical workflows, and tightly controlled GxP-relevant reporting. GxP, or Good Practice, refers to the collection of quality standards and regulations that govern pharmaceutical manufacturing, laboratory operations, and clinical trials. Data supporting batch release, regulatory submissions, or clinical decision-making falls under GxP expectations and therefore demands a significantly higher level of validation than data used for internal analytics or business intelligence.

Where electronic records and signatures are involved, procedures must mandate rigorous access controls, audit trails, and system validation consistent with 21 CFR Part 11, the FDA regulation that establishes standards for electronic records and signatures in regulated industries.^9,10 HIPAA-regulated data demands minimum-necessary access, de-identification, and strong vendor controls.¹² GDPR obligations must be woven into the design of global studies, particularly where data subjects are European residents.¹³

Validation should be risk-based rather than uniform. Reports used for regulatory submissions or batch disposition require exhaustive test scripts, traceability matrices, and formal change control. Exploratory analytics pipelines do not. A validated lakehouse does not mean freezing innovation; it means maintaining a clear separation between locked-down production data products and agile experimental sandboxes, with a defined pathway for promising analytics to mature into governed assets. This separation is sometimes called a “promotion pathway” and is one of the most important governance constructs an organization can establish, because it allows the scientific community to move quickly without inadvertently compromising the integrity of regulated data products.^14,15

Operating Model and Change Management

Technology is the easier half of a lakehouse migration. The harder half is organizational.

The most successful programs dismantle traditional silos and form cross-functional squads that blend data engineers and architects with clinical subject-matter experts, regulatory operations professionals, and privacy officers. This is not simply good project management practice. It reflects a genuine shift in how data work gets done: from a model in which IT builds pipelines for business users to consume, to one in which data products are co-owned by the teams that depend on them. That shift requires new roles, including data product owners, catalog stewards, and responsible AI practitioners, as well as new accountability structures to support them.

Executive sponsorship is essential, and not merely as a formality. Migrations of this scope inevitably surface legacy metric inconsistencies, ownership disputes, and governance gaps that have accumulated over years. Resolving them requires authority that sits above any individual function. Without visible executive commitment, these issues tend to stall in committee rather than get resolved.

Training must extend well beyond instruction in new software tools. It must encompass the new responsibilities that come with a lakehouse operating model: understanding what it means to own a data product, using a data catalog to assess fitness for purpose, and applying responsible AI practices when building on governed data assets.

A Powerful Data Foundation

A lakehouse migration in life sciences is not, at its core, a technology project. It is a business and compliance transformation enabled by cloud architecture. The organizations that do it well are the ones that resist the temptation to treat it as an IT initiative and instead engage clinical, quality, regulatory, and commercial stakeholders as co-architects of the outcome.

The most effective programs preserve what legacy warehouses do well (trusted reporting, quality controls, strict governance) while expanding the enterprise's ability to work with the full range of scientific, clinical, and operational data generated by modern drug development and manufacturing. They start with high-value use cases that demonstrate the architecture's potential without exposing the organization to unnecessary risk (Table III). They prove the approach through targeted parallel runs before cutting over. They invest heavily in metadata stewardship, because data that cannot be understood and trusted will not be used. And they scale through reusable data products, so that each new use case builds on the governed foundation rather than starting from scratch.

Table III: Illustrative Life Sciences Use Cases⁸

Source: Adapted from: Kah M, Mui J, Ames M, et al. Migrating a Research Data Warehouse to a Public Cloud: Challenges and Opportunities. J Am Med Inform Assoc. 2022;29(4):592-600.

The result, when the work is done with discipline, is a data foundation capable of powering faster trials, safer products, and responsible AI, and one that holds up under regulatory scrutiny when it matters most.

Disclaimer: The views expressed in this article are those of the authors and not of the organizations they represent.

References

1. Armbrust M, Ghodsi A, Xin R, et al. Lakehouse: a new generation of open platforms that unify data warehousing and advanced analytics. Proc CIDR. 2021.

2. Nambiar A, Mundra D. An overview of data warehouse and data lake in modern enterprise data management. Big Data Cogn Compute. 2022;6(4):132.

3. Harby AA, Zulkernine F. Data Lakehouse: a survey and experimental study. Inf Syst. 2025;127:102460.

4. Databricks. Lakehouse architecture and medallion architecture technical documentation. Accessed July 29, 2026. View source

5. Inmon B, Levins M. Evolution to the data lakehouse. Databricks Blog. Published May 19, 2021. View source

6. Datavid. From bottlenecks to breakthroughs: Modernizing clinical data at scale—Roche case study. Accessed July 2026. View source

7. Harby AA, Zulkernine F. From data warehouse to lakehouse: a comparative review. 2022 IEEE Int Conf Big Data. 2022:2889-2898.

8. Kahn MG, Mui JY, Ames MJ, et al. Migrating a research data warehouse to a public cloud: challenges and opportunities. J Am Med Inform Assoc. 2022;29(4):592-600.

9. US Food and Drug Administration. Data integrity and compliance with drug CGMP: questions and answers guidance for industry. Published 2024.

10. US Food and Drug Administration. Electronic systems, electronic records, and electronic signatures in clinical investigations: questions and answers guidance for industry. Published 2023.

11. US Food and Drug Administration. Decentralized clinical trials for drugs, biological products, and devices: guidance for industry, investigators, and other stakeholders. Published 2023.

12. US Department of Health and Human Services. Summary of the HIPAA privacy rule. HHS Office for Civil Rights.

13. European Medicines Agency. Guideline on computerized systems and electronic data in clinical trials. Published 2023.

14. International Council for Harmonization. ICH E6(R2) good clinical practice. Published 2016.

15. International Council for Harmonization. ICH E6(R3) good clinical practice draft guideline. Published 2023.

16. Clinical Data Interchange Standards Consortium. CDISC standards for clinical research data, including SDTM and ADaM.

17. Linux Foundation. Delta Lake, Apache Iceberg, and Apache Hudi open table format project documentation.

18. Amazon Web Services, Microsoft Azure, Google Cloud. Documentation on cloud data lakes, analytics governance, encryption, and healthcare/life sciences compliance controls.