Every few weeks someone on r/dataengineering asks if anyone is actually moving away from Medallion. I build these on Databricks and Fabric, and I inherit plenty that teams built themselves first, so here’s my honest answer.
Most Medallion builds are run as a project, scoped and shipped and walked away from, when they need to be built like a product that keeps working as the systems feeding it change.
The three layers are Bronze, Silver, and Gold. About 3/4 of the teams I see use exactly this shape. The naming gets misused constantly, so start with what each layer is actually for, not just what it holds.
None of this is new. Data teams were running staging, cleansing, and a presentation layer decades before anyone called it Bronze, Silver, and Gold. Inmon’s staging and enterprise warehouse, the old ODS-to-semantic-model flow, hub-and-spoke. The names are recent.
Maturing data through stages is not. That is a point in the pattern’s favor, not against it. It has survived 30 years of renaming because the underlying idea holds.
People assume Bronze is just a dumping ground, but it has a specific job. If a source system fails or breaks tomorrow, Bronze is the point you can restart your warehouse from. You stay independent. You don’t break because someone upstream broke. That is the reason it exists, and it is why skipping it to save time is a false economy.
The data moves from Bronze to Silver, and on the way you fix it. Remove the noise, correct the bad values, handle the fields the source stored badly. Silver is intermediate. It is not where reporting happens, and most business users will never touch it. What you get at the end is data you can actually rely on.
The validation we almost always have to add here is quarantine. In-house builds tend to drop records that fail a rule, silently, so a row that doesn’t parse just disappears. We route failed records to a quarantine table instead, so the failure is visible and auditable rather than a number that doesn’t add up three months later.
This is where the shift happens that data people describe as OLTP to OLAP: from transactional records to an analytical model built for fast reading.
You aggregate. You denormalize. Ten tables in Silver might collapse to 3 or 4 in Gold. You pre-calculate the numbers the business asks for over and over, so gross profit or net income is already sitting in the table when someone opens the report, not recalculated on the fly every time.
The Gold problem we see most is a governance one.
Every team builds its own Gold table for its own version of a metric, and you end up with five definitions of “active customer” in five tables, none of which agree. The fix is shared metric definitions governed at the Gold layer. Piling on more tables just deepens the mess. The inconsistency it creates is what erodes trust in the data.
Storage is cheap now. Compute is not. So you accept duplicating data into Gold, because paying for a bit more storage is cheaper than paying for the compute every time a slow report runs.
Not every setup stops at three layers. When a client is pulling in messy, varied inputs, I often add a Raw tier in front of Bronze.
Think about what a large enterprise actually ingests. Structured data from Oracle, Teradata, SQL Server. Semi-structured CSV and Excel exports. Raw JSON and XML files. Clickstream and analytics feeds that arrive as flat streams of whatever the source felt like sending. That variety is the problem Raw solves.
Bronze holds the same records as the source, but shaped into a structured, tabular form. Raw goes one step further back. It keeps the data and its original format exactly as they arrived. A JSON stays a JSON. Same number of records, same data types, nothing touched.
Say a source like Google Analytics stops sending data, because the subscription ended or the API changed. Your data is still sitting in your system in its original source format. You can restart the whole pipeline from Raw without depending on that third party at all.
Bronze gives you independence from source downtime. Raw gives you independence from the source existing.
Raw pays off when you’re consolidating messy, multi-format data from many sources or APIs. On a clean single-source pipeline it is mostly duplicated data and extra pipelines to maintain, and some very good engineers will tell you it is not worth it there. They’re right for that case. Add the tier for the problem it solves, not because four sounds more thorough than three.
| Decision | Bronze | Silver | Gold |
|---|---|---|---|
| Schema Enforcement |
Minimal. Accept the data as it comes |
Strict. Validate and quarantine bad records |
Strict. Enforce the consumption-ready structure |
| Data Retention | Full history, append-only | Validated records, keep reprocessing history | Current and historical aggregates as needed |
| Who can see it | Data engineering only | Data engineering, data science, analytics engineering | All authenticated business users |
| Transformation depth | None beyond metadata | Cleansing, deduplication, validation | Aggregation, modeling, metric calculation |
| Write pattern | Append-only | Merge or upsert to deduplicate | Full rebuild or incremental |
| Primary consumers | Pipelines, audit, reprocessing | Data scientists, ML pipelines, analytics engineers | BI tools, analysts, AI systems, executives |
| Failure handling | Accept and log, never fail on bad data | Quarantine invalid records and alert | Fail loudly, the quality gate has to hold |
| What Drives the Choice | Points to Databricks | Points to Fabric |
|---|---|---|
| Existing stack and licensing | Already invested in Databricks, team trained on it | A Microsoft shop, existing Fabric or Azure licensing |
| Team skills | Spark and notebook-heavy engineering teams | Teams centered on Power BI and the Microsoft toolset |
| Where you’re heading | Modernizing hard toward AI and ML | Standardizing on one Microsoft ecosystem |
Here’s the setup we use on client engagements, the large US retailer included, when the client runs on Fabric.
| Layer | Fabric Tools | What We Do There |
|---|---|---|
| Bronze |
Fabric pipelines, Dataflows Gen2, OneLake shortcuts |
Land source data in OneLake as-is. Keep ingestion timestamps and source identifiers on every record, lock down access and hold source history to the client’s retention rules. |
| Silver |
Notebooks, SQL, Dataflows Gen2, Delta tables in a lakehouse |
Standardize formats, remove duplicates, handle missing values and apply shared business rules. Manage schema changes and quarantine invalid records for review. |
| Gold |
Lakehouse or warehouse, Power BI semantic models |
Publish fact and dimension models and the aggregations the business asks for. Define metrics once and apply row- and object-level security so each team sees only the data it should. |
The best answer I ever read in one of those “moving away from Medallion” threads was that the job of an architect is to pick the pattern that fits the problem, and that the worst architects know one way of doing things and force it onto everything.
Three governed layers on every pipeline, no matter the data, is that mistake. So here is where I tell clients not to reach for the full pattern.
Start with the question everyone actually asks: is three layers overkill, can I simplify?
Usually the honest answer is that three is the floor, not the ceiling. Going below it tends to cost more than it saves. The real simplification lever is not how many layers you have, it is how much each one does.
There is one simplification that backfires almost every time: cutting Silver. A team decides full Medallion is too much, drops the middle, and loses its model of the business. What is left is a pile of reporting tables with duplicated logic that slowly drifts out of sync, because nothing is genuinely reusable. Simplify what the layers do. Don’t delete the layer that holds your definition of the business together.
If we look at your setup and three governed layers is more than the problem needs, we say so. If we don’t see a clear return in building it, we don’t take the engagement. That is easier to say up front than to walk back later.
Remember the rule that you never expose Bronze or Silver. That is shifting, and it is worth understanding why, because it changes how you should design today.
Traditional consumers only ever wanted Gold: clean, aggregated, ready to chart. Data scientists and the teams building on LLMs want the opposite. They don’t want your cleansed and denormalized data. They want the raw information, so they can find their own patterns and run their own inference on the source as it actually was. The cleansing that makes Silver good for reporting throws away signal they need. We already see data scientists asking for Bronze access directly for exactly this reason.
So a Medallion build meant to serve AI has to do two jobs at once. Govern Gold tightly for the business, and keep a clean path back to raw Bronze and Silver for the data science and AI work. This is also where Raw-layer retention earns its place. A model that has to be retrained on years of history needs source data going back further than most analytics ever asks for, and it needs it in its original form. Design for that now and you avoid rebuilding the whole thing the day someone asks for it.
