This months T-SQL Tuesday contribution highlights a couple of points about developing metadata-driven frameworks for Microsoft Fabric to cater for change detection.
Since this months T-SQL Tuesday invitation invites us to cover change detection I decided to put my own unique spin on it. By sharing some advice about how to develop metadata-driven frameworks efficiently and contributing to open-source solutions. Along the way I share plenty of links.
Before I go any further I just want to say a big thanks to Meagan Longoria for hosting this months T-SQL Tuesday.
Change detection with metadata-driven frameworks
I had various options for this months contribution due to my experience with various change detection solutions. Including Azure Synapse Link for SQL Server 2022. Which I covered in previous posts. Including one that covered some excessive file tests for Azure Synapse Link for SQL Server 2022.
In the end I decided to cover developing metadata-driven frameworks for Microsoft Fabric. Due to the fact that it is such a hot topic for multiple reasons. One of which is the growing availability of open-source, metadata-driven frameworks for Microsoft Fabric.
For example, the Fabric Essentials listings alone lists the three frameworks below:
Most of these frameworks contain logic that allow you to work with change detection. With varying levels of options and efficiency. Including the ability to perform incremental updates based on change detection against the source system.
For example, the ELT Framework that is deployed as part of Fabric Accelerator caters for this. By allowing you to store delta/watermark values provided by the source system. So that the next time you perform ingestion you start from that value.
Advice about metadata-driven frameworks
One piece of advice is to make sure the metadriven-framework you develop is scalable. Because even if you develop an amazing ingestion method that includes a small language model, the reality is that over time the number of sources and source types are likely to expand.
Another piece of advice is for those of you looking to develop your own metadata driven framework. I strongly suggest customizing an existing one instead of reinventing the wheel. I recommend this because a lot of existing ones have matured over time and cater for a number of sources and scenarios.
For example, the Fabric Accelerator solution by Benny Austin contains a mature IngestDefinition table that caters for a number of data sources and change detection methods.
If you do decide to customize an open-source solution you found online, I recommend contributing to that solution if you develop something that others can benefit from. For example, if you improve change detection for an open-source solution that is shared in GitHub contribute to that project.
Of course, if you do end up developing an amazing ingestion method that includes a small language model, I highly recommend sharing the details about it with others as well.
Epilogue
To summarize, I hope that this advice about developing metadata-driven frameworks for Microsoft Fabric to cater for change detection helps some of you develop more efficiently. Plus, I hope it encourages more of you to contribute to existing solutions.
1 thought on “T-SQL Tuesday 198 – Developing metadata-driven frameworks for Microsoft Fabric to cater for change detection”