If you missed it, this week is the Microsoft Fabric Community Conference in Las Vegas. After two weeks of travel, I skipped this one but truth be told, I do miss being there.
Luckily, Aron Ulag shared a blog with the main releases / sneak previews and here are some highlights from my point of view.
First, we are transforming Microsoft Fabric’s CI/CD experience. This transformation includes support for data pipelines and data warehouses in Fabric Git integration and deployment pipelines. Spark job definition and Spark environment will become available in Git integration.
announcements from the Microsoft fabric community conference
Yes! More support for all the items in Fabric to get them into source control. I sincerely hope this will make deployment from development to test to production a lot easier. Let’s see how this works out!
For data pipelines, you can now access on-premises data using the on-premises Data Gateway—the same gateway used with dataflows.
announcements from the Microsoft fabric community conference
This way of accessing data will make a huge difference. In general, data pipelines are quite quick in copying data from a source system to Azure. Until now, getting data from an on-premises SQL Server was only possible through the dataflow Gen2 experience. Or exposing your database to the open internet, which isn’t a good idea. Ever. Now with the Gateway, there’s a more secure way to connect to your data. It’s been in use for PowerBI for quite some time now. Even though I still prefer a VPN connection to my data, this might prove a very valid alternative for many use cases.
A new feature we are bringing to Microsoft Fabric is called task flows. Task flows can help you visualize a data project from end-to-end.
announcements from the Microsoft fabric community conference
You need to read the blog for this one, as it has a very neat animated gif to go with it. You’ll find it about half way through. Not only is this a very nice way of visualising your ETL process, it can also help in detecting bottlenecks or design flaws. I’m really curious to see what it will look like with very complex design patterns but the first impressions are quite nice!
Last November, we shared a new, zero-ETL way of accessing and ingesting data seamlessly in near-real time from any database or data warehouse into the Data Warehousing experience in Fabric called Mirroring. We are thrilled to announce that Mirroring is now in preview, enabling Azure Cosmos DB, Azure SQL DB, and Snowflake customers to mirror their data in OneLake.
announcements from the Microsoft fabric community conference
This one is cut short for readability and for me this is the core of the message. This technique is like Change Data Capture (CDC) but not quite. It should have a very low latency between source system and Fabric. Exact numbers are hard to define as there are variables like networking at play here. But without too much work you can ingest your data into your raw or bronze layer. Again curious to see what happens with very busy source systems.
Fabric external data sharing, coming soon, enables you to share data and assets with external organizations such as business partners, customers, and vendors in an easy, quick, and secure manner.
announcements from the Microsoft fabric community conference
I was hoping for a service like this to be honest. Sharing data remains a struggle and having a set technology other organisations can connect to has a lot of potential. Excited to see how this works out!
Is that all?
No, there was much more in the key note, like the announcement of folders in the workspace. And more announcements on AI integration and PowerBI. But as most of you readers probably know, I’m working in the back-end of data. PowerBI is something I’ll happily leave to other smart people. As for the AI part, there are some cool things coming and I’m really happy some of my colleagues have a firm understanding of what’s happening there.
To get the full grasp of everything that’s happening, read the blog and keep a sharp eye on the Fabric Blog for all the other updates!
https://blog.fabric.microsoft.com/en-us/blog
What are your favourite bits in these announcements? Let me know in the comments!
Reitse's blog 