Direct Lake Incremental Framing Effect

I was writing a blog on Direct Lake incremental framing but my colleague Chris Webb beat me to it and just published an excellent blog. To summarize, with incremental framing, when a Direct Lake semantic model refreshes, it analyzes the Delta log to see what's changed since the last refresh:

• It identifies which parquet files are new, which have been modified, and which have been removed

• For unchanged data, it maintains the existing data in memory (preserving dictionaries and other optimizations)

• It only reloads the data from new or modified parquet files

• It removes from memory any data from deleted parquet files

You can read more about it in the documentation.

I will instead highlight another update based on the work by two of my other colleagues, Phil Seamark and Michael Kovalsky. Semantic Link Lab’s .delta_analyzer_history() function estimates the incremental framing effect based on the updates to the delta table. 0 means no benefit at all and 100% means highly effective. Note that this is based on the changes to the delta table and does not account for any refresh/updates made to the semantic model.

import sempy_labs as labs
labs.delta_analyzer_history("sales").tail(2) #sales is the name of the table in the lakehouse attached to the notebook

References:

• microsoft/semantic-link-labs: Early access to new features for Microsoft Fabric's Semantic Link.

• Direct Lake overview - Microsoft Fabric | Microsoft Learn

• Performance testing Power BI Direct Lake models revisited: ensuring worst-case performance

• Chris Webb's BI Blog: Performance Testing Power BI Direct Lake Mode Datasets In Fabric