Search for a command to run...
Highly efficient refresh operation in Direct Lake models
Principal Program Manager, Microsoft Fabric CAT helping users and organizations build scalable, insightful, secure solutions. Blogs, opinions are my own and do not represent my employer.
No comments yet. Be the first to comment.
Last week Microsoft released an open-source text embedding model called Harrier in three sizes- 270M, 0.6B and 27B. I have been testing it in my RAG pipeline and so far it has crushed all my metrics.
For Power BI Copilot and Data agents with semantic models, you must use Prep Data for AI configuration to ground the responses in the context added in Prep for AI. In this blog, I will show you how yo
At FabCon Atlanta last week, the updated notebook Copilot for data engineering and data science was announced. It brings agentic capabilities to the Copilot and is much more intelligent and Fabric-awa
Using Fabric data agent Python SDK
Use workspace monitoring to monitor MCP server use
I was writing a blog on Direct Lake incremental framing but my colleague Chris Webb beat me to it and just published an excellent blog. To summarize, with incremental framing, when a Direct Lake semantic model refreshes, it analyzes the Delta log to see what's changed since the last refresh:
It identifies which parquet files are new, which have been modified, and which have been removed
For unchanged data, it maintains the existing data in memory (preserving dictionaries and other optimizations)
It only reloads the data from new or modified parquet files
It removes from memory any data from deleted parquet files
You can read more about it in the documentation.
I will instead highlight another update based on the work by two of my other colleagues, Phil Seamark and Michael Kovalsky. Semantic Link Lab’s .delta_analyzer_history() function estimates the incremental framing effect based on the updates to the delta table. 0 means no benefit at all and 100% means highly effective. Note that this is based on the changes to the delta table and does not account for any refresh/updates made to the semantic model.
import sempy_labs as labs
labs.delta_analyzer_history("sales").tail(2) #sales is the name of the table in the lakehouse attached to the notebook
