Search for a command to run...
Cleaning the lakes
Principal Program Manager, Microsoft Fabric CAT helping users and organizations build scalable, insightful, secure solutions. Blogs, opinions are my own and do not represent my employer.
Hi Sandeep,
Thank you for sharing your knowledge and findings with us.
I have a quick question: When running the code, I’m encountering the ArtifactConnection_InaccessibleDatasource error. I’m both a Capacity Admin and a Workspace Admin. How can I resolve this issue?
Last week Microsoft released an open-source text embedding model called Harrier in three sizes- 270M, 0.6B and 27B. I have been testing it in my RAG pipeline and so far it has crushed all my metrics.
For Power BI Copilot and Data agents with semantic models, you must use Prep Data for AI configuration to ground the responses in the context added in Prep for AI. In this blog, I will show you how yo
At FabCon Atlanta last week, the updated notebook Copilot for data engineering and data science was announced. It brings agentic capabilities to the Copilot and is much more intelligent and Fabric-awa
Using Fabric data agent Python SDK
Use workspace monitoring to monitor MCP server use
My friend Alex Powers (PM, Fabric CAT) wrote a blog post about cleaning the staging lakehouses generated by Dataflow Gen2. Before reading this blog, go ahead and read his blog first on the mechanics of it and the whys. Note that these are system generated lakehouses so at some time in the future, they will be automatically purged but until then the users will be paying the storage cost of these lakehouses. If you want to read more about how dataflow gen2 works and whether you should stage or not , read this and this blog.
A question was asked on Reddit, how do you get the size of these lakehouses to determine the impact. Here is how you do that :
import sempy.fabric as fabric
import pandas as pd
def get_staging_lakehouse_size():
"""
Fabric.Guru | Sandeep Pawar | 07/19/2024
Get the size of each DataflowsStagingLakehouse in the capacity in MB
"""
ppu_capacities = fabric.list_capacities().query('Sku == "PP3"') # Exclude PPU workspaces
ws = fabric.list_workspaces().query('`Is On Dedicated Capacity`==True') # Only Premium
ws = ws[~ws['Capacity Id'].isin(ppu_capacities.Id)]
results = []
for _, row in ws.iterrows():
try:
lakehouses = mssparkutils.lakehouse.list(row['Id'])
for lakehouse in lakehouses:
if lakehouse['displayName'] == "DataflowsStagingLakehouse":
abfs_path = lakehouse['properties']['abfsPath']
path = abfs_path + "/Tables"
tables = [f.path for f in mssparkutils.fs.ls(path)]
total_size_mb = max(1, sum(round(sum([f.size for f in mssparkutils.fs.ls(table)]) / (1024 * 1024)) for table in tables))
results.append({'Name': row['Name'], 'Size_MB': total_size_mb})
except:
pass
return pd.DataFrame(results).sort_values('Size_MB', ascending=False)
result_df = get_staging_lakehouse_size()
result_df
The pseudo-code is:
Get Premium/Fabric workspaces, excluding PPU
Identify workspaces with staging lakehouses
For each lakehouse, iterate over the tables and calculate the size of the delta tables in MB
I didn't know I had accumulated 25B of gunk in the Onelake that I was paying for. You can use this to decide/automate which workspaces to clean up.
If you want to nuke all the tables in these lakehouses, just adjust the code above to drop the tables. I considered sharing that, but I don't want you to blame me if your dataflows suddenly stopped working :P You do it at your own risk :D
Cleaning the (Staging) Lakeside - It's Not About The Cell (itsnotaboutthecell.com)
Fabric Dataflows Gen2: To stage or not to stage? (crossjoin.co.uk)