Search for a command to run...
Programmatically defining the default lakehouse of a notebook
Principal Program Manager, Microsoft Fabric CAT helping users and organizations build scalable, insightful, secure solutions. Blogs, opinions are my own and do not represent my employer.
Hi Sandeep, thanks a lot. I have read both blogs. I am interested in what's your approach when promoting notebook to another environment (feature branch -> dev -> test -> prod). I do not like UI assignment of lakehouses, cause its then hardcoded in notebook metadata. :D I use configure to setup default lakehouse, but how to assign another one ? I am not sure about mounting it. cause you can not change it when moving to another stage.
Thanks
I believe I have found the solution. It is possible to parameterize %%configure like this
%%configure
{ "defaultLakehouse": { "name": { "parameterName": "defaultLakehouseName", "defaultValue": "FourthLakehouse" }, "id": { "parameterName": "defaultLakehouseId", "defaultValue": "773faa37-826f-4f9b-830f-e2a7a23e3903" } } }
I am using it to programmatically drop unnecessary tables in various lakehouses, but I also think I will use it to apply optimize, vacuum and vorder operations to tables. hope it helps
Great blog!
I'm wondering, is it possible to switch the default Lakehouse multiple times within a Notebook run?
If I understand correctly, changing the default Lakehouse by using the '%% configure' approach will only work properly if this is done in the first cell in the Notebook. I guess that means we can only set the default Lakehouse once in a Notebook run.
Some users want to be able to switch default Lakehouse programmatically multiple times within a Notebook run, in order to use Spark SQL on multiple Lakehouses within the same Notebook run.
Last week Microsoft released an open-source text embedding model called Harrier in three sizes- 270M, 0.6B and 27B. I have been testing it in my RAG pipeline and so far it has crushed all my metrics.
For Power BI Copilot and Data agents with semantic models, you must use Prep Data for AI configuration to ground the responses in the context added in Prep for AI. In this blog, I will show you how yo
At FabCon Atlanta last week, the updated notebook Copilot for data engineering and data science was announced. It brings agentic capabilities to the Copilot and is much more intelligent and Fabric-awa
Using Fabric data agent Python SDK
Use workspace monitoring to monitor MCP server use
I wrote a blog post a while ago on mounting a lakehouse (or generally speaking a storage location) to all nodes in a Fabric spark notebook. This allows you to use the File API file path from the mounted lakehouse.
Mounting a lakehouse using mssparkutils.fs.mount() doesn't define the default lakehouse of a notebook. To do so, you can use the configure magic as below:
%%configure -f
{
"defaultLakehouse": {
"name": '<name of the lakehouse>',
"id": '<lakehouse id>',
"workspaceId": '<workspace id of the lakehouse>'
}
}
This should be the first cell in your notebook. It will define the default lakehouse for the scope of the session. Once configured, you can query the Tables and files from Files section using the relative path for spark or the File API path. Note that unlike the GUI option of adding the lakehouse, you will not see the default lakehouse in the lakehouse explorer. You will not see the lakehouse to notebook lineage either in the workspace.
Notes:
Configure magic can be parameterized to use it in pipelines
You can add other configuration options as well for spark, default environments etc.
If you are running the notebook in a pipeline, pipeline will fail if configure magic is not on the first cell
The %%configure used in mssparkutils.notebook.run is going to be ignored but used in %run notebook will continue executing
I will also add that beyond being able to use the local filesystem, I don't think adding a default lakehouse does anything more. If you don't add a default lakehouse, you can always use the abfs paths in spark to query any table or file from any workspace you have access to.
The limitation of this method is that since the configure magic has to be the first cell, you can't programmatically assign the lakehouse id in the configure json. I thought of using the update notebook item API (see below) but it didn't work. I got 200 response code so the request went through successfully but the default lakehouse didn't get attached. There is an update notebook definition API which may work but it needs payload to be a base64 encoded ipynb. So the user will have to first get the current notebook definition using an API, edit the json to add metadata.trident.lakehouse properties, encode again and send it in a POST request - too much work. Let me know if you get it to work.
## THIS CODE DIDN'T WORK FOR ME - AS OF 7/26/2024 ##
import sempy.fabric as fabric
payload = {
"displayName": "<notebookname>", #keep the name of the existing notebook otherwise the API call fails
"metadata": {
"trident": {
"lakehouse": {
"default_lakehouse": "xxxxxxxxxxx",
"default_lakehouse_name": "xxxxxxxxxx",
"default_lakehouse_workspace_id": "xxxxxxxxx"
}
}
}
}
client = fabric.FabricRestClient()
wsid= "xxxxxxxxxx"
nbid="xxxxxxxx"
client.patch(f"v1/workspaces/{wsid}/items/{nbid}", json=payload)
#response code 200