Search for a command to run...
Using Semantic Link & APIs
Principal Program Manager, Microsoft Fabric CAT helping users and organizations build scalable, insightful, secure solutions. Blogs, opinions are my own and do not represent my employer.
No comments yet. Be the first to comment.
Last week Microsoft released an open-source text embedding model called Harrier in three sizes- 270M, 0.6B and 27B. I have been testing it in my RAG pipeline and so far it has crushed all my metrics.
For Power BI Copilot and Data agents with semantic models, you must use Prep Data for AI configuration to ground the responses in the context added in Prep for AI. In this blog, I will show you how yo
At FabCon Atlanta last week, the updated notebook Copilot for data engineering and data science was announced. It brings agentic capabilities to the Copilot and is much more intelligent and Fabric-awa
Using Fabric data agent Python SDK
Use workspace monitoring to monitor MCP server use
It’s always a good idea to use the latest GA runtime for the Spark pool in Fabric. Unless you change it manually, the workspace will always use the previously set runtime even if a new version is available. To help identify the runtime that workspaces are using and to upgrade multiple workspaces at once, use the code below, powered by Semantic Link.
Here I scrape the MS Docs to get the latest GA runtime and use the APIs to check/upgrade the runtimes.
import sempy.fabric as fabric
import requests
import pandas as pd
client = fabric.FabricRestClient()
def get_latest_ga_runtime(url=None):
"""
Get the latest GA Fabric runtime
ref : https://learn.microsoft.com/en-us/fabric/data-engineering/runtime
"""
url = "https://learn.microsoft.com/en-us/fabric/data-engineering/lifecycle" if url is None else url
df = pd.read_html(url)[0]
return (df[df['Release stage'] == 'GA']
.assign(Version=lambda x: x['Runtime name'].str.extract(r'(\d+\.\d+)'))
['Version'].max())
latest_rt = get_latest_ga_runtime()
def get_workspace_runtime(workspaceId=None):
"""
Get current runtime of the default environment of a Fabric workspace
"""
workspaceId = fabric.get_notebook_workspace_id() if workspaceId is None else workspaceId
try:
result = client.get(f"v1/workspaces/{workspaceId}/spark/settings")
return result.json()['environment']['runtimeVersion']
except (requests.RequestException, KeyError):
return "na"
workspaces = (fabric.list_workspaces()
.query('`Is On Dedicated Capacity`==True and `Name`!="Admin monitoring"') #Only P or F
.assign(runtime=lambda df: df['Id'].apply(get_workspace_runtime))
.assign(is_latest_rt = lambda rt:rt['runtime'] ==latest_rt ))
workspaces
You need to be a workspace admin to change the runtime version
outdated_rt_ws = workspaces[workspaces['is_latest_rt']==False]
display(outdated_rt_ws)
def upgrade_runtime(runtime:str, workspaceId=None):
workspaceId = fabric.get_notebook_workspace_id() if workspaceId is None else workspaceId
payload ={"environment": {"runtimeVersion": runtime }}
return client.patch(f"/v1/workspaces/{workspaceId}/spark/settings", json = payload)
## Either iterate over the above outdated_rt_ws dataframe or upgrade the runtime for one specific workspace
## if workspaceId=None, the workspace of teh notebook is used
upgrade_runtime(runtime = latest_rt, workspaceId=None)
Update :
You can also use Semantic Link Labs to get and update the spark settings. Thanks to Michael Kovalsky for the tip.
Needless to say, before changing the runtime version, ensure your spark jobs can run in the specified runtime.