Search for a command to run...
Automation, Convenience and Flexibility
Principal Program Manager, Microsoft Fabric CAT helping users and organizations build scalable, insightful, secure solutions. Blogs, opinions are my own and do not represent my employer.
No comments yet. Be the first to comment.
Last week Microsoft released an open-source text embedding model called Harrier in three sizes- 270M, 0.6B and 27B. I have been testing it in my RAG pipeline and so far it has crushed all my metrics.
For Power BI Copilot and Data agents with semantic models, you must use Prep Data for AI configuration to ground the responses in the context added in Prep for AI. In this blog, I will show you how yo
At FabCon Atlanta last week, the updated notebook Copilot for data engineering and data science was announced. It brings agentic capabilities to the Copilot and is much more intelligent and Fabric-awa
Using Fabric data agent Python SDK
Use workspace monitoring to monitor MCP server use
First I would like to thank Edgar Cotte (Sr PM Fabric CAT) for the inspiration for this blog. Edgar shared this recently in a workshop so I got curious and explored more. Fabric CLI, as the name suggests, allows you to interact with the Fabric environment using command line interface. Whether you are a Fabric admin or a developer, it’s essential for exploration and automation from a convenient interface. But if you work with external clients like I do where there may be restrictions on installing libraries, using Fabric CLI in Fabric notebook may be an easier option especially if you are already familiar with all the commands.
To learn more about Fabric CLI, refer to:
Fabric command line interface - Microsoft Fabric REST APIs | Microsoft Learn
Microsoft Fabric CLI: Turbo-charge Ops with Retro Command-Line Power
In Fabric python notebook, first install fabric cli :
%pip install ms-fabric-cli --q
This will use the identity of the user executing the notebook. You can also use service principal authentication as shown here.
token = notebookutils.credentials.getToken('pbi')
os.environ['FAB_TOKEN'] = token
os.environ['FAB_TOKEN_ONELAKE'] = token
There are two types of commands - command line and interactive. In the notebook, you should be able to run all commands supported in command line mode. The documentation does a great job of describing if a command can be used in command line, interactive or both.
There are two ways you can execute the supported commands - using magic or using a Python wrapper for the commands.
Once installed and auth setup, use !fab magic to run the shell command. Below I get a list of all the workspaces along with the capacity attached and capacity id.
To get a list of all items in a workspace:
This works even if you have spaces in item/workspace names:
You can use JMESPath in the query or you can also use shell commands like below to filter the lines that contain word “Trial” and print the first column (i.e. workspace names)
If a command is not available, you can call an API inline as well:
To download items to an attached default lakehouse:
!fab allows only one line command. To create more complex multi line commands, you can use %%sh cell magic like below:
%%sh
echo "=== Exploring Reid Workspace ==="
# jump to workspace
fab -c "cd Reid.Workspace" || echo "Failed to navigate"
echo "Listing all items:"
fab -c "ls Reid.Workspace -l" || echo "No items or access denied"
echo ""
echo "Notebooks only:"
fab -c "ls Reid.Workspace" | grep "\.Notebook" || echo "No notebooks found"
echo ""
echo "Reports only:"
fab -c "ls Reid.Workspace" | grep "\.Report" || echo "No reports found"
echo "Exploration completed"
You can also pass Python variables to commands, making it very dynamic:
fabric-cli library is installed in the default Python runtime with the token set up so users can use !fab easily. That would be super handy.Above method is great for interactive exploration. For automation, you can create Python functions and use it like any other Python function. For example, I downloaded one notebook but for more complex patterns/automation, I can use subprocess.run() to execute the commands.
To download all notebooks from a workspace to a lakehouse location:
def download_item(workspace, item_name, local_path="/lakehouse/default/Files/tmp", force=True):
"""
Download a Fabric item to local dir
"""
try:
os.makedirs(local_path, exist_ok=True)
cmd = f"export {workspace}.Workspace/{item_name} -o {local_path}"
if force:
cmd += " -f"
result = subprocess.run(["fab", "-c", cmd], capture_output=True, text=True)
if result.returncode == 0:
print(f"Downloaded {item_name} to {local_path}")
return True
else:
print(f"Failed: {result.stderr}")
return False
except Exception as e:
print(f"Error: {e}")
return False
For loop:
def download_all_notebooks(workspace, local_path="/lakehouse/default/Files/tmp"):
"""Download all notebooks from workspace"""
try:
result = subprocess.run(["fab", "-c", f"ls {workspace}.Workspace"], capture_output=True, text=True)
notebooks = [line.strip() for line in result.stdout.split('\n') if '.Notebook' in line]
success_count = 0
for notebook in notebooks:
if download_item(workspace, notebook, local_path):
success_count += 1
print(f"Downloaded {success_count}/{len(notebooks)} notebooks")
return success_count
except Exception as e:
print(f"Error downloading: {e}")
return 0
You can generalize this to create re-usable functions, e.g. below lists all workspaces:
import subprocess
import json
from typing import Optional, Dict, Any
def exec_fabcli(command: str, capture_output: bool = False, silently_continue: bool = False) -> Optional[str]:
"""
Run a Fabric CLI command from within a notebook.
Args:
command: The fab command to run (without 'fab' prefix)
capture_output: Whether to capture and return the output
silently_continue: Whether to suppress exceptions on errors
Returns:
Command output if capture_output=True, otherwise None
"""
try:
result = subprocess.run(["fab", "-c", command], capture_output=True, text=True, timeout=60)
if not silently_continue and result.returncode != 0:
error_msg = f"Command failed with exit code {result.returncode}\n"
error_msg += f"STDOUT: {result.stdout}\n"
error_msg += f"STDERR: {result.stderr}"
raise Exception(error_msg)
if capture_output:
return result.stdout.strip()
else:
# output
if result.stdout:
print(result.stdout)
if result.stderr:
print(f"Warning: {result.stderr}")
except subprocess.TimeoutExpired:
raise Exception("Command timed out after 60 seconds")
except FileNotFoundError:
raise Exception("Fabric CLI not found. Make sure 'fab-cli' is installed")
def list_workspaces(detailed: bool = False, show_hidden: bool = False) -> str:
"""List all workspaces"""
flags = ""
if detailed:
flags += " -l"
if show_hidden:
flags += " -a"
return exec_fabcli(f"ls{flags}", capture_output=True)
print(list_workspaces(detailed=True))
Semantic Link and Semantic Link Labs also provide similar capabilities but if you are already using CLI, this is a convenient way to reuse what you already know. This will especially be helpful for automating workflows. Different tools, different use cases. Fabric CLI can be usedf or CI/CD with Github and ADO automation. Jacob Knightley shared this excellent comparison at FabCon :
Here are additional resource to learn more:
Fabric CLI to deploy FUAM : fabric-toolbox/monitoring/fabric-unified-admin-monitoring/scripts/Deploy_FUAM.ipynb at main · microsoft/fabric-toolbox
Demos by Aitor Murguzur: murggu/fab-demos
Multi-tenant scenario: alisonpezzott/pbi-ci-cd-isv-multi-tenant: CI/CD scenario Multi Tenant for Microsoft Power BI PRO projects by utilizing fabric-cli and GitHub Actions
ecotte/Fabric-Monitoring-RTI by Edgar Cotte
fabgptI can’t complete the blog without mentioning AI, can I ? :D I love how Fabric CLI allows you to query, automate so easily. I was wondering what if all of that could be achieved with natural language? In the below example, I used Open AI + notebook cell magic to wrap above Python functions in a cell magic which I am calling fabgpt :D It generates above Python code and executes it.
