Search for a command to run...
Backing up my Fabric Trial workspaces
Principal Program Manager, Microsoft Fabric CAT helping users and organizations build scalable, insightful, secure solutions. Blogs, opinions are my own and do not represent my employer.
Sandeep - how do you set up your git repos to protect the main branch. Do you create a dev branch and associate it with the dev workspace in a deployment pipeline?
Last week Microsoft released an open-source text embedding model called Harrier in three sizes- 270M, 0.6B and 27B. I have been testing it in my RAG pipeline and so far it has crushed all my metrics.
For Power BI Copilot and Data agents with semantic models, you must use Prep Data for AI configuration to ground the responses in the context added in Prep for AI. In this blog, I will show you how yo
At FabCon Atlanta last week, the updated notebook Copilot for data engineering and data science was announced. It brings agentic capabilities to the Copilot and is much more intelligent and Fabric-awa
Using Fabric data agent Python SDK
Use workspace monitoring to monitor MCP server use
I've been living on the edge. I use my personal Fabric trial capacity a lot for testing and learning. As the number of days left in the trial dwindles to single digits, I start praying to the Fabric gods to renew the trial. :D So far they have been renewing my capacity, so thank you for that. I have 80+ workspaces and hundreds, if not thousands, of items so I wanted to automatically create private Github repos for each workspace, sync it with the main branch and commit the items. I chose GitHub because that’s where I keep my personal projects. The process would be the same for Azure DevOps repos.
You will need a PAT to create a connection to the repos. I created a classic token to use it for all repos. Choose whatever is best for your scenario and limit the scope as required.
Once you have the PAT, create a cloud connection in Fabric to generate a connection Id. Choose Github -Source control as the connection type. This is under Settings > Manage Connections in Fabric.
You need to have an Admin of the workspaces you want to sync.
Enable GitHub and ADO in tenant settings.
Install Pygihub and Semantic Link Labs in Fabric Python notebook
Here is the logic, change it as needed:
Get a list of Fabric workspaces
Get a list of workspaces that are already git enabled
Make a list of workspaces that need to be sync’d
For each of the above workspaces:
create and initialize a repo in GitHub with a readme.md. I name the repo as fabric_lower_case_worskpace_name
create a connection to the above repo in Fabric using Semantic Link Labs
wait 60s
get the latest commit hash
get the ids of the items you want to commit. In my case, I only want to sync notebooks. Change this to include other supported items.
commit the items if items exist using Semantic Link Labs
Note that there can be delay in sync depending on the number of items, item types etc, so adjust the time.sleep(n) wait period based on your scenario. You can customize this further and make the logic more granular but for my case this works well. I just need it for backup more than anything else.
# !pip install PyGithub semantic-link-labs --q
from github import Github
import sempy.fabric as fabric
import pandas as pd
import time
import uuid
import sempy_labs as labs
# get github PAT
g = Github("github_pat_xxxxxx")
conn_id = "xxxxxxxxxxxxxx"
def create_and_initialize_github_repo(repo_name):
user = g.get_user()
try:
repo = user.get_repo(repo_name)
print(f"Repository '{repo_name}' already exists.")
except Exception as e:
repo = user.create_repo(
name=repo_name,
description="Repository for workspace " + repo_name,
private=True
)
repo.create_file("README.md", "Initial commit", "# " + repo_name)
print(f"Repository '{repo_name}' created and initialized with README.md.")
return repo
def get_latest_commit_hash(repo_name):
""" Fetch the latest commit hash from GitHub """
owner = g.get_user().login
repo = g.get_repo(f"{owner}/{repo_name}")
commits = repo.get_commits()
return commits[0].sha if commits.totalCount > 0 else None
# only Fabric workspaces
workspaces_df = fabric.list_workspaces().query('Type!="AdminInsights" and `Is On Dedicated Capacity`==True')
git_connections_df = labs.admin.list_git_connections()
# workspaces without git enabled
workspaces_without_git = workspaces_df[~workspaces_df["Id"].isin(git_connections_df["Workspace Id"])]
for index, row in workspaces_without_git.iterrows():
workspace_name = row["Name"]
formatted_repo_name = "fabric_" + workspace_name.replace(" ", "_").lower()
# Create (or reuse) GitHub repository
repo = create_and_initialize_github_repo(formatted_repo_name)
# Resolve workspace details
resolved_workspace_name= fabric.resolve_workspace_name(workspace_name)
workspace_id = fabric.resolve_workspace_id(workspace_name)
# Connect the workspace to GitHub
labs.connect_workspace_to_github(
owner_name=g.get_user().login,
repository_name=formatted_repo_name,
branch_name="main",
directory_name="/",
connection_id=conn_id,
workspace=workspace_id
)
print(f"🟢 The '{resolved_workspace_name}' workspace has been connected to the '{formatted_repo_name}' GitHub repository.")
# Wait until the Git connection is detected
max_retries = 10
retry_interval = 5 # seconds
connection_ready = False
for attempt in range(max_retries):
current_connections = labs.admin.list_git_connections()
if workspace_id in current_connections["Workspace Id"].values:
print(f"Git connection detected for workspace '{resolved_workspace_name}'.")
connection_ready = True
break
else:
print(f"Waiting for Git connection to initialize for '{resolved_workspace_name}' (Attempt {attempt+1}/{max_retries})...")
time.sleep(retry_interval)
if not connection_ready:
print(f"Git connection not initialized for workspace '{resolved_workspace_name}'. Skipping sync.")
continue
time.sleep(60)
remote_commit_hash = labs.initialize_git_connection(workspace=workspace_id)
print(f"🟢 Git connection initialized. Remote commit hash: {remote_commit_hash}")
time.sleep(60) # Adjust timing as needed
# Get item ids and commit only the Notebooks
print("Fetching all item IDs for commit...")
workspace_items = fabric.list_items(workspace=workspace_id).query('Type=="Notebook"')
item_ids = list(workspace_items["Id"])
if not item_ids:
print(f"❌ No items found in workspace '{resolved_workspace_name}'. Skipping commit.")
continue
print(f"🟢 Found {len(item_ids)} items to commit.")
# Commit specified items to Git
try:
labs.commit_to_git(
comment="Sync workspace with main branch",
item_ids=item_ids,
workspace=workspace_id
)
except Exception as e:
print(str(e))
print(f"✅ Workspace '{resolved_workspace_name}' has been successfully committed to Git.")
Result:
Admittedly, getting this to work wasn’t very straightforward. So, if you run into issues, I won’t be surprised. If you do improve the code, I would love to know. Thanks !