Search for a command to run...
Installing and managing Python packages in Microsoft Fabric
Principal Program Manager, Microsoft Fabric CAT helping users and organizations build scalable, insightful, secure solutions. Blogs, opinions are my own and do not represent my employer.
The configuration for packages has moved into a new "Environments" area where, much like the Environments for Runbooks you can configure a number of settings areas and then direct your notebook to use that Environment
Thoughts on the best way to go about managing Langchain, llamaindex and all the massive amount of related dependencies in the Fabric notebook environment? I’m really starting to like the latter with the various flavors and approaches to RAG indexing. Although maybe starting to worry it is becoming a Frankenstein like Langchain already is.
Or best to just pivot to and stick with Semantic Kernel, Cognitive Search, etc with Fabric?
My gut tells me the second…
Last week Microsoft released an open-source text embedding model called Harrier in three sizes- 270M, 0.6B and 27B. I have been testing it in my RAG pipeline and so far it has crushed all my metrics.
For Power BI Copilot and Data agents with semantic models, you must use Prep Data for AI configuration to ground the responses in the context added in Prep for AI. In this blog, I will show you how yo
At FabCon Atlanta last week, the updated notebook Copilot for data engineering and data science was announced. It brings agentic capabilities to the Copilot and is much more intelligent and Fabric-awa
Using Fabric data agent Python SDK
Use workspace monitoring to monitor MCP server use
Updated : Aug 31, 2023
In Microsoft Fabric, you can install Python libraries from PyPI and conda feeds. You can install at the workspace level or the session level in notebooks. Before I show how to install custom Python packages, a quick recap of installing standard libraries.
Any libraries installed at the workspace level will be available for all notebook sessions in that worskspace.
You can manually install libraries one at a time using PyPI or conda channels. You can also upload YAML file with the required libraries to install a number of libraries. If you have an existing workspace with a number of Python libraries installed, you can export the YAML file and add it to the new workspace to replicate the environment.
Currently, multiple environments are not supported so you will have only one virtual environment per workspace. This will change in the future. Also, note that installing custom libraries will slow down the spark cluster start-up time. Only install the libraries you need.
To get a list of default libraries installed on the cluster, use the notebook and run !pip list . Unlike AzureML, the list of default libraries is not published in the documentation yet.
Below is a sample YAML file:
You can install libraries at the session scope in a notebook by running !pip install <library name> or !conda install <library name> . You can also use %pip install which will install on driver and executor nodes of the spark cluster whereas !pip installs only on the driver node. If you have several libraries you want to install using PyPI, you can create a requirements.txt file, save it to the lakehouse and install. Read this article for instructions to create and use requirements.txt.
What if you have custom Python functions and packages? You can build a wheel package for your Python files and install it at the workspace or session level. Let's say you want to use my check_vorder function in all of your notebooks in a workspace. I am going to call my package needles (needles to build Fabric, get it 😜 ). This package has the above function. Below are the steps to build the wheel file.
Create a folder with the below structure:
setup.py has the information regarding the package and the dependencies:
## Sandeep Pawar | fabric.guru ##
## setup.py
from setuptools import setup, find_packages
setup(
name='needles', #needs to build fabric
version='0.0.1',
url='https://www.fabric.guru',
author='Sandeep Pawar',
author_email='pawarbi@outlook.com',
description='Utilities for Microsoft Fabric',
packages=find_packages(),
install_requires=['pyarrow'],
)
init.py can be blank. I like to add a function to check if it has been imported successfully:
## __.init__.py
def function_init():
print('Successfully Imported Init.py')
needles folder has: init and the python file with my function
check_vorder has the below function:
## check_vorder function
def get(table_name_path):
'''
Author: Sandeep Pawar | fabric.guru | Jun 6, 2023
Provide table_name_path as '//lakehouse/default/Tables/<table_name>'
If the Delta table is V-ordered, returns true; otherwise, false.
You must first mount the lakehouse to use the local filesystem API.
'''
import os
if not os.path.exists(table_name_path):
print(f'{os.path.basename(table_name_path)} does not exist')
result = None # Initialize the variable with a default value
else:
import pyarrow.dataset as ds
schema = ds.dataset(table_name_path).schema.metadata
is_vorder = any(b'vorder' in key for key in schema.keys())
if is_vorder:
result = str(schema[b'com.microsoft.parquet.vorder.enabled'])
else:
result = "Table is not V-ordered"
return result
Now you just need to build the wheel by going to the directory with the setup.py file and running the command python setup.py bdist_wheel . Be sure you have wheel installed (!pip install wheel)
This will package your Python files in a wheel file. You will find it in the dist folder in the same directory.
To install it in the workspace, go to Library Management in the workspace settings. Under Custom libraries, select upload and add the newly created whl file. This can take a while.
There is no way to download a whl file uploaded here. Also, there is no central repository where the tenant admin can upload custom packages to make them available to all/selected workspaces. That would be a great feature.
Once successfully uploaded, the package would be available for use in all future sessions:
You can also save the wheel file to a lakehouse just like any other file and install it in-line. Unlike workspace installation, this can be used across workspaces.
Once support for virtual environments becomes available, it will make managing environments and dependencies for multiple projects much easier. Until then, if you need to isolate environments, either create multiple workspaces (not practical ) or install in-line (increases run time).
Update:
While you can use the Files section in the lakehouse to upload the whl and py files, there is now a dedicated location per notebook where you can upload "notebook resources".
It will always stay attached to the notebook and can be referred to in the pipelines too. There is one catch, I am not sure if it's a bug or now. If you use the relative path "/builtin/...", installation will not work. Instead, use
path = mssparkutils.nbResPath + "/builtin/<>package_name>.whl"
%pip install path
You can download the above whl packages from here. Note that the process for py, jar and tar.gz (R) files is the same.