The 27B model is obviously too big for Fabric notebook but the 270M and 0.6B variant despite being small and extremely capable and fast even with CPU inferencing in Fabric Python notebook. For comparison, I have been using the text-embedding-ada-002 model in my pipeline with R1 score = 0.68, the 270M model topped that easily with R1 0.76 with lower latency. Bonus : no Fabric CU consumption (other than Python execution time). In fabric notebook, you can save the model in a lakehouse and load it for inferencing. The initial load, especially 0.6B model may take a while but the retrieval is fast.

Take a look at this sample notebook on how to operationalize it in Fabric.

snippets/harrier-ragbench-techqa-fabricguru.ipynb at main · pawarbi/snippets

• Use %%configure to upgrade the compute and attach a lakehouse

• Download the model

• Instantiate it

• Load the data

• Embed and retrive

• Just for teh sake of testing, I also added a harrier + BM25 pipeline to improve retrieval, the R3 score improved to 99%

Code

Below I call the GetSemanticModelSchema tool from the hosted MCP to get the Prep for AI configurations. Run below in a Fabric notebook.

import httpx
import json

async def get_semantic_model_schema(model_id):
    """Fetch and parse the semantic model prep for ai config from Power BI MCP server.
        Author : Sandeep Pawar | Fabric.guru
    """
    MCP_SERVER_URL = "https://api.fabric.microsoft.com/v1/mcp/powerbi"
    token = notebookutils.credentials.getToken("pbi")
    headers = {
        "Authorization": f"Bearer {token}",
        "Content-Type": "application/json"
    }

    payload = {
        "jsonrpc": "2.0",
        "id": 1,
        "method": "tools/call",
        "params": {
            "name": "GetSemanticModelSchema",
            "arguments": {"artifactId": model_id}
        }
    }

    def parse_sse_response(text):
        for line in text.split('\n'):
            if line.startswith('data: '):
                return json.loads(line[6:])
        return {}


    async with httpx.AsyncClient(timeout=120.0) as client:
        response = await client.post(MCP_SERVER_URL, headers=headers, json=payload)
        data = parse_sse_response(response.text)

    parsed = json.loads(data['result']['content'][0]['text'])

    return {
        "name": parsed['semanticModel']['Name'],
        "tables": parsed['schema']['Tables'],
        "relationships": parsed['schema']['ActiveRelationships'],
        "custom_instructions": parsed['schema']['CustomInstructions'],
        "verified_answers": parsed['schema']['VerifiedAnswers']
    }

# retrieve
prep4ai = await get_semantic_model_schema("<semantic_model_guid>")

Here you have it. Note the verified answer. This is often a confusion, what does the Verified Answer actually store ? As you can see it includes the projections - the tables, columns, measures, filters used in the visual and not the DAX query of the visual !

You can use this to programmatically check, verify, monitor the configurations or make it part of your best practices analyzer for AI.

• Use the Copilot without starting a session. Just open the notebook, Copilot and start asking questions and making changes. Saves you CUs.

• list items in the workspace : list all the lakehouses in this workspace

• take actions : mount the and lakehouses and make the default lakehouse

• get ABFSS paths : give me abfs path of lakehouse <lakehouse_1>.

• create spark pool configurations using %%configure : add configuration to use 4 cores

• refer to content in a cell by cell # : explain cell 11,

Give it a try and you will be surprised how well it works.

But one of my most favorite features is being able to read and refer to other notebooks. For example, I can ask the Copilot to read notebook_1 from the same workspace. Think of the implications for a second. Below is one example, how this can be helpful.

Cross-referencing notebooks

1. In a Fabric workspace I created a notebook with a markdown that includes rules from Palantir PySpark style guide. This style guide is an opinionated guide to PySpark code style for common situations and the associated best practices based on the most frequent recurring topics across the PySpark. Below is a summarized version in a markdown:

PySpark Style Guide

Purpose: This notebook is a style contract for AI-assisted code generation and review. When referenced from another notebook (e.g., refer to @pyspark_style_guide), you MUST apply every rule below to all PySpark code produced or reviewed in that session.

Adapted from Palantir PySpark Style Guide (MIT License).

---

VERSIONS

Use features and API supported by following versions:

• Spark 3.5

• Delta 3.2

• Python 3.11

Enforcement Checklist

When reviewing or generating PySpark code, walk through each check below in order. Flag every violation found. Do not skip checks.

#	Check	What to look for
C1	Imports	Any bare from pyspark.sql.functions import ... or alias other than F, T, W
C2	Column access	Any df.colName dot-access outside of a join on= clause
C3	String column refs	Any F.col('x') that could just be 'x' (Spark 3.0+)
C4	Variable names	Any single-letter dataframe names (df, o, d, t)
C5	Magic values	Any literal string, number, or threshold inline in filter, when, withColumn, select that is not a named constant
C6	Select contract	More than one function per column in a select, or a .when() expression inside a select
C7	withColumnRenamed	Any use. Replace with select + .alias()
C8	Empty columns	Any lit(''), lit('NA'), lit('N/A'). Must be lit(None)
C9	Logical density	More than 3 boolean expressions in a single .filter() or F.when() without named variables
C10	Chain length	More than 5 chained statements in one block
C11	Chain mixing	Joins, filters, withColumn, and selects mixed in the same chain
C12	Join hygiene	Any .join() missing explicit how=
C13	Right joins	Any how='right'. Swap df order, use left
C14	Window frames	Any Window.partitionBy(...).orderBy(...) without explicit .rowsBetween() or .rangeBetween()
C15	Window nulls	F.first() or F.last() without ignorenulls=True
C16	Global windows	Empty W.partitionBy() or window without orderBy used for aggregation. Use .agg() instead
C17	Otherwise fallback	.otherwise(<catch-all value>) masking unexpected data. Use None or omit
C18	Line continuation	Any \ for multiline. Wrap in parentheses instead
C19	UDFs	Any @udf or F.udf(). Rewrite with native functions
C20	Comments	Comments that describe what code does instead of why a decision was made
C21	Dead code	Commented-out code blocks. Remove them
C22	Function size	Functions over ~70 lines or files over ~250 lines

---

Anti-Patterns (find and fix these)

Each pattern below is a regex-like signature. If you see it, it is a violation.

AP1: Bare function imports

# VIOLATION: any of these
from pyspark.sql.functions import col, when, sum, lit
import pyspark.sql.functions as func

# FIX: always
from pyspark.sql import functions as F
from pyspark.sql import types as T
from pyspark.sql import Window as W

AP2: Dot-access column references

# VIOLATION: df.column_name anywhere except join on=
df.select(df.order_id, df.amount)
df.withColumn('x', df.price * df.qty)

# FIX: use string refs
df.select('order_id', 'amount')
df.withColumn('x', F.col('price') * F.col('qty'))

AP3: Inline magic values

# VIOLATION: bare literals in logic
df.filter(F.col('amount') > 500)
F.when(F.col('status') == 'shipped', 'In Transit')
df.filter(F.col('days') < 365)

# FIX: named constants at top of cell/function
HIGH_VALUE_THRESHOLD = 500
STATUS_SHIPPED = 'shipped'
LABEL_IN_TRANSIT = 'In Transit'
ONE_YEAR_DAYS = 365

df.filter(F.col('amount') > HIGH_VALUE_THRESHOLD)
F.when(F.col('status') == STATUS_SHIPPED, LABEL_IN_TRANSIT)
df.filter(F.col('days') < ONE_YEAR_DAYS)

AP4: Complex logic inside .when() or .filter()

# VIOLATION: more than 3 conditions inline
df.filter(
    (F.col('a') == 'x') & (F.col('b') > 10) & (F.col('c') != 'y')
    & ((F.col('d') == 'online') | (F.col('d') == 'partner'))
)

# FIX: named boolean expressions, max 3 in the final filter
is_valid_type = (F.col('a') == TYPE_X)
above_threshold = (F.col('b') > MIN_THRESHOLD)
not_excluded = (F.col('c') != EXCLUDED_STATUS)
is_target_channel = (F.col('d') == CHANNEL_ONLINE) | (F.col('d') == CHANNEL_PARTNER)

flagged = is_valid_type & above_threshold & not_excluded & is_target_channel
df.filter(flagged)

AP5: .when() inside select

# VIOLATION: conditional logic embedded in select
df.select(
    'order_id',
    F.when(F.col('status') == 'shipped', 'In Transit')
     .when(F.col('status') == 'delivered', 'Complete')
     .alias('status_label'),
)

# FIX: select plain columns, then withColumn for derived logic
df = df.select('order_id', 'status')
df = df.withColumn(
    'status_label',
    F.when(F.col('status') == STATUS_SHIPPED, LABEL_IN_TRANSIT)
     .when(F.col('status') == STATUS_DELIVERED, LABEL_COMPLETE)
)

AP6: Empty column sentinels

# VIOLATION
df.withColumn('notes', F.lit(''))
df.withColumn('review_date', F.lit('N/A'))

# FIX
df.withColumn('notes', F.lit(None))
df.withColumn('review_date', F.lit(None))

AP7: Missing window frame

# VIOLATION: implicit frame
w = W.partitionBy('customer_id').orderBy('order_date')

# FIX: always explicit
w = (W.partitionBy('customer_id')
      .orderBy('order_date')
      .rowsBetween(W.unboundedPreceding, 0))

AP8: Blanket .otherwise()

# VIOLATION: masks unexpected values
F.when(..., 'A').when(..., 'B').otherwise('Unknown')

# FIX: omit otherwise (returns null) or use lit(None) explicitly
F.when(..., 'A').when(..., 'B')

AP9: Monster chains

# VIOLATION: mixed concerns, too long
df = (df.select(...).filter(...).withColumn(...).join(...).drop(...).withColumn(...))

# FIX: separate by concern, max 5 per block
df = (
    df
    .select(...)
    .filter(...)
)
df = df.withColumn(...)
df = df.join(..., how='inner')

AP10: Backslash continuation

# VIOLATION
df = df.filter(F.col('a') == 'x') \
       .filter(F.col('b') > 10)

# FIX: parentheses
df = (
    df
    .filter(F.col('a') == 'x')
    .filter(F.col('b') > 10)
)

---

Quick Reference (for code generation)

When writing new code, apply these defaults:

• Imports: F, T, W only

• Columns: string refs where possible, F.col() when needed

• Descriptive df names: orders_df, active_orders, not df, o

• Constants: every literal in logic gets a SCREAMING_SNAKE name

• Selects: plain columns + one transform each, no .when() inside

• Chains: max 5 lines, group by concern (filter/select, then enrich, then join)

• Joins: always how=, always left not right, alias for disambiguation

• Windows: always explicit frame, always ignorenulls=True on first/last

• Empty cols: F.lit(None), never lit('') or lit('NA')

• No UDFs, no .otherwise() fallbacks, no \ continuations

• Comments explain why, not what. No commented-out code.

1. I named the notebook PYSPARK_STYLE_GUIDE. It's all caps intentionally (more on this later).

2. In another notebook, which already has some PySpark code , I opened Copilot.

3. Asked : List all notebooks in this workspace. I can see the PYSPARK_STYLE_GUIDE notebook:

1. My notebook has one cell with large code block (intentional). I prompted Copilot :

refer to @PYSPARK_STYLE_GUIDE and fix the code without losing the function and purpose

https://youtu.be/gtj_f7oBeuk

Copilot read the style notebook and applied the rules to the cells in this notebook. You could also use this to extract code patterns from other notebooks, e.g. how did <notebook_name> ingested the data, use the same library as <notebook_name> to create ML features etc. Super handy.

Your BI/DE/DS team could also create reference pattern notebooks, and refer them for driving consistency and quality. Note that you can list items in another workspace but can't refer cross-workspace.

This was for Copilot in Fabric notebook. In an upcoming blog, I will share how I use Skills for Fabric for development.

Reference:

• PALANTIR_STYLE_GUIDE.md

• Introducing the updated Copilot for data engineering and data science (Preview) | Microsoft Fabric Blog | Microsoft Fabric

• Python (Spark) • PySpark reference • Style guide • Palantir

You can also use the data agent SDK to test the agent programmatically. You can learn more about it here and notebook samples from this repo. Let me show you how you can compare the data agent response from the two stages.

Imagine I am testing new instructions:

• In Draft stage, I used agent instruction: Always return amounts rounded to nearest hundred, e.g. 1451 should be 1500, and 45,179 should be 45100

• For published stage, the instructions are : Always return amounts with $xyz, e.g. $123.4

I should get same answer but formatted differently based on the instructions. Rounded number for draft and precise answer with a $ for production version.

Code

The trick is to set the stage ai_skill_stage= as “sandbox” vs “production”

%pip install fabric-data-agent-sdk --q

import time
from fabric.dataagent.client import FabricOpenAI

DATA_AGENT_NAME = "<DataAgentName>"
MODEL = "gpt-4o"

sbx  = FabricOpenAI(artifact_name=DATA_AGENT_NAME, ai_skill_stage="sandbox")
prod = FabricOpenAI(artifact_name=DATA_AGENT_NAME, ai_skill_stage="production")

asst_sbx  = sbx.beta.assistants.create(model=MODEL, instructions="You are the DRAFT (sandbox) data agent.").id
asst_prod = prod.beta.assistants.create(model=MODEL, instructions="You are the PUBLISHED (production) data agent.").id


def ask(client, assistant_id, q, *, timeout_s=300):
    tid = client.beta.threads.create().id
    client.beta.threads.messages.create(thread_id=tid, role="user", content=q)
    run = client.beta.threads.runs.create(thread_id=tid, assistant_id=assistant_id)

    end = time.time() + timeout_s
    while run.status not in {"completed", "failed", "cancelled", "expired", "incomplete"}:
        if time.time() > end:
            raise TimeoutError(f"timeout (status={run.status})")
        time.sleep(2)
        run = client.beta.threads.runs.retrieve(thread_id=tid, run_id=run.id)

    if run.status != "completed":
        raise RuntimeError(f"run status={run.status}")

    for m in client.beta.threads.messages.list(thread_id=tid, order="desc").data:
        if m.role == "assistant":
            return m.content[0].text.value
    return ""


def compare(q):
    return ask(sbx, asst_sbx, q), ask(prod, asst_prod, q)


q = "what's the total transaction amount"
draft, production = compare(q)

print("DRAFT:", draft)
print("\nPRODUCTION:", production)

Result

This is handy if you want to tune the data agent performance and compare it vs production before publishing.

Power BI Modeling MCP server was launched at Ignite 2025 last month. It has quickly become an indispensable tool for working with the semantic models in Fabric workspaces. You can learn more about it below:

• Announcement by Rui Romano : https://powerbi.microsoft.com/en-us/blog/power-bi-november-2025-feature-summary/#post-31766-_Toc214035083

• Documentation : Power BI MCP server documentation - Power BI | Microsoft Learn

• Blog by Jeffrey Wang : Talk to Your Data Model: Introducing the Power BI Modeling MCP – pbidax

I was recently asked how admins can monitor usage of the Modeling MCP Server. Similar to any other external tool, if a user has permission to query a semantic model’s XMLA endpoint, you cannot restrict which client or tool they use. However, admins can monitor its usage.

1. Enable Workspace Monitoring : Turn on Workspace Monitoring, you will need to be a Workspace Admin to enable this. I have a blog on how this can be helpful.

2. Query the logs : Query the SemanticModelLogs table and identify the semantic models that are being queried and which users. Filter the ApplicationName column to find the application used for querying the model.

// Monitor Power BI Modeling Server Usage
SemanticModelLogs
| where ItemKind == 'Dataset' and ApplicationName == 'MCP-PBIModeling'
| summarize sessions = dcount(OperationId) by ItemName, ExecutingUser, ApplicationName, bin(Timestamp, 5m)

Blog: Extracting Matching Words Using A Pre-Defined List in Power Query/M | Sandeep Pawar

Following the same example I shared in the blog above, I wanted to extract certain keywords (programming languages) from text. In M, I split the text and did list lookup to find the matching words.

Fabric AI Prompt:

This shouldn’t need much explanation. Similar to AI Functions in Fabric, you specify the column, enter the prompt and let the LLM do the work.

Here is the prompt I used : Extract any programming, query, or scripting languages mentioned (including those used in BI tools such as M, LOD ). Comma-separate if multiple. NA if none or unsure

Result:

Give it a try:

let
    Source = #table(
        type table [Respondent = text, Text = text],
        {
            {"Person1", "I like Python"},
            {"Person2", "We use Python and R"},
            {"Person3", "SQL"},
            {"Person4", "My team uses Spark , SQL"},
            {"Person5", "Excel all the way"},
            {"Person6", "I hate DAX"},
            {"Person7", "M is magic"},
            {"Person8", "I don't use anything"},
            {"Person8", "I learned MDX before DAX and still use MDX for our SSAS cubes "}
        }
    ),
  #"Renamed columns" = Table.RenameColumns(Source, {{"Text", "quote"}}),
  #"Added AI prompt column" = Table.AddColumn(#"Renamed columns", "language", each FabricAI.Prompt("Extract any programming, query, or scripting languages mentioned (including those used in BI tools such as M, LOD ). Comma-separate if multiple. NA if none or unsure", Record.SelectFields(_, {"quote"})), type text)
in
    #"Added AI prompt column"

Instead of the UI, you can also use FabricAI.Prompt M function. Note that this function is not available in Power BI Desktop.

Read the documentation for more details: Fabric AI Prompt in Dataflow Gen2 (Preview) - Microsoft Fabric | Microsoft Learn

You could use this to scan your reports and detect errors before your users find them.

Steps:

• Use semantic link labs to generate pdf of the report

• Convert pdf to image and encode it as base64

• Send the base64 to LLM and ask it to detect any errors

• Optional : if you have sensitive data on the report, you could optionally add noise to the image or apply filter (e.g. gaussian blur etc.) and then encode it. I will leave that to you but can cover if you drop it in comments below.

Example Report :

In my test, I had a report with four pages. 3 pages had error like below and the last page had no errors.

Page1 : Has error

Page 4 : No error

Generate PDF:

Attach a lakehouse to the notebook and install semantic-link-labs and pymupdf4llm

Below code will save the specified report as a PDF to the Files section in a lakehouse

%pip install semantic-link-labs pymupdf4llm

from sempy_labs.report import export_report

export_report(
    report="Sales (1)", #name of your report
    export_format="PDF",
    file_name="mySalesreport",   # name of the pdf to save
    workspace="d70c1ed7-b6e4-40f1-911f-3300a13145ff",       # workspace name or id where report lives
    lakehouse="mylakehouse",       # lakehouse to save file into
    lakehouse_workspace="d70c1ed7-b6e4-40f1-911f-3300a13145ff"  # workspace for the lakehouse
)

Detect Error:

Using GPT 4.1 to detect errors on each report page

from synapse.ml.fabric.service_discovery import get_fabric_env_config
from synapse.ml.fabric.token_utils import TokenUtils
import fitz 
import base64
from io import BytesIO
from PIL import Image
import requests
import json

# Setup Fabric API
fabric_env_config = get_fabric_env_config().fabric_env_config
auth_header = TokenUtils().get_openai_auth_header()
openai_base_host = fabric_env_config.ml_workload_endpoint + "cognitive/openai/openai/"
deployment_name = "gpt-4.1"
service_url = openai_base_host + f"deployments/{deployment_name}/chat/completions?api-version=2025-04-01-preview"

auth_headers = {
    "Authorization": auth_header,
    "Content-Type": "application/json"
}
prompt= """


Analyze the image of a dashboard and determine if it contains any errors, error messages, or failed data loading. 
Look for phrases like 'Error fetching data', 'See details', error icons (X in circles), or visual indicators of failure. 
Respond with 'ERROR DETECTED' if errors are present, or 'NO ERROR' if the page appears normal. Then provide a brief explanation.

"""
def pdf_page_to_base64(pdf_path, page_num, dpi=150):
    doc = fitz.open(pdf_path)
    page = doc[page_num]
    pix = page.get_pixmap(dpi=dpi)
    img = Image.frombytes("RGB", [pix.width, pix.height], pix.samples)

    buffer = BytesIO()
    img.save(buffer, format="JPEG", quality=85)
    doc.close()

    return base64.b64encode(buffer.getvalue()).decode('utf-8')

def check_page_for_errors(base64_image):
    payload = {
        "messages": [
            {
                "role": "system",
                "content": "You are an error detection assistant analyzing Power BI dashboard and reports."
            },
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": prompt
                    },
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": f"data:image/jpeg;base64,{base64_image}",
                            "detail": "high"
                        }
                    }
                ]
            }
        ],
        "max_tokens": 500,
        "temperature": 0.1
    }

    response = requests.post(service_url, headers=auth_headers, json=payload)

    if response.status_code == 200:
        return response.json()["choices"][0]["message"]["content"]
    else:
        return f"API Error: {response.status_code}"

def analyze_pdf_for_errors(pdf_path):
    doc = fitz.open(pdf_path)
    total_pages = len(doc)
    doc.close()

    results = {}

    for page_num in range(total_pages):
        print(f"Processing page {page_num + 1}/{total_pages}...")

        base64_image = pdf_page_to_base64(pdf_path, page_num)
        result = check_page_for_errors(base64_image)

        results[page_num + 1] = result

        if "ERROR DETECTED" in result:
            print(f"  -> ERROR FOUND on page {page_num + 1}")
        else:
            print(f"  -> Page {page_num + 1} OK")

    return results


pdf_path = "/lakehouse/default/Files/mySalesreport.pdf"
error_results = analyze_pdf_for_errors(pdf_path)

error_results

Below is the JSON returned by error_results

{1: 'ERROR DETECTED\n\nExplanation: The dashboard contains an error message at the top center stating "Error fetching data for this visual See details" along with an error icon (X in a circle). This indicates that at least one visual has failed to load data. The rest of the visuals appear to be functioning normally.',
 2: 'ERROR DETECTED\n\nExplanation: The dashboard displays the message "Error fetching data for this visual" along with an error icon and a "See details" link, indicating that data failed to load for the visual.',
 3: 'ERROR DETECTED\n\nExplanation: The dashboard contains a visual with an error message stating "Error fetching data for this visual See details" along with an error icon (X in a circle), indicating that one of the visuals failed to load data.',
 4: 'NO ERROR\n\nExplanation: The dashboard displays all visuals and data without any error messages, icons, or indicators of failed data loading. All charts and figures are rendered correctly.'}

For my report, LLM correctly identified the pages with errors and described the errors as well.

I did not test all types of errors so if you wanted to make it more comprehensive, you could give few-shot example of each error and what to look for.

Defining UDF

The latest SLL version 0.12.4, thanks to my colleague Michael Kovalsky, has set_user_defined_function using TOM. Here is how you use it:

#%pip install semantic-link-labs --q #upgrade to 0.12.4
import sempy_labs as labs
from sempy_labs.tom import connect_semantic_model

dataset = '' # Enter the name or ID of your semantic model
workspace = None # Enter the name or ID of the workspace in which the semantic model resides
name = 'AddTax' # Name of the user-defined function
expression = "(amount : NUMERIC) => amount * 1.1" # Expression logic of the user-defined function

with connect_semantic_model(dataset=dataset, readonly=False, workspace=workspace) as tom:
    tom.set_user_defined_function(name=name, expression=expression)

# List user-defined functions
df = labs.list_user_defined_functions(dataset=dataset, workspace=workspace)
display(df)

Using DAX Lib

SQLBI has an open-source collection of UDFs at DAX Lib, submitted by users and community members. You can use TMDL to apply any of these UDFs in Desktop or using TE3. We can use SLL to use any of those functions from DAX Lib’s repo. Below I will use the Mode function my friend Kerry submitted.

In the repo, the functions are defined in TMDL so thanks to LLM below function extracts the name and the function using regex.

import re
from typing import Tuple, Optional

## thanks to LLM for below
def extract_dax_function(text: str) -> Tuple[Optional[str], Optional[str]]:
    """
    Returns (function_name, expression_block).
    - function_name: between 'function' and '='
    - expression_block: after '=' up to (but not including) the first 'annotation'
    """
    udf_name = re.search(r"function\s+['\"]?\s*([^'\"=\s]+)\s*['\"]?\s*=", text, re.IGNORECASE)
    if not udf_name:
        return None, None
    func_name = udf_name.group(1).strip()
    udf_expr = re.search(r"=\s*(.*?)\bannotation\b", text, re.IGNORECASE | re.DOTALL)
    if udf_expr:
        expr = udf_expr.group(1).rstrip()
    else:
        m_after_eq = re.search(r"=\s*(.*)$", text, re.DOTALL)
        expr = m_after_eq.group(1).rstrip() if m_after_eq else None

    return func_name, expr


import requests
mode_udf = (requests.get("https://raw.githubusercontent.com/daxlib/daxlib/refs/heads/main/packages/k/kolosko.summarystats/0.1.0/lib/functions.tmdl").text)
print(extract_dax_function(mode_udf)[1])

Next we take that and apply it to a published semantic model.

name = extract_dax_function(mode_udf)[0]
expression = extract_dax_function(mode_udf)[1]
dataset = "Sales_udf_sandeep" #dataset name/id
workspace=None #if notebook is in the same workspace as the dataset, else workspace name/id

with connect_semantic_model(dataset=dataset, readonly=False, workspace=workspace) as tom:
    tom.set_user_defined_function(name=name, expression=expression)

udf_df = labs.list_user_defined_functions(dataset=dataset)
display(udf_df)

Let’s check if it actually works:

dax = """

EVALUATE
{ Kolosko.SummaryStats.MODE(Sales, Sales[Net Price]) }

"""
fabric.evaluate_dax(dax_string=dax, dataset=dataset)

Centralizing UDFs

We can extend this further by automating the process. If you define one (or more) semantic model(s) with all the UDFs defined, SLL can read and apply to a list of models.

In the function below:

• It checks for compatibility level first

• If it’s <1702, it’s upgraded to 1702

• Waits for 5 seconds (for the update to go through)

• Applies the UDFs from the source semantic model to the target semantic model(s)

import time

def set_udf_with_compatibility_check(tom, name, expression, max_retries=1):
    """
    Set UDF with automatic compatibility level upgrade if needed
    """
    for attempt in range(max_retries + 1):
        try:
            tom.set_user_defined_function(name=name, expression=expression)
            print(f"Successfully set UDF: {name}")
            return True

        except ValueError as e:
            if "compatibility level of at least 1702" in str(e) and attempt == 0:
                print(f"Compatibility level error for {name}. Upgrading to 1702...")
                tom.set_compatibility_level(1702)
                time.sleep(5)  # Wait 5 seconds as requested
                continue  # Retry
            else:
                print(f"Failed to set UDF {name}: {e}")
                return False

        except Exception as e:
            print(f"Unexpected error setting UDF {name}: {e}")
            return False

    return False


for i, row in udf_df.iterrows():
    name = row['Function Name']
    expression = row['Expression']
    dataset = 'order_analysis'
    workspace = None

    with connect_semantic_model(dataset=dataset, readonly=False, workspace=workspace) as tom:
        set_udf_with_compatibility_check(tom, name, expression)

This could be a powerful pattern. You could use several approaches to centralize the UDFs:

• Define a set of golden/centralized semantic models in a workspace with UDFs defined.

• Bring these model(s) into other model as composite models

• Or use notebooks to continuously update and apply the UDFs to downstream models

Below is another approach:

Using notebooks for cataloging

I wrote a blog a while ago on getting the daylight savings adjusted US time. I converted that to a UDF. The use case here is to make this available to all the users so based on their defined timezone, they can get the daylight adjusted US time.

    ( tz: STRING ) =>
    VAR _utcnow   = UTCNOW()
    VAR _todayUTC = TRUNC(_utcnow)
    VAR _y        = YEAR(_todayUTC)

    /* US DST window: 2nd Sun Mar  1st Sun Nov (UTC-date granularity) */
    VAR _mar1     = DATE(_y, 3, 1)
    VAR _nov1     = DATE(_y,11, 1)
    VAR _secondSunMar = (_mar1 + MOD(8 - WEEKDAY(_mar1), 7)) + 7
    VAR _firstSunNov  =  _nov1 + MOD(8 - WEEKDAY(_nov1), 7)
    VAR _isDST = _todayUTC >= _secondSunMar && _todayUTC < _firstSunNov

    /* offsets (hours to ADD to UTC) */
    VAR _std =
        SWITCH(TRUE(),
            tz="useastern",  -5,
            tz="uscentral",  -6,
            tz="usmountain", -7,
            tz="usarizona",  -7,
            tz="uspacific",  -8,
            tz="usalaska",   -9,
            tz="ushawaii",  -10,
            -8  // default
        )
    VAR _dst =
        SWITCH(TRUE(),
            tz="useastern",  -4,
            tz="uscentral",  -5,
            tz="usmountain", -6,
            tz="usarizona",  -7,   /* no DST */
            tz="uspacific",  -7,
            tz="usalaska",   -8,
            tz="ushawaii",  -10,   /* no DST */
            -7  // default
        )
    VAR _usesDST = NOT (tz IN {"usarizona","ushawaii"})
    VAR _offsetHours = IF(_usesDST && _isDST, _dst, _std)

    RETURN
        _utcnow + (_offsetHours / 24.0)

RETURN
_utcnow + (_offsetHours / 24.0)

You can use Fabric notebooks for documentation, embed that into an OrgApp for self-service users to see the list of various UDFs available. Users can copy and paste the UDFs they need. In the catalog you can add all the relevant details for the users to get details and use the functions. If you want to make it very fancy, you can also include working examples, thanks to Semantic Link Labs !

These are just some ideas. DAX UDFs are super powerful and automating and centralizing them will make them even more useful & powerful.

There three steps for you to generate the SAS :

• Enable it in the workspace settings:

  

• Generate the User Delegation Key (UDK) : Read more about it here.

• Build the SAS URL : This can be the tricky part because you have to parse and create XML. This documentation provides all the details.

Once you have the SAS URL, you just make a get request like any other API to access the data.

Below I share how to do this using service principal for demonstration purposes. For user facing client application, use the appropriate authentication method to generate the token. Also note that you should use Azure Key Vault for security, for my demo I am hard coding them to keep it simple. Read the best practices before implementing.

Data

I have a finance.csv file in a Fabric Lakehouse that I want read using Onelake SAS.

Code

I created a service principal and gave it a granular access to the folder.

%pip install azure-identity --q

import base64
import hmac
import hashlib
import datetime as dt
import requests
from urllib.parse import quote
from xml.etree import ElementTree as ET
from azure.identity import ClientSecretCredential
from email.utils import formatdate
import io
import pandas as pd
import logging


TENANT_ID     = ""
CLIENT_ID     = ""
CLIENT_SECRET = ""
REGION        = "centralus" #IMPORTANT : use the regional endpoint, i.e. capacity region
WORKSPACE_ID  = "" 
ITEM_ID       = "" #lakehouse id
PATH          = "Files/raw_data/finance.csv" # relative path

cred = ClientSecretCredential(TENANT_ID, CLIENT_ID, CLIENT_SECRET)
STORAGE_BEARER = cred.get_token("https://storage.azure.com/.default").token
FABRIC_BEARER  = cred.get_token("https://api.fabric.microsoft.com/.default").token

def get_udk():

    token_obj = cred.get_token("https://storage.azure.com/.default")
    token_expiry = dt.datetime.fromtimestamp(token_obj.expires_on, dt.timezone.utc)

    now = dt.datetime.now(dt.timezone.utc)
    udk_expiry = now + dt.timedelta(minutes=55)

    if token_expiry <= udk_expiry:
        raise ValueError(f"OAuth token expires before UDK. Token: {token_expiry}, UDK: {udk_expiry}")


    url = f"https://{REGION}-onelake.blob.fabric.microsoft.com/?restype=service&comp=userdelegationkey"
    st  = (now - dt.timedelta(minutes=2)).strftime("%Y-%m-%dT%H:%M:%SZ")
    se  = (now + dt.timedelta(minutes=55)).strftime("%Y-%m-%dT%H:%M:%SZ")
    xml = f'<?xml version="1.0"?><KeyInfo><Start>{st}</Start><Expiry>{se}</Expiry></KeyInfo>'
    hdr = {"Authorization": f"Bearer {STORAGE_BEARER}", "x-ms-version":"2022-11-02",
           "x-ms-date": formatdate(usegmt=True), "Content-Type":"application/xml"}
    r = requests.post(url, data=xml, headers=hdr, timeout=20); r.raise_for_status()
    x = ET.fromstring(r.text)

    logging.info(f"UDK generated successfully, expires at {se}")
    return {t: x.findtext(t) for t in ["SignedOid","SignedTid","SignedStart","SignedExpiry","SignedService","SignedVersion","Value"]}

def build_file_sas_guid(udk: dict, workspace_id: str, item_id: str, path: str, perms="r"):

    sv  = udk["SignedVersion"]
    sr  = "b"
    spr = "https"
    key = base64.b64decode(udk["Value"])

    canonical = f"/blob/onelake/{workspace_id}/{item_id}/{path}"

    now = dt.datetime.now(dt.timezone.utc)

    signed_start = dt.datetime.strptime(udk["SignedStart"], "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=dt.timezone.utc)
    signed_expiry = dt.datetime.strptime(udk["SignedExpiry"], "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=dt.timezone.utc)

    st = max(now - dt.timedelta(minutes=2), signed_start)
    se = min(now + dt.timedelta(minutes=50), signed_expiry)

    st_s = st.replace(tzinfo=None).strftime("%Y-%m-%dT%H:%M:%SZ")
    se_s = se.replace(tzinfo=None).strftime("%Y-%m-%dT%H:%M:%SZ")

    parts = [
        perms, st_s, se_s, canonical,
        udk["SignedOid"], udk["SignedTid"], udk["SignedStart"], udk["SignedExpiry"],
        udk["SignedService"], udk["SignedVersion"],
        "", "", "",
        "",
        spr,
        sv,
        sr,
        "",
        "",
        "", "", "", "", ""
    ]
    sig = base64.b64encode(hmac.new(key, "\n".join(parts).encode(), hashlib.sha256).digest()).decode()

    enc = lambda v: quote(v, safe="")
    qs = (
        f"sp={perms}&st={enc(st_s)}&se={enc(se_s)}"
        f"&skoid={enc(udk['SignedOid'])}&sktid={enc(udk['SignedTid'])}"
        f"&skt={enc(udk['SignedStart'])}&ske={enc(udk['SignedExpiry'])}"
        f"&sks={udk['SignedService']}&skv={udk['SignedVersion']}"
        f"&sv={sv}&sr={sr}&spr={spr}&sig={enc(sig)}"
    )

    path_url = quote(path, safe="/")
    sas_url = f"https://onelake.blob.fabric.microsoft.com/{workspace_id}/{item_id}/{path_url}?{qs}"

    logging.info(f"SAS URL generated successfully, expires at {se_s}")
    return sas_url


try:
    udk = get_udk()
    sas_url = build_file_sas_guid(udk, WORKSPACE_ID, ITEM_ID, PATH, perms="r")
    print("SAS URL:", sas_url)

    r = requests.get(sas_url, timeout=30)
    print("GET:", r.status_code)

    if r.status_code == 200:
        r.raise_for_status()
        df = pd.read_csv(io.BytesIO(r.content))
        df.head(2)
    else:
        print(f"Error Status code: {r.status_code}")
        print(f"Response: {r.text}")

except Exception as e:
    print(f"Error : {e}")
    import traceback
    traceback.print_exc()

Example:

I was able to generate the SAS and read the CSV from Google Colab notebook :

Writing Files

For writing back to the Lakehouse, follow the same steps to generate the SAS and use the put method:

import requests
sas_url = build_file_sas_guid(udk, WORKSPACE_ID, ITEM_ID, "Files/raw_data/my_new_file.csv", perms="cw")
data_to_write = "column1,column2,column3\nvalue1,value2,value3"

headers = {
    'x-ms-blob-type': 'BlockBlob',
    'Content-Type': 'text/csv'
}

response = requests.put(sas_url, data=data_to_write, headers=headers, timeout=30)
response.raise_for_status()
print("Upload status:", response.status_code)
## 201 if successful

Onelake SAS has a maximum life of one hour.

References:

• OneLake shared access signatures (SAS) now available in public preview | Microsoft Fabric Blog | Microsoft Fabric

• What is a OneLake shared access signature (SAS) - Microsoft Fabric | Microsoft Learn

• Get User Delegation Key (REST API) - Azure Storage | Microsoft Learn

• It identifies which parquet files are new, which have been modified, and which have been removed

• For unchanged data, it maintains the existing data in memory (preserving dictionaries and other optimizations)

• It only reloads the data from new or modified parquet files

• It removes from memory any data from deleted parquet files

You can read more about it in the documentation.

I will instead highlight another update based on the work by two of my other colleagues, Phil Seamark and Michael Kovalsky. Semantic Link Lab’s .delta_analyzer_history() function estimates the incremental framing effect based on the updates to the delta table. 0 means no benefit at all and 100% means highly effective. Note that this is based on the changes to the delta table and does not account for any refresh/updates made to the semantic model.

import sempy_labs as labs
labs.delta_analyzer_history("sales").tail(2) #sales is the name of the table in the lakehouse attached to the notebook

References:

• microsoft/semantic-link-labs: Early access to new features for Microsoft Fabric's Semantic Link.

• Direct Lake overview - Microsoft Fabric | Microsoft Learn

• Performance testing Power BI Direct Lake models revisited: ensuring worst-case performance

• Chris Webb's BI Blog: Performance Testing Power BI Direct Lake Mode Datasets In Fabric

I highly encourage you to read his blog and subscribe to his newsletter "Humans of the Data Sphere.”

Ref: Big data quality framework: a holistic approach to continuous quality management | Journal of Big Data | Full Text

In this multi-part series, I will try to explore how AI/LLMs can possibly be used along with the rule-based DQ checks. In this first blog, I will focus on the first pass analysis, getting it set up using AI Functions in Microsoft Fabric and in the following blogs, I will operationalize it to create a robust evaluation framework.

Fabric AI Functions

I have written about AI Functions before. It allows you to use AI seamlessly in your data engineering applications with single line of code. In this introductory blog, I will show how we can use AI Functions for checking contextual DQ issues.

Data

The Consumer Financial Protection Bureau (CFPB) has comprehensive data on consumer complaints about financial products and services. It includes details like the product, sub-product, issue, complaint description, and more. When consumers log complaints, they select the product, sub-product, and issue categories. On the website, trends are shown for each product, sub-product, and issue based on the categories chosen by consumers. However, the website does not provide definitions for these categories, making it easy for consumers to assign their complaints to the wrong category. So, even though the data might be complete, the category assigned may not match the issue described which will also lead to incorrect trend analysis.

I downloaded the last 6 months data from the website (you can use API as well) with complaint narrative (~490K records).

To analyze the contextual DQ issues, we will:

• Use LLM to review the Product, Sub-product, complaint and analyze if the complaint matches the product & sub-product assigned by the user

• Return true or false based on the analysis

• Return the suggested category

• Return issue type (product mismatch, sub product mismatch or both)

• Brief explanation

• Flag for human evaluation if the LLM is unsure

As I mentioned above, this will be a first pass analysis and in the following blogs I will refine with evaluation harness to improve the process.

Prompt

The prompt follows the similar template I have used in previous blogs - instructions, examples, guidelines with tags and return the JSON in defined schema.

<INSTRUCTIONS>
You are a data quality expert analyzing CFPB consumer complaints for product categorization accuracy.
Your task is to determine if the complaint narrative matches the assigned product/sub-product categories.
</INSTRUCTIONS>

<VALID_PRODUCTS_SUBPRODUCTS>
Credit reporting or other personal consumer reports:
- Credit reporting
- Other personal consumer report

Debt collection:
- I do not know
- Credit card debt
- Other debt
- Telecommunications debt
- Rental debt
- Medical debt
- Auto debt
- Payday loan debt
- Federal student loan debt
- Private student loan debt
- Mortgage debt

Checking or savings account:
- Checking account
- Other banking product or service
- Savings account
- CD (Certificate of Deposit)

Credit card:
- General-purpose credit card or charge card
- Store credit card

Money transfer, virtual currency, or money service:
- Domestic (US) money transfer
- Mobile or digital wallet
- Virtual currency
- International money transfer
- Money order, traveler's check or cashier's check
- Check cashing service
- Foreign currency exchange

Student loan:
- Federal student loan servicing
- Private student loan

Mortgage:
- Conventional home mortgage
- FHA mortgage
- VA mortgage
- Home equity loan or line of credit (HELOC)
- Other type of mortgage
- USDA mortgage
- Manufactured home loan
- Reverse mortgage

Vehicle loan or lease:
- Loan
- Lease

Payday loan, title loan, personal loan, or advance loan:
- Installment loan
- Payday loan
- Personal line of credit
- Title loan
- Other advances of future income
- Earned wage access
- Pawn loan
- Tax refund anticipation loan or check

Prepaid card:
- General-purpose prepaid card
- Government benefit card
- Gift card
- Payroll card
- Student prepaid card

Debt or credit management:
- Debt settlement
- Credit repair services
- Mortgage modification or foreclosure avoidance
- Student loan debt relief
</VALID_PRODUCTS_SUBPRODUCTS>

<EXPECTED_OUTPUT>
{
"is_correctly_categorized": true/false,
"issue_type": "correct|product_mismatch|subproduct_mismatch|both_incorrect",
"explanation": "brief explanation",
"suggested_product": "correct product if wrong",
"suggested_subproduct": "correct sub-product if wrong",
"should_review": true/false
}
</EXPECTED_OUTPUT>

<REVIEW_CRITERIA>
Set should_review to true if:
- The complaint narrative is ambiguous or could fit multiple categories
- The technical/financial terminology is unclear or inconsistent
- Multiple financial products are mentioned making categorization difficult
- The complaint lacks sufficient detail to make a confident assessment
- You are uncertain about the correct categorization
</REVIEW_CRITERIA>

Analyze the complaint and return ONLY valid JSON.

Code

In my case, I used the Python notebook. However, if you use the Pyspark notebook, as announced recently, you do not need to install any libraries. AI Functions is now part of the Fabric RT1.3.

%pip install -q --force-reinstall openai==1.30 2>/dev/null

%pip install -q --force-reinstall https://mmlspark.blob.core.windows.net/pip/1.0.12-spark3.5/synapseml_core-1.0.12.dev1-py2.py3-none-any.whl 2>/dev/null

%pip install -q --force-reinstall https://mmlspark.blob.core.windows.net/pip/1.0.12.2-spark3.5/synapseml_internal-1.0.12.2.dev1-py2.py3-none-any.whl 2>/dev/null

import synapse.ml.aifunc as aifunc
import pandas as pd
import openai
from synapse.ml.aifunc import Conf

cols = ['Date received','Product','Sub-product','Issue','Sub-issue','Consumer complaint narrative', 'Complaint ID']
df = pd.read_csv("/lakehouse/default/Files/dq_aifunction/complaints-2025-08-01_12_34.csv", usecols =cols)
df['Date received'] = pd.to_datetime(df['Date received'],errors='coerce')
df = df.sort_values(by = ['Date received'], ascending=False)

Steps:

• Pre process the data

• Analyze the categories using AI functions

• Parse JSON

• Prepare output df

import pandas as pd
import json
import re

def clean_narrative_text(text):
 if pd.isna(text):
     return ""
 cleaned = re.sub(r'XXXX+', 'XXXX', str(text))
 cleaned = re.sub(r'\s+', ' ', cleaned).strip()
 return cleaned

def check_qlty_prmpt():
 prompt = """
<INSTRUCTIONS>
You are a data quality expert analyzing CFPB consumer complaints for product categorization accuracy.
Your task is to determine if the complaint narrative matches the assigned product/sub-product categories.
</INSTRUCTIONS>

<VALID_PRODUCTS_SUBPRODUCTS>
Credit reporting or other personal consumer reports:
- Credit reporting
- Other personal consumer report

Debt collection:
- I do not know
- Credit card debt
- Other debt
- Telecommunications debt
- Rental debt
- Medical debt
- Auto debt
- Payday loan debt
- Federal student loan debt
- Private student loan debt
- Mortgage debt

Checking or savings account:
- Checking account
- Other banking product or service
- Savings account
- CD (Certificate of Deposit)

Credit card:
- General-purpose credit card or charge card
- Store credit card

Money transfer, virtual currency, or money service:
- Domestic (US) money transfer
- Mobile or digital wallet
- Virtual currency
- International money transfer
- Money order, traveler's check or cashier's check
- Check cashing service
- Foreign currency exchange

Student loan:
- Federal student loan servicing
- Private student loan

Mortgage:
- Conventional home mortgage
- FHA mortgage
- VA mortgage
- Home equity loan or line of credit (HELOC)
- Other type of mortgage
- USDA mortgage
- Manufactured home loan
- Reverse mortgage

Vehicle loan or lease:
- Loan
- Lease

Payday loan, title loan, personal loan, or advance loan:
- Installment loan
- Payday loan
- Personal line of credit
- Title loan
- Other advances of future income
- Earned wage access
- Pawn loan
- Tax refund anticipation loan or check

Prepaid card:
- General-purpose prepaid card
- Government benefit card
- Gift card
- Payroll card
- Student prepaid card

Debt or credit management:
- Debt settlement
- Credit repair services
- Mortgage modification or foreclosure avoidance
- Student loan debt relief
</VALID_PRODUCTS_SUBPRODUCTS>

<EXPECTED_OUTPUT>
{
"is_correctly_categorized": true/false,
"issue_type": "correct|product_mismatch|subproduct_mismatch|both_incorrect",
"explanation": "brief explanation",
"suggested_product": "correct product if wrong",
"suggested_subproduct": "correct sub-product if wrong",
"should_review": true/false
}
</EXPECTED_OUTPUT>

<REVIEW_CRITERIA>
Set should_review to true if:
- The complaint narrative is ambiguous or could fit multiple categories
- The technical/financial terminology is unclear or inconsistent
- Multiple financial products are mentioned making categorization difficult
- The complaint lacks sufficient detail to make a confident assessment
- You are uncertain about the correct categorization
</REVIEW_CRITERIA>

Analyze the complaint and return ONLY valid JSON.
"""
 return prompt

def analyze_category(df, sample_size=None):
 if sample_size:
     sample_df = df.sample(min(sample_size, len(df)), random_state=42).copy()
 else:
     sample_df = df.copy()

 sample_df['cleaned_narrative'] = sample_df['Consumer complaint narrative'].apply(clean_narrative_text)
 sample_df = sample_df[sample_df['cleaned_narrative'].str.len() > 10].copy()

 sample_df['analysis_input'] = sample_df.apply(lambda row: 
     f"Current Product: {row['Product']}\n"
     f"Current Sub-product: {row['Sub-product']}\n" 
     f"Issue: {row['Issue']}\n"
     f"Complaint: {row['cleaned_narrative'][:800]}..."
 , axis=1)

 prompt = check_qlty_prmpt()
 sample_df['llm_analysis'] = sample_df[['analysis_input']].ai.generate_response(prompt, , conf=Conf(seed=0, max_concurrency=25))

 return sample_df

def parse_json(df):
 def parse_json_response(response_text):
     try:
         json_match = re.search(r'\{.*\}', response_text, re.DOTALL)
         if json_match:
             json_str = json_match.group(0)
             return json.loads(json_str)
         return None
     except:
         return None

 df['parsed_analysis'] = df['llm_analysis'].apply(parse_json_response)
 df['analysis_valid'] = df['parsed_analysis'].notna()

 valid_df = df[df['analysis_valid']].copy()

 if len(valid_df) > 0:
     valid_df['is_correctly_categorized'] = valid_df['parsed_analysis'].apply(
         lambda x: x.get('is_correctly_categorized', None) if x else None)
     valid_df['issue_type'] = valid_df['parsed_analysis'].apply(
         lambda x: x.get('issue_type', None) if x else None)
     valid_df['explanation'] = valid_df['parsed_analysis'].apply(
         lambda x: x.get('explanation', None) if x else None)
     valid_df['suggested_product'] = valid_df['parsed_analysis'].apply(
         lambda x: x.get('suggested_product', None) if x else None)
     valid_df['suggested_subproduct'] = valid_df['parsed_analysis'].apply(
         lambda x: x.get('suggested_subproduct', None) if x else None)
     valid_df['should_review'] = valid_df['parsed_analysis'].apply(
         lambda x: x.get('should_review', None) if x else None)

 return valid_df

def check_quality(df, sample_size=None):
 if sample_size:
     print(f"Analyzing {sample_size} complaints for categorization:")
 else:
     print(f"Analyzing all {len(df)} complaints for categorization:")

 analyzed_df = analyze_category(df, sample_size)
 results_df = parse_json(analyzed_df)

 output_cols = [
     'Date received', 'Product', 'Sub-product', 'Issue', 
     'Consumer complaint narrative', 'is_correctly_categorized',
     'issue_type', 'explanation', 'suggested_product', 'suggested_subproduct',
     'should_review'
 ]

 available_cols = [col for col in output_cols if col in results_df.columns]
 final_df = results_df[available_cols].copy()

 print(f"{len(final_df)} complaints analyzed.")
 print(f"Incorrectly categorized: {(~final_df['is_correctly_categorized']).sum()}")
 print(f"Flagged for review: {final_df['should_review'].sum()}")

 return final_df

## first 250 complaints for demo purposes
results_df = check_quality(df.head(250)) 
display(results_df)

Output:

Out of the 250 sample complaints analyzed, 113 were labeled as incorrectly categorized and 64 were flagged for further human evaluation. Thus we can see that we can “potentially” use AI Functions for catching data quality issues. I said potentially because more evaluation needs to be done (future blog) to use this effectively and reduce errors.

Let’s look at the very first example:

Complaint:

I am filing this new complaint because the debt collector, First Financial XXXX XXXX, and the creditor, XXXX XXXX ( XXXX XXXX, XXXX ), have provided me with debt verification documents that are factually inaccurate. This is about my previous, closed complaint ( Complaint ID # [ XXXX ] ). The 'Vehicle Valuation Report ' sent to me as verification of the debt is based on an incorrect vehicle mileage of XXXX miles. This is false. The official rental agreement ( # XXXX ) confirms that the vehicle 's actual mileage at the time of rental was XXXX miles. This discrepancy of over XXXX miles has artificially and significantly inflated the vehicle 's valuation, which is the entire basis for the debt they claim I owe. By knowingly or negligently providing a valuation based on false information and continuing to demand payment based on it, they are making false representations in an attempt to collect a debt. This is a serious issue that goes beyond a simple dispute over methods. I have sent a formal letter disputing this invalid verification and demand that this illegitimate debt be waived.

The user submitted this issue under Debt Collection → Other Debt whereas AI Functions categorized as Debt Collection → Auto Debt. This complaint is clearly about debt related a vehicle.

Some complaints were flagged for further review, e.g.

I canceled a Google Play subscription on XX/XX/XXXX through my XXXX device, but I was still charged on XX/XX/XXXX. I did not receive the service and have not received a refund or the service restored. I attempted to resolve the issue through Google Play support, but my request was denied or unresolved. They sent back several emails saying my card info had to be updated but it has been the same info since I uploaded the card which covers XX/XX/XXXX. Further, they said that restoring the subscription would have to come from the app developer, XXXX XXXX. XXXX XXXX rebutted this and said XXXX handles subscriptions. What I want : I want a full refund for the charge and for my cancellation to be properly, and promptly, honored.

This was categorized by the user as Money transfer, virtual currency, money service → digital wallet. However, there complaints is about unauthorized subscription, credit card billing rather than money transfer or bitcoin etc. One could argue that Google play uses Google wallet and hence digital wallet is the right option. So not all categorizations are straightforward and LLM can identify complaints that should be reviewed further and thus improving the DQ.

This was one example, but this approach can also be used for tabular data to catch inconsistencies and errors. One of the very first Power BI report I ever built 10 years ago was actually for catching errors Oracle ERP system. We saw more than 50% of the codes assigned were incorrect leading to delayed product shipments, incorrect processing of orders, warranty claims etc. I think using AI Functions for such tasks is a very viable solution.

Challenges

As always, we need to remember that, like rule-based checks, LLMs cannot be completely predictable. We can manage their behavior to some degree and measure their accuracy, but it's important to always verify and validate by creating baselines and evaluation tools that you can improve over time. I will write about in future blogs.

References

Transform and enrich data seamlessly with AI functions - Microsoft Fabric | Microsoft Learn

Submit a complaint | Consumer Financial Protection Bureau

About Me — Jack Vanlightly

Unstructured to Structured : Extracting Data From Messy Excel Sheets Using Fabric AI Function

Big data quality framework: a holistic approach to continuous quality management | Journal of Big Data | Full Text

But until then, if you want to check whether any reports use the default semantic models, here are two approaches using Semantic Link Labs.

If any reports use default semantic model

Below we get a list of all the reports in a workspace and is_default column shows if the report uses a default semantic model. The key function here is labs.is_default_semantic_model

#%pip install semantic-link-labs -q

import sempy_labs as labs
import sempy.fabric as fabric

#only the reports in the workspace where the notebook is hosted will be returned
#specify the workspace parameter if you want to search other workspaces & modify accordingly
df = fabric.list_reports().assign(
    is_default=lambda df: df.apply(
        lambda row: labs.is_default_semantic_model(
            fabric.resolve_dataset_name(row['Dataset Id'])
        ),
        axis=1
    )
)

df

If any default semantic models are used in reports

The other way, you may want to identify if there are any default semantic models that are used by a report.

has_reports column shows if the default model has any reports attached to it. To get the list of reports, use labs.list_reports_using_semantic_model function.

#%pip install semantic-link-labs -q

import sempy_labs as labs
import sempy.fabric as fabric
## only the models in teh workspace where the notebook is hosted will be scanned. Otherwise, specify workspace param  
df = fabric.list_datasets()
df = df[df['Dataset Name'].apply(labs.is_default_semantic_model)]
df['has_reports'] = df['Dataset Name'].apply(
    lambda name: len(labs.list_reports_using_semantic_model(dataset=name))>0
)
df = df[df['has_reports']]

df

To scan all the workspaces, either use list_items function or loop over the workspaces you are interested in.

More on this later !

References

Power BI Semantic Models - Microsoft Fabric | Microsoft Learn

Sunsetting Default Semantic Models – Microsoft Fabric | Microsoft Fabric Blog | Microsoft Fabric

Below is a crude approach I came up with to map the logical column names to physical column names.

The logic:

• Get the logical column names and physical column names to make a dictionary

• Get the parquet files from the delta transaction log

• Apply column mapping

• Read and union

As you can see from above, you lose the parallelization and efficiency in the process.

Code

#Python notebook
import polars as pl
from deltalake import DeltaTable
import os

def scan_delta_cm(path: str) -> pl.LazyFrame:
   delta_table = DeltaTable(path)

   colmaps: dict[str, str] = dict()
   for field in delta_table.schema().fields:
       logical_name = field.name
       physical_name = field.metadata.get("delta.columnMapping.physicalName", field.name)
       colmaps[logical_name] = physical_name

   all_lazy_frames = []
   for add_action in delta_table.get_add_actions(flatten=True).to_pylist():
       file_path = os.path.join(delta_table.table_uri, add_action["path"])
       lazy_df = pl.scan_parquet(file_path)

       file_schema = lazy_df.collect_schema()
       available_columns = file_schema.names()

       select_exprs = []
       for logical_name, physical_name in colmaps.items():
           if physical_name in available_columns:
               select_exprs.append(pl.col(physical_name).alias(logical_name))

       if select_exprs:
           lazy_df = lazy_df.select(select_exprs)
           all_lazy_frames.append(lazy_df)

   if all_lazy_frames:
       return pl.concat(all_lazy_frames)
   else:
       raise ValueError("No data files found")

#path is abfs path of the table
df = scan_delta_cm(path).collect()
# df.head()

Think of this more of an experiment than a solution which will work for limited cases (tables with deletion vectors wont work either as expected). For any business critical job, I would advise using Spark in such scenarios.

The other easier alternative is to use Duckdb which supports tables with columnMapping.

If you must use Polars, you can zero-copy this duckdb to polars df.

Organizational Theme

I have three report themes published in the tenant.

Semantic Link Labs

Using Semantic Link Labs, I want to apply one of the themes (the Sales Dark Theme for illustration purposes) to be applied to all reports in a workspace. To show that this works for reports in Pro workspaces as well, I published three reports to a Pro workspace. Using Semantic Link Labs, I will:

• Install SLL version >= 0.11.0

• Get the JSON of the Sales Dark Theme from the org themes

• Get a list of all the reports in the workspace

• If the report theme is different from the org theme, apply the theme to each report

  • else, skip

The base code is very simple - get org theme, apply the theme:

•           theme_json = theme.get_org_theme_json(theme='MyTheme')
            with connect_report(report=report, workspace=workspace,readonly=False) as rpt:
                rpt.set_theme(theme_json=theme_json)
  

  Below, I make it a bit more robust with error handling, catch a few edge cases and do some comparison etc:

%pip install semantic-link-labs --q

from sempy_labs.report import connect_report
from sempy_labs.theme import list_org_themes, get_org_theme_json
import sempy.fabric as fabric
import json

def theme_content(theme_dict):
    # remove the 'name' field for content comparison
    return {k: v for k, v in theme_dict.items() if k != "name"}

def apply_org_theme(workspace, org_theme_name=None):
    """
    For each report in the workspace, applies the org theme if not already set or if only the name is different.

    """
    # get org themes and select the desired one
    org_themes = list_org_themes()
    if org_themes.empty:
        raise ValueError("org themes nt found.")
    if org_theme_name:
        row = org_themes[org_themes["Theme Name"] == org_theme_name]
        if row.empty:
            raise ValueError(f"Org theme '{org_theme_name}' not found.")
        theme_id = row["Theme Id"].iloc[0]
    else:
        theme_id = org_themes["Theme Id"].iloc[0]
        org_theme_name = org_themes["Theme Name"].iloc[0]
    org_theme_json = get_org_theme_json(theme_id)
    if isinstance(org_theme_json, str):
        org_theme_json = json.loads(org_theme_json) #if theme returned is not json, just in case

    org_theme_content = theme_content(org_theme_json)
    org_theme_name_val = org_theme_json.get("name", org_theme_name)

    # all reports in the workspace; can be Fabric or Pro workspace
    reports = fabric.list_reports(workspace=workspace)
    updated = []

    for ix, row in reports.iterrows():
        report_id = row["Id"]
        report_name = row["Name"]
        try:
            with connect_report(report=report_id, workspace=workspace, readonly=False, show_diffs=False) as rpt:
                # get custom theme fallback to base theme.
                try:
                    # custom theme
                    report_theme = rpt.get_theme("customTheme")
                except Exception:
                    try:
                        report_theme = rpt.get_theme("baseTheme")
                    except Exception:
                        report_theme = {}

                if isinstance(report_theme, str):
                    report_theme = json.loads(report_theme)

                report_theme_content = theme_content(report_theme)
                report_theme_name_val = report_theme.get("name", "")

                # compare different scenarios
                # sometime the theme details could be same but name could be different
                content_match = json.dumps(report_theme_content, sort_keys=True) == json.dumps(org_theme_content, sort_keys=True)
                name_match = report_theme_name_val == org_theme_name_val

                if content_match and name_match:
                    print(f"Report '{report_name}' already has the '{org_theme_name}' org theme.")
                    continue
                elif content_match and not name_match:
                    print(f"Report theme details are the same but the theme name is different. Updated the report with the new theme '{org_theme_name}'.")
                    rpt.set_theme(theme_json=org_theme_json)
                    updated.append(report_name)
                else:
                    print(f"Applying org theme '{org_theme_name}' to '{report_name}'...")
                    rpt.set_theme(theme_json=org_theme_json)
                    updated.append(report_name)
        except Exception as err:
            print(f"failed to update '{report_name}': {err}")
    return updated

Example:

There is no API, yet, to update the Org theme but I am sure it will be made available in the future.

Note again that I executed the notebook in a Fabric workspace but the reports are in a Pro workspace ! Michael Kovalsky and other contributors have been adding many new functions to the ReportWrapper class, you should take a look at it for all the possibilities.

Recipe

• Export the Power BI report as an image to a lakehouse. I have written about this before.

• Extract all the pages of the report as png

• Use an LLM to detect broken visuals

• Save the pages with errors

Using LLM

There are several ways to do this:

• Use multimodal embedding to convert the report image to an embedding vector and do similarity search (e.g. I used Jina 2 Clip embedding and did a similarity search with this report has an error message with gray background . It worked decently. It wasn’t very robust. Plus, can’t identify the visual with the error and was prone to errors.)

• Multimodal AI Search : Similar to above but instead of text, perform similarity search using embedding of error messages. This also wasn’t too accurate.

• Use multimodal model: Use a multimodal LLM to query the image. This worked well using several different LLMs however gemini-2.5 worked the best without any additional complexities (and it’s free for testing :D). Below I show how to do that.

Using Google Gemini

Gemini >1.5 models are multimodal, i.e. they work with txt, audio, images etc. Unlike many similar models, it’s also a very capable object detection model. You can ask it to return bounding boxes for the objects you are interested in. Below, I use that to identify the error messages. I tested it on several reports, and it worked 100% of times (on the reports I tested). You will need to generate the Gemini API key (free).

Extract Report Pages

You will need to enable the tenant setting to allow exporting reports as images. Attach a lakehouse to Fabric notebook and specify the report, lakehouse details.

%pip install semantic-link-labs google-genai --q
from sempy_labs.report import export_report

(export_report(
    report = "Sales and Returns-error",
    workspace="Sales",
    export_format = "PNG",
    lakehouse="MyLakehouse",
    lakehouse_workspace="Sales")
)

Detect Visuals with Errors

The prompt is simple:

import json
import re
import os
from google import genai
from PIL import Image, ImageDraw

import json
import re
import os
import zipfile
import glob
from google import genai
from PIL import Image, ImageDraw

def detect_error_messages(zip_path, api_key, output_dir=None):
    if output_dir is None:
        output_dir = os.path.join(os.path.dirname(zip_path), "extracted")

    os.makedirs(output_dir, exist_ok=True)

    with zipfile.ZipFile(zip_path, 'r') as zip_ref:
        zip_ref.extractall(output_dir)


    png_files = glob.glob(os.path.join(output_dir, "*.png"))

    client = genai.Client(api_key=api_key)
    results = []

    for image_path in png_files:
        try:
            image = Image.open(image_path)

            response = client.models.generate_content(
                model='gemini-2.5-pro-preview-06-05',
                contents=[
                    'Analyze this image and return a JSON response with: 1) "has_error_messages": true/false if image contains error messages, 2) "coordinates": array of bounding boxes [ymin, xmin, ymax, xmax] for each error message found. Error messages typically have a black x in a circle with a gray box.',
                    image
                ]
            )

            result = _extract_json_result(response.text)
            result["image_path"] = image_path
            result["filename"] = os.path.basename(image_path)

            if result["has_error_messages"] and result["coordinates"]:
                base_name = os.path.splitext(image_path)[0]
                ext = os.path.splitext(image_path)[1]
                output_path = f"{base_name}-error{ext}"

                _draw_bounding_boxes(image_path, result["coordinates"], output_path)
                result["output_path"] = output_path

            results.append(result)

        except Exception as e:
            results.append({
                "image_path": image_path,
                "filename": os.path.basename(image_path),
                "error": str(e),
                "has_error_messages": False,
                "coordinates": []
            })

    return results

def _extract_json_result(text):
    json_pattern = r'\{[^{}]*"has_error_messages"[^{}]*\}'
    match = re.search(json_pattern, text, re.DOTALL)

    if match:
        try:
            return json.loads(match.group())
        except json.JSONDecodeError:
            pass

    coord_pattern = r'\[\s*\d+\s*,\s*\d+\s*,\s*\d+\s*,\s*\d+\s*\]'
    coordinates = [json.loads(match) for match in re.findall(coord_pattern, text)]

    return {
        "has_error_messages": len(coordinates) > 0,
        "coordinates": coordinates
    }

def _draw_bounding_boxes(image_path, coordinates, output_path):
    image = Image.open(image_path)
    draw = ImageDraw.Draw(image)
    width, height = image.size

    for i, (ymin, xmin, ymax, xmax) in enumerate(coordinates):
        x1 = int(xmin * width / 1000)
        y1 = int(ymin * height / 1000)
        x2 = int(xmax * width / 1000)
        y2 = int(ymax * height / 1000)

        dash_length = 10

        for x in range(x1, x2, dash_length * 2):
            draw.line([(x, y1), (min(x + dash_length, x2), y1)], fill='red', width=5)
            draw.line([(x, y2), (min(x + dash_length, x2), y2)], fill='red', width=5)

        for y in range(y1, y2, dash_length * 2):
            draw.line([(x1, y), (x1, min(y + dash_length, y2))], fill='red', width=5)
            draw.line([(x2, y), (x2, min(y + dash_length, y2))], fill='red', width=5)

    image.save(output_path)



api_key = "AIzaSyAxxxxxxxxxxxxxxxxxxxxxxxx"
image_path = "/lakehouse/default/Files/Sales and Returns-error.png"

result = detect_error_messages(image_path, api_key)

Above function returns a json with keys "has_error_messages" and ”coordinates” . If a visual with error is detected: has_error_messages is true and the coordinates of the visual with error are returned. The function draws a red dotted rectangle around the visuals with errors. I used Claude to refine the function.

Test

1. You can see that this report has many pages. Some pages have error like below, some don’t.

2. The function saved the report pages as a PNG and extracted the report pages:

   

   3. In the extracted folder, any pages with errors were saved with the bounding boxes

   4. 

      Here is what the output looks like:

      

      

      

      You can automate this process and send email alerts if errors are detected - all using semantic link labs.

      I do not expect anyone to use the given the pre-requisites and risks - but I just wanted to test it for my own sake. It works !

Using Semantic Link Labs if you have a pbir report, John Kerski’s solution and Kurt’s solution are more practical.

A few months ago I wrote a blog on getting workspace activities using Semantic Link Labs. I have been using it on one of my workspaces in personal tenant to generate activity data. I retrieve the logs for each day and save it to a lakehouse. The activity event logs have Client IP address and I wanted to try above function on this field. So I shortcutted the delta table to a KQL table in an Eventhouse and it worked like a charm. Super helpful for auditing & monitoring the workspace activities.

## GET ACTIVITIES FOR THE LAST 7 DAYS AND SAVE IT TO A LAKEHOUSE USING POLARS
## USING PYTHON NOTEBOOK BELOW

from datetime import datetime, timedelta
import pandas as pd
import sempy_labs as labs
import polars as pl

#last 7 days
N=7

activities = []
for n in range(N):
    day = datetime.now() - timedelta(days=n)
    start_of_day = day.replace(hour=0, minute=0, second=0, microsecond=0).strftime('%Y-%m-%dT%H:%M:%S')
    end_of_day = day.replace(hour=23, minute=59, second=59, microsecond=999999).strftime('%Y-%m-%dT%H:%M:%S')

    df = labs.admin.list_activity_events(
        start_time=start_of_day, 
        end_time=end_of_day
    ) 
    activities.append(df)

final_df = pd.concat(activities)
pl_df = pl.from_pandas(final_df)
## change to your abfss
pl_df.write_delta("abfss://Sales@onelake.dfs.fabric.microsoft.com/MyLakehouse.Lakehouse/Tables/dbo/ws_activities", mode="overwrite")

In the Eventhouse, I added the above table as a shortcut and queried to aggregate by location using geo_info_from_ip_address:

//table name "ws_activities"

external_table("ws_activities")
| extend Location = geo_info_from_ip_address(['Client IP'])
| extend LocationString = strcat_delim(", ", 
    tostring(Location.city), 
    tostring(Location.state), 
    tostring(Location.country))
| extend Latitude = todouble(Location.latitude)
| extend Longitude = todouble(Location.longitude)
| where isnotempty(LocationString) and LocationString != ", , "
| where isnotempty(Latitude) and isnotempty(Longitude)
| summarize EventCount = count() by LocationString, Latitude, Longitude

I live in Portland, OR and recently travelled to Seattle, WA so this checks out:

Rendered the result as a map which shows Portland and Seattle.

The same map is available in Real Time Dashboard so you can track the location of activities in real time and generate alerts if needed!

To learn more about Fabric CLI, refer to:

• Fabric command line interface - Microsoft Fabric REST APIs | Microsoft Learn

• Microsoft Fabric CLI | fabric-cli

• Microsoft Fabric CLI: Turbo-charge Ops with Retro Command-Line Power

• What's Coming in Fabric Automation and CI/CD | BRK205

Installation:

In Fabric python notebook, first install fabric cli :

%pip install ms-fabric-cli --q

Set up Auth Token:

This will use the identity of the user executing the notebook. You can also use service principal authentication as shown here.

token = notebookutils.credentials.getToken('pbi')
os.environ['FAB_TOKEN'] = token
os.environ['FAB_TOKEN_ONELAKE'] = token

Commands:

There are two types of commands - command line and interactive. In the notebook, you should be able to run all commands supported in command line mode. The documentation does a great job of describing if a command can be used in command line, interactive or both.

There are two ways you can execute the supported commands - using magic or using a Python wrapper for the commands.

Using cell magic

Once installed and auth setup, use !fab magic to run the shell command. Below I get a list of all the workspaces along with the capacity attached and capacity id.

To get a list of all items in a workspace:

This works even if you have spaces in item/workspace names:

You can use JMESPath in the query or you can also use shell commands like below to filter the lines that contain word “Trial” and print the first column (i.e. workspace names)

If a command is not available, you can call an API inline as well:

To download items to an attached default lakehouse:

Multi-line

!fab allows only one line command. To create more complex multi line commands, you can use %%sh cell magic like below:

%%sh
echo "=== Exploring Reid Workspace ==="

# jump to workspace
fab -c "cd Reid.Workspace" || echo "Failed to navigate"

echo "Listing all items:"
fab -c "ls Reid.Workspace -l" || echo "No items or access denied"

echo ""
echo "Notebooks only:"
fab -c "ls Reid.Workspace" | grep "\.Notebook" || echo "No notebooks found"

echo ""
echo "Reports only:"
fab -c "ls Reid.Workspace" | grep "\.Report" || echo "No reports found"

echo "Exploration completed"

You can also pass Python variables to commands, making it very dynamic:

Using Python

Above method is great for interactive exploration. For automation, you can create Python functions and use it like any other Python function. For example, I downloaded one notebook but for more complex patterns/automation, I can use subprocess.run() to execute the commands.

To download all notebooks from a workspace to a lakehouse location:

def download_item(workspace, item_name, local_path="/lakehouse/default/Files/tmp", force=True):
    """
    Download a Fabric item to local dir
    """
    try:

        os.makedirs(local_path, exist_ok=True)
        cmd = f"export {workspace}.Workspace/{item_name} -o {local_path}"
        if force:
            cmd += " -f"
        result = subprocess.run(["fab", "-c", cmd], capture_output=True, text=True)

        if result.returncode == 0:
            print(f"Downloaded {item_name} to {local_path}")
            return True
        else:
            print(f"Failed: {result.stderr}")
            return False

    except Exception as e:
        print(f"Error: {e}")
        return False

For loop:

def download_all_notebooks(workspace, local_path="/lakehouse/default/Files/tmp"):
    """Download all notebooks from workspace"""
    try:

        result = subprocess.run(["fab", "-c", f"ls {workspace}.Workspace"], capture_output=True, text=True)
        notebooks = [line.strip() for line in result.stdout.split('\n') if '.Notebook' in line]

        success_count = 0
        for notebook in notebooks:
            if download_item(workspace, notebook, local_path):
                success_count += 1

        print(f"Downloaded {success_count}/{len(notebooks)} notebooks")
        return success_count

    except Exception as e:
        print(f"Error downloading: {e}")
        return 0

You can generalize this to create re-usable functions, e.g. below lists all workspaces:

import subprocess
import json
from typing import Optional, Dict, Any

def exec_fabcli(command: str, capture_output: bool = False, silently_continue: bool = False) -> Optional[str]:
    """
    Run a Fabric CLI command from within a notebook.

    Args:
        command: The fab command to run (without 'fab' prefix)
        capture_output: Whether to capture and return the output
        silently_continue: Whether to suppress exceptions on errors

    Returns:
        Command output if capture_output=True, otherwise None
    """
    try:
        result = subprocess.run(["fab", "-c", command], capture_output=True, text=True, timeout=60)

        if not silently_continue and result.returncode != 0:
            error_msg = f"Command failed with exit code {result.returncode}\n"
            error_msg += f"STDOUT: {result.stdout}\n"
            error_msg += f"STDERR: {result.stderr}"
            raise Exception(error_msg)

        if capture_output:
            return result.stdout.strip()
        else:
            # output
            if result.stdout:
                print(result.stdout)
            if result.stderr:
                print(f"Warning: {result.stderr}")

    except subprocess.TimeoutExpired:
        raise Exception("Command timed out after 60 seconds")
    except FileNotFoundError:
        raise Exception("Fabric CLI not found. Make sure 'fab-cli' is installed")

def list_workspaces(detailed: bool = False, show_hidden: bool = False) -> str:
    """List all workspaces"""
    flags = ""
    if detailed:
        flags += " -l"
    if show_hidden:
        flags += " -a"
    return exec_fabcli(f"ls{flags}", capture_output=True)

print(list_workspaces(detailed=True))

Semantic Link/Labs vs Fabric CLI

Semantic Link and Semantic Link Labs also provide similar capabilities but if you are already using CLI, this is a convenient way to reuse what you already know. This will especially be helpful for automating workflows. Different tools, different use cases. Fabric CLI can be usedf or CI/CD with Github and ADO automation. Jacob Knightley shared this excellent comparison at FabCon :

Here are additional resource to learn more:

• RuiRomano/fabric-cli-powerbi-cicd-sample

• Fabric CLI to deploy FUAM : fabric-toolbox/monitoring/fabric-unified-admin-monitoring/scripts/Deploy_FUAM.ipynb at main · microsoft/fabric-toolbox

• Demos by Aitor Murguzur: murggu/fab-demos

• Multi-tenant scenario: alisonpezzott/pbi-ci-cd-isv-multi-tenant: CI/CD scenario Multi Tenant for Microsoft Power BI PRO projects by utilizing fabric-cli and GitHub Actions

• ecotte/Fabric-Monitoring-RTI by Edgar Cotte

Experiment : fabgpt

I can’t complete the blog without mentioning AI, can I ? :D I love how Fabric CLI allows you to query, automate so easily. I was wondering what if all of that could be achieved with natural language? In the below example, I used Open AI + notebook cell magic to wrap above Python functions in a cell magic which I am calling fabgpt :D It generates above Python code and executes it.

copy_obejct():

The method is easy to use:

#%pip install semantic_link_labs --q
import sempy.labs as labs
with labs.tom.connect_semantic_model(dataset="SourceDataset", workspace="SourceWorkspace") as tom:
    # to copy Sales table from source dataset to target dataset
    table = tom.model.Tables["Sales"]
    tom.copy_object(
        object=table,
        target_dataset="TargetDataset",
        target_workspace="TargetWorkspace" 
    )

Bulk Copy Objects:

For demo purposes, I created two models model_1 and model_2 . model_1 has relationships, calculated columns, calculated tables and measures. model_2 has the two import tables without any relationships and calc tables/calumns and measures. I want to copy from model_1 to model_2. I will define the functions:

from sempy_labs.tom import connect_semantic_model
import sempy.fabric as fabric

#if workspaces are None, current workspace is used
def copy_relationships(
    source_dataset: str,
    target_dataset: str,
    source_workspace: str = None,
    target_workspace: str = None
):
    """
    Copy all relationships from the source semantic model to the target semantic model.
    """
    with connect_semantic_model(dataset=source_dataset, workspace=source_workspace) as src_tom:
        for rel in src_tom.model.Relationships:
            print(f"Copying relationship: {rel.Name}")
            src_tom.copy_object(
                object=rel,
                target_dataset=target_dataset,
                target_workspace=target_workspace
            )
    print("All relationships have been copied.")


def copy_calculated_columns(
    source_dataset: str,
    target_dataset: str,
    source_workspace: str = None,
    target_workspace: str = None
):
    """
    Copy all calculated columns from the source semantic model to the target semantic model.
    Uses sempy.fabric.list_columns to reliably identify calculated columns.
    """
    calc_cols_df = fabric.list_columns(source_dataset, workspace=source_workspace)[['Table Name', 'Column Name', 'Type']]
    calculated_columns = set(
        tuple(x) for x in calc_cols_df.query('Type == "Calculated"')[['Table Name', 'Column Name']].values
    )

    with connect_semantic_model(dataset=source_dataset, workspace=source_workspace) as src_tom:
        for table in src_tom.model.Tables:
            for column in table.Columns:
                if (table.Name, column.Name) in calculated_columns:
                    print(f"Copying calculated column: {column.Name} from table: {table.Name}")
                    src_tom.copy_object(
                        object=column,
                        target_dataset=target_dataset,
                        target_workspace=target_workspace
                    )
    print("All calculated columns have been copied.")

def copy_calculated_tables(
    source_dataset: str,
    target_dataset: str,
    source_workspace: str = None,
    target_workspace: str = None
):
    """
    Copy all calculated tables from the source semantic model to the target semantic model.

    """

    calculated_tables = set(
        fabric.list_tables(source_dataset, workspace=source_workspace)
              .query('Type == "Calculated Table"')['Name']
    )

    with connect_semantic_model(dataset=source_dataset, workspace=source_workspace) as src_tom:
        for table in src_tom.model.Tables:
            if table.Name in calculated_tables:
                print(f"Copying calculated table: {table.Name}")
                src_tom.copy_object(
                    object=table,
                    target_dataset=target_dataset,
                    target_workspace=target_workspace
                )
    print("All calculated tables have been copied.")

from sempy_labs.tom import connect_semantic_model

def copy_all_measures(
    source_dataset: str,
    target_dataset: str,
    source_workspace: str = None,
    target_workspace: str = None
):
    """
    Copy all measures from every table in the source semantic model to the target semantic model.

    """
    with connect_semantic_model(dataset=source_dataset, workspace=source_workspace) as src_tom:
        for table in src_tom.model.Tables:
            for measure in table.Measures:
                print(f"Copying measure '{measure.Name}' from table '{table.Name}'...")
                try:
                    src_tom.copy_object(
                        object=measure,
                        target_dataset=target_dataset,
                        target_workspace=target_workspace)
                except:
                    print(f"Error with {measure.Name}, check again")
                    continue

    print("All measures have been copied.")

copy_relationships(
source_dataset="model_1",
target_dataset="model_2",
source_workspace="a79cbb27-3cc-d64bf25ca405",
target_workspace="a79cbb27-3bf64bf25ca405"
)

copy_calculated_tables(
source_dataset="model_1",
target_dataset="model_2",
source_workspace="a79cbb27-3cc-d64bf25ca405",
target_workspace="a79cbb27-3bf64bf25ca405"
)

copy_calculated_columns(
source_dataset="model_1",
target_dataset="model_2",
source_workspace="a79cbb27-3cc-d64bf25ca405",
target_workspace="a79cbb27-3bf64bf25ca405"
)

copy_all_measures(
source_dataset="model_1",
target_dataset="model_2",
source_workspace="a79cbb27-3cc-d64bf25ca405",
target_workspace="a79cbb27-3bf64bf25ca405"
)

Before:

After

When you copy tables, make sure to update the connection and refresh the target semantic model to reflect the changes and ensure all objects are copied correctly. When you copy objects, all its properties are also copied (e.g. formatting, annotations etc.) You can also copy any TOM object, such as partitions, hierarchies, perspectives, and more.

You need XMLA Write enabled in the tenant and capacity settings. You also need Contributor+ access (or build permissions) to both the semantic models. This works in any Premium/Fabric workspace.

Download notebook from here.

References:

• sempy_labs.tom package — semantic-link-labs 0.9.11 documentation

• TabularEditor/TabularEditor: This is the code repository and issue tracker for Tabular Editor 2.X (free, open-source version). This repository is being maintained by Daniel Otykier.

• Advanced C# Scripts | Tabular Editor Documentation

I presented this use case at FabCon Vegas.

The Badly Formatted Excel

I came across this post on the Power BI forum. The user wanted to import the Excel sheet below into Power BI and create a model for reporting. Sounds simple, right? If you know Power Query and some M, you can easily turn this into a table for all Excel files on a schedule—as long as all the files have the same structure, the same column names, and the same columns. If there is any difference, you'll either get an error or incorrect results. Plus, imagine if below purchase orders were in different languages from different suppliers!

This is exactly what we will try to fix using Fabric AI Functions.

Another user shared the M code to clean the data but also noted that this will work if the form doesn’t change.

Synthetic Data

I don't have the data from this user, so I created an Excel file with four variations of the purchase order data mentioned above. I randomly placed data in different cells, added extra columns and rows, changed the layout, and used different column names to create messy layout. Although the data is in a table format, the formatting is unstructured and almost impossible to process using Power Query or Python.

Sheet 1:

Sheet 2:

Compared to sheet 1, in Sheet 2, following changes were made:

• Split the PO header details in two columns instead of in one

• Excluded JOB ID from PO header and instead added it as a table header in a merged cell (typically merged cells are hard to deal with)

• PO date is in a slightly different format (mm-dd-yyyy instead of mm dd yyyy)

Sheet 3

• Randomly changed the header layout. Notice that the positions are different, and instead of having the column:value pairs in two columns next to each other, they are now one below the other.

• Added an extract NUMBER column

• Added a merged cell table header for PO details

• Changed column names

• PO date is in different format

• Missing phone number

  

Sheet 4

• Added some text at the top of the table irrelevant to the data

• Changed the column order and column header

• Changed date format

As you can see - all the PO forms are different and although they have similar data, the layouts are totally different. There is no way Power Query or any Python code with regex pattern matching can extract data in a structured form from this Excel.

You can download the Excel from here : snippets/complete_purchase_order.xlsx at main · pawarbi/snippets

Goals

As mentioned above, the main goal is to extract the data, but we also we want to :

• evaluate the accuracy of extraction to ensure AI Function can be used reliably.

• perform the extraction efficiently with minimal CU consumption

• make the process reproducible to ensure we can trace the results and performance in the future

• identify risks, errors and solution space

Fabric AI Function

Fabric AI functions use LLM (Azure Open AI deployment) for text summarization, extraction, classification etc with convenient Python functions. It can be used with any F SKU in a Fabric PySpark notebook. We will start with an easy/baseline solution first and progressively work towards the goals mentioned above to achieve the final production-ready solution.

To get started, install AI functions in a Fabric PySpark notebook with runtime 1.3

## Fabric PySpark notebook with runtimr 1.3
%pip install -q tiktoken deepdiff tabulate openai==1.30 > /dev/null 2>&1
%pip install -q --force-reinstall httpx==0.27.0 > /dev/null 2>&1
%pip install -q --force-reinstall https://mmlspark.blob.core.windows.net/pip/1.0.9/synapseml_core-1.0.9-py2.py3-none-any.whl > /dev/null 2>&1
%pip install -q --force-reinstall https://mmlspark.blob.core.windows.net/pip/1.0.10.0-spark3.4-5-a5d50c90-SNAPSHOT/synapseml_internal-1.0.10.0.dev1-py2.py3-none-any.whl > /dev/null 2>&1

#optional - I will explain later
!wget -O /synfs/nb_resource/builtin/json_diff_extraction_accuracy.py https://raw.githubusercontent.com/pawarbi/snippets/refs/heads/main/json_diff_extraction_accuracy.py

Since we want to extract the data, we can use the .extract method in AI functions to specify the data we want to extract. Note that AI functions work on text data so we need to convert the excel to some text format. In my previous blogs, I have shared how LLM love markdown format. So we will convert the dataframe to markdown before passing it to AI functions.

import os
import re
import json
import time
from datetime import datetime
import tiktoken 
import pandas as pd
from tqdm import tqdm
import openai
from notebookutils import fs
import numpy as np
from synapse.ml.aifunc import Conf
from deepdiff import DeepDiff
from pydantic import BaseModel, ValidationError, field_validator
import builtin.json_diff_extraction_accuracy as diff

path = "/lakehouse/default/Files/complete_purchase_order.xlsx"
df = pd.read_excel(path)

# convert data to markdown
md = pd.DataFrame({"data":[df.to_markdown()]})

# define fields to extract
df = md["data"].ai.extract(
    'job_id',
    'customer_id',
    'po_date',
    'name',
    'phone_number',
    'delivery_type',
    'delivery_date',
    'delivery_time',
    'delivery_address',
    'eir_code',
    'type',
    'material_code',
    'material_description',
    'quantity',
    'uom'
)
display(df)

And just like that without any prompts or efforts, AI function extracted the fields correctly from the first sheet.

We can loop over the other sheets and be done with it. But not so fast. Note that we only got one row back corresponding to the PO header and didn’t get all the rows in the details table. That’s because in the .extract() we can only extract text and not complex data types like dictionaries and lists. To achieve that we will need to write a prompt and use .generate_response() function. To do so, we will:

• write a prompt which includes the schema of the expected response we want

• one of our goals is to achieve the solution by minimizing the CU consumption. AI functions are billed by the spark meter but measured in input and output tokens. We will count the tokens to track this.

• process the response to create two dataframes - one for order detail and another for materials table.

Prompt

In the below prompt, I give instructions and define the JSON schema the LLM should use to provide the response. Notice the data types defined as well as enum (options to choose from, any other value would be invalid). This is a fairly straightforward prompt.

prompt = """
  Extract purchase order details as a valid JSON and material list from the originally from an Excel. Return ONLY the JSON without any explanation or details based on below schema.
  Ignore extraneous details.
  "response_schema": {
    "purchase_order": {
      "job_id": "string",
      "customer_id": "string",
      "po_date": "string (date format: DD/MM/YYYY)",
      "name": "string",
      "phone_number": "string",
      "delivery_type": "string" (enum: Delivery, Pickup, Shipping)
      "delivery_date": "string (date format: DD/MM/YYYY)",
      "delivery_time": "string",
      "delivery_address": "string",
      "eir_code": "string"
    },
    "materials": [
      {
        "type": "string",
        "material_code": "string",
        "material_description": "string",
        "quantity": "integer",
        "uom": "string" (unit of measure, null if not found)
      }
    ]
  }
"""

To calculate the number of tokens, we can use the tiktoken library. Currently, AI function uses gpt-3.5 model. If it changes to any other model, be sure update the model accordingly below (each model uses a different tokenizer).

model = "gpt-3.5-turbo"
tokenizer= tiktoken.encoding_for_model(model)
def get_token_count(obj):
    if not isinstance(obj, str):
        obj = str(obj)
    return len(tokenizer.encode(obj))

Extract the data

path = "/lakehouse/default/Files/complete_purchase_order.xlsx"


sheet_names = ['PO1', 'PO2', 'PO3', 'PO4', 'PO5']
df_dict = {
    sheet: pd.read_excel(path, sheet_name=sheet).dropna(how='all').dropna(how='all', axis=1)
    for sheet in sheet_names
}

df_extract = pd.DataFrame({
    "json_data": [json.loads(df_dict[f'PO{i}'].to_json()) for i in range(1, 6)],
    "string_data": [df_dict[f'PO{i}'].to_string() for i in range(1, 6)],
    "markdwn_data": [df_dict[f'PO{i}'].to_markdown() for i in range(1, 6)]
})
col = "string_data"
df_extract['data'] = df_extract[[col]].ai.generate_response(prompt, conf=Conf(seed=0, max_concurrency=50))
df_extract['input_tokens'] = df_extract[col].apply(get_token_count)
df_extract['output_tokens'] = df_extract["data"].apply(get_token_count)

# df_extract
#### Extract orders and materials

order_dfs = []
material_dfs = []

for num in range(4):
    po = f"PO{num+1}"
    current_data = json.loads(df_extract['data'][num])
    order_df = pd.json_normalize(current_data)
    order_df['source_po'] = po
    order_dfs.append(order_df.drop(columns=['materials'], errors='ignore'))

    materials_df = pd.json_normalize(current_data.get('materials', []))
    materials_df['source_po'] = po
    material_dfs.append(materials_df)

order_df = pd.concat(order_dfs, ignore_index=True).sort_values('purchase_order.customer_id').reset_index(drop=True)
materials_df = pd.concat(material_dfs, ignore_index=True).sort_values(['source_po','type','material_code']).reset_index(drop=True)

To get the number of tokens, use the .ai.stats method:

In this case, we used 3761 input tokens (prompt + input data) and 3773 output tokens (the JSON returned).

Next we still need to clean up the above dataframes a bit more to make it consumption ready by cleaning the column names, fixing the date format and data types etc.

def clean_order_df(df):
    """
    Clean the order dataframe with robust error handling
    """
    cleaned_df = df.copy()

    # Remove prefix from column names
    new_columns = {col: col.replace('purchase_order.', '') for col in cleaned_df.columns if col.startswith('purchase_order.')}
    cleaned_df = cleaned_df.rename(columns=new_columns)

    # Convert customer_id to integer with error handling
    if 'customer_id' in cleaned_df.columns:
        def safe_int_convert(x):
            if pd.isna(x) or x == '':
                return None
            try:
                return int(x)
            except (ValueError, TypeError):
                return None

        cleaned_df['customer_id'] = cleaned_df['customer_id'].apply(safe_int_convert)

    # Convert date columns with error handling
    date_columns = ['po_date', 'delivery_date']
    for date_col in date_columns:
        if date_col in cleaned_df.columns:
            def safe_date_convert(x):
                if pd.isna(x) or x == '':
                    return None
                try:
                    return pd.to_datetime(x, dayfirst=True).strftime('%d-%m-%Y')
                except (ValueError, TypeError):
                    return None

            cleaned_df[date_col] = cleaned_df[date_col].apply(safe_date_convert)

    # Convert text columns with error handling
    text_columns = ['name', 'delivery_type', 'delivery_time', 'delivery_address']
    for text_col in text_columns:
        if text_col in cleaned_df.columns:
            def safe_text_convert(x):
                if pd.isna(x) or x == '':
                    return None
                try:
                    return str(x).title()
                except (ValueError, TypeError):
                    return None

            cleaned_df[text_col] = cleaned_df[text_col].apply(safe_text_convert)

    # Sort and reset index
    try:
        cleaned_df = cleaned_df.sort_values('customer_id', na_position='last').reset_index(drop=True)
    except Exception as e:
        print(f"Warning: Sorting failed with error: {e}")
        cleaned_df = cleaned_df.reset_index(drop=True)

    return cleaned_df

final_order_df = clean_order_df(order_df)
display(final_order_df)

For materials data:

def clean_materials_df(df):
    df['quantity'] = df['quantity'].astype('float')
    df['uom'] = df['uom'].str.lower()
    return df.sort_values(['source_po','type','material_code']).reset_index(drop=True)

final_materials_df = clean_materials_df(materials_df)
display(final_materials_df)

Validation

We got the result we wanted but is it accurate? Let’s find out. To find the accuracy, we need to compare the final results with the actual data. There are many accuracy metrics but we will keep it simple. I will convert the dataframe to a JSON and then compare each field with the JSON of the original excel sheet I manually created for validation purposes.

Create a JSON of the above dfs:

purchase_order = {
    "job_id": final_order_df["job_id"].iloc[0] if "job_id" in final_order_df.columns else "",
    "customer_id": final_order_df["customer_id"].iloc[0] if "customer_id" in final_order_df.columns else "",
    "po_date": final_order_df["po_date"].iloc[0] if "po_date" in final_order_df.columns else "",
    "name": final_order_df["name"].iloc[0] if "name" in final_order_df.columns else "",
    "phone_number": final_order_df["phone_number"].iloc[0] if "phone_number" in final_order_df.columns else "",
    "delivery_type": final_order_df["delivery_type"].iloc[0] if "delivery_type" in final_order_df.columns else "",
    "delivery_date": final_order_df["delivery_date"].iloc[0] if "delivery_date" in final_order_df.columns else "",
    "delivery_time": final_order_df["delivery_time"].iloc[0] if "delivery_time" in final_order_df.columns else "",
    "delivery_address": final_order_df["delivery_address"].iloc[0] if "delivery_address" in final_order_df.columns else "",
    "eir_code": final_order_df["eir_code"].iloc[0] if "eir_code" in final_order_df.columns else ""
}


materials = final_materials_df.to_dict(orient='records')

order_json = {"purchase_order": purchase_order}
materials_json = {"materials": materials}

Fix the data types so they conform to JSON and load the ground truth as JSON.

def convert_numpy_types(obj):
    if isinstance(obj, dict):
        return {k: convert_numpy_types(v) for k, v in obj.items()}
    elif isinstance(obj, list):
        return [convert_numpy_types(i) for i in obj]
    elif isinstance(obj, np.integer):
        return int(obj)
    elif isinstance(obj, np.floating):
        return float(obj)
    elif isinstance(obj, np.ndarray):
        return obj.tolist()
    else:
        return obj


order_json = convert_numpy_types(order_json)
materials_json = convert_numpy_types(materials_json)

with open("/lakehouse/default/Files/ground_truth/order_ground_truth.json", 'r') as f:
    order_groundtruth = json.load(f)

#load materials
with open("/lakehouse/default/Files/ground_truth/materials_ground_truth.json", 'r') as f:
    material_groundtruth  = json.load(f)

To calculate accuracy by comparing each field I wrote Python function using deepdiff which you can get it from my repo.

result = diff.calculate_json_accuracy( material_groundtruth, materials_json)
accuracy_score = result['score']
total_fields = result['total_fields']
diff_stats = result['json_diff_stats']

print(f"Accuracy score: {result['score']}")
print(f"Total fields: {result['total_fields']}")
print(f"Diff stats: {result['json_diff_stats']}")

In the materials dataframe, we compared total 324 fields and all of them are valid and match the actual data. Keep in mind that this is despite missing values, different column names, column orders etc. LLM was able to understand the context and overall structure of the text to extract the fields correctly.

Other Goals

We achieved one of the goals - to extract the data accurately. But what about minimizing the cost, i.e. CU consumption?

I omitted one step above intentionally to keep things simple. In the above code, I first converted the data as string before sending it to LLM via AI function. I could have converted the data to other formats as well like markdown I did in the first baseline or as JSON as well. Let’s see what happens if I do that.

#using string
display(df_extract)

Use markdown instead:

As you can see above, the number of input tokens are about ~10-20% more ! So using string instead of markdown is better in this case example without sacrificing accuracy.

AIOps

In data science, you usually run many experiments to find the right hyperparameters, configurations, and features, including feature engineering, before deciding on the best setup to meet your goals. LLM responses are stochastic, so they should be handled like any other data science project, applying the same principles for reproducibility and accuracy. Fabric has built-in MLflow integration, allowing us to run all the experiments and finalize our solution for production. This blog is already lengthy, so I won't cover all the details here, but at a high level : I set up MLflow experiment, instrument it with metrics I want to capture (inputs, tokens, reponses, accuracy etc.), vary the formatting used, log the runs and compare the results. In a real world project, you will run many experiments with different prompts, temperatures, seeds, model configurations etc but I am skipping it here.

Putting it all together:

import os
import re
import json
import time
import mlflow
import pandas as pd
import numpy as np
import tiktoken
import openai
from tqdm import tqdm
from datetime import datetime
from synapse.ml.aifunc import Conf
from deepdiff import DeepDiff
import builtin.json_diff_extraction_accuracy as diff
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)

experiment_name = "purchase_order_extraction_format_test"
mlflow.set_experiment(experiment_name)

prompt = """
  Extract purchase order details as a valid JSON and material list from the originally from an Excel. Return ONLY the JSON without any explanation or details based on below schema.
  Ignore extraneous details.
  "response_schema": {
    "purchase_order": {
      "job_id": "string",
      "customer_id": "string",
      "po_date": "string (date format: DD/MM/YYYY)",
      "name": "string",
      "phone_number": "string",
      "delivery_type": "string" (enum: Delivery, Pickup, Shipping)
      "delivery_date": "string (date format: DD/MM/YYYY)",
      "delivery_time": "string",
      "delivery_address": "string",
      "eir_code": "string"
    },
    "materials": [
      {
        "type": "string",
        "material_code": "string",
        "material_description": "string",
        "quantity": "integer",
        "uom": "string" (unit of measure, null if not found)
      }
    ]
  }
"""


model = "gpt-3.5-turbo"
tokenizer = tiktoken.encoding_for_model(model)

def get_token_count(obj):
    if not isinstance(obj, str):
        obj = str(obj)
    return len(tokenizer.encode(obj))

## attach a lakehouse and upload the ground truth to teh below folder
def load_groundtruth():
    with open("/lakehouse/default/Files/ground_truth/order_ground_truth.json", 'r') as f:
        order_groundtruth = json.load(f)

    with open("/lakehouse/default/Files/ground_truth/materials_ground_truth.json", 'r') as f:
        material_groundtruth = json.load(f)

    return order_groundtruth, material_groundtruth

## this is for converting to int for deepdiff, otherwise not needed
def convert_numpy_types(obj):
    if isinstance(obj, dict):
        return {k: convert_numpy_types(v) for k, v in obj.items()}
    elif isinstance(obj, list):
        return [convert_numpy_types(i) for i in obj]
    elif isinstance(obj, np.integer):
        return int(obj)
    elif isinstance(obj, np.floating):
        return float(obj)
    elif isinstance(obj, np.ndarray):
        return obj.tolist()
    else:
        return obj


def clean_order_df(df):
    cleaned_df = df.copy()
    new_columns = {col: col.replace('purchase_order.', '') for col in cleaned_df.columns if col.startswith('purchase_order.')}
    cleaned_df = cleaned_df.rename(columns=new_columns)

    if 'customer_id' in cleaned_df.columns:
        def safe_int_convert(x):
            if pd.isna(x) or x == '':
                return None
            try:
                return int(x)
            except (ValueError, TypeError):
                return None

        cleaned_df['customer_id'] = cleaned_df['customer_id'].apply(safe_int_convert)


    date_columns = ['po_date', 'delivery_date']
    for date_col in date_columns:
        if date_col in cleaned_df.columns:
            def safe_date_convert(x):
                if pd.isna(x) or x == '':
                    return None
                try:
                    return pd.to_datetime(x, dayfirst=True).strftime('%d-%m-%Y')
                except (ValueError, TypeError):
                    return None

            cleaned_df[date_col] = cleaned_df[date_col].apply(safe_date_convert)


    text_columns = ['name', 'delivery_type', 'delivery_time', 'delivery_address']
    for text_col in text_columns:
        if text_col in cleaned_df.columns:
            def safe_text_convert(x):
                if pd.isna(x) or x == '':
                    return None
                try:
                    return str(x).title()
                except (ValueError, TypeError):
                    return None

            cleaned_df[text_col] = cleaned_df[text_col].apply(safe_text_convert)

    try:
        cleaned_df = cleaned_df.sort_values('customer_id', na_position='last').reset_index(drop=True)
    except Exception as e:
        print(f"Warning: Sorting failed with error: {e}")
        cleaned_df = cleaned_df.reset_index(drop=True)

    return cleaned_df

def clean_materials_df(df):
    df['quantity'] = df['quantity'].astype('float')
    df['uom'] = df['uom'].str.lower()
    return df.sort_values(['source_po','type','material_code']).reset_index(drop=True)

def process_format_test(format_col, df_extract, mlflow_run_name, temp=0.0):
    order_groundtruth, material_groundtruth = load_groundtruth()

    with mlflow.start_run(run_name=mlflow_run_name):
        mlflow.log_param("model", model)
        mlflow.log_param("format", format_col)
        mlflow.log_param("prompt", prompt)
        mlflow.log_param("timestamp", datetime.now().strftime("%Y-%m-%d %H:%M:%S"))

        start_time = time.time()
        df_extract['data'] = df_extract[[format_col]].ai.generate_response(prompt, conf=Conf(seed=0, max_concurrency=50, temperature=temp))
        execution_time = time.time() - start_time

        stats = df_extract.ai.stats
        mlflow.log_metric("num_successful", stats['num_successful'])
        mlflow.log_metric("num_exceptions", stats['num_exceptions'])
        mlflow.log_metric("num_unevaluated", stats['num_unevaluated'])
        mlflow.log_metric("prompt_tokens", stats['prompt_tokens'])
        mlflow.log_metric("completion_tokens", stats['completion_tokens'])
        mlflow.log_metric("execution_time", execution_time)

        df_extract['input_tokens'] = df_extract[format_col].apply(get_token_count)
        df_extract['output_tokens'] = df_extract["data"].apply(get_token_count)

        mlflow.log_metric("avg_input_tokens", df_extract['input_tokens'].mean())
        mlflow.log_metric("avg_output_tokens", df_extract['output_tokens'].mean())

        order_dfs = []
        material_dfs = []

        for num in range(min(4, len(df_extract))):
            po = f"PO{num+1}"
            try:
                current_data = json.loads(df_extract['data'][num])

                # order data
                order_df = pd.json_normalize(current_data)
                order_df['source_po'] = po
                order_dfs.append(order_df.drop(columns=['materials'], errors='ignore'))

                # materials data
                materials_df = pd.json_normalize(current_data.get('materials', []))
                materials_df['source_po'] = po
                material_dfs.append(materials_df)
            except Exception as e:
                mlflow.log_param(f"error_po{num+1}", str(e))
                continue

        if not order_dfs or not material_dfs:
            mlflow.log_metric("accuracy_score", 0.0)
            return

        order_df = pd.concat(order_dfs, ignore_index=True)
        materials_df = pd.concat(material_dfs, ignore_index=True)

        final_order_df = clean_order_df(order_df)
        final_materials_df = clean_materials_df(materials_df)

        # this is shaping the data for deepdiff, to convert into a record format
        purchase_order = {
            "job_id": final_order_df["job_id"].iloc[0] if "job_id" in final_order_df.columns and len(final_order_df) > 0 else "",
            "customer_id": final_order_df["customer_id"].iloc[0] if "customer_id" in final_order_df.columns and len(final_order_df) > 0 else "",
            "po_date": final_order_df["po_date"].iloc[0] if "po_date" in final_order_df.columns and len(final_order_df) > 0 else "",
            "name": final_order_df["name"].iloc[0] if "name" in final_order_df.columns and len(final_order_df) > 0 else "",
            "phone_number": final_order_df["phone_number"].iloc[0] if "phone_number" in final_order_df.columns and len(final_order_df) > 0 else "",
            "delivery_type": final_order_df["delivery_type"].iloc[0] if "delivery_type" in final_order_df.columns and len(final_order_df) > 0 else "",
            "delivery_date": final_order_df["delivery_date"].iloc[0] if "delivery_date" in final_order_df.columns and len(final_order_df) > 0 else "",
            "delivery_time": final_order_df["delivery_time"].iloc[0] if "delivery_time" in final_order_df.columns and len(final_order_df) > 0 else "",
            "delivery_address": final_order_df["delivery_address"].iloc[0] if "delivery_address" in final_order_df.columns and len(final_order_df) > 0 else "",
            "eir_code": final_order_df["eir_code"].iloc[0] if "eir_code" in final_order_df.columns and len(final_order_df) > 0 else ""
        }

        materials = final_materials_df.to_dict(orient='records')

        order_json = {"purchase_order": purchase_order}
        materials_json = {"materials": materials}

        order_json = convert_numpy_types(order_json)
        materials_json = convert_numpy_types(materials_json)

        # claculate accuracy
        result = diff.calculate_json_accuracy(material_groundtruth, materials_json)
        accuracy_score = result['score']
        total_fields = result['total_fields']
        diff_stats = result['json_diff_stats']

        # Log metrics
        mlflow.log_metric("accuracy_score", accuracy_score)
        mlflow.log_metric("total_fields", total_fields)
        mlflow.log_metric("diff_additions", diff_stats['additions'])
        mlflow.log_metric("diff_deletions", diff_stats['deletions'])
        mlflow.log_metric("diff_modifications", diff_stats['modifications'])
        mlflow.log_metric("diff_total", diff_stats['total'])

        with open("order_extracted.json", "w") as f:
            json.dump(order_json, f, indent=2)

        with open("materials_extracted.json", "w") as f:
            json.dump(materials_json, f, indent=2)

        mlflow.log_artifact("order_extracted.json")
        mlflow.log_artifact("materials_extracted.json")

        # optional i you want to log the data for tracing       
        # mlflow.log_artifact("order_dataframe.csv")
        # mlflow.log_artifact("materials_dataframe.csv")

        return {
            "format": format_col,
            "accuracy": accuracy_score,
            "execution_time": execution_time,
            "diff_total": diff_stats['total']
        }

Run the MLFlow experiments:

path = "/lakehouse/default/Files/complete_purchase_order.xlsx"
sheet_names = ['PO1', 'PO2', 'PO3', 'PO4', 'PO5']

# Load Excel data
df_dict = {
    sheet: pd.read_excel(path, sheet_name=sheet).dropna(how='all').dropna(how='all', axis=1)
    for sheet in sheet_names
}


df_extract = pd.DataFrame({
    "json_data": [json.loads(df_dict[f'PO{i}'].to_json()) for i in range(1, 6)],
    "string_data": [df_dict[f'PO{i}'].to_string() for i in range(1, 6)],
    "markdwn_data": [df_dict[f'PO{i}'].to_markdown() for i in range(1, 6)]
})


formats = ["string_data", "json_data", "markdwn_data"]
results = []

for fmt in formats:
    result = process_format_test(fmt, df_extract.copy(), f"format_test_{fmt}")
    if result:
        results.append(result)

if results:
    results_df = pd.DataFrame(results)
    results_df.to_csv("format_comparison.csv", index=False)

    with mlflow.start_run(run_name="summary"):
        mlflow.log_artifact("format_comparison.csv")
        best_format = results_df.loc[results_df['accuracy'].idxmax(), 'format']
        mlflow.log_param("best_format", best_format)

        for col in results_df.columns:
            if col != 'format':
                mlflow.log_metric(f"avg_{col}", results_df[col].mean())

MLFLow inline run comparison:

In the run comparison we can see that both text and markdown resulted in 100% accuracy however markdown generated more tokens. JSON on the other hand is <100% accurate.

Why does this matter? Because, as you receive more data, varied data, as LLMs used in AI function changes, you need to be able to trace this back and reproduce/benchmark the results (as much as possible) so you can productionize the solution confidently. Using AI is easy, building evals is hard but necessary.

Risks, Limitations

• I created four layout examples. In the real world, you might encounter a layout different from the four I used. We can address this in a few ways. We can extract features from the current forms (Azure Doc Intelligence, for example, also provides the layout). We can train a machine learning model on many examples to create a classifier or use a large language model as a classifier for incoming forms. (e.g. a layout drift model)

• Rate limits and context size: GPT-3.5 has a context window of only 16K tokens (including both input and output) and a maximum output of less than 4K tokens. This means it can't handle large Excel tables. In the example above, the average output was 1000 tokens, so for GPT-3.5, the data can't be more than about four times what I used. However, this is temporary. It was announced at FabCon that the model will be updated to newer GPT models with a larger context window (128K instead of 16K). AI functions also have a rate limit of 1000 TPM. For long documents, we can split the data into chunks to work around these limits, but that will require extra effort.

• LLMs are bound to hallucinate and are subject to prompt injection, e.g. what if in the excel sheet there is text that intentionally or unintentionally instructs the models to behave differently and fudge the numbers?

• The eval harness I created checked for two things - schema and values. But we could build more evaluation metrics for specific cases to ensure the accuracy risks are mitigated.

• If Microsoft changes the underlying model, it may affect your results. You can always use a Azure Open AI custom deployment to control for this.

• Always be aware of risks associated with sending sensitive data to LLMs and AI services.

To estimate cost, you can use my Fabric AI & Copilot cost calculator.

References

• Transform and enrich data seamlessly with AI functions - Microsoft Fabric | Microsoft Learn

• Welcome to Pydantic - Pydantic

• Unstructured To Structured : Using Fabric AI Functions To Extract Invoice Data From PDFs

• Customize the configuration of AI functions - Microsoft Fabric | Microsoft Learn

Steps:

• Get application id and livy id of the application (notebook or SJD)

• Retrieve event log zip and save it to a lakehouse

• Extract the event log JSON and save it to a lakehouse

• Parse the event log for spark metrics

Get Session Info

Use the get_latest_session_info() function from my previous blog. This will give you the latest application id and the livy id which will be required for further API calls. Note that if you want to get ids for all previous sessions modify the function accordingly.

## Fabric Python or Pyspark notebook
## Function from https://fabric.guru/extracting-fabric-spark-driver-logs-using-api

notebook_id = "996b5d64-xxxxxxxx-ff2ba76d0fc8"
workspace_id = fabric.get_notebook_workspace_id() #or replace with your workspace id
session = get_latest_session_info(notebook_id, workspace_id)
livy_id = session['livyId']
app_id = session['applicationId']

Retrieve Spark Event Log

Spark event log can be several hundred MBs or even GBs so attach a default lakehouse to save the log zip file to a lakehouse.

import os
import zipfile
import glob
import sempy.fabric as fabric

def get_spark_event_log(notebook_id, workspace_id, output_path, livy_id=None, app_id=None):
    """
    Sandeep Pawar | fabric.guru | May 19,2025
    Gets Spark event logs and saves to specified path in an attached lakehouse.
    """
    if not livy_id or not app_id:
        session_info = get_latest_session_info(notebook_id, workspace_id)
        if not session_info:
            return "Error: Could not retrieve session info"
        livy_id = session_info.get('livyId')
        app_id = session_info.get('applicationId')

    if os.path.isdir(output_path) or output_path.endswith('/'):
        os.makedirs(output_path, exist_ok=True)
        output_path = os.path.join(output_path, f"spark_log_{app_id}.zip")
    else:
        os.makedirs(os.path.dirname(output_path), exist_ok=True)

    client = fabric.FabricRestClient()
    #refer to https://learn.microsoft.com/en-us/rest/api/fabric/spark/livy-sessions for API details
    #/1/ below is for the first try
    endpoint = f"https://api.fabric.microsoft.com/v1/workspaces/{workspace_id}/notebooks/{notebook_id}/livySessions/{livy_id}/applications/{app_id}/1/logs"

    try:
        print(f"Retrieving logs from endpoint: {endpoint}")
        response = client.get(endpoint)

        if response.status_code != 200:
            return f"Error: Received status code {response.status_code}. Response: {response.text}"

        with open(output_path, "wb") as f:
            f.write(response.content)

        file_size = len(response.content)
        return f"Successfully saved event logs ({file_size/1024/1024:.2f} MB) to {output_path}"

    except Exception as e:
        return f"Error retrieving logs: {str(e)}"

Extract Event Log

The zip file contains the event log which needs to be extracted. Below functions unzips the file and save it to the specified path in the lakehouse.

## Be sure to attach a default lakehouse
# I assumed it's .zip, adjust if other zip compression in the future
def unzip_spark_log(zip_path, extract_path):
    """
    Sandeep Pawar | fabric.guru | May 19, 2025
    Extracts Spark event log zip to specified directory. 

    """
    if os.path.isdir(zip_path):
        zip_files = glob.glob(os.path.join(zip_path, "*.zip"))
        if not zip_files:
            return f"Error: {zip_path} is a directory and no zip files were found within it"
        zip_path = zip_files[0]
        print(f"Using zip file: {zip_path}")

    if not os.path.exists(zip_path):
        return f"Error: Zip file {zip_path} doesn't exist"

    os.makedirs(extract_path, exist_ok=True)

    try:
        with zipfile.ZipFile(zip_path, 'r') as zip_ref:
            file_list = zip_ref.namelist()
            zip_ref.extractall(extract_path)

        return f"Extracted {len(file_list)} files to {extract_path}: {', '.join(file_list)}"

    except Exception as e:
        return f"Error extracting logs: {str(e)}"

Example:

##python or pyspark notebook
output_path = "/lakehouse/default/Files/rawlogs"
get_spark_event_log(notebook_id, workspace_id,output_path , livy_id=livy_id, app_id=app_id) #saves the zip file
unzip_spark_log(output_path, output_path) #unzips the zip file

Parse log for spark metrics:

Parse the event log JSON to get the performance metrics. Below I am extracting the stage metrics across all the stages. From this you can get metrics like data spill, GC time, shuffle read/write, CPU time, idle time, memory used etc which can help you optimize the spark application (more on this later).

##pyspark notebook
%%pyspark
df = spark.read.json(f"Files/rawlogs/application_xxxxxxx")
df1 = df.filter("Event='SparkListenerStageCompleted'").select("`Stage Info`.*")
df1.createOrReplaceTempView("t2")
df2 = spark.sql("select 'Submission Time','Completion Time', 'Number of Tasks', 'Stage ID', t3.col.* from t2 lateral view explode(Accumulables) t3")
df2.createOrReplaceTempView("t4")
result = spark.sql("select Name, sum(Value) as value from t4 group by Name order by Name")
display(result)

Similar to stage-level, you can also get task-level metrics, or aggregate metrics for all jobs in a workspace etc. There are other couple of interesting APIs which I will cover in future blogs.

I will share more on how to interpret these metrics and how they can provide insights into the application. Note that Fabric offers many built-in monitoring capabilities. APIs give you the ability to access detailed metrics and create additional custom metrics.

You can download the event log manually as well from the spark history server:

I would like to thank Jenny Jiang from Microsoft for answering my questions.

References:

• Extracting Fabric Spark Driver Logs Using API

• Announcing the Fabric Apache Spark Diagnostic Emitter: Collect Logs and Metrics | Microsoft Fabric Blog | Microsoft Fabric

• Livy Sessions - REST API (Spark) | Microsoft Learn

• Apache Spark monitoring overview - Microsoft Fabric | Microsoft Learn

• https://github.com/LucaCanali/sparkMeasure

• Monitoring and Instrumentation - Spark 3.5.5 Documentation

• How to explore Apache Spark metrics with Spark listeners - Databricks

• groupon/spark-metrics: A library to expose more of Apache Spark's metrics system

• Workspace monitoring overview - Microsoft Fabric | Microsoft Learn