Cancelling Dataflow Gen2 Refresh

Using the PowerBIRestClient in Semantic-Link v0.5.0

Update : Mar 8, 2024

As of March 8, you can cancel the Fabric Dataflow Gen2 refresh in service using UI:

Programmatically cancel the refresh:

DFg2 can be resource intensive, and if the refresh takes longer than expected, it may consume a significant amount of CUs. Thankfully, you can use the Power BI Rest API to cancel it programmatically. My friend Alex Powers already has a PowerShell script that you can use. You can also use the Power BI VS Code extension by Gerhard Brueckl.

But I would like to show you how you can do this using the PowerBIRestClient in the latest version of Semantic-Link (v0.5.0). The API is similar to the FabricRestClient I blogged about in my last blog. The new version has many updates I love and I will write about it soon. But in the meantime, checkout the official documentation for the details.

In the below code:

Semantic Link makes calling APIs very easy. Authentication is handled automatically for you so no need to generate a token.

# Install semantic-link >= 0.5.0
!pip install semantic-link --q

import pandas as pd
import sempy.fabric as fabric
from sempy.fabric.exceptions import FabricHTTPException

# Instantiate the client
client = fabric.PowerBIRestClient()

def get_dataflow_transactions(workspace_id, dataflow_id):
    try:
        response = client.get(f"v1.0/myorg/groups/{workspace_id}/dataflows/{dataflow_id}/transactions")

        if response.status_code != 200:
            raise FabricHTTPException(response)
            return pd.DataFrame()

        transactions = response.json().get('value', [])
        if not transactions:
            return pd.DataFrame([{'status': 'No Status', 'transaction': 'No Transaction ID', 'refreshType': 'None'}])

        first_transaction = transactions[0]
        return pd.DataFrame([{
            'status': first_transaction['status'],
            'transaction': first_transaction['id'],
            'refreshType': first_transaction['refreshType']
        }])
    except Exception as e:
        print("Error getting transactions. Check workspace, dataflow id:", e)
        return pd.DataFrame()

def get_dataflows(workspace):
    try:
        workspace_id = fabric.resolve_workspace_id(workspace)
    except Exception as e:
        print("Check workspace name/id:", e)
        return pd.DataFrame()

    try:
        response = client.get(f"v1.0/myorg/groups/{workspace_id}/dataflows")
        if response.status_code != 200:
            raise FabricHTTPException(response)
            return pd.DataFrame()

        dataflows = response.json().get('value', [])
        dataflow_df = (pd.json_normalize(dataflows, max_level=1)
                       .rename(columns={"name": "dataflowName", "objectId": "dataflowID"})
                       .dropna()
                       .reset_index(drop=True))
        result = dataflow_df.join(dataflow_df.apply(lambda x: get_dataflow_transactions(workspace_id, x['dataflowID']).iloc[0], axis=1))
        return result
    except Exception as e:
        print("Error in get_dataflows:", e)
        return pd.DataFrame()


def cancel_dataflow(workspace, transaction_id):
    """
    Cancel a specific dataflow transaction.
    """
    try:
        workspace_id = fabric.resolve_workspace_id(workspace)
        response = client.post(f"v1.0/myorg/groups/{workspace_id}/dataflows/transactions/{transaction_id}/cancel")
        if response.status_code != 200:
            raise FabricHTTPException(response)
            return None

        return response.json()
    except Exception as e:
        print("Error:", e)
        return None

Example:

Get dataflow transactions:

Cancel the dataflow:

💡
As I mentioned in my last blog, you should use the FabricRestClient for calling the Fabric API endpoints and PowerBIRestClient for the Power BI APIs. Hopefully in the future, both will merge and only one will be needed.
💡
Tip: If you need to run a DFg2 with heavy transformations, run it in a pipeline so you can set the timeout and cancel the pipeline run instead.

This was one example . You can use this client to call any Power BI Rest endpoint and make GET, POST requests. If you are new to Python, note that the actual code is just a couple of lines, majority is for error/exception handling to make it robust. I love Semantic Link, and it's becoming indispensable day by day.

Did you find this article valuable?

Support Sandeep Pawar by becoming a sponsor. Any amount is appreciated!