Search for a command to run...
V-order is highly recommended for optimal Delta & Direct Lake performance in Fabric. Here I show how to check if the table has been V-order optimized.
Principal Program Manager, Microsoft Fabric CAT helping users and organizations build scalable, insightful, secure solutions. Blogs, opinions are my own and do not represent my employer.
Thanks for the post! I tried to use the basic os function you mentioned here, but I found the result is always true, even if I delete the table I'm searching for. Any idea why that would be?
if(os.path.exists('//lakehouse/default/Tables/table_name')):
Very good!
Last week Microsoft released an open-source text embedding model called Harrier in three sizes- 270M, 0.6B and 27B. I have been testing it in my RAG pipeline and so far it has crushed all my metrics.
For Power BI Copilot and Data agents with semantic models, you must use Prep Data for AI configuration to ground the responses in the context added in Prep for AI. In this blog, I will show you how yo
At FabCon Atlanta last week, the updated notebook Copilot for data engineering and data science was announced. It brings agentic capabilities to the Copilot and is much more intelligent and Fabric-awa
Using Fabric data agent Python SDK
Use workspace monitoring to monitor MCP server use
V-order is a write time optimization to the parquet file format. When the delta table is created using any of the Fabric engines (Dataflow Gen2, Spark notebooks, Pipelines, DWH), Delta tables are automatically are V-Order'd. This not only helps with size of the table but can significantly improve Direct Lake dataset read performance. While it's not required, Delta tables with V-order are highly recommended for fastest Direct Lake and Delta read performance. For more ob V-order, read this official article.
However, there is no direct way to identify if a table already has V-order or not. There three ways to check but let me show two easy ways. I will cover the third in a future blog post. When a Delt atable is created, the transaction logs have a metadata property related to V-order.
I created two Delta tables in a Fabric lakehouse, one with and the other without the V-order by changing the spark configuration.
You can manually inspect the transaction logs in the Lakehouse by going to Lakehouse > Table name > Right Click > View Files > _delta_log, inspect the latest .json file. In the transaction logs you will either see VORDER set to false or missing if the table is not V-Order'd. If the Delta table is V-order, the property is set to true .
If you have several Delta tables, many transaction logs or if you need to check it as a part of your validation process for DataOps, you can use pyarrow to check the schema metadata. This only checks the metadata so the table is not required to be read. Below is the Python code I used:
def check_vorder(table_name_path):
'''
Author: Sandeep Pawar | fabric.guru | Jun 6, 2023
Provide table_name_path as '//lakehouse/default/Tables/<table_name>'
If the Delta table is V-ordered, returns true; otherwise, false.
You must first mount the lakehouse to use the local filesystem API.
'''
import os
if not os.path.exists(table_name_path):
print(f'{os.path.basename(table_name_path)} does not exist')
result = None # Initialize the variable with a default value
else:
import pyarrow.dataset as ds
schema = ds.dataset(table_name_path).schema.metadata
is_vorder = any(b'vorder' in key for key in schema.keys())
if is_vorder:
result = str(schema[b'com.microsoft.parquet.vorder.enabled'])
else:
result = "Table is not V-ordered"
return result
There is another robust method using Spark that can be used to detect parquet files in the Delta table that are non V-Order'd but I will cover that in a future blog post.