First fabric-cicd deployments steps

This post includes what I consider to be your first fabric-cicd deployment steps from your local machine. Because I am aware that a lot of people have yet to work with fabric-cicd.

Fabric-cicd is a Python library that allows you to perform CI/CD of various Microsoft Fabric items into Microsoft Fabric workspaces. With some customization it can resolve issues like rebinding items to newly deployed items along the way.

Typically this is done by deploying the metadata for items stored in a workspace that is configured with Microsoft Fabric Git integration. This post focuses solely on local deployments with the below general flow.

General flow of first fabric-cicd deployment steps — Git Logo by Jason Long is licensed under the Creative Commons Attribution 3.0 Unported License.

To manage expectations, this post is aimed at those with little or no experience with the fabric-cicd Python library. However, those with experience can learn things in this post. Because it lays the foundation for various topics. Including configuration-based deployment which is currently in preview.

Below are my guidelines about the four sections for everybody:

1: Essential for new users to learn fabric-cicd.
2: Essential, especially when looking to deploy to multiple workspaces.
3: Essential if looking to perform CI/CD with fabric-cicd in Azure Devops or GitHub.
4: Optional, shows how to enable experimental features and highlights a feature that will make central configuration easier.

Even though this post focuses on local deployments, I do provide links to material that covers Azure DevOps and GitHub deployments as well.

Examples of fabric-cicd deployment steps

All the examples are based on working with Visual Studio Code. You can find an online example of the scripts and folder structure in my fabric-cicd-local GitHub repository.

Which contains the same samples that you can find in the fabric-cicd repository with some slight differences. For instance, the ABC report and semantic model are modified, and the other reports are removed. You can follow along with either with the sample I provide or your own Git repository.

In order perform any of the below examples you need the following installed locally:

Python from versions 3.9 to 3.12.
Azure CLI
Fabric-cicd library. I recommend installing from a command line with ‘pip install –upgrade fabric-cicd’ once Python is installed.

In addition, you need a destination workspace in Microsoft Fabric. With the permissions for your own account and a service principal added to deploy items.

1: Logged is as yourself with hard-coded values

First ever fabric-cicd script you can run to test deploying items to a workspace is one that contains hard-coded values whilst authenticated as yourself. That way you can test the permission in Microsoft Fabric easier. Plus ensure fabric-cicd deploys the right items with correct settings.

First you need to create a Python file that performs the below tasks:

Imports the required Python libraries
Adds your values for the FabricWorkspace object (workspace Id, deployment environment, repository directory and the items you want to deploy)
Publishes the items stored in the repository

Below is an example of a valid script:

from fabric_cicd import FabricWorkspace, publish_all_items

# Sample values for FabricWorkspace  object parameters
workspace_id = "00000000-0000-0000-0000-000000000000"
environment = "Prod"
repository_directory = ("workspace")
item_type_in_scope = ["Report","SemanticModel"]

# Use Azure CLI credential to authenticate
# token_credential = AzureCliCredential()

# Initialize the FabricWorkspace object with the required parameters
target_workspace = FabricWorkspace(
    workspace_id=workspace_id,
    environment=environment,
    repository_directory=repository_directory,
    item_type_in_scope=item_type_in_scope,
)

# Publish all items defined in item_type_in_scope
publish_all_items(target_workspace)

One thing I will point out is that a lot of online examples include a unpublish_all_orphan_items command at the end. Only add this if you want items that do not exist in the workspace folder to be unpublished, in other words deleted.

Afterwards, you can login as yourself with the az login command and then run the script.

python .\1-auth_local_hard_coded.py

Once it has completed go into your Microsoft Fabric workspace to confirm your items are there.

Checking items deployed in fabric workspace with fabric-cicd as part of first fabric-cicd deployment steps — Items deployed in fabric workspace with fabric-cicd

Once you have performed your first deployment you can delete the items. Alternatively, keep them if you only want to do a one-time deployment and are okay with a Python file containing your hard-coded values.

Parameterization

One thing you can start experimenting with at this early stage is the parameterization you can perform with fabric-cicd. You tend to do this with various settings in the “parameter.yml” file. Which is typically found in the root of your repository or the folder that contains your Fabric items.

Parameterization can resolve a lot of binding issues that occur when you deploy items to a new workspace. I show a few examples in the “parameter.yml” file in my sample repository. Like the one below, which is an example of find_replace.

# find_replace examples
find_replace:
    # Replace Power BI Text
    - find_value: "This is a Report in Dev!!!"
      replace_value:
        Test: "This is a Report in Test!!!"
        Prod: "This is a Report in Prod!!!"

You can start with static examples like this and then work towards more advanced methods such as dynamic replacements. Which are perfect for scenarios where you want to dynamically repalce values with those of items in your newly deployed workspace.

2: Logged is as yourself with variables

Working with hard-coded values is fine for a one-time script. However, what do you do when you want to deploy to multiple workspaces? Do you create a script for each one?

That would not be efficient, which is why it is essential to test passing values into a single script. Especially if you are looking to apply the scripts in Azure Devops or GitHub for CI/CD purposes.

Which is why my next recommended step is to start passing arguments into your Python scripts. To make the same Python script reusable. To do this locally you can create PowerShell variables which you can then pass into your Python script as arguments. Like the ones in the below example.

$ProdWorkspace = "00000000-0000-0000-0000-000000000000"
$Environment = "Prod"
$RepositoryDirectory = "workspace"
$ItemsInScope = "Report,SemanticModel"

You then prepare your Python script to accept arguments by importing the argparse module. Which will allow you to pass through arguments into your Python script. Like in the below example.

from fabric_cicd import FabricWorkspace, publish_all_items
import argparse

# Include parsed arguments
parser = argparse.ArgumentParser(description='Process some variables.')
parser.add_argument('--WorkspaceId', type=str)
parser.add_argument('--Environment', type=str)
parser.add_argument('--RepositoryDirectory', type=str)
parser.add_argument('--ItemsInScope', type=str)
args = parser.parse_args()

# Convert item_type_in_scope into a list
allitems = args.ItemsInScope
item_type_in_scope=allitems.split(",")

# Initialize the FabricWorkspace object with the required parameters
target_workspace = FabricWorkspace(
    workspace_id= args.WorkspaceId,
    environment=args.Environment,
    repository_directory=args.RepositoryDirectory,
    item_type_in_scope=item_type_in_scope,    
)

# Publish all items defined in item_type_in_scope
publish_all_items(target_workspace)

Once the script is updated you can run the Python script stating the PowerShell variables as arguments.

python .\2-auth_local_variables.py --WorkspaceId $ProdWorkspace --Environment $Environment --RepositoryDirectory $RepositoryDirectory --ItemsInScope  $ItemsInScope

Which will then populate your workspace with the items in scope.

Now you can work with the same Python script to deploy to multiple workspaces. Which is a lot more efficient. After doing this I recommend experimenting with changing the different variables. For example, increase the items in scope.

3: Authenticating as a service principal

If you are testing fabric-cicd locally with the intention of working in Azure DevOps or GitHub I recommend authenticating as a service principal. To check that the service principal can deploy items to the target workspace.

When you look to do this remember that the service principal only needs access to the workspace. No other special setup is required. You can sign in with a service principal with the az login command in various ways, including the below:

az login --service-principal --username APP_ID --password CLIENT_SECRET --tenant TENANT_ID

Another key thing you need to do is ensure that the service principal has enough permissions to create items in the target workspace. Once the above is done you can run the Python script with arguments.

python .\2-auth_local_variables.py --WorkspaceId $ProdWorkspace --Environment $Environment --RepositoryDirectory $RepositoryDirectory --ItemsInScope  $ItemsInScope

Once you are done, you can push the contents of your fabric-cicd repository to a repository in Azure DevOps or GitHub. In order to start experimenting with deployments.

Pushing from your local machine provides the opportunity to prepare everything locally beforehand. I highlighted this in my previous post about preparing Git repositories for Microsoft Fabric Git integration.

Preparing locally first allows you to include a detailed README file like the one in my sample instead of a bland auto-generated one. I generated the one in my sample repository with GitHub Copilot in Visual Studio Code and various MCP servers. Including the Microsoft Learn MCP server.

One small piece of advice if intending to automate fabric-cicd is to add the below to your Python script after stating your imports. To ensure the ordering of your output is correct in Azure DevOps or GitHub.

# Force unbuffered output like `python -u`
sys.stdout.reconfigure(line_buffering=True, write_through=True)
sys.stderr.reconfigure(line_buffering=True, write_through=True)

Anyway, you can find out more about how to operationalize fabric-cicd for Azure Devops or GitHub in the below posts.

4: Fabric-cicd Configuration deployment

This section is purely optional. However, it does introduce you to configuration deployment in fabric-cicd. Plus shows how to enable experimental features in your Python script.

Configuration deployment basically allows you to manage deployments to multiple environments through one centralized configuration file. I personally like working with configuration management as it simplifies managing a variety of settings in one place.

For example, I can specify a variety of settings for two different environments with the below code:

core: # Core configurations
  # Either workspace or workspace_id must be provided
  workspace: # Workspace names by environment
    test: fabric-cicd-config-test
    prod: fabric-cicd-config-prod

  repository_directory: "."  # Path to workspace items directory (relative to config.yml location) (required)

  item_types_in_scope:  # Item types to include in deployment (optional)
    - VariableLibrary
    - Dataflow
    - DataPipeline
    - Notebook
    - Environment
    - Report
    - SemanticModel
    - MirroredDatabase
    - Lakehouse

  parameter: "parameter.yml" # Path to parameter file (relative to config.yml location) (optional)

publish: # Publish configuration (optional)
  exclude_regex: "^.*Shortcut.*"  # Regex pattern to exclude items from publishing
  folder_exclude_regex: "^WithSchema.*"  # Regex pattern to exclude folders from publishing

  skip:  # Skip publishing for specific environments
    test: false # Enable publishing in test environment
    prod: false # Enable publishing in prod environment

unpublish: # Unpublish configuration (optional)
  exclude_regex: "^DEBUG.*"  # Regex pattern to exclude items from unpublishing

  skip:  # Skip unpublishing for specific environments
    test: false # Enable unpublishing in test environment
    prod: false # Enable unpublishing in prod environment

features:  # Feature flags to enable (optional)
  - enable_shortcut_publish
  - enable_exclude_folder

Plus, it makes the Python script becomes a lot more efficient. Because all I need to specify in my Python script are the relevant library imports, the options to enable the relevant feature flags and the location of my config file. As you can see below.

# Log in with Azure CLI (az login) prior to execution
# Syntax to run 'python .\auth_local_hard_coded.py'

from pathlib import Path

from fabric_cicd import deploy_with_config,append_feature_flag

append_feature_flag("enable_experimental_features")
# Commented out as not required from v0.2.0 onwards
# append_feature_flag("enable_config_deploy")

# Deploy using a config file
deploy_with_config(
    config_file_path="workspace/config.yml",
    environment="test"
)

Note that I commented out the enable_config_deploy feature flag. Because it is now an available feature from v0.2.0. I can then run the Python script.

python .\3-auth_local_config.py

Once completed, the process populates the workspace. As you can see below.

Items deployed in fabric workspace with fabric-cicd configuration-based deployment

Note that this can take a while when working with my full example due to the number of items and configurations. Feel free to adjust where required.

I recommend testing what other benefits working with the configuration deployment offers you. Because I often see issues it resolves appear on forums.

To learn how to deploy Azure DevOps read my post on how to operationalize configuration-based deployments to work with Microsoft Fabric and Azure DevOps.

Final words about first fabric-cicd deployment steps

I am sharing what I consider the first fabric-cicd deployment steps from a local machine to help others get started. Because getting familiar with fabric-cicd this way makes it easier for others to adopt in either Azure Devops or GitHub.

As you can see, you can experiment with multiple topics that are covered in this post. Hopefully the other links on how to operationalize fabric-cicd in Azure DevOps or GitHub will help you continue your journey there as well.