Operationalize configuration-based deployments to work with Microsoft Fabric and Azure DevOps

This post covers how to operationalize configuration-based deployments to work with Microsoft Fabric and Azure DevOps. This feature is now generally available in the latest fabric-cicd release.

Microsoft has officially announced official support for Microsoft fabric-cicd tool. With that news, this feels like the perfect time to follow up on the post read by thousands about operationalizing fabric-cicd to work with Microsoft Fabric and Azure DevOps.

For awareness, this post shows how to perform configuration-based deployments in Azure DevOps with Classic Pipelines. Those who are new to fabric-cicd can check out my other post which covers your first fabric-cicd deployments steps.

Operationalize configuration-based deployments

In order to create a pipeline in Azure DevOps I first synchronized some sample items into Azure Repos. Along with a parameter and a configuration file.

My configuration file includes various settings relating to deploying to both test and production workspaces. Including the below.

core: # Core configurations
  # Either workspace or workspace_id must be provided
  workspace: # Workspace names by environment
    test: fabric-cicd-config-test
    prod: fabric-cicd-config-prod

  workspace_id: # Workspace IDs by environment (takes precedence over workspace if both are provided)
    test: {Workspace GUID value}
    prod: {Workspace GUID value}

  repository_directory: "."  # Path to workspace items directory (relative to config.yml location) (required)

  item_types_in_scope:  # Item types to include in deployment (optional)
    - VariableLibrary
    - Dataflow
    - DataPipeline
    - Notebook
    - Environment
    - Report
    - SemanticModel
    - MirroredDatabase
    - Lakehouse

  parameter: "parameter.yml" # Path to parameter file (relative to config.yml location) (optional)

publish: # Publish configuration (optional)
  exclude_regex: "^.*Shortcut.*"  # Regex pattern to exclude items from publishing

  skip:  # Skip publishing for specific environments
    test: false # Enable publishing in test environment
    prod: false # Enable publishing in prod environment

unpublish: # Unpublish configuration (optional)
  exclude_regex: "^DEBUG.*"  # Regex pattern to exclude items from unpublishing

  skip:  # Skip unpublishing for specific environments
    test: true # Enable unpublishing in test environment
    prod: false # Enable unpublishing in prod environment

features:  # Feature flags to enable (optional)
  - enable_shortcut_publish

As you can see above, I mixed things up a bit by choosing different unpublish settings for the test and production environments. To confirm that particular setting worked.

I then created a Python file in the root of my Git repository to orchestrate the fabric-cicd deployments. Working with a central configuration file allows for the Python script to be leaner than usual. As you can see below.

from fabric_cicd import deploy_with_config,append_feature_flag

import argparse
append_feature_flag("enable_experimental_features")
# Commented out as no longer required from fabric-cicd v0.2.0 onwards for configuration based deployments
# append_feature_flag("enable_config_deploy")

parser = argparse.ArgumentParser(description='Process one variable.')
parser.add_argument('--Environment', type=str)
args = parser.parse_args()

# Deploy using a config file
deploy_with_config(
    config_file_path="fabric-cicd-sample/workspace/config.yml",
    environment=args.Environment
)

Note that I needed the enable experimental features flag to be enabled in order to work with experimental features in the config file.

Creating release pipeline to Operationalize configuration-based deployments

I first added the existing repository as the artifact for my release pipeline. Selecting the dev branch as the source for deployments.

Once I added my artifact I created two stages. One for Test and one for Production. I then had to add the tasks to both stages. Both stages had the below tasks. Only slight difference in the task names was to reflect different destinations and one argument.

Use Python 3.12

First task to run is to select the version of Python to work with via the Python version task. If you do not select a version of Python for this particular method, you will get an error about invalid syntax. You currently need between versions 3.9 and 3.12 for fabric-cicd to work.

Install necessary libraries

Afterwards I had to install the necessary libraries in a PowerShell task. In this task I ran the below code.

python -m pip install --upgrade pip
pip install fabric-cicd

I needed fabric-cicd since that was the main library required.

Authenticate as a service principal

Once the libraries are installed the next task authenticates as a service principal with the below PowerShell code.

Install-Module -Name Az.Accounts -AllowClobber -Force

$SecureStringPwd = ConvertTo-SecureString $(AZURE_CLIENT_SECRET) -AsPlainText -Force
$pscredential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $(AZURE_CLIENT_ID), $SecureStringPwd
                  
Connect-AzAccount -ServicePrincipal -Credential $pscredential -Tenant $(AZURE_TENANT_ID)

$fabricToken = (Get-AzAccessToken -ResourceUrl $(resourceUrl)).Token

As you can see you need four Azure Pipeline variables added for this task:

• AZURE_CLIENT_ID – Your service principal client ID.
• AZURE_CLIENT_SECRET – Your service principal secret. Note this is the secret value.
• AZURE_TENANT_ID – Your Microsoft Entra tenant ID.
• resourceUrl – URL value to get a bearer token for the specific API. In this case for “https://api.fabric.microsoft.com”.

One key point I must stress here is that your service principal must have permissions to change the Microsoft Fabric workspace in question.

Anyway, I chose to do this with PowerShell since this also method allows you to authenticate with trial E5 subscriptions. However, those operating with a regular tenant can also configure an Azure DevOps service connection as an alternative to this step.

In addition, these variables are capitalized due to my testing methods. Feel free to change them as required.

Run script to deploy with fabric-cicd

In the final stage I call the Python script that orchestrates the deployment with fabric-cicd with a Python script task. With the script path configured to “$(System.DefaultWorkingDirectory)/_fabric-cicd-sample/auth_spn_secret_AzDo_config.py”.

I decided on this naming convention since it is a modified version of the sample ‘auth_spn_secret.py’ provided with the fabric-cicd GitHub repository. Which I changed to work with Azure DevOps in a more efficient manner.

For each environment I only need to pass through the environment argument. Since everything else is managed in the central configuration file.

--Environment "test" 

Once done, I was ready to run my pipeline.

Test results to operationalize configuration-based deployments

When I run the pipeline the two stages complete like in the below example.

When I checked the two Fabric workspaces, I saw that all the items had deployed into them. I then checked the deployment log for test to confirm that the unpublish operation was skipped for test.

Skipping unpublish operation for environment test‌

Afterwards, I checked the logs for production to confirm that it attempted to unpublish items. proving that the configuration file works as expected.

 ‌########## Unpublishing Orphaned Items #############################################################‌

Tips to operationalize configuration-based deployments in fabric-cicd optimally

Below are some general tips to operationalize configuration-based deployments with fabric-cicd optimally.

• Keep an eye on fabric-cicd updates. Always check you are reviewing the latest version on the fabric-cicd page.
• Work with YAML pipelines instead of Release pipelines in Azure Devops.
• Create a service connection in Azure DevOps that authenticates as a service principal.
• Test with your own development items configured with Microsoft Fabric Git integration.
• Experiment with the various configuration options to suit your own strategy. Plus, alternative methods to pass through the parameters.

Final words

I think it is fitting to publish this post the same week that Microsoft announced official support for Microsoft fabric-cicd tool. because it highlights just how far the fabric-cicd library has come in the last year. Thanks to efforts by both Microsoft and members of the community.

I know a lot of people prefer a central configuration option. Feel free to consume the contents of this post and customize it to suit your needs.