Orchestrate configuration-based deployments to Microsoft Fabric with YAML Pipelines

This post covers how to orchestrate configuration-based deployments to Microsoft Fabric with YAML pipelines in Azure DevOps. This feature is now generally available in the latest fabric-cicd release.

Completed YAML pipeline for configuration-based deployment to Microsoft Fabric — Completed YAML pipeline for configuration-based deployment

In my previous post I showed you how to operationalize configuration-based deployments to work with Microsoft Fabric and the Classic Pipelines in Azure DevOps. Now that Microsoft has officially announced official support for Microsoft fabric-cicd tool this feels like the perfect time to show how to do this with YAML pipelines.

I provide a sample GitHub repository that you can clone/download and work with. Dedicated solely to config-based deployments so that the README file is dedicated to this topic. Within this post I also share plenty of links.

Those who are new to fabric-cicd can check out my other post which covers your first fabric-cicd deployments steps.

Orchestrate configuration-based deployments to Microsoft Fabric

In order to create a YAML pipeline in Azure DevOps I first synchronized some sample items into Azure Repos. Along with a parameter and a configuration file in a workspace subfolder. Along with the Python script I wish to work with.

Repository structure including workspace folder containing Fabric items

Inside the workspace folder there is a configuration file (config.yml) that includes various settings relating to deploying to both test and production workspaces. Including the below:

core: # Core configurations
  # Either workspace or workspace_id must be provided
  workspace: # Workspace names by environment
    test: fabric-cicd-config-test # Must be a valid workspace name
    prod: fabric-cicd-config-prod

# Commented out to show workspace name usage
  workspace_id: # Workspace IDs by environment (takes precedence over workspace if both are provided)
    test: 424849e1-cdfe-4011-a665-94f203fbcab7
    prod: 58d7038b-6726-41eb-8bcd-aca571ea3307

  repository_directory: "."  # Path to workspace items directory (relative to config.yml location) (required)

  item_types_in_scope:  # Item types to include in deployment (optional)
    - VariableLibrary
    - Dataflow
    - DataPipeline
    - Notebook
    - Environment
    - Report
    - SemanticModel
    - MirroredDatabase
    - Lakehouse

  parameter: "parameter.yml" # Path to parameter file (relative to config.yml location) (optional)

publish: # Publish configuration (optional)
  exclude_regex: "^NotThisitem*."  # Regex pattern to exclude items from publishing

  skip:  # Skip publishing for specific environments
    test: false # Enable publishing in test environment
    prod: false # Enable publishing in prod environment

unpublish: # Unpublish configuration (optional)
  exclude_regex: "^DEBUG.*"  # Regex pattern to exclude items from unpublishing

  skip:  # Skip unpublishing for specific environments
    test: true # Enable unpublishing in test environment
    prod: false # Enable unpublishing in prod environment

constants:  # Global constants to override (optional)
  DEFAULT_API_ROOT_URL: "https://msitapi.fabric.microsoft.com"

In addition the folder includes a parameter file (parameter.yml) that contains a few examples of working with parameterization.

In the root of my Git repository is the Python file that orchestrates the fabric-cicd deployments. Working with a central configuration file allows for the Python script to be leaner than usual. As you can see below.

from fabric_cicd import deploy_with_config,append_feature_flag

import argparse
append_feature_flag("enable_experimental_features")

parser = argparse.ArgumentParser(description='Process one variable.')
parser.add_argument('--Environment', type=str)
args = parser.parse_args()

# Deploy using a config file
deploy_with_config(
    config_file_path="./workspace/config.yml",
    environment=args.Environment
)

Note that I needed the enable experimental features flag to be enabled in order to work with experimental features in the config file.

Creating YAML Pipeline to orchestrate configuration-based deployments

Once in my YAML pipeline I specified the two variable groups that I had created. Plus, the fact that I did not want a trigger at this moment in time so that I can manually run the pipeline. In addition, I specified I wanted an Azure Pipelines Agent with the latest windows image.

variables:
- group: fabric-cicd-ns
- group: fabric-cicd-s

trigger: none

pool: 
  vmImage: 'windows-latest'

Afterwards, I configure two stages in my YAML pipeline. One to deploy to a test workspace and another to deploy to the production workspace that is connected to an Azure DevOps environment.

Stage to deploy to a test workspace in Microsoft Fabric with fabric-cicd

I first created a stage to deploy to the test workspace. Which contained a single job.

- stage: Test
  displayName: 'Deploy to Test'
  
  jobs:
  - job: 'DeployTest'
    displayName: 'Deploy To Test'

My first task in this job was to specify the Python version. Which is essential when working with fabric-cicd to avoid any errors about invalid syntax.

- task: UsePythonVersion@0
  displayName: 'Use Python 3.12'
  inputs:
    versionSpec: 3.12

Afterwards I installed the fabric-cicd library.

- task: PowerShell@2
  displayName: 'Install necessary libraries'
  inputs:
    targetType: 'inline'
    script: |
      python -m pip install --upgrade pip
      pip install fabric-cicd
    pwsh: true

Afterwards, I added the below PowerShell task to authenticate a service principal.

- task: PowerShell@2
  displayName: 'Authenticate as Service Principal'
  inputs:
    targetType: 'inline'
    script: |
      Install-Module -Name Az.Accounts -AllowClobber -Force

      $SecureStringPwd = ConvertTo-SecureString $(AZURE_CLIENT_SECRET) -AsPlainText -Force
      $pscredential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $(AZURE_CLIENT_ID), $SecureStringPwd
                        
      Connect-AzAccount -ServicePrincipal -Credential $pscredential -Tenant $(AZURE_TENANT_ID)

      $fabricToken = (Get-AzAccessToken -ResourceUrl $(resourceUrl)).Token
    pwsh: true

I decided to authenticate this way because it also works with those working with trial Microsoft 365 E5 tenants.

Another way is to create a service connection in your Azure DevOps project. As shown in this post by Richard Mintz. However, you require a subscription ID to create a service connection. Which is not available with trial tenants.

Final task created in the stage to deploy to test is a Python script task to call the Python script that orchestrates the fabric-cicd deployments. Because most of the deployment options are covered by the central configuration file this task only requires one argument.

      - task: PythonScript@0
        displayName: 'Run script to deploy with fabric-cicd to Test'
        inputs:
          scriptPath: 'auth_spn_secret_config.py'
          arguments: '--Environment "test"'

Only difference for the production stage was to specify the different environment.

      - task: PythonScript@0
        displayName: 'Run script to deploy with fabric-cicd to production'
        inputs:
          scriptPath: 'auth_spn_secret_config.py'
          arguments: '--Environment "prod"'

Test results to operationalize configuration-based deployments

When I ran the pipeline the two stages had completed like in the image shown at the beginning of this post.

When I checked the two Fabric workspaces, I saw that all the items had deployed into them. I then checked the deployment log for test to confirm that the unpublish operation was skipped for test.

[info]   07:08:57 - Skipping unpublish operation for environment 'test'

Afterwards, I checked the logs for production to confirm that it attempted to unpublish items. Proving that the configuration file works as expected.

[info]   07:11:58 - ####################################################################################################
[info]   07:11:58 - ########## Unpublishing Orphaned Items #############################################################
[info]   07:11:58 - ####################################################################################################

Tips to orchestrate configuration-based deployments in fabric-cicd optimally

Below are some general tips to orchestrate configuration-based deployments with fabric-cicd optimally.

Implement the recommended development process by Microsoft first for your dev environment.
Create a service connection in Azure DevOps that authenticates as a service principal.
Test with your own development items configured with Microsoft Fabric Git integration.
Experiment with the various configuration options to suit your own strategy. Plus, alternative methods to pass through the parameters.

Final words

I think it is fitting to publish this post this just after Microsoft announced official support for Microsoft fabric-cicd tool. because it highlights just how far the fabric-cicd library has come in the last year. Thanks to efforts by both Microsoft and members of the community.

I know a lot of people prefer a central configuration option. Feel free to consume the contents of this post and customize it to suit your needs.