Simplify CI/CD for Real-Time Intelligence

This post covers one way you can simplify CI/CD for Real-Time Intelligence in Microsoft Fabric with the fab deploy command in Azure DevOps.

I often get asked if CI/CD for Real-Time Intelligence works. Since it is not a topic that gets discussed often. So I decided to show CI/CD working for Real-Time Intelligence and how you can simplify deployments to other Fabric workspaces with the fab deploy command. Double value for post content.

I also show the new Fabric automation tools Azure DevOps extension in action. Due to the fact that I wanted an efficient way to issue two fab deploy commands. For reasons that I cover later in this post.

Along the way I share plenty of links for those looking to perform CI/CD for Real-Time Intelligence.

Plus, this post is accompanied by a sample repository that I published in GitHub. You can clone/download the ADO-rti-cicd-example repository and customize as needed. My only request is that you give this repository a star in GitHub.

You can find a brief overview about the new Fabric automation tools Azure DevOps extension, fab deploy and fabric-cicd in a previous post. Where I covered testing fab deploy with the new Fabric automation tools Azure DevOps extension.

Simplify CI/CD for Real-Time Intelligence

For this post I deployed all the Fabric items from the end-to-end tutorial by Microsoft that are currently supported by fabric-cicd. Since fabric-cicd is at the heart of this method.

I configured the initial workspace with Microsoft Fabric Git integration. So that Git repositories could serve as the source of truth for item metadata and deployments.

I initially tested deploying these items with just fabric-cicd. Since that is at the very core of the solution. I first tested to see what happened when I tried to install everything at once and got an error relating to dependencies.

To resolve the dependency issues I came up with a solution that involved two configuration files and a parameter file. You can find all of these in the workspace folder of my sample repository and I go into detail about them next.

Configuration files

When working with the fab deploy command you must perform fabric-cicd configuration-based deployments. In other words, you must point to a configuration file that contains various details about your deployment.

Because an Activator item had a dependency on a Real-Time Dashboard I needed to deploy everything else first and then the Activator item. In order to do this I needed a separate configuration file for the two deployments. As you can see below.

Configuration file for first deployment:

# Sample configuration file for fabric-cicd deployment
# This file demonstrates the YAML configuration structure for simplified deployment workflow

core: # Core configurations
  # Either workspace or workspace_id must be provided
  workspace: # Workspace names by environment
    Test: rti-cicd-test

  repository_directory: "."  # Path to workspace items directory (relative to config.yml location) (required)

  item_types_in_scope:  # Item types to include in deployment (optional)
    - Eventhouse
    - Eventstream
    - KQLDashboard
    - KQLDatabase
    - KQLQueryset
    - Lakehouse

  parameter: "parameter.yml" # Path to parameter file (relative to config.yml location) (optional)

constants:  # Global constants to override (optional)
  DEFAULT_API_ROOT_URL: "https://msitapi.fabric.microsoft.com"

Configuration file for second deployment:

# Sample configuration file for fabric-cicd deployment
# This file demonstrates the YAML configuration structure for simplified deployment workflow

core: # Core configurations
  # Either workspace or workspace_id must be provided
  workspace: # Workspace names by environment
    Test: rti-cicd-test


  repository_directory: "."  # Path to workspace items directory (relative to config.yml location) (required)

  item_types_in_scope:  # Item types to include in deployment (optional)
    - KQLDashboard
    - Reflex
  parameter: "parameter.yml" # Path to parameter file (relative to config.yml location) (optional)

constants:  # Global constants to override (optional)
  DEFAULT_API_ROOT_URL: "https://msitapi.fabric.microsoft.com"

I duplicated the KQLDashboard (Real-Time Dashboard) value in the second config file to ensure that the ID value for it was dynamically updated in the Activator item.

Parameter file

Since I needed to perform fabric-cicd parametrization to replace values during deployment I added a “parameter.yml” file. As you can see in the contents of my “parameter.yml” file below.

find_replace:
    # Replace workspace ID
    - find_value: "2991b27d-4872-479d-af7b-fd60b1916f31"
      replace_value:
       Test: "$workspace.$id"

    # Replace KQLDatabase ID entity ID according to URL for KQL Dashboard, can be found as value databaseArtifactId in RealTimeDashboard.json
    - find_value: "fbb01943-e3b9-43ff-a531-7c7ab018003b"
      replace_value:
       Test: "$items.KQLDatabase.Tutorial.$id"
      item_type: KQLDashboard

    # Replace KQLDashboard ID in order for Activator to point to right Dashboard source, can be found as dashboardId in ReflexEntities.json
    - find_value: "c098d55a-ceb3-4221-8c16-ac98d106a349"
      replace_value:
       Test: "$items.KQLDashboard.TutorialDashboard.$id"

    # Replace Lakehouse ID
    - find_value: "dd7c54ca-dc66-4d89-8cf8-5ea119a08786"
      replace_value:
       Test: "$items.Lakehouse.Tutorial.$id"

Some key points I need to highlight about the above parameter file:

• When working with a KQL Database added through an Eventhouse you must enter the value displayed in the source browsers URL as a source for KQL Dashboards. Do not enter the logical id value shown in Git.
• In addition to the above, I recommend filtering your find_replace value by item_type of KQLDashboard. Otherwise issues can occur.
• Dynamically changing the Eventhouse id value can cause issues with the KQL database deployment, which is why absent.
• When working with a KQL Database added through an Eventhouse dynamically changing the KQL Database id value to the original logic value shown in Git can cause issues.
• I did not need to change the Eventhouse Query Service URI for a local deployment. However, I added it to the sample file as suspect will need to change across tenants.

YAML pipeline to CI/CD for Real-Time Intelligence

After I put everything in place, I created the following YAML pipeline in Azure DevOps. Which utilizes the new Fabric automation tools Azure DevOps extension.

trigger: none

pool:
  vmImage: ubuntu-latest

# Change the variables to point to your variable group and variable name that contains the target environment for deployment
variables:
- group: fabric-cicd-s
- name: TargetEnvironment
  value: Test

stages:
- stage: deploy
  displayName: Deploy RTI items
  jobs:
  - job: fabdeploy
    displayName: Use fab deploy
    steps:

    - task: FabricCLI@0
      displayName: 'Use fab deploy'
      env:
        FAB_SPN_CLIENT_ID: $(Azure_Client_ID)
        FAB_TENANT_ID: $(Azure_Tenant_ID)
        FAB_SPN_CLIENT_SECRET: $(Azure_Client_Secret)
      inputs:
        scriptLanguage: 'bash'
        scriptType: 'inlineScript'
        inlineScript: |
          fab config set encryption_fallback_enabled true
          fab deploy --config "$(Build.SourcesDirectory)/workspace/config.yml" --target_env "$(TargetEnvironment)" -f
          fab deploy --config "$(Build.SourcesDirectory)/workspace/config_part2.yml" --target_env "$(TargetEnvironment)" -f
        FabricCLIVersion: 'v1.5.0'

I first ran the pipeline without EventStream enabled, so that I could test that the KQL queryset returned no results. In addition, I checked that the KQL Dashboard returned no results.

Plus, I went into Activator to confirm that the sources were correct.

Afterwards, I enabled Eventstream and started the pipeline again. Once completed I checked the database contained data. I then checked my KQL Dashboard, which this time looked very different.

Other items seemed okay as well. For example, events were flowing through Activator. So all looks well.

Epilogue

As you can see, CI/CD for Real-Time Intelligence is achievable as long as you consider complexities such as dependencies. Plus, you can also simplify CI/CD for Real-Time Intelligence by utilizing offerings such as fabric-cicd and the fab deploy command.

Another thing I like about this particular method is that it highlights the fact that you still need fabric-cicd knowledge when working with the fab deploy command.

Anyway, feel free to experiment for yourselves.