Automate FUAM deployments and updates

by

This post shows how you can automate FUAM deployments and updates with Azure DevOps. Including the creation of service connections, download of relevant items and relevant notebooks run.

Automated FUAM update performed in Azure DevOps
Automated FUAM update performed in Azure DevOps

For those yet to discover FUAM, FUAM stands for Fabric Unified Admin Monitoring and is a popular monitoring solution developed by two Microsoft employees. You deploy FUAM in a Fabric workspace. So that it can extract metrics and provide a holistic monitoring view.

FUAM has had some interesting updates recently. Including the ability to view shared objects. So it is definitely worth checking out.

About this method to automate FUAM deployments and updates

I previously created a FUAM deployment accelerator. Which allowed you to deploy FUAM automatically. That approach worked, but because I built it with fabric-cicd,updating it would require significant effort.

With this in mind, I came up with this method as an alternative. Which allows you to deploy with the latest version by downloading the latest version of the deployment notebook. Which then downloads and configures the latest version available.

I also added in the logic to automatically deploy the updates as well in a similar fashion. Doing this allows you to orchestrate the deployment and updates of FUAM to one or more Microsoft Fabric tenants with ease.

I decided to implement this with a YAML Pipeline in Azure DevOps for one very good reason. At a recent event the majority of attendees were working with Azure DevOps. To manage expectations, this method takes longer than the previous method due to the tasks performed.

You can look to adopt this method for other offerings that deploy via notebook downloads as well. For example, the new deployment method for Fabric Accelerator.

Below is a breakdown of the three sections that follow in this post:

  • Where to find the accompanying example and what the prerequisites are.
  • Working with the YAML Pipeline
  • For those interested, breakdown of the YAML pipeline

Automate FUAM deployments and updates example

You can find an online example of the YAML Pipeline covered in this post in my ADO-fuam-deployments-and-updates repository. Which you can import directly into your Azure DevOps project. From there, you can create a new YAML Pipeline based on the existing repository.

After setting up the following prerequisites you can import this into your Azure DevOps project and enter the parameters required.

All the prerequisites covered in the FUAM deployment guide still apply. Plus, the below Tenant admin settings need to be enabled. I recommend adding the service principal to a Microsoft Entra group and enabling settings for the Entra group.

  • Service principals can call Fabric can create workspaces, connections, and deployment pipelines. – Required for the workflow to complete.
  • Service principals can call Fabric public APIs. – Also required for the workflow to complete.
  • Service principals can access read-only APIs – Required for data Pipelines to work after deployment.

You must also create a variable group in Azure DevOps and add the below variables. I recommend calling the variable group fuam-ns.

  • pbi_connection_name- Name of Power BI connection created by this method.
  • pbibaseUrl – Base URL added to Power BI Connection.
  • pbiaudience – Audience added to Power BI Connection.
  • fabric_connection_name – Name of Power BI connection created by this method.
  • fabricbaseUrl – Base URL added to Fabric Connection.
  • fabricaudience – Audience added to Fabric Connection.

When starting your YAML pipeline you must enter the below parameters.

To start FUAM deployments or updates

To start the FUAM deployment or updates you must start the Pipeline in Azure DevOps. You do this by going to the pipeline in Azure DevOps. You must then click on the “Run pipeline” button and enter the requested parameters as below.

Parameters to automate FUAM deployments and updates in Azure DevOps
Parameters to automate FUAM deployment and updates in Azure DevOps

Note that when you deselect “New installation” the method updates the specified workspace. You must read the important notes in the FUAM update guide when you intend to do this.

New installation of FUAM selected

When you select new installation the below occurs:

  1. If the connections do not exist, they are created in Microsoft Fabric.
  2. Workspace is created.
  3. Notebook to deploy FUAM is downloaded.
  4. Downloaded notebook is imported alongside notebook for post deployment tasks.
  5. Deployment notebook is started.
  6. Post deployment notebook is started.

Once completed you can continue from step 5 of the FUAM deployment guide.

Update of FUAM selected

When you deselect new installation, the method performs an update and the following occurs:

  1. Connections are created in Microsoft Fabric if they do not exist.
  2. Existing notebook to deploy FUAM is deleted in the workspace specified.
  3. Latest notebook to deploy FUAM is downloaded.
  4. Downloaded notebook is imported alongside notebook for post deployment tasks.
  5. Deployment notebook is started.
  6. Post deployment notebook is started.

Once completed you can continue from step 3 of the FUAM update guide.

Manual tasks to perform after either method

After either of the above methods has completed there is one very important manual task you must do before you run the orchestration pipeline.

Which is that you must make a change to the Load_Capacity_Metrics_E2E Data Pipeline and save your change before you run the Load_FUAM_Data_E2E Data Pipeline. Otherwise this pipeline will fail due to permissions context.

I recommend copying the name of one of the notebooks into the description for the activity as below.

Making a manual change in a pipeline

In addition, you need to do this with the Load_Inventory_E2E as well if you are not using the Key Vault Parameters for the orchestration pipeline.

Even though this workaround is unorthodox it works.

Breakdown of method to automate FUAM deployment and updates

This section is for those who want to know more about the YAML pipeline itself.

At the start of the YAML pipeline the parameters shown previously are specified. It then specifies the variable group to work with which contains non-sensitive values. Plus, it sets trigger to none so you start the YAML pipeline manually and specifies a Microsoft-hosted Agent Pool.

variables:
- group: fuam-ns

trigger: none

pool: 
   vmImage: 'ubuntu-latest'

Afterwards, three separate stages take place as follows.

Stage to create the connections

First step in the job to create the connections is to specify the version of Python to work with. Via a use Python version task.

- task: UsePythonVersion@0
  displayName: 'Use Python 3.12'
  inputs:
    versionSpec: 3.12

Afterwards, a PowerShell task installs the ms-fabric-cli library in order to work with the Fabric Command Line Interface (Fabric CLI).

- task: PowerShell@2
  displayName: 'Install necessary libraries'
  inputs:
    targetType: 'inline'
    script: |
      python -m pip install --upgrade pip
      pip install ms-fabric-cli
    pwsh: true

Once done, another PowerShell task states two fab commands. One to set “command_fallback_enabled” to true and another to login with the Fabric CLI as a service principal.

fab config set encryption_fallback_enabled true
fab auth login -u ${{parameters.azure_client_id}} -p ${{parameters.azure_client_secret}} --tenant ${{parameters.azure_tenant_id}}

After the login another PowerShell task adds the “fuam pbi-service-api admin” connection in Microsoft Fabric if the connection does not exist.

$pbiconnections = fab ls .connections | Select-String "$(pbi_connection_name)"
$pbiconnections

if ($pbiconnections) {
    Write-Host "✅ Connection $(pbi_connection_name) already exists."
} else {
    Write-Host "Creating Connection $(pbi_connection_name)."
    fab create ".connections/$(pbi_connection_name).connection" -P "connectionDetails.type=WebForPipeline,connectionDetails.creationMethod=WebForPipeline.Contents,connectionDetails.parameters.baseUrl=$(pbibaseUrl),connectionDetails.parameters.audience=$(pbiaudience),credentialDetails.type=ServicePrincipal,credentialDetails.tenantId=${{ parameters.azure_tenant_id }},credentialDetails.servicePrincipalClientId=${{ parameters.azure_client_id }},credentialDetails.servicePrincipalSecret=${{ parameters.azure_client_secret }}"
}

When specifying the fab create command in a YAML Pipeline you must put double quotes around the connection name. Afterwards, another task uses the same logic to add the “fuam fabric-service-api admin” connection.

Adding the connections like this fine. However, because the service principal created them you will not be able to see them yourself in Microsoft Fabric. Even if you are a Fabric admin.

To resolve this issue, a separate task assigns permissions to the Power BI connection for the Entra user stated in the parameters with the below code.

$permissions = fab acl get .connections/$(pbi_connection_name).Connection -q [*].principal.id | Select-String ${{parameters.EntraObjectId}}

if ($permissions) {
    Write-Host "✅ Permissions for ${{parameters.EntraObjectId}} already exist on the Power BI connection."
} else {
    Write-Host "Adding permissions for ${{parameters.EntraObjectId}} to the Power BI connection."

    $pbiconnectionid = fab get .connections/$(pbi_connection_name).connection -q id

    $body = @{
    principal = @{
        id = "${{parameters.EntraObjectId}}"
        type = "User"  
    }
    role = "Owner"
    } | ConvertTo-Json -Compress
    
    # Create a temp file path and rite with UTF-8 WITHOUT BOM
    $tempFile = [System.IO.Path]::GetTempFileName() + ".json"
    $utf8Encoding = New-Object System.Text.UTF8Encoding $false
    [System.IO.File]::WriteAllText($tempFile, $body, $utf8Encoding)

    fab api -X post "connections/$pbiconnectionid/roleAssignments" -H "Content-Type=application/json" -i $tempFile
  }

As you can see above, you can call the role assignment API directly with the fab api command. Which saves creating a more complex API statement.

In the final task, the pipeline repeats the code above to add permissions to the Fabric connection.

Stage to manage workspace

Second stage of the YAML pipeline manages the workspace depending on if new deployment or an update. Which will only start after the stage to create the connections has completed thanks to the dependsOn syntax below.

- stage: WorkspaceManagement
  displayName: 'Manage Workspace'
  dependsOn: CreateConnections

First three tasks in the stage specifies the Python version, adds the ms-fabric-cli Python library and logs into Fabric CLI again.

Afterwards a PowerShell task creates a new workspace and connects it to the capacity specified if the new installation parameter was specified. Specifying the workspace name dynamically as part of the task display name.

- task: PowerShell@2
  displayName: 'Create the ${{parameters.WorkspaceName}} workspace with ms-fabric-cli'
  condition: eq('${{ parameters.new_installation }}', true)
  inputs:
    targetType: 'inline'
    script: |
      fab create ${{parameters.WorkspaceName}}.Workspace -P capacityname=${{parameters.CapacityName}}
    pwsh: true

If you don’t select a new installation as a parameter, the task deletes the existing notebook used to deploy FUAM.

# Remove the exiting deploy_fuam notebook if this is not a new installation to ensure the latest version gets imported and run later in the pipeline
- task: PowerShell@2
  displayName: 'Remove the exiting deploy_fuam notebook if this is not a new installation'
  condition: eq('${{ parameters.new_installation }}', false)
  inputs:
    targetType: 'inline'
    script: |
      fab rm ${{parameters.WorkspaceName}}.Workspace/Deploy_FUAM.Notebook -f
    pwsh: true

Another PowerShell task then adds the Entra user specified in the parameters as an admin of the workspace.

fab acl set ${{parameters.WorkspaceName}}.Workspace -I ${{parameters.EntraObjectId}} -R admin -f

Stage to import and run notebooks

Final stage imports the downloaded notebook to deploy FUAM. Along with the notebook to perform post deployment tasks which is already in the repository.

First PowerShell task creates a new subfolder for the notebook. It then downloads the latest version of the “Deploy_FUAM” notebook from the GitHub repository into a subfolder. The destination must be a notebook subfolder so the import can run.

# Create new subdirectory
$newdir = "Deploy_FUAM.Notebook"
md $newdir

# Download notebook into new directory
$notebookDir = Join-Path $(Build.SourcesDirectory) $newdir
$outputFile = Join-Path $notebookDir "Deploy_FUAM.ipynb"
Invoke-WebRequest -Uri "$(DeployURL)" -Headers @{ Accept = "application/octet-stream" } -OutFile $outputFile

Note: You can create the subdirectory in advance. However, if the directory is populated with a file you can encounter an issue when you look to import it.

Once the download has completed two other PowerShell tasks take place to modify the notebook slightly. To cater for issues caused after a recent service update. First PowerShell task adds an additional line to the first set of imports in the notebook. In order to add the notebook utilities library.

$notebookDir = Join-Path $(Build.SourcesDirectory) "Deploy_FUAM.Notebook"
$filePath = Join-Path $notebookDir "Deploy_FUAM.ipynb"
$notebook = Get-Content $filePath -Raw | ConvertFrom-Json

foreach ($cell in $notebook.cells) {
    if ($cell.cell_type -eq "code" -and $cell.source.Count -gt 0) {
        $firstLine = $cell.source[0]
        if ($firstLine -like "*import*" -and $firstLine -notlike "*from notebookutils*") {
            $cell.source = @("from notebookutils import notebook`n") + $cell.source
            break
        }
    }
}

$notebook | ConvertTo-Json -Depth 100 | Set-Content $filePath

Second task that modifies the notebook replaces the cell that starts with the %%configure syntax midway. Since it causes the notebook to fail when automated. Instead, it replaces it with a command to exit the notebook gracefully.

$notebookDir = Join-Path $(Build.SourcesDirectory) "Deploy_FUAM.Notebook"
$filePath = Join-Path $notebookDir "Deploy_FUAM.ipynb"
$notebook = Get-Content $filePath -Raw | ConvertFrom-Json

foreach ($cell in $notebook.cells) {
    if ($cell.cell_type -eq "code" -and $cell.source.Count -gt 0) {
        $firstLine = $cell.source[0]
        if ($firstLine -like "*%%configure -f*") {
            $cell.source = @("notebookutils.notebook.exit(`"Gracefully exiting`")")
        }
    }
}

$notebook | ConvertTo-Json -Depth 100 | Set-Content $filePath

After the notebook modifications a PowerShell task installs the Fabric CLI library and authenticate as the service principal. Afterwards, another PowerShell task imports both the downloaded notebook to deploy FUAM and the post deployment notebook.

$env:PYTHONIOENCODING = "utf-8"
fab import -f /${{parameters.WorkspaceName}}.Workspace/Deploy_FUAM.Notebook -i Deploy_FUAM.Notebook
fab import -f /${{parameters.WorkspaceName}}.Workspace/Deploy_FUAM_post_deployment.Notebook -i Deploy_FUAM_post_deployment.Notebook

Afterwards, another PowerShell task starts the deployment notebook in the new workspace. Which downloads and configures items as required.

$env:PYTHONIOENCODING = "utf-8"
# Run the post-deployment notebook which contains all tasks up until refreshing SQL Endpoint for Config_Lakehouse
fab job run /${{parameters.WorkspaceName}}.Workspace/Deploy_FUAM.Notebook  -P _inlineInstallationEnabled:bool=true

Finally, another PowerShell task performs the post deployment activities.

$env:PYTHONIOENCODING = "utf-8"
# Run the post-deployment notebook which contains all tasks up until refreshing SQL Endpoint for Config_Lakehouse
fab job run /${{parameters.WorkspaceName}}.Workspace/Deploy_FUAM_post_deployment.Notebook  -P _inlineInstallationEnabled:bool=true

Final words about method to Automate FUAM deployments and updates

As I mentioned earlier you can adapt this method to deploy other offerings. In addition, you can customize it to suit your needs.

I am sharing this method to automate FUAM deployments and updates to enable others to be more efficient with FUAM. Plus, encourage others to look at FUAM. Because I truly believe it can be a great addition to your governance story.

2 thoughts on “Automate FUAM deployments and updates”

  1. Pingback: Updating Microsoft Fabric Notebook Cells at Build Time with Azure Pipelines - Chantified Lens
  2. Pingback: Testing fab deploy with the new Fabric automation tools Azure DevOps extension - Chantified Lens

Leave a Comment