Updating Microsoft Fabric Notebook Cells at Build Time with Azure Pipelines

This post covers how I ended up updating Microsoft Fabric notebook cells at build time with Azure Pipelines. Which is the service that manages your pipelines in Azure DevOps.

I decided to share this to be transparent about what happened and how I resolved the issue. Plus, share how you can directly edit notebook cells whilst running pipelines in Azure DevOps.

This method works with notebooks from online sources or from Git repositories configured with Microsoft Fabric Git integration. As long as the file is ipynb format.

Last week I shared a post that covered how to automate FUAM deployments and updates. As part of that process it downloads and runs a notebook.

Up until last week the process was working as expected as you can see below. My pipeline would start the notebook to deploy FUAM and then stop running the notebook once it errored at the %%configure magic command, and then from there carry on with the rest of the pipeline in Azure DevOps.

However, just twenty-four hours after I published that post, an update somewhere caused the pipeline to stop working. Instead of continuing with the pipeline after the notebook errored at the cell that contained the %%configure command, the pipeline now stopped working completely at that point.

Because I had already published the post I had two options. Either pull the post or come up with a solution fast. I opted for the latter after realizing I had a swift solution for it.

Updating Microsoft Fabric Notebook Cells at Build Time with Azure Pipelines

As part of the existing deployment method, the pipeline downloads the notebook file onto the agent with the below code.

# Create new subdirectory
$newdir = "Deploy_FUAM.Notebook"
md $newdir

# Download notebook into new directory
$notebookDir = Join-Path $(Build.SourcesDirectory) $newdir
$outputFile = Join-Path $notebookDir "Deploy_FUAM.ipynb"
Invoke-WebRequest -Uri "$(DeployURL)" -Headers @{ Accept = "application/octet-stream" } -OutFile $outputFile

With this in mind, I decided to update the downloaded file directly. To replace the cell that contained the %%configure command during build time. Before importing the notebook into Microsoft Fabric.

After some research I discovered that a graceful way to stop a notebook running in Microsoft Fabric was to call the exit function that comes with NotebookUtils. Which meant that I had to first import NotebookUtils and then replace the cell that contains the %%configure syntax.

Because speed was of the essence, I got GitHub Copilot in Visual Studio Code to help me come up with the below code. Which adds a new first line to the first cell that contains the word “import”.

$notebookDir = Join-Path $(Build.SourcesDirectory) "Deploy_FUAM.Notebook"
$filePath = Join-Path $notebookDir "Deploy_FUAM.ipynb"
$notebook = Get-Content $filePath -Raw | ConvertFrom-Json

foreach ($cell in $notebook.cells) {
    if ($cell.cell_type -eq "code" -and $cell.source.Count -gt 0) {
        $firstLine = $cell.source[0]
        if ($firstLine -like "*import*" -and $firstLine -notlike "*from notebookutils*") {
            $cell.source = @("from notebookutils import notebook`n") + $cell.source
            break
        }
    }
}

$notebook | ConvertTo-Json -Depth 100 | Set-Content $filePath

I say helped because I had to make some changes to the code it had generated. Such as adding the “n” after the import statement myself so that it would start a new line for the existing import statements. To prevent it joining to the existing first line in the cell.

Afterwards, I got it to come up with the PowerShell to replace the cell that contained the %%configure command with a command to exit the notebook gracefully.

$notebookDir = Join-Path $(Build.SourcesDirectory) "Deploy_FUAM.Notebook"
$filePath = Join-Path $notebookDir "Deploy_FUAM.ipynb"
$notebook = Get-Content $filePath -Raw | ConvertFrom-Json

foreach ($cell in $notebook.cells) {
    if ($cell.cell_type -eq "code" -and $cell.source.Count -gt 0) {
        $firstLine = $cell.source[0]
        if ($firstLine -like "*%%configure -f*") {
            $cell.source = @("notebookutils.notebook.exit(`"Gracefully exiting`")")
        }
    }
}

$notebook | ConvertTo-Json -Depth 100 | Set-Content $filePath

Testing that Microsoft Fabric Notebook Cells were being updated

I then started the pipeline and patiently waited for it to complete. I say patiently because I had to wait for the pipeline to run in its entirety.

After it had completed, I went into the completed Pipeline in the Monitor hub and checked the snapshot of the completed notebook. To confirm that the pipeline replaced the cell value.

I was very relieved that it had done so. I then continued with the steps to deploy FUAM.

After a few iterations of deployments and updates I updated the post and my public ADO-fuam-deployments-and-updates repository. Plus, amplified the change on socials.

Final words

I hope this tale of how I ended up updating Microsoft Fabric notebook cells at build time with Azure Pipelines is insightful. I never did pinpoint the cause of the issue. Azure Devops had some updates deployed last week. However, they looked to be unrelated.

Whatever the cause, it encouraged me to come up with this nifty solution fast. Which meant others can still work with the existing method, or utilize the fix to update cells in their own pipelines. Either way, I’m glad I resolved it quickly and provided a working solution for everyone.