Use Managed DevOps Pools for Microsoft Fabric deployments

In this post I show how you can use Managed DevOps Pools for Microsoft Fabric deployments. Which can be beneficial for a variety of reasons, including the ability to access secure Azure Key Vaults with firewall enabled. Like in the below diagram.

This post is based on my previous post where I showed how to perform CI/CD for Fabric apps with Azure DevOps. However, this setup can also provide access to secrets for a variety of other applications and services when working with Azure DevOps pipelines. Such as SQL Server, Azure App Service and Microsoft Foundry.

To clarify. Managed DevOps Pools introduces the capability to easily spin up Azure DevOps agent pools based on Microsoft-hosted images. Which you can customize in many ways. Including choosing virtual machine sizes and the ability to inject agents into your own Azure Virtual Networks.

You can manage all these settings in a variety of ways. Including through the Azure Portal as shown below.

As you can see, this can save a lot of time creating and maintaining agent pools. Plus, you can potentially reduce costs by understanding how Managed DevOps Pools are priced.

I decided to base this post on Microsoft Fabric to raise awareness about Managed Devops Pools in the Fabric community. However, if you do not work with Microsoft Fabric you can utilize the concepts for your own Azure DevOps needs. Along the way I share plenty of links.

Example of using Managed DevOps Pools for Microsoft Fabric deployments

I first needed to deploy the required Azure Resources. Including an Azure Key Vault, an Azure Virtual Network to deploy the agent pools into and the Managed DevOps Pool itself. You can deploy these resources either through the Azure Portal or with an Infrastructure as Code offering such as Bicep.

Once I had deployed my Azure Key Vault, I added the necessary secrets for an Entra service principal that I intended to authenticate with. Including the tenant and client ids, as well as the secret value. After adding the secrets I secured my Azure Key Vault by disabling public access and granted the relevant permissions to the service principal.

One key point I must highlight here is that you can select to allow trusted Microsoft services to bypass this firewall. However, Azure DevOps is not a trusted service through this setting.

Anyway, after installing and configuring these services I was in a position to deploy my Managed Devops Pool.

Deploying Managed DevOps Pool

Before deploying my Managed DevOps Pool I first had to register the Managed DevOps Pool resource provider.

Afterwards, I was ready to perform the deployment the Managed DevOps Pool itself. Microsoft provides multiple quick start guides on how to do this. Including how to create a Managed DevOps Pool using the Azure Portal.

You can follow along with one of the Microsoft guides to deploy within your own environment. However, below are some configuration options I suggest you consider for your own deployment.

• When working with your first custom pool consider the maximum number of agents to work with. I recommend only one or two to begin with.
• To reduce costs be pragmatic about the agent size you require. Balance between compute required for your tasks and cost. Be aware that this also depends on the availability of the virtual machines in your region.
• Experiment with different images to maintain costs.
• I strongly recommend you select to use a fresh agent every time instead of paying to keep an agent running all the time.

Another point I want to highlight is the fact that you can configure your Managed DevOps Pools to work with other Azure DevOps organizations. I tested and can confirm that you can even configure them to work with other organizations that are in other Entra tenants, as long as you have the right permissions.

Post-deployment tasks

After the deployment was completed a private endpoint from my existing Azure Key Vault to the virtual network needed to be created. I decided to create it manually in the Key Vault itself, selecting the target sub-resource of vault.

Afterwards, I created a service connection in Azure DevOps that connected to the Azure Resource Group that contained the Azure Key Vault.

Typically after this stage I add an Azure DevOps variable group that connects to the Azure Key Vault. However, because I was also experimenting with cross-tenant access I decided to do something different. Which I cover next.

Setting up my YAML pipeline to cater for Managed Devops Pools

I started by making a copy of the ‘deploy-app.yml‘ YAML pipeline that I shared in my previous post. Afterwards, I made the following changes to the new YAML pipeline.

First, I removed the reference to the secure variable group, leaving the one for non-sensitive values:

variables:
- group: rayfin-ns

To make the YAML pipeline more flexible I added a new variable to above variable group called agentpool and set the value to be the name of the newly deployed Managed DevOps Pool. I then specified that as the agent pool to work with in my YAML Pipeline.

pool: 
  name: $(agentpool)

As I mentioned, this time I did not link a variable group to Azure Key Vault. Instead I decided to work with the Azure Key Vault task in the YAML pipeline instead. For greater flexibility.

I added this task right before my authentication task. As you can see below.

- task: AzureKeyVault@2
  inputs:
    azureSubscription: 'FabricKV' -- Service Connection name
    KeyVaultName: '{MY KEYVAULT NAME}'
    SecretsFilter: '*'
    RunAsPreJob: true

- task: PowerShell@2
  displayName: 'Login to Rayfin with service principal'
  inputs:
    targetType: 'inline'
    script: |
      npx rayfin login --service-principal --encryption-fallback-enabled -t '$(AZURE-TENANT-ID)' -u '$(AZURE-CLIENT-ID)' -p '$(AZURE-CLIENT-SECRET)'

However, I did set RunAsPreJob to true. Which meant it would get the secrets before running the tasks in the job.

I then tested my YAML pipeline. Which completed successfully. For certainty, I checked the pipeline logs to confirm that the Managed DevOps Pool had been selected and that the Azure Key Vault secrets had been acquired properly.

For certainty, I deleted the items from my Fabric workspace each time I tested as well. Each time the items re-appeared as expected.

Epilogue

As you can see, using Managed DevOps Pools for Microsoft Fabric deployments is great for various scenarios such as accessing secure Azure Key Vaults.

I remember years ago Chris Gibson did a good series about Azure DevOps scale sets. Managed DevOps Pools are definitely an evolution of those.

Personally, I think they make creating and managing custom pools for scenarios like the one in this post a lot easier. I can tell you from experience working with Managed DevOps Pools is a lot easier than self-hosted agents.

Anyway, I hope the knowledge in this post has raised awareness about Managed DevOps Pools more and helps others in the community.