Extend Azure DevOps with Azure Functions

With Azure DevOps, Microsoft provides a rich platform for building your continuous integration and continuous delivery (CI/CD) workflows. Refer to https://azure.microsoft.com/en-us/services/devops/

Our CI/CD design

Lately we came up with the following CI/CD design:

Notice the following features:

1. When a feature branch is created, an app service slot is created and the feature workload is deployed to this slot;
2. When a feature is merged with master, the slot gets removed and the master workload gets deployed to the production slot.

Learn more about Azure App Services and its deployment slot feature here: https://docs.microsoft.com/en-us/azure/app-service/deploy-staging-slots

Implementation

In order to learn how to set up the above workflow you can read the following excellent blog post of Lucas Caljé: https://levelup.gitconnected.com/manage-appservice-slots-with-azure-yaml-pipelines-3fb2a2e5da9a

Interestingly, Lucas' solution does not require any external integration. It solely uses Azure Pipelines and its trigger feature. One caveat though: it requires the developers to put in a specific commit message for the pipeline to clean up the deployment slot that is linked to the completed feature.

Let's build out his solution so that developers are not required anymore to include this commit message.

Extension

So we must extend Azure DevOps. We will be using a webhook that will call out to an Azure Function, that will kick off a pipeline, visualized in the following diagram:

GitHub Actions provide support for the branch deleted trigger; so no extension required: https://docs.github.com/en/free-pro-team@latest/actions/reference/events-that-trigger-workflows#delete-event-delete

We pick Azure Functions because of its consumption plan: you only pay for the usage and it scales automatically when demand increases.

Build the function app

Azure Functions is a versatile platform with support for many language runtime to create serverless workloads. For this blog post we will create a TypeScript function that will respond to an http call, i.e. http triggered. You can read more here:
https://docs.microsoft.com/en-us/azure/azure-functions/functions-create-first-function-vs-code?pivots=programming-language-typescript

Perform the following steps to lay out a basic function app:

1. Install Visual Studio Code (https://code.visualstudio.com/)
2. Install the Azure Functions Extension (https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-azurefunctions)
3. Create a new folder
4. Open the folder in Visual Studio Code
5. Open up the command palette (CTRL+SHIFT+P on Windows) and choose the Azure Functions: Create Function command
   a. The extension signals that there is no functions project. Confirm to create it;
   b. Choose TypeScript as the language;
   c. Choose Http trigger as the template;
   d. Choose KickOffAzurePipeline as the name;
   e. Choose Function as the authorization level to prevent unauthorized access to your function.

The extension will scaffold a working http trigger TypeScript function app for you. You can run it by issuing the following commands on the prompt:

1. npm install to install the two node modules that are specified in package.json;
2. npm start to start the local Azure Functions runtime to test the function app locally.

If all goes well, the Azure Functions runtime will state an endpoint which you can call out in the browser:

KickOffPipeline: [GET,POST] http://localhost:7071/api/KickOffPipeline

When you copy/paste this url in the browser you will get the following response:

This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response.

You can try out specifying the name parameter on the url:

http://localhost:7071/api/KickOffPipeline?name=Carl

And the app responds with:

Hello, Carl. This HTTP triggered function executed successfully.

Add Azure DevOps library

Now we will add functionality to the function that will interact with Azure DevOps. Microsoft provides a nice Node Modules package that we can include:

npm install azure-devops-node-api

More information can be found here: https://github.com/Microsoft/azure-devops-node-api
You will need to work with the samples as at this point in time the documentation is a bit limited.

Authentication

In order to authenticate against Azure DevOps you will need to create a Personal Access Token (PAT) with the appropriate authorization scope, i.e. the permission to kick off pipelines. Read the following for more information: https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate

The PAT has the following format: ackwlpvdpyreh4yx2za2f2azlg2uhxuuehltrzbkkbxhp52p3vhq

We can plug secrets into a Function App through the settings panel in the Portal. The Functions runtime will load the settings into environment variables. Did you know you can use Key Vault replacement tokens in settings to really store the secrets into Key Vault? You can read more about this here: https://docs.microsoft.com/en-us/azure/app-service/app-service-key-vault-references. You could also pull the secret at run-time within your function from Key Vault. One example is: https://github.com/Azure/azure-sdk-for-js/blob/master/sdk/keyvault/keyvault-secrets/samples/typescript/src/helloWorld.ts

In both mentioned Key Vault integration cases you need to authorize the Azure Function's Managed identity onto the Key Vault. In the future we expect to be able to authorize the Function App's Managed Identity directly to Azure DevOps; or more probably, GitHub 😉 In this way we can get rid of the PAT.

So we work with environment variables in our code. The function app runtime will pull these when run locally from the local file local.settings.json.

With all these pieces into place the following code will set up the connection:

const collectionURL = req.query["collectionURL"];
context.log(`collectionURL: ${collectionURL}`);
const token = process.env["PAT"];

let authHandler = azdo.getPersonalAccessTokenHandler(token);
let connection = new azdo.WebApi(collectionURL, authHandler);

The collectionURL is equal to "https://dev.azure.com/<your orgname>"

Next we need to call into the Azure DevOps APIs in order to kick off a build. For kicking off yaml or classic pipelines with input variables there is the following:

async function testQueueBuild(project: string, pipelineId: number, sourceBranch: string) {
  const azdoBuild = await connection.getBuildApi();
  const buildDefinition = await azdoBuild.getDefinition(project, pipelineId);
  const build : Build = {
    definition: buildDefinition,
    sourceBranch: sourceBranch,
    parameters: JSON.stringify({
      'variable1': 'value1',
      'variable': 'value2'
    }),        
    reason: BuildReason.IndividualCI 
  }
  let result = await azdoBuild.queueBuild(build, project);
  console.log(result);    
};

Although the key in the Build object is called parameters, the REST API will only process input variables. What about kicking off yaml pipelines using input parameters instead? If you inspect the REST call that the Azure DevOps Web App makes when you run a yaml pipeline, you will discover that the following end-point is called:

https://dev.azure.com/<your org>/<your project>/_apis/pipelines/<pipelineId>/runs

with a payload compared to:

{
  "stagesToSkip": [],
  "resources": {
    "repositories": {
      "self": {
        "refName": "refs/heads/master"
      }
    }
  },
  "templateParameters": {
    "slot": "123456",
    "pool": "ubuntu-latest"
  },
  "variables":{}
}

This REST API is documented at: https://docs.microsoft.com/en-us/rest/api/azure/devops/pipelines/runs/run pipeline?view=azure-devops-rest-6.0

Unfortunately, what is always the case with API libraries that they are lagging behind the actual API surface. At the time of writing, the library does not provide a straight method. This issue has been raised: https://github.com/microsoft/azure-devops-node-api/issues/392

To work around the limitation, the library provides means to build out your own custom REST call:

async function runPipeline(refName, orgUrl, project, pipelineId) {
  const azdoBuild = await connection.getBuildApi();
  const build = {
    resources: {
      repositories: {
        self: {
          refName: `refs/heads/${refName}`,
        },
      },
    },
    templateParameters
  };
  const url = `${orgUrl}/${project}/_apis/pipelines/${pipelineId}/runs`;
  const reqOpts = {
    acceptHeader: 'application/json;api-version=6.0',
  };
  return azdoBuild.rest.create(url, build, reqOpts);
}

Public end-point

With that, the first pieces of our Function App are done. Before we can create a webhook towards our function, we will need to deploy it to a public endpoint. For local development you can setup a reverse proxy such as Ngrok: https://ngrok.com/

Ultimately you would want to setup CI/CD to deploy your function app to Azure. For a quick deploy you can also use Visual Studio Code. The Azure Functions extension provides an easy deploy workflow. This is described in the mentioned tutorial: https://docs.microsoft.com/en-us/azure/azure-functions/functions-create-first-function-vs-code?pivots=programming-language-typescript.

We still need to extend our function to really ingest the webhook input and resolve the branch in order to get the related slot. We will do that after we have set up the webhook.

Register a webhook in Azure DevOps

Webhooks provide a way to send a JSON representation of an event to any service. All that is required is a public endpoint (HTTP or HTTPS). Read more at https://docs.microsoft.com/en-us/azure/devops/service-hooks/services/webhooks?view=azure-devops

As you can see in the screenshot, Web Hooks are one of the many Service Hook options that Azure DevOps provides.

For getting events on branch removals, pick the Code pushed event:

In order to understand the payload we are getting back from the web hook, you can use services such as https://requestbin.com/. The incoming request gets captured and you inspect the payload. You get the following when a branch gets created:

...
"refUpdates": [
  {
    "name": "refs/heads/test",
    "oldObjectId": "0000000000000000000000000000000000000000",
    "newObjectId": "54a8f23a21a6b3aa94e43c4ddf78ac64cc1309c9"
  }
]
...

And the following when a branch gets deleted:

...
"refUpdates": [
  {
    "oldObjectId": "54a8f23a21a6b3aa94e43c4ddf78ac64cc1309c9",
    "name": "refs/heads/test",
    "newObjectId": "0000000000000000000000000000000000000000"
  }
]
...

So our logic would be the following to pull out the relevant pieces:

const branchName = req.body.resource.refUpdates[0].name.substring(11);
const oldObjectId = req.body.resource.refUpdates[0].oldObjectId;
const newObjectId = req.body.resource.refUpdates[0].newObjectId;

if (newObjectId === '0000000000000000000000000000000000000000') {
  // kick off the pipeline
}

You can review my version of the complete function in the following repo:
https://github.com/cveld/kickoff-azdo-pipeline

You can first test out your function locally with a tools such as Postman: https://www.postman.com/

The local base url should be: http://localhost:7071/api/KickOffPipeline

Various parameters are read by our function from the query parameters:

• collectionURL: contains the base url to your Azure DevOps org; https://dev.azure.com/<your org>;
• project: the Azure DevOps project that contains the pipeline you would like to run;
• pipelineId: the id of the pipeline you would like to run;
• branch: which branch to take the yaml file from that is referenced by the pipeline. Normally you would go for master, but during development of your pipeline you could go for a feature branch.

So the complete url you will need to specify as a web hook endpoint looks like:
http://localhost:7071/api/KickOffPipeline?collectionURL=https://dev.azure.com/carlintveld&project=lucas-demo&pipelineId=19&branch=feature/test-parameter

Next you can work with a tool such as ngrok to get an Ngrok provided public endpoint that proxies to your local endpoint. With this setup you will get local logging and you can diagnose if the web hook is not processed as expected. Example:

ngrok http 7071

Starts up ngrok and will provide an endpoint such as https://257093d1c361.ngrok.io.

And finally, publish the finished function and get the public url for it. As we picked function-level authorization, don't forget to include the authorization key into the url.

The base url looks like:
https://slotsexperiment-functionapp-nodejs.azurewebsites.net/api/KickOffPipeline?code=Q9H2a788HoDMeaDdfWgr0N2VonXLjpO3ktPcbX9mnVg88vv7gNcWPA==

So the complete url you will need to specify as a web hook endpoint looks like:
https://slotsexperiment-functionapp-nodejs.azurewebsites.net/api/KickOffPipeline?code=Q9H2a788HoDMeaDdfWgr0N2VonXLjpO3ktPcbX9mnVg88vv7gNcWPA==&collectionURL=https://dev.azure.com/carlintveld&project=lucas-demo&pipelineId=19&branch=feature/test-parameter

With the following sample yaml pipeline:

# Learn more about Azure DevOps yaml pipelines through https://aka.ms/yaml
trigger: none

parameters:
- name: branchRemoved
  type: string

pool:
  vmImage: ubuntu-latest

steps:
- script: |
    echo parameters.branchRemoved: ${{parameters.branchRemoved}}

The result would look like the following:

You see in the screenshot that the branchRemoved parameter is printed out with the branch name that got removed.

The setup is ready to integrate with the solution that Lucas Caljé has laid out in his blog.

Happy Serverless September to you all!