Leveraging OpenAI to Enhance Pull Request Management in Azure DevOps

Leveraging OpenAI to Enhance Pull Request Management in Azure DevOps

Comments 0 Comment

Objective

Our goal is to create an automation that significantly enhances the efficiency of our development process. A service hook is triggered when a new Pull Request is created in Azure DevOps, sending a webhook to an Azure Function. This function analyzes the request body, gets all commits in the Pull Request, and sends it to Open AI’s API, which generates a Pull Request message detailing all of the proposed changes from the commits and writes it back to the Azure DevOps Pull Request.

The diagram below gives a high-level overview of how automation works from start to finish.

OpenAI

Generate a new API Key

The first step is to obtain an OpenAI API key to interact with the API. To do so, go to the following website.

Once you have signed in, click + Create new secret key

Give it a descriptive name, and for permissions, in my testing, I had to give it All permissions. When I gave it read & write to assistants and threads, the command would work, but I would not get anything back from AI.

Click Copy and note the key for later

Azure Functions

Create a new Function App

Next, we must go to our Azure Portal and create a new Azure Subscription. I will create mine using the consumption plan.

Give your function a proper name, and select a Resource Group. For the runtime stack, select PowerShell Core; the version should be whatever PSCore version you prefer (currently, 7.4 is in preview, so I chose 7.2).

Finally, in the Review + Create pane, review your configuration and ensure no issues with your deployment. When finished, click Create.

Create a new PowerShell Function

Now that we have created our Function App, we must create a PowerShell function to perform the automation.

Go to your newly created Function App and click.

Click Next. Select an HTTP trigger function that will be run whenever it receives an HTTP request.

Give your Function a name, and for authorization level, select Function. Once completed, click Create.

Function PowerShell Code

The next thing we need to do is replace the default function code with our own. But before we do that, let’s step through the code and explain what it does. If you don’t care to learn the details of the code and just want to get it up and running, feel free to skip to the last part of this section.

Get-PRCommits

The first function is Get-PRCommits. This function will talk back to the Azure DevOps API and gather all the commits in the Pull Request. The Azure DevOps API authentication is a base64-encoded PAT. Don’t worry about this step; we will do this later.

function Get-PRCommits {
    Param (
        [Parameter(Mandatory)]
        [system.string]$AzDoPAT,
        [Parameter(Mandatory)]
        [system.string]$organization,
        [Parameter(Mandatory)]
        [system.string]$project,
        [Parameter(Mandatory)]
        [system.string]$repositoryId,
        [Parameter(Mandatory)]
        [system.int32]$pullRequestId
    )
    begin {
        $encodedPAT = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$($AzDoPAT)"))
        $url = "https://dev.azure.com/$organization/$project/_apis/git/repositories/$repositoryId/pullRequests/$pullRequestId/commits?api-version=7.1-preview.1"
    }
    process {
        try {
            $response = Invoke-RestMethod -Uri $url -Method Get -Headers @{Authorization = "Basic $encodedPAT"}
            return $response
        } catch {
            Write-Error "Failed to get pull request commits. Error: $_"
            throw
        }
    }
}

New-PRMessage

The next function is New-PRMessage. This function has a parameter of type system.object called $body. This will be the body response from the function above, where we get all the commits in the Pull Request. If the body contains the property “commentTruncated,” we know that the commit messages within the body are not the entire message, so we must make a GET request back to Azure DevOps to obtain the full message.

Once we have all the messages, we will create a prompt for Open AI. I am not a prompt engineer, so feel free to tweak this to fit your needs best. The prompt will ask AI to write a detailed pull request messages from the commit messages. It also passes the total count of commits there are, and I ask it not to include any title, greeting, signature, or closing message. The response is to be written in markdown. Lastly, the entire description should be under 4000 characters (the limit for Azure DevOps.)

After we get the description, we ask AI to create a title for the Pull Request it just created a description for, and keep it under 100 characters.

function New-PRMessage {
    Param (
        [Parameter(Mandatory)]
        [system.object]$Body,
        [Parameter(Mandatory)]
        [system.string]$AzDoPAT
    )
    begin {
        $encodedPAT = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$($AzDoPAT)"))
        $Comment = @()
        [system.int32]$count = $Body.count
        foreach ($item in $body) {
            If ($item.commentTruncated) {
                $item | where-object { $_.commentTruncated } | foreach-object {
                    $comment += (Invoke-RestMethod -uri $_.url -Method Get -Headers @{Authorization = "Basic $encodedPAT" }).comment
                }
            }
            else {
                Write-Host "Comment is not truncated. The comment is: $($item.comment)"
                $Comment += $body.comment
            }
        }
        
        $Prompt = "Please write me a detailed pull request message. My pull request includes the following commits: $comment. There are $count commits in total. Do not include a title, greeting, signature, or closing. Please format it using markdown. Have the message begin with either an overview or a summary of the changes. Keep the entire description under 4000 characters."
        Write-Host "Prompt is: $Prompt"
    }
    process {
        try {
            $Response = Invoke-OAIChat $Prompt
            $TitlePrompt = "Please write me a title for the following pull request: $Response. The title should be a concise summary of the changes. It should be no longer than 100 characters. Do not include the word PR or Pull Request in the title."
            $title = Invoke-OAIChat $titlePrompt
            
            $lines = $Response -split "`n"
            $startIndex = $lines | Select-String -Pattern "^#" | Select-Object -First 1 | ForEach-Object { $_.LineNumber - 1 }
            if ($null -ne $startIndex) {
                $filteredLines = $lines[$startIndex..($lines.Length - 1)]
                $response = $filteredLines -join "`n"
            }
            $out = [PSCustomObject]@{
                Title   = $title
                Message = $Response
            }
            return $out
        }
        catch {
            Write-Error "Failed to generate PR message from OpenAI. Error: $_"
            throw
        }
    }
}

Update-PullRequest

We need to update our Pull Request details and title now that AI has responded. The function Update-PullRequest does just that.

function Update-PullRequest {
    param (
        [string]$organization,
        [string]$project,
        [string]$repositoryId,
        [system.int32]$pullRequestId,
        [string]$description,
        [string]$personalAccessToken,
        [string]$title
    )
    begin {
        $encodedPAT = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$($personalAccessToken)"))
        $url = "https://dev.azure.com/$organization/$project/_apis/git/repositories/$repositoryId/pullRequests/$pullRequestId" + "?api-version=7.1-preview.1"
        write-host "URL: $url"
        $body = @{
            description = $description
            title       = $title
        } | ConvertTo-Json
    }
    process {
        try {
            $response = Invoke-RestMethod -Uri $url -Method Patch -Headers @{Authorization = "Basic $encodedPAT"; "Content-Type" = "application/json"} -Body $body
            return $response
        } catch {
            Write-Error "Failed to update pull request description. Error: $_"
            throw
        }
    }
}

Variables

The function has several variables:

  1. $personalAccessToken: This is the PAT for Azure DevOps and will be stored in an environment variable.
  2. $repository: The Repository ID is extracted from the body of the incoming web-hook from Azure DevOps.
  3. $pullRequestId: The Pull Request ID is extracted from the body of the incoming web-hook from Azure DevOps.
  4. $project: The Project ID is extracted from the body of the incoming web-hook from Azure DevOps.
  5. $organization: The Azure DevOps Organization that will be stored in an environment variable.

Note: You may be wondering where the variable for the OpenAI API Key is. We will go into this in more detail later, but it is stored in an environment variable called, “$env:OpenAIKey” The module we use is programmed to look for this environment variable, so as long as it’s present, we do not need to explicitly call it out.

Deploy Function Code

Now that we have reviewed how the code works let’s replace the default function code with our own.

Click on our newly created Function in our new Function App

Select all of the default Function code and delete it. Replace it with the code below:

using namespace System.Net
param($Request, $TriggerMetadata)

$personalAccessToken = $env:AzDoPAT

try {
    $requestbody = $Request.Body 

    $url = $requestBody.resource.repository.url
    $organization = $env:organization
    $repository = $requestBody.resource.repository.id
    $pullRequestId = $requestBody.resource.pullRequestId
    $project = $requestBody.resource.repository.project.id

}
catch {
    Write-Error "Failed to parse request body. Error: $_"
    Push-OutputBinding -Name Response -Value ([HttpResponseContext]@{
            StatusCode = [HttpStatusCode]::InternalServerError
            Body       = "Failed to parse request body."
        })
    return
}
function Get-PRCommits {
    Param (
        [Parameter(Mandatory)]
        [system.string]$AzDoPAT,
        [Parameter(Mandatory)]
        [system.string]$organization,
        [Parameter(Mandatory)]
        [system.string]$project,
        [Parameter(Mandatory)]
        [system.string]$repositoryId,
        [Parameter(Mandatory)]
        [system.int32]$pullRequestId
    )
    begin {
        $encodedPAT = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$($AzDoPAT)"))
        $url = "https://dev.azure.com/$organization/$project/_apis/git/repositories/$repositoryId/pullRequests/$pullRequestId/commits?api-version=7.1-preview.1"
    }
    process {
        try {
            $response = Invoke-RestMethod -Uri $url -Method Get -Headers @{Authorization = "Basic $encodedPAT" }
            return $response
        }
        catch {
            Write-Error "Failed to get pull request commits. Error: $_"
            throw
        }
    }
}
function New-PRMessage {
    Param (
        [Parameter(Mandatory)]
        [system.object]$Body,
        [Parameter(Mandatory)]
        [system.string]$AzDoPAT
    )
    begin {
        $encodedPAT = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$($AzDoPAT)"))
        $Comment = @()
        [system.int32]$count = $Body.count
        foreach ($item in $body) {
            If ($item.commentTruncated) {
                $item | where-object { $_.commentTruncated } | foreach-object {
                    $comment += (Invoke-RestMethod -uri $_.url -Method Get -Headers @{Authorization = "Basic $encodedPAT" }).comment
                }
            }
            else {
                Write-Host "Comment is not truncated. The comment is: $($item.comment)"
                $Comment += $body.comment
            }
        }
        
        $Prompt = "Please write me a detailed pull request message. My pull request includes the following commits: $comment. There are $count commits in total. Do not include a title, greeting, signature, or closing. Please format it using markdown. Have the message begin with either an overview or a summary of the changes. Keep the entire description under 4000 characters."
        Write-Host "Prompt is: $Prompt"
    }
    process {
        try {
            $Response = Invoke-OAIChat $Prompt
            $TitlePrompt = "Please write me a title for the following pull request: $Response. The title should be a concise summary of the changes. It should be no longer than 100 characters. Do not include the word PR or Pull Request in the title."
            $title = Invoke-OAIChat $titlePrompt
            
            $lines = $Response -split "`n"
            $startIndex = $lines | Select-String -Pattern "^#" | Select-Object -First 1 | ForEach-Object { $_.LineNumber - 1 }
            if ($null -ne $startIndex) {
                $filteredLines = $lines[$startIndex..($lines.Length - 1)]
                $response = $filteredLines -join "`n"
            }
            $out = [PSCustomObject]@{
                Title   = $title
                Message = $Response
            }
            return $out
        }
        catch {
            Write-Error "Failed to generate PR message from OpenAI. Error: $_"
            throw
        }
    }
}
function Update-PullRequest {
    param (
        [string]$organization,
        [string]$project,
        [string]$repositoryId,
        [system.int32]$pullRequestId,
        [string]$description,
        [string]$personalAccessToken,
        [string]$title
    )
    begin {
        $encodedPAT = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$($personalAccessToken)"))
        $url = "https://dev.azure.com/$organization/$project/_apis/git/repositories/$repositoryId/pullRequests/$pullRequestId" + "?api-version=7.1-preview.1"
        $body = @{
            description = $description
            title       = $title
        } | ConvertTo-Json
    }
    process {
        try {
            $response = Invoke-RestMethod -Uri $url -Method Patch -Headers @{Authorization = "Basic $encodedPAT"; "Content-Type" = "application/json" } -Body $body
            return $response
        }
        catch {
            Write-Error "Failed to update pull request description. Error: $_"
            throw
        }
    }
}

try {
    $Commits = Get-PRCommits -AzDoPAT $personalAccessToken -organization $organization -project $project -repositoryId $repository -pullRequestId $pullRequestId
}
catch {
    Push-OutputBinding -Name Response -Value ([HttpResponseContext]@{
            StatusCode = [HttpStatusCode]::InternalServerError
            Body       = "Failed to get pull request commits. Error: $_"
        })
    return
}

try {
    $PR = New-PRMessage -Body $Commits.value -AzDoPAT $personalAccessToken
}
catch {
    Push-OutputBinding -Name Response -Value ([HttpResponseContext]@{
            StatusCode = [HttpStatusCode]::InternalServerError
            Body       = "Failed to generate pull request message from OpenAI. Error: $_"
        })
    return
}

try {
    Update-PullRequest -organization $organization -project $project -repositoryId $repository -pullRequestId $pullRequestId -description $pr.message -title $pr.title -personalAccessToken $personalAccessToken
}
catch {
    Push-OutputBinding -Name Response -Value ([HttpResponseContext]@{
            StatusCode = [HttpStatusCode]::InternalServerError
            Body       = "Failed to update pull request description. Error: $_"
        })
    return
}

Push-OutputBinding -Name Response -Value ([HttpResponseContext]@{
        StatusCode = [HttpStatusCode]::OK
        Body       = "Pull request description and title updated successfully."
    })

Once you have finished, click Save

Environment Variables

Next, we need to store certain values as environment variables so they are not hard-coded into the function. Back in your Function App page, under Settings, click Environment Variables.

We need to create two environment variables. Refer to the table below:

Note: We have not yet gotten the personal access token (PAT) from Azure DevOps. If you are unfamiliar with the process, skip the variable for now. We will create it later.

Variable NameValue
AzDoPAT[The PAT from Azure DevOps]
OpenAIKey[The OpenAI Key we got earlier]
OrganizationYour Organization Name in Azure DevOps

Add the PSAI Module

Lastly, we must ensure that our function downloads the PSAI module when it runs. The PSAI module provides convenient access to the OpenAI APIs from the console and from PowerShell scripts. Documentation for the module can be found here.

Back in the Function App page, under Development Tools, click Advanced Tools.

Under Debug Console, click PowerShell.

Click the Site folder, then the wwwrootfolder, and lastly, click the pencil icon for the file requirements.psd1

In the file, add the following code:

# This file enables modules to be automatically managed by the Functions service.
# See https://aka.ms/functionsmanageddependency for additional information.
#
@{
    'PSAI' = '0.2.0'
    # For latest supported version, go to 'https://www.powershellgallery.com/packages/Az'. 
    # To use the Az module in your function app, please uncomment the line below.
    # 'Az' = '12.*'
}

Note: When writing this, 0.2.0 is the latest release.

Your file should look like the picture below. Press Save when finished.

Azure DevOps

Personal Access Token

Next, we must generate a new Personal Access Token (PAT) in Azure DevOps to interact with the API.

Click the User Settings icon in the top right and then click Personal Access Tokens.

Give your new PAT a name, select the organization to which it should have access, and indicate its expiration date. For Scope, it should have Read & Write permissions for Code.

It should also have Read & Write to Pull Request Threads.

Save the displayed personal access token.

Back in the Azure Function app, under Environment Variables, we can now add the AzDoPAT variable. The value is the PAT that we copied above.

Configuring the Service Hook

Now, we need to configure the service hook. The service hook will send a webhook to our newly created PowerShell Function whenever a new Pull Request is created.

Back in Azure DevOps, click the Project for which you want to configure the service hook-in, go to Project Settings, and then click Service hooks.

Click on the green plus icon to create a new service hook.

Next, click Web Hooks

For the trigger, select “Pull request created“

If you wanted to filter it down by repository, branch, etc. This is the step to do so.

Next, jump back to our PowerShell function. If you click it and go to Code + Test, you will see “get function URL“

Once you click it, copy the default URL with the function key.

Back in Azure DevOps, paste the key in the URL and press Finish.

Demonstrating the Workflow

Now that everything is in place let’s create a new Pull Request in one of our repositories. Below is a screenshot of my initial Pull Request; it lacks a real title and description.

After a few seconds, the Pull Request is automatically updated to the following:

Leave a Reply

Your email address will not be published. Required fields are marked *