At Pretius, we generate huge amounts of code each month. We emphasize quality, but we also know that time is money (both for us and for our clients). We employ various code quality tools in our work, but we’re always looking for new, cutting-edge technologies. This is precisely how the possibility of using AI for code review caught our attention.

We’ve created an OpenAI Reviewer – a custom OpenAI task (an Azure DevOps plugin) which you can connect to the CI/CD pipeline to automatically review your code. You can get it for free at the Visual Studio Marketplace.

Why we created it

We had several reasons to create this tool. Some are more internal – we have a natural curiosity about new technologies. However, there were also some tangible business benefits we expected to get:

• We wanted to familiarize ourselves with AI – we started with a tool to review recruitment tasks, and code review followed
• We wanted to try it out internally to optimize the time our developers spend on code tasks such as code review
• We wanted to implement it in one of our projects (and probably will)

OpenAI Reviewer – How it works, and how it’s built

First, we’ll describe how our OpenAI Reviewer tool works. If you’re interested, you can also find the syntax below.

Task OpenAI Reviewer only works if the pipeline is launched during a Pull Request. Otherwise, the task is marked as Skipped with an appropriate message. Task collects all changes from a given Pull Request. Then, for each collected change (git diff), it prepares a code review by calling the OpenAI API. The received responses from the OpenAI API are added as comments in the Pull Request.

Considering that synchronous calls to the OpenAI API may take a long time (depending on the chosen OpenAI model and the number of changes in a Pull Request), a proper mechanism was implemented to control the token usage.

Pretius OpenAI Reviewer – Syntax

If you’re wondering how the OpenAI Reviewer’s syntax looks, you can find all the details below.

# Code review on Pull Request using OpenAI API
- task: openai-reviewer@0
  inputs:
    openaiApiKey: '$(openai-api-key)' # string. Required. OpenAI API Key.
   # Advanced
    #openaiApiUrl: 'https://api.openai.com/v1' # string. Optional. OpenAI API Url. Default: https://api.openai.com/v1.
    #openaiApiTimeout: 60000 # int. Optional. OpenAI API Timeout (ms). Default: 60000.
    #openaiApiMaxRetries: 1 # int. Optional. OpenAI API Retries. Default: 1.
    #openaiModel: 'gpt-3.5-turbo' # 'gpt-3.5-turbo' | 'gpt-3.5-turbo-16k' | 'gpt-4' | 'gpt-4-32k'. Optional. OpenAI Model. Default: gpt-3.5-turbo.
    #openaiTemperature: 0 # int. Optional. OpenAI Temperature. Default: 0.
    #openaiTopP: # int. Optional. OpenAI Top P. Default: None.
    #openaiPresencePenalty: 0 # int. Optional. OpenAI Presence Penalty. Default: 0.
    #openaiFrequencyPenalty: 0 # int. Optional. OpenAI Frequence Penalty. Default: 0.
    #openaiMaxTokens: 200 # int. Optional. OpenAI Max Tokens. Default: 200.
    #openaiN: 1 # int. Optional. OpenAI API N. Default: 1.
  # Repository
    #repositorySkipBinaryFiles: true # boolean. Optional. Skip Binary Files. Default: true.  
    #repositoryExcludedFiles: # multiLine. Optional. Excluded Files. Default: None.
  # Code review
    #codeReviewIgnorePullRequestIterations: false # boolean. Optional. Ignore PR iterations. Default: false.   
    #codeReviewSystemPrompt: # multiLine. Optional. Override System Prompt. Default: Fixed prompt.
    #codeReviewUserPrompt: # multiLine. Optional. Override User Prompt. Default: Fixed prompt.

Installation

The tool’s installation is quite simple. Follow these instructions.

1. Go to the OpenAI Reviewer extension page in the Visual Studio Marketplace and install it in your organization. It’s worth noting that the current version of our plugin offers expanded model options with GPT-3.5 Turbo and GPT-4 Turbo (gpt-4-1106-preview).

2. Grant the ‘Contribute to pull requests’ permission to the agent launching the pipelines at the level of the selected project in Azure DevOps. In Project settings → Repositories → Security → Your build service user, set the Contribute to pull requests option to Allow. This will allow you to add comments to the Pull Request.

3. From now on, you will have a new task available called OpenAI Reviewer.

Configuration options

After you’ve installed OpenAI Reviewer, you’ll also need to do some basic configuration. Here are some details you’ll need to get things done (these descriptions are usually quotes from the OpenAI documentation).

openaiApiKey – OpenAI API Key 

“The OpenAI API key is a secret token that authorizes access to OpenAI’s services. You can obtain it by signing up for an API account on the OpenAI platform”.

openaiApiUrl – OpenAI API Url

“The URL of the OpenAI API”. 

openaiApiTimeout – OpenAI API Timeout (ms)

“The maximum amount of time (in milliseconds) that the client should wait for a response from the server before timing out a single request”.

openaiApiMaxRetries – OpenAI Max Retries

“The maximum number of times that the client will retry a request in case of a temporary failure, like a network error or a 5XX error from the server”.

openaiModel – OpenAI Model

“ID of the model to use. See the model endpoint compatibility table for details on which models work with the Chat API”.

openaiTemperature – OpenAI Temparature

“What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. We generally recommend altering this or top_p but not both”.

openaiTopP – OpenAI Top P

“An alternative to sampling with temperature – nucleus sampling – where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered. We generally recommend altering this or temperature but not both”.

openaiPresencePenalty – OpenAI Presence Penalty

“Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far, increasing the model’s likelihood of talking about new topics. See more information about frequency and presence penalties”.

openaiFrequencyPenalty – OpenAI Frequence Penalty

“Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model’s likelihood of repeating the same line verbatim. See more information about frequency and presence penalties”.

openaiMaxTokens – OpenAI Max Tokens

“The maximum number of tokens to generate in the chat completion. The total length of input tokens and generated tokens is limited by the model’s context length”.

openaiN – OpenAI N

“Specifies how many chat completion choices to generate for each input message”.

repositorySkipBinaryFiles – Skip Binary Files

“Enable this option if you want to skip binary files (e.g. *.png, *.jar)”.

repositoryExcludedFiles – Excluded Files

“You can specify file patterns to exclude from code review, one per line, e.g.:

• *.txt
• test.json
• */exclude/path/*”

codeReviewIgnorePullRequestIterations – Ignore PR iterations

“Enable this option if you want to skip the code review for new iterations in a Pull Request”.

codeReviewSystemPrompt – Override System Prompt

“This field enables you to customize the prompt sent to OpenAI for the ‘system’ role. You can include placeholders in your prompt:

• {{GIT_DIFF}} – will be replaced with the Git diff
• {{CURRENT_COMMENTS}} – will be replaced with current comments for file in Pull Request
• {{MAX_TOKENS}} – will be replaced with current max tokens configured to OpenAI

You can also utilize EMPTY_CODE_REVIEW, which, if present in the response from the OpenAI API, will be omitted when adding code review comments”.

Default prompt:

Act as code reviewer of a Pull Request in a software development project, providing feedback on possible bugs and critical issues.

You are provided with Pull Request changes in standard git diff (unified diff) format.

As a code reviewer, your task is:
- Review only lines that have been added, edited, or deleted.
- Don't review the code that hasn't been modified.
- Don't repeat your reviews if the code line is correct.
- Ignore the lack of new line in the end of all of the files.
- If you have multiple comments, use bullet points for clarity.
- You are encouraged to use Markdown for your response to format your feedback effectively.
- If there are no bugs or critical issues, write only 'EMPTY_CODE_REVIEW' as a code review.
- If there are bugs or critical issues, don't write 'EMPTY_CODE_REVIEW'.

codeReviewUserPrompt – Override User Prompt

“This field enables you to customize the prompt sent to OpenAI for the ‘user’ role. You can include placeholders in your prompt:

1. {{GIT_DIFF}} – will be replace with git diff
2. {{CURRENT_COMMENTS}} – will be replaced with current comments for file in Pull Request
3. {{MAX_TOKENS}} – will be replaced with current max tokens configured to OpenAI”

Default prompt:

Git diff: """
{{GIT_DIFF}}
"""

OpenAI Reviewer – How to use

Using the OpenAI Reviewer tool is a relatively straightforward process. This is the recommended method, but others are possible. 

Adding OpenAPI Key in the Library

In the first step, you’ll configure a new group of variables.

1. In Azure DevOps, go to Pipelines → Library and create a new variable group via + Variable group.

2. Complete the name with open-ai, and then in the Variables section, add a new variable with the name openai-api-key, and paste your OpenAI API key as the value. Additionally, you must select the Change variable type to secret option.

3. Click the Save button

Adding the task to the new pipeline

The next step is to create a sample pipeline that will perform code review during the Pull Request.

1. In Azure DevOps, go to Pipelines → Pipelines and select the New pipeline option.

2. Select the Azure Repost Git option.

3. Indicate the repository in which the new pipeline will be created. In our example, it will be a repository called test.

4. Select the Starter pipeline option.

5. Replace the entire content of the default generated pipeline with the following:

trigger: none

pool:
  vmImage: "ubuntu-latest"

variables:
  - name: System.Debug
    value: false # enable for more info
  - group: open-ai

steps:
  - checkout: self
    persistCredentials: true

  - task: openai-reviewer@0
    displayName: 'Code review'
    inputs:
      openaiApiKey: '$(openai-api-key)'

6. Save by clicking Save and run → Save. In the new popup, select to save directly to the main branch.

Configuring the pipeline

In the last step, you must configure your pipeline to run whenever creating/modifying a Pull Request.

1. In Azure DevOps, go to Project settings → Repositories → Your repository name → Policies.

2. In the Branch Policies section, select the main branch.

3. In the Build Validation section, select the option to add a new policy.

4. In the new popup, indicate your repository in the Build pipeline field. The rest of the configuration can be left by default.

5. Save.

Finishing touches

From now on, each time a Pull request is created/modified, the pipeline will be automatically launched, and the OpenAI Reviewer task will add comments with the code review performed by OpenAI.

Additionally, when you first run the pipeline, due to the use of groups of variables, access to them must be accepted.

1. Enter the executed pipeline.

2. Select Permission needed, then click Permit in the popup and confirm access.

Conclusion – AI code review can be useful but it won’t replace developers

AI code review is a useful tool in the repertoire of software development teams. It has a visible, positive impact on code quality and requires minimal investment and effort to set up. However, it’s not perfect. It won’t allow you to automate the code review process and take it out of the developers’ hands – at least not yet. Here is a summary of the benefits and disadvantages or potential problems you might encounter.

In the end, the biggest test is still in front of us – we hope we’ll soon implement this tool in an actual project, which will result in plenty of additional feedback. We also plan to create an AWS plugin.  

We hope you’ll share your experience with OpenAI Reviewer with us and point out things to improve and change so that we can make it better. Reach out to us at azure-marketplace@pretius.com. We’ll be happy to hear from you!