GitHub Copilot Extensions: Collecting Feedback in a VS Code Copilot Extension

In the first post of this series, we introduced GitHub Copilot Extensions and discussed the two types of extensions. Client side and server side extensions.

In our previous post on Using Context in a VS Code Extension, we enhanced our VS Code Copilot extension by adding support for references to the UI and how to use explicit references in the prompt.

Goal

This post demonstrates how to implement feedback collection in your VS Code Copilot extension. We'll cover:

1. Handling user feedback through Copilot's built-in thumbs up/down mechanism
2. Accessing feedback data in your extension

By the end of this tutorial, you'll understand how to collect, process, and utilize user feedback to enhance your extension's effectiveness. We'll continue building upon our pirate/Yoda parrot example to demonstrate these concepts in practice.

TIP

The complete source code for this tutorial series is available on GitHub. These posts will guide you through building this extension step-by-step, from initial setup to the final version you see in the repository.

This simple example extension acts like a chat parrot, repeating what you type in the chat window. You can optionally have it repeat your message in pirate or Yoda style.

Collecting User Feedback

Collecting user feedback is an important step for evaluating and improving the quality of your Copilot extension's responses. Feedback helps you understand how users perceive the extension's helpfulness, identify areas for improvement, and gauge the overall user experience. Whether it's through explicit ratings, free-form comments, or implicit usage patterns, gathering feedback provides valuable insights into what works well and what needs refinement.

Copilot provides a mechanism for users to provide feedback on the quality of the responses it provides. In the form of a thumbs up or thumbs down, right under the response.

When a user clicks on thumbs up, it means this is helpful and when it clicks on thumbs down it means it's not helpful and it has to provide the reason why the response was not helpful (see screenshot below).

Extensions can optionally use this mechanism to collect feedback of its own responses. Copilot only provides a way for the extension handler to know the user feedback, it's up to the extension to collect and process it any way it see fits (send it to it's own service, open it's own UI,etc).

Warning

The information that you collect (the prompt, the response, the user or something else) it is your responsibility to handle it in a legal and privacy compliant way. If you are collecting any information associated to a user or even in an anonymous way it is something that you should make it clear to your users.

Collecting feedback on an extension

When a user triggers the feedback, the extension handler receives a vscode.ChatResultFeedback parameter.

So let's add the handler, create a file (eg: feedbackhandler.ts) and add the following code:

import * as vscode from 'vscode';

export function handleFeedback(feedback: vscode.ChatResultFeedback) {
    console.log('Feedback received:', feedback);

    switch (feedback.kind) {
        case vscode.ChatResultFeedbackKind.Helpful:
            vscode.window.showInformationMessage('🚀 Happy that you liked it.');
            break;
        case vscode.ChatResultFeedbackKind.Unhelpful:
            vscode.window.showWarningMessage('😢 Sorry that you didn\'t like our response.');
            break;
        default:
            vscode.window.showInformationMessage('Don\'t know what to do with this feedback type.');
            break;
    }
}

This handler doesn't do anything useful with the feedback, it just shows to the user a message based on the feedback type.

Now, we just need to register the handler in our extension. Open the extension.ts file and add the following code in the activate function:

participant.onDidReceiveFeedback(handleFeedback); // Handle thumbs up and down feedback

We are now ready to test this. Go ahead, try and use thumbs up or down and will see something like this:

References

• Measuring Success

Next In Series

Next up: Adding configurability to our chat parrot. Stay tuned!