10 Lessons From Building AI Powered Features

Presented at All Things Open RTP Meetup
Presented by: Dalia Abo Sheasha, Senior Product Manager @ Microsoft

Abstract: In this session, Dalia from the GitHub Copilot in Visual Studio team will share valuable lessons learned from integrating AI-powered features into one of the world’s most widely-used development environments. Drawing from hands-on experience, Dalia will discuss the unique challenges of embedding AI into existing systems, the art and science of prompt engineering, navigating COGS limitations, and adhering to Responsible AI standards. You'll learn practical strategies for balancing speed with quality, leveraging AI to enhance user experience, and overcoming common technical hurdles. Whether you’re new to AI or working on your own AI-powered projects, this talk will provide actionable insights to help you build smarter, more powerful applications.

1. 10 Lessons from
Building AI-Powered
Features
Dalia Abo Sheasha
Senior Product Manager, Microsoft

2. Who Am I?
•Now:
•Product Manager (Microsoft Visual
Studio)
•Before:
•Software Engineering Team Lead
(Modernization tooling)
•Software Engineer (Backend
development)
•Developer Advocate (Developer tools)

3. The era of the
AI Copilot
“But the thing that every
developer can do that is hard for
us at a scale to do, is to really go
into specific domains and
figure out how to make this
technology work there”

4. 6 months later…

5. Lessons
Learned

6. Lesson #1
Learn how AI works
(it’s not magic)

7. I'm a bit jet lagged. It was an 8 hour ____.
Source: Ahmed Tawfik | What is Generative AI?
Can you fill in the blank?
flight
trip
ride
journey
mission
…

8. Generative AI: Strengths

9. Generating
content

10. Having
conversations

11. Summarizing
content

12. Creating images and videos
Language translations
Personalization
Data augmentation
And more…

13. Generative AI: Current Weaknesses

14. Bias AccuracyContext

15. Lesson #2
Identify an unmet user need

16. Mission
Empower every developer
to do more

17. “What is your
least favorite part
of your job?”

18. Generative
AI
Strengths
Your User
Need
The sweet spot

19. Caution:
Analysis Paralysis

20. Lesson #3
Build.Something.

21. Hackathon (June 1-2, 2023)

22. Great Learning
Opportunity!
•Environment Setup
•Prompt Engineering
•APIs/Libraries
•AI Models
Strengths/Weaknesses
•Limitations (token limits,
COGS)
•Troubleshooting

23. Best reason to prototype?
Have something to show your users and
validate they actually want it.

24. Lesson #4
Be ready to pivot and iterate

25. Red Flags

26. New feature contradicts existing user understanding
“I would not read that”
“The code is supposed to explain
what you're doing, the doc should
explain why you're doing it”
If the suggestions are wrong,
the consequences are big
“Copilot doesn't like my code,
my self-esteem suffers”
“My problem with AI is that
sometimes it's breaking my code”

27. “It has inferred variable names
and I'm not sure how”

28. AI-Powered Rename
Suggestions
(DEMO)

29. A software developer is
using an AI rename feature
in a code editor. Your task
is to help a developer
improve their symbol…
Prompt
Response
{"resultMessage",
"outcomeMessage",
"gameResult"}
Large Language
Model (LLM)

30. Lesson #5
A good prompt is key

31. “Prompt engineering is the practice of
designing inputs for generative AI tools
that will produce optimal outputs.”
What is prompt engineering? | McKinsey

32. Early Attempts
Input
Your task is to help the Developer improve the name of one
variable. The variable is indicated by [CURRENT DOCUMENT CODE
SELECTED BY THE DEVELOPER] and [CURRENT DOCUMENT CODE SELECTED
BY THE DEVELOPER END].
Analyze the definition and usage of the variable in the current
document.
…
public static String Message(Choice choice1, Choice choice2)
{
[CURRENT DOCUMENT CODE SELECTED BY THE DEVELOPER]
String str = string.Empty;
[CURRENT DOCUMENT CODE SELECTED BY THE DEVELOPER END]
if (choice1 == choice2)
{
Console.ForegroundColor = ConsoleColor.Yellow;
…
Return three variable names that would be better than the
current document code selected by the developer. The variable
names should do the following in order of priority
- Describe how the variable is defined and is being used.
- Follow the current document naming conventions
- Follow the language best practices.
Output
```C#
// Better variable names for x in class Game
of file path
C:\Users\dabosheasha\source\repos\RockPaperS
cissors\RockPaperScissors\Game.cs
// Describes that this is the result of the
game played
String gameResult = string.Empty;
// Describes that this is the player’s
choice of Rock, Paper or Scissors
Choice playerSelection = Choice.Invalid;
//Describes that this is the computer’s
random choice of Rock, Paper, or Scissors
Choice computerSelection = Choice.Invalid;
```

33. 50+ Attempts Later…
Input
A software developer is using an AI rename feature in a
code editor.
Your task is to help a developer improve their symbol
indicated by [NameThisIdentifier]. The existing symbol
name is "str".
The following is the code:
…
public static String Message(Choice choice1, Choice
choice2)
{
String [NameThisIdentifier] = string.Empty;
if (choice1 == choice2)
{
Console.ForegroundColor = ConsoleColor.Yellow;
…
Given the provided information, generate five suggestions
to rename the selected symbol. Put the suggestions in a
JSON array called `SuggestedNames` and return the json
object only as a response. Here is an example of the
RESPONSE format: { "SuggestedNames": ["firstName",
"secondName", "thirdName", "fourthName", "fifthName"] }
Output
{ "SuggestedNames":
[
"resultMessage",
"outcomeMessage",
"gameResult"
]
}

34. Experiment and document your
attempts

35. OpenAI Prompt
Engineering Guide

36. Lesson #6
Start with simple
prompt techniques

37. LLM
Zero-shot prompting
Tell me a joke
Why can't you give Elsa a
balloon?
Because she will let it go!

38. LLM
Few-shot prompting
Tell me a joke
Example: Why don't eggs
tell jokes? They'd crack
each other up.
Example: What do you
call cheese that isn't
yours? Nacho cheese.
Example: Why did the
coffee file a police
report? It got mugged.
What do you call fake
spaghetti? An impasta!

39. Relevant
Data
Retrieval Augmented Generation
Tell me a topical joke What do you call
an AI that loves to
sing? A tune-ning
machine! LLM

40. Fine-tuned
LLM
Fine-tuning
Tell me a joke
ةرمدحاوهجدعقيىلعةوهقدعق
ىلعياش
Translation: Once upon a
time, a guy was going to
sit on a coffee shop
(another word for coffee),
he sat instead on tea

41. Which technique do you use?
Start with zero-shot,
then few-shot,
then RAG,
then fine-tune
Source: Inside Microsoft AI innovations
with Mark Russinovich

42. Lesson #7
Suggestions Quality > UI

43. A suggestion accuracy and relevance
has the highest correlation with user
satisfaction (more than usability)

44. “Data is your superpower,
even more so in an AI world”
– Taylor Mullen

45. Data
•Acceptance rate ↑
•Request rate ↑
•Error rate ↓

46. Lesson #8
Build Responsibly
(Responsible AI)

47. Microsoft
Responsible AI
Principles

48. Building Responsibly
•Need to be transparent with the
user that an AI is in use and
results may be variable
•Encourages users to pause and verify
•Accepting a bad suggestion leads to
dissatisfaction
•Transparency about context
allowed users to use the
product better
•Give developers more controls
•Ask for permission when
appropriate

49. Building Responsibly
•Privacy and Security
•GitHub Copilot Trust
Center
(https://gh.io/ctc)
•More information:
•Microsoft Responsible
AI Standard
•Responsible AI
Transparency Report

50. Lesson #9
Be careful with COGS
(Cost of Goods Sold)

51. Credit: Etienne Baudoux

52. Remove
the button! Put it back!
Remove
the button!

54. Lesson #10
Enjoy the Journey

56. Lessons
1.Learn how AI works
2.Identify an unmet user need
3.Build.Something.
4.Be ready to pivot and iterate
5.A good prompt is key
6.Start with simple prompt techniques
7.Suggestions Quality > UI
8.Build Responsibly (Responsible AI)
9.Be careful with COGS
10.Enjoy the Journey

57. Resources
•AI for Beginners
•OpenAI Prompt Engineering Guide
•Microsoft Responsible AI
Standard
•GitHub Copilot with Visual Studio
•AI-Powered Rename Suggestions
in Visual Studio

58. Thank you!
Dalia Abo Sheasha

59. Q&A