Generative AI for the rest of us

© 2023, Amazon Web Services, Inc. or its affiliates.
Massimo Re Ferrè
Senior Principal Technologist, AWS
Generative AI for the rest of us

© 2023, Amazon Web Services, Inc. or its affiliates. 2
Mainframes
Zooming out a bit
Technology wave #1
Data center

Mainframes
Zooming out a bit
Personal Computers
Technology wave #2
Technology wave #1
Data center

Mainframes
Zooming out a bit
Phyisical Servers
Virtual Machines
Personal Computers
Technology wave #2
Technology wave #1
Data center

Mainframes
Zooming out a bit
Phyisical Servers
Virtual Machines
Personal Computers
Technology wave #2
Technology wave #1
T
e
c
h
n
o
l
o
g
y
d
e
l
i
v
e
r
y
m
o
d
e
l
Data center
Cloud

Mainframes
Zooming out a bit
Phyisical Servers
Virtual Machines
Personal Computers
Containers
Functions
Technology wave #2
Technology wave #1
T
e
c
h
n
o
l
o
g
y
d
e
l
i
v
e
r
y
m
o
d
e
l
Data center
Cloud

Mainframes
Zooming out a bit
Phyisical Servers
Virtual Machines
Personal Computers
Containers
Functions
Generative AI
Technology wave #3
Technology wave #2
Technology wave #1
T
e
c
h
n
o
l
o
g
y
d
e
l
i
v
e
r
y
m
o
d
e
l
Data center
Cloud

What is Generative AI?
8

What is Generative AI (in simple terms)
9
- Traditional AI/ML: “Is this a picture of Rome or Florence?”
- [ Discriminative ]
- Gen AI: “Compare Rome Vs. Florence for someone interested in history”
- [ Generative ]

Gen AI “prompt”
10
A T I T S V E R Y C O R E ( T H E L L M - L A R G E L A N G U A G E M O D E L ) , G E N A I I S A F A K E . B U T A U S E F U L O N E
submit

This is how I like to think about an LLM
11
* or any profession that has nothing to do with a job in IT for that matter
q An LLM is akin to a … windsurfer professional*
qVery proficient in English
qAnd that had memorized all Wikipedia and all IT forums out there (and a lot more)
q They know Stack Overflow inside out! But don’t have a window to check the
weather (or a watch to check the time, etc)
q On their own, they have no relation to reality (beyond what they read)
q But they are great at generating free form content based on what they know
“have seen"

Why is Gen AI useful? [ the builder view ]
Source of unstructured knowledge
How can I use this knowledge and reason about it to create a new asset?
An asset being a piece of code, a whole program, a blog, an architecture, a troubleshooting
workflow, a db query and more outside of the IT realm (a poem, a picture, a receipt …)

Read and
memorize it all
(LOL – yeah sure)
(1)
You

You
Read and
memorize it all
(LOL – yeah sure)
Search engines (possibly
not relevant and still hard
- you are the integrator
and generator of a new
asset – text or code)
(1)
(2)
asset

15
You
Read and
memorize it all
(LOL – yeah sure)
LLM
Train on it
(doable)
(1)
(2)
(3a)
Natural language
conversation
(3b)
asset
asset
Search engines (possibly
not relevant and still hard
- you are the integrator
and generator of a new
asset – text or code)

My first Gen AI application
16

Real life use case
17
B A C K G R O U N D : I H A T E W H A T S A P P V O C A L M E S S A G E S
!!!!

Real life use case – the ClickOps version
18
Audio
file
Text
file
LLM
Audio to text translation Text summarization
Text
file

Real life use case – the ClickOps version
19
Prompt
Output
(generated asset)

Real life use case – the application version
20
M Y F I R S T ( N O N T U T O R I A L - B A S E D H E L L O - W O R L D ) G E N E R A T I V E A I A P P L I C A T I O N

Real life use case – the application version
L A M B D A C A L L S A N E X T E R N A L L L M S E R V I C E
https://it20.info/2023/08/building-a-generative-ai-application-using-aws-step-functions/

Making LLMs useful
22

Why are people talking about things like Agents, Tools, RAG..
q The LLM is just one (fundamental) component of Generative AI
q The LLM could hallucinate, don’t have knowledge of recent / private / live
information, can’t do advanced math, may have limited reasoning capabilities, etc.
q You need something to complement its capabilities and guide/help it
q Especially for “real” business use cases that go beyond “toying around”

T H E R E ’ R E T W O D I M E N S I O N S T H E L L M O P E R A T E S I N ( L E V E L O F A B S T R A C T I O N A N D D O M A I N S )
Developing code
Debugging code
Living life
Deploying code
Domains
Writing a novel
Organizing travels

Developing code
Debugging code
Deploying code
Domains
Autocomplete a
function method
Build a new ERP
from scratch
Resolve an error
message
Rearchitect the
app to avoid this
error at scale
Suggest what I
could do today
Organize my
whole life for the
next 10 years
Level of abstraction
Simple task Complex task
Writing a novel
Living life
Organizing travels
Tell me how long
it takes driving
from Florence to
Rome
Plan in details all
my 1-year long
sabbatical

Domains
Progressive complexity
Completion Chat Reasoning Acting
à
à
à
coverage
Domain
A
function
of
the
corpus
data

Domains
Large Language model
Smaller
purpose
built/tuned
model
Models may need to be
helped / guided to achieve
goals where task complexity
is too high or simply for
missing domain knowledge

A N E X A M P L E O F C O T ( C H A I N O F T H O U G H T S )
https://arxiv.org/abs/2201.11903
But sometimes in-prompt Chain of Thoughts (CoT) isn’t enough for the LLM to reason properly
W
elcom
e to
the
m
agic world
of
“prom
pt engineering”

A N E X A M P L E O F T H E F A C T C H E C K I N G W I T H P R O M P T C H A I N I N G P R O C E S S
https://it20.info/2023/6/the-dark-zone-between-the-magic-genai-experience-and-the-large-language-model/
Q: What is the biggest clock in the world?

LLM
A N E X A M P L E O F T O O L S
You
“what’s the weather like
today in Rome?”
Math function
code
Web search
code
“Calculate <very complex
formula>”
(1a)
(2a)
(1b)
(2b)

A N E X A M P L E O F R E A C T ( R E A S O N I N G A N D A C T I N G )

LLM
A N E X A M P L E O F R E A C T ( R E A S O N I N G A N D A C T I N G )
You
Iterating
reasoning
code
“Write the solution for
<very complex task>” (1)
(2)
Tool

LLM
Vector
DB
A N E X A M P L E O F R A G ( R E T R I E V A L - A U G M E N T E D G E N E R A T I O N )
You
Private corpus of data
embedding
“Write a draft email on <specific
company secret topic>”
(1)
(2)

Prompt context Vs. RAG Vs. fine-tuning
34

Prompt context Vs. RAG Vs. fine-tuning: I am lost
q Fair. There are three ways to increase an LLM answer precision and correctness
1. Provide context in the prompt
2. Augment the LLM with an external source of vectorized data at inference time (RAG)
3. Fine tune the LLM with additional data
q There isn’t a global right or wrong approach. As often happens, it depends
q Also they are not mutually exclusive
q They could (and often should) be used together to achieve optimal results

q Rate of the change of the data source
q Limits, cost, latency, speed of prompt context tokens
q Cost of fine tuning
q including the work required to “prepare the data”
q Cost of creating and maintaining the vector store
Prompt context Vs. RAG Vs. fine-tuning: when to use what?

q Complexity of the architecture
q fine tuning may make the architecture easier (with an upfront fine-tuning investment)
q Shape and location of the data source
q Precision of the outcome
q no absolute rules exist, testing may be required
q Personal experience of the team building the solution
q “I have always used RAG and that’s what I am comfortable with”
Prompt context Vs. RAG Vs. fine-tuning: when to use what?

Who’s Gen AI for?
38

Who’s Gen AI for?
q For the developer that is writing code
q e.g. code assistants e.g. AWS CodeWhisperer
q For the developer that wants to use English as a programming language
q e.g. the example of the WhatsApp vocal messages
q For the ops person that does not want to write a SQL query to extract data
q e.g. https://www.honeycomb.io/blog/introducing-query-assistant

Who’s Gen AI for?
q For the business analyst that wants to create a report off of a spreadsheet
q For the journalist that wants to draft an article on a specific topic
q Etc. etc.
q Come see me later to chat about the story of my plumber impressed by “chat …
chat …. chat something” (true story)

Conclusions
q Get ready for this new wave. It’s coming and (I think) it’s staying.
q LLMs have moved the needle of the art of possible
q But LLMs alone are not enough. You need to … make LLMs useful.
q Gen AI is for everyone, not just for “builders”. It’s for “consumers” of tech too.
q Go explore! Go build!

Massimo Re Ferrè
Senior Principal Technologist, AWS
Twitter: @mreferre
E-mail: mreferre@amazon.com
Thanks!

Generative AI for the rest of us

More Related Content

What's hot (20)

Similar to Generative AI for the rest of us (20)

More from Massimo Ferre' (14)

Recently uploaded (20)

Generative AI for the rest of us