On Programming Variability with Large Language Model-based Assistant

On Programming Variability with Large
Language Model-based Assistant
Mathieu Acher, José A. Galindo, Jean-Marc Jézéquel
https://inria.hal.science/hal-04153310/
https://github.com/acherm/progvary-withgpt/
1

Generative programming [Czarnecki2000], Model-driven engineering:
automatically generate variants from a specification written in one or more textual or
graphical domain-specific languages
Accidental and
essential complexity
Variability further
increases this
software complexity:
multiple features, code
variations, and an
exponential number of
possible variants 4

LLM
Hypothesis: Large language models (LLMs) act as a new variability compiler capable
of transforming a high-level specification (“prompt”) into variable code, features,
generators, configurable systems, etc. written in a given technological space.
Motto:
“features as
prompts”
5

https://github.com/acherm/progvary-withgpt/blob/main/conditioncompilation-helloworld/Chat
GPT-Rust_Configurable_Feature_Program.md
6

https://github.com/acherm/progvary-withgpt/blob/main/conditioncom
pilation-helloworld/ChatGPT-Java_program_with_Jinja.md 7

https://github.com/acherm/
progvary-withgpt/blob/main/
conditioncompilation-hellow
orld/ChatGPT-Java_progra
m_with_Jinja.md 8

https://github.com/acherm/progvary-withgpt/blob/main/featuret
oggle-helloworld/ChatGPT-Unleash_JS_Feature_Toggles.md
9

“would it be possible to adapt the
program as a Web application... and
provide a server-side that returns the
output while the client side prints it”
https://github.com/acherm/progvary-withgpt/blob/main/featuret
oggle-helloworld/ChatGPT-Unleash_JS_Feature_Toggles.md
10

Here is a TikZ code:
tikzset{%
something/.pic={
tikzset{x=3cm/5,y=3cm/5,shift={(0,-1/3)}}
useasboundingbox (-1,-1) (1,2);
fill [BlueGrey900] (0,-2)
.. controls ++(180:3) and ++(0:5/4) .. (-2,0)
arc (270:90:1/5)
.. controls ++(0:2) and ++(180:11/4) .. (0,-2+2/5);
foreach i in {-1,1}
scoped[shift={(1/2*i,9/4)}, rotate=45*i]{
clip [overlay] (0, 5/9) ellipse [radius=8/9];
clip [overlay] (0,-5/9) ellipse [radius=8/9];
fill [BlueGrey900] ellipse [radius=1];
fill [Purple100] ellipse [radius=1];
};
fill [BlueGrey900] ellipse [x radius=3/4, y radius=2];
fill [BlueGrey900]
(0,15/8) ellipse [x radius=1, y radius=5/6]
(0, 8/6) ellipse [x radius=1/2, y radius=1/2]
{[shift={(-1/2,-2)}, rotate= 10] ellipse [x radius=1/3, y radius=5/4]}
{[shift={( 1/2,-2)}, rotate=-10] ellipse [x radius=1/3, y radius=5/4]};
fill [BlueGrey500]
(-1/9,11/8) ellipse [x radius=1/5, y radius=1/5]
11

tikzset{%
cat/.pic={
.. controls ++(180:3) and ++(0:5/4) .. (-2,0)
arc (270:90:1/5)
.. controls ++(0:2) and ++(180:11/4) .. (0,-2+2/5);
foreach i in {-1,1}
};
fill [BlueGrey900]
fill [BlueGrey500]
( 1/9,11/8) ellipse [x radius=1/5, y radius=1/5];
fill [Purple100]
(0,12/8) ellipse [x radius=1/10, y radius=1/5]
(0,12/8+1/9) ellipse [x radius=1/5 , y radius=1/10];
foreach i in {-1,1}
scoped[shift={(1/2*i,2)}, rotate=35*i]{
fill [Yellow50] ellipse [radius=1];
};
scoped{
clip (-1,-2) rectangle ++(2,1);
fill [BlueGrey900] (0,-2) ellipse [radius=1/2];
fill [Grey100]
(-1/2,-2) ellipse [x radius=1/3, y radius=1/4]
( 1/2,-2) ellipse [x radius=1/3, y radius=1/4];
};
foreach i in {-1,1}
foreach j in {-1,0,1}
fill [Grey100, shift={(0,11/8)}, xscale=i, rotate=j*15,
12

tikzset{%
cat/.pic={
.. controls ++(180:3) and ++(0:5/4) .. (-2,0)
arc (270:90:1/5)
.. controls ++(0:2) and ++(180:11/4) .. (0,-2+2/5);
foreach i in {-1,1}
};
fill [BlueGrey900]
fill [BlueGrey500]
fill [Purple100]
foreach i in {-1,1}
I’d like to change the color of the cat’s tail
13

tikzset{%
cat/.pic={
.. controls ++(180:3) and ++(0:5/4) .. (-2,0)
arc (270:90:1/5)
.. controls ++(0:2) and ++(180:11/4) .. (0,-2+2/5);
foreach i in {-1,1}
};
fill [BlueGrey900]
fill [BlueGrey500]
fill [Purple100]
foreach i in {-1,1}
added by
ChatGPT!
14

https://github.com/acherm/progvary-withgpt/blob/main/varyTikZ/ChatGPT-Change_eye_color.md
15

https://github.com/acherm/progvary-withgpt/blob/main/varyTikZ/ChatGPT-Change_eye_color.md
16

tikzset{%
cat/.pic={
.. controls ++(180:3) and ++(0:5/4) .. (-2,0)
arc (270:90:1/5)
.. controls ++(0:2) and ++(180:11/4) .. (0,-2+2/5);
foreach i in {-1,1}
};
fill [BlueGrey900]
fill [BlueGrey500]
fill [Purple100]
foreach i in {-1,1}
};
scoped{
clip (-1,-2) rectangle ++(2,1);
fill [BlueGrey900] (0,-2) ellipse [radius=1/2];
fill [Grey100]
(-1/2,-2) ellipse [x radius=1/3, y radius=1/4]
( 1/2,-2) ellipse [x radius=1/3, y radius=1/4];
};
foreach i in {-1,1}
foreach j in {-1,0,1}
fill [Grey100, shift={(0,11/8)}, xscale=i, rotate=j*15,
shift=(0:1/2)]
ellipse [x radius=1/3, y radius=1/64];
},
Wait wait… Feature location:
Despite a positive experience
here with the cat TikZ,
replications over different TikZ
code representing squid,
chimpanzee, or dogs show that
LLMs are in fact inaccurate.
LLMs can be lucky to guess
which portion of code is
associated to a feature, but
clearly not 100% of the time.
It is problematic since then the
mapping between features and
code element should then be
revised, after manual
investigation.
18

Threats to validity
LLM = ChatGPT4 mainly (also Claude from Anthropic AI and ChatGPT3.5)
Prompts
No control over hyperparameters (eg temperature)
Very few repetitions
Programming variability involves many tasks
19

Potential
LLMs can assist developers in implementing variability in different programming
languages (C, Rust, Java, TikZ, etc.) and mechanisms (conditional compilation, feature
toggles, command-line parameters, template, etc.). With "features as prompts", there is
hope to raise the level of abstraction, increase automation, and bring more flexibility
when synthesizing and exploring software variants.
20

Limitations
LLMs are by construction stochastic, non-determinist and highly sensitive to prompt
variations – and so are corresponding implementations of features and variability.
https://github.com/acherm/progvary-wit
hgpt/blob/main/varyfloatinC/ChatGPT-V
ariable_Type_Alternatives.md
LLM
21

Discussions
(limitations & potential)
In particular, LLMs can produce different programs out of the same exact prompt.
Unfortunately, some of these programs can be incomplete or simply wrong, introducing
serious variability errors.
Potential: LLMs act as assistant that recommend some possible implementations of
variability. It is the role of programmers to review the recommendation and either
selects another recommendation or manually implement variability.
LLM
22

Future work
#1 A variability benchmark for LLM-based assistant (for assessing accuracy and
robustness)
Challenge: In our case, we target comprehensive programming scenarios that (1)
require combining different pieces of code and (2) call to combine different languages
and technologies for realizing variability. Not easy to automate the “goodness”
assessment of some solutions
LLM
23

Future work
#2 Improving the robustness of LLMs
Challenge: how to properly formulate instructions/prompts for programming variability…
informality of NLP can backfire. Dedicated “variability language” for LLM? Specialization
of LLMs for variability tasks?
LLM
24

#1 A variability benchmark for LLM-based assistant
(for assessing accuracy and robustness)
#2 Improving the robustness of LLMs
https://inria.hal.science/hal-04153310/
https://github.com/acherm/progvary-withgpt/
25

Remainder of the talk
How LLMs can be concretely and originally used for programming
software variability
Variability implementations of a classical Hello world
Variability for reproducibility and floating-points
Transforming an unfamiliar code with an end-user, Web-based
customization tool
Discussions (potential and limitations)
27

https://github.com/acherm/progvary-withgpt/blob/main/varyfloatinC/ChatGPT-C_Variations_with_%23ifdef.md
https://github.com/acherm/progvary-withgpt/blob/main/varyfloatinC/approx.c
28

https://github.com/acherm/progvary-withgpt/blob/main/varyfloatinC/ChatGPT-C_Variations_with_%23ifdef.md
https://github.com/acherm/progvary-withgpt/blob/main/varyfloatinC/approx_eval.py
29

retrieve the result of S. Boldo et al.
30

On Programming Variability with Large Language Model-based Assistant

More Related Content

More from University of Rennes, INSA Rennes, Inria/IRISA, CNRS (20)

Recently uploaded (20)

On Programming Variability with Large Language Model-based Assistant