![Introduction Hierarchical model Extensions Implementation Conclusions Extra
Limitation
b
e
Choose either
d or e after aa
c
d
∆c = {3}
∆e = {2}
∆d = {1}
∆b = [1, 2]
=
=
=
=
=
=
=](https://image.slidesharecdn.com/sony-slides-180901185802/85/Structured-Interactive-Scores-with-formal-semantics-21-320.jpg)
![Introduction Hierarchical model Extensions Implementation Conclusions Extra
Structural definition
Verse 1 (v)
Chorus (ch)
Verse 3 (x)
Verse 2 (w)
Verse 4 (y)
Chorus (c)
∆v
∆w∆ch
∆c
∆ = 0
∆
=
0
∆ = 0
∆ = [2, 5]
∆ = [1, 4]
ϕ > 0 ∨ ψ ≤ 3
φ
>
0 ∧
ψ
≤
5
d
∆x ∆y
∆a
∆b
V = {φ, ψ, ϕ}
main (m)](https://image.slidesharecdn.com/sony-slides-180901185802/85/Structured-Interactive-Scores-with-formal-semantics-22-320.jpg)
![Introduction Hierarchical model Extensions Implementation Conclusions Extra
Interactive Scores: Temporal objects
Karplus (k1)
Karplus (k2)
Karplus (k3)
∆k1 = [10s, 10s]
∆k2 = [5s, 10s]
∆k3 = [4s, 4s]
Output](https://image.slidesharecdn.com/sony-slides-180901185802/85/Structured-Interactive-Scores-with-formal-semantics-26-320.jpg)
![Introduction Hierarchical model Extensions Implementation Conclusions Extra
Interactive Scores: Temporal relations
Karplus (k1)
Karplus (k2)
Karplus (k3)
100smp
[2s, 4s]
[0s, 0s]
[0s, 0s]
Output](https://image.slidesharecdn.com/sony-slides-180901185802/85/Structured-Interactive-Scores-with-formal-semantics-27-320.jpg)
![Introduction Hierarchical model Extensions Implementation Conclusions Extra
Extension of Interactive Scores
Karplus (k1)
Karplus (k2)
Karplus (k3)a
b100smp
[2s, 4s]
[0s, 0s]
[0s, 0s]
∆k1 = [10s, 10s]
∆k2 = [5s, 10s]
∆k3 = [4s, 4s]
ThreeStrings(f)
Interactive score
Temporal relation
High-precision
temporal relation
Temporal object
[i,j] Duration
Interactive object
Output
Dataflow relation](https://image.slidesharecdn.com/sony-slides-180901185802/85/Structured-Interactive-Scores-with-formal-semantics-30-320.jpg)
![Introduction Hierarchical model Extensions Implementation Conclusions Extra
Extension of Interactive Scores
Karplus (k1)
Karplus (k2)
Karplus (k3)a
b100smp
[2s, 4s]
[0s, 0s]
[0s, 0s]
∆k1 = [10s, 10s]
∆k2 = [5s, 10s]
∆k3 = [4s, 4s]
ThreeStrings(f)
Interactive score
Temporal relation
High-precision
temporal relation
Temporal object
[i,j] Duration
Interactive object
Output
Dataflow relation](https://image.slidesharecdn.com/sony-slides-180901185802/85/Structured-Interactive-Scores-with-formal-semantics-43-320.jpg)
![Introduction Hierarchical model Extensions Implementation Conclusions Extra
Extension of Interactive Scores
Karplus (k1)
Karplus (k2)
Karplus (k3)a
b100smp
[2s, 4s]
[0s, 0s]
[0s, 0s]
∆k1 = [10s, 10s]
∆k2 = [5s, 10s]
∆k3 = [4s, 4s]
ThreeStrings(f)
Interactive score
Temporal relation
High-precision
temporal relation
Temporal object
[i,j] Duration
Interactive object
Output
Dataflow relation](https://image.slidesharecdn.com/sony-slides-180901185802/85/Structured-Interactive-Scores-with-formal-semantics-45-320.jpg)
![Introduction Hierarchical model Extensions Implementation Conclusions Extra
Example: An arpeggio with three strings
Karplus (k1)
Karplus (k2)
Karplus (k3)a
b[100smp, 100smp]
[2s, 4s]
[0s, 0s]
[0s, 0s]
∆k1 = [10s, 10s]
∆k2 = [5s, 10s]
∆k3 = [4s, 4s]
ThreeStrings(f)](https://image.slidesharecdn.com/sony-slides-180901185802/85/Structured-Interactive-Scores-with-formal-semantics-46-320.jpg)
![Introduction Hierarchical model Extensions Implementation Conclusions Extra
Example: An arpeggio with three strings
a
b
[2s, 4s]
∆k1 = [10s, 10s]
∆k2 = [5s, 10s]
∆k3 = [4s, 4s]
Figure: The constraint graph](https://image.slidesharecdn.com/sony-slides-180901185802/85/Structured-Interactive-Scores-with-formal-semantics-47-320.jpg)
![Introduction Hierarchical model Extensions Implementation Conclusions Extra
Application: An arpeggio without “clicks”
a
b
[100smp, 100smp]
[2s, 4s]
[0s, 0s]
[0s, 0s]
∆k1 = [10s, 10s] Anti Click Three Karplus (f)Karplus (k1)
Karplus' AC[9.5s, 9.5s] [0.5s, 0.5s]
Karplus (k2)
Karplus' AC [0.5s, 0.5s]
∆k2 = [5s, 10s]
[4.5s, 9.5s]
Karplus (k3)
Karplus' AC [0.5s, 0.5s]
∆k3 = [4s, 4s]
[3.5s, 3.5s]](https://image.slidesharecdn.com/sony-slides-180901185802/85/Structured-Interactive-Scores-with-formal-semantics-51-320.jpg)
![Introduction Hierarchical model Extensions Implementation Conclusions Extra
Application: Sound source perception
Karplus (k1)
L Output (o1)
[0s, 0s]
∆k1 = [10s, 10s]
∆o1 = [10s, 10s]
R Output (o2) ∆o2 = [10s, 10s]
Karplus (k1)
L Output (o1)
∆k1 = [10s, 10s]
∆o1 = [10s, 10s]
R Output (o2) ∆o2 = [10s, 10s]
500µs
Time Time](https://image.slidesharecdn.com/sony-slides-180901185802/85/Structured-Interactive-Scores-with-formal-semantics-52-320.jpg)
Structured Interactive Scores with formal semantics
- 1. Introduction Hierarchical model Extensions Implementation Conclusions Extra Structured interactive scores: From a structural description of a multimedia scenario to a real-time capable implementation with formal semantics Mauricio TORO∗ Talk at Sony Computer Science Laboratory, Paris ∗LaBRI, Universit´e de Bordeaux. July 20th 2012
- 2. Introduction Hierarchical model Extensions Implementation Conclusions Extra Some domains shaped by the arose of technology Concrete Music Electroacoustic Music Installations Interactive Multimedia Experimental Music Non-linear Multimedia
- 3. Introduction Hierarchical model Extensions Implementation Conclusions Extra Problems with existing tools 1. Time models are unrelated
- 4. Introduction Hierarchical model Extensions Implementation Conclusions Extra Time models are unrelated: Cue Lists Vs. Timeline Vs.
- 5. Introduction Hierarchical model Extensions Implementation Conclusions Extra Problems with existing tools 1. Time models are unrelated 2. Schedulers are not appropriate for soft real-time
- 6. Introduction Hierarchical model Extensions Implementation Conclusions Extra Scheduler problem: aggravated under high cpu load • A Karplus-Strong implementation in Pure Data1 . • A Karplus-Strong implementation in Faust. 1 Colin Barry’s implementation found at www.loomer.co.uk
- 7. Introduction Hierarchical model Extensions Implementation Conclusions Extra Problems with existing tools 1. Time models are unrelated 2. Schedulers are not appropriate for soft real-time 3. No hierarchy 4. No formal semantics 5. No unified model for conditional branching and temporal relations
- 8. Introduction Hierarchical model Extensions Implementation Conclusions Extra Solution: Interactive Scores Green (g) Red (r) Sound (u) Lights (l) a b Scenario (c) d overlaps starts starts ends ends ends
- 9. Introduction Hierarchical model Extensions Implementation Conclusions Extra Outline of the presentation 1. Hierarchical model 2. Time Conditional model 3. Signal Processing model 4. Simulation 5. Verification 6. Conclusions
- 10. Introduction Hierarchical model Extensions Implementation Conclusions Extra Structural definition: Temporal objects Green (g) Red (r) Sound (u)
- 11. Introduction Hierarchical model Extensions Implementation Conclusions Extra Structural definition: Hierarchy of temporal objects Green (g) Red (r) Sound (u) Lights (l) Scenario (c)
- 12. Introduction Hierarchical model Extensions Implementation Conclusions Extra Structural definition: Interactive objects Green (g) Red (r) Sound (u) Lights (l) a b Scenario (c) d
- 13. Introduction Hierarchical model Extensions Implementation Conclusions Extra Structural definition: Temporal relations Green (g) Red (r) Sound (u) Lights (l) a b Scenario (c) d overlaps starts starts ends ends ends
- 14. Introduction Hierarchical model Extensions Implementation Conclusions Extra Structural definition Green (g) Red (r) Sound (u) Lights (l) a b Scenario (c) d ∆r ∆g ∆u ∆l ∆c {0} {0} {0} {0} {0} [0, ∞) [0, ∞) [0, ∞)
- 15. Introduction Hierarchical model Extensions Implementation Conclusions Extra tes semantics: Temporal objects ∆r ∆g ∆u ∆c ∆l sc su sb sl sr sg sa sd ec eu er eg el
- 16. Introduction Hierarchical model Extensions Implementation Conclusions Extra tes semantics: Temporal relations 0 0 0 0 (0,∞ )(0,∞ ) (0,∞ ) sc su sb sl sr sg 0 sa sd ec eu er eg
- 17. Introduction Hierarchical model Extensions Implementation Conclusions Extra tes semantics: Interactive scores 0 0 0 0 (0,∞ )(0,∞ ) (0,∞ ) ∆r ∆g ∆u ∆c ∆l sc su sb sl sr sg 0 sa sd ec eu er eg
- 18. Introduction Hierarchical model Extensions Implementation Conclusions Extra Operational semantics: Intuition sc su sb sl sr sg sa sd ec eu er eg 0 0 0 0 (0 ,∞ )(0,∞) (0 , ∞ ) ∆r ∆g ∆u ∆c ∆l sc su sb sl sr sg 0 sa sd ec eu er eg Constraint Store
- 19. Introduction Hierarchical model Extensions Implementation Conclusions Extra Correctness of the operational semantics with respect to the event structures semantics Theorem The possible time units in which a point process can be launched are subsummed by the possible time units in which an event can appear in an event trace. Proof. By induction on the structure of an event structure semantics of a score.
- 20. Introduction Hierarchical model Extensions Implementation Conclusions Extra Time conditional model • Time conditional relations and choices are used to state weather some objects launch a silent action or a visible action. • All temporal objects are executed and all branches starting on the same point have the same set of durations.
- 21. Introduction Hierarchical model Extensions Implementation Conclusions Extra Limitation b e Choose either d or e after aa c d ∆c = {3} ∆e = {2} ∆d = {1} ∆b = [1, 2] = = = = = = =
- 22. Introduction Hierarchical model Extensions Implementation Conclusions Extra Structural definition Verse 1 (v) Chorus (ch) Verse 3 (x) Verse 2 (w) Verse 4 (y) Chorus (c) ∆v ∆w∆ch ∆c ∆ = 0 ∆ = 0 ∆ = 0 ∆ = [2, 5] ∆ = [1, 4] ϕ > 0 ∨ ψ ≤ 3 φ > 0 ∧ ψ ≤ 5 d ∆x ∆y ∆a ∆b V = {φ, ψ, ϕ} main (m)
- 23. Introduction Hierarchical model Extensions Implementation Conclusions Extra tes semantics 0 0 sa sb eb ea ecscsv ev d s a s b
- 24. Introduction Hierarchical model Extensions Implementation Conclusions Extra Extension with signal processing To have control over the microform of sound during performance, we need two new types of relations • high-precision temporal relations that can be expressed in the order of samples, milliseconds and microseconds, and • data-flow relations to connect the sound output of a temporal object to the input of another.
- 25. Introduction Hierarchical model Extensions Implementation Conclusions Extra Interactive Scores: Temporal objects Karplus (k1) Karplus (k2) Karplus (k3) Output
- 26. Introduction Hierarchical model Extensions Implementation Conclusions Extra Interactive Scores: Temporal objects Karplus (k1) Karplus (k2) Karplus (k3) ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] Output
- 27. Introduction Hierarchical model Extensions Implementation Conclusions Extra Interactive Scores: Temporal relations Karplus (k1) Karplus (k2) Karplus (k3) 100smp [2s, 4s] [0s, 0s] [0s, 0s] Output
- 28. Introduction Hierarchical model Extensions Implementation Conclusions Extra Interactive Scores: Dataflow relations Karplus (k1) Karplus (k2) Karplus (k3) Output
- 29. Introduction Hierarchical model Extensions Implementation Conclusions Extra Interactive Scores: Interactive objects Karplus (k1) Karplus (k2) Karplus (k3)a b Output
- 30. Introduction Hierarchical model Extensions Implementation Conclusions Extra Extension of Interactive Scores Karplus (k1) Karplus (k2) Karplus (k3)a b100smp [2s, 4s] [0s, 0s] [0s, 0s] ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] ThreeStrings(f) Interactive score Temporal relation High-precision temporal relation Temporal object [i,j] Duration Interactive object Output Dataflow relation
- 31. Introduction Hierarchical model Extensions Implementation Conclusions Extra Simulation OpenMusic interface Ntccrt compiler Pure Data external Max/Msp external Common Lisp interface C++ interface Stand-alone program User Programmer
- 32. Introduction Hierarchical model Extensions Implementation Conclusions Extra Results on the conditional branching extension 0 5 10 15 20 25 30 35 40 45 50 0 200 400 600 800 1000 1200 1400 1600 1800 Average dura*on of a *me unit (ms) Number of points in the Interac*ve Score
- 33. Introduction Hierarchical model Extensions Implementation Conclusions Extra Results on the signal processing extension Implementation 85% cpu load 3% cpu load Using object block˜ 1 in Pd 7991 ms 1 ms Using object z˜ 1 in Pd 9231 ms 2 ms Using Faust object in Pd 0.5 ms 0.5 ms Table: Average relative jitter of each string of an arpeggio with respect to the beep. All implementations use Karplus-Strong algorithm to simulate the sound of the strings.
- 34. Introduction Hierarchical model Extensions Implementation Conclusions Extra Verification P F YES NO P F NFA NFA DFA DFA DFA' DFA DFA' IS EMPTY? P Model checking algorithm: Does a formula F satisfies a process P?
- 35. Introduction Hierarchical model Extensions Implementation Conclusions Extra cltl properties Examples of constraint linear-time logic (cltl) formulae are • “always the constraint pitch = 60 can be deduced from the output store”, namely (pitch = 60); and • “eventually, object a and object b are launched at the same time”, namely ♦(launcha ˙∧launchb).
- 36. Introduction Hierarchical model Extensions Implementation Conclusions Extra Summary • A Formal model for multimedia interaction that combines • timeline model • cue-list model • An extension for conditional branching • An extension for audio processing • A simulation tool • A verification tool
- 37. Introduction Hierarchical model Extensions Implementation Conclusions Extra Related work to interactive scores • Score Following • Machine Improvisation • Computer Assisted Composition • Globally Asynchronous, locally Synchronous
- 38. Introduction Hierarchical model Extensions Implementation Conclusions Extra Discussion • Comparison with previous models of interactive scores • Comparison with Csound, Max, Live, QLab, Protools...
- 39. Introduction Hierarchical model Extensions Implementation Conclusions Extra Discussion • Comparison with previous models of interactive scores • Comparison with Csound, Max, Live, QLab, Protools...
- 40. Introduction Hierarchical model Extensions Implementation Conclusions Extra Solutions to the problem statements • Time models are unrelated • Schedulers are not appropriate for soft real-time • No hierarchy • No formal semantics • No unified model for conditional branching and temporal relations
- 41. Introduction Hierarchical model Extensions Implementation Conclusions Extra Future work • inedit • react+
- 42. Introduction Hierarchical model Extensions Implementation Conclusions Extra M E R C I. Avez-vous des questions?
- 43. Introduction Hierarchical model Extensions Implementation Conclusions Extra Extension of Interactive Scores Karplus (k1) Karplus (k2) Karplus (k3)a b100smp [2s, 4s] [0s, 0s] [0s, 0s] ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] ThreeStrings(f) Interactive score Temporal relation High-precision temporal relation Temporal object [i,j] Duration Interactive object Output Dataflow relation
- 44. Introduction Hierarchical model Extensions Implementation Conclusions Extra The novelty of our extension: Faust and ntcc interconection 1. A model based on the Non-deterministic Timed Concurrent Constraint (ntcc) calculus for concurrency, user interactions and temporal relations, and 2. a model based on the Faust programming language for sound processing and micro controls. buton Label mouse down mouse up 0 1 user ntcc
- 45. Introduction Hierarchical model Extensions Implementation Conclusions Extra Extension of Interactive Scores Karplus (k1) Karplus (k2) Karplus (k3)a b100smp [2s, 4s] [0s, 0s] [0s, 0s] ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] ThreeStrings(f) Interactive score Temporal relation High-precision temporal relation Temporal object [i,j] Duration Interactive object Output Dataflow relation
- 46. Introduction Hierarchical model Extensions Implementation Conclusions Extra Example: An arpeggio with three strings Karplus (k1) Karplus (k2) Karplus (k3)a b[100smp, 100smp] [2s, 4s] [0s, 0s] [0s, 0s] ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] ThreeStrings(f)
- 47. Introduction Hierarchical model Extensions Implementation Conclusions Extra Example: An arpeggio with three strings a b [2s, 4s] ∆k1 = [10s, 10s] ∆k2 = [5s, 10s] ∆k3 = [4s, 4s] Figure: The constraint graph
- 48. Introduction Hierarchical model Extensions Implementation Conclusions Extra Example: An arpeggio with three strings Karplus (k1) Karplus (k2) Karplus (k3) @100 threeCords(f) output sk1 ek1 sk1 ek2 sk2 Figure: Faust’s Block diagram
- 49. Introduction Hierarchical model Extensions Implementation Conclusions Extra Example: An arpeggio with three strings Figure: Implementation
- 50. Introduction Hierarchical model Extensions Implementation Conclusions Extra Application: Three user controled arpeggios ∆ = 10 ∆ = 10 ∆ = 10 Figure: The double-headed arrow represents an inequality (≤) and a white-headed arrow represents an equality relation (=).
- 51. Introduction Hierarchical model Extensions Implementation Conclusions Extra Application: An arpeggio without “clicks” a b [100smp, 100smp] [2s, 4s] [0s, 0s] [0s, 0s] ∆k1 = [10s, 10s] Anti Click Three Karplus (f)Karplus (k1) Karplus' AC[9.5s, 9.5s] [0.5s, 0.5s] Karplus (k2) Karplus' AC [0.5s, 0.5s] ∆k2 = [5s, 10s] [4.5s, 9.5s] Karplus (k3) Karplus' AC [0.5s, 0.5s] ∆k3 = [4s, 4s] [3.5s, 3.5s]
- 52. Introduction Hierarchical model Extensions Implementation Conclusions Extra Application: Sound source perception Karplus (k1) L Output (o1) [0s, 0s] ∆k1 = [10s, 10s] ∆o1 = [10s, 10s] R Output (o2) ∆o2 = [10s, 10s] Karplus (k1) L Output (o1) ∆k1 = [10s, 10s] ∆o1 = [10s, 10s] R Output (o2) ∆o2 = [10s, 10s] 500µs Time Time