QOrchestra and FlowML. High-level audio synthesis tools on top of SAOL

High-level audio synthesis tools on top of SAOL

1. Bert Schiettecatte, VUB
QOrchestra & FlowML
High-level audio synthesis tools
on top of SAOL

2. Introduction (1)
This presentation:
•Introduce an audio synthesis diagram format
(FlowML)
•Present a compiler from FlowML to SAOL
(FlowML2SAOL)
•Present a high-level authoring tool for SAOL
(QOrchestra)

3. Introduction (2)
The SA (Structured Audio) standard of Mpeg-4
audio consists of 5 elements:
•SAOL (SA Orchestra Language):a signal-
processing language for describing
musical instruments
•SASL (SA Score Language):a language
for describing a performance using the
instruments of a SAOL program

4. Introduction (3)
•SASBF (SA Sample Bank Format):allows
transmission of sample banks
•Scheduler description:used to translate
MIDI or SASL to events internally, in a
decoder
•Reference to MIDI:supported MIDI
messages and their meaning

5. Available technology (1)
Command-line encoders, decoders, profilers:
•saolc:reference implementation of
encoder/decoder
•sfront:SAOL-to-C compiler with realtime
I/O and some profiling support
•sarun:compiles SAOL to a program for a
virtual machine with a special instruction
set, for efficient decoding

6. Available technology (2)
Authoring tools:
•QOrchestra:high-level synthesis diagram
tool with SAOL export function, “non-
realtime”, supports re-use of diagrams
•CPS:low-level software synthesis tool
with SAOL save function, “realtime”
•VisOrc:Similar to CPS, with integrated
sequencer

7. Available technology (3)
IDEs:
•SFX:SAOL editing environment with
syntax highlighting, auditioning, some
profiling support

8. QOrchestra (1)
•Goal:experimental visual authoring tool for
SAOL (GPL license), drawing space with
primitive building blocks, connection tool,
library for new blocks, …
•Main idea:tool for people who don’t know
SAOL, or don’t want to write SAOL code
•Also a goal:case study of a complex
diagram editor and object-oriented
programming technology

9. QOrchestra (2)
QOrchestra is NOT:
•A low-level SAOL editor (e.g. CPS)
•A complete SA authoring environment
•A real-time software synthesis application
•An IDE
•Finished

10. QOrchestra (3)

11. FlowML (1)
•QOrchestra project requirement: save, load
and re-use synthesis diagrams
•Available storage technology: XML
(eXtensibleMarkup Language), a way to add
meta-information to information (e.g.
<author>Bert Schiettecatte</author>)

12. FlowML (2)
•Tags for a specific document class are
described in an XML DTD(document type
definition)
•XML technology is freely available, tested,
reliable, very easy to use, …
•FlowML = XML DTDsdescribing synthesis
diagram formats on a high level + collection
of synthesis constructs + synthesis
environment-specific extension mechanism
(~ C++ pragma’s)

13. FlowML (3)
•Solved storage problems in QOrchestra: store
diagrams and libraries as FlowML
•Generating SAOL code from FlowML is
relativelyeasy
•Very easy to understand (stored as text file)
•Could be a bridgebetween various software
synthesis systems
•Could be the right high-level tool for
describing bigsynthesis systems (<> SAOL)

14. FlowML (4)
•“Web-friendly”: easy to create browser plugins
•Is nota programming language, but a high-
level modelling language, in a way similar to
UML (methodology for structuring OO designs)
•FlowML is open and free
•Extensions for future versions can be proposed
and will be examined prior to becoming official
(same license as HTML)

15. FlowML (5)
FlowML diagram rules (informal):
•Multiple outputs per block
•Multiple connections per output
•Multiple inputs per block
•Only one connection per input
•All signals are mono

16. FlowML (6)
FlowML diagram rules (informal):
•Every component instance has a unique ID
used for referring
•Every definition contains a description and a
unique nameused for instantiating
•2 types of signals: audio and control
•Diagram/component I/O is done through
special primitivesdefined in the standard

17. FlowML (7)
Audio synthesis diagram example: Reverbmixer
lvl1
lvl2
in1
in2
out
delay
audio_in
time_in
out
xfade
in1
in2
x
out
audio_in
out
out
audio_outin
in
time_in
out
fback_in
out
level
out
drywet
out

18. FlowML code for the reverb diagram
FlowML (8)
<?xml version='1.0'?>
<GraphPrototype name="revbfilt">
<Description>
Reverb (filtered delay).
</Description>
<Implementation>
<Instance id="audio_in" type="audio_in" x="0" y="0">
<Connection to="in1" from="out" targetid="mixer"/>
<Connection to="in1" from="out" targetid="xfade"/>
</Instance>
<Instance id="fback_in" type="control_in" x="0" y="0">
<Connection to="lvl2" from="out" targetid="mixer"/>
</Instance>
<Instance id="level" type="control_in" x="0" y="0">
<Connection to="lvl1" from="out" targetid="mixer"/>
</Instance>
<Instance id="mixer" type="mixer" x="0" y="0">
<Connection to="in" from="out" targetid="dly"/>
</Instance>
<Instance id="time_in" type="control_in" x="0" y="0">
<Connection to="time" from="out" targetid="dly"/>
</Instance>

19. FlowML (9)
FlowML code for the reverb diagram (cont’d)
<Instance id="dly" type="delay" x="0" y="0">
<Connection to="in2" from="out" targetid="xfade"/>
<Connection to="in2" from="out" targetid="mixer"/>
</Instance>
<Instance id="xfade" type="xfade" x="0" y="0">
<Connection to="in" from="out" targetid="audio_out"/>
</Instance>
<Instance id="drywet" type="control_in" x="0" y="0">
<Connection to="x" from="out" targetid="xfade"/>
</Instance>
<Instance id="audio_out" type="audio_out" x="0" y="0"/>
</Implementation>
</GraphPrototype>

20. SAOL Generator (1)
•Originally intended to be a part of
QOrchestra, but now a separate program
•Reads FlowML synthesis diagrams and
generates SAOLcode with the behaviour
of the diagram
•Uses libraries of SAOL opcodes to
simplifygenerated code: every block in
the diagram translates to an opcode call

21. SAOL Generator (2)
•Will support libraries containing FlowML
diagrams and SAOL code (currently only
FlowML)
•Generated code will require preprocessing
(e.g. #include)
•Problem:separation authoring tool (including
synthesizer) & code generator can cause
differences in resulting sound

22. SAOL Generator (3)
•Problem: Integrating software
synthesizer/code generator might cause a
problem: what about intellectual property?

23. SAOL Generator (4)
generateOpcDecl(Prototype, Program, DeclsDone)
VarDecls = newList
Code = newList
InstsDone = newSet
Params = getInputNames(P)
Outputs = getOutputNames(P)
foreachO inOutputs
add(Results, generateOpcResult(Prototype, O, Null, Program,
VarDecls, Code, DeclsDone,
InstsDone))
add(Code, makeReturn(Results))
returnmakeOpcDecl(getRate(getElement(Outputs)),
getName(P), Params, VarDecls, Code)
Code generation: opcode declaration

24. SAOL Generator (5)
generateOpcResult(Prototype, N, C, Program, VarDecls, Code,
DeclsDone, InstsDone)
ifisOutput(Prototype, N)
X = getConnectionTo(N)
returngenerateOpcResult(Prototype, getFrom(X),
getFromConnector(X), Program, VarDecls,
Code, DeclsDone, InstsDone)
elseifisInput(Prototype, N)
returnmakeVarRef(N)
elseifcontains(InstsDone, N)
ifhasMultipleOutputs(N)
returnmakeArrayLookup(last(N), getNumber(N, C))
else
returnmakeVarRef(last(N))
Code generation: opcode result (1/3)
Output instance
Input instance
Instance already calculated

25. SAOL Generator (6)
else
add(InstsDone, N)
ifisGraphPrototype(N) and notcontains(DeclsDone, N)
add(DeclsDone, N)
add(Program, generateOpcDecl(N, Program, DeclsDone))
Params = newList
ConnectionsTo = getConnectionsTo(N)
foreachX inConnectionsTo
add(Params, generateOpcResult(Prototype, getFrom(X),
getFromConnector(X), Program,
VarDecls, Code, DeclsDone,
InstsDone))
A = makeOpcCall(N, Params)
R = getRate(N, C)
S = getNrOutputs(N)
Code generation: opcode result (2/3)
Instance in middle of diagram
Satisfy instance inputs

26. SAOL Generator (7)
if hasMultipleOutputs(N) or hasMultipleConnectionsPerOutput(N)
add(Code, makeAssignment(last(N), next(N)))
add(Code, makeAssignment(next(N), A)
if hasMultipleOutputs(N)
add(Vars, makeArrayDecl(last(N), R, S))
add(Vars, makeArrayDecl(next(N), R, S))
return makeArrayLookup(last(N), getNumber(N, C))
else
if hasMultipleConnectionsPerOutput(N)
add(Vars, makeVarDecl(last(N), R))
add(Vars, makeVarDecl(next(N), R))
return makeVarRef(last(N))
else
return A
Code generation: opcode result (3/3)

27. SAOL Generator (8)
aopcode revbfilt(asig audio_in, ksig drywet, ksig fback_in,
ksig level, ksig time_in)
{
asig dly_next, dly_last;
dly_last = dly_next;
dly_next = delay(mixer(audio_in, level, dly_last, fback_in), time_in);
return xfade(drywet, audio_in, dly_last);
}
Generated SAOL code (reverb diagram)mixer
lvl1
lvl2
in1
in2
out
delay
audio_in
time_in
out
xfade
in1
in2
x
out
audio_in
out
out
audio_outin
in
time_in
out
fback_in
out
level
out
drywet
out

28. The future
•FlowML needs inputto mature and
become really interesting
•Free software (e.g. FlowML parser, SAOL
and CSound code generators, …) needs
to be developed
•Serious authoring tools are needed to
make Mpeg-4 popular
•Artists need to make a difficult transition

29. Conclusion (1)
•FlowML might be interesting as a bridge
between synthesis systems
•FlowML hides much of the engineering side
of audio synthesis and provides a high-level
description format for musicians
•FlowML allows to prototypea synthesizer
quickly before optimizing it (e.g. compile to
SAOL and hand-optimize)

30. Conclusion (2)
•The popularity of SAOL will depend on its
support in commercially available
software synthesis applications

31. References and contact info
•FlowML and QOrchestra
http://www.bertschiettecatte.com/
•Mpeg-4 SA at the MLG, MIT Media Lab
http://sound.media.mit.edu/mpeg4/
•Mpeg-4 SA book at Berkeley
http://www.cs.berkeley.edu/~lazzaro/