aim/README.md

---
theme: gaia
_class: lead
<!-- paginate: true -->
backgroundColor: #000000
color: #B3ACA2
---

# **Towards a universal button grid interface**
## And its open implementation
Thibaud Keller
[ossia.io](https://ossia.io) - [Maynooth University](https://www.maynoothuniversity.ie/) 
*made with [marp](https://marp.app)*

![bg right:40%](images/deluge-slider-2.jpg)
![bg](images/UGS.png)
![bg blur:2px](images/monome-gridlab.jpg)

---

# An enduring practice

- Thoroughly documented at LMU Munich by Beat Rossmy [[1]](https://doi.org/10.21428/92fbeb44.563bfea9),
with Alexander Wiethoff [[2]](https://doi.org/10.21428/92fbeb44.6a2451e6)[[12]](https://doi.org/10.1145/3294109.3295646), Maximilian Rauh[[3]](https://doi.org/10.21428/92fbeb44.db84ecd0)
and Sebastian Unger [[4]](https://doi.org/10.21428/92fbeb44.303223db).

- With a dedicated course at Berkley college of Music
"Grid Controller Studies" ISEL-112 [[5]](https://github.com/stretta/gridlab)

- And an active community of artists and developers [[6]](https://llllllll.co/tag/grid)
Due, in part, to the open source, and open-ended nature of monome's grid

---

# The monome way

![w:540](images/monome-gridlab.jpg) ![h:297](images/grids.jpg)
As presented in [Maker Stories [7]](https://stories.maker.co/monome-open-source-tools-for-music-makers).

<!-- 
* Demo of the MLR max patch 
and a sonified Conway's game of life 

* Made in America
-->

---
# Origins of backlit button grids

![bg vertical right:40% 70%](images/202edited.png)
![bg 70%](images/sp808_gal.jpg)

1998 - SP series (202 & 808)
Performance samplers

The grid allows:
- Sampling and triggering
- Copy and paste
- Step recording

---

<!--
_backgroundColor: #ffffff
_header: 'Annual grid device releases ([data set](https://gothub.ducks.party/BeatRossmy/Musical-Grid-Interfaces)), separated in five product categories'
_footer: 'Figure 2 in Rosmy, B. & Wiethoff, A. “Musical Grid Interfaces: Past, Present, and Future Directions.”  2021[[2]](https://doi.org/10.21428/92fbeb44.6a2451e6)' 
-->
![bg 67%](images/grid_releases.svg)

---

# Notable instruments and sequencers

- 2007 - Tenori-on, Yamaha
- 2014 - BeatStep, Arturia
Circadian Rhythm, Tip Top audio (eurorack)
- 2015 - Circuit, Novation
Pocket operators, Teenage Engineering (hand held)
- 2016 - Deluge, Synthstrom (open source)
- 2017 - Seq, Polyend (wide)
- 2019 - Zoia, Empress (nodal patcher)
- 2023 - Push 3, Ableton (export to and from a PC)

---

# Notable controllers

- 2006 - 40h, monome (open source)
- 2009 - launchpad, Novation
- 2013 - Push, Ableton
- 2014 - Untztrument, Adafruit (open source)
- 2015 - Linstrument, Roger Lynn (continuous surface)
- 2019 - Grid, Intech Studio (modular)
- 2020 - Mine S, Special Waves (modular)

<!--  
An alternative open source firmware for the Novation Launchpad Pro

Interesting, but Ideally, a library should work with any controller out of the box
-->

---

# Problem

- A majority of closed source, proprietary hardware.
- Either a blank slate (monome), or overly standardized practices
- Software specific libraries, with divergent approaches
- Many overlapping features

---
# Limitation

### Menus!

The case of the
Yamaha FS1R [[8]](https://fs1r.skerjanc.de/)
and the dread of
"menu diving"
against 
[Pamela's PRO Workout](https://busycircuits.com/alm034/)

![bg right:60% 110%](images/controller-b-fur-yamaha-fs1r-730x548.jpg)
![bg 26%](images/pamela.jpg)

---

# Button grids can be:

- Keyboards / drum pads
- Step sequencers
- "Clip" launchers (Ardour (Ableton))
- Arrangement views (Deluge)
- Routing matrixes (Deluge)
- Rudimentary waveform croppers (Deluge)
- Text displays (Launchapd)
- Basic spin boxes (Circuit)
- Nodal patchers (Zoia) ...

---

# Tangible timelines

![w:540](images/UGS.png) ![w:540](images/superbrain.jpg)
Samuel J. Hunt's [UGS](https://iv.nboeck.de/watch?v=KHFJPfa38wY) [[9]](https://doi.org/10.1145/3411109.3411122) & Beat Rossmy's [SuperBrain](https://vimeo.com/546873594)

---

# Challenges

- Compatibility: supporting a wide selection of hardware devices
- Performance: low compute cost & optimal responsiveness
- Extensibility: ease of adding more devices
- Low resolution: reducing graphical elements to their bare minimum
- Portability: ease of integration in software
- Modularity: option to aggregate devices

---

#  Approach

- Starting with the Launchpad Pro: common & full-featured 
- Implement a minimalist widget system
- Porting [score](https://ossia.io)'s user interface to a grid
- Drawing from the [remote control plugin](https://ossia.io/score-docs/in-depth/remote.html)
- Search for Intuitive mappings of physical actions to software functionalities
- Aim for full integration of the hardware device, allowing complete use of score without a screen, mouse or keyboard

<!-- 
score contains: 
Independent timelines, 
nodal patchers,
waveform displays, 
and widgets of all kind 

Heavily rely on WIMP interactions
-->

---

# Beyond

![bg vertical right:70% 90%](images/touchgrid.png)
![bg 90%](images/kinephone.png)
Beat Rossmy's [TouchGrid](https://vimeo.com/387381202) [[4]](https://nime.pubpub.org/pub/touchgrid/release/1) & MIT labs [Kinéphone [10]](https://doi.org/10.5281/zenodo.1176145)/[SoundFORMS [11]](https://doi.org/10.1145/2851581.2892414)

<!-- 
edge and basal interactions 

Accessibility: convert light lo cell height
or vibration

small screen in each cell
or dot matrix
or 14 segment displays

displaying characters requires at least 3x5 cell fonts
https://fontstruct.com/fontstructions/show/1371235/3x5-dot-matrix-1
-->

---

# Honorable mention

![bg vertical right:40% 90%](images/impulse_curve.jpg)
![bg 90%](images/comb.png)
Eli Fieldsteel's 
[Impulse Curve](https://inv.nadeko.net/watch?v=6iF5FZ9DAY4&list=PLLnA8B3ZULyHtPZUaIBUtMy3xtxq_uC_I&index=8)[[14]](https://www.icfo.eu/event/3292) 
& Beat Rossmy's 
[COMB](https://vimeo.com/231299236)[[12]](https://doi.org/10.1145/3294109.3295646)

<!-- 
Impulse curve filmed performance
https://inv.nadeko.net/watch?v=k8N0iMYd9H8&t=0 
-->

---

# Bibliography

[[1]](https://doi.org/10.21428/92fbeb44.563bfea9) Rossmy, B. “Buttons, Sliders, and Keys – A Survey on Musical Grid Interface Standards.” In International Conference on New Interfaces for Musical Expression, 2022.

[[2]](https://doi.org/10.21428/92fbeb44.6a2451e6) Rossmy, B., & Wiethoff, A. “Musical Grid Interfaces: Past, Present, and Future Directions.” In International Conference on New Interfaces for Musical Expression, 2021.

[[3]](https://doi.org/10.21428/92fbeb44.db84ecd0) Rossmy, B., Rauh, M., & Wiethoff, A. “Towards User Interface Guidelines for Musical Grid Interfaces.” In International Conference on New Interfaces for Musical Expression, 2022.

---

[[4]](https://doi.org/10.21428/92fbeb44.db84ecd0) Rossmy, B., Unger, S., & Wiethoff, A. “TouchGrid – Combining Touch Interaction with Musical Grid Interfaces.” In International Conference on New Interfaces for Musical Expression, 2021.

[[5]](https://github.com/stretta/gridlab) Matthew Davidson, aka stretta, gridalb patches repository, June 21, 2024. https://github.com/stretta/gridlab.

[[6]](https://llllllll.co/tag/grid) Lines “Latest Grid Topics”. The monome comunity forum. https://llllllll.co/tag/grid.

[[7]](https://stories.maker.co/monome-open-source-tools-for-music-makers) Maker Stories. “Monome / Open Source Tools for Music Makers.” https://stories.maker.co/monome-open-source-tools-for-music-makers.

---

[[8]](https://fs1r.skerjanc.de/) Robert Skerjanc, “HARDWARE-CONTROLLER FOR YAMAHA FS1R,” August 26, 2023. https://fs1r.skerjanc.de/.

[[9]](https://doi.org/10.1145/3411109.3411122) Hunt, Samuel J. “Exploring Polyrhythms, Polymeters, and Polytempi with the Universal Grid Sequencer Framework.” In Proceedings of the 15th International Audio Mostly Conference, 101–6. AM ’20. New York, NY, USA: Association for Computing Machinery, 2020.

---

[[10]](https://doi.org/10.5281/zenodo.1176145) Xiao, Xiao, Donald Derek Haddad, Thomas Sanchez, Akito van Troyer, Rébecca Kleinberger, Penny Webb, Joe Paradiso, Tod Machover, and Hiroshi Ishii. “Kin\’{e}phone: Exploring the Musical Potential of an Actuated Pin-Based Shape Display.” Proceedings of the International Conference on New Interfaces for Musical Expression. June 1, 2016.

[[11]](https://doi.org/10.1145/2851581.2892414) Colter, Aubrey, Patlapa Davivongsa, Donald Derek Haddad, Halla Moore, Brian Tice, and Hiroshi Ishii. “SoundFORMS: Manipulating Sound Through Touch.” In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, 2425–30. CHI EA ’16. New York, NY, USA: Association for Computing Machinery, 2016.

---

[[12]](https://doi.org/10.1145/3294109.3295646) Rossmy, Beat, and Alexander Wiethoff. “COMB -- Shape as a Meaningful Element of Interaction.” In Proceedings of the Thirteenth International Conference on Tangible, Embedded, and Embodied Interaction, 287–95. TEI ’19. New York, NY, USA: Association for Computing Machinery, 2019.

[[13]](https://www.icfo.eu/event/3292)Fieldsteel, Eli. “LightMatrix: A Light-Sensitive Musical Control Interface.” QUANTUM SOUNDS SYMPOSIUM - ICFO (blog). Accessed November 24, 2024.