HomeMAL' // MANUFACTORUM
datacrypt
Blob Opera MIDI / README.md
README.md4.7 KB

Blob Opera MIDI

Drop a MIDI or MusicXML file in your browser and let Google Arts & Culture's Blob Opera sing it — four blobs, four voices. MusicXML lyrics are sung as phonemes.

How it works

Blob Opera plays a recording: a protobuf describing four monophonic voices (soprano, alto, tenor, bass), each a list of (timeSeconds, midiPitch, phoneme) notes. Given a recording id, the embeddable player at gacembed.withgoogle.com/blob-opera/#/r/<id> fetches and performs it.

This app turns a MIDI or MusicXML file into one of those recordings:

  1. Parse in the browser — MIDI (public/lib/midi.js, ticks → seconds via

the tempo map) or MusicXML/.mxl (public/lib/musicxml.js, including a minimal zip inflater for .mxl; ties merge, chords keep their top note).

  1. Arrange into four monophonic voices (public/lib/arrange.js). Two modes:
  • Fold all tracks into 4 voices (MIDI default): every note from every

track is pooled and packed into four monophonic lines, ordered soprano→bass. Nothing is discarded unless more than four notes sound at the same instant — which four blobs physically cannot sing — and the UI reports exactly how many were dropped.

  • Manual: assign each track to a voice yourself (used for MusicXML, or

MIDI with the fold toggle off). Overlapping notes in one voice flatten to the highest; a 1–2 track file is cloned up to four voices so it still sings.

  1. Sing lyrics (public/lib/lyrics.js): MusicXML lyric syllables are

matched greedily against Blob Opera's phoneme alphabet; a note must start on a vowel, so leading consonants migrate to the previous note's suffix, multi-vowel syllables subdivide the note, and lyric-less notes melisma on the last vowel. MIDI has no lyrics, so everything is sung on "ah".

  1. Encode the RecordingMessage protobuf by hand (public/lib/protobuf.js).
  2. Upload the bytes and load the returned id into an <iframe>.

No backend — it's a static page

Everything, including the upload, runs in the browser. There is one wrinkle: Google's recording endpoint (cilex-aeiopera.uc.r.appspot.com/recording) sends no Access-Control-Allow-Origin header, so the response — which carries the recording id — is unreadable by a direct cross-origin fetch. So the PUT is bounced through a public CORS relay (public/app.js, tried in order with fallback). Only the recording bytes (musical notes, no personal data) pass through. If Google ever adds CORS, the relay can be dropped for a direct call.

The main experiment page can't be iframed (X-Frame-Options: SAMEORIGIN), which is why the player is the gacembed.withgoogle.com embed — it sets no such header.

Run it

It's a static page — open it from any static file server (browsers block ES module imports over file://, so you do need a server, any will do):

npm start                       # tiny static server, http://localhost:8080
# or:  python3 -m http.server -d public 8080

Drop a .mid, .musicxml or .mxl file, assign each track to a voice (auto-guessed by pitch), and hit Sing it.

As a site tool

Published in the wider repository under site/tools/blob_opera_midi/ (a copy of public/), linked from site/tools/index.html. Being fully static, it needs no special server route — it runs anywhere the site is hosted. Edit the sources here and re-copy public/ into the tool folder to update the published copy.

Test

npm test              # synthesize a MIDI, run the full pipeline, upload it
npm test -- --no-network   # skip the live upload

Notes & limits

  • Best results come from SATB arrangements — one melodic line per track/part.
  • Blob Opera has exactly four singers, all monophonic. Fold mode keeps every

note up to four-way simultaneity; beyond that, extra notes are dropped and counted. Manual mode reduces polyphony within a voice to the top note.

  • MIDI files have no lyrics, so they are sung on an open "ah" vowel. MusicXML

lyrics are converted with a generic letter-to-phoneme mapping (no real language model) — letters outside the phoneme alphabet (k, w, x) are dropped.

  • Multi-voice MusicXML staves use the first voice only.
  • Recordings are uploaded to Google's public Blob Opera backend to obtain a

shareable link, exactly as the official "share" button does.

Credits

by David Li and Google Arts & Culture.

and blob-opera-midi.