fluidstudio

SKILL.md (8955B)

---

 ---
 name: fluidstudio
 description: Programmable music notation with audio+video frame sync — write music as text, compile to MIDI/WAV, generate per-frame CSVs for video sync with PIL, ffmpeg, Blender, and more.
 category: media
 ---
 
 # FluidStudio — Interactive Music + Visual Sync
 
 ## What This Is
 
 FluidStudio is a programmable music notation system with **frame-accurate audio+video synchronization**. You write music as text (`.flsp` files), compile it to MIDI and WAV stems, then generate per-frame CSVs that drive visual output — shapes, text, glitch effects, 3D scenes — all locked to the beat.
 
 **The goal:** interactive, real-time sync between audio and visuals. Not pre-rendered video — a system where every frame knows what every instrument is doing.
 
 ## Setup
 
 ```bash
 # System deps (Debian)
 sudo apt install fluidsynth ffmpeg
 
 # Python
 python3 -m venv .venv
 source .venv/bin/activate
 pip install -r requirements.txt
 ```
 
 Requires: `mido`, `numpy`, `Pillow`. Optional: Blender with MCP for 3D scenes.
 
 ## The Project (You Are Here)
 
 ```
 fluidstudio/
 ├── asset/soundfonts/        # SF2 files — check INDEX.md for ranges
 ├── tools/                   # Python package
 │   ├── __init__.py          # from tools import compile_project
 │   ├── parser.py            # .flsp notation parser
 │   ├── midi_gen.py          # MIDI generation + FluidSynth rendering
 │   ├── compiler.py          # partition → stems → mix
 │   ├── clips.py             # per-frame CSV generator (the sync engine)
 │   ├── fps.py               # BPM → FPS calculator
 │   └── video.py             # basic video from clips (reference impl)
 ├── projects/                # songs go here
 └── SKILL.md                 # this file
 ```
 
 ## Project Structure (Per Song)
 
 ```
 projects/<song_name>/
 ├── partition/
 │   ├── header.json          # bpm, key, octave, mode, channels
 │   └── <voice>.flsp         # one file per stem (filename = channel name)
 ├── stems/                   # compiled WAV stems (exact beat duration)
 ├── mix/                     # mixed WAV
 ├── midi/                    # compiled MIDI files
 ├── clip/                    # per-frame CSVs (the sync data)
 └── render/                  # final output (video, etc.)
 ```
 
 ## Header Format (`partition/header.json`)
 
 ```json
 {
   "project": "my_song",
   "bpm": 120,
   "key": 0,
   "octave": 4,
   "mode": "ionian",
   "channels": [
     {"name": "piano", "sf2": "rkhive/jrhodes3a-looped/jrhodes3a-looped.sf2", "octave": 5},
     {"name": "drums", "sf2": "rkhive/ghosdrum/ghosdrum.sf2", "octave": 1}
   ]
 }
 ```
 
 - `key`: semitone root (0=C, 7=G). Combined with `mode` → chord suggestions.
 - `octave`: base octave (4 = C4 = MIDI 60).
 - `mode`: chord suggestion only — fully chromatic, no restrictions.
 - `sf2`: path relative to `asset/soundfonts/`. Check `INDEX.md` for ranges.
 
 **Choosing instruments:** Check `asset/soundfonts/INDEX.md` for optimal ranges. Some SF2s only sound good in specific octaves. Always verify before assigning.
 
 ## Notation Format (`.flsp`)
 
 Chromatic semitone system — `0=C, 1=C#, 2=D, ..., 11=B` relative to octave in header.
 
 ```
 # melody
 - 0:b/2 (legato: on)  // C, portamento
 - 4:b/2               // E
 ---
 - 7:b                 // G
 ---
 - 9:b/2               // A
 - 11:b/2              // B
 ---
 - 0:b (legato: off)   // C
 ```
 
 - `---` = beat divider. Lines between dividers = subdivisions.
 - Duration: `b` = 1 beat, `b/2` = half, `b/3` = triplet, `2b` = 2 beats.
 - Chords: `0xm69` = C as m69 chord. Velocity: `(v:60)`. Legato: `(legato: on)`.
 - Rests: empty section between `---` = full beat rest.
 
 ## Compiling
 
 ```bash
 python -m tools.compiler projects/my_song
 ```
 
 Or from Python:
 ```python
 from tools.compiler import compile_project
 compile_project("projects/my_song")
 ```
 
 ## The Frame Sync System
 
 This is the core of FluidStudio. The frame sync system generates per-frame CSVs that tell you exactly what every instrument is doing at every frame.
 
 ```bash
 python -m tools.clips projects/my_song
 # Output: projects/my_song/clip/*.csv (one per bar)
 ```
 
 **Each CSV row = one frame. Columns = instruments. Cell values = MIDI notes playing at that frame.**
 
 ### Frame Rate
 
 Auto-calculated from BPM for clean sync:
 
 ```
 fps × 60 = frames per minute
 frames_per_beat = (fps × 60) / BPM
 ```
 
 For clean sync, `frames_per_beat` must be integer. `tools/fps.py` calculates this:
 
 ```bash
 python -m tools.fps 120    # → fps=24, frames/beat=12
 python -m tools.fps 100    # → fps=40, frames/beat=24
 ```
 
 Override with `--fps N` if you need a specific rate.
 
 ### What You Can Do With Frame Data
 
 The clip CSVs are the bridge between audio and visuals. Each frame knows which notes are active, which instruments are playing, which beats are hitting. **Use this to drive anything:**
 
 **PIL / Pillow** — Generate frames programmatically. Draw shapes that pulse with instruments, colors that shift with chords, particles that spawn on note-on events. The `tools/video.py` is a reference implementation (disks on note-on, fade over 1s) — go beyond it.
 
 **ffmpeg filters** — Apply glitch effects at precise moments. `rgbashift` on snare hits, `curve` shifts on chord changes, `zoompan` on bass drops. The frame CSV tells you exactly which frames to target.
 
 **Text / Matrix effects** — Scroll text at beat subdivisions. Spawn characters on note-on. Change font size with velocity. Glitch text on off-beats.
 
 **Blender + MCP** — Set Blender's frame rate to match the project's fps. Read the clip CSVs to keyframe object positions, material colors, light intensities, camera moves. Flash lights on kick drums. Rotate objects on chord changes. Spawn geometry on note-on events.
 
 **Any visual tool** — TouchDesigner, Processing, p5.js, shader uniforms, LED strips, projection mapping. The CSVs are just data — (frame, instrument, notes). Map it to anything.
 
 ### Blender Workflow
 
 1. Set Blender scene fps to match project fps (from `tools/fps.py`)
 2. Read clip CSVs in a script or via MCP
 3. Keyframe objects per frame: position, rotation, scale, material, visibility
 4. Sync camera moves to beat structure
 5. Render frames → combine with audio in ffmpeg
 
 ## Music Theory — Be Creative
 
 **Key + mode are suggestions, not rules.** The notation is fully chromatic — any note works anywhere. Key and mode exist to suggest which chords feel "at home" together, but dissonance, tension, and outside notes are powerful tools.
 
 **Don't default to basic major/minor.** If the user hasn't specified a key/scale preference, explore:
 
 - **Misheberak** — Phrygian dominant, Middle Eastern flavor, great for tension
 - **Freygish** — Similar to misheberak, Jewish folk scale, haunting
 - **Hungarian minor** — Double harmonic, dramatic, cinematic
 - **Whole tone** — Dreamy, unresolved, floating
 - **Lyidan #4** — Bright but strange, sci-fi feel
 - **Persian** — Exotic, contemplative
 - **Diminished / augmented** — Symmetrical, unsettling, great for transitions
 
 Use `tools/chromatic_check.py` to list scale-pure chords per degree:
 
 ```bash
 python tools/chromatic_check.py 0 hungarian_minor
 ```
 
 **As the user works on more projects, learn their preferences.** Do they favor minor keys? Modal interchange? Chromatic runs? Adjust your suggestions over time.
 
 ## Available Chord Types
 
 M, M6, M69, M7m, M7M, M9, Madd2, Madd9, m, m6, m7m, m7M, m9, m69, madd2, madd9, sus2, sus26, sus27, sus27M, sus4, sus46, sus47, sus47M, dimb3, dim, dim7, b5, aug, aug7, p4, TT, p5
 
 ## Available Modes
 
 ionian, dorian, phrygian, lydian, mixolydian, aeolian, locrian, misheberak, freygish, hungarian_minor, hungarian_major, persian, whole_tone
 
 ## 48-Tick Resolution
 
 48 ticks per bar = LCM(16, 12):
 - Binary (16th notes): 3 ticks each
 - Ternary (triplets): 4 ticks each
 - Both land on integer ticks → polyrhythm locked
 
 At 120 BPM: 48 ticks/bar = 12 ticks/beat. 120 × 12 = 1440 ticks/min = 24 fps × 60. One tick = one frame at 24fps/120BPM.
 
 ## Pitfalls
 
 - **Stem length mismatch:** `amix` truncates to shortest input. Loop patterns to match longest part.
 - **WAV for stitching:** MP3 adds padding (~0.05s). Keep stems as WAV until final delivery.
 - **Trim accuracy:** Use `ffmpeg -af "atrim=end=N"` (sample-accurate), not `-t` (ms off).
 - **Relative imports:** All tools use relative imports. Use `python -m tools.compiler`, not `python tools/compiler.py`.
 
 ## Design Philosophy
 
 **Simplification over special cases.** Chromatic semitones + octave offset eliminated the need for separate percussion channels — drums use the same system.
 
 **Unified systems.** All channels (melodic, chord, percussion) use the same notation and resolution logic.
 
 **Interactive, not pre-rendered.** Frame sync is a live data feed. The CSVs know what's happening at every frame. Use that to drive any visual system — don't just render video, build an instrument.