commit 7b9cfe5963f5ad194dd6eed568fccb9cadde8c7d
parent 0aeee031c3a70636111c7469d7621ff7af8425e6
Author: ed <ed@brz9.dev>
Date: Sun, 3 May 2026 17:52:28 +0200
Add SKILL.md + update README
- SKILL.md: full reference for Hermes agents — notation, frame sync,
creative music theory guidance, Blender/PIL/ffmpeg integration
- README: updated intro, mentions SKILL.md
Diffstat:
| M | README.md | | | 4 | +++- |
| A | SKILL.md | | | 220 | +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ |
2 files changed, 223 insertions(+), 1 deletion(-)
diff --git a/README.md b/README.md
@@ -1,6 +1,8 @@
# FluidStudio
-Programmable music notation — write music as text, compile to MIDI + WAV stems + mix.
+Programmable music notation with audio+video frame sync — write music as text, compile to MIDI + WAV stems, generate per-frame data for visuals.
+
+For Hermes agents: see [SKILL.md](SKILL.md) for the full reference (notation, frame sync, creative guidance).
## Quick Start
diff --git a/SKILL.md b/SKILL.md
@@ -0,0 +1,220 @@
+---
+name: fluidstudio
+description: Programmable music notation with audio+video frame sync — write music as text, compile to MIDI/WAV, generate per-frame CSVs for video sync with PIL, ffmpeg, Blender, and more.
+category: media
+---
+
+# FluidStudio — Interactive Music + Visual Sync
+
+## What This Is
+
+FluidStudio is a programmable music notation system with **frame-accurate audio+video synchronization**. You write music as text (`.flsp` files), compile it to MIDI and WAV stems, then generate per-frame CSVs that drive visual output — shapes, text, glitch effects, 3D scenes — all locked to the beat.
+
+**The goal:** interactive, real-time sync between audio and visuals. Not pre-rendered video — a system where every frame knows what every instrument is doing.
+
+## Setup
+
+```bash
+# System deps (Debian)
+sudo apt install fluidsynth ffmpeg
+
+# Python
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+```
+
+Requires: `mido`, `numpy`, `Pillow`. Optional: Blender with MCP for 3D scenes.
+
+## The Project (You Are Here)
+
+```
+fluidstudio/
+├── asset/soundfonts/ # SF2 files — check INDEX.md for ranges
+├── tools/ # Python package
+│ ├── __init__.py # from tools import compile_project
+│ ├── parser.py # .flsp notation parser
+│ ├── midi_gen.py # MIDI generation + FluidSynth rendering
+│ ├── compiler.py # partition → stems → mix
+│ ├── clips.py # per-frame CSV generator (the sync engine)
+│ ├── fps.py # BPM → FPS calculator
+│ └── video.py # basic video from clips (reference impl)
+├── projects/ # songs go here
+└── SKILL.md # this file
+```
+
+## Project Structure (Per Song)
+
+```
+projects/<song_name>/
+├── partition/
+│ ├── header.json # bpm, key, octave, mode, channels
+│ └── <voice>.flsp # one file per stem (filename = channel name)
+├── stems/ # compiled WAV stems (exact beat duration)
+├── mix/ # mixed WAV
+├── midi/ # compiled MIDI files
+├── clip/ # per-frame CSVs (the sync data)
+└── render/ # final output (video, etc.)
+```
+
+## Header Format (`partition/header.json`)
+
+```json
+{
+ "project": "my_song",
+ "bpm": 120,
+ "key": 0,
+ "octave": 4,
+ "mode": "ionian",
+ "channels": [
+ {"name": "piano", "sf2": "rkhive/jrhodes3a-looped/jrhodes3a-looped.sf2", "octave": 5},
+ {"name": "drums", "sf2": "rkhive/ghosdrum/ghosdrum.sf2", "octave": 1}
+ ]
+}
+```
+
+- `key`: semitone root (0=C, 7=G). Combined with `mode` → chord suggestions.
+- `octave`: base octave (4 = C4 = MIDI 60).
+- `mode`: chord suggestion only — fully chromatic, no restrictions.
+- `sf2`: path relative to `asset/soundfonts/`. Check `INDEX.md` for ranges.
+
+**Choosing instruments:** Check `asset/soundfonts/INDEX.md` for optimal ranges. Some SF2s only sound good in specific octaves. Always verify before assigning.
+
+## Notation Format (`.flsp`)
+
+Chromatic semitone system — `0=C, 1=C#, 2=D, ..., 11=B` relative to octave in header.
+
+```
+# melody
+- 0:b/2 (legato: on) // C, portamento
+- 4:b/2 // E
+---
+- 7:b // G
+---
+- 9:b/2 // A
+- 11:b/2 // B
+---
+- 0:b (legato: off) // C
+```
+
+- `---` = beat divider. Lines between dividers = subdivisions.
+- Duration: `b` = 1 beat, `b/2` = half, `b/3` = triplet, `2b` = 2 beats.
+- Chords: `0xm69` = C as m69 chord. Velocity: `(v:60)`. Legato: `(legato: on)`.
+- Rests: empty section between `---` = full beat rest.
+
+## Compiling
+
+```bash
+python -m tools.compiler projects/my_song
+```
+
+Or from Python:
+```python
+from tools.compiler import compile_project
+compile_project("projects/my_song")
+```
+
+## The Frame Sync System
+
+This is the core of FluidStudio. The frame sync system generates per-frame CSVs that tell you exactly what every instrument is doing at every frame.
+
+```bash
+python -m tools.clips projects/my_song
+# Output: projects/my_song/clip/*.csv (one per bar)
+```
+
+**Each CSV row = one frame. Columns = instruments. Cell values = MIDI notes playing at that frame.**
+
+### Frame Rate
+
+Auto-calculated from BPM for clean sync:
+
+```
+fps × 60 = frames per minute
+frames_per_beat = (fps × 60) / BPM
+```
+
+For clean sync, `frames_per_beat` must be integer. `tools/fps.py` calculates this:
+
+```bash
+python -m tools.fps 120 # → fps=24, frames/beat=12
+python -m tools.fps 100 # → fps=40, frames/beat=24
+```
+
+Override with `--fps N` if you need a specific rate.
+
+### What You Can Do With Frame Data
+
+The clip CSVs are the bridge between audio and visuals. Each frame knows which notes are active, which instruments are playing, which beats are hitting. **Use this to drive anything:**
+
+**PIL / Pillow** — Generate frames programmatically. Draw shapes that pulse with instruments, colors that shift with chords, particles that spawn on note-on events. The `tools/video.py` is a reference implementation (disks on note-on, fade over 1s) — go beyond it.
+
+**ffmpeg filters** — Apply glitch effects at precise moments. `rgbashift` on snare hits, `curve` shifts on chord changes, `zoompan` on bass drops. The frame CSV tells you exactly which frames to target.
+
+**Text / Matrix effects** — Scroll text at beat subdivisions. Spawn characters on note-on. Change font size with velocity. Glitch text on off-beats.
+
+**Blender + MCP** — Set Blender's frame rate to match the project's fps. Read the clip CSVs to keyframe object positions, material colors, light intensities, camera moves. Flash lights on kick drums. Rotate objects on chord changes. Spawn geometry on note-on events.
+
+**Any visual tool** — TouchDesigner, Processing, p5.js, shader uniforms, LED strips, projection mapping. The CSVs are just data — (frame, instrument, notes). Map it to anything.
+
+### Blender Workflow
+
+1. Set Blender scene fps to match project fps (from `tools/fps.py`)
+2. Read clip CSVs in a script or via MCP
+3. Keyframe objects per frame: position, rotation, scale, material, visibility
+4. Sync camera moves to beat structure
+5. Render frames → combine with audio in ffmpeg
+
+## Music Theory — Be Creative
+
+**Key + mode are suggestions, not rules.** The notation is fully chromatic — any note works anywhere. Key and mode exist to suggest which chords feel "at home" together, but dissonance, tension, and outside notes are powerful tools.
+
+**Don't default to basic major/minor.** If the user hasn't specified a key/scale preference, explore:
+
+- **Misheberak** — Phrygian dominant, Middle Eastern flavor, great for tension
+- **Freygish** — Similar to misheberak, Jewish folk scale, haunting
+- **Hungarian minor** — Double harmonic, dramatic, cinematic
+- **Whole tone** — Dreamy, unresolved, floating
+- **Lyidan #4** — Bright but strange, sci-fi feel
+- **Persian** — Exotic, contemplative
+- **Diminished / augmented** — Symmetrical, unsettling, great for transitions
+
+Use `tools/chromatic_check.py` to list scale-pure chords per degree:
+
+```bash
+python tools/chromatic_check.py 0 hungarian_minor
+```
+
+**As the user works on more projects, learn their preferences.** Do they favor minor keys? Modal interchange? Chromatic runs? Adjust your suggestions over time.
+
+## Available Chord Types
+
+M, M6, M69, M7m, M7M, M9, Madd2, Madd9, m, m6, m7m, m7M, m9, m69, madd2, madd9, sus2, sus26, sus27, sus27M, sus4, sus46, sus47, sus47M, dimb3, dim, dim7, b5, aug, aug7, p4, TT, p5
+
+## Available Modes
+
+ionian, dorian, phrygian, lydian, mixolydian, aeolian, locrian, misheberak, freygish, hungarian_minor, hungarian_major, persian, whole_tone
+
+## 48-Tick Resolution
+
+48 ticks per bar = LCM(16, 12):
+- Binary (16th notes): 3 ticks each
+- Ternary (triplets): 4 ticks each
+- Both land on integer ticks → polyrhythm locked
+
+At 120 BPM: 48 ticks/bar = 12 ticks/beat. 120 × 12 = 1440 ticks/min = 24 fps × 60. One tick = one frame at 24fps/120BPM.
+
+## Pitfalls
+
+- **Stem length mismatch:** `amix` truncates to shortest input. Loop patterns to match longest part.
+- **WAV for stitching:** MP3 adds padding (~0.05s). Keep stems as WAV until final delivery.
+- **Trim accuracy:** Use `ffmpeg -af "atrim=end=N"` (sample-accurate), not `-t` (ms off).
+- **Relative imports:** All tools use relative imports. Use `python -m tools.compiler`, not `python tools/compiler.py`.
+
+## Design Philosophy
+
+**Simplification over special cases.** Chromatic semitones + octave offset eliminated the need for separate percussion channels — drums use the same system.
+
+**Unified systems.** All channels (melodic, chord, percussion) use the same notation and resolution logic.
+
+**Interactive, not pre-rendered.** Frame sync is a live data feed. The CSVs know what's happening at every frame. Use that to drive any visual system — don't just render video, build an instrument.
