b1rdmania
4fdc6984bc
- Improved search results table with detailed metadata display - Added functionality to preview original MIDI alongside generated motifs - Implemented automatic selection of the best result with visual confidence indicators - Enhanced user experience with a "Try Next Result" button for A/B testing These updates provide users with better insights into MIDI quality and facilitate easier comparisons between original and generated compositions.
164 lines
6.8 KiB
Markdown
164 lines
6.8 KiB
Markdown
## MIDI playback MVP — detailed technical integration plan
|
||
|
||
### Product target (MVP)
|
||
- **User flow**:
|
||
1) User types a song name (e.g. “Hotel California”)
|
||
2) App searches MIDI sources and shows a ranked list
|
||
3) User selects a result and can **play the MIDI “correctly”** using **General MIDI soundfonts**
|
||
4) User can then click **Generate Motif** to synthesize a “similar-but-different” version from the same parsed MIDI
|
||
|
||
### Current code reality (gaps to close)
|
||
- **Search is currently biased toward mock/synthetic**, not real MIDI:
|
||
- `server/src/services/MIDISearchService.ts` includes `MockAdapter` first.
|
||
- **Fetch can silently replace real URLs with synthetic MIDI**:
|
||
- `server/src/services/MIDIFetchService.ts` generates synthetic MIDI when URL contains `bitmidi.com/uploads`, preventing true BitMidi playback.
|
||
- **Frontend preview is oscillator-based**, not GM soundfont playback.
|
||
|
||
---
|
||
|
||
## Architecture (what we’ll ship)
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
User -->|typesQuery| FrontendUI
|
||
FrontendUI -->|GET /api/midi/search?q=...| BackendSearch
|
||
BackendSearch -->|rankedResults| FrontendUI
|
||
FrontendUI -->|selectResult + GET /api/midi/fetch?u=...| BackendFetch
|
||
BackendFetch -->|midiBytes| FrontendParse
|
||
FrontendParse -->|NoteEvents + TrackMeta| PreviewPlayerGM
|
||
FrontendParse -->|NoteEvents| MotifEngine
|
||
MotifEngine -->|roles + chords| SynthesisEngine
|
||
PreviewPlayerGM --> AudioOut
|
||
SynthesisEngine --> AudioOut
|
||
```
|
||
|
||
---
|
||
|
||
## Phase 0 — “Real MIDI mode” defaults (backend hardening)
|
||
|
||
### 0.1 Gate mock adapter behind env flag
|
||
- **Change**: In `server/src/services/MIDISearchService.ts`, make adapters:
|
||
- Default: `[BitMidiAdapter, DongraysAdapter]`
|
||
- Optional: prepend `MockAdapter` only if `USE_MOCK_ADAPTER=1` (or similar)
|
||
- **Acceptance**:
|
||
- Searching “Hotel California” returns **non-`synthetic:*`** results when internet is available.
|
||
- Devs can still run offline with `USE_MOCK_ADAPTER=1`.
|
||
|
||
### 0.2 Stop auto-synth overriding real URLs in fetch
|
||
- **Change**: In `server/src/services/MIDIFetchService.ts`, only synthesize when:
|
||
- `url.startsWith('synthetic:')` (and optionally if `USE_SYNTHETIC_FETCH=1`)
|
||
- **Remove**: `url.includes('bitmidi.com/uploads')` synthetic shortcut
|
||
- **Acceptance**:
|
||
- Selecting a BitMidi result triggers a real network fetch and caches the real bytes.
|
||
- If the remote file is invalid, it fails explicitly with a clear error.
|
||
|
||
### 0.3 Make backend behavior explicit in responses (optional but recommended)
|
||
- **Change**: Add response fields or headers indicating source:
|
||
- Example: `X-Motif-Source: real|synthetic|cache`
|
||
- **Acceptance**:
|
||
- Frontend can display “cached”/“live”/“synthetic fallback” badges (helps debugging + trust).
|
||
|
||
---
|
||
|
||
## Phase 1 — GM soundfont playback (frontend “Preview” becomes correct MIDI playback)
|
||
|
||
### 1.1 Dependency choice & asset strategy
|
||
- **Dependency**: add a browser-friendly soundfont player dependency (e.g. `soundfont-player`).
|
||
- **Soundfont hosting**:
|
||
- Prefer static hosting under `/public/soundfonts/` (versioned with the app)
|
||
- Or use a CDN, but pin versions and handle CORS
|
||
- **MVP instrument set**:
|
||
- Minimum viable: **Acoustic Grand Piano** for all melodic tracks, and a basic drum fallback
|
||
- Better: load instruments on demand per track program
|
||
|
||
### 1.2 Implement `SoundfontMIDIPlayer`
|
||
Create `src/synthesis/SoundfontMIDIPlayer.ts` with:
|
||
- **Responsibilities**
|
||
- Load instruments (program → soundfont instrument name)
|
||
- Schedule note-on/note-off with WebAudio timing
|
||
- Provide `load(midi)` / `play()` / `stop()` / `setVolume()` APIs
|
||
- **Inputs**
|
||
- Best: use `@tonejs/midi`’s `Midi` object (tracks include `instrument.number`, `notes`, `channel`)
|
||
- Alternate: keep using `NoteEvent[]`, but you’ll lose program/channel unless you extend the event model
|
||
- **Timing correctness**
|
||
- Use seconds-based timing from `@tonejs/midi` notes (`time`, `duration` are in seconds)
|
||
- Ensure AudioContext resumes on user gesture
|
||
- **Drums**
|
||
- If channel 9/10 is detected: either map to a percussion kit if supported, or skip drums for MVP (but be explicit in UI)
|
||
|
||
### 1.3 Wire UI to use soundfont preview
|
||
- In `src/main.ts`, replace/augment the current oscillator `MIDIPlayer` usage:
|
||
- Preview buttons should play via `SoundfontMIDIPlayer`
|
||
- Keep existing Motif buttons intact
|
||
- Keep oscillator preview only as a fallback if soundfonts fail to load (optional).
|
||
|
||
---
|
||
|
||
## Phase 2 — “Play the real song correctly” UX loop
|
||
|
||
### 2.1 Search UX requirements
|
||
- **Result list must show**:
|
||
- title, source, confidence
|
||
- parsed metadata: duration, track count, issues
|
||
- **Selection behavior**:
|
||
- Selecting a row fetches + parses once; enables Preview + Generate Motif
|
||
|
||
### 2.2 Error handling (user-facing)
|
||
Define user-facing error classes/messages:
|
||
- **Search**: “No results”, “Backend unavailable”, “Rate limited / source blocked”
|
||
- **Fetch**: “MIDI file blocked”, “Invalid MIDI header”, “Quality rejected”
|
||
- **Parse**: “Unsupported MIDI features” / “Parse failed”
|
||
- **Preview playback**: “Soundfont failed to load” / “Audio not allowed until click”
|
||
|
||
### 2.3 Observability (dev-facing)
|
||
- Backend: log adapter failures per source + timings
|
||
- Frontend: log selected MIDI URL, parse duration, instrument load times
|
||
- Optional: a small “Debug” accordion showing chosen URL, cache hit, parse issues, loaded instruments
|
||
|
||
---
|
||
|
||
## Phase 3 — Motif generation stays step 2 (but align data model)
|
||
- Keep the current path:
|
||
- `MotifEngine.generateFromMIDI(NoteEvent[])` then `MotifEngine.play()`
|
||
- Recommended alignment work:
|
||
- Decide whether Motif should later consume richer track metadata (program/channel) to improve role mapping.
|
||
|
||
---
|
||
|
||
## Integration milestones & acceptance checks
|
||
|
||
### Milestone A — Real MIDI end-to-end
|
||
- Search returns results from BitMidi/Dongrays with mock disabled by default
|
||
- Fetch returns real bytes and caches them
|
||
- Parse endpoint `/api/midi/parse` works for metadata
|
||
|
||
### Milestone B — “Correct” preview playback
|
||
- Preview produces recognizable instrument playback (piano at minimum)
|
||
- Stop reliably stops scheduled notes
|
||
- Works in Chrome/Safari with autoplay policies (requires click)
|
||
|
||
### Milestone C — Motif as second step
|
||
- Generate Motif still works on the same loaded MIDI
|
||
- Preview and Motif can be A/B tested without reloading the page
|
||
|
||
---
|
||
|
||
## Recommended team task breakdown
|
||
|
||
### Backend engineer
|
||
- Implement env gating for mock/synthetic
|
||
- Tighten fetch behavior + ensure BitMidi URLs are truly fetched
|
||
- Add explicit “source = real/cache/synthetic” marker
|
||
|
||
### Frontend engineer
|
||
- Add the chosen soundfont library
|
||
- Implement `SoundfontMIDIPlayer`
|
||
- Wire `src/main.ts` preview buttons to soundfont playback
|
||
- Add user-facing error messaging for soundfont failures
|
||
|
||
### QA / test harness
|
||
- Maintain a short list of known-good queries (3–5 songs) and confirm:
|
||
- search results appear
|
||
- at least one MIDI fetches and plays
|
||
- Motif plays afterward
|