Enhance MIDI search results and dual player UI

- Improved search results table with detailed metadata display - Added functionality to preview original MIDI alongside generated motifs - Implemented automatic selection of the best result with visual confidence indicators - Enhanced user experience with a "Try Next Result" button for A/B testing These updates provide users with better insights into MIDI quality and facilitate easier comparisons between original and generated compositions.
2025-12-18 12:18:44 +00:00
parent 177e8fbed3
commit 4fdc6984bc
2 changed files with 199 additions and 0 deletions
@@ -0,0 +1,163 @@
+## MIDI playback MVP — detailed technical integration plan
+
+### Product target (MVP)
+- **User flow**:
+  1) User types a song name (e.g. “Hotel California”)
+  2) App searches MIDI sources and shows a ranked list
+  3) User selects a result and can **play the MIDI “correctly”** using **General MIDI soundfonts**
+  4) User can then click **Generate Motif** to synthesize a “similar-but-different” version from the same parsed MIDI
+
+### Current code reality (gaps to close)
+- **Search is currently biased toward mock/synthetic**, not real MIDI:
+  - `server/src/services/MIDISearchService.ts` includes `MockAdapter` first.
+- **Fetch can silently replace real URLs with synthetic MIDI**:
+  - `server/src/services/MIDIFetchService.ts` generates synthetic MIDI when URL contains `bitmidi.com/uploads`, preventing true BitMidi playback.
+- **Frontend preview is oscillator-based**, not GM soundfont playback.
+
+---
+
+## Architecture (what we’ll ship)
+
+```mermaid
+flowchart LR
+User -->|typesQuery| FrontendUI
+FrontendUI -->|GET /api/midi/search?q=...| BackendSearch
+BackendSearch -->|rankedResults| FrontendUI
+FrontendUI -->|selectResult + GET /api/midi/fetch?u=...| BackendFetch
+BackendFetch -->|midiBytes| FrontendParse
+FrontendParse -->|NoteEvents + TrackMeta| PreviewPlayerGM
+FrontendParse -->|NoteEvents| MotifEngine
+MotifEngine -->|roles + chords| SynthesisEngine
+PreviewPlayerGM --> AudioOut
+SynthesisEngine --> AudioOut
+```
+
+---
+
+## Phase 0 — “Real MIDI mode” defaults (backend hardening)
+
+### 0.1 Gate mock adapter behind env flag
+- **Change**: In `server/src/services/MIDISearchService.ts`, make adapters:
+  - Default: `[BitMidiAdapter, DongraysAdapter]`
+  - Optional: prepend `MockAdapter` only if `USE_MOCK_ADAPTER=1` (or similar)
+- **Acceptance**:
+  - Searching “Hotel California” returns **non-`synthetic:*`** results when internet is available.
+  - Devs can still run offline with `USE_MOCK_ADAPTER=1`.
+
+### 0.2 Stop auto-synth overriding real URLs in fetch
+- **Change**: In `server/src/services/MIDIFetchService.ts`, only synthesize when:
+  - `url.startsWith('synthetic:')` (and optionally if `USE_SYNTHETIC_FETCH=1`)
+- **Remove**: `url.includes('bitmidi.com/uploads')` synthetic shortcut
+- **Acceptance**:
+  - Selecting a BitMidi result triggers a real network fetch and caches the real bytes.
+  - If the remote file is invalid, it fails explicitly with a clear error.
+
+### 0.3 Make backend behavior explicit in responses (optional but recommended)
+- **Change**: Add response fields or headers indicating source:
+  - Example: `X-Motif-Source: real|synthetic|cache`
+- **Acceptance**:
+  - Frontend can display “cached”/“live”/“synthetic fallback” badges (helps debugging + trust).
+
+---
+
+## Phase 1 — GM soundfont playback (frontend “Preview” becomes correct MIDI playback)
+
+### 1.1 Dependency choice & asset strategy
+- **Dependency**: add a browser-friendly soundfont player dependency (e.g. `soundfont-player`).
+- **Soundfont hosting**:
+  - Prefer static hosting under `/public/soundfonts/` (versioned with the app)
+  - Or use a CDN, but pin versions and handle CORS
+- **MVP instrument set**:
+  - Minimum viable: **Acoustic Grand Piano** for all melodic tracks, and a basic drum fallback
+  - Better: load instruments on demand per track program
+
+### 1.2 Implement `SoundfontMIDIPlayer`
+Create `src/synthesis/SoundfontMIDIPlayer.ts` with:
+- **Responsibilities**
+  - Load instruments (program → soundfont instrument name)
+  - Schedule note-on/note-off with WebAudio timing
+  - Provide `load(midi)` / `play()` / `stop()` / `setVolume()` APIs
+- **Inputs**
+  - Best: use `@tonejs/midi`’s `Midi` object (tracks include `instrument.number`, `notes`, `channel`)
+  - Alternate: keep using `NoteEvent[]`, but you’ll lose program/channel unless you extend the event model
+- **Timing correctness**
+  - Use seconds-based timing from `@tonejs/midi` notes (`time`, `duration` are in seconds)
+  - Ensure AudioContext resumes on user gesture
+- **Drums**
+  - If channel 9/10 is detected: either map to a percussion kit if supported, or skip drums for MVP (but be explicit in UI)
+
+### 1.3 Wire UI to use soundfont preview
+- In `src/main.ts`, replace/augment the current oscillator `MIDIPlayer` usage:
+  - Preview buttons should play via `SoundfontMIDIPlayer`
+  - Keep existing Motif buttons intact
+- Keep oscillator preview only as a fallback if soundfonts fail to load (optional).
+
+---
+
+## Phase 2 — “Play the real song correctly” UX loop
+
+### 2.1 Search UX requirements
+- **Result list must show**:
+  - title, source, confidence
+  - parsed metadata: duration, track count, issues
+- **Selection behavior**:
+  - Selecting a row fetches + parses once; enables Preview + Generate Motif
+
+### 2.2 Error handling (user-facing)
+Define user-facing error classes/messages:
+- **Search**: “No results”, “Backend unavailable”, “Rate limited / source blocked”
+- **Fetch**: “MIDI file blocked”, “Invalid MIDI header”, “Quality rejected”
+- **Parse**: “Unsupported MIDI features” / “Parse failed”
+- **Preview playback**: “Soundfont failed to load” / “Audio not allowed until click”
+
+### 2.3 Observability (dev-facing)
+- Backend: log adapter failures per source + timings
+- Frontend: log selected MIDI URL, parse duration, instrument load times
+- Optional: a small “Debug” accordion showing chosen URL, cache hit, parse issues, loaded instruments
+
+---
+
+## Phase 3 — Motif generation stays step 2 (but align data model)
+- Keep the current path:
+  - `MotifEngine.generateFromMIDI(NoteEvent[])` then `MotifEngine.play()`
+- Recommended alignment work:
+  - Decide whether Motif should later consume richer track metadata (program/channel) to improve role mapping.
+
+---
+
+## Integration milestones & acceptance checks
+
+### Milestone A — Real MIDI end-to-end
+- Search returns results from BitMidi/Dongrays with mock disabled by default
+- Fetch returns real bytes and caches them
+- Parse endpoint `/api/midi/parse` works for metadata
+
+### Milestone B — “Correct” preview playback
+- Preview produces recognizable instrument playback (piano at minimum)
+- Stop reliably stops scheduled notes
+- Works in Chrome/Safari with autoplay policies (requires click)
+
+### Milestone C — Motif as second step
+- Generate Motif still works on the same loaded MIDI
+- Preview and Motif can be A/B tested without reloading the page
+
+---
+
+## Recommended team task breakdown
+
+### Backend engineer
+- Implement env gating for mock/synthetic
+- Tighten fetch behavior + ensure BitMidi URLs are truly fetched
+- Add explicit “source = real/cache/synthetic” marker
+
+### Frontend engineer
+- Add the chosen soundfont library
+- Implement `SoundfontMIDIPlayer`
+- Wire `src/main.ts` preview buttons to soundfont playback
+- Add user-facing error messaging for soundfont failures
+
+### QA / test harness
+- Maintain a short list of known-good queries (3–5 songs) and confirm:
+  - search results appear
+  - at least one MIDI fetches and plays
+  - Motif plays afterward
@@ -0,0 +1,36 @@
+# MOTIF — Aims, Scope, and Guidelines (for AI agents)
+
+## Aims (what we’re trying to achieve)
+- **Primary aim (MVP)**: Let a user search any song name, find real MIDI files online, **load one**, and **play it back in-browser** in a way that feels like a proper MIDI performance.
+- **Secondary aim (next step)**: From the same MIDI, generate a **similar-but-different** “Motif” version and play that.
+- **Future aim (later)**: Add stylistic transforms (“more ominous”, “dance/remix”), controls, and improved musical intelligence.
+
+## Scope (what is in-bounds right now)
+- **Search + ranking** of MIDI candidates from multiple sources
+- **Fetch + validate + cache** MIDI bytes via the backend
+- **Parse** MIDI and show basic metadata (duration, track count, issues)
+- **Playback of the fetched MIDI** (this is the current priority)
+- Keep Motif generation working, but don’t block MVP playback on it
+
+## Non-scope (avoid for now)
+- Perfect matching/similarity, chord/key analysis, advanced arrangement
+- Big infra (DB/CDN/monitoring/analytics), large refactors
+- New features unrelated to search/fetch/parse/playback reliability
+
+## Guidelines (how to work in this repo)
+- **Bias toward real MIDI**: don’t silently replace real results with synthetic/mock in the default user path.
+- **Make degradation explicit**: if mock/synthetic is used, it must be clearly indicated (UI/logs).
+- **Keep changes incremental**: prioritize reliable end-to-end playback over architecture rewrites.
+- **Respect browser audio constraints**: playback must work with autoplay policies (user gesture → resume AudioContext).
+- **Add practical observability**: when fixing issues, add minimal logs/errors that explain which step failed (search vs fetch vs parse vs playback).
+- **Definition of done for any PR**:
+  - Search returns results for common queries
+  - Selecting a result fetches bytes successfully (or shows a clear error)
+  - Playback starts/stops reliably without breaking subsequent plays
+
+## “Done means demo-able”
+A change is successful if someone can:
+- search “Hotel California”
+- select a result
+- hit Play and hear it
+- hit Stop and try another result