Multimodal music recommendation / cross-modal ML
MelodyMind
A multimodal AI music companion whose recommendation core aligns a frozen CLAP audio encoder to a text embedding space with InfoNCE contrastive learning — arrived at only after a naive baseline collapsed and the failure was diagnosed.
The problem
Emotion-aware music recommendation is hard because emotional intent lives in text ("something for a rainy focus session") while the signal lives in audio, and the two have no shared representation. The naive first attempts — text sentiment classifiers over lyrics and Reddit, spectrogram CNNs, concatenated audio-text pipelines — suffered feature collapse and rode on imbalanced data, producing embeddings that did not separate by emotion. The core question became how to align audio and text into one space reliably enough to recommend across modalities.
Constraints
- Academic thesis with no labeled emotion-to-audio ground truth — training data had to be assembled from public sources (Reddit, Last.fm, Genius lyrics, Deezer 30-second previews) with real class imbalance.
- Audio access limited to short public previews, not full tracks.
- A three-person team on a fixed academic timeline, iterating from research toward a demonstrable prototype.
- No compute budget to train an audio encoder from scratch — the design had to lean on a frozen pretrained encoder and train only a lightweight head.
Approach & key decisions
Freeze a pretrained CLAP audio encoder and train only a small projection head into the Nomic text embedding space, rather than training an audio model end to end.
It reuses CLAP's audio understanding, needs far less data and compute, and reduces the alignment problem to learning one projection — the tractable path given no ground-truth dataset.
Align modalities with an InfoNCE in-batch contrastive loss instead of a regression or classification objective.
Contrastive in-batch pairing optimizes both audio-text matching and song-to-song discrimination at once, which is what fixed the feature collapse the earlier baselines exhibited.
Run a structured zero-shot CLAP evaluation across ten emotion prompts before committing to fine-tuning.
It surfaced an honest split — the pretrained model handled direct emotions, tempo, and timbre but was weak on abstract/situational cues like 'romantic dinner' or 'focus' — which is the evidence that justified the projection-head approach rather than assuming it.
Build recommendation on weighted embedding centroids in Pinecone with mood filtering by vector distance.
Once audio and text share a space, playlist construction and mood filtering reduce to vector geometry, keeping the serving path simple and inspectable.
Route the conversational layer through Gemini with GROQ LLaMA 3 as a latency fallback.
The real-time voice loop needs low latency; a fallback protects responsiveness when the primary model is slow.
What I built
- Trained a lightweight fully-connected projection head mapping frozen CLAP audio embeddings into the dimensionality of pre-computed Nomic text embeddings, enabling direct audio-text similarity search.
- Built the earlier naive baselines (lyric/Reddit sentiment classifiers, spectrogram CNNs, concatenated audio-text pipelines) that exposed the data imbalance and feature collapse motivating the CLAP approach.
- Assembled a data pipeline over Reddit (emotional language), Last.fm (mood/genre tags), Genius lyrics, and Deezer previews with concurrent download and feature extraction.
- Served the system from a service-oriented FastAPI backend with async SQLAlchemy and Alembic migrations, uploading aligned embeddings to Pinecone for similarity search and centroid-based playlist construction.
- Implemented the 'Talk to Your DJ' loop: audio → Whisper (via GROQ) transcript → LangChain agent → playlist update → streamed TTS reply, adapting the playlist mid-session as the user's stated mood changes.
- Added image-to-playlist support via multimodal LLM mood extraction mapped to embedding queries.
- Built a Demucs-based stem separation service with an asynchronous job lifecycle (validation, model loading, separation, output writing), waveform preview metadata, and WAV/MP3 downloads.
- Built the React Native (Expo) app: auth, chat playlist UI, image upload, the animated Talk-to-DJ voice screen, and stem upload/progress/preview, verified on real devices via Expo Go.
Outcomes
- 17 epochs The CLAP–InfoNCE model stabilized after 17 training epochs with clear embedding separation, eliminating the feature collapse of the earlier baselinesThesis
- Diagnosed, not assumed Zero-shot CLAP evaluation documented concrete strengths (direct emotions, tempo, timbre) and weaknesses (abstract/situational prompts) — an honest negative result used to justify fine-tuning, not a headline metricThesis
- Working prototype Iteration 2 delivered a working multimodal prototype — text, image, and voice playlist generation plus the Talk-to-Your-DJ loop — verified on real devicesThesis
- Shipped feature Iteration 3 delivered end-to-end asynchronous stem separation (backend API + mobile UI) isolating vocals, drums, bass, and other stems with downloadable outputsThesis
Selected media
Limitations & what's next
- No quantitative recommendation-quality benchmark (accuracy, MAP, nDCG, latency) was measured — outcomes are qualitative, and the honest next step is a held-out evaluation against a labeled emotion set.
- The pretrained encoder remains weak on abstract and situational prompts; closing that gap likely needs targeted training data for those cases, not just more epochs.
- Some integrations are partial: Google OAuth was in progress and Spotify export was only finalized in the final iteration — the prototype is a research demonstrator, not a launched app with users.
- Training relied on public sources with real class imbalance and short audio previews; a production system would need broader, balanced, fully-licensed audio.