TalkPlay: An LLM-based music recommendation — Its model, data, and the future”

Keunwoo's Invited Talk about "TalkPlay" at RIT (Rochester Institute of Technology), 2025/10/24.

1. Keunwoo Choi, 2025 October
An LLM-based music recommendation — Its model, data, and the future”
TalkPlay

2. Presenter: Keunwoo Choi

3. Keunwoo Choi
is a Machine Learning Engineer in..
LLM for Drug
Discovery
(Genentech)
LLM Music Recsys
(TalkPlay)Music AI
a
GenAudio
(Ga Music AI
(ByteDa Music AI
(Spotify) Music Recsys
Deep Lea
(PhD,
Queen Ma 3D Audio
(ETRI)
202320202014 2025/10/10today
NYU KAISTApplied
Acoustics MIR LLM BOTH

4. Keunwoo Choi
is a Machine Learning Engineer in..
LLM for Drug
Discovery
(Genentech)
LLM Music Recsys
(TalkPlay)Music AI
a
GenAudio
(Ga Music AI
(ByteDa Music AI
(Spotify) Music Recsys
Deep Lea
(PhD,
Queen Ma 3D Audio
(ETRI)
202320202014 2025/10/10today
NYU KAISTApplied
Acoustics

5. Scatterplot of ContentsMIR LLM BOTH

6. Music Information RetrievalMIR

7. ISMIR (2000 — now)
Music Information Retrieval
extract info!"
https://ismir.net/
https://ismir2025.ismir.net/

8. Industry
E.g., Sponsors of ISMIR
2005 2015 2025

9. My MIR Tasks
•Tagging (various types of music classification) // during PhD & at ByteDance
•Drum Transcription // for fun & at Spotify
•Representation Learning // for Spotify
•Music Captioning // at Genentech, side project
•Lyrics Alignment // at Gaudio
•Source Separation // at ByteDance and Gaudio

10. LLM for Drug
Discovery
(Genentech) LLM Music Recsys
(T Music AI
a
GenAudio
(Ga Music AI
(ByteDa Music AI
(Spotify) Music Recsys
Deep Lea
(PhD,
Queen Ma 3D Audio
(ETRI) 202320202014 2025/10/10todayApplied
Acoustics
# 2022 Nov
ChatGPT release$MIR

11. Large Language ModelsLLM

12. LLM?
•You know, ChatGPT, Gemini, Claude, Deepseek, Grok, Llama, Qwen, ..

13. Next Remark [1 min]:
Training LLMs

14. Next Token Prediction
LLM
input
output
Imagine all the people
living
sharing
that

15. Next Token Prediction
LLM
input
output
Imagine all the peopleliving
life
as
their
in
like
as
life

16. Next Token Prediction
•Given a sequence, LLMs are trained to make every auto-regressive prediction.
•In this talk, illustrations are simplified as if it’s sequence-to-sequence.
input
Corpus
Imagine
Imagine all the people living for today
alloutput
inputImagine all theoutput
inputImagine all the peopleoutput
inputImagine all the people livingoutput
inputImagine all the people living foroutput
inputImagine all the people living for todayoutput

17. Training with Next Token Prediction
= Learning to Solve Tasks
= Injecting knowledge
GPT-2

18. Remark 1:
Next-Token Prediction
can pretrain a powerful model
that works in many domains.

19. Next Remark [3 min]:
Making LLMs Talk

20. Pretraining vs. Post-training
•After pre-training LLMs on huge corpus, they can auto-complete accurately on the domain
included in the corpus.
Pretrained
LLM
input
The capital city of France is
output
Paris, which is the biggest..
The capital city of South Korea is Seoul. The capital city of Japan is
The human genes consists of DNA, a molecule made of ..
What is the capital city of France? This obvious question has been..
1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 ..

21. •But, you (may) want it to talk back!
input
What’s the capital city of France?
output
Paris!
Tell me the capital city of S. Korea. It’s Seoul.
What do human genes consists of? They consists of DNAs.
What’s the capital city of France? It is still Paris!
Calculate 1 + 2 + 3 + 4? The answer is 10.
Post-
Trained
LLM

22. •Post-training / Conversational / Instruction-following
Pretrained
LLM
ChatGPT,
Gemini,
Claude,
Deepseek,
Post-
Training

23. Pretraining Post-training
Training
Method
Next Token Prediction
Corpus Documents, books, ..
(Instruction-following)
Conversations
Goal Knowledge injection Behavior control

24. Pretraining Post-training
Training
Method
(loss
function)
Next Token Prediction
Corpus Documents, books, ..
(Instruction-following)
Conversations
Goal Knowledge injection Behavior control

25. Training is the same, next-token prediction
•Using a lot of {instruction, response} pairs.
•Technically, the post-trained LLM still performs the “auto-complete”.
•But the completion - the LLM response - would be about following the instruction.
What’s the capital city of France? Paris!
Tell me the capital city of S. Korea. It’s Seoul.
What do human genes consists of? They consists of DNAs.
What’s the capital city of France? It is still Paris!
Calculate 1 + 2 + 3 + 4? The answer is 10.
Post-
trained
LLM
Pretrained
LLM

26. Remark 2:
Next-Token Prediction
with {instruction, response}
makes LLMs talk.

27. Next Remark [5 min]:
Multimodality

28. Recap
•LLMs are huge auto-completion machines, that are perhaps very useful.
•Why so useful?
•The model architecture is general enough for any sequences.
•“any” = any domain
•“any” = any behaviors
•and..
Pretrained
LLM
Post-
Trained
LLM

29. Multimodal LLM
some kind of multimodal data
output

30. Advanced: Actual audio-in, audio-out
(But today, we will focus on audio “analysis” only)
https://kyutai.org/next/codec-explainer

31. Multimodal LLM
text tokenizer
audio
tokenizer
This music is a 90’s rock music, with a fast tempo and funky guitar.
input
output

32. Advanced: Multimodal Inputs
•How?
•“Patchify” the image or audio to, process it, and feed it to the LLM
•I.e. make whatever data a sequence, and treat them like a sequence
•LLM learns to understand the input information, along with text (as usual).

33. E.g., LLark

34. E.g., Google Gemini

35. Let’s ask Gemini!
“Describe this music clip in the best detail you can. "
you and i - keunwoo.OOO

36. "gemini-1.5-flash": "The audio clip is a short excerpt of what sounds like Korean vocal music.
It's characterized by:* **Vocal Style:** The vocals are primarily melodic, with a focus on
a single, clear voice . The style is expressive, with subtle variations in pitch and volume that
suggest emotion. The singer uses a style that could be described as lyrical and perhaps
slightly melancholic . The pronunciation is clear, but without the strong, forceful quality of
some traditional Korean vocal styles.*
**Musical Accompaniment:** There is no discernible instrumental accompaniment in this short
clip. The audio focuses solely on the voice .*
**Tempo and Rhythm:** The tempo is relatively slow to moderate . The rhythm is somewhat free-
flowing, not strictly adhering to a rigid beat, but still possessing a distinct melodic pulse.*
**Pitch and Melody:** The melody is relatively simple, but pleasing. It moves primarily within
a limited range, suggesting a focus on expression rather than virtuosic display . There's a
gentle rise and fall in the pitch, creating a sense of movement and narrative within the
melody.
**Overall Impression:** The overall impression is one of quiet intimacy and emotional depth . It
sounds like a segment of a larger vocal piece, possibly a song or aria, characterized by a
refined and expressive vocal delivery. The lack of instrumental accompaniment enhances the
focus on the voice and its emotional conveyance.”

37. "gemini-2.0-flash-lite" : "This music clip features a simple yet effective arrangement.
Acoustic guitar plays a gentle, fingerpicked melody, possibly with a light reverb effect. The
music begins with a clear and slightly echoey acoustic guitar introduction, followed by a male
vocalist. The vocalist sings a soft, melancholic melody . The production emphasizes the intimacy
and emotional weight of the performance, with the simple instrumentation allowing the singer's
voice to take center stage. The overall feel is calm, a little wistful, and intimate, creating
a peaceful atmosphere. It seems like a nice piece of singer-songwriter material.”
“gemini-2.0-flash": "This is a slow, melancholic Korean pop song . It features a male vocalist
with a gentle, slightly grainy voice singing in Korean, accompanied by a simple piano melody
that emphasizes a sense of longing and reflection . The music has a heartfelt, emotional
quality."

38. "gemini-2.5-flash-preview-05-20" : "This is a **serene and mellow K-R&B (Korean R&B) or K-Soul ballad **, with
a strong influence of smooth jazz and contemporary R&B . The tempo is slow and unhurried , creating a very
relaxed and intimate atmosphere.
Here's a detailed breakdown:
* **Tempo:** Very slow (Adagio/Largo), estimated around 60-70 BPM, giving it a peaceful, unhurried feel.
* **Mood/Atmosphere:** Calm, contemplative, gentle, warm, and somewhat introspective. It evokes a
feeling of quiet comfort or peaceful reflection.
* **Vocalist:** A male vocalist with a smooth, gentle, and slightly breathy delivery sings in Korean.
His voice is clear and has a comforting, almost crooning quality, resting comfortably in a baritone or lower
tenor range.
* **Instrumentation:** The arrangement is sparse and tasteful, focusing on a warm, layered texture.
* **Keyboards/Synth:** A soft, sustained synth pad creates an airy, spacious, and warm backdrop
throughout the clip. There may be subtle electric piano or piano chords layered beneath the pad.
* **Drums/Percussion:** The drums are exceptionally subtle, providing a soft, understated rhythm
section. A gentle kick drum lays down the beat, accompanied by a very light snare (possibly brushed or
played with soft sticks) and delicate hi-hat work, ensuring the rhythm is felt rather than explicitly heard.
* **Bass:** A warm, smooth electric bass provides a foundational, melodic pulse without being overly
prominent, blending seamlessly into the background.
* **Guitar:** A clean-toned electric guitar plays sparse, delicate melodic lines and arpeggiated chords,
often in the higher register. It functions as a decorative, atmospheric element rather than a lead
instrument, adding touches of warmth and introspection.
**Timeline Breakdown:**
* **0:00 - 0:02:** The clip begins with a brief instrumental intro featuring the sustained synth pad and
a short, clean electric guitar motif, setting the mellow tone.
* **0:02 - 0:09:** The male vocalist enters smoothly, carrying the melody in Korean, accompanied by the
full, subtle instrumentation of the bass, drums , synth pad, and the occasional delicate guitar lick. The
dynamics remain consistently soft and gentle throughout this section.
The overall texture is very soft, airy, and layered, with no harsh or loud elements. The dynamics remain
relatively consistent and low, maintaining a sense of calm."

39. Remark 3:
LLMs can understand
multimodal data,
just like texts

40. Remark 1 + 2 + 3
•1. LLMs are large, and can learn from a large amount of data, by next token prediction.
•2. We can make it a strong instruction follower with the same, next token prediction.
•3. We can also make it understand audio with next token prediction.
•1 + 3: We can let LLM learn from a large amount of audio data.
•2 + 3: We can make it perform music information retrieval tasks.
•1 + 2 + 3: We can make it a very strong audio analyzer.

41. Tokens

42. Tokens
•Token is the minimum discrete & distinguished unit of language in LLMs.
•One token ~= one concept
•A sentence == A sequence of token
•A LLM usually has about 100k tokens
•Inside LLM, a single token is represented by a single vector
•100k tokens x 4096-dim x 2 bytes/number ~=410M parameters ~= 800 MB in memory/disk
•Long sequence → Many tokens → More compute and memory, sometimes O(n**2)

43. Tokens
•Word-by-word tokenization is not great, because there are too many words.
•Say, select top 1M words:
•[Memory Issue] Tokens already take 4B parameters (8G), not even counting the language model itself.
•[Performance] It still would have out-of-vocab issues.
•[Modeling Efficiency] There would be too many similar words; LLMs need to understand all of them,
separately.
•Character-wise tokenization is also not great.
•(+) Just 26 * 2 = 52 tokens to represent English!
•(-) Sequences become too long. (E.g. 4 times longer → 16 times more memory usage)
•(-) The semantic of each character is too vague; making it harder for the LM to learn.

44. Tokens
•Therefore, sub-word tokenization.
https://belladoreai.github.io/llama-tokenizer-js/example-demo/build/

45. Advanced: Deep dive to Tokenization
•Deep dive: https://www.fast.ai/posts/2025-10-16-karpathy-tokenizers#gpt-2-and-gpt-4-
tokenizers

46. Tokens, anyway.
•LLMs break down texts into sub-word units, and it’s called tokens.
•A token == a vector in LLM
•Thus, a sentence == a sequence of vectors.
•LLMs don’t read texts. They take a sequence of vectors!
LLM
https://jalammar.github.io/illustrated-transformer/

47. What to worry about Tokens
•Everything:
•Model Performance
•Memory Efficiency
•Modeling Efficiency
•Sub-word tokenizer: Best among we have.
•~100k vocab: good enough for 1+ languages

48. Q&A
End of the first 30ish minutes

49. TalkPlayBOTH

50. LLM Recommendation
also known as Generative Recommendation

51. Actually, since early October..
•This latest feature is very similar
to TalkPlay, and something that
has been discussed in Recsys
research community in the last
1-2 years.
•So.. great!
But let’s get back to the
motivation, and to do that, I need
to pretend this feature doesn't
exist.
Spotify x ChatGPT

52. Why LLM is not enough?
•LLMs know so many music, artists, albums, and musical concepts, and can understand our
query, respond well, speak better English than me, etc, etc..
•But - LLM itself has no connection to actual music catalog
•How can we do that?

53. Connect LLM <> Item DB
• == next token prediction machine
•We want LLM to recommend items just like it predict next tokens…
•(because that’s how LLM becomes so powerful at understanding,
and we want it to be same good at recommending music)
•A “token” (its vector) represents a (sub)word, a concept, an entity.
•So.. we need to represent songs using tokens. How?
•→ LET’S ADD MORE TOKENS THAT REPRESENT MUSIC ITEMS!!
•Recommendation can be made just like generative models
•Generative Recommendation
LLM
Music Database

54. Connect LLM <> Item DB
•One song == one token?
•Similar to word-as-a-token
•e.g. 100M different vectors for 100M songs — Not good, because,
•Too much memory usage
•Too many vectors for model to learn
•A better solution is..
~100M songs, ~1M artists
5+ B videos on YouTube
600M items on Amazon

55. •Naively (like word-as-a-token):
•one item is represented by one token
•16 items in DB → 16 new tokens needed
•1M items in DB → 1M new tokens needed
•Efficiently (like sub-word tokens)
•one item is represented by two tokens
•16 items → 8 (sqrt(16) * two) new tokens needed
•1M items → 2000 (sqrt(1M) * two) needed
item representation
1
token_1
2
token_2
3
token_3
4
token_4
5
token_5
6
token_6
7
token_7
8
token_8
9
token_9
10
token_10
11
token_11
12
token_12
13
token_13
14
token_14
15
token_15
16
token_16
item representation
1
(token_1, token_5)
2
(token_1, token_6)
3
(token_1, token_7)
4
(token_1, token_8)
5
(token_2, token_5)
6
(token_2, token_6)
7
(token_2, token_7)
8
(token_2, token_8)
9
(token_3, token_5)
10
(token_3, token_6)
11
(token_3, token_7)
12
(token_3, token_8)
13
(token_4, token_5)
14
(token_4, token_6)
15
(token_4, token_7)
16
(token_4, token_8)

56. item representation
1
(category_1, subcategory_1)
2
(category_1, subcategory_2)
3
(category_1, subcategory_3)
4
(category_1, subcategory_4)
5
(category_2, subcategory_1)
6
(category_2, subcategory_2)
7
(category_2, subcategory_3)
8
(category_2, subcategory_4)
9
(category_3, subcategory_1)
10
(category_3, subcategory_2)
11
(category_3, subcategory_3)
12
(category_3, subcategory_4)
13
(category_4, subcategory_1)
14
(category_4, subcategory_2)
15
(category_4, subcategory_3)
16
(category_4, subcategory_4)
item representation
1
item_1
2
item_2
3
item_3
4
item_4
5
item_5
6
item_6
7
item_7
8
item_8
9
item_9
10
item_10
11
item_11
12
item_12
13
item_13
14
item_14
15
item_15
16
item_16
•Imagine: the the double-token representation may mean something like: %

57. •As known as “SemanticID”
•Currently de factor method for generative
recommendation
https://arxiv.org/abs/2305.05065

58. LLM recommender
Give me music like keunwoo.OOO’s songs
category_subcategory_1
item representation
1
(category_1, subcategory_1)
2
(category_1, subcategory_2)
3
(category_1, subcategory_3)
4
(category_1, subcategory_4)
5
(category_2, subcategory_1)
6
(category_2, subcategory_2)
7
(category_2, subcategory_3)
8
(category_2, subcategory_4)
9
(category_3, subcategory_1)
10
(category_3, subcategory_2)
11
(category_3, subcategory_3)
12
(category_3, subcategory_4)
13
(category_4, subcategory_1)
14
(category_4, subcategory_2)
15
(category_4, subcategory_3)
16
(category_4, subcategory_4)

59. •If everything went well:
•“category_3” token indicates some musical quality similar to keunwoo.OOO
•“subcategory_5” token also indicates something like that.
•LLM recommender knows all the meanings of category_* to subcategory_*
perhaps because it’s trained on relevant recommendation data.
LLM recommender
Give me music like keunwoo.OOO’s songs
category_subcategory_1

60. •Efficiently (like sub-word tokens)
•one item is represented by two tokens
•16 items → 8 (sqrt(16) * two) new tokens needed
•1M items → 2000 (sqrt(1M) * two) needed
item representation
1
(token_1, token_5)
2
(token_1, token_6)
3
(token_1, token_7)
4
(token_1, token_8)
5
(token_2, token_5)
6
(token_2, token_6)
7
(token_2, token_7)
8
(token_2, token_8)
9
(token_3, token_5)
10
(token_3, token_6)
11
(token_3, token_7)
12
(token_3, token_8)
13
(token_4, token_5)
14
(token_4, token_6)
15
(token_4, token_7)
16
(token_4, token_8)
•BUT we can’t randomly assign the tokens!!
•Effectively
•Item 1, 2, 3, 4 should have something in
common, since they’re all “token_1”
•Item 5, 6, 7, 8 too.
•Item 9, 10, 11, 12; item 13, 14, 15 16 too.


AND..
•Item 1, 5, 9, 13 should have something in
common, since they’re all “token_5”

61. •We need to assign tokens to a lot of songs in
•By using 2 (or more!) tokens, we keep the token space manageable (Efficiency)
•But the token assignment needs to make sense, semantically.
DB
https://arxiv.org/abs/2502.13713

62. TalkPlay 1

63. TalkPlay 1
How to make the music tokenizer make sense?

64. •A song is represented by five tokens
•Each token dimension has its own, dedicated meaning.
•1st token: indicates the playlist embedding of the song
•2nd token: indicates the tags embedding
•3rd token: the metadata embedding
•4th token: the lyrics embedding
•5th token: the audio signal embedding

65. •Train a LLM with the multimodal music tokens
•replacing music item
•one music item == 5 multimodal music tokens
•e.g.
•LLM learns the meanings of multimodal music
tokens , associated with the
query and all the other contexts.
<p-13><t-1><m-11><l-143><a-94>
<p-13> <t-1> <m-11><l-143><a-94>
What was the Michael Jackson’s song about piece?
I think you’re looking for “Heal the World”
Training TalkPlay

66. TalkPlay recommends (predicts) a song
In the music DB,
each song is converted into 5 tokens
( )
based on its
audio, metadata, lyrics, and other
information
TalkPlay
Query: play me something like ..
(predicted multimodal music tokens)

67. Evaluation of TalkPlay
•It works fairly well.
•It is difficult to evaluate it because there’s no actual baseline model, but we did our best.

68. TalkPlayData 2

69. Motivation of TPD2
•TalkPlay models are trained on conversational music recommendation datasets.
•that has queries, music recommendations, maybe feedback, multi-turn chats, ..
•but…
•..
•…
•…..
•….……
•Such a dataset doesn’t exists!

70. Previously: TalkPlay
Essentially, “TalkPlayData 1”
•For each playlist in MPD ( ):
•ask an LLM to convert the playlist to a conversation
•, using the technique we introduced in .
(which is like .. see our ISMIR 2023 paper)

71. Previously: TalkPlay
•What we did it well &
•First LLM-based chat synthesis for music recommendation
•Realistic conversation
•Large scale
•Music sequence is grounded by playlists
•Limitations '
•Although it’s multi-turn, the turn-by-turn coherence was weak
•Each conversation flow feels too simple
•Lack of conversation goal; every conversation sounds too similar to each other
•Lack of personalization

72. Solution
•ISSUE: Without any input, LLM just generates something around the “average”
•average music chat topic, average likes/dislikes, average listener type, average goal, ..
•Speak like the same person, with the same musical taste, and the same goal for a music conversation.
•So, we have to condition the LLM highly specifically.

73. TalkPlayData 2
•More realism: Four separate LLMs — each of them has a dedicated role
•Enhance/Provide a listener profile
•Generate a conversation goal
https://arxiv.org/abs/2509.09685

74. Listener Profile
•User demographics
•Musical preferences
https://talkpl.ai/p/talkplaydata2/

75. Conversation Goal

76. Examples:
https://talkpl.ai/p/talkplaydata2/

77. •A listening session (sequence of tracks + other information) is sampled
•Profiling tracks + user info goes into to generate a
•Recommendation Pool and goes into that generates a

78. •Separation of Listener LLM and Recsys LLM:
•They are exposed to different subsets of information
•Listener wants to achieve Conversation Goal
•Listener doesn’t know which songs are available exactly
•Recsys wants to satisfy Listener
•Recsys doesn’t know too much about the listener; has to rely on the current chat.

79. Data Generation
•RecsysLLM can see audio, album art, lyrics,
metadata, tags, etc., of recommendation
pool.
•Listener LLM also can see __same_info__ of
each recommended track.
•Multiple calls to Gemini 2.5 Flash per chat
•More calls, more costs
•More calls, more problems
•→ pretty complicated generation pipeline

80. Does the goal/profile help?
•Yes, they make the overall conversation much more diverse.

81. A lot more details in the paper
( Paper: On arXiv: "TalkPlayData 2: An Agentic Synthetic Data Pipeline for Multimodal Conversational Music Recommendation"
) Dataset: Hugging Face. 16,500 conversations in total. 15,500 in training split and 1,000 in test split.
* Generation Code: GitHub — self-contained and runnable with a small dummy set, designed to expose the exact prompts and the end-to-end process.

82. TalkPlay-Tools [1 min]

83. https://talkpl.ai/p/talkplay_tools/
https://arxiv.org/abs/2510.01698

84. Future Works

85. Public Challenge
•TalkPlayData 2 is adopted as the official dataset
for NLP4MUSA 2026 CMR Challenge
•Please join there!
https://sites.google.com/view/nlp4musa-2026

86. More Realistic Data
•TalkPlayData 1 and 2 are outcomes out of our data struggle of academia x Recsys
•Little user data, only preview audios if lucky, no user interaction, ..

87. More adoption from LLM Research
•RAG, Multimodal RAG
•Tools
•Reinforcement Learning:
•DeepSeek, ChatGPT O-1 and O-3, Grok, Gemini — all extensively used RL
•Works best / easiest when the answer is verifiable (e.g. math and coding)
•Is music recommendation a (very quickly) verifiable problem? Hmm.. do you know?

88. More expansion in Recsys
•TalkPlay (embedding prediction; somewhat close to “similar song search”

→ TalkPlay-Tools (Combine keyword search, attribute filtering, and embedding-based
approach into a system)

→ then, what?
•Obvious answer: Personalization, podcast /audiobooks / samples and stems
•A lot more works to be done; towards a singe LLM-only recommendation system

89. What’s stopping me?
•There’s no data — really no data, to the point I already used so many adjacent data to
synthesize TalkPlayData 2. But we want real user interaction, real audio, larger, up-to-date!
•No one pays me for this, lol

90. What has encouraged me?
•Being an indie / academic researcher has some benefits!
•No conflicts of interests
•Always publish my result
•Can follow my heart and do what I believe is important
•Now that I have re-re-started Diablo 2 Resurrected, things are highly uncertain..

91. LLM for Drug
Discovery
(Genentech)
LLM Music Recsys
(TalkPlay)Music AI
a
GenAudio
(Ga Music AI
(ByteDa Music AI
(Spotify) Music Recsys
Deep Lea
(PhD,
Queen Ma 3D Audio
(ETRI)
202320202014 2025/10/10today
NYU KAISTApplied
Acoustics MIR LLM BOTH
Keunwoo Choi
and the journey of TalkPlay
(with Seungheon Doh , the other TalkPlayer)

92. Q&A
thank you for listening! - Keunwoo