Building a Musical AI Model for Melodex: Tansen

A few months ago someone asked me why we don't just plug a language model into a DAW and call it done.

Fair question. Language models are pretty good at talking about music. Ask one what a Dorian mode is, or how to voice a Cmaj9, and you'll get a decent answer. So why build something else?

Because talking about music and working with music are two different jobs.

Here's the request that broke every demo we tried early on:

"Make the second chorus bigger, but don't touch the vocal melody or the bassline."

That's not a generation problem. The user isn't asking for a new song. They're asking us to change one part of a thing that already exists, while leaving two other parts exactly alone, and making sure the whole thing still holds together.

Anyone who has produced anything knows how much is packed into that sentence. Bigger how? Louder drums? More layers? Wider chords? A busier build into it? And what does "don't touch" actually cover? The notes? The rhythm? The sound? You answer those questions by listening to the song, not by reading the prompt in isolation.

That's why we're building Tansen.

Tansen is our attempt at a model layer that understands music the way a working producer does. As structure, not just sound. I'll try to explain why we think we need it, without the pitch deck version.

Most "AI music" tools right now are really good at one thing: you type some words, you get audio back. Sometimes that audio is impressive. But try to go one step further and it falls apart.

"Cool, now keep everything but change the chords under the verse to feel a bit darker."

Suddenly you're stuck. The system gave you a stereo file. It doesn't really know where the verse is. It doesn't know what the chords are versus the melody. It can't pull those apart cleanly because it flattened everything down to a waveform at the end.

We ran into this constantly in the first version of Melodex. We could generate a decent starting idea. Then the user would want to edit it, which is, you know, the actual job of making music, and we'd have nowhere solid to stand. Regenerating from scratch just annoyed people. They'd liked 80% of what they had.

So we changed how we think about it. Instead of treating a song like a file to generate, we started treating it like a little system to edit.

If you've used Cursor or Copilot, you already get the mental model. You don't ask those tools to rewrite your whole repo every time. You point at a function and say "refactor this, leave the API alone." The tool looks at the codebase, makes a diff, you review it.

We want the same thing for music. A song has parts. Rhythm, harmony, melody, arrangement, sounds. Those parts relate to each other. Change the harmony and you affect what the melody means, even if you didn't move a single melody note. Add drums and you change how dense the chorus feels. It's all connected.

A four-bar piano part is a good example. You could store it as millions of audio samples. Or you could store it as a list of notes with timing and velocity. Or you could describe it as "a little two-bar motif in A minor, then a variation." All three are true. A useful music model has to move between those levels without getting lost.

Right now language models are great at the top level. "Make the intro darker" - they can parse that into something like: okay, we're talking about the intro section, darker probably means lower register, sparser, maybe minor harmony. That's genuinely useful. We use them for that in Melodex today.

But then someone has to actually do the musical work. Pick the notes. Keep the timing tight. Make sure the bass still fits the new chords. Make sure the hook still sounds like the hook in the chorus two minutes later. That's not a language problem anymore.

Music is also weird compared to text. Text mostly goes left to right. Music goes left to right and up and down at the same time. At any given second you've got kick, bass, chords, vocal all hitting together, and they have to agree with each other. Plus there's feel. Swing, ghost notes, pushing slightly ahead of the beat. You can't quantize all of that away without making everything sound like a ringtone.

And then there's memory. Not context window memory. Musical memory. If your verse introduces a little melodic idea, and your chorus never refers back to it, the song feels random even if every bar sounds fine on its own. Keeping that thread alive across three minutes is hard. Longer context windows help, but they don't solve it by themselves. The model has to know what to hold onto.

So here's the boring-sounding principle we're building around: edits should be small, scoped, and undoable.

Don't regenerate the song. Edit the chorus. Keep a list of what you promised not to change, and actually keep it. Check your work before you show it to the user. Are all the notes playable? Did you break the arrangement? Did the melody actually stay the same like you said it would?

This sounds obvious. It's surprisingly rare in AI music right now.

Under the hood, that means we don't want one giant model that does everything. Drums are a rhythm problem. Harmony is a constraint problem. Arrangement is a structure problem. Mixing is signal processing. Jamming all of that into one black box gives you less control, not more. And Melodex is supposed to be about control.

Our current setup is pretty simple to describe: symbolic notes in the middle, because you can edit notes. Rules around the edges to keep things sane, learning where it actually helps with style and taste. When you type "make the chorus more energetic," we turn that into a little musical plan first. More drum density here, wider voicing there, maybe add a layer. Then we apply it to just that section. You can see what changed. You can undo it.

For training, we've stopped obsessing over "more audio." Finished songs tell you what music sounds like. They don't tell you how someone got there. What we actually want is edit histories. What did people try, keep, throw away? In Melodex, when you accept an edit or hit undo or tweak what we gave you, that's signal. It's messy, but it's real.

I'll be honest about where this gets fuzzy. We don't have perfect answers for evaluation. A four-bar loop can sound great and still fail as a song because the second verse goes nowhere. Coherence across a whole track, keeping variation interesting without losing the idea, respecting constraints like "don't touch the melody" - those are the things we test for, and we're still figuring out good ways to measure them.

Long term, I think of it as three pieces that have to fit:

The musical brain that understands composition and intent. That's Tansen.

The project itself, with tracks and sections and history. That's Melodex.

And the engine that turns notes into actual sound you can hear and export.

Models will keep getting better. That's fine. We're betting the useful part isn't just a smarter model in isolation. It's the representation underneath, the workflow around it, the fact that you can see your song as parts and keep editing. That's harder to copy than a checkpoint.

What we're not building: a button that spits out finished songs while you watch. No "AI replaces producers" story. If you can't open what we made and change it, we failed. If you can't undo it, we failed. You're supposed to stay in the loop the whole time.

Why now? Because generation proved it can make audio. The next problem is making AI useful in the middle of the process. Analyzing what's there, remembering the hook, trying a variation, backing out when it doesn't work. That's music-to-music, not text-to-music. It needs analysis and generation to share the same structure, otherwise every edit is just a guess.

And the name. Tansen was a musician in the Hindustani classical tradition, famous for deep knowledge of raga and for improvising inside strict form. That tension - total mastery of structure, plus real freedom inside it - is exactly what we want from this thing. Respect the form. Enable the weird idea.

There's already more generated music out there than anyone will ever listen to. We don't need more audio. We need tools that help someone take an idea in their head and turn it into something real, without taking the fun part away.

That's what we're trying to build.

We're just getting started.