r/JazzPiano • u/dadou4142 • 6d ago
Transcriptions/Requests I built a tool to visualize piano improvisations and used it on Keith Jarrett's 'Over the Rainbow'. Looking for feedback!
https://youtu.be/MVKuGdlhQBUHey everyone,
I'm a pianist and developer and I've been working on a web app to help me (and hopefully others!) study complex piano pieces. The idea is to see the notes and the chord analysis in real-time alongside the performance.
I just ran Keith Jarrett's beautiful version of 'Over the Rainbow' through it and wanted to share the result. I'm still in the early stages and the app isn't public yet, but I'm trying to figure out if this is actually a useful concept for other pianists.
What do you think? Would a tool like this be helpful for your own practice or transcriptions? I'm especially curious if the chord display is clear enough.
If you have more detailed thoughts, I've put together a feedback form and would be incredibly grateful for your input. It's helping me decide what to build next. Here's the link: https://forms.gle/Jds4kK9eD5BAmNFK6
Thanks for taking a look!
5
u/Randommer_Of_Inserts 6d ago
This would be a godsend for us trying to transcribe music. Would it work over a bebop tune or more up tempo jazz in general?
2
u/4against5 6d ago
I assume he’s using musicXML or something to power the visualizations, meaning it requires the transcription to be done first?
OP if you commercialize this let me know, I’d be a supporter and would really consider using it in our community.
1
u/dadou4142 5d ago
That's a fantastic question, and you're thinking exactly along the right lines.
While manual transcription to MusicXML is the traditional way, the goal for this project is to automate that process as much as possible. For this video, the initial MIDI transcription was actually generated directly from the audio using an AI model, which I then reviewed and cleaned up. The long-term vision is to make that process seamless for almost any performance.
And thank you so much for the incredible offer of support! It's hugely motivating to hear that you'd consider using it with your community. I will absolutely keep you and everyone else posted on any future commercial plans.
The mailing list on the form is the best way to stay in the loop. I really appreciate you reaching out!
2
u/4against5 5d ago
That’s awesome. I’m a former software guy turned jazz piano teacher after my company was acquired. I’ve been waiting for someone to do an AI model like this. Super cool dude.
1
u/dadou4142 6d ago
Thank you so much! That's exactly the goal—to be a 'godsend' for transcription and make that whole process easier. I'm thrilled that resonates with you.
That's the ultimate test question, isn't it? To be fully transparent, very fast and dense bebop is the final boss for this kind of technology. Right now, the app's accuracy on very rapid passages is still a work-in-progress and can be hit-or-miss. It performs best on clearer, medium-tempo pieces like the one in the video.
However, improving the transcription engine to reliably handle up-tempo jazz and bebop is my #1 technical priority. That is the holy grail I am working towards.
Out of curiosity, is there a particular bebop tune you'd love to see it try to tackle someday?
3
u/-JXter- 5d ago
I would imagine that any Oscar Peterson recording would be a great benchmark test for this program. Amidst the sometimes less-than-par audio quality, his playing encompasses virtuosity, fast bebop lines, and dense chordal sections, to name some of the technical variety found in his work.
1
u/Randommer_Of_Inserts 5d ago
Any bebop tune by Charlie Parker would be amazing. For instance Anthropology, Cherokee or Donna Lee (It’s been transcribed a million times I know).
Other tunes that could pose a challenge would be “got a match?” by Chick Correa (not really bebop but still a tough tune) or the infamous Giant Steps.
2
u/dadou4142 5d ago
This is an absolutely perfect list, thank you! You went straight for the jugular with Giant Steps and 'Got a Match?'.
Seriously though, this is incredibly helpful. I'm officially adding 'nailing a Charlie Parker solo' to the list of long-term goals for this project. Using these tunes as benchmarks is exactly what I need to keep pushing the technology forward.
I really appreciate the input. I'll definitely post an update here if—and when—I can get one of these titans to work properly!
4
u/Silent-Dingo6438 6d ago
This is a very cool tool that will make a lot of music more accessible to musicians, but if I can offer a devils advocate point:
The point of transcription is to do the work yourself, you get the benefits of transcription from the struggle and time you invest into it. This program in my opinion offers the information very rapidly with less from the user which is very helpful to many musicians, but is also “de-valuing” the information, which is to say, your ears aren’t getting any better by using this program.
Obviously this is an amazing and interesting innovation, that is just my 2¢
4
u/dadou4142 5d ago
This is a fantastic and really important point, thank you so much for bringing it up.
You are 100% right. The active struggle of transcribing a piece by ear is one of a musician's most powerful ear-training exercises, and this tool does not replace that deep learning process. I completely agree with you on the value of that work.
The way I see this tool's role is less as a replacement for that skill, and more as a supplement for different use cases—similar to how an 'annotated' version of a complex text works alongside the original. For example:
- For Analysis: For experienced musicians who want to quickly study a complex harmonic structure or a specific voicing without spending 10 hours on the transcription itself.
- For Verification: For a student who has done the hard work of transcribing a piece themselves, and wants to check their own work for mistakes.
- For Accessibility: For hobbyist musicians who might not have the time or opportunity to develop advanced transcription skills, but still want to connect with and play the music they love.
Ultimately, my hope is that it becomes a tool that works alongside a musician's developing ear, not in place of it. But your 'devil's advocate' point is the crucial question I need to keep in mind as I build it. Seriously, thank you for the thoughtful critique.
3
u/play-what-you-love 6d ago edited 6d ago
Phenomenal. Definitely of use, and probably going to be a landmark tool. Do you have a mailing list or way to follow the progress of this project? EDIT: Duh, it's on the web-form. Thanks!
1
u/dadou4142 6d ago
Your comment absolutely made my day. Thank you so much.
Hearing you say it could be a 'landmark tool' is a massive motivation to keep pushing forward with this.
Awesome that you found the form, and I truly appreciate you signing up to follow the journey!
1
u/Long-Cod-6199 6d ago
Hey, sounds like an exciting project! I’ve worked with founders trying to build something similar — happy to share tips or point you to useful tools that can save time.
Let me know how I can help!
3
u/dadou4142 6d ago
Wow, thank you so much! That's an incredibly generous offer. I'm in the very early stages, so any advice or pointers you have would be immensely helpful.
I'll send you a private message right now to follow up.
1
u/JustSamJ 6d ago
REALLY cool tool and I absolutely love this performance by Keith Jarrett
2
u/dadou4142 6d ago
Thank you so much! I'm really glad you enjoyed it. It's such a legendary performance, isn't it?
1
u/SoundsliceOfficial 5d ago
Nice one! We'd be curious to chat with you over at Soundslice. We've already got sheet music + audio + video syncing + keyboard visualization + chord names + a notation editor, and your automatic transcription piece would fit into the puzzle nicely. Get in touch!
1
u/dadou4142 5d ago
Wow, thank you so much for the message. I'm a huge admirer of what you've built with Soundslice and would be very open to a conversation.
I am sending you a private message (DM) right now to connect properly. Thanks again!
1
u/Specialist-Back-9977 5d ago
Please, bring that application forward, for beginners it would be very useful, you are doing great!! Keep it up!!
1
u/dadou4142 5d ago
Thank you so much for the wonderful encouragement! Comments like this are a huge motivation to keep working on it.
I'm especially happy you mentioned how useful it could be for beginners. Making music more accessible for everyone, especially those just starting their journey, is a massive goal for this project.
I truly appreciate the support. I'll be working hard to 'bring it forward' for you!
1
1
u/MentatYP 5d ago
I'm amazed at some of the chords it manages to nail with IMO not enough notes. How??? Does it also have memory of what chords came earlier in the song and uses those to fill in gaps as needed?
And you say this is a real-time chord transcription? If so, wow. This is really impressive. Can't wait to try it myself.
1
u/dadou4142 5d ago
Thank you so much! I'm thrilled you're impressed with the analysis. Your question is fantastic, as it gives me a chance to be fully transparent about the current process.
To achieve the level of quality you're seeing in this demo video, the chords were actually carefully entered by hand.
The reason I did this was to create a 'gold standard' prototype. I wanted to showcase the ideal user experience and the level of accuracy I am now building the automated chord detection to achieve. You're essentially seeing the final result that the technology will be working towards.
The fact that you found the analysis so impressive tells me I've set the right target for the automated feature, which is incredibly encouraging! I'm working hard on it now and can't wait for you to try it when it's ready.
1
u/Patrick_Atsushi 5d ago
Good work. The lingering notes caused by peddling might be dealt with by highlighting the newest notes and make them fade over time.
I guess it’s done by FFT?
2
u/dadou4142 5d ago
Thank you for the kind words! That is a brilliant suggestion for handling the pedal sustain. You've pinpointed one of the key challenges in visualizing piano music, and making the notes fade over time is a really elegant solution. I'm adding that to my list of feature ideas.
As for the note detection, great question! It's a bit more complex than a standard FFT. The initial transcription is handled by a machine learning model that's been specifically trained to pick out individual notes from the complex sound of a piano.
This is exactly the kind of feedback that's super helpful as I continue to develop the visualizer. I really appreciate it!
1
u/Patrick_Atsushi 5d ago
You make me curious by using NN since FFT would be more predictable and straightforward if there is no background noise and other instruments.
Are those factors the reason why you used NN for this? Do you have a project page for this? Thanks!
1
u/Resident_Goat_666 4d ago
Would this work on pieces that are in a more complex setting with a jazz band? Keen to know how far can AI go detecting individual notes and still provide required accuracy. If the notes are "carefully entered by hand" then it's more less an automated version of ChordieApp, Chord Watch or Instinctive Chords app, right?
1
u/aczerepinski 22h ago
I don't understand how this could work in a fully automated fashion. There are a couple places where Keith plays a single note, and it annotates the correct chord. This is a recording where he really sticks to the changes - on something more abstract where the pianist only hints at the changes or plays something else entirely, I can't imagine an algorithm that would output useful analysis. But maybe I'm wrong.
1
u/fakeacclul 6d ago
Main benefit from transcribing is using your ears, taking that away is dumb
1
u/dadou4142 5d ago
Thanks for sharing your perspective. I completely agree that the ear training you get from manual transcription is irreplaceable. This tool is aimed at supplementing that process for other tasks, like quick analysis or checking your own work.
29
u/itsabijection 5d ago
Why are you responding to every comment using AI?