Audio to MIDI Converter

How Audio to MIDI Conversion Works: From Sound Waves to Note Data

Spotify released Basic Pitch in June 2022 as an open-source automatic music transcription model. The model weighs under 1 MB and runs in a browser using TensorFlow.js with a WebGL backend. It converts polyphonic audio (multiple notes at once) into MIDI note data, including pitch bend information. This tool uses that exact model, loaded from a local copy of the model weights, to transcribe your audio file without sending anything to a server.

The model was trained on a dataset of synthesized instrument recordings covering piano, guitar, strings, and wind instruments. According to the ICASSP 2022 paper by Donahue et al., Basic Pitch achieves note F1-scores competitive with much larger models while being small enough to run on consumer hardware. The tradeoff is that it works best on single-instrument recordings, not full mixed songs.

What the Model Actually Does

When you click Convert to MIDI, the tool performs three sequential steps. First, the audio is down-mixed to mono and resampled to 22,050 Hz (the model's expected input rate). Second, the model processes the audio in overlapping frames and outputs three probability matrices: a note frame matrix (which pitches are active at each time step), an onset matrix (when new notes begin), and a contour matrix (fine-grained pitch tracking for pitch bend detection). Third, post-processing algorithms convert those matrices into discrete MIDI note events with start times, durations, pitches, and velocities.

The onset matrix is what triggers new notes. A high onset probability at a given time and pitch means the model detected a new note starting there. The frame matrix sustains notes after their onset. The contour matrix provides sub-semitone pitch resolution, which the tool converts into MIDI pitch bend values for notes that are slightly out of tune or feature vibrato and glissando.

Understanding the Sensitivity Controls

The three sliders in this tool control the post-processing thresholds that convert the model's raw output into MIDI notes. These are the same parameters exposed by the original Basic Pitch library.

Onset Sensitivity (default 0.25) sets the threshold for detecting note onsets. Lowering this value makes the model more sensitive and detects more note starts, including quiet or ambiguous ones. Raising it makes detection more conservative, which reduces false positives but may miss soft notes. For a clean piano recording, 0.2 to 0.3 works well. For a breathy vocal with soft attacks, try 0.15. For a percussive guitar with sharp attacks, 0.3 to 0.4 prevents double-triggering on the same note.

Frame Threshold (default 0.25) sets how loud a pitch must be in the frame matrix to count as part of a sustained note. Lowering this value extends notes longer because quieter frames pass the threshold. Raising it shortens notes by cutting off quieter tail frames. If your notes are being truncated, lower this to 0.15. If notes are bleeding into each other, raise it to 0.35.

Minimum Note Length (default 5 frames) filters out detections shorter than the specified number of frames. Each frame is about 30 ms, so the default of 5 frames means notes shorter than 150 ms are discarded. This removes noise artifacts and spurious detections. For fast passages with short notes, lower this to 2 or 3. For sustained pad or string recordings, raise it to 10 or more to filter out transient noise.

Best Audio Sources for Transcription

Basic Pitch is designed for single-instrument recordings. The model was trained on isolated instrument tracks, not full band mixes. If you upload a complete song with drums, bass, vocals, and guitars all playing at once, the model will attempt to transcribe everything it hears, producing a noisy and inaccurate MIDI file. For best results, isolate the instrument you want to transcribe first.

You can use the Vocal Remover to extract a vocal or instrumental from a stereo mix before transcribing. For isolating a specific section of a longer recording, the Audio Trimmer lets you cut out the relevant portion. If your recording has background noise or hiss, run it through the Noise Gate Simulator first to clean up the signal before transcription.

Ideal input sources include: a solo vocal recording (a cappella or with minimal backing), a solo piano performance, a guitar recording (acoustic or clean electric), a flute or wind instrument recording, or a monophonic synth line. Reverb and delay on the source audio reduce transcription accuracy because they smear the onset detection. Dry recordings transpose to MIDI more cleanly.

What to Do with the MIDI File

The downloaded .mid file is a standard MIDI file compatible with every DAW (Ableton Live, Logic Pro, FL Studio, Cubase, Pro Tools, GarageBand) and every MIDI-compatible plugin or hardware synth. Import the file into your project, assign a virtual instrument to the MIDI track, and you have a playable version of your audio recording.

Common workflows include: transcribing a vocal melody to a MIDI track so you can harmonize it, counterpoint it, or replace it with a synth. Converting a piano improvisation to MIDI so you can edit individual notes, change the tempo, or try different instrument sounds. Extracting a guitar riff to MIDI for use in a sampler or for notation purposes. If you need to transpose the MIDI to a different key after transcription, the MIDI File Transposer handles that. To view the MIDI as sheet music, the MIDI to Sheet Music tool renders it as standard notation using VexFlow.

For a deeper look at the notes, the MIDI Chord Player visualizes MIDI files as an interactive piano roll with per-track playback and velocity inspection. And if you want to build on the transcribed melody with arpeggios, the Arpeggio Generator can create arpeggio patterns from the chords in your transcribed MIDI.

Limitations and Honest Expectations

Basic Pitch is a lightweight model, not a professional-grade transcription system. It will not produce perfect MIDI from every recording. Here is what to expect.

Timing accuracy is generally within 30 to 50 ms of the actual note onset. For most musical purposes this is fine, but if you need sample-accurate timing for tight MIDI sequencing, you will need to manually adjust note positions in your DAW after import.

Pitch accuracy depends on the instrument and recording quality. Clean monophonic sources (a single vocal, a single flute) transcribe at high accuracy. Polyphonic sources (chords on a piano, fingerpicked guitar) are harder. The model may miss notes in dense chords or add phantom notes in noisy passages. The pitch bend data is approximate and should be treated as a suggestion rather than precise pitch tracking.

Velocity data is derived from the model's amplitude estimates and is not calibrated to match the actual dynamics of your recording. Use it as a rough guide and adjust velocities in your DAW after import.

Processing time depends on audio length and your device's GPU. A 30-second clip typically processes in 5 to 15 seconds on a modern laptop with WebGL enabled. A 5-minute recording may take 1 to 3 minutes. The progress bar shows the model's internal progress estimate. Very long files (over 10 minutes) may exhaust browser memory. For long recordings, trim the section you need first.

The Basic Pitch GitHub repository contains the full source code, training data description, and benchmark results. The model is released under the Apache 2.0 license, and the TypeScript package is available on npm as @spotify/basic-pitch. This tool uses the model weights directly from that package, served from the same domain as this page, so no third-party requests are made during transcription.