Generate music with an RNN

View on TensorFlow.org Run in Google Colab View source on GitHub Download notebook

This tutorial shows you how to generate musical notes using a simple RNN. You will train a model using a collection of piano MIDI files from the MAESTRO dataset. Given a sequence of notes, your model will learn to predict the next note in the sequence. You can generate a longer sequences of notes by calling the model repeatedly.

This tutorial contains complete code to parse and create MIDI files. You can learn more about how RNNs work by visiting Text generation with an RNN.

Setup

This tutorial uses the pretty_midi library to create and parse MIDI files, and pyfluidsynth for generating audio playback in Colab.

sudo apt install -y fluidsynth
The following packages were automatically installed and are no longer required:
  libatasmart4 libblockdev-fs2 libblockdev-loop2 libblockdev-part-err2
  libblockdev-part2 libblockdev-swap2 libblockdev-utils2 libblockdev2 libnuma1
  libparted-fs-resize0
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  fluid-soundfont-gm libdouble-conversion3 libfluidsynth2 libinstpatch-1.0-2
  libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5gui5 libqt5network5 libqt5svg5
  libqt5widgets5 libsdl2-2.0-0 libxcb-xinerama0 libxcb-xinput0 qsynth
  qt5-gtk-platformtheme qttranslations5-l10n timgm6mb-soundfont
Suggested packages:
  fluid-soundfont-gs timidity qt5-image-formats-plugins qtwayland5 jackd
  musescore
The following NEW packages will be installed:
  fluid-soundfont-gm fluidsynth libdouble-conversion3 libfluidsynth2
  libinstpatch-1.0-2 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5gui5
  libqt5network5 libqt5svg5 libqt5widgets5 libsdl2-2.0-0 libxcb-xinerama0
  libxcb-xinput0 qsynth qt5-gtk-platformtheme qttranslations5-l10n
  timgm6mb-soundfont
0 upgraded, 19 newly installed, 0 to remove and 139 not upgraded.
Need to get 136 MB of archives.
After this operation, 203 MB of additional disk space will be used.
Get:1 http://us-central1.gce.archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB]
Get:2 http://us-central1.gce.archive.ubuntu.com/ubuntu focal/main amd64 libpcre2-16-0 amd64 10.34-7 [181 kB]
Get:3 http://us-central1.gce.archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2006 kB]
Get:4 http://us-central1.gce.archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB]
Get:5 http://us-central1.gce.archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB]
Get:6 http://us-central1.gce.archive.ubuntu.com/ubuntu focal/main amd64 libxcb-xinerama0 amd64 1.14-2 [5260 B]
Get:7 http://us-central1.gce.archive.ubuntu.com/ubuntu focal/main amd64 libxcb-xinput0 amd64 1.14-2 [29.3 kB]
Get:8 http://us-central1.gce.archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5gui5 amd64 5.12.8+dfsg-0ubuntu2.1 [2971 kB]
Get:9 http://us-central1.gce.archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5widgets5 amd64 5.12.8+dfsg-0ubuntu2.1 [2295 kB]
Get:10 http://us-central1.gce.archive.ubuntu.com/ubuntu focal/universe amd64 libqt5svg5 amd64 5.12.8-0ubuntu1 [131 kB]
Get:11 http://us-central1.gce.archive.ubuntu.com/ubuntu focal/universe amd64 fluid-soundfont-gm all 3.1-5.1 [119 MB]
Get:12 http://us-central1.gce.archive.ubuntu.com/ubuntu focal/universe amd64 libinstpatch-1.0-2 amd64 1.1.2-2build1 [238 kB]
Get:13 http://us-central1.gce.archive.ubuntu.com/ubuntu focal/universe amd64 libsdl2-2.0-0 amd64 2.0.10+dfsg1-3 [407 kB]
Get:14 http://us-central1.gce.archive.ubuntu.com/ubuntu focal/universe amd64 timgm6mb-soundfont all 1.3-3 [5420 kB]
Get:15 http://us-central1.gce.archive.ubuntu.com/ubuntu focal/universe amd64 libfluidsynth2 amd64 2.1.1-2 [198 kB]
Get:16 http://us-central1.gce.archive.ubuntu.com/ubuntu focal/universe amd64 fluidsynth amd64 2.1.1-2 [25.6 kB]
Get:17 http://us-central1.gce.archive.ubuntu.com/ubuntu focal/universe amd64 qsynth amd64 0.6.1-1build1 [245 kB]
Get:18 http://us-central1.gce.archive.ubuntu.com/ubuntu focal-updates/universe amd64 qt5-gtk-platformtheme amd64 5.12.8+dfsg-0ubuntu2.1 [124 kB]
Get:19 http://us-central1.gce.archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1486 kB]
Fetched 136 MB in 10s (13.8 MB/s)

7[0;23r8[1ASelecting previously unselected package libdouble-conversion3:amd64.
(Reading database ... 139818 files and directories currently installed.)
Preparing to unpack .../00-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ...
7[24;0fProgress: [  0%] [..........................................................] 87[24;0fProgress: [  1%] [..........................................................] 8Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ...
7[24;0fProgress: [  3%] [#.........................................................] 8Selecting previously unselected package libpcre2-16-0:amd64.
Preparing to unpack .../01-libpcre2-16-0_10.34-7_amd64.deb ...
7[24;0fProgress: [  4%] [##........................................................] 8Unpacking libpcre2-16-0:amd64 (10.34-7) ...
7[24;0fProgress: [  5%] [###.......................................................] 8Selecting previously unselected package libqt5core5a:amd64.
Preparing to unpack .../02-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ...
7[24;0fProgress: [  6%] [###.......................................................] 8Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ...
7[24;0fProgress: [  8%] [####......................................................] 8Selecting previously unselected package libqt5dbus5:amd64.
Preparing to unpack .../03-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ...
7[24;0fProgress: [  9%] [#####.....................................................] 8Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ...
7[24;0fProgress: [ 10%] [######....................................................] 8Selecting previously unselected package libqt5network5:amd64.
Preparing to unpack .../04-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ...
7[24;0fProgress: [ 12%] [######....................................................] 8Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ...
7[24;0fProgress: [ 13%] [#######...................................................] 8Selecting previously unselected package libxcb-xinerama0:amd64.
Preparing to unpack .../05-libxcb-xinerama0_1.14-2_amd64.deb ...
7[24;0fProgress: [ 14%] [########..................................................] 8Unpacking libxcb-xinerama0:amd64 (1.14-2) ...
7[24;0fProgress: [ 16%] [#########.................................................] 8Selecting previously unselected package libxcb-xinput0:amd64.
Preparing to unpack .../06-libxcb-xinput0_1.14-2_amd64.deb ...
7[24;0fProgress: [ 17%] [#########.................................................] 8Unpacking libxcb-xinput0:amd64 (1.14-2) ...
7[24;0fProgress: [ 18%] [##########................................................] 8Selecting previously unselected package libqt5gui5:amd64.
Preparing to unpack .../07-libqt5gui5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ...
7[24;0fProgress: [ 19%] [###########...............................................] 8Unpacking libqt5gui5:amd64 (5.12.8+dfsg-0ubuntu2.1) ...
7[24;0fProgress: [ 21%] [############..............................................] 8Selecting previously unselected package libqt5widgets5:amd64.
Preparing to unpack .../08-libqt5widgets5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ...
7[24;0fProgress: [ 22%] [############..............................................] 8Unpacking libqt5widgets5:amd64 (5.12.8+dfsg-0ubuntu2.1) ...
7[24;0fProgress: [ 23%] [#############.............................................] 8Selecting previously unselected package libqt5svg5:amd64.
Preparing to unpack .../09-libqt5svg5_5.12.8-0ubuntu1_amd64.deb ...
7[24;0fProgress: [ 25%] [##############............................................] 8Unpacking libqt5svg5:amd64 (5.12.8-0ubuntu1) ...
7[24;0fProgress: [ 26%] [###############...........................................] 8Selecting previously unselected package fluid-soundfont-gm.
Preparing to unpack .../10-fluid-soundfont-gm_3.1-5.1_all.deb ...
7[24;0fProgress: [ 27%] [###############...........................................] 8Unpacking fluid-soundfont-gm (3.1-5.1) ...
7[24;0fProgress: [ 29%] [################..........................................] 8Selecting previously unselected package libinstpatch-1.0-2:amd64.
Preparing to unpack .../11-libinstpatch-1.0-2_1.1.2-2build1_amd64.deb ...
7[24;0fProgress: [ 30%] [#################.........................................] 8Unpacking libinstpatch-1.0-2:amd64 (1.1.2-2build1) ...
7[24;0fProgress: [ 31%] [##################........................................] 8Selecting previously unselected package libsdl2-2.0-0:amd64.
Preparing to unpack .../12-libsdl2-2.0-0_2.0.10+dfsg1-3_amd64.deb ...
7[24;0fProgress: [ 32%] [##################........................................] 8Unpacking libsdl2-2.0-0:amd64 (2.0.10+dfsg1-3) ...
7[24;0fProgress: [ 34%] [###################.......................................] 8Selecting previously unselected package timgm6mb-soundfont.
Preparing to unpack .../13-timgm6mb-soundfont_1.3-3_all.deb ...
7[24;0fProgress: [ 35%] [####################......................................] 8Unpacking timgm6mb-soundfont (1.3-3) ...
7[24;0fProgress: [ 36%] [#####################.....................................] 8Selecting previously unselected package libfluidsynth2:amd64.
Preparing to unpack .../14-libfluidsynth2_2.1.1-2_amd64.deb ...
7[24;0fProgress: [ 38%] [#####################.....................................] 8Unpacking libfluidsynth2:amd64 (2.1.1-2) ...
7[24;0fProgress: [ 39%] [######################....................................] 8Selecting previously unselected package fluidsynth.
Preparing to unpack .../15-fluidsynth_2.1.1-2_amd64.deb ...
7[24;0fProgress: [ 40%] [#######################...................................] 8Unpacking fluidsynth (2.1.1-2) ...
7[24;0fProgress: [ 42%] [########################..................................] 8Selecting previously unselected package qsynth.
Preparing to unpack .../16-qsynth_0.6.1-1build1_amd64.deb ...
7[24;0fProgress: [ 43%] [########################..................................] 8Unpacking qsynth (0.6.1-1build1) ...
7[24;0fProgress: [ 44%] [#########################.................................] 8Selecting previously unselected package qt5-gtk-platformtheme:amd64.
Preparing to unpack .../17-qt5-gtk-platformtheme_5.12.8+dfsg-0ubuntu2.1_amd64.deb ...
7[24;0fProgress: [ 45%] [##########################................................] 8Unpacking qt5-gtk-platformtheme:amd64 (5.12.8+dfsg-0ubuntu2.1) ...
7[24;0fProgress: [ 47%] [###########################...............................] 8Selecting previously unselected package qttranslations5-l10n.
Preparing to unpack .../18-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ...
7[24;0fProgress: [ 48%] [###########################...............................] 8Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ...
7[24;0fProgress: [ 49%] [############################..............................] 8Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ...
7[24;0fProgress: [ 51%] [#############################.............................] 87[24;0fProgress: [ 52%] [##############################............................] 8Setting up libxcb-xinput0:amd64 (1.14-2) ...
7[24;0fProgress: [ 53%] [##############################............................] 87[24;0fProgress: [ 55%] [###############################...........................] 8Setting up libpcre2-16-0:amd64 (10.34-7) ...
7[24;0fProgress: [ 56%] [################################..........................] 87[24;0fProgress: [ 57%] [#################################.........................] 8Setting up libxcb-xinerama0:amd64 (1.14-2) ...
7[24;0fProgress: [ 58%] [#################################.........................] 87[24;0fProgress: [ 60%] [##################################........................] 8Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ...
7[24;0fProgress: [ 61%] [###################################.......................] 87[24;0fProgress: [ 62%] [####################################......................] 8Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ...
7[24;0fProgress: [ 64%] [####################################......................] 87[24;0fProgress: [ 65%] [#####################################.....................] 8Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ...
7[24;0fProgress: [ 66%] [######################################....................] 87[24;0fProgress: [ 68%] [#######################################...................] 8Setting up fluid-soundfont-gm (3.1-5.1) ...
7[24;0fProgress: [ 69%] [#######################################...................] 87[24;0fProgress: [ 70%] [########################################..................] 8Setting up libsdl2-2.0-0:amd64 (2.0.10+dfsg1-3) ...
7[24;0fProgress: [ 71%] [#########################################.................] 87[24;0fProgress: [ 73%] [##########################################................] 8Setting up timgm6mb-soundfont (1.3-3) ...
7[24;0fProgress: [ 74%] [##########################################................] 8update-alternatives: using /usr/share/sounds/sf2/TimGM6mb.sf2 to provide /usr/share/sounds/sf2/default-GM.sf2 (default-GM.sf2) in auto mode
update-alternatives: using /usr/share/sounds/sf2/TimGM6mb.sf2 to provide /usr/share/sounds/sf3/default-GM.sf3 (default-GM.sf3) in auto mode
7[24;0fProgress: [ 75%] [###########################################...............] 8Setting up libinstpatch-1.0-2:amd64 (1.1.2-2build1) ...
7[24;0fProgress: [ 77%] [############################################..............] 87[24;0fProgress: [ 78%] [#############################################.............] 8Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ...
7[24;0fProgress: [ 79%] [#############################################.............] 87[24;0fProgress: [ 81%] [##############################################............] 8Setting up libfluidsynth2:amd64 (2.1.1-2) ...
7[24;0fProgress: [ 82%] [###############################################...........] 87[24;0fProgress: [ 83%] [################################################..........] 8Setting up libqt5gui5:amd64 (5.12.8+dfsg-0ubuntu2.1) ...
7[24;0fProgress: [ 84%] [################################################..........] 87[24;0fProgress: [ 86%] [#################################################.........] 8Setting up libqt5widgets5:amd64 (5.12.8+dfsg-0ubuntu2.1) ...
7[24;0fProgress: [ 87%] [##################################################........] 87[24;0fProgress: [ 88%] [###################################################.......] 8Setting up qt5-gtk-platformtheme:amd64 (5.12.8+dfsg-0ubuntu2.1) ...
7[24;0fProgress: [ 90%] [###################################################.......] 87[24;0fProgress: [ 91%] [####################################################......] 8Setting up fluidsynth (2.1.1-2) ...
7[24;0fProgress: [ 92%] [#####################################################.....] 8Created symlink /etc/systemd/user/multi-user.target.wants/fluidsynth.service → /usr/lib/systemd/user/fluidsynth.service.
7[24;0fProgress: [ 94%] [######################################################....] 8Setting up libqt5svg5:amd64 (5.12.8-0ubuntu1) ...
7[24;0fProgress: [ 95%] [######################################################....] 87[24;0fProgress: [ 96%] [#######################################################...] 8Setting up qsynth (0.6.1-1build1) ...
7[24;0fProgress: [ 97%] [########################################################..] 87[24;0fProgress: [ 99%] [#########################################################.] 8Processing triggers for desktop-file-utils (0.24-1ubuntu3) ...
Processing triggers for mime-support (3.64ubuntu1) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for gnome-menus (3.36.0-1ubuntu1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.7) ...
Processing triggers for man-db (2.9.1-1) ...

7[0;24r8[1A[J
pip install --upgrade pyfluidsynth
pip install pretty_midi
import collections
import datetime
import fluidsynth
import glob
import numpy as np
import pathlib
import pandas as pd
import pretty_midi
import seaborn as sns
import tensorflow as tf

from IPython import display
from matplotlib import pyplot as plt
from typing import Dict, List, Optional, Sequence, Tuple
seed = 42
tf.random.set_seed(seed)
np.random.seed(seed)

# Sampling rate for audio playback
_SAMPLING_RATE = 16000

Download the Maestro dataset

data_dir = pathlib.Path('data/maestro-v2.0.0')
if not data_dir.exists():
  tf.keras.utils.get_file(
      'maestro-v2.0.0-midi.zip',
      origin='https://storage.googleapis.com/magentadata/datasets/maestro/v2.0.0/maestro-v2.0.0-midi.zip',
      extract=True,
      cache_dir='.', cache_subdir='data',
  )
Downloading data from https://storage.googleapis.com/magentadata/datasets/maestro/v2.0.0/maestro-v2.0.0-midi.zip
59243107/59243107 [==============================] - 1s 0us/step

The dataset contains about 1,200 MIDI files.

filenames = glob.glob(str(data_dir/'**/*.mid*'))
print('Number of files:', len(filenames))
Number of files: 1282

Process a MIDI file

First, use pretty_midi to parse a single MIDI file and inspect the format of the notes. If you would like to download the MIDI file below to play on your computer, you can do so in colab by writing files.download(sample_file).

sample_file = filenames[1]
print(sample_file)
data/maestro-v2.0.0/2013/ORIG-MIDI_02_7_8_13_Group__MID--AUDIO_14_R2_2013_wav--1.midi

Generate a PrettyMIDI object for the sample MIDI file.

pm = pretty_midi.PrettyMIDI(sample_file)

Play the sample file. The playback widget may take several seconds to load.

def display_audio(pm: pretty_midi.PrettyMIDI, seconds=30):
  waveform = pm.fluidsynth(fs=_SAMPLING_RATE)
  # Take a sample of the generated waveform to mitigate kernel resets
  waveform_short = waveform[:seconds*_SAMPLING_RATE]
  return display.Audio(waveform_short, rate=_SAMPLING_RATE)
display_audio(pm)
fluidsynth: warning: SDL2 not initialized, SDL2 audio driver won't be usable

Do some inspection on the MIDI file. What kinds of instruments are used?

print('Number of instruments:', len(pm.instruments))
instrument = pm.instruments[0]
instrument_name = pretty_midi.program_to_instrument_name(instrument.program)
print('Instrument name:', instrument_name)
Number of instruments: 1
Instrument name: Acoustic Grand Piano

Extract notes

for i, note in enumerate(instrument.notes[:10]):
  note_name = pretty_midi.note_number_to_name(note.pitch)
  duration = note.end - note.start
  print(f'{i}: pitch={note.pitch}, note_name={note_name},'
        f' duration={duration:.4f}')
0: pitch=46, note_name=A#2, duration=0.0651
1: pitch=74, note_name=D5, duration=0.1237
2: pitch=62, note_name=D4, duration=0.3216
3: pitch=74, note_name=D5, duration=0.1133
4: pitch=53, note_name=F3, duration=0.2956
5: pitch=58, note_name=A#3, duration=0.3333
6: pitch=62, note_name=D4, duration=0.3346
7: pitch=74, note_name=D5, duration=0.4727
8: pitch=50, note_name=D3, duration=0.2552
9: pitch=70, note_name=A#4, duration=0.3021

You will use three variables to represent a note when training the model: pitch, step and duration. The pitch is the perceptual quality of the sound as a MIDI note number. The step is the time elapsed from the previous note or start of the track. The duration is how long the note will be playing in seconds and is the difference between the note end and note start times.

Extract the notes from the sample MIDI file.

def midi_to_notes(midi_file: str) -> pd.DataFrame:
  pm = pretty_midi.PrettyMIDI(midi_file)
  instrument = pm.instruments[0]
  notes = collections.defaultdict(list)

  # Sort the notes by start time
  sorted_notes = sorted(instrument.notes, key=lambda note: note.start)
  prev_start = sorted_notes[0].start

  for note in sorted_notes:
    start = note.start
    end = note.end
    notes['pitch'].append(note.pitch)
    notes['start'].append(start)
    notes['end'].append(end)
    notes['step'].append(start - prev_start)
    notes['duration'].append(end - start)
    prev_start = start

  return pd.DataFrame({name: np.array(value) for name, value in notes.items()})
raw_notes = midi_to_notes(sample_file)
raw_notes.head()

It may be easier to interpret the note names rather than the pitches, so you can use the function below to convert from the numeric pitch values to note names. The note name shows the type of note, accidental and octave number (e.g. C#4).

get_note_names = np.vectorize(pretty_midi.note_number_to_name)
sample_note_names = get_note_names(raw_notes['pitch'])
sample_note_names[:10]
array(['D5', 'A#2', 'D4', 'D5', 'A#3', 'F3', 'D5', 'D4', 'A#4', 'D3'],
      dtype='<U3')

To visualize the musical piece, plot the note pitch, start and end across the length of the track (i.e. piano roll). Start with the first 100 notes

def plot_piano_roll(notes: pd.DataFrame, count: Optional[int] = None):
  if count:
    title = f'First {count} notes'
  else:
    title = f'Whole track'
    count = len(notes['pitch'])
  plt.figure(figsize=(20, 4))
  plot_pitch = np.stack([notes['pitch'], notes['pitch']], axis=0)
  plot_start_stop = np.stack([notes['start'], notes['end']], axis=0)
  plt.plot(
      plot_start_stop[:, :count], plot_pitch[:, :count], color="b", marker=".")
  plt.xlabel('Time [s]')
  plt.ylabel('Pitch')
  _ = plt.title(title)
plot_piano_roll(raw_notes, count=100)

png

Plot the notes for the entire track.

plot_piano_roll(raw_notes)

png

Check the distribution of each note variable.

def plot_distributions(notes: pd.DataFrame, drop_percentile=2.5):
  plt.figure(figsize=[15, 5])
  plt.subplot(1, 3, 1)
  sns.histplot(notes, x="pitch", bins=20)

  plt.subplot(1, 3, 2)
  max_step = np.percentile(notes['step'], 100 - drop_percentile)
  sns.histplot(notes, x="step", bins=np.linspace(0, max_step, 21))

  plt.subplot(1, 3, 3)
  max_duration = np.percentile(notes['duration'], 100 - drop_percentile)
  sns.histplot(notes, x="duration", bins=np.linspace(0, max_duration, 21))
plot_distributions(raw_notes)

png

Create a MIDI file

You can generate your own MIDI file from a list of notes using the function below.

def notes_to_midi(
  notes: pd.DataFrame,
  out_file: str, 
  instrument_name: str,
  velocity: int = 100,  # note loudness
) -> pretty_midi.PrettyMIDI:

  pm = pretty_midi.PrettyMIDI()
  instrument = pretty_midi.Instrument(
      program=pretty_midi.instrument_name_to_program(
          instrument_name))

  prev_start = 0
  for i, note in notes.iterrows():
    start = float(prev_start + note['step'])
    end = float(start + note['duration'])
    note = pretty_midi.Note(
        velocity=velocity,
        pitch=int(note['pitch']),
        start=start,
        end=end,
    )
    instrument.notes.append(note)
    prev_start = start

  pm.instruments.append(instrument)
  pm.write(out_file)
  return pm
example_file = 'example.midi'
example_pm = notes_to_midi(
    raw_notes, out_file=example_file, instrument_name=instrument_name)

Play the generated MIDI file and see if there is any difference.

display_audio(example_pm)
fluidsynth: warning: SDL2 not initialized, SDL2 audio driver won't be usable

As before, you can write files.download(example_file) to download and play this file.

Create the training dataset

Create the training dataset by extracting notes from the MIDI files. You can start by using a small number of files, and experiment later with more. This may take a couple minutes.

num_files = 5
all_notes = []
for f in filenames[:num_files]:
  notes = midi_to_notes(f)
  all_notes.append(notes)

all_notes = pd.concat(all_notes)
n_notes = len(all_notes)
print('Number of notes parsed:', n_notes)
Number of notes parsed: 28365

Next, create a tf.data.Dataset from the parsed notes.

key_order = ['pitch', 'step', 'duration']
train_notes = np.stack([all_notes[key] for key in key_order], axis=1)
notes_ds = tf.data.Dataset.from_tensor_slices(train_notes)
notes_ds.element_spec
TensorSpec(shape=(3,), dtype=tf.float64, name=None)

You will train the model on batches of sequences of notes. Each example will consist of a sequence of notes as the input features, and next note as the label. In this way, the model will be trained to predict the next note in a sequence. You can find a diagram explaining this process (and more details) in Text classification with an RNN.

You can use the handy window function with size seq_length to create the features and labels in this format.

def create_sequences(
    dataset: tf.data.Dataset, 
    seq_length: int,
    vocab_size = 128,
) -> tf.data.Dataset:
  """Returns TF Dataset of sequence and label examples."""
  seq_length = seq_length+1

  # Take 1 extra for the labels
  windows = dataset.window(seq_length, shift=1, stride=1,
                              drop_remainder=True)

  # `flat_map` flattens the" dataset of datasets" into a dataset of tensors
  flatten = lambda x: x.batch(seq_length, drop_remainder=True)
  sequences = windows.flat_map(flatten)

  # Normalize note pitch
  def scale_pitch(x):
    x = x/[vocab_size,1.0,1.0]
    return x

  # Split the labels
  def split_labels(sequences):
    inputs = sequences[:-1]
    labels_dense = sequences[-1]
    labels = {key:labels_dense[i] for i,key in enumerate(key_order)}

    return scale_pitch(inputs), labels

  return sequences.map(split_labels, num_parallel_calls=tf.data.AUTOTUNE)

Set the sequence length for each example. Experiment with different lengths (e.g. 50, 100, 150) to see which one works best for the data, or use hyperparameter tuning. The size of the vocabulary (vocab_size) is set to 128 representing all the pitches supported by pretty_midi.

seq_length = 25
vocab_size = 128
seq_ds = create_sequences(notes_ds, seq_length, vocab_size)
seq_ds.element_spec
(TensorSpec(shape=(25, 3), dtype=tf.float64, name=None),
 {'pitch': TensorSpec(shape=(), dtype=tf.float64, name=None),
  'step': TensorSpec(shape=(), dtype=tf.float64, name=None),
  'duration': TensorSpec(shape=(), dtype=tf.float64, name=None)})

The shape of the dataset is (100,1), meaning that the model will take 100 notes as input, and learn to predict the following note as output.

for seq, target in seq_ds.take(1):
  print('sequence shape:', seq.shape)
  print('sequence elements (first 10):', seq[0: 10])
  print()
  print('target:', target)
sequence shape: (25, 3)
sequence elements (first 10): tf.Tensor(
[[0.359375   0.         0.30989583]
 [0.578125   0.0546875  0.49088542]
 [0.484375   0.48567708 0.2890625 ]
 [0.453125   0.29296875 0.4765625 ]
 [0.578125   0.00130208 0.13802083]
 [0.4140625  0.00911458 0.26953125]
 [0.578125   0.296875   0.45052083]
 [0.484375   0.01953125 0.09505208]
 [0.390625   0.3046875  0.20572917]
 [0.546875   0.04296875 0.41666667]], shape=(10, 3), dtype=float64)

target: {'pitch': <tf.Tensor: shape=(), dtype=float64, numpy=60.0>, 'step': <tf.Tensor: shape=(), dtype=float64, numpy=0.44661458333333304>, 'duration': <tf.Tensor: shape=(), dtype=float64, numpy=0.26953125>}

Batch the examples, and configure the dataset for performance.

batch_size = 64
buffer_size = n_notes - seq_length  # the number of items in the dataset
train_ds = (seq_ds
            .shuffle(buffer_size)
            .batch(batch_size, drop_remainder=True)
            .cache()
            .prefetch(tf.data.experimental.AUTOTUNE))
train_ds.element_spec
(TensorSpec(shape=(64, 25, 3), dtype=tf.float64, name=None),
 {'pitch': TensorSpec(shape=(64,), dtype=tf.float64, name=None),
  'step': TensorSpec(shape=(64,), dtype=tf.float64, name=None),
  'duration': TensorSpec(shape=(64,), dtype=tf.float64, name=None)})

Create and train the model

The model will have three outputs, one for each note variable. For step and duration, you will use a custom loss function based on mean squared error that encourages the model to output non-negative values.

def mse_with_positive_pressure(y_true: tf.Tensor, y_pred: tf.Tensor):
  mse = (y_true - y_pred) ** 2
  positive_pressure = 10 * tf.maximum(-y_pred, 0.0)
  return tf.reduce_mean(mse + positive_pressure)
input_shape = (seq_length, 3)
learning_rate = 0.005

inputs = tf.keras.Input(input_shape)
x = tf.keras.layers.LSTM(128)(inputs)

outputs = {
  'pitch': tf.keras.layers.Dense(128, name='pitch')(x),
  'step': tf.keras.layers.Dense(1, name='step')(x),
  'duration': tf.keras.layers.Dense(1, name='duration')(x),
}

model = tf.keras.Model(inputs, outputs)

loss = {
      'pitch': tf.keras.losses.SparseCategoricalCrossentropy(
          from_logits=True),
      'step': mse_with_positive_pressure,
      'duration': mse_with_positive_pressure,
}

optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)

model.compile(loss=loss, optimizer=optimizer)

model.summary()
Model: "model"
__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to                     
==================================================================================================
 input_1 (InputLayer)           [(None, 25, 3)]      0           []                               
                                                                                                  
 lstm (LSTM)                    (None, 128)          67584       ['input_1[0][0]']                
                                                                                                  
 duration (Dense)               (None, 1)            129         ['lstm[0][0]']                   
                                                                                                  
 pitch (Dense)                  (None, 128)          16512       ['lstm[0][0]']                   
                                                                                                  
 step (Dense)                   (None, 1)            129         ['lstm[0][0]']                   
                                                                                                  
==================================================================================================
Total params: 84,354
Trainable params: 84,354
Non-trainable params: 0
__________________________________________________________________________________________________

Testing the model.evaluate function, you can see that the pitch loss is significantly greater than the step and duration losses. Note that loss is the total loss computed by summing all the other losses and is currently dominated by the pitch loss.

losses = model.evaluate(train_ds, return_dict=True)
losses
442/442 [==============================] - 8s 4ms/step - loss: 5.0546 - duration_loss: 0.1709 - pitch_loss: 4.8489 - step_loss: 0.0348
{'loss': 5.054589748382568,
 'duration_loss': 0.17090104520320892,
 'pitch_loss': 4.848920822143555,
 'step_loss': 0.03476677089929581}

One way balance this is to use the loss_weights argument to compile:

model.compile(
    loss=loss,
    loss_weights={
        'pitch': 0.05,
        'step': 1.0,
        'duration':1.0,
    },
    optimizer=optimizer,
)

The loss then becomes the weighted sum of the individual losses.

model.evaluate(train_ds, return_dict=True)
442/442 [==============================] - 3s 4ms/step - loss: 0.4481 - duration_loss: 0.1709 - pitch_loss: 4.8489 - step_loss: 0.0348
{'loss': 0.4481140077114105,
 'duration_loss': 0.17090104520320892,
 'pitch_loss': 4.848920822143555,
 'step_loss': 0.03476677089929581}

Train the model.

callbacks = [
    tf.keras.callbacks.ModelCheckpoint(
        filepath='./training_checkpoints/ckpt_{epoch}',
        save_weights_only=True),
    tf.keras.callbacks.EarlyStopping(
        monitor='loss',
        patience=5,
        verbose=1,
        restore_best_weights=True),
]
%%time
epochs = 50

history = model.fit(
    train_ds,
    epochs=epochs,
    callbacks=callbacks,
)
Epoch 1/50
442/442 [==============================] - 4s 5ms/step - loss: 0.3407 - duration_loss: 0.0984 - pitch_loss: 4.2120 - step_loss: 0.0317
Epoch 2/50
442/442 [==============================] - 2s 5ms/step - loss: 0.3261 - duration_loss: 0.0917 - pitch_loss: 4.0778 - step_loss: 0.0305
Epoch 3/50
442/442 [==============================] - 2s 5ms/step - loss: 0.3211 - duration_loss: 0.0894 - pitch_loss: 4.0290 - step_loss: 0.0302
Epoch 4/50
442/442 [==============================] - 2s 5ms/step - loss: 0.3177 - duration_loss: 0.0887 - pitch_loss: 3.9848 - step_loss: 0.0298
Epoch 5/50
442/442 [==============================] - 2s 5ms/step - loss: 0.3124 - duration_loss: 0.0855 - pitch_loss: 3.9504 - step_loss: 0.0294
Epoch 6/50
442/442 [==============================] - 2s 5ms/step - loss: 0.3092 - duration_loss: 0.0837 - pitch_loss: 3.9296 - step_loss: 0.0290
Epoch 7/50
442/442 [==============================] - 2s 5ms/step - loss: 0.3057 - duration_loss: 0.0812 - pitch_loss: 3.9189 - step_loss: 0.0286
Epoch 8/50
442/442 [==============================] - 2s 5ms/step - loss: 0.3051 - duration_loss: 0.0809 - pitch_loss: 3.9156 - step_loss: 0.0284
Epoch 9/50
442/442 [==============================] - 2s 5ms/step - loss: 0.3014 - duration_loss: 0.0777 - pitch_loss: 3.9135 - step_loss: 0.0280
Epoch 10/50
442/442 [==============================] - 2s 5ms/step - loss: 0.3044 - duration_loss: 0.0807 - pitch_loss: 3.9119 - step_loss: 0.0281
Epoch 11/50
442/442 [==============================] - 2s 5ms/step - loss: 0.3052 - duration_loss: 0.0813 - pitch_loss: 3.9132 - step_loss: 0.0283
Epoch 12/50
442/442 [==============================] - 2s 5ms/step - loss: 0.3015 - duration_loss: 0.0781 - pitch_loss: 3.9165 - step_loss: 0.0276
Epoch 13/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2983 - duration_loss: 0.0763 - pitch_loss: 3.9034 - step_loss: 0.0268
Epoch 14/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2944 - duration_loss: 0.0734 - pitch_loss: 3.8927 - step_loss: 0.0264
Epoch 15/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2919 - duration_loss: 0.0722 - pitch_loss: 3.8834 - step_loss: 0.0256
Epoch 16/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2899 - duration_loss: 0.0705 - pitch_loss: 3.8814 - step_loss: 0.0253
Epoch 17/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2897 - duration_loss: 0.0699 - pitch_loss: 3.8801 - step_loss: 0.0258
Epoch 18/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2898 - duration_loss: 0.0705 - pitch_loss: 3.8767 - step_loss: 0.0255
Epoch 19/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2846 - duration_loss: 0.0670 - pitch_loss: 3.8656 - step_loss: 0.0243
Epoch 20/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2862 - duration_loss: 0.0677 - pitch_loss: 3.8634 - step_loss: 0.0254
Epoch 21/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2827 - duration_loss: 0.0661 - pitch_loss: 3.8430 - step_loss: 0.0244
Epoch 22/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2834 - duration_loss: 0.0664 - pitch_loss: 3.8386 - step_loss: 0.0251
Epoch 23/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2798 - duration_loss: 0.0646 - pitch_loss: 3.8223 - step_loss: 0.0241
Epoch 24/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2798 - duration_loss: 0.0658 - pitch_loss: 3.8098 - step_loss: 0.0235
Epoch 25/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2842 - duration_loss: 0.0692 - pitch_loss: 3.8145 - step_loss: 0.0243
Epoch 26/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2763 - duration_loss: 0.0645 - pitch_loss: 3.7876 - step_loss: 0.0225
Epoch 27/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2707 - duration_loss: 0.0600 - pitch_loss: 3.7790 - step_loss: 0.0218
Epoch 28/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2687 - duration_loss: 0.0595 - pitch_loss: 3.7588 - step_loss: 0.0213
Epoch 29/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2666 - duration_loss: 0.0583 - pitch_loss: 3.7494 - step_loss: 0.0209
Epoch 30/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2652 - duration_loss: 0.0578 - pitch_loss: 3.7432 - step_loss: 0.0202
Epoch 31/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2680 - duration_loss: 0.0605 - pitch_loss: 3.7344 - step_loss: 0.0208
Epoch 32/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2611 - duration_loss: 0.0544 - pitch_loss: 3.7291 - step_loss: 0.0202
Epoch 33/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2574 - duration_loss: 0.0532 - pitch_loss: 3.7088 - step_loss: 0.0188
Epoch 34/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2566 - duration_loss: 0.0525 - pitch_loss: 3.7087 - step_loss: 0.0186
Epoch 35/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2555 - duration_loss: 0.0518 - pitch_loss: 3.7002 - step_loss: 0.0186
Epoch 36/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2532 - duration_loss: 0.0508 - pitch_loss: 3.6877 - step_loss: 0.0180
Epoch 37/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2508 - duration_loss: 0.0488 - pitch_loss: 3.6811 - step_loss: 0.0180
Epoch 38/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2485 - duration_loss: 0.0476 - pitch_loss: 3.6681 - step_loss: 0.0174
Epoch 39/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2534 - duration_loss: 0.0503 - pitch_loss: 3.6813 - step_loss: 0.0191
Epoch 40/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2538 - duration_loss: 0.0522 - pitch_loss: 3.6683 - step_loss: 0.0182
Epoch 41/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2488 - duration_loss: 0.0485 - pitch_loss: 3.6546 - step_loss: 0.0175
Epoch 42/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2428 - duration_loss: 0.0442 - pitch_loss: 3.6382 - step_loss: 0.0167
Epoch 43/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2406 - duration_loss: 0.0428 - pitch_loss: 3.6387 - step_loss: 0.0159
Epoch 44/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2375 - duration_loss: 0.0407 - pitch_loss: 3.6240 - step_loss: 0.0156
Epoch 45/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2470 - duration_loss: 0.0480 - pitch_loss: 3.6451 - step_loss: 0.0167
Epoch 46/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2354 - duration_loss: 0.0396 - pitch_loss: 3.6093 - step_loss: 0.0153
Epoch 47/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2340 - duration_loss: 0.0388 - pitch_loss: 3.6043 - step_loss: 0.0150
Epoch 48/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2352 - duration_loss: 0.0398 - pitch_loss: 3.6002 - step_loss: 0.0153
Epoch 49/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2312 - duration_loss: 0.0370 - pitch_loss: 3.5893 - step_loss: 0.0148
Epoch 50/50
442/442 [==============================] - 2s 5ms/step - loss: 0.2343 - duration_loss: 0.0397 - pitch_loss: 3.5839 - step_loss: 0.0153
CPU times: user 2min 40s, sys: 33.3 s, total: 3min 13s
Wall time: 1min 49s
plt.plot(history.epoch, history.history['loss'], label='total loss')
plt.show()

png

Generate notes

To use the model to generate notes, you will first need to provide a starting sequence of notes. The function below generates one note from a sequence of notes.

For note pitch, it draws a sample from softmax distribution of notes produced by the model, and does not simply pick the note with the highest probability. Always picking the note with the highest probability would lead to repetitive sequences of notes being generated.

The temperature parameter can be used to control the randomness of notes generated. You can find more details on temperature in Text generation with an RNN.

def predict_next_note(
    notes: np.ndarray, 
    keras_model: tf.keras.Model, 
    temperature: float = 1.0) -> int:
  """Generates a note IDs using a trained sequence model."""

  assert temperature > 0

  # Add batch dimension
  inputs = tf.expand_dims(notes, 0)

  predictions = model.predict(inputs)
  pitch_logits = predictions['pitch']
  step = predictions['step']
  duration = predictions['duration']

  pitch_logits /= temperature
  pitch = tf.random.categorical(pitch_logits, num_samples=1)
  pitch = tf.squeeze(pitch, axis=-1)
  duration = tf.squeeze(duration, axis=-1)
  step = tf.squeeze(step, axis=-1)

  # `step` and `duration` values should be non-negative
  step = tf.maximum(0, step)
  duration = tf.maximum(0, duration)

  return int(pitch), float(step), float(duration)

Now generate some notes. You can play around with temperature and the starting sequence in next_notes and see what happens.

temperature = 2.0
num_predictions = 120

sample_notes = np.stack([raw_notes[key] for key in key_order], axis=1)

# The initial sequence of notes; pitch is normalized similar to training
# sequences
input_notes = (
    sample_notes[:seq_length] / np.array([vocab_size, 1, 1]))

generated_notes = []
prev_start = 0
for _ in range(num_predictions):
  pitch, step, duration = predict_next_note(input_notes, model, temperature)
  start = prev_start + step
  end = start + duration
  input_note = (pitch, step, duration)
  generated_notes.append((*input_note, start, end))
  input_notes = np.delete(input_notes, 0, axis=0)
  input_notes = np.append(input_notes, np.expand_dims(input_note, 0), axis=0)
  prev_start = start

generated_notes = pd.DataFrame(
    generated_notes, columns=(*key_order, 'start', 'end'))
1/1 [==============================] - 0s 490ms/step
1/1 [==============================] - 0s 25ms/step
1/1 [==============================] - 0s 23ms/step
1/1 [==============================] - 0s 23ms/step
1/1 [==============================] - 0s 24ms/step
1/1 [==============================] - 0s 23ms/step
1/1 [==============================] - 0s 23ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 23ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 23ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 20ms/step
1/1 [==============================] - 0s 23ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 24ms/step
1/1 [==============================] - 0s 24ms/step
1/1 [==============================] - 0s 23ms/step
1/1 [==============================] - 0s 23ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 23ms/step
1/1 [==============================] - 0s 23ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 23ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 23ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 23ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 23ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 23ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 23ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 23ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 22ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 21ms/step
1/1 [==============================] - 0s 22ms/step
generated_notes.head(10)
out_file = 'output.mid'
out_pm = notes_to_midi(
    generated_notes, out_file=out_file, instrument_name=instrument_name)
display_audio(out_pm)
fluidsynth: warning: SDL2 not initialized, SDL2 audio driver won't be usable

You can also download the audio file by adding the two lines below:

from google.colab import files
files.download(out_file)

Visualize the generated notes.

plot_piano_roll(generated_notes)

png

Check the distributions of pitch, step and duration.

plot_distributions(generated_notes)

png

In the above plots, you will notice the change in distribution of the note variables. Since there is a feedback loop between the model's outputs and inputs, the model tends to generate similar sequences of outputs to reduce the loss. This is particularly relevant for step and duration, which has uses MSE loss. For pitch, you can increase the randomness by increasing the temperature in predict_next_note.

Next steps

This tutorial demonstrated the mechanics of using an RNN to generate sequences of notes from a dataset of MIDI files. To learn more, you can visit the closely related Text generation with an RNN tutorial, which contains additional diagrams and explanations.

An alternative to using RNNs for music generation is using GANs. Rather than generating audio, a GAN-based approach can generate a entire sequence in parallel. The Magenta team has done impressive work on this approach with GANSynth. You can also find many wonderful music and art projects and open-source code on Magenta project website.