On the urging of Rob, I looked at Root Mean Square (RMS) for determining onsets and offsets. RMS does and excellent job of identifying note onsets and offsets if they are completely disjunct. If two notes are played without space in between the RMS does not find it because RMS just looks at amplitude of the incoming signal. I am going to compensate for this by checking for note onsets using Spectral Flux and a combination of spectral flux and RMS to find offsets. I have to experiment a lot with the values of what is an acceptable limit to call turning on. But that is a lot of tuning and just trial and error.
Now it is time to combine both parts, the pitch detection and onset detection. I haven't finished implementing it yet but I outlined some psuedo code to make sure I understood the project flow.

          
findOnsetPitch(buffer){
  run entire onset_detection routine
  var noteOn = false
  Iterate through array of flux peaks {
    if(value != 0 && note){
      pass next 1024 samples to pitch_detection
      create new Note instance (startSamples, pitch in Hz returned from pitch_detect)
      set NoteOn to true
    } else if(value != 0 && noteOn == true){
      if(current RMs value < .3){
        noteOn = false
        add where note stopped to the last created Note instance
      } else {
        add where note stopped to last Note instance
        pass next 1024 samples to pitch_detect
        create new Note instance (startSamples, pitch in Hz returned from pitch_detect)
      }
    } else if(current RMS value < .3){
      noteOn = false
      add where note stopped to last Note instance
    }
  }
}
          
        

The Note instance is a new class I created to handle each note. It will contain where a note starts, where it stops, and it's pitch. It is to help contain all the information I need for a single note. I started putting together the combined function, but it is very Frankenstein's Monster right now. I will continue to work on it this week. I don't have high hopes for it working right the first time at all.

An update to last week, turns out the issue with my code had nothing to do with pass by reference. The issue, which is quite blaringly obvious now, is that when trying to get the last element of the array I called the array currentNoteArray not currNoteArray. So the functionality of pitch detection now works properly and will save each new pitch that was found.

This week I started looking at this onset detection. This method uses spectral flux. Spectral flux is a measure of how quicly the power spectrum of a signal is changing. The power is found by performing a fast fourier transform and then summing the total value of the signal.

The method that was implemented by default only summed increases in the energy from the previous bin. The application provides a lovely visualization. This is the visualization with the current method for 3 sine wave blips.

The red line is the input signal. The green line is the spectral flux. The purple line is the activation threshold determined by the local average of the signal. The blue line indicates where the spectral flux exceeded the threshold value. As you can see, it only activates on the onset of the note not when it ends. My thought was to change the inclusion of only increases to both increases and decreases by using absolute value.

This method is able to capture the onset and offset of notes. Well maybe not notes but definitely sine waves. There is some error with the middle of the notes, but that was present in the original method so I will have to dive into the algorithm further. I have not implemented this into my tool displayed on the website. I have been playing with this offline.

So first step of the new project was getting pitch detection working. This was simple enough, I just had to add the the module I mentioned in last week's post into my webpage. This went swimmingly besides issues with CSS, which I chose not to address because that is not the point of this class. So I got that working with all three forms of audio, live input, recorded sample, and an oscilltor.

Later on in the week I tried to make the program to track each new pitch that was made and hold onto it to display it when the recording was done. This kind of worked, but also didn't work at all. I am fairly certain the path I am taking is right, but I think fundamentally I am doing the wrong thing. I believe the issue is arrising from an error with pass by reference vs pass by value. Currently I have written:

          
var note =  noteFromPitch( pitch );
if (currNoteArray.length == 0) {
	currNoteArray.push(note);
} else if (currNoteArray[currentNoteArray.length - 1] != note){
	currNoteArray.push(note);
}
      

Pitch is a frequency and note is a midi note value. Hypothetically, this code should add the value of the midi note value to the end of my note array if it is the first one or if it is different from the last note added to the array. I think what might be happening, is an error with passing by reference. I would like currNoteArray to store just the value of note, but I think it is storing the address location which is messing up displaying the note later on in the program. I tried googling a solution to it, but I think I can't find a solution because I am thinking in a c++ mindset and not a scripting mindset. I was going to get Rob's opinion because I can't seem to get it to work.

So now that I have web audio working on my website, the future now includes inplementing my project idea as a web app. The essential componnents are pitch identification, onset identification (for the start of notes and their finishes), and some way to record the information and convert it to some form of common format (probably music xml). I have found some code for a potential pitch detection software on github. I'm not sure if there is a way to include this as a module or if I just copy the code over. The options for onset detection are Web audio specific one or this generic javascript one. The web audio specific one looks like it is too complicated for my needs, but it is already setup to work with web audio. The generic javascript one looks like it is more appropriately scoped, but I would have to do all the web audio setup myself. This week I will try to get the pitch detection working as I think that will be quicker, but I will move on to onset detection shortly after. I will do all the transfering to musicxml data type by myself unless there is a super easy module to do it for me.

I have created another tab on my website for the work I am going to be doing with Web Audio. As of today I have it playing some lovely royalty free music. I followed this tutorial for setting up this very basic playback. This took me about an hour and a half (less if I would stop spelling things wrong ¯\_(ツ)_/¯). From here I am going to look into how to including libraries/packages in web audio. I assume it integrates with the javascript, but this is a whole new world for me.