Send MIDI Sequencer to a MIDI Output

[novaXire]

Hello,

I would like to know if you have plan to add the possibility to send MIDI information to a MIDI Output or if you want to lock alphaTab to use only TinySoundFont?

I made a new class in AlphaTab.Windows and replace _synthesizer in alphaTab by this new class and MIDI works.

[Danielku15]

Hi.
AlphaTab is not locked to use only TinySoundFont, also not today. But I also think that the underlying synthesizer components are not suitable to send midi directly to another device. AlphaTab does not have a real time synthesizer or a real time sequencer. It does not send the midi events exactly at the time when they should be played. Instead it internally does the synthesizing always as fast as possible and buffers the samples until some extend.

The logic internally works like this: On each platform there is a raw PCM audio output (Web Audio Context, NAudio WaveProvider32 etc. When this audio output requests samples to be played, alphaTab grabs them from an internal audio buffer. If this audio buffer is filled less than 50%, it requests another 50% of the audio samples. Internally a loop starts which sends midi events to TinySoundFont and generates the audio samples.

But to connect an appropriate midi output, you need a real time sequencer, which plays the midi events in the exact speed. Today you would always receive in regular intervals, based on the internal audio buffers, a list of buffers which have been synthesized.

I don't know your exact use case yet. But in general there are two features related to the midi you might be interested:

1. You can generate the midi of a song independently using the low level APIs.
2. You can receive any played midi event in quasi-real time via a special event.

Depending on what you want to achieve, you might have the features already available. You can use any components understanding Midi and send the generated midi along. Some conversion between the event classes is obviously needed or you go via binary writing/reading. This can be any synth of midi library depending on your preference.

Or you use the original synth of alphaTab, but you don't load any SoundFont, but just use the internals to have a real time playback of "silence". Based on your needs you can forward the midi events or react on them manually to do something.

What I do not plan to support, is a separate real time sequencer which does a playback of the events in real time. There are enough libs for that.

[novaXire]

I didn't see the second link, i'll check this.

I just want to be able to use an external midi device or the built-in Microsoft GS Wavetable Synth or a virtual midi port to use a VST in a DAW for example in sync with the score.

[Danielku15]

I think in this scenario, you might be better set with a completely separate playback system independent from the default synthesizer. You are better set with a real time midi sequencer to send the output to any other synth or audio components.

There are various midi libraries you might use:
https://github.com/atsushieno/managed-midi
https://www.nuget.org/packages/Sanford.Multimedia.Midi/

Regarding the cursor display you would either need to use a complete own cursor overlay which works with your synth/playback system, or you might be able to call some low level apis like the normal synth:

alphaTab/src/AlphaTabApiBase.ts

Line 792 in f1f2af5

private cursorUpdateTick(tick: number, stop: boolean = false): void {

Theoretically the components within alphaTab are quite decoupled and can be used independent. But of course you might have to wire them up correctly to your execution path and logic.

Some parts of the cursor might be a bit tightly coupled with the built-in synth, but I guess we can easily work around or adapt this. Like: currently the cursors are animated according to the built-in player state and speeds.

[serenity81]

Rather than change midi output I am looking for a way to route the player audio to a different audio output device on my window machine. according to links below it seems that web audio api is being used:
https://next.alphatab.net/docs/reference/alphasynth/
https://alphatab.net/docs/reference/settings/player/outputmode/#description
https://next.alphatab.net/docs/guides/lowlevel-apis/#playing-files-via-alphasynth

I can use code like below to get Audio Devices on my PC:
` // Function to get available audio output devices
async function getAudioDevices() {
try {
// Request access to the audio device to force detection
await navigator.mediaDevices.getUserMedia({ audio: true });

            // Get all devices, filter only audiooutput devices
            const devices = await navigator.mediaDevices.enumerateDevices();
            return devices.filter(device => device.kind === 'audiooutput');
        } catch (err) {
            console.error("Error getting audio devices:", err);
            return [];
        }
    }

    // Function to populate the device dropdown
    async function populateAudioDeviceList() {
        const devices = await getAudioDevices();
        const select = document.getElementById("device-select");

        select.innerHTML = '<option value="">Select Output Device</option>'; // Reset list

        // Add each device to the dropdown
        devices.forEach(device => {
            const option = document.createElement("option");
            option.value = device.deviceId;
            option.textContent = device.label || `Device ${select.length + 1}`;
            select.appendChild(option);
        });

        if (devices.length === 0) {
            console.warn("No audio output devices found! Try granting microphone access.");
        }
    }

    // Function to change the audio output device
    async function changeOutputDevice(deviceId) {
        const audioElement = document.getElementById("audio-output");

        if (audioElement.setSinkId) {
            try {
                // Set the audio output device using the selected device ID
                await audioElement.setSinkId(deviceId);
                console.log(`Audio output changed to: ${deviceId}`);
            } catch (err) {
                console.error("Error setting audio output:", err);
            }
        } else {
            console.warn("setSinkId() is not supported in this browser.");
        }
    }

    // Function to setup the Web Audio Context and play an audio tone
    async function setupAudioContext() {
        const audioContext = new AudioContext();
        const destination = audioContext.createMediaStreamDestination();
        const audioElement = document.getElementById("audio-output");

        audioElement.srcObject = destination.stream;

        // Create an oscillator to generate sound (A4 note, 440Hz)
        const oscillator = audioContext.createOscillator();
        oscillator.type = "sine";
        oscillator.frequency.setValueAtTime(440, audioContext.currentTime);
        oscillator.connect(destination);
        oscillator.start();
    }

    // Event listener to handle the device selection
    document.getElementById("device-select").addEventListener("change", (event) => {
        changeOutputDevice(event.target.value);
    });

    // Event listener to start audio when the button is clicked
    document.getElementById("start-audio").addEventListener("click", setupAudioContext);

    // Populate the device list when the page loads
    populateAudioDeviceList();
</script>`

My question s:

1. Is there a way to route entire audio output to one other than default windows device selected in task bar
2. Can I route individual tracks output if above is possible for whole playback sound

[Danielku15]

1. This is not directly supported at this point but there are some tricks to access the internal audio context. I would still need to fiddle around with the possibilities but:

• You can access the audio context via: api.player._output._context. The audio context should be persistent and not changing. So you could reconfigure it before starting the playback. https://github.com/CoderLine/alphaTab/blob/develop/src/platform/javascript/AlphaSynthWebAudioOutputBase.ts
• Its connected to the destination when playback starts, changing things during playback will likely not work unless web audio changes the destination under-the-hood. Likely we get a new node when changing we have to connect to. Needs some prototyping to see how it behaves. https://github.com/CoderLine/alphaTab/blob/develop/src/platform/javascript/AlphaSynthAudioWorkletOutput.ts#L207
• The class definition for AlphaSynthWebAudioOutputBase is accessible via alphaTab.synth.AlphaSynthWebAudioOutputBase So you could change the method implementation before any the alphaTab code runs:

const proto = Object.getPrototypeOf(new alphaTab.synth.AlphaSynthWebAudioOutputBase());
const oldCreateAudioContext= proto.createAudioContext;
proto.createAudioContext = function() {
    console.log('Inside createAudioContext', this);
	return new AudioContext({sinkId : fromSomewhere});
};

Maybe you can prepare a small jsfiddle on your preferred setup from an end-user experience (with some audio selection and alphaTab in place)? This would allow me best to design an API that will actually work out for you to change the output. Just add a placeholder at the spot where you would tell alphaTab what output to use.

3. No this is not available and will likely also not become a feature in near future due to its complexity handling the output side.

Internally the synthesizer works with midi and midi channels (not tracks) and one track can consist of multiple channels. It would require quite some significant rearchitecturing to support multiple parallel audio streams to be sent to different outputs instead of mixing them together. And these multiple outputs have to be synchronized in playback. All this somehow needs to be configurable and in worst case even changable dynamically.

This is already very deep in the solution space, I don't understand yet how a end-user would want to have such flexibility. I'm not aware of any software providing such a functionality either (Guitar Pro, MuseScore, Audacity, Capella, Timidity, Fluidsynth..). So I currently rate the effort rather not worth the gain.

But I might be missing information to judge things right, so feel free to share more insights trying to convince me of the value 😉

Here some references and thoughts on how the implementation could be organized:

• When the output needs new audio to be generated (buffering) the next samples across all voices and channels are generated here

  alphaTab/src/synth/AlphaSynth.ts

  Lines 187 to 228 in 159729d

  	if (
  	this.state == PlayerState.Playing &&
  	(!this._sequencer.isFinished || this._synthesizer.activeVoiceCount > 0)
  	) {
  	let samples: Float32Array = new Float32Array(
  	SynthConstants.MicroBufferSize * SynthConstants.MicroBufferCount * SynthConstants.AudioChannels
  	);
  	let bufferPos: number = 0;
  	for (let i = 0; i < SynthConstants.MicroBufferCount; i++) {
  	// synthesize buffer
  	this._sequencer.fillMidiEventQueue();
  	const synthesizedEvents = this._synthesizer.synthesize(
  	samples,
  	bufferPos,
  	SynthConstants.MicroBufferSize
  	);
  	bufferPos += SynthConstants.MicroBufferSize * SynthConstants.AudioChannels;
  	// push all processed events into the queue
  	// for informing users about played events
  	for (const e of synthesizedEvents) {
  	if (this._midiEventsPlayedFilter.has(e.event.type)) {
  	this._playedEventsQueue.enqueue(e);
  	}
  	}
  	// tell sequencer to check whether its work is done
  	if (this._sequencer.isFinished) {
  	break;
  	}
  	}
  	// send it to output
  	if (bufferPos < samples.length) {
  	samples = samples.subarray(0, bufferPos);
  	}
  	this._notPlayedSamples += samples.length;
  	this.output.addSamples(samples);
  	} else {
  	// Tell output that there is no data left for it.
  	let samples: Float32Array = new Float32Array(0);
  	this.output.addSamples(samples);
  	}

  • Here we likely need to do a "separation" into multiple distinctive streams based on some input config using midi channels.
• Active voices copy their synthesized audio into the final output buffer here: https://github.com/CoderLine/alphaTab/blob/develop/src/synth/synthesis/Voice.ts#L234
  • If we pass in separate buffers from the outside these synth parts don't need any adaption.
• For Web Audio with AudioWorklets we have the possibility to chain "nodes" and have multiple output nodes. Splitting up the audio on output side could work if the node graph is configured accordingly.
  • AudioWorklets:
    • We would need to create a worklet node with the correct number of outputs needed.

      alphaTab/src/platform/javascript/AlphaSynthAudioWorkletOutput.ts

      Line 198 in 159729d

      numberOfOutputs: 1,

    • The outputs have to be connected to the respective destinations. https://github.com/CoderLine/alphaTab/blob/159729d6f66e7d827fed5ded7adf7ef663165b03/src/platform/javascript/AlphaSynthAudioWorkletOutput.ts#L207ahttps://github.com/CoderLine/alphaTab/blob/159729d6f66e7d827fed5ded7adf7ef663165b03/src/platform/javascript/AlphaSynthAudioWorkletOutput.ts#L104-L105
  • ScriptProcessorNode: It can only have multiple output channels (e.g. stereo) but not multiple output nodes.
• For .net using NAudio we'd need multiple DirectSoundOut instances for multiple devices. But synchronization would have to happen manually, respecting aspects like latency and buffering (not easy).
• For Android its the same challenge like on .net of manual output synchronization.

[Danielku15]

Update regarding (1): I made some local tests and changing the audio device works fine (even on-the-fly) via await at.player._output._context.setSinkId(devices.deviceId). You can use this solution as intermediate workaround until we add official support without relying on internals.

I added #1904 to track this feature. Please let me know early if something on the planned feature design does not match your needs.

[serenity81]

Thanks for the update, I will try this simpler approach and revert back tomorrow. As prior solution felt too complicated for a simple problem in my eyes (I know its hard to implement coding wise at your end as its not standard). But this at.player._output._context.setSinkId(devices.deviceId) approach makes more sense.

I need user to select each various audio devices for various scores/tracks for control via midi devices. I know that per track you said is almost impossible, I wish it was the case as it will make my job a lot easier. But I have alternative solution where I detect various midi controls being made externally via JS and apply them to AlphaTab. For example, if user moved a Hardware fader the JS code will move SW slider for a particular track via https://alphatab.net/docs/reference/api/changetrackvolume. So I am faking what i want to do.

So I give the user flexibility in controlling what happening on screen and also allow user to control output and other real physical instruments using one hardware mixer. Its kinda working for now, and for me atleast its very exciting. I will keep you updated

[serenity81]

Update regarding (1): I made some local tests and changing the audio device works fine (even on-the-fly) via await at.player._output._context.setSinkId(devices.deviceId). You can use this solution as intermediate workaround until we add official support without relying on internals.

I added #1904 to track this feature. Please let me know early if something on the planned feature design does not match your needs.

That works amazing simple, it took me a min to implement Thanks. I wish there was a way to give each channel/track a separate audio device. I know its very hard as you said, but mybe there is a way to fake it like I did for the hardware side.