Re: [boost] Fwd: [asio]Extension for audio device service

Subject: Re: [boost] Fwd: [asio]Extension for audio device service
From: Brendon Costa (brendon.j.costa_at_[hidden])
Date: 2013-04-24 06:37:12

Hi Adrian,

I am interested in helping out with this. I have also been thinking about
developing a boost library for use of audio hardware for about a year now.
But have not had the time and motivation to get around to it.

The general idea of using boost::asio to provide an interface that is
consistent with other IO in boost is a very good one if we can achieve the
realtime requirements through it (which it sounds like we can from what you
have said).

I would suggest that you need to support multiple audio "backends" per OS
when it comes to device selection. For example, on linux there is a
plethora of choices and no one choice is "correct". The user should be
given the option to choose which backend(s) to use/support. You will notice
that some apps like to provide all options, and others work with only one
backend.

I was thinking of proposing three related audio API's for boost a while
ago. I do believe that to do this properly is a *LOT* of work (which is why
I never really started).

1) Audio DSP Module
This is for DSP related tasks and takes care of common things like audio
format conversion, resampling etc.
It is possible to write all of this I think in a platform agnostic way,
however some platforms provide helpers for some of these things like MacOSX
and maybe we should consider how we could use them or at least integrate
with them.
Possibly define both a compile time and runtime interface as both have
advantages
This would be used in the main interface for the play/record API for the
pull/push callback
Additionally this may expand in the future with all sorts of algorithms
defined for audio processing. Though at first I would suggest it be a
minimal of what is required to support the other two modules described

2) Audio HW Play/Record Module
This is for accessing the play/record of audio on audio hardware.
I would suggest as you mentioned both push and pull modes. You can
"emulate" one with the other though be it with a lot of work (and some
latency) and some backends support both. So we should provide the interface
for all backends but export a "capabilities" of the backend indicating what
it supports natively (same as we should do for audio format).

3) Audio Mixer Module
This is to control mixer settings including things like default device
selections, volume controls, mutes, supported formats and device
enumeration.
Also, it should recieve events indicating external changes to these things.

I have designed and implemented an API at work that did a lot of this
(though the separation was not so well defined) and supported a number of
backends including DirectSound, WASAPI, CoreAudio, portaudio, PulseAudio,
OpenSLES.

I might spend some time next week to writeup a proposal for a basic outline
of what I think an interface could look like. Maybe we can compare ideas
and notes?

--- snip ----
 errcode audio_port::open(audio_format &,const audio_direction,
audio_device_mode ).
--- snip ----

Could you define what you mean by an audio_port?

Do you consider an audio_port to be an audio hardware device or a port on
that device?

For example: I consider a hadware device as representing a single physical
device having one or more ports. Where a port is one of:
* Recording port : Normal wave device input records from mic/line in etc
* Playback port : Normal wave device output plays out to line out/speakers
etc
* Monitor port : Not always available but is similar to a Recording port,
except it returns exactly what is going out of the device speakers (for
example may include CD noise or MIDI or WAVE output from other processes).
This is sometimes used for doing echo cancellation on input audio that
cancels system sounds generated from other applications and not just your
application.

I also think some professional sound cards may have additional "ports" of
type recording/playback. I think the MOTU is like that for example.

If you take that definition of a port, then the direction may become
un-necessary (at least I haven't seen a case where it would allow
bi-directional).

As for enumeration of "ports" it makes sense to have a simple function as
you described that actually wraps something that is possibly more complex
from the "Audio Mixer Module". I guess it *may* look something like:
std::list<boost::audiohw::port>
boost::audiohw::get_ports(boost::audiohw::recording_port|boost::audiohw::playback_port|boost::audiohw::monitor_port);

What do you propose to be part of the audio_format struct?
I can think of:
* Channel Count : Integer >= 1
* Data format : float32, int16, ...
* Interleaving : yes/no
* Sample rate: 48000
* Channel mapping : (have a default defined mapping, but allow user
overrides)

There should also be the ability to define some form of latency parameters
I think. Possibly even a place holder for extensible backend specific
configuration (latency details could be part of that).

--- snip ----
 errcode audio_port::connect(audio_callback &);
--- snip ----

This makes sense, would you allow multiple callbacks to be connected to the
single open port?
For recording ports, I guess this is simple in that it just calls all
connected callbacks with the same data
For playback ports, would you call multiple and then mix the audio into a
block given to the device?
If so what are the restrictions on the audio produced by the callbacks?
Should they provide exactly as much data as requested?
If so, how much data does the callback request in one call? Is it fixed,
variable, configurable (latency parameters I mentioned in the config)
If they could return less than requested, then the mixing becomes more
complex
Can you connect callbacks after you have started the device?
I would assume that because the audio format has been defined already, then
the callback can be verified based on the callback type.
I.e. If defining a float32 format, then this could verify the callback data
type is a float. Or you could go the C way and use void* but there are
advantages to the type safety

I guess a proposal for the callback could be:
result on_audio_cb(float* data_out, size_t data_out_size, const
audiohw::format& audio_format);

How do you plan to handle synchronous input/output audio?
What I mean by this is that the consumption and production of audio is
synchronized so when you get 20msec of audio from the input you must also
output 20msec.
This has benefits to various audio algorithms, and can be achieved across
different devices with special clock monitoring and resampling techniques.
It is generally the case for input/output on the same physical audio
hardware that they are synchronized, but is not across different hardware
as often the audio card clocks can be un-synchronized.
This decision can affect how to structure the open() call. For example, you
may open both input/output at the same time if you want to support this.

There are a few options the first is my preferred though is more difficult
to use:
* You can define for synchronous handling, that the record/playback/monitor
callbacks will always be called in order
* You could do what portaudio does and define the callback to have (but
that doesn't work well with the previous design)
result on_audio_cb(float* data_out, const float* data_in, size_t data_size,
const audiohw::format& audio_out_format, const audiohw::format&
audio_in_format);

How do you plan to handle asynchronous errors in the audio device?
Maybe an error code passed to the on_audio_cb() or a seperate callback that
can be registered for errors?

On Wednesday, 24 April 2013, adrien courdavault wrote:

> Hello.
>
> I make this new thread to be clearer.
> There is currently no way to manage audio endpoints cconnection easily.
> It looks like some people might find this usefull (as I do), and I've
> been suggested on the boost dev list to try to detail this, as an
> extension to Boost.ASIO.
>
> For this reason I create this thread.
>
> I'm trying to make a very basic first draft of the concepts and see if
> this may be a good idea.
>
> I attached here the first things I've written. This is very short and
> general.
>
> I would like to know:
> * do you think I'm going in the right direction by seing this as
> Boost.ASIO extension.
> * do you have suggestions
> * would someone like to participate ?
>
> Thank you
>