Need for a baseline
what is a baseline ?
A baseline is a minimum or starting point used for comparisons.
@raulsenaferreira, you raised the point of the importance to find a baseline for comparison. Can you provide more clearly what are the benefits of this ? were you suggesting for making a baseline study ?
purpose of a baseline study
The purpose of a baseline study is to provide an information base against which to monitor and assess an activity’s progress and effectiveness during implementation and after the activity is completed. Sometimes the data needed for a baseline, against which to measure the degree and quality of change during an activity’s implementation, will already exist. [source]
If we want to assess the performance/effectiveness of our localization model, we need to find a localization model that is comparable to our. That is, a model which tries to localize a sound source in the exact same conditions. If the conditions are different, than I think that we cannot assess anything unless we are able to prove/argue rigorously what these conditions changes implies and how they affect the performance assessment.
Finding such performance baseline seems complicated to me. I think also that the training/cross-validation loss on repeated training is enough to discuss about the ability of our model to generalize or to overfit on our data. In my opinion, the problem is not really to know if the model overfit our data, but if the data is representative enough of the problem we are trying to tackle.
For example:
- do we want that our model perform well with microphones that are not calibrated and that may have different frequency responses, or do we only require our model to perform well when microphones have identical frequency responses ?
- do we want our model to be able to localize a dirac sound only, or different sounds ? If so, which kind of sounds should these be ? For example, it is predictable that the lesser the bandwidth of a sound, the lesser the model will be able to predict the sound source azimuth ; in particular, it is not possible to localize a sine sound, because it has the lowest possible spectrum (only one frequency).
starting point for a baseline study
I found this article, which discuss the sound source localization with tones having different bandwidths. Maybe it could be a starting point for a baseline study. However, it seems that the experiments have been done in rooms, and this could affect the localization accuracy, making our comparison irrelevant to this baseline.
what do you think @all ?