Random initialization, unit test and threshold
I open a new issue to gather all the discussions. Please don't make any action to these problems until the discussion is closed.
We have a mix of problems in the unit tests:
- we have some unknown source of numerical noise (somewhere between pinocchio and numpy) that makes it difficult to use a unique threshould for our numerical unittests.
- the values used in the unittests are random, and we don't have any mean of selecting the seed (at least for the random values of Pinocchio, that does not expose the seed setter of Eigen).
- @cmastall started to change the values of the unittest in a tentative to unified and make more clean the unittest.
This mix of problems is even more difficult when using the unittest for CI, as the random initialization may fail only some times.
We may consider several solutions:
- increase the values used for threshold, as done in !121 (merged)
- set the seed
- run the tests several times and do statistic, instead of deciding on the first failure.
Increasing the values should be done with care. One compromise could be to only increase the value during CI, and keep in small (more conservative test) for personal tests.
Setting the seed implies new modification in eigenpy, that I don't want to do. @jcarpent anybody exposed the seed in the past?
Running the tests several times with some conservative values is possibly the best solution.