Commit 7c0791b4 authored by ehebrard's avatar ehebrard
Browse files

ok

parent dcd17ffd
......@@ -85,13 +85,13 @@ int main(int argc, char *argv[]) {
WeightedDataset<int> input;
////// READING
// try {
// read_binary(input, opt);
// } catch (const std::exception &e) {
if (opt.verbosity >= DTOptions::NORMAL)
cout << "c format not recognized or input non-binary, binarizing...\n";
read_non_binary(input, opt);
// }
try {
read_binary(input, opt);
} catch (const std::exception &e) {
if (opt.verbosity >= DTOptions::NORMAL)
cout << "c binarizing...\n";
read_non_binary(input, opt);
}
if (opt.sample < 1)
input.sample(opt.sample);
......
......@@ -210,14 +210,12 @@ The solver DL8.5 is a dynamic programming algorithm for the minimum error bounde
\KwResult{The minimum error on $\negex,\posex,\features$ for decision trees of depth $\maxd$}
$\error \gets \min(|\negex|,|\posex|)$\;
\lIf{$\maxd = 0$ or $\error = 0$} {
\Return $\error$
}
\If{$\maxd > 0$ and $\error > 0$} {
\ForEach{$\afeat \in \features$} {
$\error \gets \min(\error, \dynprog(\negex(\{\afeat\}),\posex(\{\afeat\}),\features \setminus \{\afeat\},\maxd-1)$\;
\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ $ + \dynprog(\negex(\{\bar{\afeat}\}),\posex(\{\bar{\afeat}\}),\features \setminus \{\afeat\},\maxd-1))$\;
}
}
\Return $\error$\;
\end{algorithm}
......@@ -1260,6 +1258,7 @@ Moreover, at every branch, ``informationless'' features (i.e., features $\afeat$
\section{Experimental Results}
\label{sec:exp}
All experiments were run
on 7 cluster nodes, each with 36 Intel Xeon CPU E5-2695 v4 2.10GHz cores
......@@ -1279,6 +1278,335 @@ As \dleight does not provide a solution for every data set (on some instance it
instead of absolute values, we provide the average relative difference in error and accuracy w.r.t. \budalg, however, and only for the data sets where a decision tree was found. Similarly, we report the average cpu time ratio w.r.t. \budalg, however, only for instances which were proven optimal by both algorithms\footnote{every instance proven optimal by \dleight is also proven optimal by \budalg and \murtree}.
\begin{table}[htbp]
\begin{center}
\begin{footnotesize}
\tabcolsep=1.7pt
\input{src/tables/summaryclasses.tex}
\end{footnotesize}
\end{center}
\caption{\label{tab:summaryaccsmall} Comparison with the state of the art}
\end{table}
\subsection{Computing deep classifiers}
Next, we shift our focus to how fast can we obtain accurate trees and how fast can we improve the accuracy over basic solutions found by heuristics.
Therefore, in order to have a baseline, we use a well known heuristic: \cart (we ran its implementation in scikit-learn).
Here we report the average error after a given period of time (3 seconds, 10 seconds, 1 minute or 5 minutes) in Table~\ref{tab:summaryspeed}.
\medskip
We can see that the first solution found by our algorithm has comparable accuracy (actually, often slightly better) to the one found by \cart. The implementation of \cart in scikit-learn does not seem to be very efficient computationally. However, this is not so relevant as it is clear that one greedy run of the heuristic can be implemented to be as fast as the first dive of \budalg. The point of this experiment is threefold: first, it shows that the first solution is very similar to that found by \cart (here we believe that the only difference is to use the feature yielding the minimum error instead of the feature with minimum Gini impurity for the deepest feature tests); second, this tree
is found extremely quickly, and there is no scaling issue with respect to the depth of the tree or with respect to the size of the data set; third, even for large data sets and deep trees, the accuracy of the initial classifier can be significantly improved given a reasonable computation time.
%Moreover, although we would need larger data sets to be confident about that, it seems that our algorithm is faster than \cart to find this first decision tree. %One can conjecture that \cart uses more sophisticated heuristic choices to explain these two observations.
%Then, in most cases, it is possible to improve the first solution significantly within a few seconds. Notice that for larger depth, improving the initial solution is harder and the 3s time limit is comparatively tighter than for smaller trees, so the gain of \budalg over \cart is more sensible for small trees.
\begin{table}[htbp]
\begin{center}
\begin{footnotesize}
\tabcolsep=1.8pt
\input{src/tables/summaryspeed.tex}
\end{footnotesize}
\end{center}
\caption{\label{tab:summaryspeed} Comparison with state the of the art: computing accurate trees}
\end{table}
Figure~\ref{fig:cactus} reports the evolution of accuracy over time, giving a good view of the difference between \murtree and \budalg during search. The accuracy of the tree returned by \cart is given for reference.
We can see in those graphs that \murtree finds an initial tree extremely quickly, although with very low accuracy. This is because \murtree show progress even when the tree is not complete, i.e., the first solution is always a single node with the most promising feature. We can see in Table~\ref{tab:summaryspeed} that this is indeed always the same feature, irrespective of the maximum depth.
\begin{figure}
\subfloat[depth=3]{\input{src/tables/xscerror3.tex}}
\subfloat[depth=7]{\input{src/tables/xscerror7.tex}}
\subfloat[depth=10]{\input{src/tables/xscerror10.tex}}
% \subfloat[depth=8]{\input{src/tables/xscerror8.tex}}
% \subfloat[depth=9]{\input{src/tables/xscerror9.tex}}
% \subfloat[depth=10]{\input{src/tables/xscerror10.tex}}
\caption{\label{fig:cactus}Accuracy over time, averaged across all data sets}
\end{figure}
\bibliographystyle{plain}
\bibliography{src/references}
% \end{document}
\clearpage
% \newgeometry{bottom=2cm,top=2cm,margin=1cm}
\section*{Appendix}
The benchmark of classification data set we used is described in Table~\ref{tab:info}. It consists of 30 data sets
commonly used in related work articles, to which we added some large data sets from Kaggle: \texttt{bank}, \texttt{titanic}, \texttt{surgical-deepnet} and \texttt{weather-aus}, as well as the \texttt{mnist} data sets, \texttt{adult\_discretized} and \texttt{compas\_discretized}. We report the number of data points ($|\allex|$), the number of features ($|\features|$), the same parameters after preprocessing (respectively $|\allex|^*$ and $|\features|^*$), and the ``noise'' ratio, that is: $2|\posex \cap \negex|/(|\posex|+|\negex|)$.
\medskip
Then we report the raw data from our experimental comparison with the state of the art for $\mdepth=3,4,5,7,10$ and for the four size catagories in the following tables:
\tabcolsep=10pt
\begin{center}
\begin{tabular}{lrrrr}
\toprule
\multirow{2}{*}{$\mdepth$}& \multicolumn{2}{c}{$\numex < 1500$} & \multicolumn{2}{c}{$\numex \geq 1500$} \\
& $\numfeat < 100$ & $\numfeat \geq 100$ & $\numfeat < 100$ & $\numfeat \geq 100$ \\
% & $\numex < 1500; \numfeat < 100$ & $\numex < 1500; \numfeat \geq 100$ & $\numex \geq 1500; \numfeat < 100$ & $\numex \geq 1500; \numfeat \geq 100$ \\
\midrule
$3$ & Table~\ref{tab:all31} & Table~\ref{tab:all32} & Table~\ref{tab:all33} & Table~\ref{tab:all34} \\
$4$ & Table~\ref{tab:all41} & Table~\ref{tab:all42} & Table~\ref{tab:all43} & Table~\ref{tab:all44} \\
$5$ & Table~\ref{tab:all51} & Table~\ref{tab:all52} & Table~\ref{tab:all53} & Table~\ref{tab:all54} \\
$7$ & Table~\ref{tab:all71} & Table~\ref{tab:all72} & Table~\ref{tab:all73} & Table~\ref{tab:all74} \\
$10$ & Table~\ref{tab:all101} & Table~\ref{tab:all102} & Table~\ref{tab:all103} & Table~\ref{tab:all104} \\
\bottomrule
\end{tabular}
\end{center}
% in Tables~\ref{tab:all3},
% \ref{tab:all4},
% \ref{tab:all5},
% \ref{tab:all6},
% \ref{tab:all7},
% \ref{tab:all8},
% \ref{tab:all9} and \ref{tab:all10}, respectively.
For every instance, we give the classification error of the best tree found within a time limit of 1h for every method. Moreover, we give the CPU time taken by each method to prove optimality when optimality is proven (in which case we mark it by a ``$^*$''), and to find the best solution otherwise. Notice that \cp\ and \dleight\ are not anytime and hence only report a solution at the end of the time limit when optimality is not proven. In this case, we write $\geq1h$.
\medskip
Every process was first run with a memory limit of 3.5GB. Many runs of \dleight, \cp\ and \binoct\ went well over that limit and were rerun with a limit of 50GB. Still, 138 runs of \binoct and 164 runs of \dleight (out of 460) went over the limit. As \binoct can output trees anytime, the data for these runs (up until the memory blow-out) are in the tables. For \dleight, however, this is marked as a ``-'' since there was no output.
%\binoct 138
%\dleight 164 %5+10+20+23+30+27+25+24
\begin{table}[htbp]%
\begin{center}%
\begin{scriptsize}%
\tabcolsep=10pt%
\input{src/tables/datasetinfo.tex}%
\end{scriptsize}%
\end{center}%
\caption{\label{tab:info} Benchmark and preprocessing data}%
\end{table}%
\begin{table}[htbp]%
\begin{center}%
\begin{scriptsize}%
\tabcolsep=2pt%
\input{src/tables/allclasses_smallsmall_3.tex}%
\end{scriptsize}%
\end{center}%
\caption{\label{tab:all31} Comparison with state of the art: $\numex<5000, \numfeat<250$, depth 3}%
\end{table}%
\begin{table}[htbp]%
\begin{center}%
\begin{scriptsize}%
\tabcolsep=2pt%
\input{src/tables/allclasses_smallbig_3.tex}%
\end{scriptsize}%
\end{center}%
\caption{\label{tab:all32} Comparison with state of the art: $\numex<5000, \numfeat \geq 250$, depth 3}%
\end{table}%
\begin{table}[htbp]%
\begin{center}%
\begin{scriptsize}%
\tabcolsep=2pt%
\input{src/tables/allclasses_bigsmall_3.tex}%
\end{scriptsize}%
\end{center}%
\caption{\label{tab:all33} Comparison with state of the art: $\numex \geq 5000, \numfeat < 250$, depth 3}%
\end{table}%
\begin{table}[htbp]%
\begin{center}%
\begin{scriptsize}%
\tabcolsep=2pt%
\input{src/tables/allclasses_bigbig_3.tex}%
\end{scriptsize}%
\end{center}%
\caption{\label{tab:all34} Comparison with state of the art: $\numex \geq 5000, \numfeat \geq 250$, depth 3}%
\end{table}%
\begin{table}[htbp]%
\begin{center}%
\begin{scriptsize}%
\tabcolsep=2pt%
\input{src/tables/allclasses_smallsmall_4.tex}%
\end{scriptsize}%
\end{center}%
\caption{\label{tab:all41} Comparison with state of the art: $\numex<5000, \numfeat<250$, depth 4}%
\end{table}%
\begin{table}[htbp]%
\begin{center}%
\begin{scriptsize}%
\tabcolsep=2pt%
\input{src/tables/allclasses_smallbig_4.tex}%
\end{scriptsize}%
\end{center}%
\caption{\label{tab:all42} Comparison with state of the art: $\numex<5000, \numfeat \geq 250$, depth 4}%
\end{table}%
\begin{table}[htbp]%
\begin{center}%
\begin{scriptsize}%
\tabcolsep=2pt%
\input{src/tables/allclasses_bigsmall_4.tex}%
\end{scriptsize}%
\end{center}%
\caption{\label{tab:all43} Comparison with state of the art: $\numex \geq 5000, \numfeat < 250$, depth 4}%
\end{table}%
\begin{table}[htbp]%
\begin{center}%
\begin{scriptsize}%
\tabcolsep=2pt%
\input{src/tables/allclasses_bigbig_4.tex}%
\end{scriptsize}%
\end{center}%
\caption{\label{tab:all44} Comparison with state of the art: $\numex \geq 5000, \numfeat \geq 250$, depth 4}%
\end{table}%
\begin{table}[htbp]%
\begin{center}%
\begin{scriptsize}%
\tabcolsep=2pt%
\input{src/tables/allclasses_smallsmall_5.tex}%
\end{scriptsize}%
\end{center}%
\caption{\label{tab:all51} Comparison with state of the art: $\numex<5000, \numfeat<250$, depth 5}%
\end{table}%
\begin{table}[htbp]%
\begin{center}%
\begin{scriptsize}%
\tabcolsep=2pt%
\input{src/tables/allclasses_smallbig_5.tex}%
\end{scriptsize}%
\end{center}%
\caption{\label{tab:all52} Comparison with state of the art: $\numex<5000, \numfeat \geq 250$, depth 5}%
\end{table}%
\begin{table}[htbp]%
\begin{center}%
\begin{scriptsize}%
\tabcolsep=2pt%
\input{src/tables/allclasses_bigsmall_5.tex}%
\end{scriptsize}%
\end{center}%
\caption{\label{tab:all53} Comparison with state of the art: $\numex \geq 5000, \numfeat < 250$, depth 5}%
\end{table}%
\begin{table}[htbp]%
\begin{center}%
\begin{scriptsize}%
\tabcolsep=2pt%
\input{src/tables/allclasses_bigbig_5.tex}%
\end{scriptsize}%
\end{center}%
\caption{\label{tab:all54} Comparison with state of the art: $\numex \geq 5000, \numfeat \geq 250$, depth 5}%
\end{table}%
\begin{table}[htbp]%
\begin{center}%
\begin{scriptsize}%
\tabcolsep=2pt%
\input{src/tables/allclasses_smallsmall_7.tex}%
\end{scriptsize}%
\end{center}%
\caption{\label{tab:all71} Comparison with state of the art: $\numex<5000, \numfeat<250$, depth 7}%
\end{table}%
\begin{table}[htbp]%
\begin{center}%
\begin{scriptsize}%
\tabcolsep=2pt%
\input{src/tables/allclasses_smallbig_7.tex}%
\end{scriptsize}%
\end{center}%
\caption{\label{tab:all72} Comparison with state of the art: $\numex<5000, \numfeat \geq 250$, depth 7}%
\end{table}%
\begin{table}[htbp]%
\begin{center}%
\begin{scriptsize}%
\tabcolsep=2pt%
\input{src/tables/allclasses_bigsmall_7.tex}%
\end{scriptsize}%
\end{center}%
\caption{\label{tab:all73} Comparison with state of the art: $\numex \geq 5000, \numfeat < 250$, depth 7}%
\end{table}%
\begin{table}[htbp]%
\begin{center}%
\begin{scriptsize}%
\tabcolsep=2pt%
\input{src/tables/allclasses_bigbig_7.tex}%
\end{scriptsize}%
\end{center}%
\caption{\label{tab:all74} Comparison with state of the art: $\numex \geq 5000, \numfeat \geq 250$, depth 7}%
\end{table}%
\begin{table}[htbp]%
\begin{center}%
\begin{scriptsize}%
\tabcolsep=2pt%
\input{src/tables/allclasses_smallsmall_10.tex}%
\end{scriptsize}%
\end{center}%
\caption{\label{tab:all101} Comparison with state of the art: $\numex<5000, \numfeat<250$, depth 10}%
\end{table}%
\begin{table}[htbp]%
\begin{center}%
\begin{scriptsize}%
\tabcolsep=2pt%
\input{src/tables/allclasses_smallbig_10.tex}%
\end{scriptsize}%
\end{center}%
\caption{\label{tab:all102} Comparison with state of the art: $\numex<5000, \numfeat \geq 250$, depth 10}%
\end{table}%
\begin{table}[htbp]%
\begin{center}%
\begin{scriptsize}%
\tabcolsep=2pt%
\input{src/tables/allclasses_bigsmall_10.tex}%
\end{scriptsize}%
\end{center}%
\caption{\label{tab:all103} Comparison with state of the art: $\numex \geq 5000, \numfeat < 250$, depth 10}%
\end{table}%
\begin{table}[htbp]%
\begin{center}%
\begin{scriptsize}%
\tabcolsep=2pt%
\input{src/tables/allclasses_bigbig_10.tex}%
\end{scriptsize}%
\end{center}%
\caption{\label{tab:all104} Comparison with state of the art: $\numex \geq 5000, \numfeat \geq 250$, depth 10}%
\end{table}%
\end{document}
% We report in Table~\ref{tab:summaryacc} data averaged over the 47 data sets described above, for
%
%
......@@ -1352,36 +1680,6 @@ instead of absolute values, we provide the average relative difference in error
\end{table}
\subsection{Computing deep classifiers}
Next, we shift our focus to how fast can we obtain accurate trees and how fast can we improve the accuracy over basic solutions found by heuristics.
Therefore, in order to have a baseline, we use a well known heuristic: \cart (we ran its implementation in scikit-learn).
Here we report the average error after a given period of time (3 seconds, 10 seconds, 1 minute or 5 minutes) in Table~\ref{tab:summaryspeed}.
\medskip
We can see that the first solution found by our algorithm has comparable accuracy (actually, often slightly better) to the one found by \cart. The implementation of \cart in scikit-learn does not seem to be very efficient computationally. However, this is not so relevant as it is clear that one greedy run of the heuristic can be implemented to be as fast as the first dive of \budalg. The point of this experiment is threefold: first, it shows that the first solution is very similar to that found by \cart (here we believe that the only difference is to use the feature yielding the minimum error instead of the feature with minimum Gini impurity for the deepest feature tests); second, this tree
is found extremely quickly, and there is no scaling issue with respect to the depth of the tree or with respect to the size of the data set; third, even for large data sets and deep trees, the accuracy of the initial classifier can be significantly improved given a reasonable computation time.
%Moreover, although we would need larger data sets to be confident about that, it seems that our algorithm is faster than \cart to find this first decision tree. %One can conjecture that \cart uses more sophisticated heuristic choices to explain these two observations.
%Then, in most cases, it is possible to improve the first solution significantly within a few seconds. Notice that for larger depth, improving the initial solution is harder and the 3s time limit is comparatively tighter than for smaller trees, so the gain of \budalg over \cart is more sensible for small trees.
\begin{table}[htbp]
\begin{center}
\begin{footnotesize}
\tabcolsep=1.8pt
\input{src/tables/summaryspeed.tex}
\end{footnotesize}
\end{center}
\caption{\label{tab:summaryspeed} Comparison with state the of the art: computing accurate trees}
\end{table}
Figure~\ref{fig:cactus} reports the evolution of accuracy over time, giving a good view of the difference between \murtree and \budalg during search. The accuracy of the tree returned by \cart is given for reference.
We can see in those graphs that \murtree finds an initial tree extremely quickly, although with very low accuracy. This is because \murtree show progress even when the tree is not complete, i.e., the first solution is always a single node with the most promising feature. We can see in Table~\ref{tab:summaryspeed} that this is indeed always the same feature, irrespective of the maximum depth.
......
\begin{tabular}{lrrrrrrrrrr}
\begin{tabular}{lrrrrrrrrrrrr}
\toprule
\multirow{2}{*}{}& \multicolumn{2}{c}{\budalg} & \multicolumn{2}{c}{\murtree} & \multicolumn{2}{c}{\dleight} & \multicolumn{2}{c}{\cp} & \multicolumn{2}{c}{\cart}\\
\cmidrule(rr){2-3}\cmidrule(rr){4-5}\cmidrule(rr){6-7}\cmidrule(rr){8-9}\cmidrule(rr){10-11}
& \multicolumn{1}{c}{error} & \multicolumn{1}{c}{cpu} & \multicolumn{1}{c}{error} & \multicolumn{1}{c}{cpu} & \multicolumn{1}{c}{error} & \multicolumn{1}{c}{cpu} & \multicolumn{1}{c}{error} & \multicolumn{1}{c}{cpu} & \multicolumn{1}{c}{error} & \multicolumn{1}{c}{cpu} \\
\multirow{2}{*}{}& \multicolumn{2}{c}{\budalg} & \multicolumn{2}{c}{\murtree} & \multicolumn{2}{c}{\dleight} & \multicolumn{2}{c}{\cp} & \multicolumn{2}{c}{binoct} & \multicolumn{2}{c}{\cart}\\
\cmidrule(rr){2-3}\cmidrule(rr){4-5}\cmidrule(rr){6-7}\cmidrule(rr){8-9}\cmidrule(rr){10-11}\cmidrule(rr){12-13}
& \multicolumn{1}{c}{error} & \multicolumn{1}{c}{cpu} & \multicolumn{1}{c}{error} & \multicolumn{1}{c}{cpu} & \multicolumn{1}{c}{error} & \multicolumn{1}{c}{cpu} & \multicolumn{1}{c}{error} & \multicolumn{1}{c}{cpu} & \multicolumn{1}{c}{error} & \multicolumn{1}{c}{cpu} & \multicolumn{1}{c}{error} & \multicolumn{1}{c}{cpu} \\
\midrule
\texttt{monk3-bin} & 0 & 0.00$^*$ & 0 & 0.00$^*$ & 0 & 0.00$^*$ & 0 & 0.99$^*$ & 0 & 0.00\\
\texttt{monk1-bin} & 0 & 0.00$^*$ & 0 & 0.00$^*$ & 0 & 0.00$^*$ & 0 & 0.64$^*$ & 0 & 0.00\\
\texttt{hepatitis} & 0 & 0.00$^*$ & 0 & 0.01$^*$ & 0 & 0.00$^*$ & 0 & 1.3$^*$ & 0 & 0.00\\
\texttt{lymph} & 0 & 0.00$^*$ & 0 & 0.02$^*$ & 0 & 0.00$^*$ & 0 & 1.2$^*$ & 0 & 0.00\\
\texttt{iris-bin} & 1 & 0.00$^*$ & 1 & 0.00$^*$ & 1 & 0.01$^*$ & 1 & 21$^*$ & 1 & 0.00\\
\texttt{monk2-bin} & 0 & 0.00$^*$ & 0 & 0.01$^*$ & 0 & 0.00$^*$ & 0 & 0.89$^*$ & 0 & 0.00\\
\texttt{wine-bin} & 0 & 0.00$^*$ & 0 & 0.01$^*$ & 0 & 0.00$^*$ & 0 & 0.86$^*$ & 0 & 0.01\\
\texttt{wine3} & 16 & 272 & 28 & 1555 & - & - & 19 & $\mathsmaller{\geq}1$h & 19 & 0.01\\
\texttt{wine2} & 24 & 399 & 43 & 625 & - & - & 29 & $\mathsmaller{\geq}1$h & 29 & 0.02\\
\texttt{wine1} & 22 & 545 & 33 & 1439 & - & - & 27 & $\mathsmaller{\geq}1$h & 25 & 0.01\\
\texttt{audiology} & 0 & 0.00$^*$ & 0 & 0.02$^*$ & 0 & 0.00$^*$ & 0 & 1.4$^*$ & 0 & 0.00\\
\texttt{heart-cleveland} & 0 & 0.00$^*$ & 0 & 0.04$^*$ & 0 & 0.08$^*$ & 0 & 1.2$^*$ & 0 & 0.00\\
\texttt{primary-tumor} & 15 & 0.00$^*$ & 15 & 2088 & - & - & 82 & $\mathsmaller{\geq}1$h & 20 & 0.00\\
\texttt{ionosphere} & 0 & 0.00$^*$ & 0 & 2.1$^*$ & 0 & 110$^*$ & 0 & 8.1$^*$ & 0 & 0.01\\
\texttt{Ionosphere-bin} & 0 & 0.00$^*$ & 0 & 0.06$^*$ & 0 & 0.09$^*$ & 0 & 23$^*$ & 1 & 0.00\\
\texttt{vote} & 0 & 0.00$^*$ & 0 & 0.01$^*$ & 0 & 0.00$^*$ & 0 & 2.3$^*$ & 0 & 0.00\\
\texttt{forest-fires} & 113 & 942 & 174 & 3052 & - & - & 247 & $\mathsmaller{\geq}1$h & 146 & 0.02\\
\texttt{balance-scale-bin} & 0 & 19$^*$ & 0 & 3.2$^*$ & 0 & 1.5$^*$ & 0 & 16$^*$ & 6 & 0.00\\
\texttt{soybean} & 2 & 0.00$^*$ & 7 & 1766 & - & - & 92 & $\mathsmaller{\geq}1$h & 2 & 0.00\\
\texttt{australian-credit} & 0 & 0.04$^*$ & 0 & 69$^*$ & - & - & 0 & 464$^*$ & 12 & 0.01\\
\texttt{breast-wisconsin} & 0 & 0.00$^*$ & 0 & 0.08$^*$ & 0 & 3.4$^*$ & 0 & 7.8$^*$ & 0 & 0.00\\
\texttt{breast-cancer} & 0 & 0.00$^*$ & 0 & 0.02$^*$ & 0 & 0.00$^*$ & 0 & 2.4$^*$ & 0 & 0.00\\
\texttt{diabetes} & 0 & 0.67$^*$ & 0 & 2157$^*$ & - & - & 0 & 463$^*$ & 35 & 0.01\\
\texttt{IndiansDiabetes-bin} & 8 & 4.7$^*$ & 43 & 433 & - & - & 268 & $\mathsmaller{\geq}1$h & 63 & 0.00\\
\texttt{anneal} & 34 & 23$^*$ & 121 & 1210 & - & - & 187 & $\mathsmaller{\geq}1$h & 59 & 0.00\\
\texttt{vehicle} & 0 & 0.00$^*$ & 0 & 0.34$^*$ & 0 & 0.37$^*$ & 0 & 4.2$^*$ & 0 & 0.01\\
\texttt{titanic} & 35 & 3059 & 215 & 2804 & - & - & 342 & $\mathsmaller{\geq}1$h & 78 & 0.01\\
\texttt{tic-tac-toe-bin} & 0 & 0.00$^*$ & 0 & 0.01$^*$ & 0 & 0.00$^*$ & 0 & 0.97$^*$ & 0 & 0.00\\
\texttt{tic-tac-toe} & 0 & 0.00$^*$ & 0 & 0.04$^*$ & 0 & 0.03$^*$ & 0 & 0.81$^*$ & 6 & 0.00\\
\texttt{german-credit} & 0 & 69$^*$ & 0 & 388$^*$ & - & - & 0 & 28$^*$ & 66 & 0.01\\
\texttt{biodeg-bin} & 1 & 1169$^*$ & 159 & 1049 & - & - & 356 & $\mathsmaller{\geq}1$h & 27 & 0.02\\
\texttt{messidor-bin} & 66 & 604 & 223 & 1204 & - & - & 540 & $\mathsmaller{\geq}1$h & 211 & 0.03\\
\texttt{banknote-bin} & 2 & 0.00$^*$ & 2 & 463$^*$ & - & - & 610 & $\mathsmaller{\geq}1$h & 2 & 0.00\\
\texttt{yeast} & 28 & 1008 & 237 & 3555 & - & - & 463 & $\mathsmaller{\geq}1$h & 185 & 0.01\\
\texttt{winequality-red-bin} & 2 & 0.00$^*$ & 5 & 1076 & - & - & 10 & $\mathsmaller{\geq}1$h & 2 & 0.00\\
\texttt{car\_evaluation-bin} & 80 & 0.00$^*$ & 80 & 31$^*$ & 80 & 9.2$^*$ & 80 & $\mathsmaller{\geq}1$h & 80 & 0.00\\
\texttt{car} & 0 & 0.26$^*$ & 0 & 0.03$^*$ & 0 & 0.03$^*$ & 0 & 3.3$^*$ & 11 & 0.00\\
\texttt{segment} & 0 & 0.00$^*$ & 0 & 0.36$^*$ & 0 & 0.08$^*$ & 0 & 1.9$^*$ & 0 & 0.01\\
\texttt{seismic\_bumps-bin} & 38 & 2591 & 128 & 143 & - & - & 170 & $\mathsmaller{\geq}1$h & 101 & 0.01\\
\texttt{splice-1} & 5 & 1160 & 1319 & 189 & - & - & 1535 & $\mathsmaller{\geq}1$h & 12 & 0.05\\
\texttt{kr-vs-kp} & 0 & 1897$^*$ & 295 & 1333 & - & - & 784 & $\mathsmaller{\geq}1$h & 12 & 0.01\\
\texttt{chess-bin} & 0 & 0.00$^*$ & 0 & 0.06$^*$ & 0 & 0.01$^*$ & 0 & 0.66$^*$ & 0 & 0.00\\
\texttt{hypothyroid} & 17 & 0.96$^*$ & 115 & 3040 & - & - & 277 & $\mathsmaller{\geq}1$h & 31 & 0.01\\
\texttt{Statlog\_satellite-bin} & 3 & 219 & 116 & 2837 & - & - & - & - & 15 & 0.13\\
\texttt{bank\_conv-bin} & 169 & 2794 & 371 & 2954 & - & - & 521 & $\mathsmaller{\geq}1$h & 207 & 0.10\\
\texttt{spambase-bin} & 262 & 546 & 1047 & 3379 & - & - & 1813 & $\mathsmaller{\geq}1$h & 332 & 0.09\\
\texttt{compas\_discretized} & 1828 & 0.73$^*$ & 1828 & 2248 & - & - & 2809 & $\mathsmaller{\geq}1$h & 1871 & 0.01\\
\texttt{pendigits} & 0 & 0.00$^*$ & 0 & 17$^*$ & 0 & 1247$^*$ & 0 & 5.3$^*$ & 0 & 0.07\\
\texttt{mushroom} & 0 & 0.00$^*$ & 0 & 0.13$^*$ & 0 & 1.1$^*$ & 0 & 1.2$^*$ & 0 & 0.04\\
\texttt{surgical-deepnet} & 965 & 2865 & 2638 & 3583 & - & - & 3690 & $\mathsmaller{\geq}1$h & 1089 & 14\\
\texttt{HTRU\_2-bin} & 219 & 550 & 399 & 2695 & - & - & 1639 & $\mathsmaller{\geq}1$h & 293 & 0.08\\
\texttt{magic04-bin} & 1635 & 2746 & 3456 & 2402 & - & - & 6688 & $\mathsmaller{\geq}1$h & 2145 & 0.13\\
\texttt{letter} & 0 & 79$^*$ & 636 & 3519 & 725 & $\mathsmaller{\geq}1$h & 813 & $\mathsmaller{\geq}1$h & 21 & 0.31\\
\texttt{letter\_recognition-bin} & 11 & 712 & 649 & 2356 & - & - & - & - & 28 & 0.45\\
\texttt{taiwan\_binarised} & 4217 & 1001 & 5908 & 2274 & - & - & 6636 & $\mathsmaller{\geq}1$h & 4710 & 0.54\\
\texttt{default\_credit-bin} & 4547 & 2019 & 5838 & 1485 & - & - & - & - & 4762 & 1.3\\
\texttt{adult\_discretized} & 3841 & 2632 & 5549 & 2236 & - & - & 7511 & $\mathsmaller{\geq}1$h & 4148 & 0.12\\
\texttt{Statlog\_shuttle-bin} & 0 & 0.02$^*$ & 0 & 49$^*$ & 0 & 99$^*$ & 0 & 16$^*$ & 0 & 3.6\\
\texttt{bank} & 3242 & 800 & 5287 & 0.00 & 4826 & $\mathsmaller{\geq}1$h & 5289 & $\mathsmaller{\geq}1$h & 3327 & 102\\
\texttt{mnist\_0} & 383 & 413 & 2556 & 579 & 3314 & $\mathsmaller{\geq}1$h & 5923 & $\mathsmaller{\geq}1$h & 477 & 8.5\\
\texttt{mnist\_1} & 331 & 360 & 3450 & 3550 & 4544 & $\mathsmaller{\geq}1$h & 6742 & $\mathsmaller{\geq}1$h & 439 & 7.8\\
\texttt{mnist\_2} & 1522 & 2520 & 3927 & 1659 & - & - & 5958 & $\mathsmaller{\geq}1$h & 1959 & 8.7\\
\texttt{mnist\_3} & 1079 & 376 & 4352 & 105 & 5171 & $\mathsmaller{\geq}1$h & 6131 & $\mathsmaller{\geq}1$h & 1169 & 6.7\\
\texttt{mnist\_4} & 801 & 398 & 4709 & 1076 & 5580 & $\mathsmaller{\geq}1$h & 5842 & $\mathsmaller{\geq}1$h & 1010 & 10\\
\texttt{mnist\_5} & 1973 & 491 & 3539 & 87 & 4379 & $\mathsmaller{\geq}1$h & 5421 & $\mathsmaller{\geq}1$h & 2266 & 6.9\\
\texttt{mnist\_7} & 1082 & 1277 & 3483 & 265 & - & - & 6265 & $\mathsmaller{\geq}1$h & 1263 & 11\\
\texttt{mnist\_6} & 965 & 1742 & 2684 & 512 & 2699 & $\mathsmaller{\geq}1$h & 5918 & $\mathsmaller{\geq}1$h & 1211 & 7.4\\
\texttt{mnist\_8} & 696 & 2141 & 3583 & 207 & - & - & 5851 & $\mathsmaller{\geq}1$h & 916 & 7.9\\
\texttt{mnist\_9} & 1594 & 2124 & 4590 & 306 & - & - & 5949 & $\mathsmaller{\geq}1$h & 1722 & 7.1\\
\texttt{hand\_posture-bin} & 334 & 39 & 14635 & 670 & - & - & - & - & 530 & 88\\
\texttt{weather-aus} & 1601 & 2591 & 5868 & 2441 & - & - & 1761 & $\mathsmaller{\geq}1$h & 1642 & 32\\
\texttt{monk3-bin} & 0 & 0.00$^*$ & 0 & 0.00$^*$ & 0 & 0.00$^*$ & 0 & 0.99$^*$ & - & - & 0 & 0.00\\
\texttt{monk1-bin} & 0 & 0.00$^*$ & 0 & 0.00$^*$ & 0 & 0.00$^*$ & 0 & 0.64$^*$ & - & - & 0 & 0.00\\
\texttt{hepatitis} & 0 & 0.00$^*$ & 0 & 0.01$^*$ & 0 & 0.00$^*$ & 0 & 1.3$^*$ & 19 & 2032 & 0 & 0.00\\
\texttt{lymph} & 0 & 0.00$^*$ & 0 & 0.02$^*$ & 0 & 0.00$^*$ & 0 & 1.2$^*$ & 30 & 576 & 0 & 0.00\\
\texttt{iris-bin} & 1 & 0.00$^*$ & 1 & 0.00$^*$ & 1 & 0.01$^*$ & 1 & 21$^*$ & - & - & 1 & 0.00\\
\texttt{monk2-bin} & 0 & 0.00$^*$ & 0 & 0.01$^*$ & 0 & 0.00$^*$ & 0 & 0.89$^*$ & - & - & 0 & 0.00\\
\texttt{wine1} & 22 & 545 & 33 & 1439 & - & - & 27 & $\mathsmaller{\geq}1$h & - & - & 25 & 0.01\\
\texttt{wine2} & 24 & 399 & 43 & 625 & - & - & 29 & $\mathsmaller{\geq}1$h & - & - & 29 & 0.02\\
\texttt{wine3} & 16 & 272 & 28 & 1555 & - & - & 19 & $\mathsmaller{\geq}1$h & - & - & 19 & 0.01\\
\texttt{wine-bin} & 0 & 0.00$^*$ & 0 & 0.01$^*$ & 0 & 0.00$^*$ & 0 & 0.86$^*$ & - & - & 0 & 0.01\\
\texttt{audiology} & 0 & 0.00$^*$ & 0 & 0.02$^*$ & 0 & 0.00$^*$ & 0 & 1.4$^*$ & 25 & 17 & 0 & 0.00\\
\texttt{heart-cleveland} & 0 & 0.00$^*$ & 0 & 0.04$^*$ & 0 & 0.08$^*$ & 0 & 1.2$^*$ & 127 & 7.6 & 0 & 0.00\\
\texttt{primary-tumor} & 15 & 0.00$^*$ & 15 & 2088 & - & - & 82 & $\mathsmaller{\geq}1$h & 31 & 3329 & 20 & 0.00\\
\texttt{ionosphere} & 0 & 0.00$^*$ & 0 & 2.1$^*$ & 0 & 110$^*$ & 0 & 8.1$^*$ & 225 & 0.00 & 0 & 0.01\\
\texttt{Ionosphere-bin} & 0 & 0.00$^*$ & 0 & 0.06$^*$ & 0 & 0.09$^*$ & 0 & 23$^*$ & - & - & 1 & 0.00\\
\texttt{vote} & 0 & 0.00$^*$ & 0 & 0.01$^*$ & 0 & 0.00$^*$ & 0 & 2.3$^*$ & 132 & 9.9 & 0 & 0.00\\
\texttt{forest-fires} & 113 & 942 & 174 & 3052 & - & - & 247 & $\mathsmaller{\geq}1$h & - & - & 146 & 0.02\\
\texttt{balance-scale-bin} & 0 & 19$^*$ & 0 & 3.2$^*$ & 0 & 1.5$^*$ & 0 & 16$^*$ & - & - & 6 & 0.00\\
\texttt{soybean} & 2 & 0.00$^*$ & 7 & 1766 & - & - & 92 & $\mathsmaller{\geq}1$h & 84 & 11 & 2 & 0.00\\
\texttt{australian-credit} & 0 & 0.04$^*$ & 0 & 69$^*$ & - & - & 0 & 464$^*$ & 357 & 0.00 & 12 & 0.01\\
\texttt{breast-cancer} & 0 & 0.00$^*$ & 0 & 0.02$^*$ & 0 & 0.00$^*$ & 0 & 2.4$^*$ & 239 & 0.00 & 0 & 0.00\\
\texttt{breast-wisconsin} & 0 & 0.00$^*$ & 0 & 0.08$^*$ & 0 & 3.4$^*$ & 0 & 7.8$^*$ & 444 & 0.00 & 0 & 0.00\\
\texttt{diabetes} & 0 & 0.67$^*$ & 0 & 2157$^*$ & - & - & 0 & 463$^*$ & 500 & 0.00 & 35 & 0.01\\
\texttt{IndiansDiabetes-bin} & 8 & 4.7$^*$ & 43 & 433 & - & - & 268 & $\mathsmaller{\geq}1$h & - & - & 63 & 0.00\\
\texttt{anneal} & 34 & 23$^*$ & 121 & 1210 & - & - & 187 & $\mathsmaller{\geq}1$h & 625 & 0.00 & 59 & 0.00\\
\texttt{vehicle} & 0 & 0.00$^*$ & 0 & 0.34$^*$ & 0 & 0.37$^*$ & 0 & 4.2$^*$ & - & - & 0 & 0.01\\
\texttt{titanic} & 35 & 3059 & 215 & 2804 & - & - & 342 & $\mathsmaller{\geq}1$h & - & - & 78 & 0.01\\
\texttt{tic-tac-toe} & 0 & 0.00$^*$ & 0 & 0.04$^*$ & 0 & 0.03$^*$ & 0 & 0.81$^*$ & 332 & 194 & 6 & 0.00\\
\texttt{tic-tac-toe-bin} & 0 & 0.00$^*$ & 0 & 0.01$^*$ & 0 & 0.00$^*$ & 0 & 0.97$^*$ & - & - & 0 & 0.00\\
\texttt{german-credit} & 0 & 69$^*$ & 0 & 388$^*$ & - & - & 0 & 28$^*$ & 700 & 0.00 & 66 & 0.01\\
\texttt{biodeg-bin} & 1 & 1169$^*$ & 159 & 1049 & - & - & 356 & $\mathsmaller{\geq}1$h & - & - & 27 & 0.02\\
\texttt{messidor-bin} & 66 & 604 & 223 & 1204 & - & - & 540 & $\mathsmaller{\geq}1$h & - & - & 211 & 0.03\\
\texttt{banknote-bin} & 2 & 0.00$^*$ & 2 & 463$^*$ & 2 & 738$^*$ & 610 & $\mathsmaller{\geq}1$h & - & - & 2 & 0.00\\
\texttt{yeast} & 28 & 1008 & 237 & 3555 & - & - & 463 & $\mathsmaller{\geq}1$h & 463 & 0.00 & 185 & 0.01\\
\texttt{winequality-red-bin} & 2 & 0.00$^*$ & 5 & 1076 & - & - & 10 & $\mathsmaller{\geq}1$h & - & - & 2 & 0.00\\
\texttt{car} & 0 & 0.26$^*$ & 0 & 0.03$^*$ & 0 & 0.03$^*$ & 0 & 3.3$^*$ & 518 & 0.00 & 11 & 0.00\\
\texttt{car\_evaluation-bin} & 80 & 0.00$^*$ & 80 & 31$^*$ & 80 & 9.2$^*$ & 80 & $\mathsmaller{\geq}1$h & - & - & 80 & 0.00\\
\texttt{segment} & 0 & 0.00$^*$ & 0 & 0.36$^*$ & 0 & 0.08$^*$ & 0 & 1.9$^*$ & - & - & 0 & 0.01\\
\texttt{seismic\_bumps-bin} & 38 & 2591 & 128 & 143 & - & - & 170 & $\mathsmaller{\geq}1$h & - & - & 101 & 0.01\\
\texttt{splice-1} & 5 & 1160 & 1319 & 189 & - & - & 1535 & $\mathsmaller{\geq}1$h & - & - & 12 & 0.05\\
\texttt{kr-vs-kp} & 0 & 1897$^*$ & 295 & 1333 & - & - & 784 & $\mathsmaller{\geq}1$h & - & - & 12 & 0.01\\
\texttt{chess-bin} & 0 & 0.00$^*$ & 0 & 0.06$^*$ & 0 & 0.01$^*$ & 0 & 0.66$^*$ & - & - & 0 & 0.00\\
\texttt{hypothyroid} & 17 & 0.96$^*$ & 115 & 3040 & - & - & 277 & $\mathsmaller{\geq}1$h & - & - & 31 & 0.01\\
\texttt{Statlog\_satellite-bin} & 3 & 219 & 116 & 2837 & - & - & 1072 & $\mathsmaller{\geq}1$h & - & - & 15 & 0.13\\
\texttt{bank\_conv-bin} & 169 & 2794 & 371 & 2954 & - & - & 521 & $\mathsmaller{\geq}1$h & - & - & 207 & 0.10\\
\texttt{spambase-bin} & 262 & 546 & 1047 & 3379 & - & - & 1813 & $\mathsmaller{\geq}1$h & - & - & 332 & 0.09\\
\texttt{compas\_discretized} & 1828 & 0.73$^*$ & 1828 & 2248 & - & - & 2809 & $\mathsmaller{\geq}1$h & 2809 & 0.00 & 1871 & 0.01\\
\texttt{pendigits} & 0 & 0.00$^*$ & 0 & 17$^*$ & 0 & 1247$^*$ & 0 & 5.3$^*$ & - & - & 0 & 0.07\\
\texttt{mushroom} & 0 & 0.00$^