Ensambles of best models

To move accuracy few percents up, ensambles of models can be used. Two similar options are presented:

• ensamble of three models with averaged outputs,
• ensamble of nine models with voting strategy. For the latter, 80% accuracy is achieved.

After re-creating test dataset, models selected to cooperate can be loaded from disk:

            String dir = EnsambleAvg.class.getClassLoader().getResource("models").getFile();
            MultiLayerNetwork n1 = ModelSerializer.restoreMultiLayerNetwork(new File(dir, "0.7596314907872697"));
            MultiLayerNetwork n2 = ModelSerializer.restoreMultiLayerNetwork(new File(dir, "0.7763819095477387"));
            MultiLayerNetwork n3 = ModelSerializer.restoreMultiLayerNetwork(new File(dir, "0.7646566164154104"));

or loaded all at once:

List<MultiLayerNetwork> nets = Arrays
    .stream(new File(EnsambleVote.class.getClassLoader().getResource("models").getFile()).listFiles())
    .map(f -> {
        MultiLayerNetwork n = null;
        try {
            n = ModelSerializer.restoreMultiLayerNetwork(f);
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
        return n;
    }).collect(Collectors.toList());

By using function that averages their outputs, ensamble can be established:

static double[] predict(int numLabels, DataSet ds, MultiLayerNetwork... nets) {

    List<double[]> outputs = Arrays.stream(nets)
        .map(net -> net.output(ds.getFeatureMatrix(), false).data().asDouble())
        .collect(Collectors.toList());

    double[] result = new double[numLabels];
    Arrays.fill(result, 0d);

    outputs.forEach(d -> {
        for (int i = 0; i < numLabels; ++i) {
            result[i] += d[i];
        }
    });

    for (int i = 0; i < numLabels; ++i) {
        result[i] /= nets.length;
    }

    return result;
}

Misclassified images can be shown together with their predicted labels:

test.filter(
        ds -> label(predict(numLabels, ds, nets.toArray(new MultiLayerNetwork[0]))) != label(ds
            .getLabels()))
    .foreach(
            ds -> log.info("predicted {}, label {}",
                    asString(predict(numLabels, ds, nets.toArray(new MultiLayerNetwork[0]))),
                    label(ds.getLabels()))
    );

Evaluation looks almost the same as for single model:

JavaPairRDD<Object, Object> predictionsAndLabels = test
    .mapToPair(
    ds -> new Tuple2<>(label(predict(numLabels, ds, n1, n2, n3)), label(ds.getLabels()))
    );

MulticlassMetrics metrics = new MulticlassMetrics(predictionsAndLabels.rdd());
double accuracy = 1.0 * predictionsAndLabels.filter(x -> x._1.equals(x._2)).count() / test.count();
log.info("accuracy {} ", accuracy);
predictionsAndLabels.take(10).forEach(t -> log.info("predicted {}, label {}", t._1, t._2));
log.info("confusionMatrix {}", metrics.confusionMatrix());

Function for voting:

static double[] predict(int numLabels, DataSet ds, MultiLayerNetwork... nets) {

    List<double[]> outputs = Arrays.stream(nets)
        .map(net -> net.output(ds.getFeatureMatrix(), false).data().asDouble())
        .collect(Collectors.toList());

    double[] result = new double[numLabels];
    Arrays.fill(result, Double.MIN_VALUE);

    outputs.forEach(d -> {
        double max = Arrays.stream(d).max().getAsDouble();
        for (int i = 0; i < numLabels; ++i) {
            if (d[i] < max) {
                continue;// use only max value
            }
            if (result[i] == Double.MIN_VALUE) {
                result[i] = 1d;
            } else {
                result[i] += 1d;
            }
        }
    });
    return result;
}