标签：

CIFAR-10 Competition Winners: Interviews with Dr. Ben Graham, Phil Culliton, & Zygmunt Zaj?c Dr. Ben Graham

Dr. Ben Graham is an Assistant Professor in Statistics and Complexity at the University of Warwick. With a categorization accuracy of 0.95530 he ranked first place.

Congratulations on winning the CIFAR-10 competition! How do you feel about your victory?

Thank you! I am very pleased to have won, and quite frankly pretty amazed at just how competitive the competition was.

When I first saw the competition, I did not think the test error would go below about 8%. I assumed 32x32 pixels just wasn‘t enough information to identify objects very reliably. As it turned out, everyone in the top 10 got below 7%, which is roughly on a par with human performance.

Can you tell us about the setup of the network? How many layers?

It is a deep convolutional network trained using SparseConvNet with architecture:

input=(3x126x126) - 320C2 - 320C2 - MP2 - 640C2 - 10% dropout - 640C2 - 10% dropout - MP2 - 960C2 - 20% dropout - 960C2 - 20% dropout - MP2 - 1280C2 - 30% dropout - 1280C2 - 30% dropout - MP2 - 1600C2 - 40% dropout - 1600C2 - 40% dropout - MP2 - 1920C2 - 50% dropout - 1920C1 - 50% dropout - 10C1 - Softmax output

It was trained taking advantage of:

spatial-sparsity in the 126x126 input layer,

batchwise dropout,

(very) leaky rectified linear units, and

affine spatial and color-space training data augmentation.

The same architecture produces a test error of 20.68% for CIFAR-100.

These cats evaded the DeepCNet solution by looking a lot like a fighter jet and a car.

Can you tell us a little about the hardware used to train the nets? How long did it take to train? What was the development cycle like?

The network took about 90 hours to train on an NVIDIA GeForce GTX780 graphics card. I had already written a convolutional neural network for spatially-sparse inputs to learn to recognise online Chinese handwriting.

Over the course of the competition I upgraded the program to allowdropout to be applied batchwise, and cleaned up some kernels that were accessing memory inefficiently. That made it feasible to train pretty large networks.

Which papers/approaches authored by other scientists did contribute the most to your top score?

The network architecture is the result of borrowing ideas from a number of recent papers

Reading each of those papers was jaw-dropping as the ideas would not have occurred to me.

These images were all correctly classified. To the net they look the most like their respective classes. From DeepCNet‘s extremes.

Where do you see convnets in the future? Anything in particular that you are excited about?

I am very interested in the idea of spatially-sparse 3d convolutional networks. For example, given a length of string, you might be able to pull both ends to produce a straight line. Alternatively, the string might contain a knot which you cannot get rid of no matter how hard you pull. That is an idea that is obvious for humans, but hard to solve by computer as there are so many different kinds on knots.

Hopefully 3d convolutional networks can develop some of the physical intuition humans take for granted.

Besides convnets, I am very interested in machine learning techniques for time-series data, such as recurrent neural networks.

Thank you very much for sharing your code on the forums. What is your opinion on sharing code?

My pleasure; it was nice to see a couple of the other teams in the top 10 ("Jiki" and "Phil & Triskelion & Kazanova") use the code. Another Kaggler, Nagadomi, also made his code available during the competition. It was fascinating to see him implement some of the ideas to come out of the ILSVRC2014 competition such as "C3-C3-MP2" layers and Inception layers.

Do you think your convnet could be improved even more on this task, or do you feel it is close to its limit?

After the competition, I re-ran my top network on the 10,000 images from the original CIFAR-10 test set, resulting in 446 errors.

Here is a confusion matrix for showing where the 446 errors come from:

airplane automobile bird cat deer dog frog horse ship truck airplane 0 3 10 2 2 0 2 0 16 3 automobile 1 0 1 0 0 0 0 0 3 12 bird 8 1 0 14 19 8 9 5 2 0 cat 4 1 8 0 9 57 20 2 5 2 deer 3 1 12 7 0 5 4 8 0 0 dog 4 1 7 39 10 0 1 7 1 1 frog 4 0 7 7 3 1 0 1 0 1 horse 6 0 3 4 7 8 0 0 0 0 ship 2 3 2 0 0 1 0 0 0 3 truck 3 20 0 2 0 0 1 0 7 0