COMP 551 - Project 4 - Reproducible Machine Learning
Teamname: The Deathstars
Seara Chen Benjamin Paul-Dubois-Taine
siyu.chen2@mail.mcgill.ca benjamin.paul-dubois-taine@mail.mcgill.ca
260670714 260673170
AbstractThis is a reproduction of the paper compressing en-
coding of words for use in character-level convolutional networks
for text classification submitted to the ICLR2018 conference.
We reproduce its proposed encoding scheme and convolutional
network. We try to replicate their computation for three of the
analyzed datasets. We were able to achieve similar results in both
test error as well as running time for all three datasets. While
most of the main ideas in the paper are explained well and could
therefore be replicated, some implementation details are missing.
I. INTRODUCTION
This project is part of the ICLR2018 Reproducibility Chal-
lenge. The goal of the challenge is to improve the quality of
submissions by highlighting the importance of reproducibility.
In this paper we will present our review of reproducibility
by reproducing one of the papers submitted to the ICLR2018
Conference.
Zhang, Zhao, and LeCun [1] show, that convolutional neural
networks (CNNs) can be used effectively on text classification
tasks given enough data. For this, they encode their training
data on a character level, and let a multi-layered CNN classify
it. They use a one-hot encoding scheme to represent the
different characters.
For this project, we try to replicate results of the paper
compressing encoding of words for use in character-level
convolutional networks for text classification, submitted on
the openReview platform for ICLR2018. The authors of this
paper try to improve the model of Zhang, Zhao, and LeCun [1]
by using a more efficient character encoding scheme, leading
to faster computation times of the network.
Instead of using a one-hot encoding, they propose an
encoding that depends on the frequency of occurrence of
the characters. It is claimed that this encoding scheme will
be competitive in accuracy while allowing for faster training
times. With the new encoding, characters that appear often
in the document can be represented in a compressed way,
resulting in smaller input matrices of the network, and thus
potentially shorter training time.
We chose to replicate this paper because we found the
approach of using convolutional networks in a natural lan-
guage setting to be very interesting. This approach might
revolutionize the way we see natural language problems, since
it shows that classification can be done, even when solely
looking at the smallest pieces of language.
We will verify the papers claims by replicating their in-
vestigation process. We implement their proposed encoding
scheme from the description in the paper, try to replicate their
Lino Toran Jenner
lino.toranjenner@mail.mcgill.ca
260793554
chosen CNN architecture and compare the results for three
of the datasets used in the original paper. We analyze both
accuracy as well as running times. We further discuss general
reproducibility of the paper, and focus on aspects that were
difficult to reproduce without further information.
II. T ECHNICAL S UMMARY OF M ETHOD E MPLOYED
In this section we quickly present the approach of the
authors. We first describe the encoding of documents, secondly
we present the network architecture used to train the model,
and finally we give an overview of how this model was tested.
A. Encoding Procedure
Given a number of documents, each belonging to a distinct
category, each document is first encoded to be interpretable
by a CNN. The documents are encoded at the character level.
First, the characters are ranked by frequency over the entire
training set. Each character is then encoded depending on its
rank: if a character has rank i, its encoding will start and
end with a 1, and will have i times the digit 0 in between.
For example, if the three most frequent characters in a corpus
are e, a and t, then their encoding will be 11, 101
and 1001, respectively. The same scheme is applied until
the character with lowest rank is reached. This results in a
distinct encoding for each character in the documents.
Once this is done, each document is encoded as a matrix,
with each row representing a word of the document. Words
are encoded by concatenating the codes of each character, up
to a maximum size of 256 digits. The maximum number of
words represented per document is set to 128. Hence each
document is represented by a matrix of size 128x256. If words
representations are less than 256 digits, or if documents consist
of less than 128 words, then the remaining columns or rows
are simply padded with 0s.
Compared to the model by Zhang, Zhao, and LeCun [1],
which used the first 1012 characters of each document as input
and used a one-hot encoding for each character (resulting in a
1012 by 70 matrix for each document), this approach is much
more efficient, since it can fit more information in a smaller
matrix.
B. Network Architecture
For classification, the authors use a CNN with multiple
layers. From the figure posted in the original paper we
reconstructed the following network architecture: The input
is fed into 4 parallel convolutional layers (256 filters, kernel
size = 3). The output of each of these is maxpooled (poolsize
= 3). The outputs of these layers are then concatenated in
a merge layer. Two more convolutional (filter = 256, kernel
size = 5 for both) and two MaxPool layers (pool sizes 3
and 4) follow this step (in the order Conv->MaxPool->Conv-
>MaxPool). The result of this is flattend and passed to a fully
connected layer consisting of 128 neurons. The final fully
connected layer consists of a number of neurons consistent
with the distinct classes of the current problem. The activation
functions of the network are not specified in the paper.
C. Testing
The performance of the model is assessed in two ways: the
training time and the error rate. They compare the running
time on 2 GPUs, an Nvidia GeForce 1080ti and an Nvidia
GeForce 930M, and the compare the test error rate between
their model and the results provided in the paper by Zhang,
Zhao, and LeCun [1]. They claim that the training time is
faster, even on a less powerful GPU, and that the error rate
is comparable to the model proposed by Zhang, Zhao, and
LeCun [1].
III. R EPRODUCIBILITY M ETHODOLOGY
In this section we discuss which aspects of the original
paper we tried to reproduce and how we approached the
reproduction.
A. Data Collection
The original paper contains little description of the datasets
used, except mentioning that they are the same as the ones in
Zhang, Zhao, and LeCun [1], and that a detailed description
can be found in that paper. In the paper by Zhang, Zhao,
and LeCun [1], the way in which the data is procured is
not entirely clear. Some of the datasets are publicly available,
well-maintained with proper documentation, and available for
access, such as the Yahoo! Answers Topic Classification.
Other datasets were extracted from public websites, such as
the DBpedia Ontology Classification dataset. However, it was
not clear as to how the information was scrapped from the
websites, and no corresponding code was provided. Moreover,
there were other datasets for which a link is provided, but
the information provided through the link is outdated, and no
longer useful, as it was the case for the AG news dataset.
Specifically, a link was provided to the raw data where the
news articles were extracted from, however almost all the links
to the news articles were no longer functional.
The lack of clarity in terms of data access caused confusion
when trying to reproduce the results. A significant amount
of time had to be spent tracking down the used datasets.
However, upon examining the website of the lab at which the
paper by Zhang, Zhao, and LeCun [1] was written, a link to
the actual data was discovered, but the paper itself does not
include a direct reference to it. Within the content provided,
train and test sets are provided separately, making the split
easily reproducible.
To make the paper under review more reproducible in the
future, it should include a direct link to dataset it used, as well
as an original description of the data and the corresponding
code used. But overall, if the link to the dataset is directly
provided, the data collection process is highly reproducible.
To account for the scope of this project, we decided to only
focus on three of the eight provided datasets. We chose the
following 3 datasets:
AGs news dataset: a corpus made up of news articles
classified in 4 classes, with 120,000 training examples
and 7,600 testing examples [1].
DBpedia ontology dataset: a corpus of wikipedia articles
classified into 14 categories with 560,000 training exam-
ples and 70,000 testing examples [2].
Yelp polarity dataset: a corpus of Yelp reviews classified
as either positive or negative, with 560,000 training
examples and 38,000 testing examples [3].
B. Implementation of the Model
In order to reproduce the results from the paper, it was first
necessary to implement the model given in the article, namely
both the encoding and the CNN architecture, since the code
was not provided to us. We chose to use an environment as
close as possible to the one used by the authors of the article.
The coding was hence done in Python 3.6 and the CNN was
implemented using Keras 2.0 [4], with Theano as backend [5].
a) Encoding and Preprocessing: Using the description
of the original paper, we wrote an encoding function. The
compressed code of each character in the training set was
calculated in the way described above. We then created a
dictionary of characters present in the training set and mapped
them to their respective encodings (based on frequency in the
training set). Each batch of data sent to the CNN is encoded
using this function.
Several difficulties emerged while preprocessing the data.
The authors wrote, that they preprocessed the data by using
only lowercase characters. However no further instructions of
data preprocessing were given. This lead to some confusion
when we took a closer look at the datasets: For example,
some datasets contained many occurrences of the character
\ between 2 words, instead of a space (the readme files
informed us, that these represent newline characters). However
no explanation was given in the paper on how to encode or
whether to ignore these characters in the original paper. We
chose the easiest approach and treated all characters as regular
characters.
The datasets include a title and a description. It is not stated
in the paper, which of these should be used for classification.
We decided, coherent with the procedure used by Zhang and
LeCun [6], to use the concatenation of title and description
as data samples. For easier reproduction, clear explanations of
all the preprocessing decisions used should be provided.
b) Convolutional Neural Network: More challenges for
reproducibility came up for the implementation of the CNN.
The authors described their network architecture solely using a
table with the columns Layer type, shape, # of parameters,

connected to. We deduced our final implemented network
structure from this table. Implementing the CNN was not
a straightforward task, as many parameters of the network
had to be inferred by looking at the values in the table and
making deductions about the network structure. For instance,
we deduced, that the convolutional layers use 256 filters and
the valid padding with stride of 1, by looking at the described
output: (None,126,256). For reproducibility purposes, it would
have been helpful (and time saving), to state all the relevant
parameters of the layers, or to provide the code.
One parameter, that was completely missing from the paper,
and that was therefore hard to correctly reproduce, were the
chosen activation functions. We contacted the authors but
received no response in time for submission. In the end, we
chose to implement ReLU (rectified linear units) activation
functions for the convolutional layers and the first fully con-
nected layer. We expect that reLu activations are the ones used
by the author, since they are typically used in CNNs. We chose
to implement a softmax activation function for the output layer.
While some of the hyperparameters of the model were stated
very clearly in the paper, others were missing. The number
of epochs, the optimizer (with parameters) and batch sizes
were provided and copied for our implementation. However,
no information was provided on the loss function used, or
if the batches were provided to the CNN in shuffled order.
Additionally, hardware, operating system and programming
environment was specified by the authors. They also provided
a good description of the used libraries (with version numbers).
This made it easy to compare the final running times and make
deductions as to why there might be differences.
C. Testing Accuracy on Different Corpora
Once we had implemented this model, the second step was
to try and reproduce the performance the authors claimed it
achieved on the different corpora used.
One issue we ran into was figuring out how the loss
used to compare models in Table 6 of the original paper
was calculated. The authors titled the table "Loss comparison
among traditional models", but failed to include more details.
We contacted the authors using the openReview platform and
they quickly answered. They informed us, that the values
represented in table 6 were the error rates on the test set.
Once this was made clear, we were able to try and reproduce
the error rates on the three datasets mentioned earlier.
D. Comparing Running Time
In the original paper, the authors stress out on several
occasions that their model achieves fast training time (espe-
cially compared with the running times in [6]). It therefore
seems important to us to be able to reproduce the same
training time. However, the authors trained their network with
GPUs (either Nvidia 1080ti or Nvidia 930M) that were not
available to us. Even if they were, the exact same running
time might not be possible to achieve because of differences
in the implementations of the model. For example, because of
limited memory, we are not able to store the encoding of all the
Dataset Original Results Reproduced Results
ag_news 12.33 11.89
dbpedia 2.07 2.33
yelp_polarity 7.96 7.84
TABLE I: Comparison of the test error (in %) between original
and replicated results.
documents of a large corpus. Therefore we need to recompute
the encoding of a document at each epoch. This means that
the encoding of a document is calculated 5 times in total (one
time for each epoch). We do not know if the authors had to
use a similar trick, or if they had a larger amount of memory
to work with.
In order to train the model, we obtained Google Cloud
Computing credits which enabled us to train the model on
a Nvidia K80 GPU.
Since we did not have the same GPU, our goal was not
to get the same training time, but to get the same proportion
between training times on different datasets. For example, on
the Nvidia 930M GPU, the authors report a training time of
25 minutes per epoch on the AG news corpus, and a training
time of 116 minutes per epoch on the DBpedia dataset. It
thus took about 4.6 times longer to train the model on the
DBpedia dataset. For reproduction, we wanted to see if we
would reach a similar factor between the training time of our
implementation of the model on those 2 datasets. Note that the
factors are not the same between the two GPUs used in the
original paper. We consider the factors obtained on the 930M
GPU to be more reliable as the training time on this GPU
is longer, and therefore the time reported is less sensitive to
rounding approximations.
IV. E MPIRICAL R ESULTS
In this section we provide the empirical results obtained
during our approach and compare them with the results from
the original paper.
A. Error Rates
Table I shows the error rates obtained on the test set, for
the three different datasets that were analyzed. The error rates
obtained are very close to the ones from the original paper, but
not exactly the same. Interestingly, we achieve better results
for two of the datasets (ag_news and yelp_polarity) but worse
results on the dbpedia dataset.
B. Running Times
Table II shows the training time per epoch of the original
paper and our reproduction. For better comparison, the original
paper tested their running times on two different GPUs. Our
times are similar to the times achieved by the original times
when testing on the 1080ti GPU. These times are significantly
faster than the ones obtained on a 930M GPU. What is
important to notice is that we obtained a similar proportion
between training times on the different datasets, even though
the model seems to be training a little faster than expected on
the yelp_polarity dataset.
 Dataset Original Time on Nvidia 1080ti GPU Original Time on Nvidia 930M GPU
ag_news 3 min 25 min
dbpedia 18 min 116 min
yelp_polarity 21 min 119 min
TABLE II: Time comparison of training time (per epoch) between the reports of the original paper and our reproduction.
V. D ISCUSSION
In this section we discuss our results and the overall
reproducibility of the paper.
Our results (test error as well as running times) look very
similar to the results of the original paper, and we were able
to reproduce it for the most part. Our running times (on a
Tesla K80) were also similar to the authors. This is especially
important, since the authors tried to show, that their encoding
approach is more effective than the one by Zhang, Zhao, and
LeCun [1]. Since our running times are consistent with theirs,
it is very probable, that this encoding scheme is in fact more
time-efficient while achieving similar accuracies. To make sure
of this, one would have to experiment the model on the exact
same machines used by the authors.
We could however not reproduce the exact results of the
original paper. This was due to the fact, that not all of the
details of preprocessing, network architecture and hyperpa-
rameters were available to us. In the following, we address
the main difficulties we faced in our replication:
The original paper did not contain any direct links or other
information on the datasets. Therefore, some extra work and
time had to be spent tracking them down. Once we had found
the datasets, we were able to exactly replicate the split into
training and test set, because this split was already provided.
Overall, this paper is well-written and easy to follow. The
description of the encoding is very clear, and the network ar-
chitecture, although missing a few important details like what
the activation functions are, is pretty clear as well. However,
for easier replication, it would be helpful to include the exact
configuration so that deductions about the parameters used
are not necessary. Additionally, more exact description of the
preprocessing employed is also needed for exact replication.
The computing infrastructure (including library versions)
used was clearly explained, and even though we did not
possess the exact same environment, we believe that one
would be able to set up the exact same infrastructure with
the information provided.
Unfortunately, the authors did not provide any code to
support their findings, and therefore we had to implement the
full model on our own, sometimes making arbitrary decisions.
However, the clarity of the paper and its level of details
were of big help in this implementation. Implementing the
full encoding and the network architecture did require some
time on our hand, but we found that once this was done, the
experiments ran rather smoothly. It is important to note that
without Google Cloud computing services, reproducing the
results from the paper would have been significantly harder
as the training time would have been very slow, and thus we
would have been limited in our different trials.
Reproduced Time (on Nvidia Tesla K80)
4.87 min
23.11 min
20.17 min
Finally, interactions with the authors were necessary to
clarify a few points that were left ambiguous after reading
their paper. As mentioned earlier, the authors provided us
with a clarification of the loss used to compare models. We
also contacted them about the activation functions used in the
network but received no reply until the submission deadline.
VI. CONCLUSIONS
Overall, the work under investigation was fairly repro-
ducible, with both the reproduced test error and running
time resembling closely to the results reported by the paper.
However, an exact reproduction was not possible mainly
due to the fact that not all of the details of preprocessing,
network architecture as well as hyperparameters involved were
available to us.
VII. S TATEMENT O F C ONTRIBUTIONS
Benjamin was in charge of implementing the encoding of
the documents and wrote some of the report.
Seara was reponsible for data collecting, and some portion
of the report.
Lino implemented the CNN and wrote part of the report.
We hereby state, that all the work presented in this report
is that of the authors.
ACKNOWLEDGMENT
We would like to thank Google Compute Cloud for provid-
ing us access to a computing instance. We also want to thank
the authors of the original paper for answering our questions.
R EFERENCES
[1] X. Zhang, J. Zhao, and Y. LeCun, Character-level con-
volutional networks for text classification, in Advances
in neural information processing systems, 2015, pp. 649
657.
[2] DBpedia, DBpedia Ontology, http : / / wiki . dbpedia . org /
services-resources/ontology, accessed 2017-12-14.
[3] Yelp, Yelp Dataset Challenge, https : / / www. yelp . com /
dataset/challenge, accessed 2017-12-14.
[4] F. Chollet et al., Keras, https://github.com/fchollet/keras,
2015.
[5] R. Al-Rfou, G. Alain, A. Almahairi, C. Angermueller,
D. Bahdanau, N. Ballas, F. Bastien, J. Bayer, A. Belikov,
A. Belopolsky, Y. Bengio, A. Bergeron, J. Bergstra, V.
Bisson, J. Bleecher Snyder, N. Bouchard, N. Boulanger-
Lewandowski, X. Bouthillier, A. de Brbisson, O.
Breuleux, P.-L. Carrier, K. Cho, J. Chorowski, P. Chris-
tiano, T. Cooijmans, M.-A. Ct, M. Ct, A. Courville,
Y. N. Dauphin, O. Delalleau, J. Demouth, G. Desjardins,
S. Dieleman, L. Dinh, M. Ducoffe, V. Dumoulin, S.
Ebrahimi Kahou, D. Erhan, Z. Fan, O. Firat, M. Germain,
X. Glorot, I. Goodfellow, M. Graham, C. Gulcehre, P.
Hamel, I. Harlouchet, J.-P. Heng, B. Hidasi, S. Honari, A.
Jain, S. Jean, K. Jia, M. Korobov, V. Kulkarni, A. Lamb,
P. Lamblin, E. Larsen, C. Laurent, S. Lee, S. Lefrancois,
S. Lemieux, N. Lonard, Z. Lin, J. A. Livezey, C. Lorenz,
J. Lowin, Q. Ma, P.-A. Manzagol, O. Mastropietro,
R. T. McGibbon, R. Memisevic, B. van Merrinboer,
V. Michalski, M. Mirza, A. Orlandi, C. Pal, R. Pascanu,
M. Pezeshki, C. Raffel, D. Renshaw, M. Rocklin, A.
Romero, M. Roth, P. Sadowski, J. Salvatier, F. Savard,
J. Schlter, J. Schulman, G. Schwartz, I. V. Serban,
D. Serdyuk, S. Shabanian, . Simon, S. Spieckermann,
S. R. Subramanyam, J. Sygnowski, J. Tanguay, G. van
Tulder, J. Turian, S. Urban, P. Vincent, F. Visin, H. de
Vries, D. Warde-Farley, D. J. Webb, M. Willson, K. Xu,
L. Xue, L. Yao, S. Zhang, and Y. Zhang, Theano: A
Python framework for fast computation of mathematical
expressions, ArXiv e-prints, vol. abs/1605.02688, May
2016. [Online]. Available: http : / / arxiv. org / abs / 1605 .
02688.
[6] X. Zhang and Y. LeCun, Text understanding from
scratch, ArXiv preprint arXiv:1502.01710, 2015.
A PPENDIX
Our review on OpenReview:
This is an executive summary of a more detailed report that
can be found here: (link will be included in final submission)
This project is part of the ICLR2018 Reproducibility Chal-
lenge. The goal of the challenge is to improve the quality of
submissions by highlighting the importance of reproducibility.
In this review we present the main challenges we faced
while reproducing the ideas presented in this paper as well
as highlighting which aspects were easily reproducible.
We gathered three of the analyzed data sets, implemented
the encoding scheme and the convolutional neural network
according to the description given in the paper.
Our results (test error as well as running times) are very
similar to the results of the original paper, and we were able
to reproduce it for the most part. The test errors achieved on
the datasets were the following (original results in parenthesis):
ag_news 11.89 (12.33), dbpedia 2.33 (2.07) & yelp_polarity
7.84 (7.96). Our training time per epoch (trained on a Nvidia
Tesla K80 whereas the authors used a Nvidia 1080ti): ag_news
4.87 min (3 min), dbpedia 23.11 min (18 min) & yelp_polarity
20.17 min (21 min).
 However, as can be seen from these values, we could not a more detailed description would have been helpful.
reproduce exact results. This was mainly due to the fact, that The computing infrastructure (including library versions)
not all of the details of preprocessing, network architecture used was clearly explained, and even though we did not
and hyperparameters were available to us. In the following, we possess the exact same environment, we believe that one
address the most relevant points we faced in our replication: would be able to set up the exact same infrastructure with
the information provided.
The original paper does not contain any direct links or
We did not have access to the code of the authors and
any other information on the data sets used. All the
therefore had to implement the full model on our own. In
information on the data sets was gathered from papers
some cases we were missing information on parameters
cited. Therefore, extra work and time had to be spent
or how exactly things were implemented (see above).
tracking these data sets down. Once we had found the
Our implementation could therefore be different from the
data sets, we were able to exactly replicate the split
one of the original authors, affecting computation times.
into training and test set, because this split was already
However, since the paper was pretty clear for the most
provided.
part, and our results resemble the ones of the authors, we
The description of the encoding function is very clear
are relatively certain that this has not been a big issue.
and examples make it easy to understand. Were confident
Finally, some interaction with the authors was necessary
in our replication of the encoding process. However, the
to clarify a few points that were left ambiguous after
paper doesnt go into detail on the preprocessing of the
reading their paper. Some of the results and tables were
data sets. We were unsure how newline characters in the
not described very extensively by the authors and there-
documents were preprocessed for instance. For replica-
fore needed clarification. We contacted them using this
tion purposes, a detailed description of the preprocessing
platform and received a quick answer. We also contacted
employed would have been helpful.
them about the activation functions used in the network
The network architecture is presented in form of a table.
but received no reply until the submission deadline.
Some important implementation details are missing (ac-
tivation functions, loss function used), and others have to
be deduced by observing the output dimensions of the
individual layers of the network. This makes it difficult
to exactly replicate the network the authors used. Again,