A Sensitivity Analysis of (and Practitioners Guide to) Convolutional Neural Networks for Sentence Classification

Convolutional Neural Networks (CNNs) have recently achieved remarkably strong performance on the practically important task of sentence classification (kim 2014, kalchbrenner 2014, johnson 2014). However, these models require practitioners to specify an exact model architecture and set accompanying hyperparameters, including the filter region size, regularization parameters, and so on. It is currently unknown how sensitive model performance is to changes in these configurations for the task of sentence classification. We thus conduct a sensitivity analysis of one-layer CNNs to explore the effect of architecture components on model performance; our aim is to distinguish between important and comparatively inconsequential design decisions for sentence classification. We focus on one-layer CNNs (to the exclusion of more complex models) due to their comparative simplicity and strong empirical performance, which makes it a modern standard baseline method akin to Support Vector Machine (SVMs) and logistic regression. We derive practical advice from our extensive empirical results for those interested in getting the most out of CNNs for sentence classification in real world settings.

💡 Research Summary

The paper conducts a thorough sensitivity analysis of one‑layer convolutional neural networks (CNNs) for sentence classification, aiming to demystify the many architectural and hyper‑parameter choices that practitioners must make. Using nine widely‑used benchmark datasets—including MR, SST‑1, SST‑2, Subj, TREC, CR, MPQA, Opi, and Irony—the authors adopt a consistent preprocessing pipeline and perform 10‑fold cross‑validation (CV). To capture stochastic variability inherent in training neural networks, each experimental setting is replicated ten times (each replication being a full 10‑fold CV), and for the baseline they even repeat CV 100 times to report mean, minimum, and maximum performance.

The baseline architecture mirrors Kim (2014): sentences are represented as matrices of pre‑trained word embeddings (either Google word2vec or GloVe), filters slide over the matrix with region (height) sizes of 3, 4, and 5, each size having 100 feature maps, ReLU activation, 1‑max pooling, dropout 0.5, and an L2 norm constraint of 3. Optimization uses AdaDelta with a minibatch size of 50 and early stopping on a 10 % validation split.

The authors systematically vary one component at a time while keeping all others fixed, exploring: (1) static vs. non‑static embeddings, (2) alternative embedding sources (word2vec vs. GloVe), (3) filter region sizes from 2 to 6, (4) number of feature maps (50, 100, 200, 300), (5) activation functions (ReLU, tanh, sigmoid), (6) pooling strategies (1‑max, average, k‑max), (7) dropout rates (0.2, 0.5, 0.7), and (8) L2 norm caps (1, 3, 5).

Key findings:

• Embedding finetuning: Non‑static embeddings (i.e., allowing the word vectors to be updated during training) consistently outperform static embeddings by 2–3 percentage points across all datasets, confirming that task‑specific adaptation of word vectors is beneficial. The choice between word2vec and GloVe matters far less (differences ≤ 2 pp).
• Filter region size: Sizes 3–5 provide the most robust performance. Size 2 captures only very short n‑grams and underperforms on longer sentences; size 6 or larger inflates the parameter count and leads to over‑fitting without measurable gains.
• Number of feature maps: Increasing from 100 to 200–300 maps yields noticeable accuracy improvements, especially on larger corpora (e.g., SST‑1, MR). Going beyond 300 offers diminishing returns while substantially increasing training time.
• Activation function: ReLU outperforms tanh and sigmoid in both convergence speed and final test accuracy (≈ 1 pp gain), likely because its sparsity‑inducing property works well with dropout and L2 regularization.
• Pooling: 1‑max pooling is superior to average pooling, which discards the most salient feature, and to k‑max pooling, which introduces an extra hyper‑parameter without clear benefit.
• Dropout: A dropout rate of 0.5 strikes the best balance; lower rates (0.2) lead to over‑fitting, while higher rates (0.7) cause unstable training and reduced performance.
• L2 norm constraint: Setting the L2 cap to 3 provides the optimal regularization strength. Values ≤ 1 overly shrink weights, harming expressive power; values ≥ 5 effectively remove the constraint.

The authors also highlight the intrinsic variance of CNN training: even with identical hyper‑parameters, the mean accuracy across 100 repetitions of 10‑fold CV can vary by ±2–3 pp. This underscores the importance of reporting performance ranges and, when possible, averaging over multiple random seeds.

When compared against strong SVM baselines (linear SVM on bag‑of‑words, RBF‑SVM on averaged word vectors, and hybrid bag‑of‑words + word‑vector features), the non‑static CNN consistently achieves 3–7 pp higher accuracy, with the largest gains on sentiment tasks (SST‑2, MR) and question classification (TREC).

Practical recommendations derived from the study:

1. Use pre‑trained embeddings (word2vec or GloVe) and allow them to be fine‑tuned (non‑static).
2. Adopt a filter region size of 3 (or 3–5 if computational budget permits) and allocate 200–300 feature maps per size.
3. Employ ReLU activation, 1‑max pooling, dropout 0.5, and an L2 norm cap of 3.
4. If hyper‑parameter search resources are limited, the above defaults already yield near‑optimal performance across diverse sentence classification tasks.
5. Always report mean performance together with variance (e.g., min/max or standard deviation) obtained from multiple CV repetitions to convey the stochastic nature of the results.

In sum, the paper provides a data‑driven, empirically validated guide for configuring one‑layer CNNs for sentence classification, clarifying which design choices matter most and offering concrete default settings that practitioners can adopt with confidence.