Facial Emotion Recognition Using Convolutional Neural Networks: Dataset, Model Architecture, and Evaluation

Human facial expressions can be categorized into seven basic emotions—happy, sad, surprise, fear, anger, disgust, and neutral—providing rich information for applications such as retail satisfaction measurement, medical monitoring, and entertainment content optimization.

The study uses the FER2013 dataset from a 2013 Kaggle challenge, containing 35,887 48×48 grayscale face images labeled with the seven emotions. Because the "disgust" class is severely under‑represented (only 113 samples), it is merged with "anger" to create a balanced six‑class dataset of 28,709 training images and 3,589 images each for validation and testing.

The model is a convolutional neural network (CNN) implemented with Keras' Sequential() API. Input images are pre‑processed with OpenCV: faces are detected using a Haar cascade, converted to grayscale, resized to 48×48, and reshaped to a (1,48,48) NumPy array before being fed to the network.

The architecture consists of an input layer, multiple convolutional layers (with 3×3 kernels and shared weights), max‑pooling layers (2×2 windows), dense (fully‑connected) layers, dropout regularization, and a softmax output layer that predicts the probability of each emotion.

Initial experiments with a simple three‑conv‑layer CNN achieved only 15% accuracy, essentially random guessing. To improve performance, the model was expanded to nine layers (three conv layers followed by a max‑pooling layer, repeated three times) and trained on an AWS g2.2xlarge GPU instance, dramatically reducing training time.

Cross‑validation of the final model yields an overall accuracy of 58%. The model distinguishes positive emotions (happy, surprise) well, with accuracies of 76.7% and 69.3% respectively, but performs poorly on negative emotions such as sad (39.7%) and often confuses sad with neutral due to limited samples.

Visualization of feature maps after the second and third max‑pooling layers shows increasingly abstract representations, illustrating how deeper layers capture higher‑level patterns.

The author, Jostine Ho, is a data scientist and deep‑learning researcher with a background in petroleum systems engineering. References include the original FER2013 dataset paper, Stanford CS231n lecture notes, and the Dropout paper by Srivastava et al.

"Dataset: Facial Emotion Recognition (FER2013)" ICML 2013 Workshop in Challenges in Representation Learning.

"Andrej Karpathy's Convolutional Neural Networks (CNNs / ConvNets)" CS231n, Stanford University.

Srivastava et al., 2014. "Dropout: A Simple Way to Prevent Neural Networks from Overfitting", JMLR.

Duncan, D., Shine, G., English, C., 2016. "Report: Facial Emotion Recognition in Real-time", CS231n.

Related links: fer2013datagen.py , Keras Sequential API , Data visualization notebook .