Study 02 · Computer vision · Image classification

A model for classifying clothing photos with ResNet

2021 · Undergraduate research student

I trained a model to classify K-Fashion photos as dresses, outerwear, trousers or tops and analyzed errors for each class.

14 types of clothing photos
Sampling based on the class with the fewest instances
2ResNet50V2
Start with ImageNet pretrained weights
3Error analysis by class
Precision, recall, confusion matrix

Utilizing ImageNet pre-training ResNet50V2

TensorFlowResNet50V2Class imbalanceError analysis

Roles and outcomes

Assigned role: Implemented code for clothing classification using a pre-trained ResNet50V2 model, along with a workflow for class-based sampling and evaluation; misclassifications were analyzed using a confusion matrix.
Results: Built a training and analysis pipeline that classifies K-Fashion photos into dresses, outerwear, pants and tops. Later, on the public Fashion-MNIST dataset, I changed settings one at a time to separate how data augmentation and the shortcut connection affect predictions.

Implemented and analyzed personally

ResNet50V2 classification training code
Downsampling training and validation data by class
Reviewing class-specific metrics and confusion matrices
Analysis of classification errors for dresses and outerwear

Based research and disclosure model

ImageNet pretrained ResNet50V2 weights
K-Fashion data

Data analyzed in the original study

Confusion matrix for 4 classes in a public K-fashion experiment. Predictions for several actual classes are concentrated in Class 2. — Confusion matrix from the original K-Fashion experiment. Rows are actual classes and columns are predicted classes: 0 = dress, 1 = outerwear, 2 = pants, 3 = top. Several actual classes skew toward being predicted as pants (2). This is separate data from the Fashion-MNIST experiment below.

Number of class 0 to 3 learning and verification images from K-Fashion original research. Bar graph showing the difference in number of samples by class — Class distribution of the original K-Fashion dataset. This illustrates the rationale for sampling based on the minority class in both the training and validation sets.

Experiment results and trial and error

Traced the prediction skew to the augmentation settings and a shortcut bug

When running the same code on the public Fashion-MNIST dataset, the predictions became heavily biased toward one side. This was caused by issues in the augmentation settings and the shortcut connections.

RTX A6000
TensorFlow 2.15
Fashion-MNIST: 4 classes (dress, coat, trousers, T-shirt)
10 epoch

In Fashion-MNIST independent testing, accuracy comparison was 48.9% with original settings, 78.7% with shortcut modification, 94.4% with augmentation off, and 96.4% with multiple settings modified together. — Independent test accuracy on 4 Fashion-MNIST classes (seed 0). Turning off augmentation and fixing the shortcut were compared separately; the last bar changes the shortcut, augmentation and L2 settings together.

48.9% → 94.4%Comparison with augmentation disabled · Fashion-MNIST · seed 0
+29.8%pModification of shortcut connections only · Same seed 0
0%Reproduction rate with original settings · Same seed 0

Problem and solution

ProblemThe number of photos varies by clothing type, leading to a bias in the training sample toward certain classes.
ResolveDownsampled the training and validation datasets to match the number of samples in the least frequent class for each.
ProblemIt is difficult to determine which garments are being confused with one another based solely on the overall score.
ResolveWe recorded precision, recall, and confusion matrices for each class, and analyzed instances of misclassification—such as confusing dresses with outerwear—using accompanying images.
ProblemWhen predictions skew to one side, it is hard to tell whether the data or the code settings are the cause.
Root cause analysisOn Fashion-MNIST, I changed the augmentation, shortcut and regularization settings one at a time and traced the skew back to the code settings.

Experiment comparison and detailed results

8 Fashion-MNIST test photos. In the original setting, the two coat predictions were dresses and t-shirts, and the model with augmentation turned off guessed all 8 cards correctly. — Prediction examples for 8 Fashion-MNIST photos. The two models behind this image reached 47.2% and 94.2% overall accuracy, in runs separate from seed 0 in the chart above. The number of correct answers among these 8 photos does not represent overall accuracy.

Independent test accuracy · 1,000 images per class · Values for seeds 0 and 1
Modified settings (one at a time)	Test Accuracy	Proportion of the most frequently predicted class
Original settings	48.9% / 39.1%	36% / 49%
Augmentation off	94.4% / 94.3%	26% / 26%
Shortcut connection modification	78.7% / 63.1%	35% / 53%
L2 coefficient 0.01 → 0.0001	39.4%	48%
Blend precision off	52.2%	51%
Momentum: 0.1 → 0.9	24.9%	99.9%
Shortcut modification + augmentation disabled + L2 0.0001	96.4%	25%

Observed causes and remaining challenges.

Augmentation was stronger than intended
Intended as light augmentation, but it actually resulted in ±72° rotation, vertical flipping, and scaling down to half-size; due to caching, the transformations from the first epoch remained fixed throughout.
Shortcut connection error
The identity shortcut passed the value that went through BN and ReLU instead of the original input. The ImageNet weights went in without error and were invisible.
Evaluation method
The recall of the README was based on a threshold of 0.5 and the validation data was rewritten in the evaluation. The accuracy posted here is measured on a non-overlapping test set.

Criteria for interpreting results

The evaluation in the README is not an independent test as it rewrites the validation data.
Does not verify the root cause of the error (e.g., imbalances, viewing angle/composition)
The figures below come from training on 4 Fashion-MNIST classes for 10 epochs

Points connecting to recent projects

The habit of breaking errors down by class and questioning what an evaluation split means has stayed with me. For the CAD machining-feature model, too, I look at errors per class separately and split the data by geometry family for evaluation.

AI that reads 3D models and designs processing strategies