Study 03 · Computer vision · Virtual try-on
Virtual Try-On App for Clothing on Photos of People
2021 · Undergraduate research studentI built the Android-to-server workflow that returns a virtual try-on image after the user selects a person photo and a garment.
- 1Select a photo of a person and the clothing to "try on."
Send people photos and clothing IDs from your Android app
- 2Human body parsing, pose, and clothing masks
Organized in ACGPN input format
- 3Generation of "virtual try-on" images
Flask server returns to app
Integrated implementation combining ACGPN, human body parsing, and pose estimation public models
Roles and outcomes
- Assigned role
- Configured input preprocessing and clothing masks for a virtual try-on model; integrated a Flask upload API with an Android client built using Kotlin and Retrofit.
- Results
- We developed an integrated workflow that takes a person's photo and a selected garment, performs preprocessing, generates an image of the person wearing the item, and returns the result to the app. We reused existing generative models as well as pose and body-parsing models; our own implementation focused on input configuration, mask processing, and the integration between the server and the app.
Implemented and analyzed personally
- Organizing human body parsing, pose, and clothing masks into the ACGPN input format
- Generate clothing mask with OpenCV GrabCut
- Flask upload, result return API
- Kotlin, Retrofit Android client connection
Based research and disclosure model
- ACGPN virtual wear model
- Publicly available models for human body parsing and pose estimation
Experiment results and trial and error
Verified the process from upload to the return of the virtual try-on result, and analyzed mask errors.
Public code run · VITON-HD test images (CC BY-NC 4.0) · Faces mosaicked · Times below are CPU measurements
Requesting try-on images through the server upload API showed that for 13 of 40 garments the GrabCut rectangle extended outside the image, so the mask bled into the background.
- Apple M1 Pro CPU
- PyTorch
- Public model weights
- Server upload API run
- 13 / 40 setsClothing where the mask has bled into the background
- 0.93 → 0.96Mask IoU after refining the bounding box
- 4.9 secondsMedian CPU time per request
Problem and solution
ProblemVirtual try-on models require not only a photograph of a person but also specific, formatted data regarding poses, body parts, and clothing regions.
ResolveThe outputs from public human parsing and pose estimation models have been formatted to meet the input requirements for ACGPN, with an added preprocessing step that uses GrabCut to generate clothing masks.
ProblemThe process for handling file input/output across multiple models is too complex for a user to manage manually.
ResolveWe configured the Flask upload API to run preprocessing and inference and return the resulting image, and connected the Android client to Retrofit.
ProblemIf the background is blended into the clothing mask, the generated clothing and the torso may become blurred together.
Supplementary experimentWhile running the public code, I found that the initial GrabCut rectangle extended past the image boundary; in a comparison that kept the rectangle inside the image, the median mask IoU rose from 0.929 to 0.957.
Experiment comparison and detailed results
| Item | Result | Condition |
|---|---|---|
| Processing time for a single request | Median 4.90 seconds (interquartile range 3.57–6.57 seconds) | 40 requests, CPU |
| Creation of ACGPN | Median 2.27 seconds | Same 40 iterations |
| SSIM for the reconstruction of the same garment. | Mean 0.869 · Median 0.884 | 20 individuals |
| GrabCut Mask IoU | Median 0.929 → 0.957 | 40 garments · Code as is → Place the rectangle inside the image |
Observed causes and remaining challenges.
- GrabCut rectangle error
The rectangle (1, 1, w, h) extended beyond the image boundaries, leaving background cues only along the top and left edges; in 13 out of 40 instances, the mask bled into the background.
- Three types of failure
If the mask leaks into the background, the torso area will bleed; if the sleeve length changes, "ghosting" (residual images) may appear on the arms; and if the overall length differs, the original clothing pattern may remain visible below the hem.
- How to read SSIM
These values are derived by "redressing" the subject in the same garment and comparing the result to the original image. Since the area outside the garment is taken directly from the input, the scores are inflated and cannot be compared against the values reported in the ACGPN paper.
- Data and weights
VITON data is restricted from distribution, so VITON-HD test images were used, and ACGPN and OpenPose weights are released for non-commercial research use.
Criteria for interpreting results
- Integrated implementation, not an enhancement for image quality or speed
- Results obtained by running the model on several dozen test images using publicly available weights
- Does not display product-level performance or commercial operational status
Connecting the inputs and outputs of different modules is much like the current process of executing, within Fusion, the processing plans generated by the server.
AI that reads 3D models and designs processing strategies