Find the hand
Locate the relevant hand region before sending pixels to the image classifier, rather than analyzing the whole camera frame.
ASL Vision is an interactive computer vision project for recognizing 24 static American Sign Language (ASL) handshapes. It connects a PyTorch convolutional neural network to MediaPipe hand localization, optimized webcam inference, temporal prediction smoothing and Grad-CAM visual explanations.

The original educational CNN recognized small, grayscale images, but an interactive application needs to find a hand in a natural scene, process frames efficiently and present predictions in a way people can actually read.
Locate the relevant hand region before sending pixels to the image classifier, rather than analyzing the whole camera frame.
Separate camera display from heavier detection and CNN inference so slower predictions do not stall the stream.
Use consecutive-prediction confirmation and minimum display time to reduce rapid, distracting letter changes.
A shared hand-detection and preprocessing workflow powers both image uploads and live webcam inference.
Upload a photo or acquire frames through the browser camera.
MediaPipe estimates 21 hand points and a padded square crop.
Convert the cropped hand to 28 × 28 grayscale and classify it.
Confirm repeated predictions and display the letter clearly.
MediaPipe handles localization; the CNN classifies the resulting crop. These are separate models.
Inference is performed locally by Python; the browser provides the user interface and the camera stream. Heavy recognition work does not block video rendering.
Editorial-style interface with image upload, camera snapshot, live video and model evaluation.
Finds hand landmarks; dynamically crops only the relevant region.
Predicts among 24 static ASL classes using a compact grayscale input.
Maintains responsive video with an independent inference worker and latest-frame processing.
The interface supports photo upload, single camera captures and a live webcam. Each screenshot below comes from the working application.



Evaluation uses 7,172 samples from the Sign Language MNIST validation dataset. It measures performance on that dataset, not on uncontrolled camera scenes.
Overall classification accuracy on the validation split.
Class-balanced F1 summary across 24 letters.
Static letters; J and Z involve movement.
The project evolved from educational CNN exercises into a reproducible, interactive demonstration with additional computer-vision and inference components.
Recovered CNN weights in a reusable PyTorch model definition and evaluated the classifier.
Added landmark detection, square crops and 28 × 28 grayscale preprocessing.
Moved MediaPipe and CNN work out of the video callback and discarded stale frames.
Used repeated predictions and a minimum hold time to keep letters readable.
Added visual maps of influential regions for predictions on static hand crops.
Separated application and model modules with automated tests and validation reporting.
The biggest lesson was understanding the gap between a model that scores well on a curated dataset and a complete system that performs usefully for someone in front of a camera.
Good model metrics are a starting point, not a finished user experience. This project pushed me to reason about domain shift, latency, explainability and the interaction between several models in one application.
MediaPipe localizes the hand; the CNN identifies its static configuration. Separating these tasks improved framing and made failures easier to diagnose.
Processing every frame synchronously made the video unusable. A latest-frame queue and background inference preserved a fluid camera display.
Rapid letter changes were difficult to follow. Confirmation rules and display hold time improved the practical clarity of predictions.
Strong Sign Language MNIST results do not automatically transfer to different hands, backgrounds and lighting. Grad-CAM and explicit scope statements help interpret—not overstate—the system.
ASL Vision connected deep learning with image preprocessing, streaming performance and user-facing explanations. The result is a more complete demonstration of how to turn an academic model into an interactive application.
ASL is a complete language with grammar, spatial structure and movement. This system recognizes a subset of static manual alphabet handshapes; it cannot translate full ASL sentences or recognize dynamic J and Z.
Static handshape classification, hand tracking, photo and webcam inference, baseline evaluation and Grad-CAM visualization.
Independent evaluation on diverse real-camera images, backgrounds, lighting and signers; calibrated confidence; and dynamic gesture understanding.
Source modules, tests, reproducibility instructions and model evaluation are documented on GitHub.