>_ Jorge Uc / projects← All projects
~/projects/asl-vision // COMPUTER VISION
◆ Interactive AI demoPyTorch + MediaPipeExplainable AI

From hand movement to model insight.

ASL Vision is an interactive computer vision project for recognizing 24 static American Sign Language (ASL) handshapes. It connects a PyTorch convolutional neural network to MediaPipe hand localization, optimized webcam inference, temporal prediction smoothing and Grad-CAM visual explanations.

ASL Vision: real prediction for letter Y
PROJECT FILE / ASL-001PYTORCH / MEDIAPIPE
VALIDATION ACCURACY98.19%Sign Language MNIST
MACRO-F10.9804Validation split
CLASSES24Static ASL letters
SAMPLES7,172Validation images
01 / PROJECT CONTEXT

Classification is only one part of the problem.

The original educational CNN recognized small, grayscale images, but an interactive application needs to find a hand in a natural scene, process frames efficiently and present predictions in a way people can actually read.

01PERCEPTION

Find the hand

Locate the relevant hand region before sending pixels to the image classifier, rather than analyzing the whole camera frame.

LANDMARKS / CROPPING
02RESPONSIVENESS

Keep video fluid

Separate camera display from heavier detection and CNN inference so slower predictions do not stall the stream.

REAL-TIME / THREADING
03READABILITY

Stabilize output

Use consecutive-prediction confirmation and minimum display time to reduce rapid, distracting letter changes.

TEMPORAL SMOOTHING
02 / RECOGNITION WORKFLOW

From a raw frame to a readable letter.

A shared hand-detection and preprocessing workflow powers both image uploads and live webcam inference.

> pipeline.inferenceCONCEPTUAL VIEW
01 / INPUT

Photo or video

Upload a photo or acquire frames through the browser camera.

02 / LOCALIZE

Hand landmarks

MediaPipe estimates 21 hand points and a padded square crop.

03 / PREDICT

PyTorch CNN

Convert the cropped hand to 28 × 28 grayscale and classify it.

04 / PRESENT

Stable output

Confirm repeated predictions and display the letter clearly.

MediaPipe handles localization; the CNN classifies the resulting crop. These are separate models.

03 / SYSTEM ARCHITECTURE

Designed for clarity and responsiveness.

Inference is performed locally by Python; the browser provides the user interface and the camera stream. Heavy recognition work does not block video rendering.

> system.componentsIMPLEMENTATION OVERVIEW
01 / UI

Streamlit

Editorial-style interface with image upload, camera snapshot, live video and model evaluation.

02 / HAND LOCALIZATION

MediaPipe

Finds hand landmarks; dynamically crops only the relevant region.

03 / CLASSIFIER

PyTorch CNN

Predicts among 24 static ASL classes using a compact grayscale input.

04 / VIDEO PIPELINE

WebRTC + OpenCV

Maintains responsive video with an independent inference worker and latest-frame processing.

04 / INTERACTIVE EXPERIENCE

One application. Three ways to explore it.

The interface supports photo upload, single camera captures and a live webcam. Each screenshot below comes from the working application.

05 / MODEL EVALUATION

Strong benchmark results, with a clear boundary.

Evaluation uses 7,172 samples from the Sign Language MNIST validation dataset. It measures performance on that dataset, not on uncontrolled camera scenes.

ACCURACY98.19%

Overall classification accuracy on the validation split.

MACRO-F10.9804

Class-balanced F1 summary across 24 letters.

CLASS COVERAGE24 / 26

Static letters; J and Z involve movement.

Evaluation scope: A high validation score does not establish equivalent accuracy on real webcam imagery. Predictions are educational, and the displayed softmax score is not calibrated confidence.
06 / ENGINEERING

What I implemented beyond a notebook.

The project evolved from educational CNN exercises into a reproducible, interactive demonstration with additional computer-vision and inference components.

01 / MODEL

Checkpoint recovery

Recovered CNN weights in a reusable PyTorch model definition and evaluated the classifier.

02 / PREPROCESSING

Hand-first inference

Added landmark detection, square crops and 28 × 28 grayscale preprocessing.

03 / REAL TIME

Non-blocking webcam

Moved MediaPipe and CNN work out of the video callback and discarded stale frames.

04 / OUTPUT

Temporal smoothing

Used repeated predictions and a minimum hold time to keep letters readable.

05 / EXPLAINABILITY

Grad-CAM

Added visual maps of influential regions for predictions on static hand crops.

06 / QUALITY

Tests and metrics

Separated application and model modules with automated tests and validation reporting.

07 / PERSONAL RETROSPECTIVE

What I learned building ASL Vision.

The biggest lesson was understanding the gap between a model that scores well on a curated dataset and a complete system that performs usefully for someone in front of a camera.

BEYOND VALIDATION ACCURACY

Good model metrics are a starting point, not a finished user experience. This project pushed me to reason about domain shift, latency, explainability and the interaction between several models in one application.

01 / COMPUTER VISION

Detection and classification solve different problems

MediaPipe localizes the hand; the CNN identifies its static configuration. Separating these tasks improved framing and made failures easier to diagnose.

MediaPipeLandmarksCropping
02 / REAL-TIME ENGINEERING

Throughput is not the same as responsiveness

Processing every frame synchronously made the video unusable. A latest-frame queue and background inference preserved a fluid camera display.

WebRTCOpenCVThreading
03 / HUMAN-CENTERED AI

Correct-looking output can still be unreadable

Rapid letter changes were difficult to follow. Confirmation rules and display hold time improved the practical clarity of predictions.

Temporal smoothingUX
04 / EVALUATION

Benchmarks and real scenes need different evidence

Strong Sign Language MNIST results do not automatically transfer to different hands, backgrounds and lighting. Grad-CAM and explicit scope statements help interpret—not overstate—the system.

Domain shiftGrad-CAMValidation
KEY TAKEAWAY

A usable AI product needs more than an accurate classifier.

ASL Vision connected deep learning with image preprocessing, streaming performance and user-facing explanations. The result is a more complete demonstration of how to turn an academic model into an interactive application.

08 / RESPONSIBLE SCOPE

A recognition demo, not a language translator.

ASL is a complete language with grammar, spatial structure and movement. This system recognizes a subset of static manual alphabet handshapes; it cannot translate full ASL sentences or recognize dynamic J and Z.

WHAT IS SUPPORTED

Educational demonstration

Static handshape classification, hand tracking, photo and webcam inference, baseline evaluation and Grad-CAM visualization.

WHAT REMAINS OPEN

Real-world generalization

Independent evaluation on diverse real-camera images, backgrounds, lighting and signers; calibrated confidence; and dynamic gesture understanding.

Project origin: The original CNN work began with NVIDIA Deep Learning Institute educational material. The interactive application, hand localization, webcam optimization, smoothing and presentation are project extensions. Dataset and checkpoint reuse remain subject to their applicable terms.
OPEN PROJECT

Explore the code and methodology.

Source modules, tests, reproducibility instructions and model evaluation are documented on GitHub.

GitHub repository ↗