Abstract

Understanding the fine-grained articulation of human hands is critical in high-stakes settings such as robot-assisted surgery, chip manufacturing, and AR/VR-based human-AI interaction. Despite achieving near-human performance on general vision-language benchmarks, current vision-language models (VLMs) struggle with fine-grained spatial reasoning, especially in interpreting complex and articulated hand poses. We introduce HandVQA, a large-scale diagnostic benchmark designed to evaluate VLMs' understanding of detailed hand anatomy through visual question answering. Built upon high-quality 3D hand datasets (FreiHAND, InterHand2.6M, FPHA), our benchmark includes over 1.6M controlled multiple-choice questions that probe spatial relationships between hand joints, such as angles, distances, and relative positions. We evaluate several state-of-the-art VLMs (LLaVA, DeepSeek and Qwen-VL) in both base and fine-tuned settings, using lightweight fine-tuning via LoRA. Our findings reveal systematic limitations in current models, including hallucinated finger parts, incorrect geometric interpretations, and poor generalization. HandVQA not only exposes these critical reasoning gaps but provides a validated path to improvement. We demonstrate that the 3D-grounded spatial knowledge learned from our benchmark transfers in a zero-shot setting, significantly improving accuracy of model on novel downstream tasks like hand gesture recognition (+10.33%) and hand-object interaction (+2.63%).

Understanding a human hand gesture requires more than recognizing fingers. An AI model must also make sense of how those fingers bend, how joints related to one another, and how the entire hand changes with viewpoint. Current vision-language models remain surprisingly weak at this kind of finger-grained reasoning. 

A research team, led by Professor Seungryul Baek of the Graduate School of Artificial Intelligence at UNIST has developed HandVQA, a new benchmark that tests this ability in detail. Drawing on 3D hand data, the benchmark contains more than 1.6 million questions about joint angles, distances, and relative positions. It also gives researchers a way to improve the skills it is designed to measure.

Despite strong performance on general image-and-language tasks, vision-language models can struggle when spatial differences become subtle. Existing benchmarks rarely examine hand anatomy at the level of individual joints, making these weaknesses difficult to measure.

HandVQA fills that gap with questions generated from hand images and precise 3D joint coordinates. For each image, the benchmark asks 25 questions about properties such as finger flexion, the distance between joints, and whether one joint is above, below, in front of, or behind another.

The results showed just how much current models miss. Without specialized training, several leading vision-language models performed near chance on some spatial questions and had particular difficulty judging distances between joints. They also made geometric errors and, in some cases, referred to finger parts that did not exist in the image.

But the same benchmark that exposed these weaknesses also helped correct them. After LLaVA was fine-tuned with HandVQA, its accuracy on questions about joint distances increased from 16.20% to 90.79%.

The benefits also carried over to tasks outside the benchmark. Qwen-VL, after learning from HandVQA, improved on two tasks it had not been directly trained for: hand gesture recognition by 10.33 percentage points and hand-object interaction by 2.63 percentage points. This transfer suggests that the model learned a broader understanding of hand geometry rather than simply becoming better at answering HandVQA questions.

“HandVQA not only helped the model answer questions it had previously struggled with, but also improved its ability to handle new tasks,” said MD Khalequzzaman Chowdhury Sayem, first author of the study. “The spatial knowledge learned from the benchmark transferred to other hand-related tasks without requiring additional task-specific training.”

Professor Baek added, “Accurately understanding hand pose is important in applications where even small errors can matter, from robotic manipulation and AR/VR interfaces to assistive technologies. HandVQA helps identify where current models fall short and provides a way to improve those capabilities.”

Their findings have been accepted to the  IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2026, one of the leading international conferences in computer vision. The study was supported by the National Research Foundation of Korea (NRF) through the Mid-Career Researcher Program and the Basic Science Research Program, along with programs administered by the Institute for Information communication Technology Planning and Evaluation (IITP), including the AI Star Fellowship, AI Graduate School, and the LG AI STAR Talent Development Program for Leading Large-Scale Generative AI Models in the Physical AI Domain programs.

Journal Reference

MD Khalequzzaman Chowdhury Sayem, Mubarrat Tajoar Chowdhury,  et al. , “HandVQA: Diagnosing and Improving Fine-Grained Spatial Reasoning about Hands in Vision-Language Models,”  CVPR '26, (2026).