Why gestures
I got into computer vision because I wanted to see how code could change something in the real world, not just on a screen. Gesture recognition is a very direct version of that idea: a person moves a hand, and a machine understands and responds.
So during my undergraduate years at Xidian University, I joined Prof. Yunan Li’s group and worked on a national-level undergraduate innovation project called Smart with Hands (智随手动), together with my teammate Yang Jiawen. The project ran from November 2022 to June 2024.
Why the dark
Gesture recognition already works quite well in good conditions. But most gesture datasets are recorded in daylight, often with people sitting in front of a laptop camera. Real interaction doesn’t always happen like that. It happens in dim corridors, outdoors at night, or anywhere the lights are off. In the dark, an ordinary camera sees very little: the image becomes noisy, details disappear, and models trained on bright videos quickly lose their footing.
Our goal was to make gesture recognition hold up in exactly those conditions, and to build a lightweight system that could still work in real time.
Learning a theory I didn’t understand yet
The core of our approach was the information bottleneck: the idea that a model should keep only the information that matters for the task and compress away the rest. In a dark, noisy video, most of what the camera records is noise, so this idea fit the problem well. We built on a variational version of it, using distillation so that different kinds of input could help each other.
My part was to work through the derivation and verification of this variational distillation information bottleneck, and then to build and train the recognition models. I started by reading the papers the method came from, but reading alone didn’t get me all the way there. It was Prof. Li’s explanations that helped me connect the math to what the model was actually doing. Prof. Li was a wonderful teacher.
Collecting gestures in the dark
A model for the dark needs data from the dark, and there was very little of it. So, under the guidance of Prof. Li and the senior students in the lab, I took part in collecting a new dataset across many locations on our campus.
We recorded at night: in car parks, corridors and small groves outside, and in halls, stairwells and laboratories inside. A Kinect camera captured color and depth, and a separate infrared camera recorded at the same time, so every gesture was saved in three versions. Participants performed ten gestures chosen for real situations — such as forward, stop, turn left, look for, quiet and dangerous. Before each session, we showed everyone the standard form of each gesture, and afterwards we checked the recordings. Later came the slow part: marking by hand where every single gesture began and ended.
The darkness was real. In many of the raw color videos, it was hard to see the gesture at all with the naked eye. That was the point of the dataset: the color stream alone was not enough, but depth and infrared still captured the person clearly.
The dataset, MGR-Dark, was later published by the lab at ACM Multimedia 2024 (MGR-Dark: A Large Multimodal Video Dataset and RGB-IR Benchmark for Gesture Recognition in Darkness).
From zero to one
My teammate Yang Jiawen built the real-time gesture recognition system, the part people could actually see working. Between us, we carried the project from an idea to a finished system and a completed national project, reporting our progress to Prof. Li every month along the way. It was my first lab research project, and the first time I followed one through from beginning to end.
What stayed with me
When the project ended, I realized the question I cared about most was not a particular model. It was about information itself: what a model really relies on when it makes a decision, and how it can still recognize things when the environment works against it.
In our experiments, infrared gave the most stable signal in the dark, because it doesn’t depend on visible light. That idea — look beyond what an ordinary camera sees, and keep only what helps — indirectly shaped what I chose to study next. In my master’s research on coastal marine litter detection, I found myself asking a very similar question in a very different place, and near-infrared turned out to matter there too.
I’m grateful to Prof. Yunan Li, to the senior students who guided our data collection, to my teammate Yang Jiawen, and to everyone who stood in the dark and performed gestures for our cameras.
Looking back, the project taught me that seeing is not only a matter of having more light. Even in the darkest videos, the gesture was still there. It just had to be separated from everything that didn’t matter. I think that is a good way to approach research as well: when a problem is noisy and unclear, look for the part that really carries the meaning, and let the rest go. The world rarely offers ideal conditions, and I hope to keep building things that can still see what matters when the lights are low.