Hand Tracking and Virtual Mouse using Computer Vision

Track hand movements and create a virtual mouse using computer vision.

2 min readOct 23, 2021


Photo by Greg Rakozy on Unsplash


In previous articles, I went through with you the latest development in computer vision and optical character recognition using machine learning. In this article, let’s go through interesting stuff that we can build.

This Youtube channel is a good start for anyone who wants to learn the practical usage of computer vision.

The sample code I demonstrated here is adapted from the channel with the following improvements.

  • For GUI automation, instead of using autopy, I refactored the code to use PyAutoGUI.
  • Instead of an infinite loop, allow the user to type “q” to quit the application and then perform a proper resource clean-up.
  • Better exception handling when the movement is not detected.

Hand Tracking

Using the open-source Google MediaPipe library, we can track hand movements easily.

  • MediaPipe provides 21 landmarks or 3D coordinates we can use to track and check any particular motion easily, e.g. coordinate 4, 8, 12, 16, 20 are the fingertips.
  • By checking the particular landmarks, and measuring the distance, we can detect a particular movement, e.g. fingers up.

Virtual Mouse

Now let’s create a virtual mouse.




Software engineer, Data Science and ML practitioner.