Low-Cost and Unobtrusive Human Motion Analysis

For the evaluation of human motion, high quality analysis is usually performed in specialized laboratories using high-end multi-camera motion capturing systems. To obtain accurate results, subjects must be prepared with reflective markers, which is a very expensive and time-consuming process and requires presence in the laboratory. In the following we present related work on motion tracking systems and algorithms for cost-effective 2D/3D pose estimation, where we also use some of the algorithms to record our own data sets.

2D/3D Human Pose Estimation from RGB Images

The task of human pose estimation is to estimate the position of certain joints of the human body in either 2D or 3D coordinates for one or multiple given input images. Convolutional Neural Networks (CNN) enjoyed huge success in this computer vision task. These networks are generally well suited for image processing tasks because the architecture of the models is designed to work with data arranged in grid structures. One of the first papers (Toshev et al. [1]) in the field of  pose estimation with CNNs aimed at regressing the joint locations as \((x,y)\) pixel coordinates directly for a given image, resulting in noisy data. A better strategy for estimating joint locations in images than the simple regression of pixel coordinates is to predict a heatmap where higher values indicate higher confidence of the joint location. In two landmark papers, Stacked Hourglass [2] and Simple Baseline [3], this regression method was used, each with a different arrangement of convolutional layers and optimization strategies but both achieving state-of-the-art performance at the time. 

Another commonly used system called OpenPose [7] is a real-time multi-person 2D and 3D pose estimation model that consists of a two-step approach that first identifies key points of one or more persons in a given image and then selects all joints belonging to the same person from the set of all joints found. Figure 1 shows pictures taken from an own training process of a 2D human pose estimation model (implementation of Simple Baseline) utilizing data augmentation techniques by rotating and scaling the images in the training set.

^{Figure 1: Training Process of a Human Pose Estimation Model}

Validation of Pose Estimation Quality of a novel RGB-D Camera Device

As mentioned above, the Microsoft Kinect cameras (or similar depth camera models) are inexpensive and unobtrusive devices, but they also lack accuracy compared to high-end motion capturing systems. In the past, the first and second generation of the Microsoft Kinect camera have been evaluated for a specific application in comparison to a gold standard system to determine whether the Kinect is suitable for a particular physical assessment task. One study (Capecci et al. [5]) found that the Kinect camera could be used to evaluate physical activity in a rehabilitation scenario. Another study (Galna et al. [6]) evaluated how the Kinect sensor could be used to evaluate typical low-back pain exercises. The subject of the evaluation is often the spatial agreement between joints (using certain distance measures) or the range of motion (RoM) of certain limbs or, in gait analysis, the comparison of discrete spatio-temporal gait parameters. The aim of this current project is to investigate the tracking quality of the latest Azure Kinect camera (as mentioned above) in relation to a specific use case of physical activity. The high-end motion detection system Vicon (Vicon, UK) with a full-body 32 reflective marker setup will be used to compare the resulting kinematic data.

^{Figure 2: Application of Deep Learning Model to Measure Gait on a Treadmill}

1. Toshev, A.; Szegedy, C.; DeepPose: Human Pose Estimation via Deep Neural Networks, arXiv:1312.4659, doi: 10.1109/CVPR.2014.214

2. Newell A., Yang K., Deng J. (2016) Stacked Hourglass Networks for Human Pose Estimation. In: Leibe B., Matas J., Sebe N., Welling M. (eds) Computer Vision – ECCV 2016. ECCV 2016. Lecture Notes in Computer Science, vol 9912. Springer, Cham

3. J. Martinez, R. Hossain, J. Romero and J. J. Little, "A Simple Yet Effective Baseline for 3d Human Pose Estimation," 2017 IEEE International Conference on Computer Vision (ICCV), Venice, 2017, pp. 2659-2668.

4. Shotton, J.; Fitzgibbon, A.; Cook, M.; Sharp, T.; Finocchio, M.; Moore, R.; Kipman, A.; Blake, A. Real-time Human Pose Recognition in Parts from Single Depth Images 2011. pp. 1297–1304. doi:10.1109/CVPR.2011.5995316.
5. Capecci, M.; Ceravolo, M.G.; Ferracuti, F.; Iarlori, S.; Longhi, S.; Romeo, L.; Russi, S.N.; Verdini, F. Accuracy Evaluation of the Kinect v2 Sensor during Dynamic Movements in a Rehabilitation Scenario 2016. pp. 5409–5412
6. Galna, B.; Barry, G.; Jackson, D.; Mhiripiri, D.; Olivier, P.; Rochester, L. Accuracy of the Microsoft Kinect Sensor for Measuring Movement in People with Parkinson’s Disease. Gait and Posture 2014, 39, 1062-1068. doi:10.1016/j.gaitpost.2014.01.008.

7. Cao, Z.; Hidalgo Martinez, G.; Simon, T.; Wei S.; Sheikh, Y.; OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017