Trinity School of Engineering Professor Ciaran Simms Presents: KineMo | Athlete Movement Capture Directly from Video

Introduction

As Professor in Biomechanics at Trinity College Dublin doing research related to injury biomechanics, a focus has been on how equipment design and impact configurations relate to injury outcomes in pedestrian and cyclist collisions, but also in sports. I have collaborated since 2016 with Leinster Rugby and their lead physiotherapist (Garreth Farrell), particularly relating to head injuries in rugby. Together with Leinster Rugby we combined manual video analysis of rugby collisions with biomechanical models and staged impact testing to better understand how tackle height influences head and neck kinematics, particularly through the PhD work of Greg Tierney, now at the University of Ulster. In around 2018 we started thinking about automated video analysis of sporting collisions, and it became clear in collaboration with Prof Aljosa Smolic and Dr Richard Blythman (then both at Trinity College Dublin) that technological advances in human pose estimation from single camera video could find very strong applications in sports biomechanics, particularly if combined with an understanding of the underlying physical processes of human movement. This was the start of the KineMo journey.

How Athlete Movement Is Currently Assessed & Its Challenges

During discussions on rugby-related injuries we learned that high-end clubs like Leinster Rugby send some of their athletes to motion analysis clinics (where calibrated cameras and force-plates are used by trained experts with proprietary systems such as VICON) to test return-to-movement competency following injury, particularly in the case of lower limb injuries. Some sports clubs have these facilities in-house, using either marker-based or markerless motion capture. These assessments provide valuable quantification of dynamic movements, which would be impractical for coaches and physiotherapists to track by eye. However, these assessments are also expensive and time-consuming and are therefore sparingly used, and most athletes do not benefit from this technology. Below elite levels it is generally not practical to consider assessments using these technologies, yet the duty of care remains. As a result, there is almost no objective assessment of athlete movement competency, with the overwhelming majority of assessments being the visual (by-eye) assessments that coaches and physios make. However, a recent study found the accuracy of visual assessments of human joint angles even in slow movements to be limited to around 12°. For dynamic and whole-body movements there is no real objectivity in by-eye assessments, except partially via slow-motion replay.

Value Of Whole Body Assessment

The importance of movement competency is well understood, as it is a fundamental step on the path of safer athlete development, injury return, understanding the potential risk of future injury burden and sports-specific athletic skills progression. However, assessing human movement is also complex. For example, a coach might focus on knee flexion and torso tilt in a squatting exercise, as well as knee varus/valgus and rib-flare when squatting with weights. This means there is substantial value in a whole-body assessment of human form (or pose) during exercise completion, especially if the tracking can be done with sufficient anatomical detail and if it can occur ‘on demand’ over regular time-points relevant to the athlete’s development or return from injury.

What Does KineMo Address?

There is therefore a significant need for a simple and quick method to accurately and consistently measure and track athletes (or indeed any individual) completing common exercises, and to record how their movement form changes over time as movement competency changes. Further, there is the need to present the resulting data to sports professionals and their athletes in an actionable way. This has been the goal of KineMo. Over the last number of years I have worked with a team of scientists, technologists, sports physios, coaches and business advisors to develop the underlying technology to achieve this, and we are now working to transition the accompanying user platform into a commercially available product through our recently formed startup. KineMo initially received research support from Science Foundation Ireland and significant research and commercialisation backing from Enterprise Ireland, for which we as a team are very grateful.

What Areas of Assessment Is KineMo Used For?

KineMo is currently used to track athlete development and/or return from injury. It is best used by first establishing a baseline recording for a cohort of athletes, against which any changes can be compared, either on a cohort or on an individual basis. Baseline recordings can be recorded with KineMo or potentially other platforms like VICON:

• Injured athlete: KineMo is used to track return to movement competency over time by comparing metrics such as knee flexion and torso angle in a squat, in accordance with the recovery programme set by their physiotherapist.

• Developing athlete: KineMo is used to track whole-body form during dynamic movements like countermovement jumps and drop jumps. Where existing forceplate measures only assess performance, KineMo tracks whole-body movement form and permits an assessment of movement competency. In some recent studies we undertook, we achieved some very interesting results combining KineMo’s whole-body form with forceplate measurements – performance and form combined.

Because KineMo is quick to use, recording athlete movements can be done at multiple time-points, giving athletes objective goals to aim for. I believe this facility will lead to a general increase in literacy around human movement: for example, if I can easily record my trunk angle during a squat, I can readily check if I incorrectly hinge at the hips halfway through my squat because my lower back muscles are not strong enough. Similarly, if my knees flex too much in a dropjump landing, I can see this directly in the KineMo outputs. Or I can measure my knee varus/valgus and then seek training programmes to correct any excessive movements. KineMo’s ability to present not only these key actionable datapoints and metrics by exercise but also represent and interact with this information in a 3D visualisation provides significant value for users.

How Does KineMo Work?

KineMo uses a single camera view video recording of exercises to measure the relevant body angles during that exercise. Using a single camera view presents challenges compared to multiple calibrated cameras, but the calibration step needed for multiple cameras remains cumbersome and requires technical expertise and is therefore challenging in busy sports environments. Further, although there are depth-sensing features on some cameras, KineMo does not need this and operates using regular ubiquitous RGB video. While depth sensors provide an additional data dimension, their accuracy has not so far led to major benefits in human pose estimation during exercising, probably because an estimate of depth is not a major benefit in most poses, especially when the depth-measuring accuracy is low. Of course, as computer vision technology continues to evolve, in future this may add some additional value to our computations.

Reconstruction process (see Figure 1): A video clip of the exercise is recorded within a range of allowable positions which maintains a full view of the participant during the exercise with minimised occlusion (Figure 1A). The main body “key points” such as the ankle, knees, hips, spine, neck, etc are first identified in the image plane (pixel coordinates, Figure 1B). Our machine learning models are trained on our in-house datasets covering a wide range of exercises and movements with many participants. The models then infer the three-dimensional positions of the body key points at each time-point from the two-dimensional pixel coordinates. This is used to animate an avatar and compute metrics relevant to physios and coaches, along with aggregated scores, providing actionable data to design/adjust training, development or rehabilitation programmes. These are presented in the KineMo analysis app (Figure 1C). The score provides a quick overview of changes between measurement time-points to identify high level issues, while the detailed metrics drill into the specific movements associated with these issues. The scores can also provide athlete insight, gamification and motivation over time.

Scientific Validation

Validation is critical to the adoption and use of new technologies like KineMo and in a recent journal paper, we set out a detailed skeletal model and showed the accuracy that can be achieved using our approach. From this we set out our vision for the requirements in using single camera video to extract actionable data on exercise proficiency for physios and coaches. These are:

1. A suitable single camera recording;
2. A sufficiently detailed biomechanical model;
3. A set of metrics which can be computed from the biomechanical model;
4. A 2D key point estimator;
5. A set of ground truth training data linking 2D pixel coordinates to 3D spatial data;
6. A model for “lifting” from 2D pixel coordinates to 3D spatial coordinates and hence computing the exercise metrics;
7. A user-friendly app and platform to record and upload athlete videos and present tracked changes over time for metrics of interest, both at cohort and individual level.

This is just the starting point, we have significant development underway on our biomechanical model, our metrics, accuracy, consistency and breadth of movements, with further studies and papers in the pipeline.

Early Large-Scale Studies

In 2023 we carried out two large scale studies with rugby and Gaelic football athletes at multiple time points, recording them performing squats, jumps and lifts. These large-scale studies showed that KineMo could track whole cohorts for key exercises within an hour with a single mobile device, opening the door to large scale implementation through our commercial platform. We also tested the accuracy of the KineMo reconstructions in a sub-study where we used VICON marker-based data as a ground truth. We aim to submit the results of these collaborative case studies for publication shortly.

Alternative approaches

Most problems can be potentially addressed by various technologies, and the problem of athlete movement quantification is no different with a number of approaches to tracking athlete kinematics. These include using marker-based systems, multiple cameras, depth-sensing cameras and single/multiple wearable sensors. In a very large sports and commercial market, several of these will likely gain long-term traction. It seems clear that successful solutions will need to balance accuracy, consistency, speed and ease-of-use with suitable models and relevant training data. My perspective is also that there will be significant value in the combination of data across some of these providers for end-users and clients.

The KineMo journey

In summer and autumn 2023 we went through a user-engagement process with professionals from several sports, with our internal team and external partners, we designed and developed the first version of the KineMo platform and mobile application. Through the first half of 2024 we have been road testing this with a group of key sports and performance organisations while we also make the transition from university research to a startup company. The second half of 2024 already promises to be a busy and exciting time as we rollout our user testing to a waitlist of sports, performance organisations and more. If you are a professional coach, physio, sport or performance organisation who is interested in what we are doing at KineMo please get in touch: we would be happy to speak with you.

The Future

Despite starting my research with Garreth Farrell, Aljosa Smolic and others in this area already six years ago, the KineMo journey still feels like a beginning, and it is truly a team effort. Our business is led by Leo Peyton and he has broad ambitions for our current focus and future uses of KineMo including sports skills progression, gamification, MR/VR visualisation, and several business to consumer use cases. Des Ryan and Lara Coyne have brought a depth of knowledge to the team and much insight on the importance of movement competency in youth athlete development and they have also brought athlete maturation clearly into focus. Clara Mercadal leads our technical team (currently Jorge Gonzales, Robert McCarthy and Augustin Plaza Reino, previously also including Molly Boyne, Chao Liu, Manan Saxena, Stephen Smyth and Andrew Dai), and David Jones brings long expertise in software architecture and implementation, and collectively they have delivered so much already.

Personally, I am fascinated by the potential that accompanies the ability to extract usable measures of human movement directly from video and developing what Garreth Farrell calls ‘A new language of Kinematics’. In one of my favourite poems (under Ben Bulben), WB Yeats proclaims that “measurement began our might”. Another recent quote is that “Without assessment, training is guessing”. To me the KineMo concept chimes with both of these claims. I am convinced that the ability to quantify human movement proficiency will bring many benefits in sport and general human health and also in injury biomechanics, and I am very pleased and also excited to be part of this team and that challenging but rewarding journey.

(a)

(b)

(c)

Figure 1: The KineMo reconstruction pipeline, with adapted images from the app: (A): single camera view video of exercise movement; (B) 2D keypoint estimation in pixel coordinates (manually overlaid here, as not shown in app); (C) KineMo app screenshot with original video, 3D avatar, detailed metric time-histories and scores.

This content also appears in Engineers Ireland journal link (June ’24):
KineMo:How to capture athlete movement directly from video - Engineers Ireland