As people live longer, the incidence and severity of health problems increases, placing strain on healthcare systems. There is an urgent need for resource-wise approaches to healthcare. We present a system built using open-source tools that allows health and functional capacity data to be collected remotely. The app records performance on functional tests using the phone’s built-in camera and provides users with immediate feedback. Pose estimation is used to detect the user in the video. The x, y coordinates of key body landmarks are then used to compute further metrics such as joint angles and repetition durations. In a proof-of-concept study, we collected data from 13 patients who had recently undergone knee ligament or knee replacement surgery. Patients performed the sit-to-stand test twice, with an average difference in test duration of 1.12 s (range: 1.16–3.2 s). Y-coordinate locations allowed us to automatically identify repetition start and end times, while x, y coordinates were used to compute joint angles, a common rehabilitation outcome variable. Mean difference in repetition duration was 0.1 s (range: −0.4–0.4 s) between trials 1 and 2. Bland-Altman plots confirmed general test-retest consistency within participants. We present a mobile app that enables functional tests to be performed remotely and without supervision. We also demonstrate real-world feasibility, including the ability to automate the entire process, from testing to analysis and the provision of real-time feedback. This approach is scalable, and could form part of national health strategies, allowing healthcare providers to minimise the need for in-person appointments whilst yielding cost savings.
1 Introduction
The global population continues to grow, and with advances in living standards, life expectancy has gradually increased in most nations over recent decades. As people live longer, the incidence and severity of health problems increases, placing strain on national health systems. There is an urgent need for resource-wise approaches to healthcare that free up medical staff to focus on life-threatening cases, whilst also minimising wait times for patients with less critical needs. The recent COVID pandemic also highlights the need for remote solutions that reduce the need to get in-person access to a healthcare professional [1].
Recent advances in technology, particularly in the field of AI, have made the prospect of remote healthcare solutions feasible. For example, it is now possible to monitor heart rate dynamics, blood pressure and sleep behaviour via smartphone applications [2, 3]. However, these applications tend to be fragmentary and narrow in scope, focusing on a single variable or function, and thus only giving a limited window into a person’s health status. Moreover, existing solutions are almost always proprietary, making it difficult to scale up their use or add new functionality.
Anzeige
Open-source tools would give healthcare providers more opportunities to monitor patient function, whilst also giving patients more freedom and flexibility, by allowing new tools and functionality to be developed based on patient needs. Moreover, in line with the recent rise in citizen science applications [4], mobile apps enable healthcare interventions to reach more people, including those in poorer or more remote regions. This in turn allows data to be collected from larger and more diverse populations. The aims of this paper are: 1) to present a system built using open-source tools that allows health and functional capacity data to be collected remotely with minimal user input, and that also provides patients with immediate test feedback. 2) To demonstrate a practical use case of this approach in a hospital setting.
State-of-the-Art.
Several studies have presented methods for remote testing of functional performance in different clinical groups. For example, Brooks [5] developed a self-administered 6-min walking test mobile application (SA-6MWTapp) for independent use at home, and Hwang [6] did similar work using video conferencing to supervise the tests. Boswell [7] created a smartphone app to examine sit-to-stand test performance remotely. As well as the sit-to-stand test, the timed up and go test and step tests can be performed at home by patients with chronic respiratory disease [8‐10]. Netz [11] used smartphone accelerometer data to remotely examine balance, strength and flexibility. Hellsten [12] summarised the potential of markerless AI algorithms for remote monitoring, and these applications have proliferated in recent years [13‐15]. However, existing applications for remote testing often focus on a single task, limiting their practical value as monitoring tools. Moreover, data are typically post-processed, so participants may not receive performance feedback, and interactive applications (e.g. gamification) are not possible.
2 Methods
We have developed a smartphone app that records individuals performing various functional tests using the phone’s built-in camera and provides users with immediate feedback. The app currently offers the following functionality: individual sign-in using a QR code; access to different research projects via the app home screen; the ability to perform functional exercise tests fully remotely, and to receive instant feedback about performance. Here we first describe the technical implementation of the app, and then report results from a study performed in a clinical setting.
2.1 Approach
The first key step is to detect a person from a video automatically and in real-time. This can be achieved using a number of existing open-source pose estimation algorithms such as OpenPose [16]. Since we require real-time tracking on current smartphones, we instead use MoveNet [17], which runs with reasonable accuracy in real-time (3–6 Hz; see below). The algorithm detects x, y coordinates of key body landmarks in an image, and the coordinates are used to compute further metrics such as joint angles, distances, rep times/durations etc.
Anzeige
2.2 Technical Details
A schematic of how the app works is shown in Fig. 1. The mobile app is interfaced with open-source software developed at the University of Jyväskylä called Vasara, which is a Hyperautomation platform. From the user’s perspective, a test session begins when the user scans a QR code using the phone’s camera (QR codes can be supplied to study participants by email, for example). After scanning the code, the user enters some basic demographic data such as their age and body mass, and can then proceed to performing a functional test. In the schematic shown in Fig. 1, there are 2 different tests, but in theory we can add as many as desired. When a test is selected, the user is first presented with an instructional video demonstrating how the test is done. After this, they are given audible instructions via the app, for example “take a seat within view of the camera”. Pose estimation is used to check that all body keypoints that need to be visualized are in view before the test can start. If, for example, part of the participant’s body is obscured or outside of the camera view, the user is instructed to move forward/backward etc. Once the test is initiated, pose estimation detects the key body landmarks (see right side of Fig. 1), and the x, y coordinates of these points are saved in JSON format. Once the test is finished, the video is deleted immediately, thereby minimising the potential for applicants to be identifiable from the data, as well as minimising the data footprint of the app.
Fig. 1.
Schematic of the mobile application. The screenshot to the right is taken from the actual app and shows the detected body landmarks overlaid on the participant in real-time.
×
2.3 Use Case – Functional Testing of Patients After Knee Surgery
In this proof-of-concept study, we collected data from patients who had recently undergone knee ligament reconstruction or knee replacement surgery (n = 13). As part of their regular appointment with a physiotherapist, patients were offered the opportunity to participate in this study, which involved performing brief functional tests, and repeating each test after a short rest period.
In this study we used the sit-to-stand test as an example. The sit-to-stand test involves the functional movement of rising from a seated position and then sitting again, which is repeated several times. The test is often used as a proxy measure of lower limb muscle strength [18] and is suitable for a wide range of populations, including hip and knee osteoarthritis, and adults of different ages [19]. The test is also well suited to remote applications because it is quick and easy to administer and interpret, whilst requiring minimal equipment and space. Importantly, there is also no need to calibrate the camera since output variables are based on angles rather than distances.
Patients who agreed to participate provided written informed consent before testing. For the sit-to-stand test, participants were advised (via the app) to start the test by sitting in the chair. Once they were seated within view of the camera, the following instruction was: “when you are ready to start, raise your hand”. This motion was detected with pose estimation, and a countdown (5 to 1) was initiated. Participants were instructed to perform 5 sit-to-stand repetitions to complete the test. Upon completion, the maximum knee joint angle and test time were displayed on the screen. In this study, each participant repeated the test approximately 2 min after the first test, but in theory, tests can be repeated at any interval, for example as part of a long-term intervention protocol performed over several months or even years.
3 Results and Interpretation
The average time difference between the duration of the first and second trials- i.e. the time taken to perform 5 sit-to-stand repetitions- was 1.12 s (range: 1.16–3.2 s). Figure 2A shows examples of y-coordinate trajectories for the right hip during the test for 4 different participants. Most participants performed the movement with high repeatability, although some performed one test clearly faster than the other, likely due to the learning effect associated with an unfamiliar task. The y-coordinates of the key body parts can be used for simple analyses such as identifying the start and end of each rep, or the maximum/minimum height of a body part (e.g. using peak finding or gradient-based techniques), as shown in Fig. 2B. By combining x, y coordinate data with simple mathematics, we can also compute joint angles (Fig. 2C), which are a common outcome variable in rehabilitation and sports movement analysis. Similarly, by differentiating over time, limb velocities can be computed, which could be useful for biofeedback applications where patients are given a target movement profile to follow.
Fig. 2.
A: Example of test-retest results from four different participants. Each panel shows the y-coordinate trajectories over time for the right hip, one panel per participant. B: Using peak detection to identify the transitions between ascending and descending motion. C: The corresponding hip joint angle based on the data in B.
×
As the segregation of individual repetitions within a trial can be automated, it is also easily possible to perform repetition-level analyses. For example, the mean difference in repetition duration- i.e. the time taken to stand up fully and return to the seated position once- was 0.1 s (range: −0.4–0.4 s) between trials 1 and 2. Data for all repetitions by all participants are compared in the Bland-Altman plot [20] in Fig. 3.
Fig. 3.
Bland-Altman plot comparing the individual repetition durations between trials 1 and 2 for all participants. The dotted red line denotes the mean difference between trials 1 and 2 (bias), and the dashed black lines indicate the limits of agreement.
×
From this figure it is clear that the majority of datapoints fall within the limits of agreement, demonstrating general consistency within participants, i.e. between trials 1 and 2. Although not performed here, the analysis can be further enhanced by comparing the duration of the ascending and descending phases of a repetition, which can give important information about muscle function.
4 Discussion and Future Perspectives
Around 85% of the global population owns a smartphone, which are typically equipped with a camera and movement sensors. Phone literacy is also extremely high and improving continually, including among older adults [21] and in poorer countries [22]. Thus, smartphones have huge potential as health monitoring tools in large populations, enabling the remote completion of functional tests from anywhere. We present a mobile app that enables functional tests to be performed remotely and without supervision. We also present real-world evidence of the feasibility of this approach, including the ability to automate the entire process, from testing to analysis and the provision of real-time feedback. Using computer vision to detect people in images, we can quantify movement metrics such as the time taken to complete a task, joint angles, or limb velocities, with levels of accuracy that generally match those of a human physiotherapist [13, 14]. Our mobile app is also sufficiently user-friendly to be used in a real clinical environment by patients and/or medical staff.
A limitation of our approach is that only angle and repetition-related metrics can easily be extracted. In clinical settings it is often also desirable to compute distances during a test (e.g. maximum distance between left and right ankles). Such metrics require camera calibration, introducing possible scaling errors [14]. Nonetheless, our approach offers several advantages. We have tested our app on 4 different smartphones and as the underlying pose estimation algorithms are quite robust to image quality and resolution, the main performance difference between phones is inference speed, which ranged between 10–30 frames per second, which is satisfactory for clinical applications. We also minimise the risk of GDPR issues as our app performs real-time, on-device analysis and then discards the video, thus also minimising data storage needs. In future work we will expand the app’s functionality in several ways. Most importantly, we will develop strong protocols for handling sensitive data. We will also add more functional tests (e.g. range of motion, balance) and tailor the specific feedback that is given to users immediately after each test. Moreover, we will include the ability to administer e-questionnaires, with the option of completing the forms in writing or by recording spoken responses, which will then be transcribed using large language models.
The approach presented here could be used as a tool to implement follow-up research protocols, such as rehabilitation, training interventions, or monitoring of at-risk groups. Taking a citizen science approach will help to grow our datasets, and in turn could enable new applications in the future such as predictive modelling. By allowing patients to give consent dynamically, we can also facilitate biobank type applications, which could allow longitudinal profiles of individuals to be built up. Finally, this approach is scalable, and could form part of national health strategies, allowing healthcare providers to minimise the need for in-person appointments, and freeing up medical staff for other tasks. This could in turn lead to significant cost savings for healthcare providers.
The infrastructure on which our app is built is highly customisable. We recently used the core components of the app and added new functionality to produce an entirely different application that allows users to record the sound of birdsong outdoors. The soundtrack is then fed to an AI algorithm trained to detect different species. To date, this app has attracted over 140,000 users, who have collectively submitted over 3 million recordings. In ongoing work funded by the Jane and Aatos Erkko Foundation, we are developing several more mobile citizen science applications that span numerous domains, including human-nature interaction and learning difficulties in children. By developing open-source tools, we aim to increase the broad participation of regular citizens in scientific research.
Acknowledgments
The authors thanks Dr. Juhani Multanen for assistance with study logistics and the staff physiotherapists at Nova hospital for helping with data collection. This study did not receive any separate funding.
Disclosure of Interests
The authors have no competing interests to declare that are relevant to the content of this article.
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
