Undergoing rehabilitation through physiotherapy exercises plays a vital role in maintaining and regaining the physical strength of patients who suffer from physical impairments due to accidents or ageing. However, this repetitive and regular process can be boring over a long time and often causes the subject to lose interest in performing the required exercises. The exergaming approach tries to alleviate such problems by integrating physical exercise into a virtual gaming environment. We aim to create an interactive and intelligent rehabilitation exergaming system called RehabMe by exploiting Natural User Interface (NUI) devices such as the Xbox Kinect and openspace3D game engine. RehabMe aims to sustain patients' interest in performing exercises through immersive and interactive gameplays while carrying out an embedded assessment (of both quantitative and qualitative measures) of rehabilitation progress. With our solution, patients are no longer required to wear any sensors on the body as advanced imaging and skeleton tracking provided by Kinect can be used to detect and recognize both static and dynamic movements according to the exercises performed. The interactive exergaming encourages subjects to perform physiotherapy exercises actively by breaking the monotony of repetitive exercises and immersing in virtual gameplay environments. The embedded assessment features enable physiotherapists to remotely monitor the progress or improvements made by subjects using quantifiable parameters such as repetition, response time, accuracy, number of correct moves, etc. Thus, our solution facilitates the growing needs of home-based rehabilitation by enabling an enjoyable and relaxing interactive experience for patients and fulfilling clinical requirements of therapists at the same time.

Background

According to the World Health Organization, 15 million people suffer stroke worldwide each year. It is Singapore's third leading cause of death. The number of deaths and admissions to Singapore hospitals for stroke have been rising (1) and thus, physiotherapy is imperative to maintain and restore maximum movement and functional ability through regular exercises.

Rehabilitation aims to restore a patient's ability to independently perform basic physical activities and tasks by carrying out exercises on a regular basis. Among various forms of physical therapy, orthopaedic physical therapy deals with diagnosing, managing, and treating disorders and injuries of the musculoskeletal system including rehabilitation after orthopaedic surgery. This type of therapy is chosen to test the effectiveness of our game as it is the most common form of physiotherapy (2) and one that can be administered without close assistance with a therapist in a home environment.

Clinical Problems

A crucial problem with conventional therapy is the monotonous nature of the exercises which usually leads to the loss of patients' motivation to do the exercises (3). As a result, they spend little time each day doing the physical activity and little independent exercise takes place (4). As a lot of patients find these exercises boring (5) and repetitive, they do not feel motivated enough to continue doing them on their own (6). The lack of performing rehabilitation exercises and proper evaluation of their progress results in adverse effects to their physical conditions, such as delays in recoveries and failures to perform basic tasks (7).

The application of interactive and entertaining aspects of games facilitates rehabilitation by engaging users in a training environment where programmable tasks can be tailored to personal abilities and requirements, helping to motivate the patient and create a more enjoyable experience, leading to improved results (8). Games offer the potential to engage a patient such that they forget the monotony of the exercise they are doing. Studies have shown gaming environments enhance motivation amongst physical therapy patients. Performance feedback is also important in facilitating the patient learning process during rehabilitation (9). Feedback can take both auditory and visual forms, and can be used to signify correct or incorrect actions or responses. Gratifying messages or sounds can be presented when a task is successfully completed, leading to an increase in user motivation (10).

Proposed Exergaming Solution: RehabMe

1. Games: 3D game to perform exercise by following gameplay.
2. UI: Provides the game data, assessment and schedules.
3. Assessment: Recognizes user movements and sends outcomes to the therapists.
4. Schedule: Displays and sets personalized exercise schedules for patients.

We propose to create an exergaming system called RehabMe in which the user completes a series of games designed to act as an interactive exercise experience via the program interface, and the program also records and stores the information into the database in the physiotherapy clinic. The doctor is able to track the progress of the patients as they complete the exercise schedules every day, as well as make adjustments to the intensity of the exercises, which is reflected back onto the program interface. A complete data flow diagram of high-level activities and operations performed by the respective patient and therapist as well as their interaction through a high-speed communication network is shown in Figure 2.

Our main objective is to create an interactive exergaming system which can utilize the Kinect's NUI to sustain patients' interest in performing exercises while helping them to exercise effectively and perform embedded qualitative and quantitative assessment of the patients' exercise regimen, as part of an effective home-based rehabilitation system as shown in Figure 3. Our hypothesis is that using an interactive game in the patients' rehabilitation routine will help prolong the amount of time spent by the patients on doing the activity as compared to without any external stimuli, which will in turn, help their rehabilitation process. The game is also able to provide an assessment that is sent to the therapist, enabling the therapist to assess the patients' performance easily and more flexibly as the patients do not need to come to the clinical site for rehabilitation and assessment.

Materials

For this project, we use the Microsoft Kinect Sensor, which consists of an RGB camera and twin IR-sensors that can determine depth from 0.85m to 4.0m rather than other sensors due to the accessibility, ease of use, and full body interaction with the patients (refer to Appendix A). For our program we used the OpenSpace 3D open source development platform for interactive 3D projects (www.openspace3d.com) as it is already integrated with the Kinect using the OpenNI driver and allows flexibility for therapists to make modifications to the program. In order to gather data, we used the Upper Body Motion Tracking application developed in C++ programming using openFrameworks (www.openframeworks. cc).This enables us to track and record both the coordinates as well as the angles of the joint involved in the exercise for exercise assessment and progress/performance evaluation. This program is able to track the coordinates of the upper body, including neck, shoulders, elbows, and hands.

Methods

Using the OpenSpace 3D with openNI Kinect integration available from the open-source OpenSpace 3D community, we developed a proof of concept exergame application for rehabilitation as well as a more challenging version for sustaining the patients' interest. After developing the game, we utilized the upper body tracking program while playing the game in order to gather data from the exercise itself and processed the data in order to show how to process the raw data gathered from the tracking program to obtain a clearer assessment data which is useful for the therapist.

In this interactive program, the user can first calibrate with the Kinect sensor before proceeding. Any subsequent selections in the menu are done via hand gestures. The user's hand acts as a mouse to point to a specific item, and by pushing forward, they can select the option that is being pointed at.

There are two modes to the program, Schedule and Free Play. When the user selects Schedule, the user is presented with the list of exercises that needs to be completed (Figure 4a). The user can select any of the undone exercises to complete.

Using 'Sideways Movement' as an example, the user will perform physiotherapy routines through repetitive hand movements (Figure 5), where they need to raise their arms from the side upwards. Their movements are reflected in the 3D character in the game (Figure 4b). This rehabilitation movement trains the Range of Motion of the patient's arms. In order to gather data, we use the upper body tracking program such that we can see the performance of the patients as well as their progress. Data collected will be processed and analyzed and delivered to therapists as patient assessment reports (Figure 4c).

The second mode is the Free Play mode, where the user can select any of the 'Schedule' exercises and set the parameters to the game, such as the number of repetitions and the speed of exercise, both limited by the doctor's recommendations of how intensive the person is allowed to play the game. Also, there will be a Jewel Game, in which the user has to touch 8 objects that will appear in a semicircle on the same axis as the user. These objects will appear at random and the user has to touch the object by reaching out within a 5 second limit, or the object will disappear. To progress, the patient must touch the objects in order to score points and advance to a higher level. When the level increases, the time allowed to touch the objects will decrease accordingly. This game allows for an interactive experience for rehabilitation (refer to Appendix B for more details about the game).

We simultaneously ran the RehabMe game (Figure 6) and the upper body tracking program, (Figure 7) and were able to track and record the absolute coordinates of many parts of the upper body, including neck, shoulders, elbows, hands, and torso over a period of time. For this experiment, we collected the data from doing the exercise with the game and processed the data for the right hand from coordinates into graphs against time as shown in Figure 8 below for five repetitions of the exercise described above (moving the arms from the side upwards and then downwards is considered one repetition).

In the graphs above, only the X and Y coordinates of the right hand are shown. This is because the exercise involved, which requires the patients to move their arms from the side upwards, only involves the two axes, namely X and Y. Thus, the Z axis value is constant.

In addition to the X and Y coordinates of the right hand, the angles of the joints, especially the shoulder is important as well. This exercise trains the range of motion of the patients' shoulder joints and one of the ways to assess the range of motion is to measure the angle. The upper body tracking program is also able to track and evaluate the angle of the shoulder and the arms during the exercise and record it for the period of the exercise as shown in Figure 9.

IV. Exercise Assessment and Data Evaluation Analysis

From the first graph, we can obtain the number of repetitions that the patient has done for the period of time. The number of repetitions can be obtained from both graphs, but it is easier to see from the graph of the Y-axis values. In one repetition, the Y-axis value reaches a maximum value once (when the arm is at the top) and a minimum value once (when the arm is at the bottom). Thus, each peak corresponds to one repetition of the rehabilitation exercise, making it possible to find the number of repetition from the number of peaks in that period of time.

Moreover, the graph of the X and Y axes can be used to deduce whether the patients are struggling or whether the patient is doing the exercise properly. The therapist can detect if the patient is struggling as there will be visible fluctuations in the values recorded during their exercise. The therapist can then help to modify the regimen to help alleviate this problem.

In order to obtain the maximum range of motion of the patient, we need to interpret graph 2 and the upper body tracking program first. The upper body tracking program is able to track the angle of the shoulder joints. However, it only tracks acute angle ( 0º≥ Ø ≥ 180º). Thus, the maximum angle, 180º (Ømax) is reached when the arm is in line with the shoulders. This means that in one repetition, the value of the angle will reach a maximum twice, when the arm is brought up and down respectively. Furthermore, in one repetition, the value of the angle reaches a minimum twice. This occurs when the arm is at the bottom and at the top. In the graph, the minimum values alternate from the angle at the bottom with the angle at the top since the exercise starts from the bottom. The maximum range of motions can thus be found by subtracting the minimum values of the angles Ømin1 and Ømin2 from 360º. The minimum values of these angles are 102.2º and 123.2º respectively as seen from the graph. Thus, the maximum range of motion during this exercise is 360º - 102.2º - 123.2º= 134.6º.

V. Discussion

Our game system helps to break the monotony of the rehabilitation exercise through interactive gaming using the Kinect, and enables an effective data gathering process for therapists. Its flexibility allows both therapists and patients to modify the exercise regimen. Thus, it improves the current physiotherapy system in terms of its flexibility, ease of use, and the stimuli used to do the exercise.

Albeit these improvements, there are several limitations to our implementation. Firstly, the game program (using OpenSpace 3D), the upper body tracking program, and the user interface are not integrated yet. The game is also limited to the use for upper body rehabilitation only. Furthermore, OpenSpace 3D does not have advanced game programming functions which restricts some of the ideas that we had for the game. However, we chose this as it is the only development platform that is integrated with the Kinect through OpenNI.

In the future, we aim to integrate the game program (OpenSpace 3D), the upper body tracking program used to collect data, and the user interface together into one program to make it more convenient for users. Moreover, the game can also be expanded to focus on patients who require lower body rehabilitation and/or exercise while sitting down, since the Kinect currently requires full-body tracking to work. Furthermore, a more advanced development platform such as Unity 3D can also be used to allow more flexibility in developing the game. We would also like to harness the Kinect's built in microphone to increase the interaction between patients and the game so as to increase the patient's interest.

VI. Conclusion

Physiotherapy practices are continually evolving in conjunction with technological development, ultimately producing new ideas and equipment to enhance the effectiveness of therapeutic activities. Utilizing this game which utilizes the Kinect's NUI in physiotherapy will add excitement and provide a unique stimulus for patients to continue their rehabilitation regimen. It also enables the collection of data during the game in order to make the therapists' assessment easier and much more flexible as the patient does not need to come to the clinic to do the rehabilitation exercise and assessment but in their own time instead. This data can be further processed by the therapists in order to obtain more useful and meaningful data of the patients' progress. Thus, the game has the potential to aid the physiotherapy regimen due to its flexibility and its ability to sustain the patients' interest in performing the exercise.

Acknowledgements

We would like to express our utmost gratitude to our research mentor, Mr Aung Aung Phyo Wai from Institute for Infocomm Research (I2R), for his invaluable guidance, encouragement and patience with us throughout our attachment program. We would also like to thank KeDeng Yao Brendan for the upper body tracking that he developed during his attachment at I2R. We would also like to express our gratitude to I2R and Young Researchers' Attachment Program organized by A*STAR Graduate Academy (AGA) for providing us with this opportunity to conduct the current research work.

References

1. Venketasubramanian, N. (1999). Stroke in Singapore - an overview. Singapore Medical Journal, 40((01)), Retrieved from https://www.sma. org.sg/smj/4001/articles/4001ra1.html
2. Inverarity, Laura. (2010). Types of physical therapy. Retrieved from https://physicaltherapy.about.com/od/typesofphysicaltherapy/a/typesofpt
3. G. Burdea. (2002). "Key note address: Virtual rehabilitation - benefits and challenges," in 1st Intl. Workshop on Virtual Reality Rehabilitation, VRMHR, pp. 1-11.
4. J. H. Carr and R. Shepherd. (2002). Stroke Rehabilitation: Guidelines for Exercise and Training to Optimize Motor Skill, 3rd ed. Butterworth- Heinemann, pp.301.
5. L?vquist, E, & Dreifaldt, U. (2006). The design of a haptic exercise for post-stroke arm rehabilitation.Informally published manuscript, Interaction Design Centre (IDC), Computer Science and Information Systems, University of Limerick, IRELAND, Limerick, Ireland. Retrieved from https://citeseerx.ist.psu.edu/viewdoc
6. Den Brok, W.L.J.E. (2008). Motivational factors influencing.Informally published manuscript, Department of Technology Management, Eindhoven University of Technology, Eindhoven, Netherlands. Retrieved from https://alexandria.tue.nl/extra2/afstversl/tm/Brok%202008.pdf
7. Campbell, R, Evans, M, Tucker, M, & Quilty, B. (2000). Understanding non-compliance with physiotherapy in patients with osteoarthritis of the knee. Journal of Epidemiology and Community Health, 55(02), Retrieved from https://jech.bmj.com/content/55/2/132.abstract doi: 10.1136/jech.55.2.132
8. A. Rizzo and G. J. Kim. (2005). "A SWOT Analysis of the Field of Virtual Reality Rehabilitation and Therapy, "Presence: Teleoperators and Virtual Environments, vol. 14, pp. 119-146.
9. D. Jack, R. Boian, A. Merians, M. Tremaine, G.Burdea and S. Adamovic. (2001). "Virtual Reality-Enhanced Stroke Rehabilitation," IEEE Trans. on Neural Systems and Rehabilitation Engineering, vol. 9, pp.308-318.
10. G. Burdea. (2003)."Virtual Rehabilitation - Benefits and Challenges". Intl. Medical Informatics Association Yearbook of Medical Informatics, pp. 170-176.

Appendix A: Comparison of Hardware and Software Used for Kinect

From the table of comparison above, it is evident that the Kinect is superior to non-Natural User Interface Devices as well as other Natural User Interface devices such as the Wiimote. It's inexpensiveness as well as ease of use and allowing more interaction with the users as it tracks whole body movements makes it a viable choice. Thus, by using the Kinect, it is possible to create solutions that sustain the patients' interest further and to target a larger group of people as well.

From the table comparison above, we have decided to choose OpenNI as our software of choice. As compared to other Kinect hacks prior to the release of the Microsoft Kinect SDK, it is the most developed software available when compared to FAAST and libfreenect. Compared to the Microsoft Kinect SDK, the OpenNI falls short only due to its inability to harness the Kinect's Microphone Array unlike the Microsoft SDK. However, the OpenNI is usable on any platform, not only on Windows and thus be able to reach a bigger audience. Furthermore, OpenSpace 3D is the only development platform that is available to integrate with the Kinect runs on OpenNI, which eliminates the need to integrate the game with the Kinect after it is made. Thus, we decided to stick with OpenNI as we are going to use the OpenSpace 3D due to its integration with the Kinect.

Appendix B: About the Game

Our interactive game starts with a splash screen (Figure B.1) that prompts the user to calibrate with the Kinect sensor. Once this is done, the user is brought to the Home Page (Figure B.2) which displays the schedule of the exercise regimen as well as the option to play any of the provided exercises freely.

When the schedule option is selected, the user is brought to the Schedule page (Figure B.3) which displays the exercises that the user needs to complete. The user can also select Free Play mode (Figure B.4), and would be able to select the choice of exercise. In addition to the exercises, the patients can also choose to play the jewel game, which is a more challenging version of the rehabilitation program.

When one of the Free Play exercises is chosen, the user is presented with the options screen (Figure B.5) which lets them configure the amount of repetition and the speed of exercise. Figure B.6 shows the interface of the game which consists of the player facing a 3D space with objects that needs to be touched at a static position. When the user completes the game, the user is presented with the scores screen (Figure B.7) that displays their game statistics. Meanwhile, the diagnostics of the game (Figure B.8) is sent to the therapist for further evaluation together with the data collected by the upper body tracking program. The rehabilitation program is made using OpenSpace 3D, which is integrated with the Kinect using the OpenNI middleware. Before the game begins, the user has to calibrate with the Kinect. Afterwards, the game checks the value of the counter/repetition, which in this case is the level of the game. If the level is less than 10, the user will have to do the right hand rehabilitation exercise first for ten repetitions, after which the game proceeds to the left hand exercise. After another 10 repetitions of the left hand, the game finishes. The therapist can easily increase or decrease the number of repetitions the patient needs to do by easily changing the values in the 'if' conditions. This increases the flexibility of the program to suit different rehabilitation needs.

The rehabilitation program is made using OpenSpace 3D, which is integrated with the Kinect using the OpenNI middleware. Before the game begins, the user has to calibrate with the Kinect. Afterwards, the game checks the value of the counter/repetition, which in this case is the level of the game. If the level is less than 10, the user will have to do the right hand rehabilitation exercise first for ten repetitions, after which the game proceeds to the left hand exercise. After another 10 repetitions of the left hand, the game finishes. The therapist can easily increase or decrease the number of repetitions the patient needs to do by easily changing the values in the 'if' conditions. This increases the flexibility of the program to suit different rehabilitation needs.

The other more challenging mode for this game is the jewel game which is based on the same concept but instead of appearing in a particular order for the sideways rehabilitation movements, the jewels appear at random and a timer is incorporated to make it more challenging and fun for the patients. As the level increases according to the score, the difficulty increases - the user has to react faster as the time allowed to touch the jewels becomes shorter.

Billy Andika is a Year 6 student from Hwa Chong Institution, Singapore and he co-authored this research paper together with Johandy Tantra from Anglo-Chinese School (Independent). They participated in the 2012 Singapore Science and Engineering Fair and clinched the Gold medal for this project. For more information about this article, please contact Tan Ai Chin at tanac@hci.edu.sg.

Click here to download the full issue for USD 6.50