Ahmad Shams, Year 2 Engineering
Abstract
This paper aims to explore the effect the sense of touch has on developing muscle memory in the context of typing. If the sense of touch can improve the development of muscle memory, it could potentially help in developing motor skills at a faster rate. To investigate this, a system of vibration motors was connected to a pair of gloves and when paired with software and a graphical user interface (GUI), the motors would vibrate a certain finger depending on which character needed to be typed. Participants were tasked with learning two keyboard layouts, the Dvorak and Colemak; however, the use of the vibration motors was limited to learning the Dvorak layout. It was found that this design of the device used did not have a significant impact on performance of the tests performed, suggesting that involving the sense of touch when learning a new keyboard format is insignificant.
Introduction
Haptics refers to the science of sensing surrounding environments and is a niche in technology that combines biomechanics, engineering, and computational science (Saddik, 2007). Haptic systems allow for human-computer interactions through the sense of touch via forces and vibrations (Rakkolainen et al, 2020). This is also known as kinaesthetic communication and can enrich the way humans interact with computers (Ultraleap, 2019). Both the human and mechanical haptic systems contain: sensory, processing, and mechanical/motor components (Mandayam et al, n.d.) (Figure 1).
Figure 1: Flow chart of a haptic system Salman, n.d.
Machine Haptics:
The sensory components of the machine haptic system are simply anything that allows humans to interface with the machine or computer such as touch screens, buttons, and pressure sensors (Mandayam et al, n.d.). The processing unit, a computer system, is in charge of converting the input from the sensory components into instructions delivered to the mechanical and motor components. These mechanical and motor components are devices responsible for replacing and augmenting human touch and is the part of the machine haptic system that acts on the skin (Saddik, 2007).
Human Haptics (Figure 2):
Figure 2: Human haptic system Sassik et al, 2011
The sensory system comprises of the human sensory system. The sensory system is made up of countless receptors in and on the skin that send signals to the brain and nervous system, outlining the interaction leading to perception. The processing unit, the nervous system and the brain receives these signals and issues the appropriate motor commands to activate specific muscles, the mechanical aspect of the human haptic system, resulting in movement (Saddik, 2007).” These two systems working together is what allows haptic technologies to be feasible (figure 3).
Figure 3: Flowchart visualising how different haptic systems are connected Saddik, 2007
Muscle Memory and Myelination
It is important to recognise how practice and repetition improves performing a specific action and the biology behind developing muscle memory. This is relevant in terms of the experiment as the device created was meant to make the user perform repetitive motions with additional touch feedback in order to maximize retention of the movements. Whenever a certain action is performed and practised, the brain is optimised to perform that activity via myelination (Osika, 2022).
Myelin is a fatty tissue covering the axons extending out of neurons, acting as an insulative sheathe. Much like the insulation around electrical wires, the myelin sheath increases nerve impulses’ speed and strength, meaning that myelinated nerves serve as more efficient pathways for these nerve impulses (Hartline, 2008).
There are two glial cells that are responsible for the development of myelin, astrocytes and oligodendrocytes. When an astrocyte monitors a system of axons and observes a substantial amount of repeat signals in those axons, oligodendrocytes are activated, producing a myelin sheath. Therefore, a system of axons used to send repeat signals when repeating an action increases myelin density in the nervous system (Ishibashi et al, 2006).
Although there is no definitive proof that greater myelin density improves performance, there are studies that provide evidence for it. A study conducted by Bengtsson et al looked at the hours of piano practised by an experienced pianist during adolescence and its correlation to myelin/white matter in the brain. It was found that the more hours practised correlated to white matter density in the parts of the brain responsible for finger motor skills and visual and auditory processing centres compared to regular people (Bengtsson et al, 2005).
Previous Studies
Previous studies investigating similar concepts have found that an increase in touch information and tactile feedback can help with retaining new movements. In a study published in the American Journal of Surgery (Pinzon, Vega, Sanchez & Zheng, 2016), a group of people were instructed to recreate five 2-D walking paths of varying complexity while blindfolded. After walking through these paths while being guided, they were asked to recreate the paths on a piece of paper which they did with several errors. Afterwards, another group of participants repeated the same procedure; however, they had nine training sessions to obtain more kinaesthetic information. This group was able to recreate the paths with far less error. This shows that an increase in kinaesthetic information involving the sense of touch can aid in developing muscle memory
The Aim of This Paper
This project aims to take advantage of haptic systems and the biology of muscle memory to find a relationship between the sense of touch stimulated through vibrations and the ability to learn a new keyboard layout apart from the popular and widely used QWERTY keyboard layout. The vibrations, created by vibration motors, work in tandem with a GUI to vibrate the correct finger based on what letter needs to be typed in a word displayed on the GUI. A group of participants were given a chance to try and learn a new keyboard layout with the system of vibration motors, while another group was given the same challenge; however, without the vibrations to aid them.
Materials and Methods
Making The Gloves
To create the apparatus (Figure 4), a five-foot long ethernet cable (Amazon Basics) was cut in half, with the plugs being cut off to reveal the cable’s internal wires with a pair of wire cutters. The ends of the two halves of the wires were stripped with wire strippers so that approximately five centimeters of the internal wires were exposed. The internal wires were then stripped themselves with wire strippers, leaving one centimetre of exposed wire on each wire. It should be noted that each ethernet cable is composed of eight internal wires of four colours. Four vibration motors (Lee’s Electronics) were then soldered to each wire with the exposed ends covered with electrical tape such that the exposed wires do not come into contact and short circuit. The fingertips of a pair of cycling gloves (Humrad) were then cut off so that when it was worn, the fingertips could make direct contact with the keyboard for comfort. After the vibration motors are correctly wired, the vibration motors are attached to the second knuckle of each finger of the gloves using both double-sided tape and electrical tape.
Figure 4: Vibration motors on gloves with the microcontroller and keyboard cover
Software
A microcontroller was used to control the vibration of the motors and in this project (an Arduino Nano). The foundation of the code in this project came from a GitHub repository (David Schneider, 2014) which was used to program the Arduino via Arduino integrated development environment (IDE) and the python file, which also contained the GUI (Figure 5). The code was modified to accommodate the Dvorak and Colemak keyboard layouts and to ensure compatibility with the text editor (Pycharm) and the Arduino IDE by encoding the strings, which are the letters being typed, into UTF-8.
Each digital pin from three to twelve on the Arduino Nano was assigned to a finger on each hand. The microcontroller’s pins were connected to the vibration motors through the other end of the ethernet cable with jumper wires using a breadboard.
Figure 5: The GUI used with the system of vibration motors
Testing
When allowing participants to practice a new keyboard layout, a transparent keyboard cover was placed atop a traditional QWERTY keyboard with keyboard stickers stuck on it to mimic the Dvorak and Colemak keyboards.
During testing, each participant was given twenty minutes to practice with the corresponding keyboard layout and cover – either Dvorak or Colemak. During this practice period, the participant practices typing the phrase “the quick brown fox”. Once this practice period elapses, the keyboard cover would be removed, and the participant would be tasked to type out the aforementioned phrase without the visual aid of the GUI. The number of times the participant hits the backspace will be counted and recorded as the number of errors made when typing the phrase.
This process was conducted using both keyboard layouts on each participant; however, they were only be able to practice the Dvorak layout with the apparatus while they had to practise the Colemak layout without any vibrations. This was done to increase the data points for each keyboard layout given the already limited time and available sample size.
Results
After the participants had twenty minutes to practice learning the new keyboard layout, they were tasked with writing a sentence without the aid of the corresponding keyboard cover or any other visual aids. When typing out the sentence, the number of mistakes when learning the Dvorak layout with the vibration motors and Colemak layout without the vibration motors were noted and are portrayed in Table 1. There was no uncertainty associated with the number of mistakes.
Table 1: Number of errors typed with the Dvorak keyboard when learned with vibration motors per participant and Colemak layout without the vibration motors
It should be noted that the environment and testing conditions varied per participant as tests were performed in different rooms with varying levels of noise and distractions. The average numbers of errors when learning the Colemak layout were greater than the average number of errors when learning the Dvorak layout with the help of vibrations, although by a slim margin.
To visualise the data, a double bar graph was used to compare the performance of each participant when learning the Dvorak and Colemak layouts (Figure 6). Then, the averages were also compared using a bar graph, too (Figure 7). The data from Figure 6 is very varied, and no immediate trend was determined. As can be seen in tables one and two, Figure 7 also shows that fewer errors were made when learning the Dvorak layout on average.
Figure 6: Graph visualising the number of typing errors on each keyboard layout per participant
Figure 7: Graph visualising the average number of typing errors on each keyboard layout
Additionally, each participant’s typing speed and accuracy on the QWERY keyboard was tested, with the results being tabulated in table 2
Table 2: Typing speeds and accuracy on QWERTY keyboard of each participant
Discussion
On average, when using the vibration motors to learn the Dvorak layout, the number of typed errors between the eight participants was 41, while when learning the Colemak layout without the vibrations, it was 45. The similarity of the two data sets suggests that the vibration motors do not help in the learning of a new keyboard layout and that the device brought no difference.
This study only considers an extremely limited sample size of only eight participants. In addition to this, each participant attempted to learn both the keyboard layouts using the vibration motors for one of them in an attempt to generate the most data with the number of participants. It should be noted that having the participants learn one layout directly after trying to learn the other would have affected their final results. As previously mentioned, the environments in which the tests were conducted varied per participant. In some cases, the environment was a quiet, distraction-free environment, while in others contained several different distractions that could have impaired the participants’ ability to learn the new keyboard layout.
The biggest limitations of the study is the fact that all of the participants had only ever used the QWERTY keyboard, and as a result, they had built a strong sense of muscle memory and familiarity with it. Because of how used the participants were with typing using on a QWERTY keyboard, learning the new formats proved to be a larger challenge than anticipated as their fingers would default to the key locations that they were used to.
From the data from Table 2, it can be seen that out of the eight participants, those who learned the new keyboard layouts more successfully than others had slower typing speeds than those who performed poorly during the tests. If using typing speed and accuracy as a measure of how well a participant could type on the QWERTY layout, it could be possible that the better a participant knows the QWERTY layout, the more challenging it would be for them to learn a new one. Most participants commented on this, stating that this was a factor affecting their abilities to adapt to the new layout.
Other limitations came in the form of how the vibration motors were programmed to function. The motors vibrate as a letter appears; however, nearly all participants mentioned that the vibrations would have been more helpful had there been a delay between the vibration and the letter appearing. The speed of the vibrations was also another variable. Although the speed or the rate at which new characters would appear was adjustable, most mentioned that the vibrations were too quick.
Each participant went about learning the new keyboards in different ways, too. For example, some participants decided to ignore the vibrations completely, instead electing to commit the key placement to memory right away instead of having the vibrations guide them – others chose to follow the vibrations.
Additional limitations came in the form of the hardware of the apparatus, namely the wearability of the gloves. Firstly, wearing the gloves in the first place restricts the participants’ movement to a certain degree making for a more uncomfortable typing experience. Secondly, the vibration motors were fastened to the gloves’ knuckles with electrical tape, limiting the region’s flexibility where the fingers fitted through. For some, this was not as major of an issue as wearing the gloves while typing, but others reported it to have really limited finger mobility.
To determine whether or not involving the sense of touch in improving muscle memory in the context of typing is significant, several modifications need to be made procedurally and construction of the device. Firstly, a larger sample size is required to find a more reliable trend. Also, the testing conditions need to be consistent throughout trials, and each participant should only be assigned to one method of learning a new keyboard format. In this case, the experimental group would be those using the vibration motors, while those in the control group would learn the new format via conventional means. This would produce more reliable data than having each participant learn a new format using both methods – especially one after the other.
To improve the gloves’ wearability, more discrete and flexible gloves would have been ideal. Much like the gloves, instead of repurposed ethernet cables, which were used as a matter of convenience given their internal make-up, slimmer and more flexible wires should be used instead, such that they would not get in the way of the hands as they type. Instead of using tape to fasten the vibration motors to the gloves and have them restrict finger space, brackets should be made to keep the vibration motors secure and pressed to the finger without limiting the fingers in any way. An added benefit of doing this would be that the vibration motors could easily be taken out and repaired if need be.
In terms of the future direction of this study, it would be interesting to extrapolate the concepts explored in this paper and have them be applied to similar motor activities requiring repetitive finger motions. Perhaps in the future, a study can be conducted using similar means in learning to play a musical instrument such as the guitar or piano instead of in the context of typing. The device talked about in this paper could also be used in a study trying to determine whether or not it can improve typing speeds on the QWERTY keyboard. Lastly, it would be interesting if the device was used in tandem with a program created specifically for learning a new keyboard format in the same way sites such as typingclub.com teach the QWERTY keyboard.
References
Bengtsson, Sara L et al. “Extensive Piano Practicing Has Regionally Specific Effects On White Matter Development”. Nature Neuroscience, vol 8, no. 9, 2005, pp. 1148-1150. Springer Science And Business Media LLC, https://doi.org/10.1038/nn1516. Accessed 27 Mar 2022.
Blenkinsopp, Robert. “What Is Haptics?”. Ultraleap.Com, 2022, https://www.ultraleap.com/company/news/blog/what-is-haptics/.
Hartline, Daniel K. “What Is Myelin?”. Neuron Glia Biology, vol 4, no. 2, 2009, pp. 153-163. Cambridge University Press (CUP), https://doi.org/10.1017/s1740925x09990263. Accessed 27 Mar 2022.
Hutmacher, Fabian, and Christof Kuhbandner. “Long-Term Memory For Haptically Explored Objects: Fidelity, Durability, Incidental Encoding, And Cross-Modal Transfer”. Psychological Science, vol 29, no. 12, 2018, pp. 2031-2038. SAGE Publications, https://doi.org/10.1177/0956797618803644. Accessed 27 Mar 2022.
Ishibashi, Tomoko et al. “Astrocytes Promote Myelination In Response To Electrical Impulses”. Neuron, vol 49, no. 6, 2006, pp. 823-832. Elsevier BV, https://doi.org/10.1016/j.neuron.2006.02.006. Accessed 27 Mar 2022.
Osika, Alexandra. “The Myelin Sheath And Myelination”. Kenhub, 2022, https://www.kenhub.com/en/library/anatomy/the-myelin-sheath-and-myelination.
Pinzon, David et al. “Skill Learning From Kinesthetic Feedback”. The American Journal Of Surgery, vol 214, no. 4, 2016, pp. 721-725. Elsevier BV, https://doi.org/10.1016/j.amjsurg.2016.10.018. Accessed 27 Mar 2022.
Rakkolainen, Ismo et al. “A Survey Of Mid-Air Ultrasound Haptics And Its Applications”. Ieeexplore.Ieee.Org, 2020, http://eprints.gla.ac.uk/225807/2/225807.pdf.
Saddik, Abdulmotaleb El et al. “Human Haptic Perception”. Springer Series On Touch And Haptic Systems, 2011, pp. 45-66. Springer Berlin Heidelberg, https://doi.org/10.1007/978-3-642-22658-8_3. Accessed 28 Mar 2022.
Saddik, Abudlmotaleb. “The Potential Of Haptics Technology”. IEEE Instrumentation & Measurement Magazine, 2007, https://webpages.charlotte.edu/~jmconrad/ECGR6185-2008-01/notes/potential%20of%20haptics%20technologies.pdf. Accessed 27 Mar 2022.
Salman, Arisha. “Haptic Technology – Feedback, Devices, Working Principle, Applications”. Electricalfundablog.Com, https://electricalfundablog.com/haptic-technology/.
Schneider, David. Tactile Glove Project. 2014.
Srinivasan, Mandayam et al. “Human And Machine Haptics”. Rle.Mit.Edu, https://www.rle.mit.edu/media/pr151/24.pdf.
Williamson, Jill M., and David A. Lyons. “Myelin Dynamics Throughout Life: An Ever-Changing Landscape?”. Frontiers In Cellular Neuroscience, vol 12, 2018. Frontiers Media SA, https://doi.org/10.3389/fncel.2018.00424. Accessed 27 Mar 2022.
Leave a Reply