Motion-capture exoskeletons let violinists feel each others’ subtlest moves

World-renowned science fiction author Arthur C. Clarke (2001: A Space Odyssey, Childhood’s End) once quipped that “Any teacher who can be replaced by a computer should be.” Some people think Clarke’s statement means we should replace all teachers.

But instead, Clarke’s comment highlights the indispensable value of the socio-emotional connection to students that great teachers foster, as well a vast body of subtle skills and behavior that computers can’t easily replicate. But what if you combine them? What if a teacher could cybernetically link to a student to transmit her own dance, judo, or surgical techniques instantaneously?

Enter twinned, motion-capture, haptic-response exoskeletons for teaching music.

In a recent Science Robotics paper, Aleksandra Michalko at Belgium’s Ghent University, Francesco Di Tommaso at Università Campus Bio-Medico (UCBM) in Rome, and their colleagues at various institutions explain why their exoskeletal system works so well at harmonizing performance – and thereby improves teaching.

When we learn how to perform physical tasks involved with throwing and catching a ball, creating calligraphy, or soldering a circuit board, we’re probably using our eyes to mimic the actions of a proficient mentor. But what if we have poor vision? What if we can see perfectly well, but we can’t see our teacher because of poor in-person lighting, or a broken web camera during a remote session? And what if the fine motions are too subtle or obscured by difficult angles (as with medical procedures deep inside the body) for anyone’s vision to capture?

The exoskeletal system from Michalko and colleagues teaches by touch, just like athletic coaches, music instructors, and other teachers have automatically done for ages by repositioning the bodies and limbs of their pupils. But as the authors note, even though “haptic feedback provides a direct, implicit channel for sensorimotor communication […] its contribution to fine motor coordination in joint actions remains largely unexplored.”

That’s why Michalko’s team harnessed “the power of haptic communication, rendered through bidirectionally coupled wearable robots.” Their challenging test-case was 20 violinist duos (10 pairs of professionals, and 10 pairs of amateurs) performing live under four conditions: players could a) hear each other only, b) hear and see each other, c) hear each other and exoskeletally feel each other’s movements, and d) hear and see each other, and feel each other using the exoskeletal connection.

With “two–degree-of-freedom upper-limb” movement, the exoskeletons used sensors that transmitted precise mo-cap and force-cap data between partners, and when the motions didn’t match, servo-motors pushed each player to split the difference, promoting synchronized, natural movement.

And there was a twist: all the violinists were exoskeleton-newbies, and none knew they were haptically connected – but that connection “substantially enhanced spatiotemporal coordination and dynamic musical alignment,” or in simple terms, cybernetically-connected exoskeletons made violinists precisely align their arms and bows better, especially when the musicians could see and hear each other.

As project coordinator and UCBM NeXTlab contributing author Domenic Formica says, “We are entering an era where robots can mediate physical communication between humans in entirely new ways. This study is a first step toward systems that physically connect people, enhancing their coordination, learning, and rehabilitation.”

Di Tommaso goes further. As co-lead author and postdoctoral researcher at the Advanced Robotics and Person-Centered Technologies Research Unit (CREO Lab) at UCBM, he explains, "Haptics, or tactile and kinesthetic perception, provides information in a fundamentally different way than sight. It's physical, direct, and immediate. Our results suggest that the human motor system can integrate this information very efficiently, even in highly skilled artists."

Whether a new technology is good for humanity depends on several factors. For instance, does that technology delete enjoyable human work (and with it ambition, skill, livelihood, workplace camaraderie, and community connection) to maximize profits for the few and misery for the many? Is it replacing dangerous but vital work that improves health and even saves lives, and thus makes a better world for all?

Or does the new technology accelerate genuine learning and thus grant people more power and time to do whatever they want with skills rather than spending massive amounts of time money gaining those skills? Clearly, the UCBM cybernetically linked exoskeleton fits into that final category.

Of course, exoskeletons have countless uses beyond this remarkable new one, including helping seniors regain mobility, augmenting the strength of caregiving and industrial workers, boosting upper-body endurance and underwater swimming range, expanding hiking range, keeping Parkinson’s and paralyzed patients walking, or even becoming a mech-monster for $1,515 per hour. Adding haptic feedback – with VR – offers even more immersive use.

As innovators develop the technology from a bulky exoskeleton to something like the snug pieces of mo-cap suits but with non-motorized haptic stimulation (as with the vibrations from a cell phone or game controller, and remote hand-holding and hugging), other forms of haptic teaching will contribute to gaining large-motor skills in dance and combat sports, and fine motor skills in visual art and surgery, or improved speaking using mouth-mounted sensor-stimulators in speech therapy.

"These wearable robots,” says the exoskeleton designer and co-author Nicola Vitiello at the BioRobotics Institute of the Scuola Superiore Sant’Anna in Pontedera, Italy, “could support collaborative training, motor learning, and even rehabilitation, where therapists and patients could be physically connected."

Source: UCBM