University of Southern California Dissertations and Theses

Perception and haptic interface design for rendering hardness and stiffness

(USC Thesis Other)

Perception and haptic interface design for rendering hardness and stiffness

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Perception and Haptic Interface Design for Rendering Hardness and Stiffness
by
Naghmeh Zamani
A Dissertation Presented to the
FACULTY OF THE USC GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF PHILOSOPHY
(COMPUTER SCIENCE)
May 2023
Copyright 2023 Naghmeh Zamani
Acknowledgements
I am deeply appreciative of all those who have supported and encouraged me on this remarkable journey
of completing my PhD.
First and foremost, I extend my heartfelt appreciation to my advisor, Dr. Heather Culbertson. Her
compassion, flexibility, and dedication to my success have been pivotal in shaping me into the scholar
and researcher I am today. Beyond her role as an advisor, Heather has been a supportive friend, always
ready to offer guidance and lend a listening ear. Her genuine care has instilled in me a profound sense of
confidence and perseverance. I am sincerely grateful for her commitment to my growth and development
in the field of research. Heather’s mentorship and friendship have had a lasting impact on my life, and I
will carry the invaluable lessons and support she has provided with me throughout my career.
To my fellow researchers and friends at USC, I want to express my deepest appreciation for the cama-
raderie and support you have provided. The cherished memories we have forged together have made my
time at USC truly unforgettable and fulfilling. Additionally, I would like to extend a special thanks to my
friend Sandeep Kollanur for his caring friendship, consistent backing, and continuous support throughout
this journey.
I would like to express my gratitude to my colleagues at Meta Reality Lab, especially Benjamin
Stephens-Fripp and Jess Hartcher-O’Brien. Working with them during my internship has been an incred-
ible opportunity for which I am truly grateful. Their expertise, guidance, and collaborative efforts have
significantly enhanced my research experience, and broadened my perspectives.
ii
I would also like to express my profound gratitude to my husband, Ashkan. Our journey together,
from being labmates in graduate school to supporting each other throughout my PhD program, has been
immeasurable. His expertise and profound knowledge in our shared field have provided me with invaluable
insights and perspectives, enriching my research and academic pursuits. His support, continuous reminder
of my strengths, and unwavering belief in my capabilities have been constant sources of inspiration and
drive. His love and companionship have served as a pillar of strength through the challenges of graduate
school, and I feel truly fortunate to have him by my side. Without him, this academic milestone would not
have been possible.
I extend my sincere thanks to my mom, dad, and brothers, for being my guiding light and the foun-
dation of my success by their a consistent source of love, support, and encouragement from a distance.
Since the beginning of my journey, they have been my greatest motivation and inspiration. The strong
work ethic and passion for learning instilled in me by my parents have been the driving forces behind my
academic pursuits. I am forever grateful for everything they have done for me, and I cannot adequately
express how much their presence in my life means to me.
My heartfelt gratitude also goes to Mohsen, Parvaneh, and Donya for their unyielding love and sup-
port. Their constant encouragement and faith in my abilities have propelled me forward throughout my
academic journey. Their presence has brought immense joy into my life, and the support they have shown
me is truly immeasurable. I feel incredibly honored to have such incredible individuals by my side, who
have stood beside me, and celebrated my achievements with genuine enthusiasm.
I would also like to express my appreciation to my master’s advisor at New Mexico Tech, Dr. David
Grow, and his family. I am very grateful to him for providing me with the opportunity to work in his lab,
where I discovered my passion for research and was inspired to pursue a PhD. His commitment to fostering
a nurturing and intellectually stimulating environment has propelled my growth and development as a
researcher. I am deeply indebted to them for their support and encouragement.
iii
TableofContents
Acknowledgements ........................................................................................................................ ii
List of Tables.................................................................................................................................. vii
List of Figures ................................................................................................................................ viii
Abstract ......................................................................................................................................... xii
Chapter 1: Introduction................................................................................................................. 1
1.1 Kinesthetic and Tactile Perception ..................................................................................... 3
1.1.1 Haptic Perception of Hardness and Stiffness ........................................................... 4
1.2 Haptic Rendering Algorithms ............................................................................................ 5
1.2.1 Stiffness Rendering ............................................................................................... 6
1.2.2 Hardness Rendering .............................................................................................. 7
1.3 Haptic Interfaces for Rendering Hardness and Stiffness ...................................................... 8
1.4 Contributions ................................................................................................................... 10
Chapter 2: Effects of Dental Glove Thickness on Tactile Perception Through a Tool ......................... 12
2.1 Background...................................................................................................................... 13
2.2 Experimental Methods ...................................................................................................... 14
2.2.1 Experimental design.............................................................................................. 14
2.2.1.1 Participants............................................................................................ 14
2.2.1.2 Gloves ................................................................................................... 14
2.2.1.3 Variables................................................................................................ 15
2.2.1.4 Dental Tools .......................................................................................... 15
2.2.1.5 Force/Torque Application ....................................................................... 15
2.2.1.6 Vibration Application ............................................................................. 16
2.2.1.7 Experiment Setup................................................................................... 16
2.2.2 Vibration Recording Procedure .............................................................................. 16
2.2.3 Perception Test Procedure ..................................................................................... 18
2.3 Results ............................................................................................................................. 19
2.3.1 Tool Grasping ....................................................................................................... 19
2.3.2 Recorded Vibration ............................................................................................... 20
2.3.3 Force Perception ................................................................................................... 21
2.3.4 Torque Perception................................................................................................. 23
2.3.5 Vibration Perception Through the Tool .................................................................. 24
2.3.6 Direct Contact Vibration Perception....................................................................... 26
iv
2.4 Discussion........................................................................................................................ 26
2.5 Summary ......................................................................................................................... 28
Chapter 3: Combining Haptic Augmented Reality with a Stylus-Based Encountered-Type Display
to Modify Perceived Hardness....................................................................................... 29
3.1 Background...................................................................................................................... 30
3.2 Encountered-Type Haptic Display Design .......................................................................... 33
3.2.1 Hardware Design .................................................................................................. 33
3.2.2 Evaluation and Calibration of the Haptic Device ..................................................... 35
3.2.3 Stylus ................................................................................................................... 36
3.2.4 End-effector Plate ................................................................................................. 36
3.3 Experimental Methods ...................................................................................................... 37
3.3.1 Position-Based Spring Model ................................................................................. 38
3.3.2 Position-Based Spring-Damping Model .................................................................. 39
3.3.3 Event-Based Transient Force Feedback Model......................................................... 40
3.3.4 Encountered-Type Haptic Display Models .............................................................. 41
3.4 User Study ....................................................................................................................... 41
3.5 Results ............................................................................................................................. 43
3.6 Discussion........................................................................................................................ 47
3.7 Summary ......................................................................................................................... 49
Chapter 4: Effects of Physical Hardness on the Perception of Rendered Stiffness in an Encountered-
Type Haptic Display ..................................................................................................... 51
4.1 Related Work.................................................................................................................... 52
4.2 Encountered-Type Haptic Display Design .......................................................................... 52
4.3 Experimental Methods ...................................................................................................... 53
4.3.1 Experiment 1 ........................................................................................................ 53
4.3.2 Result of Experiment 1 .......................................................................................... 55
4.3.3 Discussion of Experiment 1 ................................................................................... 57
4.3.4 Experiment 2 ........................................................................................................ 58
4.3.4.1 Modified Staircase method...................................................................... 60
4.3.4.2 User Study ............................................................................................. 61
4.3.5 Results of Experiment 2......................................................................................... 63
4.4 Discussion........................................................................................................................ 65
4.5 Summary ......................................................................................................................... 69
Chapter 5: Rendering Hardness and Stiffness Using a Dynamic End-Effector in an Encountered-
Type Haptic Display ..................................................................................................... 70
5.1 Rendering Methods........................................................................................................... 71
5.1.1 Dynamic End-Effector Design................................................................................ 72
5.1.2 Device Control...................................................................................................... 73
5.2 Experimental Methods ...................................................................................................... 75
5.2.1 Setup.................................................................................................................... 76
5.2.2 Procedure ............................................................................................................. 77
5.2.3 Experimental Conditions ....................................................................................... 78
5.3 Results ............................................................................................................................. 79
5.3.1 Experiment 1: Stylus ............................................................................................. 79
5.3.2 Experiment 2: Bare Finger ..................................................................................... 81
v
5.4 Discussion........................................................................................................................ 82
5.5 Applications ..................................................................................................................... 85
5.5.1 Application #1: Medical Simulation ........................................................................ 85
5.5.2 Application #2: Online shopping............................................................................ 87
5.5.3 Application #2: Gaming......................................................................................... 87
5.6 Summary ......................................................................................................................... 88
Chapter 6: Energy Spectrum as the Key Feature for Hardness Perceptual Cue .................................. 90
6.1 Related Work.................................................................................................................... 91
6.2 Methods........................................................................................................................... 92
6.2.1 Material Selection and Recording ........................................................................... 92
6.2.2 Recording Tapping Vibrations:............................................................................... 93
6.2.3 Signal Rendering Techniques ................................................................................. 96
6.2.3.1 Data-driven model ................................................................................. 97
6.2.3.2 Decaying Sinusoidal model ..................................................................... 98
6.2.3.3 Ricker wavelet model ............................................................................. 98
6.2.4 Signal Tuning:....................................................................................................... 99
6.2.5 Finger elasticity compensation: ............................................................................. 99
6.2.5.1 Surface compensation: ........................................................................... 100
6.2.6 Processing: ........................................................................................................... 100
6.3 Experimental Methods ...................................................................................................... 101
6.3.1 Participants .......................................................................................................... 101
6.3.2 User Studies.......................................................................................................... 102
6.3.2.1 Update the Fingertip elasticity Properties: Experiment One ...................... 102
6.3.2.2 Update the Fingertip Properties: Experiment Two .................................... 103
6.3.2.3 Update the Surface Properties: Experiment One....................................... 104
6.3.2.4 Update the Surface Properties: Experiment Two ...................................... 105
6.4 Results ............................................................................................................................. 106
6.5 Discussion........................................................................................................................ 110
6.6 Summary ......................................................................................................................... 112
Chapter 7: Conclusion .................................................................................................................. 113
7.1 Contributions ................................................................................................................... 113
7.1.1 Effects of Dental Glove Thickness on Tactile Perception Through a Tool .................. 113
7.1.2 Combining Haptic Augmented Reality with a Stylus-Based Encountered-Type
Display to Modify Perceived Hardness ................................................................... 114
7.1.3 Effects of Physical Hardness on the Perception of Rendered Stiffness in an
Encountered-Type Haptic Display.......................................................................... 115
7.1.4 Rendering Hardness and Stiffness Using a Dynamic End-Effector in an
Encountered-Type Haptic Display.......................................................................... 115
7.1.5 Energy Spectrum as the Key Feature for Hardness Perceptual Cue........................... 116
7.2 Future Directions.............................................................................................................. 117
Bibliography .................................................................................................................................. 118
vi
ListofTables
2.1 Mean perceptual thresholds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.2 Mean ratings for difficulty of distinguishing sensations . . . . . . . . . . . . . . . . . . . . 22
3.1 Participants’ preferred rendering method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.1 Selected hardnesses covering a wide range . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.2 Selected plates in different hardness categories . . . . . . . . . . . . . . . . . . . . . . . . . 59
5.1 The results of a 2-way ANOVA analysis, examining the main effect of hardness and its
interaction with stiffness when interacting with rendered blocks using a stylus. . . . . . . 80
5.2 The results of a 2-way ANOVA analysis, examining the main effect of stiffness and its
interaction with hardness when interacting with rendered blocks using a stylus. . . . . . . 81
5.3 The results of a 2-way ANOVA analysis, examining the main effect of hardness and its
interaction with stiffness when interacting with rendered blocks using a bare finger. . . . . 82
5.4 The results of a 2-way ANOVA analysis, examining the main effect of stiffness and its
interaction with hardness when interacting with rendered blocks using a bare finger. . . . 83
6.1 Tuning parameters for wavelet and decaying sinusoid rendering techniques while
compensating for finger elasticity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
6.2 Tuning parameters for wavelet and decaying sinusoid rendering techniques while
compensating for surface properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
vii
ListofFigures
1.1 Illustration of the difference between two stiffness levels. . . . . . . . . . . . . . . . . . . . 2
1.2 Illustration of the difference between two hardness levels. . . . . . . . . . . . . . . . . . . 2
2.1 Experimental setup for vibration recording procedure. Vibrations were displayed through
a voicecoil attached to the dental tool, and participants were required to apply a constant
pressure to the force sensor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2 Experimental setup for determining perceptual thresholds for (a) force, (b) torque, (c)
vibration through the tool, and (d) vibration with direct contact. In each subfigure, one of
the glove conditions is shown. A small plastic piece was used to create a flat surface for
applying force by the SWM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3 Tool contact points on the hand. Participants were split into three contact point groupings
based on images recorded during the study. . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.4 Comparison of the power of recorded vibrations across different glove conditions.
Statistically significant differences marked (∗∗∗≡ p≤ 0.001,∗∗≡ p≤ 0.01,∗≡ p≤ 0.05). 21
2.5 Comparison of the force perception thresholds across different glove conditions.
Statistically significant differences marked (∗≡ p≤ 0.05). . . . . . . . . . . . . . . . . . . 22
2.6 Comparison of the torque perception thresholds across different glove conditions.
Statistically significant differences marked (∗∗∗≡ p≤ 0.001,∗≡ p≤ 0.05). . . . . . . . 23
2.7 Comparison of the vibration perception thresholds through a tool across different glove
conditions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.8 (a) Average vibration threshold separated by tool contact location. (b) Comparison of
the vibration perception thresholds across tool contact locations. Statistically significant
differences marked (∗∗∗≡ p≤ 0.001). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.9 Comparison of the vibration perception thresholds in direct contact across the different
glove conditions. Statistically significant differences marked (∗≡ p≤ 0.05). . . . . . . . . 26
viii
3.1 (a) Real object, (b) position-based virtual object, (c) event-based virtual object, and (d)
ETHD virtual object. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.2 Encountered-Type Haptic Display created from kinesthetic haptic device and unthethered
stylus. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.3 Desired stiffness versus actual stiffness provided by the device before and after calibration. 34
3.4 A 3d printed stylus with embedded position sensor and a Force/Torque sensor . . . . . . . 36
3.5 Three experimental setups: (a) a setup with untethered stylus and ETHD, (b) real object,
and (c) a setup with tethered stylus for rendering position- and event-based methods. . . . 37
3.6 Six real surfaces rendered in the study. Three materials (insole, felt, and wood) were
placed on foam and wood. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.7 End-effector plates to simulate hardness in the range of hard, medium, and soft. . . . . . . 42
3.8 (a) Realism, (b) hardness, and (c) compressibility ratings across all materials and rendering
methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.1 Data processing steps for tapping data. (left) hardest plate, (right) softest plate. . . . . . . . 55
4.2 Measured spectral centroid of force for each hardness of plates versus device stiffness . . . 56
4.3 Magnitude of spectral centroid of force for each plate hardness. . . . . . . . . . . . . . . . 56
4.4 Average of the spectral centroid of force for every stiffness and hardness . . . . . . . . . . 57
4.5 Magnitude of the dominant frequency of forces versus plates hardness and stiffness. . . . . 58
4.6 Selected hardnesses based on the durometer standard table. The plates with different
hardnesses are attached to a 3D-printed mount for connection to the haptic device. . . . . 59
4.7 (a) Weber fraction of stiffness versus hardness of plates. Within each plate, a smaller set
of boxplots indicates how Weber fraction of stiffness varies among different reference
stiffnesses. (b) Weber fractions of stiffnesses versus test stiffnesses, and (c) versus tapping
times. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
4.8 Comparison of Weber fraction of stiffnesses versus plate hardness for people who think
hard surfaces are easier to distinguish versus people who think soft surfaces are easier.
Each point indicates the average Weber fraction for all four reference stiffnesses (500 N/m,
1000 N/m, 1500 N/m, and 2000 N/m) for that plate hardness. . . . . . . . . . . . . . . . . . 65
5.1 Flow diagram of our control system. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
ix
5.2 Our ETHD system, (a) The base of our system, which is a modified haptic device to render
stiffness with a custom dynamic end-effector to render hardness. One tracker is located
on the body of the device. (b) and (c) Experiments in which the user interacts with the
end-effector through a stylus or a bare finger (note that all the colored tapes are added
only for the visual aid purpose of images and they did not exist throughout the study). . . 72
5.3 Visual scene of our experiments consisting of four yellow blocks that represent different
levels of hardness and stiffness. Participants were instructed to order the blocks from soft
to hard using a keyboard. As each block was selected, it disappeared from the blue box and
moved to the left side. The position of the participant’s finger or stylus was represented
by a red cursor, which showed its location relative to the center of the device. . . . . . . . 76
5.4 Confusion matrices indicating how participants sorted the blocks from soft to hard while
tapping with a stylus. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
5.5 These confusion matrices indicate how people sorted the blocks from soft to hard while
tapping with a finger. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
5.6 Abdominal palpation simulation. (a) Our ETHD with a custom end-effector for the
purpose of palpation training. Each surface has a different hardness and combined with a
wide range of stiffnesses rendered by the device we can simulate a variety of conditions.
(b) The virtual scene that is rendered in VR. . . . . . . . . . . . . . . . . . . . . . . . . . . 86
6.1 Recording vibration signals on different locations of a finger while tapping on metal. . . . 94
6.2 Recorded vibration signals in the time domain and frequency domain while tapping on
different materials. A1 shows the location of the accelerometer during recording. . . . . . 95
6.3 Data-driven rendering process for soft bubble actuator. . . . . . . . . . . . . . . . . . . . . 96
6.4 Data-driven rendering process for foam. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
6.5 Generated signals for each method while compensating for finger elasticity. . . . . . . . . 100
6.6 Generated signals for each method while compensating for surface properties. . . . . . . . 102
6.7 Discrimination task experiment while compensating for finger elasticity. . . . . . . . . . . 103
6.8 Matching task experiment while compensating for finger properties. . . . . . . . . . . . . 104
6.9 Experiment setup while compensating for A) finger elasticity, and B) surface properties. . 107
6.10 Discrimination task: tapping with the bubble actuator attached to the right and left
fingers: Percent of the times signal A selected as harder than B’.) . . . . . . . . . . . . . . 108
6.11 Matching task while compensating for fingertip elasticity. . . . . . . . . . . . . . . . . . . 109
x
6.12 Tapping with the bare finger on the foam: Percent of the times signal A selected as harder
than B’) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
6.13 Matching task while compensating for surface properties. . . . . . . . . . . . . . . . . . . . 111
xi
Abstract
This thesis tackles the challenge of accurately rendering the sensations of hardness and stiffness in haptic
applications, such as surgical simulation, virtual assembly, and the teleoperation of robots. Existing meth-
ods suffer from limitations in fidelity and realism when simulating hard virtual objects. I aim to provide
hardware and algorithm solutions that deliver highly accurate and realistic haptic feedback, enabling users
to perceive the sensation of hardness and stiffness in virtual objects with greater precision.
The first part of this work investigates the perception of vibration, force, and torque in tool-mediated
systems through a set of experiments designed to measure human tactile perception sensitivity. The study
examines how the thickness of dental gloves affects tactile sensitivity when using a tool. The results show
that the force and torque perceptual thresholds increase with increasing glove thickness, but the vibration
perceptual threshold is affected more by how the tool is held than by the gloves. However, the gloves do
dampen the vibrations, and the amount of damping increases with thickness.
Building upon this understanding of human perception in a tool-mediated system, I then explore the
rendering of stiffness and hardness using a haptic device. The traditional methods used to render virtual
hard objects cannot match the hardness of real objects. Therefore, a new method is presented to render
hard objects by combining augmented reality and a stylus-based encountered-type haptic display (ETHD)
using a modified haptic device to independently render stiffness and hardness. This thesis compares our
method to traditional rendering methods for rendering soft, medium, and hard objects. The results show
that our ETHD method yielded the most realistic hardness rendering.
xii
This work then evaluates how changing the hardness of the end-effector can mask the device’s stiffness
and affect the user’s perception of the interaction. The human subject study results indicate that when the
end-effector is made of a hard material, it is difficult for users to perceive the stiffness change rendered by
the device. On the other hand, this stiffness change is easily distinguished when the end-effector is made
of a soft material. These results indicate the presence of stiffness masking in our system and show that
our combined hardness-stiffness display is a promising approach to avoid the innate limitations of haptic
devices when rendering hard surfaces.
Our ETHD system is then modified by incorporating a dynamic end-effector to enable accurate and
realistic simulation of hardness and stiffness for real applications. Leveraging the physics of object inter-
actions in the ETHD device, this approach allows the dynamic end-effector to simultaneously adjust the
displayed hardness and stiffness based on the application and virtual environment requirements. More-
over, this approach provides insights into how users perceive and categorize virtual materials based on
their stiffness and hardness properties. Two experiments investigated the users’ ability to perceive differ-
ences in stiffness and hardness using the ETHD device. The results indicated that the physical hardness of
the blocks was a more noticeable perceptual feature than the rendered stiffness and that users could dis-
criminate between different levels of hardness and stiffness regardless of the method of touch. However,
discriminating hardness through tapping appeared to be easier with a stylus than using a bare finger.
Finally, this work explores the underlying events on the skin during the interaction between a bare
finger and the environment. To determine this effect, the study explored two conditions during a finger tap
event against a physical object. The researchers manipulated the finger and surface properties to under-
stand which mechanical information is exploited by the brain and explored surface hardness discrimination
and natural material identification. The results showed that the spectral content of vibration feedback con-
tains important mechanical details about the interaction. Specifically, key information is contained in the
spectral energy.
xiii
Chapter1
Introduction
Haptic feedback is a key part of our daily lives and is an essential part of how we interact with the world
around us, enabling us to navigate our surroundings and perform intricate tasks with ease. Without it, our
ability to grasp and manipulate objects would be severely impaired, and our overall sensory experience
would be greatly diminished. For this reason, the development of haptic technologies has become an
increasingly important area of research in recent years. Haptic technologies are designed to simulate
the sense of touch, allowing users to perceive the physical properties of virtual objects in a way that
closely resembles real-world interactions. By providing tactile and kinesthetic feedback, haptic devices
can enhance the user experience, improve the efficiency and safety of tasks that require precise haptic
perception, and enable the development of new forms of interaction between humans and machines.
One of the challenges in haptic technology is accurately simulating the hardness and stiffness of virtual
objects, which is essential for tasks such as medical procedures, teleoperation, and device assembly [84].
Stiffness and hardness are two properties that are often associated with real-world materials and objects
(Figures 1.1 and 1.2). Stiffness refers to the degree to which a material resists deformation in response to
an applied force and is a measure of how easily a material can be bent or compressed [14]. The stiffer the
material, the more force is required to deform it. For example, a steel bar is much stiffer than a rubber band,
as it requires much more force to bend or compress. Hardness, on the other hand, refers to a material’s
1
Figure 1.1: Illustration of the difference between two stiffness levels.
Figure 1.2: Illustration of the difference between two hardness levels.
ability to resist indentation or scratching and is a measure of the resistance of the material’s surface to
penetration by a harder object. For example, a diamond is considered one of the hardest materials because it
is extremely difficult to scratch or indent its surface. Stiffness is typically measured using Young’s Modulus
of the materials, and hardness is rated using a range of different scales including the Shore durometer
scale [8].
To accurately simulate the hardness and stiffness of virtual objects, haptic technology must replicate
the tactile and kinesthetic feedback that occurs when a user interacts with a physical object. This requires
a deep understanding of the human perception of hardness and stiffness, including how we perceive each
2
individually, and how these perceptions interact with one another. By developing a comprehensive un-
derstanding of haptic perception, we can then design haptic devices that can accurately reproduce the
tactile sensations associated with hardness and stiffness, leading to more realistic and immersive haptic
simulations.
1.1 KinestheticandTactilePerception
There are two main types of sensory perception that allow humans to gather information about the phys-
ical world: kinesthetic and tactile perception. Kinesthetic perception provides information about the po-
sition, movement, and force of body parts through receptors located in muscles, tendons, and joints [53].
This sense is essential in perceiving stiffness by providing information about the applied force. On the
other hand, tactile, or cutaneous, sensations provide information about the properties of objects through
the changes of strain senses in the skin. This sense allows individuals to perceive cues such as temperature,
texture, and pressure [77, 17, 15]. The ability to use both kinesthetic and tactile perception actively through
exploratory motions allows individuals to gather detailed information about objects and their properties.
This active exploration is crucial in understanding differences in physical stimuli [97].
The human skin contains a variety of mechanoreceptors that are sensitive to different types of physical
stimuli, including pressure, shear forces, slip, and vibration [50, 24]. Cutaneous mechanoreceptors located
in the skin convert its deformation (i.e., strain) to electrical signals and they are responsible for detecting
and transmitting haptic information to the brain [26]. There are four types of mechanoreceptors in the
skin that are distributed in varying densities and respond to different ranges of stimuli intensities and fre-
quencies. They include Merkel cells, Meissner’s corpuscles, Pacinian corpuscles, and Ruffini endings [51].
Merkel cells are located near the surface of the skin and preferentially fire due to the high-frequency sig-
nals from fine details and textures. Meissner’s corpuscles are located in the dermal papillae of hairless skin
and are triggered by light touch and low-frequency vibrations. Pacinian corpuscles are located deep in the
3
skin and are triggered by high-frequency vibrations and deep pressure. Ruffini endings are located in the
deep layers of the skin and are triggered by stretch and changes in skin position. Each type of mechanore-
ceptor has different properties that allow it to detect specific types of mechanical stimuli. Together, they
provide a comprehensive representation of the haptic environment. The information from these receptors
is transmitted to the brain through the somatosensory pathway, where it is integrated and processed to
create a perception of the object’s properties, such as its texture, shape, hardness, and stiffness [51, 67].
1.1.1 HapticPerceptionofHardnessandStiffness
The existing literature has extensively investigated the individual properties of perceived hardness and
perceived stiffness (sometimes referred to as softness [8]). Previous work has shown that the perception of
stiffness and hardness is influenced by a combination of kinesthetic and tactile cues, as well as other factors
such as visual and auditory information [8]. The perception of hardness and stiffness is also influenced
by various other factors, including the material properties of an object, the size and shape of an object,
and the intensity of the applied force [38, 66, 30]. However, relatively little research has explored the
relationship between these two properties. The perception of these properties is complex, and it is essential
to understand the factors that influence this perception to design haptic devices that can accurately render
these properties in virtual environments. This thesis aims to investigate how the haptic perception of one
property can influence the perception of the other.
Physical hardness refers to the intrinsic property of a material to resist deformation or scratching by
an external force. It is typically measured using the Shore durometer scale. Perceived hardness, on the
other hand, is the subjective sensation of hardness perceived by a human through touch or manipulation.
Perceived hardness can be different from physical hardness, as some materials may feel harder or softer
than what their physical hardness would suggest [54]. The perception of hardness relies more on tactile
cues than kinesthetic cues and is perceived most easily through tapping [42]. The main cue used for
4
perceiving the hardness of an object during tapping is the dominant frequency of the transient vibrations.
This has been demonstrated in several studies where higher frequencies were found to correspond with
greater perceived hardness [86, 58, 40].
In contrast, the perception of stiffness mostly relies on kinesthetic sensations and is perceived through
either pressing or tapping on an object [8]. When a force is applied to an object, kinesthetic feedback
indicates the amount of force being applied and the resulting deformation of the object [52].
The perception of hardness and stiffness differs between tool-based interactions and bare-finger con-
tact. Focusing on tool-mediated interactions is important because it reflects the realistic scenarios of daily
life, where many interactions with objects are mediated by tools such as pens, and utensils. While many
studies have evaluated human tactile sensitivity and perception through bare-finger contact, the pressure
map resolution on the skin for tool-mediated interactions is determined by the contact area between the
tool and fingers. This differs from the one-millimeter resolution provided by mechanoreceptors for bare
finger contact [56] and can impact our tactile sensitivity. Furthermore, our brain attributes the sensation
felt through the tool to the touched object or world, a phenomenon known as "distal attribution" [72].
Therefore it is important to understand the hardness and stiffness perception through a tool versus a bare
finger separately.
1.2 HapticRenderingAlgorithms
Haptic rendering refers to the process of creating and delivering feedback to users through haptic inter-
faces. This technology allows users to interact with virtual objects as if they were real, by simulating the
physical properties of the objects, including their hardness and stiffness. The need for the haptic render-
ing of virtual objects arises from the increasing use of technology in our daily lives. As we interact more
and more with virtual objects through devices such as computers, smartphones, virtual and augmented
reality glasses, and tablets, it becomes important to provide a more realistic and immersive experience that
5
includes the sense of touch. Haptic rendering has potential applications in fields such as manufacturing,
entertainment, medicine, and education, as it can be used for training, simulation, and tactile learning
experiences [96]
1.2.1 StiffnessRendering
To simulate the sensation of stiffness, haptic devices must accurately replicate the resistance of an object
to deformation. Kinesthetic haptic devices are designed to apply forces to the user’s hand or fingers in
response to their interactions with a virtual object [60]. These forces are generated based on the physical
properties of the virtual object, such as its stiffness, and the user’s interactions with it, such as the applied
force and direction of movement.
One common method used for stiffness rendering is based on modeling the virtual object as a simple
spring model to mimic the stiffness of the virtual object [13]. The amount of force applied is directly related
to the distance that the end-effector penetrates into the surface of the object, which can be measured as
the difference between the position of the end-effector and the closest point on the object’s surface:
⃗
F =− K⃗ x (1.1)
whereK is the stiffness of virtual object, and ⃗ x is the penetration distance. Another technique employed
in haptic rendering is the Vector Field approach [121], which represents force feedback as a field of vectors
that simulate the interaction between the user and virtual objects. This approach simplifies the compu-
tation of haptic feedback and enhances its realism. The kinesthetic haptic device is then programmed to
apply forces based on the movements of the user’s hand or fingers holding on to the device’s end-effector.
These devices use encoders to track the user’s motion in the virtual environment, detect collisions with
virtual objects, and provide force feedback to the user.
6
1.2.2 HardnessRendering
Similar to stiffness rendering, to simulate the sensation of hardness, haptic devices must be able to accu-
rately replicate the amount of force required to indent or scratch the surface of an object. This involves
detecting the deformation of the skin caused by the applied force and providing feedback that corresponds
to the level of indentation or scratchiness. Traditional haptic rendering techniques use the stiffness of
a kinesthetic device to mimic the sensation of hardness for a virtual surface. However, since hardness
perception predominantly relies on tactile cues and it is difficult to accurately replicate the sensation of
hardness using haptic devices, researchers have investigated alternative techniques to offer a comparable
experience to users. Research in this area has primarily focused on either algorithmic or device-based
approaches to enhance the perceived hardness and stiffness of virtual objects. Algorithmic approaches
are mostly event-based and involve altering the output force signals, such as by adding a high-frequency
transient force response [58], hybrid force-moment braking pulse [88], or damping [109]. Device-based ap-
proaches alter the haptic device itself, such as in haptic augmented reality (AR) [47] and Encountered-Type
Haptic Displays (ETHD) [116, 118].
Previous studies have shown that event-based algorithmic approaches, such as adding a high-frequency
transient force response, can be effective in increasing the perceived hardness of virtual objects. But there
are limits to the strength of the transients that can be displayed and the stiffness of the underlying sur-
face [58]. Researchers have also used tactile cues to increase perceived hardness, which has been found
to be more effective than kinesthetic cues [10, 86, 58, 35], but do not provide the same level of detail and
realism as true haptic feedback.
The integration of haptic augmented reality (AR) has been also been explored as a means of enhancing
the realism of virtual objects. A haptic AR framework developed by Jeon et al. [47] involves additional
virtual force feedback to modulate a real 3D object’s stiffness. However, this approach has limitations,
7
since it only works with moderately stiff real objects and assumes that they have a uniform dynamic
response.
This thesis aims to address the unsolved and challenging problem of realistic hardness rendering and
the interplay between hardness and stiffness perception.
1.3 HapticInterfacesforRenderingHardnessandStiffness
A haptic interface is a device that enables users to interact with a virtual environment by providing tactile
or kinesthetic haptic feedback. The haptic feedback can be rendered as forces, vibrations, or other physical
sensations to the user’s body. A haptic interface requires sensors to detect the user’s input and to generate
output that is dependent on the user’s input. The mechanical signals that are generated by the haptic
interface can stimulate human kinesthetic and/or tactile channels [39].
One of the primary challenges in haptic interface design is creating realistic and accurate haptic feed-
back. This involves accurately simulating the physical properties of virtual objects, such as their stiffness,
hardness, or texture. These mechanisms can range from simple vibration motors to more complex force
feedback systems that provide users with a range of tactile sensations, such as pressure, texture, or tem-
perature.
A kinesthetic force-feedback device uses motors or other mechanisms to apply forces to the user’s hand
or fingers during interaction with virtual environments. These devices are commonly used in various
applications such as virtual reality, gaming, teleoperation, and medical simulations. Kinesthetic force-
feedback devices can be classified as either impedance-type or admittance-type based on their control
system. In impedance-type control, the haptic device applies a force that is proportional to the difference
between the desired motion and the actual motion of the device. Admittance-type control, on the other
hand, applies a motion that is proportional to the force applied to the device [115].
8
Admittance-type force-feedback devices and impedance-type force-feedback devices both have their
advantages and disadvantages for rendering stiffness and hardness in haptic systems. Admittance-type
devices have the advantage of being more stable and providing high-force feedback. These devices are often
better at rendering stiffer and harder objects since they can quickly respond to changes in the environment.
However, these devices have a few drawbacks. They are generally more expensive, and they can also have
more significant friction and inertia, which can limit their responsiveness and make it difficult to simulate
free space and very fine movements. On the other hand, impedance-type devices have less friction and
inertia, which makes them ideal for applications that require fine movement control with no resistance.
They are also generally less expensive than admittance-type devices. However, impedance-type devices
have some limitations when it comes to stiffness and hardness rendering, and they are unable to provide
as high of a force as admittance-type devices. This can limit their effectiveness in applications where users
need to feel a high level of force feedback.
Tactile interfaces offer unique advantages over traditional kinesthetic force-feedback devices. While
the latter is often stationary and requires a dedicated workspace, tactile interfaces can be used in a variety
of settings and can provide haptic feedback in real-world contexts. Wearable haptic devices are a common
type of tactile interface. These devices are beneficial because the user does not have to hold a grounded
stylus and allows for free movement without being tethered to a stationary setup. Hardness and stiffness
rendering is also required in wearable haptic interfaces, especially in virtual reality gloves. These gloves
use sensors to detect hand and finger movements and provide haptic feedback in response based on the
virtual objects [61]. HaptX gloves G1 [36] and Meta haptic gloves [31] are designed to render stiffness using
pneumatic actuators on the fingertip. However, these devices have a relatively low force output, which may
not be sufficient to accurately simulate harder objects. SenseGlove Nova gloves are created to provide both
kinesthetic and tactile feedback using magnetic friction brakes and vibrotactile actuators [98]. However,
these gloves use strings that pull the fingers and the sensation is different than pushing an object [104].
9
Therefore in this thesis, we work to find a solution to overcome these limitations of hardness rendering
in both kinesthetic haptic devices and wearable type devices.
1.4 Contributions
The goal of this thesis is to investigate haptic perception and rendering in relation to hardness and stiff-
ness. The aim is to explore different methods and technologies for accurately simulating and rendering
these properties in haptic displays, with the ultimate goal of improving haptic experiences and enhancing
training and simulation applications. Through a series of experiments and analyses, this thesis seeks to
contribute to the understanding of the role of haptic cues in hardness and stiffness perception, as well as
the design of effective haptic rendering systems. By addressing key challenges and limitations in current
methods, this thesis aims to provide insights and recommendations for future research and development
in this field.
Chapter 2 focuses on exploring the perception of force, torque, and vibration in tool-mediated sys-
tems, which is particularly relevant since the interaction with virtual objects in kinesthetic haptic devices
is through a stylus. The experiments conducted in this chapter aim to investigate human tactile perception
sensitivity in a real-life context where high sensitivity is required. Specifically, the study examines the im-
pact of dental glove thickness on tactile sensitivity when using a tool, by measuring perceptual thresholds
for force and torque, as well as the vibrotactile perception threshold while holding a tool.
In Chapter 3, we propose a novel encountered-type haptic display system that can simultaneously and
separately render hardness and stiffness. We compare our method with well-known rendering methods
for rendering different common materials at varying hardness levels, which we find a higher realism in
our method.
10
Chapter 4 investigates the perceptual basis behind the high realism of our ETHD method compared
to other methods and studies how the hardness of the end-effector can mask the stiffness provided by the
device.
Chapter 5 introduces a dynamic end-effector for our ETHD to render both hardness and stiffness in
a real application. We show that our approach provides insights into how users perceive and sort virtual
materials based on their stiffness and hardness properties. Our results show that participants can easily
discriminate between the varying levels of hardness, regardless of the stiffness level of the blocks. However,
there is a potential impact of the method of touch, where material properties like hardness and stiffness is
easier to distinguish when tapping with a stylus than with a bare finger
In Chapter 6, we explore hardness perception through bare-finger interactions. We show that the en-
ergy spectrum of the vibration signal produced by tapping on a material can be used as a cue for its hardness
perception. Our experiments demonstrate the effectiveness of the proposed approach in accurately and
reliably distinguishing between different levels of hardness.
11
Chapter2
EffectsofDentalGloveThicknessonTactilePerceptionThroughaTool
Many studies have evaluated human tactile sensitivity and perception with direct finger contact. However,
in many applications, like dentistry or orthopedics, interaction with objects is through a tool [90]. When
we touch a surface with a bare finger, the mechanoreceptors provide about one-millimeter pressure map
resolution on the skin [56]. However, for tool-mediated interactions, this pressure map is the contact area
between tool and fingers [56], which affects our tactile sensitivity. Our brain attributes the sensation that
we feel through the tool to the touched object or world in a phenomenon called “distal attribution" [72].
We are also interested in evaluating the effects of glove thickness in clinical conditions, in which clinicians
rely significantly on tactile sensations during procedures and diagnoses. For instance, dentists use tactile
sensations felt through their tools to detect and diagnose carious lesions. Based on tactile exploration,
dentists can identify the roughness of the tissue and apply a specific amount of force to the tissues through
the tools [57, 49, 94].
Therefore, in this chapter, we perform a psychophysical study to evaluate the role of glove thickness
in tactile sensitivity while holding a dental tool. We first record the acceleration transferred through each
glove to determine its impact on tactile sensitivity. We then assess the force and torque perceptual thresh-
olds by applying force with a monofilament to the dental tooltip, and finally we measure the vibrotactile
perception threshold.
12
The goal of this chapter is to provide a deeper understanding of tactile perception while holding a tool
to help in designing dental and medical haptic simulators to match the required sensitivity (for instance, in
choosing the actuators or designing the control signals). Also, these studies assist clinicians and dentists
in making an informed decision on the gloves to use in clinical practice. One must balance the tactile
sensitivity needed for the task and the strength of the gloves required for safety. This chapter presents
work published in the Proceedings of the IEEE World Haptics Conference [117].
2.1 Background
A large effort has been made towards implementing universal precautions in medical fields. Towards this
goal, medical gloves are strongly recommended by national guidelines [34]. Gloves provide a barrier to
prevent the transference of microorganisms between the patient and the healthcare professional. However,
it is essential to consider their effect on tactile sensitivity and performance [81, 62]. The potential risks of
not wearing gloves during medical procedures are unquestioned, but many physicians prefer to not wear
gloves during sensitive tasks [29, 81] because minor errors due to decreased sensitivity could cause serious
injuries [82].
A prior study indicated a difference in the tactile performance of clinicians for different glove types [81],
but other studies failed to find such a difference [108, 5]. The standard methods used in these studies are
Semmes-Weinstein Monofilament Tests (SWMT) and Roughness Discrimination Tests (RDT). However,
these methods are artificial and do not represent the conditions clinicians experience in real life. In a recent
study, the authors used Simulated Medical Examination Tactile Tests (SMETT) to analyze participants’
ability to detect changes in geometry and stiffness [80]. Though there are conflicting results concerning
the effect of gloves on tactile sensitivity, a survey to 28 clinicians found that 68% of clinicians thought that
there was a link between thickness and cutaneous sensibility, with some prioritizing sensitivity over tear
resistance [81]. Clinicians must make decisions such as this in balancing safety with their ability to do
13
their job. For example, nitrile gloves, which are commonly used due to latex allergies, have a higher rate
of tearing in dental procedures compare to both latex and non-latex gloves [79].
2.2 ExperimentalMethods
In this study, we compare tactile sensitivity while holding a tool for different glove thicknesses. The study
consisted of two phases: (1) we recorded the amount of vibration transmission through each glove, (2) we
determined the threshold of perceivable force, torque, and vibration.
2.2.1 Experimentaldesign
2.2.1.1 Participants
In total, 15 volunteers (ten female, five male, 22-35 years old) participated in this study. All but one
were right-handed. The study was approved by the University of Southern California Institutional Re-
view Board.
2.2.1.2 Gloves
Due to the risk of latex allergies, nitrile gloves are more commonly used in hospitals and dentist offices.
Thus, we use nitrile gloves of varying thickness in this study:
• GloveOn Eureka, 0.05 mm thick
• Adenna Precision, 0.09-0.12 mm thick
• Adenna Empower, 0.2 mm thick
In this chapter, we refer to these gloves as “thin", “medium", and “thick", respectively. There are five
standard sizes ranging from X-small to X-large. The participants selected the size with the best fit for each
glove type.
14
2.2.1.3 Variables
Four conditions were tested in each experiment: bare hand, thin glove, medium glove, and thick glove.
2.2.1.4 DentalTools
We evaluated the perceptual thresholds while the participants held one of two standard dental tools.
• Dental scraper (tip: SU 15-33, OD=10 mm, length= 172 mm, mass=15 grams). Used for scraping
tartar off teeth. The amount of force applied by this tool to the tooth is important, so we use this
tool to evaluate the force and torque thresholds.
• Dental explorer (tip: 5H, OD=6 mm, length=172 mm, mass=23 grams). Used for probing teeth. This
task requires good transference of vibration, so we use this tool to evaluate the vibration thresholds.
2.2.1.5 Force/TorqueApplication
For applying force, 20 piece Aesthesio Precision Tactile Sensory Evaluators (Semmes-Weinstein Monofila-
ments (SWM)) were used. SWM is a common method for testing touch detection threshold by applying a
controlled force. These monofilaments bend proportional to their diameter and length. Their sizes range
from 1.65 to 6.65, which are the logarithm of 10 times their target force (0.008 - 300 grams) [2]. The range
of force we apply is less than 10 grams. In this range, an electromechanical device or force sensor has high
noise, and unlike monofilaments, would not be reliable enough for creating a repeatable force. Also, for
avoiding mechanical noise, applied force by the monofilaments was done by hand, instead of connecting
it to a mechanical device.
A small cylindrical piece of plastic was attached to both tooltips of the scraper to create a flat surface
for applying force. On one side, the flat surface was aligned with the axis of the tool for applying force by
the monofilament. On the other side, the flat surface was aligned perpendicular to the axis of the tool for
15
applying torque. The acting lever-arm for the applied torque was the distance between the tooltip and the
grip point, which was on average 75 mm.
2.2.1.6 VibrationApplication
A Haptuator vibrotactile transducer (model no. TL002-14-A, TactileLabs Inc.) was attached 50 mm from
the tip of the dental explorer to generate vibrations. The frequency of the generated vibration signal was
100 Hz with an amplitude 2.5 m/s
2
. This signal was chosen to lie within the frequencies at which the
human sense of touch is most sensitive [46], and the amplitude was chosen to be easily perceivable to the
participant.
2.2.1.7 ExperimentSetup
The participant was seated and wore headphones playing white noise to ensure that any external sounds
did not affect their perception. A visual barrier with a cutout for their arm prevented the participant from
seeing their hand during the experiment.
2.2.2 VibrationRecordingProcedure
Nine participants (four male, five female) completed the vibration recording phase of the study, shown
in Figure 2.1. This study phase was used to determine the effect of glove thickness in dampening out
tool vibrations by measuring the reduction in power of the vibrations felt by the participant. Vibrations
were recorded using a± 16 G, 3-axis analog accelerometer (ADXL 326); no input was required from the
participant. The 500 Hz bandwidth accelerometer was attached to the tip of the participant’s index finger
with tape. The participant was asked to hold the dental explorer with the vibrotactile transducer connected
to its tip, while wearing gloves of varying thickness or while wearing no gloves.
16
Periodontal probe
Accelerometer Support plate
Force sensor
Vibrotacle transducer
Figure 2.1: Experimental setup for vibration recording procedure. Vibrations were displayed through a
voicecoil attached to the dental tool, and participants were required to apply a constant pressure to the
force sensor.
Pressing force is important in active touch because it can affect vibration detection thresholds [87]. A
change in applied force would cause a corresponding change in grip force, affecting the recorded accelera-
tion. Thus, we asked participants to hold the tool so that its tip touched a plate connected to a force sensor
(Mini 45, ATI Industrial Automation). A visual indicator on a monitor was displayed to the participant,
and they were asked to maintain the force within a certain range by keeping the indicator’s color green.
The indicator turned yellow or red if the participant pressed too softly (<1.65 N) or too hard (>1.95 N),
respectively.
We applied a 20 Hz high-pass filter to the three acceleration signals (X, Y, and Z) to remove gravity
and effects of human motion. We then mapped these three signals onto a single axis using the DFT321
algorithm, which maintains the spectral and temporal properties of these signals [64]. Combining the
signals was necessary to determine the true power of the vibrations and is possible because humans cannot
discriminate the direction of high-frequency vibrations [6].
17
(a) (b) (c) (d)
Figure 2.2: Experimental setup for determining perceptual thresholds for (a) force, (b) torque, (c) vibration
through the tool, and (d) vibration with direct contact. In each subfigure, one of the glove conditions is
shown. A small plastic piece was used to create a flat surface for applying force by the SWM.
2.2.3 PerceptionTestProcedure
In the perceptual threshold phase, 15 participants completed the first three experiments (A, B, C) in pseudo-
random order.
ExperimentsAandB: We assessed the effect of glove thickness on perceptual thresholds of force (Exp.
A) and torque (Exp. B). Participants held the dental scraper in midair while wearing no gloves or gloves of
varying thickness. The order of the glove condition was pseudo-randomized. Participants were instructed
to hold the tool like a pencil. Forces were applied to the tooltip using the monofilaments, either in-line
with the tool (simulates force, Figure 2.2-(a)) or perpendicular to the tool (simulates torque, Figure 2.2-(b)).
The experimenter applied each force for three seconds before the participant responded either “Yes” or
“No” to indicate whether they could perceive the applied force or torque. The experimenter then waited
5 seconds before applying the next force following a staircase method. The applied stimulus begins at the
smallest sized monofilament that could not be detected (1.65); the monofilament size is then repeatedly
increased until the participant could detect the touch. Next, the size of the monofilament is decreased over
successive trials until the participant indicates they could not detect it. This process was repeated for five
reversals, and the average of these reversals is recorded as the perceptual threshold.
Experiment C: We also assessed the effect of gloves on vibration perception (Figure 2.2-(c)). Partic-
ipants held the dental explorer in midair while wearing no gloves or gloves of varying thicknesses. The
18
order of the glove condition was pseudo-randomized. A Haptuator was attached to the end of the tool to
generate 100 Hz vibrations with varying amplitude. The lowest vibration amplitude was 0.01 m/s
2
, which
was lower than the minimum detectable amplitude at 100 Hz [12], and the vibrations varied in increments
of 0.01 m/s
2
following a staircase method. The vibration was played to the participant for three seconds
before they responded indicating if they could perceive the applied vibration. This procedure was repeated
for five reversals, and the average of these reversals is recorded as the perceptual threshold.
ExperimentD: This experiment assessed the effect of tool grasping on vibration perception. Six par-
ticipants completed this experiment at a later time. This task’s procedure was the same as the Exp. C, but
the participant held the vibrotactile transducer directly in a two-fingered pinch grasp (Figure 2.2-(d)).
Participants completed a post-experiment survey after each procedure (Exp. A-D), rating their diffi-
culty in perceiving the signals for each glove condition.
2.3 Results
2.3.1 ToolGrasping
Participants were instructed to hold the tool like a pencil. Because grasp configuration is unique to each
person and an uncomfortable grasp might affect the results, we did not constrain the participant’s grasp.
We recorded an image of the participant’s hands while holding the tool. Visually comparing the tool
contact points, we observed three main groups between their grasps. These groups were self-selected
based on the participants’ chosen grasp. Other than a contact point between the tool and fingertips, a
second contact point on hand was among one of the three areas shown in Figure 2.3.
19
1
2
3
Figure 2.3: Tool contact points on the hand. Participants were split into three contact point groupings
based on images recorded during the study.
2.3.2 RecordedVibration
To determine the amount of vibration dampening caused by the gloves, we computed the power of each
recorded vibration signal:
P =
s
P
(A
i
2
)
N
(2.1)
whereA is the vibration amplitude andN is the signal length. Figure 2.4 shows the power of the recorded
vibration for each glove condition.
A one-way ANOVA on the vibration signals’ power with glove condition as the factor indicated that
the vibration signals’ power was statistically different across the different glove conditions ( F(3,56) =
8.24,p < 0.001,η 2
= 0.44). A Tukey’s post-hoc pairwise comparison test further evaluated the effects
of the glove type on vibration power. Vibration power was statistically different between bare hand and
all other gloves (thin (p = 0.005), medium (p < 0.001), and thick (p = 0.01)), but there is no statistical
significance among different glove types ( p≥ 0.1).
20
Bare hand Thin Medium Thick
Glove Condition
0 4
0 6
0 7
0 5
0 8
0 9
**
***
*
(m/s
2
) Power of Recorded Vibration
Figure 2.4: Comparison of the power of recorded vibrations across different glove conditions. Statistically
significant differences marked (∗∗∗≡ p≤ 0.001,∗∗≡ p≤ 0.01,∗≡ p≤ 0.05).
2.3.3 ForcePerception
Figure 2.5 shows the calculated force threshold for each glove condition, and the mean thresholds are
shown in Table 2.1. A one-way ANOVA on the force threshold with glove condition as the factor indi-
cated that the perceived force threshold was statistically different across the different glove conditions
(F(3,56) = 3.05,p = 0.04,η 2
= 0.14). A Tukey’s post-hoc pairwise comparison test further evaluated
the glove type’s effects on the force perception threshold. It was found to be statistically different between
the bare hand and the thick gloves (p = 0.03), but no difference between the force thresholds of any other
pairs (p≥ 0.05).
In the post-experiment survey, participants rated the difficulty of distinguishing forces for each glove
condition on a 5-point Likert scale. The means of these ratings are shown in Table 2.2. A one-way ANOVA
on the difficulty ratings with glove condition as the factor indicated that the rated difficulty was statistically
different across different glove conditions ( F(3,56) = 10.4,p < 0.001,η 2
= 0.41).A Tukey’s post-hoc
21
Bare hand Thin Medium Thick
Glove Condition
0
1
2
3
4
5
6
7
8
9
10
Force Threshold (grams)
*
Figure 2.5: Comparison of the force perception thresholds across different glove conditions. Statistically
significant differences marked (∗≡ p≤ 0.05).
Table 2.1: Mean perceptual thresholds
Gloves Bare handBare hand ThinThin MediumMedium ThickThick
mean std mean std mean std mean std
Force (grams) 0.6 0.4 1.6 1.5 1.9 1.7 2.2 2.0
Torque (grams) 0.3 0.2 0.6 0.4 0.8 0.7 1.2 0.7
Vibration (m/s
2
) 0.06 0.03 0.08 0.04 0.08 0.04 0.09 0.04
pairwise comparison test further evaluated the glove type’s effects on the rated difficulty of force percep-
tion. Glove type was statistically significant between the thick gloves and all other glove conditions (bare
hand (p < 0.001), thin (p = 0.003), and medium (p = 0.002)), and between bare hand and the medium
glove (p = 0.01).
Table 2.2: Mean ratings for difficulty of distinguishing sensations
Gloves Bare handBare hand ThinThin MediumMedium ThickThick
mean std mean std mean std mean std
Force 2 0.8 3 0.9 3 0.8 4 0.8
Torque 2 0.8 3 1 3 0.9 4 0.9
Vibration 2 0.9 3 0.9 3 1 4 0.7
22
Bare hand Thin Medium Thick
Glove Condition
0
0.5
1
1.5
2
2.5
3
3.5
Torque Threshold (grams)
*
***
Figure 2.6: Comparison of the torque perception thresholds across different glove conditions. Statistically
significant differences marked (∗∗∗≡ p≤ 0.001,∗≡ p≤ 0.05).
2.3.4 TorquePerception
Figure 2.6 shows the calculated torque thresholds for each glove condition, and the mean thresholds are
shown in Table 2.1. A one-way ANOVA on the torque threshold with glove condition as the factor indicated
that the threshold was statistically different across the different glove conditions ( F(3,56) = 6.61,p <
0.001).A Tukey’s post-hoc pairwise comparison test further evaluated the glove type’s effects on the torque
perception threshold. It was found to be statistically different between the bare hand and the thick glove
(p < 0.001), and between the thin and thick glove (p = 0.04), but not between the torque thresholds of
any other condition (p≥ 0.05).
In the post-experiment survey, participants rated the difficulty of distinguishing torques for each glove
condition on a 5-point Likert scale. The means of these ratings are shown in Table 2.2. A one-way ANOVA
on the difficulty ratings with glove condition as the factor indicated that the rated difficulty was statistically
different across different glove conditions ( F(3,56) = 13.07,p < 0.001,η 2
= 0.36).A Tukey’s post-hoc
23
Bare hand Thin Medium Thick
Glove Condition
Vibration Threshold (m/s
2
)
0.24
0.2
0.16
0.12
0.08
0.04
Figure 2.7: Comparison of the vibration perception thresholds through a tool across different glove condi-
tions.
pairwise comparison test further evaluated the effects of glove type on the difficulty of torque perception.
The rated difficulty was statistically different between the thick glove and all other conditions (bare hand
(p < 0.001), thin (p = 0.003), and medium (p = 0.002)), and between bare hand and medium glove
(p = 0.01).
2.3.5 VibrationPerceptionThroughtheTool
Figure 2.7 shows the calculated vibration threshold for each glove condition, and the mean thresholds
are shown in Table 2.1. A one-way ANOVA on the vibration threshold with glove condition as the
factor indicated no statistical difference in the vibration threshold across the different glove conditions
(F(3,56) = 1.25,p = 0.3,η 2
= 0.06).
Figure 2.8-(a) shows the average vibration perception threshold for each glove condition separated
into the tool contact location groups shown in Figure 2.3. Figure 2.8-(b) shows the calculated vibration
thresholds for each of the tool contact locations. A one-way ANOVA on the vibration threshold with the
24
Bare hand Thin Medium Thick
Glove Condition
Vibration Threshold (m/s
2
)
0.18
0.16
0.14
0.12
0.1
0.08
0.06
Location 1
Location 2
Location 3
(a)
1 2 3
Location
Vibration Threshold
0.2
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
***
***
(m/s
2
)
(b)
Figure 2.8: (a) Average vibration threshold separated by tool contact location. (b) Comparison of the
vibration perception thresholds across tool contact locations. Statistically significant differences marked
(∗∗∗≡ p≤ 0.001).
tool contact point as the factor indicated that the vibration threshold was statistically different across the
different tool contact locations ( F(2,57) = 22.0,p< 0.001,η 2
= 0.44).
We ran a Tukey’s post-hoc pairwise comparison test to further evaluate the effects of the tool contact
point on the vibration perception threshold. The threshold was found to be statistically different between
the first location and both the second ( p < 0.001), and the third (p < 0.001) locations. There was no
statistical difference between the second and the third locations ( p = 0.1).
In the post-experiment survey, participants were asked to rate the difficulty of distinguishing vibrations
for each glove condition on a 5-point Likert scale. Their means are shown in Table 2.2. We performed a one-
way ANOVA on the difficulty ratings with glove condition as the factor. This was found to be statistically
different across different glove conditions ( F(3,56) = 10.8,p < 0.001,η 2
= 0.37).We ran a Tukey’s
post-hoc pairwise comparison test to further evaluate the effects of glove type on the rated difficulty of
vibration perception. The rated difficulty was found to be statistically different between bare-hand and
both medium (p = 0.02) and thick (p =< 0.001) gloves, and between the thin glove and the thick glove
(p = 0.01).
25
Bare hand Thin Medium Thick
Glove Condition
Vibration Threshold (m/s
2
)
0.16
0.14
0.12
0.1
0.08
0.06
0.04
0.02
*
Figure 2.9: Comparison of the vibration perception thresholds in direct contact across the different glove
conditions. Statistically significant differences marked (∗≡ p≤ 0.05).
2.3.6 DirectContactVibrationPerception
Figure 2.9 shows a box plot of the calculated vibration threshold for each glove condition in the direct
contact experiment. A one-way ANOVA on the vibration threshold with glove condition as the factor
indicated that the perceived vibration threshold was statistically different across the different glove con-
ditions (F(3,56) = 3.73,p = 0.03,η 2
= 0.44). We ran a Tukey’s post-hoc pairwise comparison test to
further evaluate the effects of glove type on the vibration perception threshold. The vibration threshold
was found to be statistically different between the bare hand and the thick glove conditions ( p = 0.02),
but not between any other pairs (p≥ 0.05).
2.4 Discussion
The results from the force and torque perception experiments show a clear correlation between glove
thickness and the force and torque perception thresholds. Although there was not a significant difference
26
between the force and torque perception thresholds of the bare hand and thin or medium gloves, Figs. 2.5
and 2.6 show a trend of increasing threshold with increasing glove thickness. We note that the thickest
gloves in the experiment were thicker than is typically used in dental practice, a decision supported by the
decreased perception.
The vibration power was significantly larger for the bare-hand condition than for all glove conditions.
This power decrease indicates that the gloves act as a barrier, dampening the vibrations, but the level
of damping does not significantly vary with thickness. The results of our initial study of the vibration
perceptual threshold (Exp. C) did not match the results of the measured vibration power difference with
gloves. Since gloves reduce the vibration power, we expected to see a similar increase in the perceptual
threshold. However, the results of this study were muddled by the different grasps that participants used.
Separating the data by grasp type, we found that tool contact location affected the perceptual threshold
more than glove type. This motivated us to complete a follow-up study to measure the perceptual threshold
of the vibrations without the tool, asking the participant to grasp the vibration actuator directly (Exp. D).
Figure 2.9 shows the trend we expected with the perceptual threshold increasing with increasing thickness.
The results of these two experiments indicate that a clinician can increase their vibration sensitivity by
altering their tool grip to mitigate the loss of sensitivity caused by the gloves.
The difficulty ratings for distinguishing force, torque and vibration show that participants did not
perceive a significant difference between the thin and medium gloves, but did notice a difference between
the other conditions.
Our results indicate that the thickest gloves remove too much sensitivity to be effective for use in
clinical tasks. However, these gloves are also the safest and least likely to tear. This shows the trade-off that
clinicians must make between safety and effectiveness when choosing their gloves. Of the gloves tested
in our study, the medium gloves strike a good balance because there is not a large change in sensitivity
between them and the thinnest gloves.
27
In addition to the glove thickness, there are many other conditions in the study that may affect the
user’s perception. In a post-experiment survey, five participants mentioned that a tighter grip increased
their perception, while four participants mentioned that a looser grip increased sensitivity. Three partic-
ipants indicated that touching a larger part of the tool would increase perception. Another participant
mentioned that wearing a tight glove increased pressure on the fingers and decreased sensitivity. Most
gloves come in the five standard sizes used in this study, but there is infinite variation in user hand size.
As such, there is a high possibility that clinicians would not be able to find a glove that fits well in all areas
of the hand, which would likely affect their perception.
2.5 Summary
This chapter evaluated the effect of glove thickness on tactile sensitivity while holding a dental tool. Our
results showed that increasing glove thickness causes an increase in force and torque perceptual threshold
and an increase in vibration dampening that can affect perception. We also concluded that tool contact
location influences the perceptual threshold of vibration more than glove thickness. The results from this
study can be used to inform clinicians and dentists in choosing the best gloves for their needs.
28
Chapter3
CombiningHapticAugmentedRealitywithaStylus-Based
Encountered-TypeDisplaytoModifyPerceivedHardness
The work of the previous chapter, which focused on human perception of force, torque, and vibration in
tool-based interactions, provides a foundation for the work in this chapter, which aims to enhance the
haptic rendering of hardness in tool-based interactions.
Precise hardness rendering is required in teleoperation [109], device assembly [1, 95], and medical
training in dentistry [32], orthopedics [119, 89, 120], or dermatology [20]. Force-feedback haptic devices
are often used in simulating these tasks. The accuracy of haptic rendering depends partially on how
closely the haptic device matches our ideal assumptions that the device moves effortlessly without friction
and inertia when there is no virtual contact, and provides high force when there is a stiff virtual wall.
With traditional haptic rendering methods, the stiffness of a kinesthetic device is used to simulate the
hardness of a virtual surface. When rendering a stiff virtual wall, time-discretization, quantization of
encoder measurements and commanded force, and limited bandwidth can create an unstable system, which
the user might feel as an oscillation [27]. Due to these issues, the maximum stiffness that can be stably
rendered using a kinesthetic device is limited.
Often an object’s hardness and stiffness are lumped together in haptic rendering, but these are two
distinct properties that uniquely contribute to the perception of the object. A material’s hardness is the
29
resistance towards indentation or localized deformation, whereas stiffness is the resistance of its elastic
body to the deformation. An object’s stiffness can also be represented by the material’s Young’s modu-
lus, which is the ratio between pressure and displacement [107]. Although Young’s modulus is large in
hard materials, there is no direct relationship between the two properties. The hardness of a material is
measured using hardness tests, such as the Rockwell, Brinell, and Vickers tests. In [41], they found that
hardness perception is likely based on the relationship between the force acting on the object’s surface and
its surface vibration. Due to this relationship, tactile cues play an important role in hardness perception,
unlike stiffness which relies on kinesthetic cues. Haptic devices provide only global force and motion and
do not provide any sensory information about local object deformation or surface vibration cues. There-
fore, hardness must be rendered separately and in addition to stiffness, or alternative methods must be
used to increase perceived hardness during rendering.
This chapter presents a novel approach for rendering hard objects using an off-the-shelf impedance
haptic device that separately renders hardness and stiffness. Our approach uses an encountered-type haptic
display (ETHD), which allows the user to move freely when not in contact with a virtual object. To create
an ETHD, we modified a haptic device by attaching a hard plate to its end-effector. The user then interacts
with this plate using an untethered stylus. We then compare our ETHD method with well-known rendering
methods, including spring, spring damper, and acceleration-matching event-based transient force models
(Fig. 3.1) for rendering three different common materials at different hardness levels. This chapter presents
work published in the Proceedings of the IEEE Haptics Symposium [116].
3.1 Background
Researchers have worked on solving the issue of rendering stiffer virtual objects by either improving the
device or improving the rendering algorithms (Figure 3.1). One of these methods, virtual coupling, in-
creases stability by generating energy into the system when penetrating a stiff object [19]. This method
30
F orce
(a ) (b) (c ) (d)
Figure 3.1: (a) Real object, (b) position-based virtual object, (c) event-based virtual object, and (d) ETHD
virtual object.
restricted the actuators’ force, but could not render a very stiff object due to the computational load. In
[55], the authors used a passivity-based approach to bound the output energy and created a stiffer and
more stable virtual object, but could not reach the stiffness of a real object. Authors have also explored
novel rendering algorithms to increase the stiffness of virtual objects. One solution gradually increased the
stiffness as close as possible to the desired value while the physical damping of the device dissipated the
generated energy [102]. This method increased stiffness but decreased rate-hardness, a metric for measur-
ing hardness perception [65]. The method was later improved to increase stiffness without significantly
reducing rate-hardness [101].
Event-based haptic rendering has also been used to improve perceived hardness [58]. During real-
world interactions with hard objects, a high-frequency transient force occurs at contact; it has been shown
that simulating this high-frequency transient in virtual interactions increases realism and perceived hard-
ness [88, 58, 86, 35]. In event-based rendering, open-loop control is used during the initial contact between
31
the user’s hand and the virtual object. A high-frequency acceleration is generated over a very short du-
ration and cancels the user’s momentum into the object. Some of the methods for creating event-based
transients include decaying sinusoids [114], short-duration pulses [43], and acceleration matching [58].
Among them, acceleration matching has been shown to have the best performance in rendering hard ob-
jects. Although this method improves hard object rendering, there is a limit to the strength of the transients
that can be displayed and the stiffness of the underlying surface (due to the limit of the haptic device’s mo-
tors) [58].
Haptic augmented reality (AR) is another method that has been used to improve the realism of virtual
objects. In this method, the user interacts with a real object, which is overlaid with virtual haptic signals.
Jeon et al. created an AR system that augmented the stiffness of a real object by applying additional forces
using a kinesthetic device [47]. One issue with this previous method is that only real objects with moderate
stiffness could be used because of the limited force-feedback of current haptic devices, and the system could
not render sufficiently hard objects.
ETHDs create a tangible surface in front of the user when feedback is required [76]. With an ETHD,
the user only touches the end-effector when in contact with a virtual object. This can be a solution for
removing unwanted friction and inertia when simulating free space using a haptic device. Previously,
ETHDs have been used to create on-demand tangibles in VR. For example, in [4] and [75], authors attach
different textures to the surfaces on the end-effector of a robot. Researchers in [106], created HapticBot to
make ETHDs more scalable and deployable in VR. A full review of previous grounded ETHDs is presented
in [76]. These previous ETHDs have been created using admittance robotic arms, and their purpose was
to create on-demand tangibles in VR.
32
X
Z
Y
Figure 3.2: Encountered-Type Haptic Display created from kinesthetic haptic device and unthethered sty-
lus.
3.2 Encountered-TypeHapticDisplayDesign
Building on the benefits of ETHDs and AR, we use a stylus-based ETHD system to simultaneously render
hardness and stiffness. Due to the decoupled nature of ETHDs, we have a virtual stiffness (controlled by
the haptic device) and a real hardness (due to the collision impact between the stylus and the device’s end-
effector). Our design transforms a traditional kinesthetic haptic device into the base of an ETHD system.
With this design, we eliminate the effects of friction and inertia during free-space rendering. The haptic
device could be replaced with any device that provides a high vertical force, including an admittance device.
3.2.1 HardwareDesign
We modify a Novint Falcon haptic device by detaching the stylus from the body and using the end-effector
as the primary interaction point between the stylus and the body, as shown in Figure 3.2. Surfaces of
33
0 1000 2000 3000
Desired Stiffness (N/m)
0
1000
2000
3000
Actual Stiffness (N/m)
Commanded Stiffness
Output Stiffness Before Calibration
Output Stiffness After Calibration
Figure 3.3: Desired stiffness versus actual stiffness provided by the device before and after calibration.
different hardnesses can be directly attached to the end-effector, as discussed below. The Falcon device
is an inexpensive haptic device with limited force rendering capacity, making it impossible to perfectly
render a stiff material. We chose to work with the Falcon device to test our methods for improving the
rendering capabilities of a device with limitations; our methods can easily be applied to other kinesthetic
devices. The maximum force that this device can provide is in its X-direction, normal to the body of the
device. Since our goal is to study the relationship between stiffness and hardness, we expect the primary
interaction behavior of our users will be tapping on the surface [63]. Therefore, we rotate the device 90
degrees to maximize the output force in the vertical direction. We then add a new gravity compensator to
our system in the X-axis to negate the effect of the weight of the end-effector.
This design eliminates the effect of friction and inertia of the joints and links of the device while the
user interacts with free space. The system design is also not device-dependent and does not require an
impedance device. The kinesthetic device itself could be replaced with any device that allows vertical
application of force, including an admittance device that may provide higher force and stiffness.
34
3.2.2 EvaluationandCalibrationoftheHapticDevice
Since the Falcon haptic device uses open-loop control, we need to ensure the device’s output force is close
to our commanded force. We evaluated and manually re-calibrated the device after the auto calibration
created by Force Dimension for this device. We placed a constant weight (100 g) on the end-effector and
increased the stiffness from 100 N/m to 4000 N/m in increments of 100 N/m. We recorded the distance
traveled by the end-effector after each increment using the device’s encoders. For each stiffness, we ran
this experiment and recorded the position five times, averaging the distance traveled. The actual stiffness
was calculated by dividing the constant weight by the average traveled distance.
Figure 3.3 shows the results from the calibration. The maximum stiffness the device could provide
was around 2000 N/m; the plot shows a saturation after this point. Therefore, we use 2000 N/m as the
maximum stiffness for our experiment to ensure the device output is accurate.
Figure 3.3 shows a falsely low output stiffness for all commanded stiffness values. We fit a second-
degree polynomial to this data to determine this offset. We then removed the offset using the inverse of
this polynomial as the relationship between the commanded stiffness and desired output:
k
out
=− 3.49e− 04∗ k
2
des
+2.03∗ k
des
− 147.27 (3.1)
where k
out
is the output stiffness commanded to the device and k
des
is the desired stiffness. To test the
stiffness calibration of the device, we re-ran the above data collection with the new commanded stiffness
values. Figure 3.3 shows a significant improvement in the match between desired and commanded stiffness,
especially in the range of interestk < 2000 N/m.
35
Figure 3.4: A 3d printed stylus with embedded position sensor and a Force/Torque sensor
3.2.3 Stylus
The stylus is ungrounded and separated from the body of the haptic device. We embed sensors in the stylus
to measure its speed and the force it applies to the end-effector. This data is used to find the relationship
between human perception of hardness versus speed and force of tapping; these sensors are not needed
for rendering with this device.
We designed, and 3D printed a 12 cm long pen-like stylus with 1 cm diameter (Figure 3.4). We embedded
a Nano17 F/T sensor (ATI Industrial Automation) in the stylus to record the force of the interaction between
the stylus and the end-effector. The sensors’ data was logged by a Sensoray 826 PCI Express board. We
also included an Ascension trakSTAR magnetic tracker (0.5 mm resolution) to track the tool’s position,
which was used to calculate its speed right before the tap. The magnetic tracker is embedded in the back
of the stylus to avoid interference or noise with the force sensor. The stylus has a 4 mm hemispherical tip
that is 15 mm in length to ensure there is only one interaction point with the end-effector surface.
3.2.4 End-effectorPlate
Since contact between the stylus and the plate attached to the end-effector directly mimics the transient
force during impact with a real object, it is critical to closely match the hardness of the plate on the ETHD
to the hardness of the virtual object. It is not practical to have an infinite set of plates that can exactly match
all objects. Therefore, we use a small set of plate materials in this chapter, which was chosen to span a
36
Figure 3.5: Three experimental setups: (a) a setup with untethered stylus and ETHD, (b) real object, and
(c) a setup with tethered stylus for rendering position- and event-based methods.
large portion of the Shore hardness scale: extra-soft (Shore 10OO), medium (Shore 60A), and extra-hard
(Shore 75D).
3.3 ExperimentalMethods
We conducted an experiment to compare the realism of virtual objects rendered using different hardness/s-
tiffness methods, including position-based spring and spring-damper models, event-based acceleration
matching transient force model, and our ETHD model (Fig. 3.1). The experiment setup is shown in Fig. 3.5.
To determine the objects to test, we first selected a set of 22 common materials like fabric, foam, eraser,
felt, leather, paper, and wood. We recorded their acceleration response during tapping with the stylus. The
data was recorded using an ADXL326 accelerometer and a Sensoray 826 PCI board. We tapped 20 times
on each material and analyzed the tap with the average amplitude. We calculated the Discrete Fourier
Transform (DFT) for the selected taps for all 22 materials and compared the spectral centroid of their
37
Figure 3.6: Six real surfaces rendered in the study. Three materials (insole, felt, and wood) were placed on
foam and wood.
frequency responses, which has been shown to be an effective measure of perceived hardness [22]. We
chose three materials (shoe insole, felt, and wood) to give us a set of materials spanning a wide range
of hardness in the categories extra-soft, medium, and extra-hard. We also wanted to test the effect of
stiffness of the real objects, so we placed the materials both on a rigid wooden backing and on a 1.5-inch
thick piece of foam. The materials and their backings were placed in small boxes to avoid visual feedback
of the underlying material (Figure 3.6).
To have a fair comparison between the different rendering methods, we used two similar haptic devices,
one with an attached stylus (Figure 3.5-(c)) to run the first three methods (spring, spring-damper, and
acceleration matching), and a second one with an untethered stylus (Figure 3.5-(a)) to run the ETHD model.
Both systems used a Novint Falcon haptic device that was rotated90
◦ as discussed above.
3.3.1 Position-BasedSpringModel
The spring model, also known as a Hooke’s Law model, is a traditional rendering method in which a force
is applied to the end-effector when the user penetrates into a virtual object [13]. The applied force is
38
proportional to the penetration distance, which is the difference between the closest point on the object’s
surface and the position of the end-effector:
⃗
F =− K⃗ x (3.2)
whereK is the stiffness of virtual object, and ⃗ x is the penetration distance.
We implemented this method on the haptic device in Fig. 3.5-(c). We replaced the original stylus of the
device with a 3d-printed pen-like stylus (12 cm length,1 cm diameter) that was connected to the device’s
end-effector. A virtual object was rendered using this method for each of the three materials (insole, felt,
and wood). The stiffness for each material was tuned by two haptic experts following a staircase method
with an initial stiffness of K
i
= 200 N/m and a step-size of 100 N/m. The stiffness was capped at the
maximum reliable stiffness of the device K=2500 N/m. During tuning, the experimenter compared the
feel of tapping on the real object with tapping on the rendered virtual object, and adjusted the stiffness
accordingly. The tuning was terminated after three reversals, and the chosen stiffness was set as the
average of the reversals. The final values chosen for rendering are K
insole
= 1500 N/m, K
felt
= 1700
N/m, andK
wood
= 2500 N/m. The realK
wood
was higher than2500 N/m, but we stopped tuning when it
reached the maximum possible stiffness to maintain stability.
3.3.2 Position-BasedSpring-DampingModel
The spring-damper model is a common rendering method used to increase the hardness of a virtual surface
when stiffness is limited by stability concerns [13]. A unidirectional damper is added to the spring model
so that additional energy is dissipated as the user contacts the surface:
⃗
F =







− K⃗ x− B
⃗
x
′
x
′
≥ 0
− K⃗ x x
′
< 0
(3.3)
39
whereK is the stiffness coefficient of virtual object, B is the damping coefficient, and ⃗ x is the pene-
tration distance.
This method was also implemented on the haptic device in Figure 3.5-(c), with the attached stylus. We
rendered a virtual object using this method for each of the three study materials (insole, felt, and wood).
In this method, we used the same stiffness coefficients that were found in the above spring model. The
damping coefficient for each material was tuned by two haptic experts. The stiffness was held constant,
and the damping coefficient was varied following a staircase method with initial damping B
i
=0 and a
step-size of 0.04 N/(m/s). The tuning was terminated after three reversals, and the chosen damping was
set as the average of the reversals. The final values chosen for rendering are B
insole
= 0.06 N/(m/s),
B
felt
= 0.20 N/(m/s), andB
wood
= 0.25 N/(m/s).
3.3.3 Event-BasedTransientForceFeedbackModel
We implemented an acceleration matching event-based rendering method [58] on the haptic device in
Figure 3.5-(c). In this method, transient forces are recorded from interactions with the real objects and
played back when the user first contacts the object. To record the impact acceleration and force, the
experimenter tapped vertically on each plate (insole, felt, and wood) 20 times with different speeds using
the stylus described in Section 3.2. The acceleration data was high-pass filtered with a cutoff frequency
of 20 Hz to avoid noise due to human motion and the effects of gravity. We selected five taps spanning
the range between the minimum and maximum speeds for rendering. For each of these five taps for each
plate,100 ms of vibration data after the contact was stored and labeled with the speed of the tap.
The momentum of the haptic device is a function of the speed and mass of the user’s hand. Since mass is
constant, the momentum linearly changes with tapping speed. The tapping transient should quickly cancel
out the user’s momentum the same way that contact with a hard object would. Therefore, when rendering
the tapping transient as force, the amplitude of the rendered force should be scaled with the incoming
40
speed of the stylus before contacting the virtual object. We interpolate between the recorded transients
based on the user’s contact speed. Speeds higher than the maximum are saturated at the maximum speed,
and taps for speeds lower than the minimum speed are appropriately scaled down.
We must also convert the recorded vibration transients to a force before playing them through the
motors of the haptic device. We apply a constant scaling factor equal to the effective mass of the user’s hand
and stylus, chosen as0.05 kg based on [23]. For each plate (insole, felt, and wood), the related acceleration-
matched transient forces are added to the corresponding normal force for that material, computed using
the spring model.
3.3.4 Encountered-TypeHapticDisplayModels
In this system, we used the haptic device with the detached stylus (Figure 3.5-(a)). A plate was attached to
the device’s end-effector for each rendered object to closely match the object’s hardness. For this model, we
used the end-effector plates mentioned in Section 3.2 with Shore durometers in the categories of extra-soft
for insole, medium for felt, and extra-hard for wood (Figure 3.7).
Since we tested two versions of each material with different underlying stiffnesses, and because the
interplay between hardness and stiffness in rendering with this device is still unclear, we decided to test
two different stiffness values for each material. The insole was rendered using both low ( 1500 N/m) and
high (2500 N/m) stiffness. Felt was rendered using both medium ( 1700 N/m) and high (2500 N/m) stiffness.
Wood was rendered using both low (1500 N/m) and high (2500 N/m) stiffness.
3.4 UserStudy
We recruited 20 participants (20-35 years old; 7 female, 13 male; one left-handed; 7 with no haptic expe-
rience, 8 limited experience, 2 moderate experience and 3 extensive experience). The study was approved
by the USC IRB under protocol UP-20-01131. The participants wore headphones playing white noise to
41
Hard Medium Soft Hard Medium Soft
Figure 3.7: End-effector plates to simulate hardness in the range of hard, medium, and soft.
block audio cues. They sat on a chair, and both devices and the real object were located on a table next to
them on their dominant side. The real object was placed on a stand, so it was at the same height as the
end-effector (Figure 3.5-(b)). Participants used the attached stylus for tapping on a virtual object rendered
with the tethered haptic device (Figure 3.5-(c)). They used the untethered stylus for tapping on both the
real object or virtual object of the ETHD device (Figure 3.5-(a),(b)).
During the experiment, participants were presented with one object and one rendering method at a
time. The objects and rendering methods were both randomized for a total of 30 conditions (6 objects
(insole on foam, insole on wood, felt on foam, felt on wood, wood on foam, and wood on wood)× 5
rendering methods (spring, spring-damper, amplitude matching event-based tapping transient, ETHD with
lower stiffness, and ETHD with higher stiffness)). For each object-method pair, the participant compared
the feel of the real and rendered virtual object during tapping. Participants subjectively rated the rendered
object by placing a mark on a continuous 100-point scale using a tablet, as in [23]. They rated the match
between real and virtual objects on a realism scale from 0=“very unrealistic" to 100=“very realistic". They
also compared the feeling of the virtual object to the real object on a hardness scale (-50=“softer", 0=“same”,
50=“harder") and a compressibility scale (-50=“more rigid", 0=“same”, 50=“more compressible"). To avoid
any bias, hardness and compressibility were not defined and participants were asked to use their own
intuition for these concepts. After rating all rendering methods for a single object, they were asked to
specify which rendering method(s) they preferred the most.
42
Participants were allowed to switch back and forth between the real and virtual plates as long as desired
before rating. They were instructed not to look at the device during tapping and to focus on two small balls
displayed on the monitor that tracked the location of each device’s stylus. No virtual surface or additional
deformation cues were displayed. They were allowed to look at the device and adjust the stylus’ location
between taps. Participants were instructed to tap on the center of the plates and keep the untethered stylus
vertical during tapping on the plates. They were asked to hold the stylus like a pen. We did not otherwise
limit how they should hold the stylus (i.e., which parts of the hand should be in contact with stylus) in
order to have a variety of grasping styles, since it has been shown that different grasps influence tactile
perception [117].
3.5 Results
Figure 3.8 shows the ratings for each rendering method for realism, hardness, and compressibility. For
statistical analysis, we use the z-score of ratings in order to normalize the individual’s ratings.
For simplicity in our discussion of the results, we denote spring as S, spring damper as SD, acceleration
matching as AM, and encounter-type haptic display at ETHD. Also, we denote L = 1500 N/m, M =
1700 N/m, and H = 2500 N/m for the generated stiffnesses by the device. These values were chosen
based on the tuned stiffnesses for each material.
Figure 3.8-(a) shows the realism rating for each rendering method, separated by object. We ran a one-
way ANOVA on the z-score of realism rating for each object with the rendering method as a factor. This
analysis indicated that there was a statistically significant difference in the z-score of realism rating for
insole on foam (F(4,95) = 5.69,p < 0.001), insole on wood (F(4,95) = 7.8,p < 0.001), felt on foam
(F(4,95) = 4.4,p = 0.003), felt on wood (F(4,95) = 9.26,p < 0.001), wood on foam (F(4,95) =
8.4,p< 0.001), and wood on wood ((F(4,95) = 15.67,p< 0.001).
43
0
20
40
60
80
100
Realism Rating
Insole on Foam Felt on Foam Wood on Foam
S
L
SD
L
AM
L
ETHD
L
ETHD
H
0
20
40
60
80
100
Insole on Wood
S
M
SD
M
AM
M
ETHD
M
ETHD
H
Rendering Method
Felt on Wood
S
H
SD
H
AM
H
ETHD
L
ETHD
H
Wood on Wood
Very
Real
Very
Unreal
Very
Real
Very
Unreal
(a)
-50
0
50
Hardness Rating
Insole on Foam Felt on Foam Wood on Foam
S
L
SD
L
AM
L
ETHD
L
ETHD
H
-50
0
50
Insole on Wood
S
M
SD
M
AM
M
ETHD
M
ETHD
H
Rendering Method
Felt on Wood
S
H
SD
H
AM
H
ETHD
L
ETHD
H
Wood on Wood
Softer
Harder
Softer
Harder
(b)
-50
0
50
Compressibility Rating
Insole on Foam Felt on Foam Wood on Foam
S
L
SD
L
AM
L
ETHD
L
ETHD
H
-50
0
50
Insole on Wood
S
M
SD
M
AM
M
ETHD
M
ETHD
H
Rendering Method
Felt on Wood
S
H
SD
H
AM
H
ETHD
L
ETHD
H
Wood on Wood
More
Rigid
More
Comp
More
Rigid
More
Comp
(c)
Figure 3.8: (a) Realism, (b) hardness, and (c) compressibility ratings across all materials and rendering
methods.
44
We then ran a Tukey’s post-hoc pairwise comparison test on each ANOVA to further evaluate the
effects of the rendering methods on the z-score of realism. The results showed significantly higher realism
for ETHD
H
than for methods S, SD, and AM for insole on foam, insole on wood, felt on wood, wood on
foam, and wood on wood. Felt on foam showed significantly higher realism for ETHD
H
than methods S
and AM.
There was also a significantly higher realism for ETHD
L
than methods S and AM for insole on wood,
and between ETHD
L
and methods S, SD, and AM for both wood on foam and wood on wood. Finally,
ETHD
M
was significantly more realistic than methods S and AM for felt on wood. There was no significant
difference between any other methods.
Figure 3.8-(b) shows the hardness rating for each rendering method, separated by object. We ran a one-
way ANOVA on the z-score of hardness rating for each plate with rendering method as factor. This analysis
indicated a statistically significant difference in the z-score of hardness rating for felt on foam ( F(4,95) =
11.25,p< 0.001), felt on wood (F(4,95) = 5.95,p< 0.001), wood on foam (F(4,95) = 7.0,p< 0.001),
and wood on wood (F(4,95) = 9.06,p < 0.001). There was no statistically significant differences in
hardness ratings for insole on foam (F(4,95) = 0.39,p = 0.8) and insole on wood (F(4,95) = 0.99,p =
0.4).
We then ran a Tukey’s post-hoc pairwise comparison test on each ANOVA to further evaluate the
effects of the rendering methods on the z-score of hardness. The results showed significant differences
between the z-score of hardness ratings for method ETHD
H
and methods S and AM for both felt on foam
and felt on wood. There were also significant differences between ETHD
M
and methods S and AM for
felt on foam, and between ETHD
M
and method AM for felt on wood. Felt on foam showed additional
differences between methods S and SD. Both wood on foam and wood on wood showed significant differ-
ences in hardness between ETHD
H
and methods S, SD, and AM. Finally, there were significant differences
45
Table 3.1: Participants’ preferred rendering method
Insole
on
foam
Insole
on
foam
Insole
on
wood
Insole
on
wood
Felt
on
foam
Felt
on
foam
Felt
on
wood
Felt
on
wood
Wood
on
foam
Wood
on
foam
Wood
on
wood
Wood
on
wood
E
L,M
E
H
and/or
1
7
7
1
7
7
3
6
8
3
6
8
2
3
7
2
3
7
1
7
7
1
7
7
2
8
8
2
8
8
1
5
10
1
5
10
Total
14
83%
14
83%
17
94%
17
94%
12
67%
12
67%
15
83%
15
83%
18
100%
18
100%
16
89%
16
89%
S
SD
AM
and/or
0
0
1
2
0
0
1
2
0
0
1
0
0
0
1
0
1
4
1
0
1
4
1
0
0
2
0
1
0
2
0
1
0
0
0
0
0
0
0
0
0
0
2
0
0
0
2
0
Total
3
17%
3
17%
1
6%
1
6%
6
33%
6
33%
3
17%
3
17%
0
0%
0
0%
2
11%
2
11%
between ETHD
L
and methods S,SD, and AM for wood on foam, and between ETHD
L
and method S for
wood on wood. There was no significant difference between any other methods.
Figure 3.8-(c) shows the compressibility rating for each rendering method, separated by object. We ran
a one-way ANOVA on the z-score of compressibility rating for each plate with rendering method as factor.
This analysis indicated a statistically significant difference in the z-score of compressibility rating for felt
on foam across different rendering methods ( F(4,95) = 4.29,p = 0.003). There was no statistically
significant difference between the ratings of compressibility for insole on foam, insole on wood, felt on
wood, wood on foam, or wood on wood across the different rendering methods.
We ran a Tukey’s post-hoc pairwise comparison test to further evaluate the effects of the rendering
methods on the compressibility ratings for felt on foam. The results showed significant differences between
the z-score of compressibility ratings for method ETHD
H
and methods S and AM. There was no significant
difference between any other methods.
Participants were asked to indicate their preferred rendering method after experiencing all of the meth-
ods for each object. Table 3.1 summarizes participants’ responses.
46
Participants were also asked to provide comments about their experience with the ETHD and other
rendering methods. We include some representative comments here:
– “I prefer tapping on the plate with the pen because it is easier for me to make distinctions"
– “Moving the pen without a robot attached (ETHDs) felt better / more realistic."
– “Prefer (AM), although it isn’t hard enough it has some vibrations which make me imagine its texture."
– “(ETHD
H
) was VERY REALISTIC."
– “In (ETHD) the hardness of the surface was similar to the real one, but in (non-ETHD) the stylus felt
heavy and the simulation felt unrealistic.
3.6 Discussion
This experiment compared traditional hardness/stiffness rendering methods with our new ETHD method
for rendering soft, medium, and hard objects. The realism of our method was rated significantly higher
than many of the traditional rendering methods, but the relative difference between the rendering methods
changed with the object being rendered. The realism was also dependent on the rendered stiffness. Figure
3.8-(a) shows that objects rendered with the ETHD using a higher stiffness were more realistic than the
same object rendered with a lower stiffness, regardless of whether the material was backed with foam or
wood. The high stiffness was the maximum stiffness that could be stably rendered by the device, indicating
that a higher-stiffness haptic device could further increase realism.
The result in Figure 3.8 shows that our ETHD method outperforms the other methods in realism of
rendering all six objects. ETHD
H
was rated as significantly more realistic than the traditional rendering
methods (S, SD, AM) using the tethered haptic device. But our results showed no significant difference
in the realism between these three methods. ETHD
L
and ETHD
M
were also rated as significantly more
realistic than at least two of the three traditional methods, depending on the object. We saw a larger
improvement in realism between our ETHD method and the traditional methods for the objects backed
47
with wood, and a smaller improvement for the objects backed with foam, indicating that the benefits of
our system are greater for stiffer objects. For wood, even the lower stiffness ETHD
L
showed higher realism
than the traditional methods rendered using the higher stiffness. For all materials except felt, the maximum
realism rating was 100, which shows that some participants perceived the rendered object as identical to
the real object.
The results showed significant differences in the hardness ratings across rendering methods for both
felt and wood objects, but there were no significant differences in hardness ratings for insole. The ETHD
methods for felt were rated as feeling harder than the real objects, which means that even though these
virtual objects were more realistic than other methods, the end-effector plate was too hard. For wood on
foam and wood on wood, the average hardness ratings in ETHD methods were close to zero, which means
they were very similar to real objects. The wood objects rendered with the ETHD were also rated as the
most realistic. These results show the importance of choosing a plate that closely matches the hardness of
the object to be rendered.
Our results showed a significant difference in compressibility ratings between different rendering
methods only for felt on foam. This trend does not follow our realism results, indicating that there is
likely little connection between the realism of a rendering method and how compressible the material
was. For wood and insole on foam, and insole on wood the average rated compressibility for ETHD
H
was
close to zero, meaning that its compressibility was similar to the real object. However, the compressibility
for felt on foam rendered with ETHD
H
was much greater than zero, showing that it was perceived to be
more rigid than the real surface. All traditional methods (S, SD, and AM) were rated as more compressible
than the real object, showing that these rendering methods create virtual objects that feel too squishy.
These methods were better at matching the compressibility of the objects backed with foam than the ob-
jects backed with wood, highlighting the importance of the device’s limited stiffness in these rendering
methods.
48
Table 3.1 shows that a high percentage of participants preferred ETHD methods regardless of their
realism, hardness, and compressibility ratings. Felt on foam had the lowest preference rate (67%), which
matches the realism ratings in Figure 3.8-(a). Wood on foam had the highest preference percentage (100%)
and also the highest realism rating.
Another benefit of our system compared to other methods is that its passivity allows us to render
higher forces without instability, whereas rendering very high force using other methods often creates an
unstable system due to the energy introduced into the device by the user. We found the effect of this in
calibration when we could not increase damping or stiffness coefficient too much.
Previous work [58] used a constant stiffness when comparing different rendering methods. Similarly
we tuned the stiffness in S and used the same value in SD, AM, and ETHD. We added a second case of ETHD
with different stiffness. As could be seen in some cases like felt on foam, the realism rating for the ETHD
with the same stiffness was not significantly different from the realism of the other methods. However,
the realism for the ETHD with a higher stiffness was greater than the realism of the other methods. This
result shows that it would be better to tune these parameters separately for each method in the future.
3.7 Summary
Traditionally, force-feedback haptic devices have been used to render virtual objects using stiffness alone.
Recent results have shown the benefits of simultaneously rendering stiffness and hardness to increase re-
alism. This combination is complex due to the need to combine and synchronize high-frequency transient
forces generated by an object’s hardness and low-frequency forces generated by its stiffness. In this work,
we created an ETHD device that rendered hardness using a plate with a specific hardness on the end-
effector and rendered stiffness using the motors of the device. We also modified the device to work with
an untethered stylus, improving our rendering of free space. By comparing this method with previous
49
haptic rendering methods, we found our ETHD had the highest realism ratings and was most preferred by
the users.
50
Chapter4
EffectsofPhysicalHardnessonthePerceptionofRenderedStiffnessin
anEncountered-TypeHapticDisplay
In chapter 3, we used an ETHD to increase the realism of virtual haptic interactions, given the limitations
of the haptic device, by matching the hardness of real objects. Simultaneous and independent rendering
of hardness and stiffness allowed us to simulate both the transient- and extended-response force during
the haptic interaction. The results showed that our methods outperformed traditional position-based and
event-based rendering methods, including spring, spring-damper, and acceleration matching methods.
Our ETHD method presented multi-modal cues to the user because of the distinct gap between stiffness
and hardness modalities. However, in an ETHD setup the effect of presenting multiple haptic modalities
on the user is unknown. Similarly, in our setup, the relationship between the rendered object stiffness
and perceived hardness is not as straightforward as for natural objects [65]. This could result in tactile
masking between the two cues. Therefore, in this chapter, we explore the perceptual basis behind the high
realism of our ETHD method compared to other methods and study how the hardness of the end-effector
can mask the stiffness provided by the device. This chapter presents work published in IEEE Transactions
on Haptics [118].
51
4.1 RelatedWork
In multi-modal haptics, a more complicated stimulus is applied to the user by combining the cues from
different haptic modalities (e.g., vibration and force). Usually, multi-modal haptics is used to convey more
information to the user. However, the different modalities can interfere with each other and limit the per-
ceptual ability to perceive each channel separately, which is called tactile masking. These channels could
be all the same or different types of cues and stimulus, for example, all vibration cues in different locations
and with different signals, or vibration with a squeeze and stretch [105]. In [25] force feedback and surface
displacement were separately applied through passive touch to better convey object compliance. Touch
modalities could also be combined with other non-haptic feedback, such as visual and audio. There has
been a lot of work studying how different haptic and/or non-haptic feedback can affect each other [110,
33, 122, 21], which focus mostly on vibration along with other senses. In [100], they study how perceived
stiffness is affected by visual and haptic feedback. There has not yet been an experiment studying how
people differentiate stiffness and hardness, mainly because of the limited hardware to represent both. With
an ETHD, we can analyze these two haptic modalities separately. Therefore, in this chapter, we explore
how the hardness of the end-effector can affect stiffness perception in our ETHD.
4.2 Encountered-TypeHapticDisplayDesign
In this work we use the same ETHD setup that was developed in chapter 3.In this ETHD, we use the
device’s actuators to render stiffness and the end-effector surface to render hardness. The hardness of the
end-effector can be directly altered by attaching different materials to its surface, as shown in Fig. 3.2. The
hardness and thickness of the materials will affect the interaction impact of the stylus with this surface.
In this chapter, we explore the effect of interacting with plates of different hardnesses on the end-effector
and study their influence on human perception of stiffness and the overall interaction.
52
4.3 ExperimentalMethods
We conducted two experiments to understand the interplay between the haptic device’s rendered stiffness
and the physical hardness of the end-effector on perceived hardness. We first quantitatively evaluated this
effect by measuring the spectral centroid of force during tapping for varied combinations of stiffness and
hardness. We then qualitatively evaluated the effect of stiffness and physical hardness through a human
subject study.
4.3.1 Experiment1
Previous studies agree that a significant cue in hardness perception is the frequency of the transient vibra-
tion produced during tapping [58, 86, 45, 40]. Therefore, in this chapter, we study hardness perception in
our device during tapping. We determine the effect of varying stiffness on the spectral centroid (Hz) of the
tapping transient, which is a quantitative measure of the perceived hardness [22]. Harder surfaces pro-
duce tapping transients with a high spectral centroid, and softer surfaces have a lower spectral centroid.
We used a set of 11 plates with different hardness attached to the device’s end-effector using a custom
3D printed attachment (Figure 3.2). These plates, shown in Table 4.1 were 50.8 mm× 50.8 mm wide and
ranged from extra soft to extra hard (with a thickness of 12.7 mm for soft materials and 3.2 mm for hard
materials). The hardness of these materials is specified on the Shore durometer scale, which measures the
penetration depth under a specific load. The thickness was chosen to ensure the tooltip did not penetrate
to the end of the plate during tapping.
Magnitude and frequency spectra are essential in determining the perception of specific haptic prop-
erties of a surface, such as hardness. To understand the relative effects of stiffness and physical hardness
in the perceptual hardness of the ETHD, we compare the force responses during the interaction between
the stylus tip and end-effector plates. The experimenter tapped 30 times on each plate for each stiffness
around the same range of speed. We varied the rendered stiffness between the minimum and maximum
53
Table 4.1: Selected hardnesses covering a wide range
Shore No. Range
Shore 10OO Extra Soft
Shore 40OO Extra Soft
Shore 50OO Soft
Shore 60OO Soft
Shore 70OO Medium
Shore 40A Medium
Shore 60A Medium
Shore 80A Hard
Shore 90A Hard
Shore 95A Extra Hard
Shore 75D Extra Hard
stiffness (200 N/m and 2000 N/m) in increments of 100 N/m. Figure 4.1 shows an example of how we
processed the tapping data. Although we did not directly control contact velocity, previous research has
shown that the frequency of the vibrations produced during the tap does not change with speed, and the
peak force varies linearly with this speed due to the conservation of momentum [22]. When comparing
the spectral centroids of different materials, we select one representative tap with the average peak force
of the taps for that material, corresponding to a tap recorded at the average tapping speed. This step is
shown in Figure 4.1-(a,b). We then calculate the spectral centroid for this tap using Eq. 2, which is shown
in Figure 4.1-(c). Note that the first and last 3 seconds of the recorded tap signals (10 seconds in total) were
cropped to avoid noise. The spectral centroid (SC) is calculated as:
SC =
P
N− 1
n=0
f(n)x(n)
P
N− 1
n=0
x(n)
(4.1)
whereX(n) is the amplitude of the DFT in thenth window atf(n) frequency. This value is shown as the
green vertical dashed line in Figure 4.1-(c)).
54
01234567
Time (s)
0
5
10
15
20
Force (N)
012 3 4 5 6 7
Time (s)
0
5
10
15
20
Force (N)
(a) Tapping force, horizontal line shows average of peaks.
0 100 200 300
Time (ms)
0
5
10
15
Force (N)
0 100 200 300
Time (ms)
0
5
10
15
Force (N)
(b) Selected closest tap to the vertical average line.
0 200 400 600
Frequency (Hz)
0
0.5
1
1.5
Power
0 200 400 600
Frequency (Hz)
0
0.5
1
1.5
Power
(c) DFT of selected tap, vertical line shows spectral centroid.
Figure 4.1: Data processing steps for tapping data. (left) hardest plate, (right) softest plate.
4.3.2 ResultofExperiment1
Figure 4.2 shows the spectral centroid of the tapping force for 11 plates of varying hardness under a range of
rendered stiffness values. A two-way ANOVA with hardness and stiffness as factors show that the spectral
centroid of force is statistically significant across different hardnesses ( F(10,180) = 1192.6,p < 0.001),
but is not significant across different stiffnesses.
55
Spectral Centroid of Force (Hz)
10 OO
40 OO
50 OO
60 OO
70 OO
40 A
60 A
80 A
90 A
95 A
75 D
Plate Hardness (Shore Durometer)
200
400
600
800
1000
1200
1400
1600
1800
2000
Stiffness (N/m)
50
100
150
200
250
Figure 4.2: Measured spectral centroid of force for each hardness of plates versus device stiffness
10 OO
40 OO
50 OO
60 OO
70 OO
40 A
60 A
80 A
90 A
95 A
75 D
Plate Hardness (Shore Durometer)
0
0.1
0.2
0.3
Magnitude of Spectral Centroid of Force (N)
Figure 4.3: Magnitude of spectral centroid of force for each plate hardness.
Figure 4.3 shows the magnitude of the spectral centroid of forces across all stiffnesses and hardnesses.
To better understand the differences between the different hardnesses, Fig. 4.4 shows the boxplot of the
spectral centroid of force versus hardness, with a connected line indicating the averages, and their colors
show the different categories of shore durometers.
56
10 OO
40 OO
50 OO
60 OO
70 OO
40 A
60 A
80 A
90 A
95 A
75 D
Plate Hardness (Shore Durometer)
0
50
100
150
200
250
Spectral Centroid of Force (Hz)
Shore 00
Shore A
Shore D
Figure 4.4: Average of the spectral centroid of force for every stiffness and hardness
For each hardness and stiffness, we also found the dominant frequency of the tapping force (i.e., the
frequency at which the maximum magnitude in the DFT happens). The results are shown in Figure 4.5.
A two-way ANOVA shows that the dominant frequency of force is statistically significant across different
hardnesses (F(10,180) = 14,p < 0.001) and across different stiffnesses ( F(18,180) = 3.22,p < 0.001).
A second two-way ANOVA shows that the magnitude of the dominant frequency is also statistically
significant across different hardnesses ( F(10,180) = 32.96,p < 0.001) and across different stiffnesses
(F(10,180) = 3.6,p< 0.001).
4.3.3 DiscussionofExperiment1
Figure 4.2 shows that the spectral centroid of force only changes based on the hardness of the plates
and is not affected by the rendered stiffness. We tested 11 plates in the Shore durometer scale, covering
the range from soft to hard. Figure 4.3 shows that the magnitude of the spectral centroid of force does
not change among different plate hardnesses except for the softest material. By comparing the vertical
lines in Fig. 4.1-(c) that indicate the taps’ spectral centroid, we could infer that the difference in Fig. 4.3 is
because of the squishy structure of the softest material. This material dampens the frequency and causes
57
Figure 4.5: Magnitude of the dominant frequency of forces versus plates hardness and stiffness.
a lower-frequency spectral centroid at a higher power amplitude. Figure 4.4 indicates that the variation
in the spectral centroid of force across different plate hardnesses highly depends on the materials’ Shore
category.
To shorten the next experiment, we used these results to reduce the materials tested. We divided the
minimum and maximum spectral centroid of force into five levels and found the hardness closest to each
division. This resulted in a set of five materials of different hardnesses, one in each of the categories extra
soft, soft, medium, hard, and extra hard. These materials are listed in Table 4.2 and shown in Fig. 4.6.
4.3.4 Experiment2
In this experiment, we tested the participants’ ability to distinguish between different rendered stiffnesses
when interacting with a constant hardness surface attached to the device’s end-effector. For each plate,
we determined the Weber fraction for stiffness, which is the ratio of the just noticeable difference to the
intensity of the stimulus, by having the participant compare their perception of two rendered stiffnesses
at a time through tapping. As discussed above, we tested a set of five plates, one from each category of
extra soft, soft, medium, hard, and extra hard (Fig. 4.6, Table 4.2).
58
Table 4.2: Selected plates in different hardness categories
Plate No. Shore No. Range
P1 Shore 10 OO Extra Soft
P2 Shore 60 OO Soft
P3 Shore 60 A Medium Soft
P4 Shore 90 A Hard
P5 Shore 75 D Extra Hard
Shore 00
Shore A
Shore D
0 10 20 30 40 50 60 70 80 90 100
Extra Soft Soft
0 10 20 30 40 50 60 70 80 90 100
0 10 20 30 40 50 60 70 80 90 100
Extra Hard Hard Medium
Figure 4.6: Selected hardnesses based on the durometer standard table. The plates with different hardnesses
are attached to a 3D-printed mount for connection to the haptic device.
We conducted a pilot study with four participants to select the set of reference stiffnesses to test. We
used a set of eight reference stiffnesses from 500 N/m to 2250 N/m in increments of 250 N/m. We found
that the Weber fraction showed a monotonic linear trend with stiffness, indicating that it was unnecessary
to test all eight reference stiffnesses in the final experiment. Therefore, we selected a set of four stiffnesses
from 500 N/m to 2000 N/m with an increment of 500 N/m to shorten the experiment and minimize fatigue.
We only varied one parameter, stiffness, in the experiment. Participants rated their perception of the
interaction relative to a scale of soft–hard, following previous work that used this rating as a measure of
perceptual hardness [83, 16, 68, 69]. We note that this qualitative measure of perceptual hardness is not
equivalent to the physical hardness of the plates and captures the combined effect of both the rendered
stiffness and physical hardness of the interaction.
Participants followed a Two Alternative Forced Choice (2AFC) procedure by tapping on the end-
effector plate with the reference stiffness and test stiffness in randomized order and responded to the
59
prompt: "Which surface feels harder?". They controlled the stimuli and responded to the prompt using
four labeled keys on a keyboard. Two keys toggled back and forth between the test and reference stiffness,
and the other two were used to respond to the prompt by selecting the stimuli (1 or 2) that they perceived to
be harder. The test stiffness was chosen for each reference stiffness following a modified staircase method,
as explained below. This psychophysics method was integrated with Chai3d to control the haptic loop of
the device.
4.3.4.1 ModifiedStaircasemethod
We used an adapted staircase method to select the test stiffness based on the participant’s previous re-
sponses. For each plate and each reference stiffness, the test stiffness began at the minimum stiffness
(200 N/m) and varied with the related step size. The trial terminates after five reversals. However, in our
pilot study, we found that if there was a reversal by mistake at early steps, the average threshold would be
much lower than the actual threshold because the final threshold is the average of all reversals.
After uncovering this issue in our pilot study, we implemented an additional rule we found to be
helpful in the accuracy of detecting the threshold by ignoring responses that are likely mistakes. After
each wrong answer, if the participant can answer the following four times correctly, we ignore the last
change of direction. The reason is that the probability of answering correctly four times in a row by chance
is only 6.25%, which is very low. Therefore, it is likely that the last change of direction was a mistake,
which could have been caused by a change in tapping speed, maintaining contact with the surface, change
of body position, or other distractions.
In a pilot study, we tested this staircase method with a step size of 50 N/m and 100 N/m. We found the
Weber fraction for reference stiffnesses higher than 500 N/m was similar for both 50 N/m and 100 N/m.
Therefore, in the final study, we used a step size of 50 N/m for the reference stiffness 500 N/m and 100 N/m
60
for all other reference stiffnesses. This variation in step size was chosen to limit the experiment length and
avoid muscle fatigue, which would affect the accuracy of the results.
4.3.4.2 UserStudy
We recruited 20 participants (20-35 years old; 8 female, 12 male; one left-handed). They wore headphones
playing white noise to block audio cues from the device. They sat on a chair with the device next to them
on a table and held the stylus in their dominant hand. A keyboard was placed in front of them with four
keys labeled for switching between stiffnesses and responding to the prompt, which they used with their
non-dominant hand.
When the experiment started, participants were instructed not to look at the device during tapping to
avoid any visual effects. They could look at the device and adjust the stylus’ location between taps so they
could not be fully blindfolded. The device was located in line with where they sat, below the armrest on
a low table. They could not see the device’s motion if they were looking in front of them. We presented a
stationary ball on the monitor in front of them that changed color from yellow to white when there was an
interaction force between the tooltip and the plate. This visual indicator was not sensitive to small forces
and did not give feedback about the rendered stiffness. Note that in the pilot study, there was no monitor
in front of the user, but we found that participants lost focus quickly. Having this stationary ball in front
of them helped them gain focus and attention without any feedback about the tool’s motion or force. All
participants in the pilot study preferred having the screen in front of them, and they confirmed that color
change gave no feedback to help make their decision about the haptic feedback.
Participants were instructed to tap on each stimulus for about 10 seconds while keeping the stylus
vertical. We did not constrain the users’ grasp because we wanted them to focus on the perception rather
than maintaining a specific grasp. All participants were instructed to grasp the tool as they would hold a
pencil and keep it vertical. We saw in our work presented in Chapter 2 that these grasping styles could be
61
1 2 3 4 5
0
0.2
0.4
0.6
0.8
Weber Fraction
Plate Hardness (1=Softest, 5=Hardest)
(a)
Reference Stiffness 2000 N/m
Reference Stiffness 1500 N/m
Reference Stiffness 1000 N/m
Reference Stiffness 500 N/m
500 1000 1500 2000
Stiffness (N/m)
(b)
0
0.2
0.4
0.6
0.8
Weber Fraction
Very long Long Short Very short
Tapping Time
(c)
0
0.2
0.4
0.6
0.8
Weber Fraction
Figure 4.7: (a) Weber fraction of stiffness versus hardness of plates. Within each plate, a smaller set of
boxplots indicates how Weber fraction of stiffness varies among different reference stiffnesses. (b) Weber
fractions of stiffnesses versus test stiffnesses, and (c) versus tapping times.
62
divided into three main categories, with variations in perception based on the number of contact points on
the hand. We also did not limit how many times they had to tap to avoid any focus disturbance. Participants
were allowed to switch back and forth between the two presented stiffnesses using the keyboard as many
times as desired before responding.
We did not instruct participants on the difference between hardness and stiffness to avoid bias and
confusion. There was only a training session before the actual experiment, in which we taught them how
and approximately with what speed to tap. The participants’ contact velocity of taps was not controlled
during the experiment, but we showed them an example of tapping speed and asked them to use around
the same speed for all trials. Additionally, previous research has shown that an individual’s exploration
dynamics affect stiffness perception [103], motivating us to not further limit tapping styles. During train-
ing, we also placed a different plate on the end-effector, and we asked them to try changing the modes by
the buttons (to feel both stiffnesses of the device), and we mentioned that one of the modes (low stiffness)
is soft and the other (high stiffness) is hard.
4.3.5 ResultsofExperiment2
Figure 4.7-(a) shows a boxplot of Weber fraction for all reference stiffnesses versus the hardness of plates
(large boxplots). To understand and visualize the effect of reference stiffness, inside each large box, we
show a different boxplot that groups the results based on their reference stiffness.
A two-way ANOVA on the Weber fraction for all stiffnesses with plate hardness and reference stiff-
nesses (500 N/m,1000 N/m,1500 N/m, and 2000 N/m) as factors indicated that the Weber fractions
were statistically different across different hardnesses ( F(4,395) = 18.58,p< 0.001), and across different
reference stiffnesses ( F(4,396) = 28.63,p < 0.001). The interaction between these two factors was not
significant. A Tukey’s post-hoc pairwise comparison test further evaluated the effects of the plate hardness
on the Weber fraction of stiffnesses. The results showed that the Weber fractions for P1 was significantly
63
lower than that of P3 (p< 0.01), P4 (p< 0.001), and P5 (p< 0.001), but not from P2. The Weber fraction
of P2 was also significantly lower than that of P3 ( p = 0.05), P4 (p < 0.001), and P5 (p < 0.001). There
was also a significant difference between the Weber fractions of P3 from P5 ( p < 0.01) and not from P4.
There was no significant difference in the Webster fraction between P4 and P5.
Figure 4.7-(b) shows a boxplot of the Weber fraction of stiffnesses versus the reference stiffnesses
combined for all hardnesses. We ran a one-way ANOVA on this combined Weber fraction of stiffness with
reference stiffness as a factor. This analysis indicated a statistically significant difference in Weber fraction
across reference stiffnesses ( F(3,396) = 24.31,p < 0.001). A Tukey’s post-hoc pairwise comparison
test further evaluated the effects of the reference stiffness on the Weber fraction of stiffnesses. The Weber
fraction for reference stiffness 500 N/m was significantly lower than that of 1500 N/m ( p < 0.001) and
2000 N/m (p < 0.001). The Weber fraction for reference stiffness 1000 N/m was also significantly lower
than that of 1500 N/m (p< 0.01) and 2000 N/m (p< 0.001). There were no other significant differences.
Figure 4.7-(c) shows the boxplot of Weber fraction of stiffnesses versus tapping time. Tapping time is
the duration of one average tap (among all the taps) for each trial (finding the average tap among all taps
is shown in Figures 4.1-(a),(b). The minimum and maximum of the selected tap durations were 20 ms and
288 ms, respectively. We grouped these durations into four categories: “Very long" (221 ms-288 ms), “Long"
(154 ms-221 ms), “Short" (87 ms-154 ms), and “Very short" (20 ms-87 ms). We ran a one-way ANOVA on
the Weber fraction of stiffness with tapping time as a factor. This analysis showed that the Weber fraction
is statistically significant across different tapping times ( F(3,396) = 10.55,p < 0.001). A Tukey’s post-
hoc pairwise comparison test further evaluated the effects of the tapping times on the Weber fraction of
stiffnesses. The results showed that the Weber fraction for very short duration taps was significantly lower
than that for taps of long (p< 0.01) and very long (p< 0.001) duration. Similarly, the Weber fraction for
short-duration taps was significantly lower than that for very long taps ( p < 0.01). There were no other
significant differences.
64
In a post-experiment survey, we asked the participants to indicate for which type of surface, hard or
soft, they thought it was easier to distinguish the difference in stiffness (Fig. 4.8. Nine participants found
hard surfaces easier to distinguish, and 11 found softer surfaces easier. Note that almost all participants
could distinguish the stiffnesses for softer surfaces better, as indicated by a smaller Weber fraction, even if
they thought harder surfaces were easier to distinguish.
4.4 Discussion
In this chapter, we discussed our new approach for rendering both hard and soft surfaces shown in [116]
to be more realistic than traditional spring, spring-damper, and event-based methods. Our approach over-
comes some limitations of existing impedance and admittance-type haptic devices by rendering hardness
and stiffness together. In our approach, we modified an impedance haptic device by detaching the stylus
from the body of the device and using the end-effector as the primary interaction point, creating an ETHD.
Our design has two significant benefits; the system can generate a hard surface even given the stability and
stiffness constraints of the impedance device, and the user can move the stylus effortlessly while there is no
contact due to its untethered nature. The studies conducted in this chapter focused on determining if the
P1 P2 P3 P4 P5
Plate Hardness (P1=Softest, P5=Hardest)
0.1
0.2
0.3
0.4
0.5
0.6
Weber Fraction
Softer plates are easier to distinguish
Harder plates are easier to distinguish
Figure 4.8: Comparison of Weber fraction of stiffnesses versus plate hardness for people who think hard
surfaces are easier to distinguish versus people who think soft surfaces are easier. Each point indicates the
average Weber fraction for all four reference stiffnesses (500 N/m, 1000 N/m, 1500 N/m, and 2000 N/m) for
that plate hardness.
65
rendered hardness could mask the rendered stiffness, allowing users to perceive a hard surface even when
the underlying stiffness was low. We evaluated the hardness perception of tapping on the end-effector
of the ETHD while both end-effector hardness and device stiffness were changed. Our studies provided
important insights into the strengths of our rendering approach and the importance of the masking effects
between stiffness and hardness.
We first quantitatively tested the effect of stiffness on the perceived hardness using measurements of
the spectral centroid of the tapping transient, which is shown to be a good predictor of hardness perception
ratings [22]. The tapping transient has a higher spectral centroid for harder surfaces and a lower spectral
centroid for softer surfaces, as shown in Figure 4.1. Interestingly, we did not see a noticeable difference
in the spectral centroid for a given hardness when the rendered stiffness was changed. This indicates
that the hardness of the surface dominated the transient portion of the force during contact. However,
the stiffness may have affected the contact’s lower frequency or steady-state forces. This quantitative
result suggests that underlying rendered stiffness might not significantly affect the hardness perception.
However, the dominant frequency of force changes with both hardness and stiffness, indicating that the
rendered stiffness affects the frequency of the tapping transients. The magnitude of the spectral centroid
was fairly constant across the 11 tested materials except for the softest material (Shore 10 OO), which
showed a significantly higher magnitude. One possible reason is that the force is overdamped during the
tap on the softest plate (Figure 4.1 shows an example of this signal behavior.
We then qualitatively tested the effect of stiffness on perceived hardness in Experiment 2. In this
experiment, we found that the Weber fraction of stiffness is highly dependent on the hardness of the end-
effector. The harder the plate is, the higher the Weber fraction is. These results show that when interacting
with a hard surface, it is difficult for users to recognize and understand variations in forces from the device
due to a change in stiffness. The ability of a user to distinguish changes in stiffness becomes worse as
the hardness increases. As such, the hardness of the end-effector acts to mask the stiffness of the device.
66
Similar masking effects on other haptic modalities have also been found [92, 122]. This result shows that in
our approach, as long as we augment the interaction point of the end-effector with a small hard material,
the device does not need to generate the actual desired force because the perceived hardness increases
without actually increasing the stiffness. For example, if the user cannot distinguish between 1500 N/m
and 2000 N/m, there is no need to use a device with better actuators to generate 2000 N/m.
Our results also show that the Weber fraction increases with increased reference stiffness, regardless
of the hardness. This means that users cannot easily distinguish between changes in rendered stiffness
when the underlying stiffness is high. This also shows promise for our system being used with low-cost
haptic devices that cannot generate high stiffnesses or do not have accurate stiffness control. This masking
effect is a key advantage of our system because it allows us to still render a perceptually hard surface, even
if the device is limited in its stiffness output. Interestingly, these results are matched with [9], in which the
authors found that the force/displacement ratio influences the judgment about material compliance and
hardness perception in softer materials.
On average, the Weber fraction of stiffness was lower when participants tapped on softer plates. Fig-
ure 4.8 compares the Weber fraction based on the participants’ answers of which plates are easier to dis-
tinguish. Subjects who chose softer plates as being easier to distinguish always had higher Weber fraction,
meaning they had a lower perception of the underlying stiffness compared to the other group. Regardless
of which plates they chose as easier to distinguish, both groups always distinguished the soft plates better.
Since stiffness is the resistance of an object to elastic deformation and hardness is its resistance to localized
surface deformation, materials with a lower stiffness could also have a lower hardness. The compliance of
softer end-effectors, in addition to contributing to the hardness, might affect the rendered stiffness because
of their noticeable elastic deformation. This dual modality is one reason why users can perceive the device
stiffness more when a softer end-effector material is used.
67
The analysis shows that additional factors, including how the user interacts with the device, may also
affect their perception of the hardness during tapping. For example, the more quickly a user taps (i.e.,
the less time they stay in contact with the end-effector), the more difficult it is for them to understand
changes in stiffness. This difference is important because users will likely stay in contact with the surface
for longer during interactions with virtual objects. We leave it for future research to determine how the
masking effects that were seen in tapping translated to static and sliding contact. We also saw an effect
of force during the tap on the Weber fraction for reference stiffnesses 1000 N/m and 1500 N/m; users that
applied a larger force to the end-effector could more easily distinguish between the rendered stiffnesses.
However, we did not see an effect of tapping speed on the Weber fraction in our experiment, meaning that
hardness perception and the masking effect were not affected by the speed of contact.
These results show that our new system takes advantage of these masking effects to render virtual
surfaces more realistically and with a lower cost over traditional methods. Our system has the added
benefit of minimizing friction and inertia of the interaction through its untethered design. Note that our
results and trends indicate the existence of a stiffness masking effect rather than reporting a general Weber
fraction of stiffness and hardness. Weber fractions could be different for each device depending on its
accuracy.
Our system can be used effectively in many applications where rendering hard surfaces are required.
This could be done by augmenting a general model of the object to be displayed and then separately
controlling the stiffness, hardness, and texture of the model. For example, in orthopedic training, PVC
pipes are mainly used to simulate bone drilling because no haptic device can simulate free space and hard
surfaces. This haptic system could enable a shift in current education training systems. It could also play an
important role in different teleoperation applications because it overcomes previous limitations, including
generating high and low-force feedback and tactile feedback simultaneously.
68
Although our system has many advantages, some limitations exist, including a limited workspace
and the need to manually change the end-effector to alter the displayed hardness. We address this latter
limitation in the next chapter by making mechanical changes to the system so that it can display multiple
hardnesses simultaneously. Another limitation of the system is the need to interact with the device through
a tool. However, it is still unclear if our results can be applied directly to bare-finger interactions since
previous work [63] showed that hardness discrimination in bare-finger interactions is predominately based
on static pressing rather than tapping as in tool-based interactions. We additionally explore the direction
of bare-finger hardness and stiffness perception in Chapters 5 and 6.
4.5 Summary
In this chapter, we discussed the perception of our ETHD rendering approach, which has been shown to
be more realistic than previous methods for rendering hard objects. This system is created by detaching
the stylus of a haptic device and attaching plates with different hardnesses to the end-effector; an un-
tethered stylus is used to interact with the end-effector. Stiffness is rendered using the haptic device, and
hardness is rendered using the physical hardness of end-effector plates. We conducted two experiments to
quantitatively and qualitatively determine how changing the hardness affects the perception of the inter-
action regarding the device’s stiffness. The results show the importance of hardness in masking the haptic
device’s stiffness and indicate that more attention should be paid to rendering hardness.
69
Chapter5
RenderingHardnessandStiffnessUsingaDynamicEnd-Effectorinan
Encountered-TypeHapticDisplay
In this chapter, we build on prior findings from Chapters 3 and 4 in using an ETHD to render both hardness
and stiffness. Our novel approach renders both hardness and stiffness using a modified haptic device with
a dynamic end-effector. This system allows for simultaneous adjustment of both hardness and stiffness
based on the needs of the application and virtual environment, which is crucial for realistic simulations in
various fields such as medicine and engineering.
To evaluate the effectiveness of our approach, we conducted two human subject experiments where
participants were asked to tap on virtual blocks and sort them based on their perceived hardness and stiff-
ness. The first experiment involved tapping with a stylus, while participants tapped on the blocks using
their bare-finger in the second experiment. Through this approach, we aim to address the limitations of
existing hardness rendering methods in haptics and provide a more realistic and accurate haptic feed-
back system for various applications, particularly in fields where haptic feedback is critical for realistic
simulations.
70
Figure 5.1: Flow diagram of our control system.
5.1 RenderingMethods
Our approach involves the modification of a haptic device into an ETHD so that it can render both stiffness
and hardness. We use the same modified haptic device that was used in the previous chapters and make
it the base of our system; we then add a dynamic end-effector to enable real-time adjustments of the
displayed hardness based on the user’s location in the virtual environment. The decoupled nature of
ETHDs allows for the generation of a realistic hardness sensation through a collision impact between
the stylus and the device’s dynamic end-effector. This collision generates both transient and continuous
sensations that mimic the tactile experience of touching a surface with that specific hardness. The virtual
stiffness, controlled by the haptic device, complements this tactile sensation by providing feedback about
the level of force required to penetrate the virtual object. The full rendering approach is shown in Fig. 5.1.
71
Tracker
Dynamic
End-effector
Tracker
(a) (b) (c)
Figure 5.2: Our ETHD system, (a) The base of our system, which is a modified haptic device to render
stiffness with a custom dynamic end-effector to render hardness. One tracker is located on the body of the
device. (b) and (c) Experiments in which the user interacts with the end-effector through a stylus or a bare
finger (note that all the colored tapes are added only for the visual aid purpose of images and they did not
exist throughout the study).
5.1.1 DynamicEnd-EffectorDesign
Our system utilizes the haptic device’s actuators to render stiffness, while the end-effector surface is re-
sponsible for rendering hardness. We create a dynamic end-effector to enable real-time switching between
different levels of hardness depending on the location of the interaction. This end-effector consists of a
hollow 3D-printed triangular prism drum, with each of its three sides featuring a plate with a different
hardness (Figure 5.2-(a)). Depending on the user’s position in the virtual scene, only one of these plates
is pointed upwards at a given time and serves as the current interaction point between the stylus and the
device.
Our previous study in Chapter 3, demonstrated that it is possible to simulate a wide range of materials
with varying levels of stiffness and hardness using different stiffnesses and only three different hardness
levels (in the categories of soft, medium, and hard). Therefore, to make the system more generic and
to ensure equal time spent rotating the end-effector from one plate to another during an interaction, our
72
system includes three plates with different hardnesses. We use the Shore Durometer scale, which measures
the penetration depth under a certain load, to determine the hardness of these materials. The selected
hardness levels were extra soft (Shore 10 OO), medium (Shore 60 A), and extra hard (Shore 75 D). Each
plate is 50.8 mm× 50.8 mm in size and 12.7 mm in thickness. The thickness was chosen to prevent the
tooltip from reaching the end of the plate when tapping. By employing this design, we ensure that our
system can simulate a range of materials with different hardness levels while maintaining accuracy and
consistency in our results. Note that it is always possible to use more plates with different hardness levels
or in different ranges based on the resolution of the application.
We positioned a DC motor (Maxon, model 344516) at the center of the hollow triangular prism drum
to switch between the three different surfaces. This actuator was chosen because it is small in size, non-
backdrivable, and has a gearbox with a 16.58:1 ratio (the 2-stage planetary type with ball bearings), making
it suitable for our needs. Additionally, the actuator has a two-channel encoder and a no-load speed of
11000 rpm at 12V. To drive this actuator, we employed a Maxon EPOS4 Compact 50/5 CAN motor driver.
To connect the actuator to the end-effector of the device, we designed a custom 3D-printed joint with a
90-degree angle. The joint’s design was such that the center of the plates aligned with the center of the
end-effector on the X-Y axes. This allowed for smooth and precise rotation between the different surfaces,
ensuring an accurate and reliable rendering of the different levels of hardness and stiffness. After we
assembled the dynamic end-effector, we used EPOS studio software to tune the PID values in the position
control mode.
5.1.2 DeviceControl
Typical kinesthetic haptic devices have a stylus directly attached to the end-effector, making it easy to
determine the user’s hand location. However, our ETHD system cannot determine the hand location until
73
it touches the end-effector. Therefore, we must use additional sensors to externally locate the hand in
order to facilitate interaction with the end-effector in different workspace locations.
To enable real-time rendering of stiffness and hardness in every workspace location, we must first
locate and track the user’s finger or stylus. Since the system is an ETHD, it must follow the user so that the
surface is ready to be displayed at all times when the user contacts a virtual object. When the user enters
the region of a virtual object but has not yet made contact with it, we rotate the end-effector to display the
appropriate hardness for that object and update the rendered stiffness. This anticipation of the device is
necessary to ensure a real-time response. Our system allows for a variety of interaction methods with the
end-effector, including the use of different stylus types or even a bare finger.
Into the stylus, we integrated an Ascension trakSTAR 6 DOF magnetic tracker (Model 180) with a 0.5
mm resolution, positioned at the front of a custom 3D-printed pen-shaped tool measuring 12 cm in length
and 1cm in diameter (Fig. 5.2-(c)). The magnetic tracker is located at the front of the stylus, and the stylus
has a hemispherical tip with a 4 mm diameter and 15 mm length to ensure that only one point interacts
with the end-effector surface.
For interaction with the end-effector using a bare finger, our system uses a second Ascension trakSTAR
6 DOF magnetic tracker (Model 180) attached to the user’s fingernail to track the position of the fingertip
(Fig. 5.2-(b)). Careful consideration was taken when selecting the tracker’s size and position to ensure
optimal tracking accuracy and minimal interference with the user’s natural movement. This magnetic
tracker is a small, lightweight sensor with a 2 mm diameter and 9.9 mm length. With this fingertip-tracking
method, our system offers a more intuitive approach for interacting with the end-effector without requiring
a dedicated tool or stylus. This enables a wider range of interactions and greater flexibility in the tasks
that the haptic device can be used for.
To accurately locate the position of the stylus or fingertip relative to the haptic device, we attach an
Ascension trakSTAR Model 800 magnetic tracker to the center of the device (Figure 5.2-(a)). By subtracting
74
the position of the haptic device from the position of the stylus or fingertip, we obtain the relative position
between the two, which we then use to determine the position of the stylus or fingertip with respect to
the haptic device in 3D space. To obtain an accurate relative position, it is important to factor in the offset
between the end-effector and the stationary body of the device when the position in the x-y axes is zero.
Therefore, during the setup process, we enter a calibration mode and locate the stylus tip or fingertip on
the center of the end-effector to find the offset between the two sensors. Once the calibration is completed,
we subtract the offset from the position of the stylus or fingernail to obtain the correct relative position.
After finding the relative position between the body of the haptic device and the stylus we have to
find the relative position with respect to the end-effector. We know the end-effector’s position based on
the encoder values and we always want to move the end-effector in the x and y axis to be matched with
the relative position of the stylus, meaning that the end-effector always follows the user’s hand so that we
can create a tangible experience as soon as the user wants to interact with a virtual object in that location.
Therefore, we find where the user is going and one cycle after we temporarily switch the haptic device
to position-control mode and move the end-effector to that position. In the meantime, we command the
EPOS driver to rotate the end-effector to the correct angle to display the appropriate surface. Based on
our virtual environment we already know our desired stiffness and hardness for each location, so we also
adjust the rendered force of the device to get the desired stiffness and adjust the end-effector’s angle to
reach the desired hardness.
5.2 ExperimentalMethods
We conducted two experiments to understand how users perceive material properties while both hardness
and stiffness are rendered. In the first experiment the user interacted with the device using a stylus and in
the second experiment using a bare finger.
75
Figure 5.3: Visual scene of our experiments consisting of four yellow blocks that represent different levels of
hardness and stiffness. Participants were instructed to order the blocks from soft to hard using a keyboard.
As each block was selected, it disappeared from the blue box and moved to the left side. The position of
the participant’s finger or stylus was represented by a red cursor, which showed its location relative to the
center of the device.
5.2.1 Setup
During our experiments, we presented four virtual blocks separated by a wall (Figure 5.3). The walls in
between the blocks are tall so that the user has to lift their finger or stylus off of the end-effector when
switching between the blocks, ensuring that there is no interaction with the device when the end-effector
is rotating.
The blocks, labeled 1–4 were all of the same colors and represented surfaces with varying stiffness
and hardness. We rendered the stiffness using our haptic device, and the hardness using the device’s end-
effector. The device was located under the user’s dominant hand. Note that in Figure 5.3 the blue box was
stationary, and the red ball moved based on the location of the user’s finger or stylus with respect to the
center of the haptic device. The virtual scene was created using OpenGL in Chai3D and displayed on a
monitor screen in front of the user.
76
In Experiment 1, the user interacted with the end-effector of the haptic device using a stylus that
tracked their location (Figure 5.2-(c)). In Experiment 2, the user interacted with the end-effector using a
bare finger with a tracker attached to the fingernail and tape attached to mask tactile cues on the fingertip
(Figure 5.2-(b)). A small sphere cursor on the screen indicated the relative position of the user’s stylus or
finger with respect to the device. Once the relative position of the stylus or finger was identified to fall
within the x-y range of any of the virtual blocks, the end-effector immediately moved to that location.
However, if the relative position of the tracker remained within the x-y limits of the current block, the
end-effector did not move. After we moved the block, we use the magnetic tracker to visually show the
tracker’s location in the virtual environment (red cursor in Figure 5.3); the user’s position as measured by
the magnetic tracker is not used for the rendering calculations. To render the haptic surface, we use the
encoder data of the haptic device.
To complete the task, the participant had to sort the blocks from softest to hardest by tapping on
them. The user selected a block by pressing a key on the keyboard. As soon as the user pressed a key, the
corresponding block disappeared from the virtual scene and appeared in the ordered stack on the left-hand
side. The user repeated this process until all blocks were sorted, and then pressed the enter button to start
a new set of hardness and stiffness values.
We chose tapping as the primary interaction behavior in our experiments because our goal is to study
the relationship between stiffness and hardness. Studies have shown that tapping is a common way to
explore the perception of properties of materials [63]. Therefore, we expect tapping to be a suitable method
for users to interact with the virtual objects and provide feedback on the perceived hardness.
5.2.2 Procedure
We recruited 10 participants for the study (4 female, 6 male; 22-36 years old), all of whom were right-handed
and completed both experiments. Prior to the experiments, participants were provided with headphones
77
to block out any audio cues from the device and sat on a chair with the device placed on a table beside
them. They used their dominant hand to interact with the device by holding the stylus or using their index
finger. A keyboard with four keys enabled was placed in front of them for selecting one of the blocks with
their non-dominant hand.
Participants were instructed not to look at the device during tapping to avoid any visual effects, but
they were allowed to look at the device and adjust the stylus’ location between taps. The device was
placed below the armrest on a low table, so that the end-effector was in line with where participants sat,
and the motion of the device was not visible to them if they were looking straight ahead. Participants were
instructed to only focus on the blocks on the monitor.
During the experiment, participants were free to tap on each stimulus as much as they preferred while
keeping the stylus vertical for the first experiment. Participants were not instructed on how to grasp the
stylus, even though it has been shown that grasping styles can affect perception [117] because we wanted
them to concentrate on perception rather than maintaining a particular grasp. They were advised to hold
the stylus vertically, much like a pencil, and were not restricted in terms of how many times they could
tap. They were also not instructed on the distinction between hardness and stiffness to prevent any bias
or confusion. Before the actual experiment, there was only a training session where participants were
taught how to tap and with what approximate speed to tap. Although the contact velocity of taps was not
controlled during the experiment, an example of tapping speed was shown to participants, and they were
requested to use the same speed for all trials. Additionally, previous research suggested that an individual’s
exploration dynamics affect stiffness perception, thus we did not further limit tapping styles [103].
5.2.3 ExperimentalConditions
Each trial involved four blocks, each with a unique combination of two stiffness and two hardness levels.
We used three different stiffness levels (low (1000 N/m), medium (1500 N/m), and high (2000 N/m)) and
78
three different hardness levels (extra soft (Shore 10 OO), medium (Shore 60 A), and extra hard (Shore
75 D)), which were selected based on the previous chapter.In total, there were nine unique cases, made
up of all possible combinations of stiffness and hardness levels. To ensure reliability and validity, each
experiment was repeated twice, resulting in a total of 18 trials per experiment. The order of conditions
was pseudo-randomized for each participant and the order of Experiments 1 and 2 was counter-balanced
across participants to minimize any potential order effects.
5.3 Results
Below we present the results of our experiments. For simplicity in our discussion of the results, we denote
Hardness asH and Stiffness as S For each hardness we denotel for extra soft (Shore 10 OO),m for medium
(Shore 60 A), and h for extra hard (Shore 75 D). Also, we denote l = 1000 N/m, m = 1500 N/m, and
h = 2000 N/m for the stiffnesses generated by the device.
5.3.1 Experiment1: Stylus
Figure 5.4 shows the results of Experiment 1 for nine different combinations of stiffness and hardness levels.
This result indicates how participants sorted the blocks from soft to hard while tapping with a stylus. Since
in each of the nine cases, the combination of stiffness and hardness levels is different, for each case we
separately ran a two-way ANOVA on the sorting results with rendered stiffness and hardness as factors.
The results of the ANOVAs with hardness as a factor are shown in Table 5.1 and with stiffness as a factor in
Table 5.2. In these tables, the yellow cells indicate statistically significant values, while the red cells indicate
non-significant ones. This analysis indicated that regardless of the different combinations of stiffness and
hardness values, hardness was a significant factor in participants’ sorting. However, stiffness was only a
significant factor in less than half of the cases (44%).
79
H
l
-S
l
H
l
-S
h
H
h
-S
l
H
h
-S
h
Rendered Surface
80
80
60
40
40
60
20
0
0
20
0
0
0
0
0
0
0
20
40
60
80
H
l
-S
l
H
l
-S
m
H
h
-S
l
H
h
-S
m
70 30
70
55
45
45
50
25
0
5
0
0
0
0
0
5
0
20
40
60
80
H
l
-S
m
H
l
-S
h
H
h
-S
m
H
h
-S
h
60
40
40
60
50
50
50
50
0
0
0
0
0
0
0
0
0
20
40
60
80
H
l
-S
l
H
l
-S
h
H
m
-S
l
H
m
-S
h
Rendered Surface
70
70
60
35
35
60
25
5
0
25
0
5
0
5
5
0
0
20
40
60
80
H
l
-S
l
H
l
-S
m
H
m
-S
l
H
m
-S
m
60
35
30
50
45
45
30
55
5
0
20
0
5
5
5
10
0
20
40
60
80
H
l
-S
m
H
l
-S
h
H
m
-S
m
H
m
-S
h
60
35
35
55
50
45
45
45
0
5
5
5
0
5
5
5
0
20
40
60
80
First
Second
Third
Fourth
H
m
-S
l
H
m
-S
h
H
h
-S
l
H
h
-S
h
Rendered Surface
45
40
45
45
45
35
35
60
15
0
5
5
10
10
0
5
0
20
40
60
80
First
Second
Third
Fourth
Selection from softest to hardest
H
m
-S
l
H
m
-S
m
H
h
-S
l
H
h
-S
m
55
45
45
50
40
60
55
40
0
0
5
0
0
0
0
5
0
20
40
60
80
First
Second
Third
Fourth
H
m
-S
m
H
m
-S
h
H
h
-S
m
H
h
-S
h
45
50
55
45
50
50
45
50
5
0
0
0
0
0
0
5
0
20
40
60
80
Sorting surfaces from soft to hard while tapping on them with a stylus
Percent
Figure 5.4: Confusion matrices indicating how participants sorted the blocks from soft to hard while tap-
ping with a stylus.
Table 5.1: The results of a 2-way ANOVA analysis, examining the main effect of hardness and its interaction
with stiffness when interacting with rendered blocks using a stylus.
Hardness: LandH
Stiffness: LandH
Hardness: LandH
Stiffness: LandM
Hardness: LandH
Stiffness: MandH
F(1,77) = 366.67
p <0.001 ,η 2
= 0.80
F(1,77) = 173.95
p <0.001 ,η 2
= 0.69
F(1,77) = 311.11
p <0.001 ,η 2
= 0.80
Hardness: LandM
Stiffness: LandH
Hardness: LandM
Stiffness: LandM
Hardness: LandM
Stiffness: MandH
F(1,77) = 152.23
p <0.001 ,η 2
= 0.65
F(1,77) = 77.24
p <0.001 ,η 2
= 0.48
F(1,77) = 105.97
p <0.001 ,η 2
= 0.58
Hardness: MandH
Stiffness: LandH
Hardness: MandH
Stiffness: LandM
Hardness: MandH
Stiffness: MandH
F(1,77) = 74.74
p <0.001 ,η 2
= 0.48
F(1,77) = 200.34
p <0.001 ,η 2
= 0.72
F(1,77) = 168.12
p <0.001 ,η 2
= 0.69
We also measured the time of completion for each trial, but there was no correlation between the time
and the various combinations of stiffness and hardness.
80
Table 5.2: The results of a 2-way ANOVA analysis, examining the main effect of stiffness and its interaction
with hardness when interacting with rendered blocks using a stylus.
Hardness: LandH
Stiffness: LandH
Hardness: LandH
Stiffness: LandM
Hardness: LandH
Stiffness: MandH
F(1,77) = 14.67
p <0.01 ,η 2
= 0.03
F(1,77) = 3.18
p = 0.078 ,η 2
= 0.01
F(1,77) = 0.78
p = 0.38 ,η 2
= 0
Hardness: LandM
Stiffness: LandH
Hardness: LandM
Stiffness: LandM
Hardness: LandM
Stiffness: MandH
F(1,77) = 5.76
p <0.05 ,η 2
= 0.03
F(1,77) = 6.51
p <0.05,η 2
= 0.04
F(1,77) = 0.37
p = 0.55 ,η 2
= 0
Hardness: MandH
Stiffness: LandH
Hardness: MandH
Stiffness: LandM
Hardness: MandH
Stiffness: MandH
F(1,77) = 3.81
p <0.05 ,η 2
= 0.02
F(1,77) = 0.14
p = 0.71 ,η 2
= 0
F(1,77) = 0.49
p = 0.48 ,η 2
= 0
5.3.2 Experiment2: BareFinger
Figure 5.5 shows the result of Experiment 2 for the same nine different combinations of stiffness and
hardness as Experiment 1. This result indicates how participants sorted the blocks from soft to hard while
tapping on these blocks with a bare finger. Since in each of the nine cases, the combination of stiffness and
hardness levels is different, for each case we ran a two-way ANOVA on the sorting results with rendered
stiffness and hardness as factors. The results of the ANOVAs with hardness as a factor are shown in Table
5.3 and with stiffness as a factor are shown in Table 5.4. In these tables, the yellow cells indicate statistically
significant values, while the red cells indicate non-significant ones. This analysis indicated that regardless
of the different combinations of stiffness and hardness, hardness was a significant factor in participants’
sorting. However, stiffness was only a significant factor in 1/3 of the cases.
We also measured the time of completion for each trial, but there was no correlation between the time
and various combinations of stiffness and hardnesses.
81
H
l
-S
l
H
l
-S
h
H
h
-S
l
H
h
-S
h
Rendered Surface
65
65
55
35 45
10
20
5
15
5
15
5
5
15
20
20
0
20
40
60
H
l
-S
l
H
l
-S
m
H
h
-S
l
H
h
-S
m
60
30
30
55
50
35
40
50
10
0
0
15
0
15
10
0
0
20
40
60
H
l
-S
m
H
l
-S
h
H
h
-S
m
H
h
-S
h
55
30
35
55
45
40
40
50
5
10
10
0
5
10
5
5
0
20
40
60
H
l
-S
l
H
l
-S
h
H
m
-S
l
H
m
-S
h
Rendered Surface
65
60
50
35
30
50
20
5
10
20
15
5
0
15
15
5
0
20
40
60
H
l
-S
l
H
l
-S
m
H
m
-S
l
H
m
-S
m
50
35
40
50
55
35
30
55
15
0
0
10
5
5
5
10
0
20
40
60
H
l
-S
m
H
l
-S
h
H
m
-S
m
H
m
-S
h
60
25
25
55
45
35
35
50
5
10
15
5
15
5
0
15
0
20
40
60
First
Second
Third
Fourth
H
m
-S
l
H
m
-S
h
H
h
-S
l
H
h
-S
h
Rendered Surface
55
30
35
55
45
35
40
55
10
5
5
5
5
15
5
0
0
20
40
60
First
Second
Third
Fourth
Selection from softest to hardest
H
m
-S
l
H
m
-S
m
H
h
-S
l
H
h
-S
m
55 30
45
30
40
40
40
20
20
5
10
15
15
15
0
20
0
20
40
60
First
Second
Third
Fourth
H
m
-S
m
H
m
-S
h
H
h
-S
m
H
h
-S
h
45
45
45
30
35
50
45
35
5
5
15
10
5
10
5
15
0
20
40
60
Sorting surfaces from soft to hard while tapping on them with index Finger
Percent
Figure 5.5: These confusion matrices indicate how people sorted the blocks from soft to hard while tapping
with a finger.
Table 5.3: The results of a 2-way ANOVA analysis, examining the main effect of hardness and its interaction
with stiffness when interacting with rendered blocks using a bare finger.
Hardness: LandH
Stiffness: LandH
Hardness: LandH
Stiffness: LandM
Hardness: LandH
Stiffness: MandH
F(1,77) = 18.31
p <0.001 ,η 2
= 0.18
F(1,77) = 72.33
p <0.001 ,η 2
= 0.48
F(1,77) = 63.93
p <0.001 ,η 2
= 0.45
Hardness: LandM
Stiffness: LandH
Hardness: LandM
Stiffness: LandM
Hardness: LandM
Stiffness: MandH
F(1,77) = 85.06
p <0.001 ,η 2
= 0.51
F(1,77) = 58.34
p <0.001 ,η 2
= 0.42
F(1,77) = 45.95
p <0.001 ,η 2
= 0.36
Hardness: MandH
Stiffness: LandH
Hardness: MandH
Stiffness: LandM
Hardness: MandH
Stiffness: MandH
F(1,77) = 72.98
p <0.001 ,η 2
= 0.48
F(1,77) = 23.27
p <0.001 ,η 2
= 0.22
F(1,77) = 44.33
p <0.001 ,η 2
= 0.36
5.4 Discussion
The results of the experiments provide insights into how users perceive and sort virtual materials based on
their stiffness and hardness properties. Figures 5.4 and 5.5 and the two-way ANOVA analyses showed that
there was a statistically significant difference in how participants sorted the blocks based on their hardness,
82
Table 5.4: The results of a 2-way ANOVA analysis, examining the main effect of stiffness and its interaction
with hardness when interacting with rendered blocks using a bare finger.
Hardness: LandH
Stiffness: LandH
Hardness: LandH
Stiffness: LandM
Hardness: LandH
Stiffness: MandH
F(1,77) = 6.14
p <0.05 ,η 2
= 0.06
F(1,77) = 1.2
p = 0.28 ,η 2
= 0
F(1,77) = 1.14
p = 0.29 ,η 2
= 0
Hardness: LandM
Stiffness: LandH
Hardness: LandM
Stiffness: LandM
Hardness: LandM
Stiffness: MandH
F(1,77) = 4.07
p <0.05 ,η 2
= 0.03
F(1,77) = 3.4
p = 0.07 ,η 2
= 0.02
F(1,77) = 3.1
p = 0.09 ,η 2
= 0.02
Hardness: MandH
Stiffness: LandH
Hardness: MandH
Stiffness: LandM
Hardness: MandH
Stiffness: MandH
F(1,77) = 5.28
p <0.05 ,η 2
= 0.05
F(1,77) = 1.9
p = 0.17 ,η 2
= 0.01
F(1,77) = 0.24
p = 0.62 ,η 2
= 0
independent of the various combinations of stiffness and hardness levels and independent of if they tap
with a bare finger or a stylus. This result suggests that participants were able to distinguish between
different levels of hardness, regardless of the specific stiffness level of the block. Therefore, hardness may
be a more salient perceptual feature for individuals than stiffness when it comes to object perception. This
finding is particularly interesting because it suggests that the tactile feedback provided by the blocks may
play an important role in how users perceive and categorize the materials.
Interestingly, the results also showed that only in a few cases were participants able to differentiate
between blocks with different levels of stiffness. In general, it seems that the easiest cases to sort were
those with a combination of low and high stiffness. Note that high stiffness is twice as stiff as low stiffness
(1000 N/m versus 2000 N/m). This indicates that only when the difference between stiffness levels is large,
the stiffness of the blocks could impact how accurately participants were able to sort the blocks. Otherwise,
the effect of hardness on how participants perceive the block properties is much stronger.
Our results indicate the presence of stiffness masking, which refers to the phenomenon where per-
ceived differences in stiffness between objects are reduced when those objects have a high level of hard-
ness. This stiffness masking was previously shown to exist in Chapter 4. The confusion matrices are shown
in Figures 5.4 and 5.5 support this phenomenon. In all cases where the hardness is high or medium, people
83
randomly chose between different stiffness levels, suggesting that the hardness level of the blocks may
have been high enough to mask the perceived differences in stiffness. In other words, the participants may
have had a harder time distinguishing between the stiffness levels of the blocks when the hardness level
was high.
These findings highlight the complex nature of object perception and how multi-modal cues can work
together to influence our perceptions and judgments. It also emphasizes the importance of considering
multiple factors when designing studies or interventions aimed at improving object perception or haptic
discrimination abilities.
Since our study shows that the ANOVA results were consistent whether participants used their bare
fingers or a stylus to tap on the blocks, this indicates that the perceptual cues used to distinguish different
levels of hardness and stiffness may not rely on the method of touch. However, there was a notable differ-
ence in the magnitude of F-values and effect size between the two tapping methods. When using a stylus,
all combinations of hardness and stiffness yielded highly significant F-values and effect size as well as low
p-values. Using a bare finger resulted in lower F-values and effect size with slightly higher p-values. It
is important to consider both statistical significance and effect size when interpreting the results. A high
effect size and F-value indicate a large difference between the groups being compared, whereas a low ef-
fect size and low F-value indicate a small difference. This suggests that tapping with a stylus may produce
more accurate and consistent haptic feedback, leading to stronger statistical results.
One possible explanation for the significant results in both tapping methods is that since transient
vibration is a crucial component used to differentiate between hardness levels, tapping on a surface with
a stylus or a fingertip produces similar vibration content and leads to statistically significant results in
both cases. However, the potential reason for the difference in statistical significance between the two
methods is the varying tactile feedback and vibration content provided. Tapping with a stylus may offer
more precise and consistent feedback compared to tapping with a bare finger, which can be influenced
84
by factors such as skin moisture and pressure. This difference in feedback could potentially affect how
participants perceive the hardness and stiffness of the blocks and impact their sorting decisions.
Another interpretation of the observed differences may be related to the kinesthetic movement and
joint dynamics involved in the two methods of touch. It is well known that kinesthetic movement can
influence our perception of stiffness, and thus the effect of this sensory cue combined with a tactile cue
may lead to different responses. This is evident from the confusion matrices in Figs. 5.4 and 5.5, where the
former appears to have more distinct selections across different levels of hardness, while the latter appears
to show more variability with the comparisons across different levels of hardness and stiffness.
It is worth noting that the completion time for each trial did not have any significant correlation with
the different cases. This suggests that the time taken by participants to sort the blocks did not significantly
influence their ability to differentiate between the blocks based on their hardness or stiffness.
5.5 Applications
Our device and findings could have practical implications in fields where understanding how users perceive
and interact with different materials is important. Three potential applications that require rendering of
both hardness and stiffness include medical simulation, online shopping, and gaming. Here we describe
implementations of different applications for each category.
5.5.1 Application#1: MedicalSimulation
We can develop a medical simulation application using our ETHD device that can render both hardness
and stiffness. This application would allow us to simulate different tissue types, such as organs, muscles,
and tumors, with varying hardness and stiffness properties.
One medical application would be for palpation, in which medical students or practitioners can interact
with the virtual organs or tissues using the haptic device and try to identify the different tissue types based
85
Figure 5.6: Abdominal palpation simulation. (a) Our ETHD with a custom end-effector for the purpose of
palpation training. Each surface has a different hardness and combined with a wide range of stiffnesses
rendered by the device we can simulate a variety of conditions. (b) The virtual scene that is rendered in
VR.
on their perceived hardness and stiffness (Figure 5.6). Our system can provide feedback to the user about
their accuracy in identifying the tissue types, as well as provide information about the specific tissue
types and their corresponding hardness and stiffness properties. Our system can also be used to simulate
different pathologies or conditions, such as tumors or inflammation, which can alter the hardness and
stiffness properties of the tissue. This simulation could provide valuable training for medical professionals
in identifying and diagnosing these conditions through palpation.
Furthermore, this device is low-cost and can be used to assess the proficiency of medical professionals
and to track their progress in improving their skills over time. For example, the system can record the time
taken by the user to identify the different tissue types and provide feedback on areas for improvement.
86
5.5.2 Application#2: Onlineshopping
With the increasing trend of online shopping, consumers are unable to touch and feel the products before
making a purchase. This lack of haptic feedback often leads to customer dissatisfaction and product returns.
However, with the use of our ETHD device that can render both hardness and stiffness, online shoppers
can get a more realistic sense of the product’s physical properties.
One possible application could be online shopping for furniture or home decor. Consumers can use
our device to feel the hardness and stiffness of materials such as wood, leather, or fabric. This would give
them a better idea of the product’s quality and durability. For instance, a potential buyer can use the device
to feel the stiffness and hardness of a sofa’s cushions or the hardness of a coffee table’s surface.
The use of a haptic device for online shopping can enhance the customer’s shopping experience by
providing them with a more realistic sense of the physical properties of the products they are interested
in and can lead to a more positive shopping experience.
5.5.3 Application#2: Gaming
One potential gaming application of our device that can render both hardness and stiffness is to enhance
the player’s immersion and realism in virtual environments. By incorporating haptic feedback based on
the stiffness and hardness of virtual objects, players can have a more realistic tactile experience, allowing
for more engaging and immersive gameplay.
For example, in a game where the player is required to interact with various virtual objects, such as a
puzzle game, the haptic device can provide feedback on the stiffness and hardness of these objects. This
could allow players to better distinguish between different materials, such as metal, wood, or plastic, and
provide a more realistic tactile sensation when interacting with these objects.
Also, in a game where the player is tasked with repairing a vehicle, this device with some modifications
to its end-effector could be used to simulate the sensation of tightening or loosening bolts, which could
87
vary their stiffness and hardness properties based on their material. By replacing the stylus with a wrench
the player could also interact with the end-effector using the wrench (note: for this application, we need
to replace our base haptic device with a 6-DOF device that can provide both force and torque). This device
could also be used to simulate the sensation of collision or impact when the player’s hand interacts with
different objects in the game. With the growing popularity of virtual reality and other gaming technologies,
the potential for this haptic technology in gaming applications is vast.
5.6 Summary
In this chapter, we design an ETHD system with a dynamic end-effector that can render different hard-
nesses based on the location of virtual objects. The results of the experiments provide insights into how
users perceive and sort virtual materials based on their stiffness and hardness properties. The findings
suggest that participants were able to distinguish between different levels of hardness, regardless of the
specific stiffness level of the blocks, indicating that hardness may be a more salient perceptual feature for
individuals when it comes to object perception. The tactile feedback provided by the blocks may also play
an important role in how people perceive and categorize the materials.
Furthermore, the results showed that only in a few cases were participants able to differentiate between
blocks with different levels of stiffness. The presence of stiffness masking was shown, where the perceived
differences in stiffness between objects are reduced when those objects have a high level of hardness.
The complexity of object perception and how multiple sensory cues can work together to influence our
perceptions and judgments were highlighted. It is important to consider multiple factors when designing
studies or interventions aimed at improving object perception or haptic discrimination abilities.
Finally, findings suggest that both tapping with a stylus and tapping with a bare finger can lead to
significant differences in the way people distinguish between materials’ properties. However, the method
of touch may have an impact on the strength of statistical results in perceptual experiments. It is also likely
88
that our brains use a combination of different types of information, including vibrations and other sensory
cues, to create a perceptual experience of the world around us.
89
Chapter6
EnergySpectrumastheKeyFeatureforHardnessPerceptualCue
The work presented in Chapters 2-4 focused on perception and rendering through a tool, but most of our
daily touch-based interactions are through bare fingers. In Chapter 5 we studied rendering both hardness
and stiffness through both a tool and bare finger and showed that discriminating hardness through tapping
was easier with a stylus than through a bare finger. This result motivates the deeper dive into bare finger
perception of hardness in this chapter.
Humans effortlessly construct a coherent, robust, and stable impression of the physical world through
their sense of touch. This is remarkable given that a consistent experience of material by feeling its texture,
for example, can result from a large variation in the finger’s mechanical responses. For sensory processing
involving touch, identifying features responsible for a stable representation of the world remains a major
challenge. For example, bare-finger contact with a surface produces vibrations from which the brain can
extract surface properties.
To understand the role of vibration in communicating material properties, such as the elasticity or
hardness of a surface, we simulate changes in the rendered hardness of an object through local vibration at
the fingertip. In order to change the hardness we generate additional vibration cues to match the vibration
during impact to the desired hardness by compensating for the missing contents from the impact vibration.
90
Fingertip elasticity changes over development and adapts to changes in hydration levels etc. on multi-
ple timescales [28], but our experience of material properties remains stable. To account for such variations
in real-time, and to determine what features of the vibration are robust reproducible features, we tested
under two different conditions. In this chapter, we first explore the estimated hardness and features of
transferred vibrations by tapping on different surfaces with a bare finger. Secondly, we update the finger-
tip properties via a wearable inflatable silicone layer, referred to as a soft bubble. We then lastly update
the properties of the surface being tapped.
We conducted four experiments to investigate the effect of updating fingertip or surface properties on
hardness rendering. In order to render hardness, we used a vibration actuator attached to the back of the
finger, which generated a vibration signal representing the material’s hardness. Three signal rendering
techniques were used. The energy of the signals was matched between different techniques. To evaluate
the effectiveness of our approach, we conducted a hardness discrimination and matching task experiment.
6.1 RelatedWork
When we explore or grasp an object we depend on interaction and object properties such as the interfa-
cial frictional forces, texture, hardness, or softness of an object. During any physical interaction between
fingers and the environment, vibration is transferred to the skin. We distinguish different properties of ma-
terials, such as hardness and roughness, by interpreting these transferred vibrations that are transformed
via the skin’s mechanoreceptors [111, 99]. Mechanoreceptors are somatosensory receptors that transmit
extracellular stimuli to intracellular signal transduction via mechanically gated ion channels. Touch, pres-
sure, stretching, sound waves, and motion are common examples of external stimuli [44].
In order to simulate transient contact interactions between the skin and a solid object (e.g., a surface)
researchers frequently invoke a theory based on the frequency specificity of mechanoreceptor populations,
where a mechanoreceptor tuning to∼ 200 Hz has informed device design [11]. However, there is evidence
91
to suggest that this tuning is primarily relevant for threshold stimulation [18]. An alternative hypothesis
is that the mechanoreceptors respond to the total energy of the incoming signal, corresponding to the
RMS of the power spectral density of the mechanical input [3, 7]. Based on this hypothesis it is possible
to capture the spectral power from surface interactions, to determine what information the brain uses to
recover material properties and recreate realistic haptic interactions.
Here we focus on perceived hardness, a property of a material that is directly accessible via decoding
vibration information that arises during a transient contact event [41]. In our experiments, we consider
tapping for the perception of hardness, as our key dynamic interaction between the finger and the envi-
ronment, given its dominant mode of energy transmission is in vibration (unlike pressing, sliding contacts,
etc.) [67, 116, 118].
6.2 Methods
In this section, we investigate the effect of updating fingertip or surface properties on hardness rendering
using vibration signals. Specifically, we explore whether the energy spectrum of the vibration signal pro-
duced by tapping on a material can be used as a cue for its hardness perception. To achieve this, we conduct
four experiments using a vibration actuator attached to the back of the finger to generate vibration sig-
nals representing materials with different hardnesses. We use three different signal rendering techniques
and match the energy of the signals between them. Finally, we evaluate the effectiveness of our approach
through hardness discrimination and matching task experiments.
6.2.1 MaterialSelectionandRecording
We conducted a pilot study to determine how people perceive the differences between materials and to see
if they can distinguish the differences between the hardness of materials if we mask their tactile cues. We
92
blindfolded a participant and asked them to wear a thin glove to mask the temperature cues, and noise-
canceling headphones playing white noise to mask audio cues. The participant then tapped on 20 different
materials that we interact with in our daily lives, including candles, glass, acrylic, carpet, silicone, soft
foam type 1, soft foam type 2, hard foam, plywood, Wood type 2, metal, rock, cement, ceramic, eraser,
insole, fabric, felt, hard plastic, cardboard. We randomly put two sets in front of them each time and found
they could not distinguish between hard materials, including rock, wood, glass, metal, etc. The rubber
and silicone felt similar, but they had different bounciness responses. Then we explored the soft materials
and found it easy to distinguish the differences between soft materials. Based on these perceptual pilot
experiments, we chose the materials: metal (Aluminum), wood (plywood), rubber (Poron performance
cushioning SRVSMA-15500-84), gel silicone, and foam (polyester batting, Simple Foam Cushion by Loops
& Threads) (these materials are shown in Figure 6.8, where it was used for an experiment.)
6.2.2 RecordingTappingVibrations:
We examined how the vibration from surfaces of different hardness and material properties dissipates
along the finger using wearable accelerometers by measuring the vibrations captured at the nail, on the
distal phalanx (below the nail), and on the middle phalanx while the finger tapped on different materials
(Figure 6.1). We used a NIDAQ (NI-9205 module) to log data with a 2000 Hz sampling frequency from
a three-axis Endevco 35A accelerometer (Endevco, NY, USA) and a signal conditioner (Endevco Model
133 PS/ISOTRON). We attached this accelerometer to the index fingernail using the accelerometer adhe-
sive wax (Petro Wax, Model 080A109-PCB PiezoElectronics). As can be seen in Figure 6.1 we observed
frequency-dependent propagation where high-frequency information is captured closest to the contact lo-
cation and dissipates rapidly. Therefore, to capture the most accurate local information, we rendered our
data-driven approach based on the vibrations we measured on the fingernail.
93
To encode a wide range of material interactions for materials we experience in our daily lives, we
captured the vibrations resulting from a bare finger tapping on five representative materials (Figure 6.2)).
These materials were chosen as indicated above, including metal, wood, rubber, silicone, and foam, as
well as the vibration that was recorded from tapping with a bubble attached to the fingertip on a block of
wood. These materials are typical in everyday interactions and the five samples provide a large enough
comparison set to be representative, while not adding noise to the estimates due to cognitive limitations.
A metronome controlled the tapping speeds with 90 BPM. We selected one representative tap with the
average peak vibration of the taps for that material, corresponding to a tap recorded at the average tapping
speed. The tap’s peak was selected and cropped from 5 ms before the peak to 15 ms after (time domain
signals in Figure 6.2-B).
The captured vibrations, represented in the frequency domain (Figure 6.2-B), highlight the importance
of high-frequency content encoding (more than 500 Hz) for identifying the differences between the mate-
rials during our simple interactions. The relevance of these high-frequency components already belies a
frequency-based encoding as this crucial information is predicted to lie outside the sensitivity bandwidth
0 200 400 600 800 1000
Frequency (Hz)
0
0.05
0.1
0.15
0.2
0.25
0.3
Power (m
2
/s
4
)
Nail
Distal Phalanx
Middle Phalanx
Figure 6.1: Recording vibration signals on different locations of a finger while tapping on metal.
94
of the mechanoreceptors [11]. The spectral information could provide a feature that is unique to each ma-
terial interaction but not limited to the mechanoreceptor frequency tuning identified in the literature [11].
We compensated for changes in the finger pad elasticity since this property updates based on different
activities during our life and tested our hypothesis that the spectral energy produced from a bare finger
tapping a surface communicates the unique material properties of the object in contact with the skin.
A soft bubble actuator(two-layer heat-sealed TPU in a circular shape) matching the circumference of
the finger pad was worn during the finger-compensated surface recordings. The bubble was inflated to
generate a change in fingertip elasticity. Our recordings show a minimal difference between 5 and 10psi of
bubble inflation. Therefore, for energy efficiency, we used 5psi for both recording and rendering. In this
experiment, we compensate for the frequency content in Figure 6.2-B that is missing between the vibration
from this inflated bubble and any of the surfaces.
When we update the finger elasticity properties and attach a bubble to the fingertip, we need to com-
pensate for the vibration between the bubble and the surface we interact with in order to alter the vibration
to feel like a different material. Therefore, we subtract the vibration signals in the frequency domain (for
any of these five materials) from the vibration of the bubble. Figure 6.3 shows these new signals.
0 5 10 15 20
Time (ms)
-4
-2
0
2
4
6
8
10
Acceleration (m/s
2
)
Metal
Wood
Rubber
Silicone
Foam
Bubble
200 400 600 800 1000
Frequency (Hz)
0
0.2
0.4
0.6
0.8
1
Normalized power (m
2
/s
4
)
Metal
Wood
Rubber
Silicone
Foam
Bubble
A1 recorded signals A1 frequency domain
A1
A2
A1
bubble compensation surface compensation
Time (ms)
-5
0
5
10
500 1000
Frequency (Hz)
0
0.5
1
1.5
Time (ms)
-5
0
5
10
500 1000
Frequency (Hz)
0
0.5
1
1.5
0 10 20
Time (ms)
-5
0
5
10
Metal
500 1000
Frequency (Hz)
0
0.5
1
1.5
0 10 20
Time (ms)
-5
0
5
10
Foam
500 1000
Frequency (Hz)
0
0.5
1
1.5
Bare finger
Bubble
New signal
Material tap
Foam tap
New signal
Metal Silicon
0 10 20 0 10 20
B)
signal recording setup
A)
C)
D)
Material samples
Figure 6.2: Recorded vibration signals in the time domain and frequency domain while tapping on different
materials. A1 shows the location of the accelerometer during recording.
95
When we update the surface properties, we must compensate for the vibration between the fingertip
and the soft surface we interact with. Therefore in the frequency domain, we subtract this vibration from
the vibration of the desired material that we want to render. Figure 6.4 shows these new signals. These
new signals are converted from the frequency domain to the time domain, then upsampled and stored as
wave files; these wave files are used as our reference signals for the data-driven method. The signals then
generated using decaying sinusoid, or Ricker wavelet techniques using the tuning methods that is shown
below. All of these signals were computed in real-time.
6.2.3 SignalRenderingTechniques
In this section the three approaches used for the hardness discrimination and matching task experiment
are explained in detail - the data-driven approach, the decaying sinusoid approach, and the Ricker wavelet
approach. The data-driven approach was chosen as the reference method. Since this approach captures the
actual vibrations that are generated during physical interactions, it provides a more accurate representation
of the material properties and how they are perceived by the user. This was compared with the commonly
used decaying sinusoid approach in which we used a matching fundamental frequency and spectral energy,
and a Ricker wavelet approach where we matched the spectral energy, but the frequency content differs,
0 10 20
Time (ms)
-5
0
5
10
Metal
500 1000
Frequency (Hz)
0
0.5
1
1.5
0 10 20
Time (ms)
-5
0
5
10
Wood
500 1000
Frequency (Hz)
0
0.5
1
1.5
0 10 20
Time (ms)
-5
0
5
10
Rubber
500 1000
Frequency (Hz)
0
0.5
1
1.5
0 10 20
Time (ms)
-5
0
5
10
Silicone
500 1000
Frequency (Hz)
0
0.5
1
1.5
0 10 20
Time (ms)
-5
0
5
10
Foam
500 1000
Frequency (Hz)
0
0.5
1
1.5
Bare finger
Bubble
New signal
Figure 6.3: Data-driven rendering process for soft bubble actuator.
96
especially for the soft materials. These methods are explained below. For each condition, updating fingertip
or surface properties, we ran a hardness discrimination and matching task experiment.
6.2.3.1 Data-drivenmodel
In a data-driven model, we capture the output response of a system, like acceleration in our case, given the
user inputs, like the finger position [71]. To improve the realism of virtual haptic interactions, researchers
have used this method and recorded data from real interactions and represented the same sensation virtu-
ally, like texture, tapping [59], or cutting [85]. This method also has been used in haptic augmented reality,
in which virtual feedback is provided to modulate the haptic properties of real objects [37, 48]. However,
recordings require a large amount of storage space and are time costly.
0 10 20
Time (ms)
-5
0
5
10
Metal
500 1000
Frequency (Hz)
0
0.5
1
1.5
0 10 20
Time (ms)
-5
0
5
10
Wood
500 1000
Frequency (Hz)
0
0.5
1
1.5
0 10 20
Time (ms)
-5
0
5
10
Rubber
500 1000
Frequency (Hz)
0
0.5
1
1.5
0 10 20
Time (ms)
-5
0
5
10
Silicone
500 1000
Frequency (Hz)
0
0.5
1
1.5
Tap on Materials
Tap on Foam
New Signal
Figure 6.4: Data-driven rendering process for foam.
97
6.2.3.2 DecayingSinusoidalmodel
Impact transients for each material have a characteristic decay rate, in which the amplitude increases
linearly with impact velocity, and have a constant slope [86]. However, many resonant modes and inter-
mittent contacts produce a more complex response. These very short impulses generate high-frequency
accelerations and allow the user to deduce the object’s material properties [59].
Equation 6.1 shows this model, in whichA is the amplitude slope,B is the decay rate,ω is the frequency,
andt is the time of the sinusoid.
Q(t) =A
Bt
sin(ωt) (6.1)
Researchers have long used decaying sinusoids to simulate impact transients [85]. Kuchenbecker et al.
[59] used a data-driven model, an input-output approach, and Hooke’s Law model to compare the realism
of virtual surfaces. According to the findings, playing either the velocity-scaled recorded acceleration
transients (data-driven) or the manually tuned decaying sinusoids created a virtual surface that most nearly
matched the perceived hardness of the real surface. Therefore, in addition to a data-driven model, we also
use decaying sinusoids as one of our generic and lower-cost signal rendering techniques.
6.2.3.3 Rickerwaveletmodel
Another signal rendering technique that we used was Ricker wavelet [93], which is the second derivative
of a Gaussian function. This wavelet assumes that acceleration rather than displacement better represents
the energy content, that there is a transient contact between two solids, and that the displacements go
from zero to a maximum with a Gaussian-like behavior. As such, a two-parameter function can define the
impact between a finger and a solid, and communicate the material properties of the impact by updating
the width orΣ of our Gaussian. The Ricker wavelet is shown in Equation 6.2:
98
R(t) = (1− 1
2
ω
2
p
t
2
)exp(
− 1
4
ω
2
p
t
2
) (6.2)
wheret is time andω
p
is the most energetic frequency or the peak frequency with the greatest spectral
content [112].
6.2.4 SignalTuning:
6.2.5 Fingerelasticitycompensation:
Table 6.1 shows the parameters that were tuned for wavelet and decaying sinusoid methods and RMS
values recorded from the data-driven method. The resulting signals for the three methods are shown in
Figure 6.5.
Table 6.1: Tuning parameters for wavelet and decaying sinusoid rendering techniques while compensating
for finger elasticity.
Metal Wood Rubber Silicone Foam
RMS 11.42 7.09 2.29 0.85 1.50
Wavelet
Gain 1 0.7364 0.361 0.12 −0.162
Sigma 0.0276 0.0315 0.042 0.144 0.138
DecayingSinusoid
Amplitude 0.482 0.382 0.22 0.16 0.1855
Frequency 250 200 150 50 100
99
-1
0
1
Data Driven
Metal
-1
0
1
Wood
-1
0
1
Rubber
-1
0
1
Silicone
-1
0
1
Foam
-1
0
1
Normalized Acceleration
Ricker Wavelet
-1
0
1
-1
0
1
-1
0
1
-1
0
0 10 20
Time (ms)
-1
0
1
Decaying Sinusoid
0 10 20
Time (ms)
-1
0
1
0 10 20
Time (ms)
-1
0
1
0 10 20
Time (ms)
-1
0
1
0 10 20
Time (ms)
-1
0
1
Figure 6.5: Generated signals for each method while compensating for finger elasticity.
6.2.5.1 Surfacecompensation:
Table 6.2 shows the parameters that were tuned for wavelet and decaying sinusoid methods and RMS
values recorded from the data-driven method. The resulting signals for the three methods are shown in
Figure 6.6.
6.2.6 Processing:
A Bela board was used to generate these signals. This board uses Xenomai, a Linux-based RTOS with hard
real-time computing support. Generating Audio output using this board guarantees a latency of less than
one millisecond, which means the latency between the moment that the finger is detected on the surface
and the new vibration signal is triggered would be less than one millisecond. Therefore, the user would
not perceive a delay.
The finger is detected using a compact photoelectric sensor (Panasonic, Model CX-400, Osaka, Japan)
located in front of the finger. It detects the finger, or the finger with the bubble attached to it, as soon as
100
the finger contacts the surface. After contact, the Bela audio output generates the vibration signal that
was stored as wave signals. These signals are then amplified through an audio amplifier. The amplified
signals are generated through a vibration actuator (Lofelt L5, Berlin, Germany) that is taped to the index
fingernail. This actuator was selected because it is broadband and can provide a frequency from 30 Hz to
1000 Hz.
6.3 ExperimentalMethods
6.3.1 Participants
We recruited 40 participants (age: 22-52, 13 female-27 male, one left-handed. Their experience with haptics
was: 2 no experience, 16 limited, 13 moderate, and nine extensive), and 10 participants took part in each
experiment.
Upon arrival, the participant sat on a chair and was informed how to tap on two blocks under their
right and left index fingers. The blocks were behind a curtain that was used to blindfold the participants.
Table 6.2: Tuning parameters for wavelet and decaying sinusoid rendering techniques while compensating
for surface properties.
Metal Wood Rubber Silicone
RMS 6.95 4.63 2.32 0.59
Wavelet
Gain 1 0.74 0.36 0.12
Sigma 0.02 0.03 0.07 0.16
DecayingSinusoid
Amplitude 0.38 0.33 0.22 0.11
Frequency 200 175 125 125
101
After giving them instructions on the process, they wore noise-canceling headphones playing white noise.
The blocks and attachments to their index finger were different for each experiment.
6.3.2 UserStudies
6.3.2.1 UpdatetheFingertipelasticityProperties: ExperimentOne
Experiment one is a discrimination task. In this experiment, we change the fingertip properties by tap-
ing a soft bubble actuator to the right and left index fingertips. The bubble was inflated with a manual
compressor to 5 psi. Two vibration actuators are taped to the index fingernails.
The participants tapped on two identical wood blocks under the right and left index fingers. Two
compact photoelectric sensors were located in the center between the hands and detected the bubble as
soon as they made contact with the surface (a paper shield was placed in front of the bubble for better
detection by the sensor). When the contact for each finger was detected, a vibration cue is generated on
0 10 20
-1
0
1
Data Driven
Metal
0 10 20
-1
0
1
Wood
0 10 20
-1
0
1
Rubber
0 10 20
-1
0
1
Silicone
-1
0
1
Normalized Acceleration
Ricker Wavelet
-1
0
1
-1
0
-1
0
1
0 10 20
Time (ms)
-1
0
1
Decaying Sinusoid
0 10 20
Time (ms)
-1
0
1
0 10 20
Time (ms)
-1
0
1
0 10 20
Time (ms)
-1
0
1
Figure 6.6: Generated signals for each method while compensating for surface properties.
102
Figure 6.7: Discrimination task experiment while compensating for finger elasticity.
the related actuator. In each trial, two different signals were played, one to each finger. Participants were
asked a two-alternative forced choice question, "Which side feels harder/stiffer?", to compare the two sides.
The participant could tap on each block and go back and forth as often as they wanted.
In total, there were three different methods, and for each method, five different vibration signals were
generated (including three with no vibration). These signals were designed for rendering four different
materials when interacting with foam. Note that a no-vibration signal was used for discrimination between
no feedback and any additional feedback.
Each of these five signals was compared with the other four signals, creating 20 permutations for
each method and 60 permutations in total. Each permutation was repeated twice for each hand in this
discrimination task. All trials were randomized. This experiment lasted 30-45 minutes (Figure 6.7 ).
6.3.2.2 UpdatetheFingertipProperties: ExperimentTwo
Experiment two is a matching task experiment for when we update the fingertip properties, in which
the participant tries to match a rendered signal to a real material. A soft bubble actuator is taped to the
non-dominant index fingertip. The bubble was inflated with a manual compressor to 5 psi. Two vibration
actuators are taped to the index fingernails.
In this experiment a compact photoelectric sensor is located in the center in front of the non-dominant
hand. The user was instructed to tap on a foam block with their non-dominant index finger. As soon as
103
the photoelectric sensor detects contact between the non-dominant index finger and the foam surface, a
vibration is generated with the actuator on the back of the finger. In each trial, a different vibration cue
was played. The user was asked to tap with their dominant hand index finger on five real materials to find
the one closest to the rendered signal. The actuator taped to the dominant hand was used only to keep the
weight and conditions consistent between the right and left hands because that could affect the perception.
In total, there were three different methods, and five different vibration signals were generated for
each method. These signals were designed for rendering five different materials when interacting with
foam.Each of these 15 signals (5 signals x 3 methods) was repeated five times, creating a total of 75 trials.
All trials were randomized. Note that we ran one training trial when participants began so that they
learned the process. The experiment was designed to be completed within 1 hour, and all participants took
between 30-60 minutes (Figure 6.8).
6.3.2.3 UpdatetheSurfaceProperties: ExperimentOne
Experiment one is a discrimination task in which the participants tap on two similar blocks of foam under
the right and left index fingers. Two vibration actuators are taped to the index fingernails. Two compact
photoelectric sensors are located in the center between the hands and detect fingers as soon as they contact
the surface. When the contact for each finger is detected, a vibration cue is played on the related actuator.
In each trial, two different signals were played, and participants were asked to compare those two sides.
Figure 6.8: Matching task experiment while compensating for finger properties.
104
The question that was asked was, "Which side feels harder/stiffer?". The participant could tap a few seconds
on each block and go back and forth as often as they wanted. Their response was a two-alternative forced
choice, either right or left.
In total, there were three different methods, and for each method, five different vibration signals were
generated (including three with no vibration). These signals were designed for rendering four different
materials when interacting with foam. Note that a no-vibration signal was used for discrimination between
no feedback and any additional feedback.
Each of these five signals was compared with the other four signals, creating 20 permutations for
each method and 60 permutations in total. Each permutation was repeated twice for each hand in this
discrimination task. All of the trials were randomized. Note that we conducted one training trial at the
beginning of the experiment so that the participant learned the process. After that, we recorded all the
responses. This experiment lasted 30-45 minutes.
6.3.2.4 UpdatetheSurfaceProperties: ExperimentTwo
Experiment two is a matching task experiment in which the participant tries to match a rendered signal
to a real material. Two vibration actuators are taped to the index fingernails, and a compact photoelectric
sensor is located in the center in front of the non-dominant hand. The user was instructed to tap on a
foam block with their non-dominant index finger. As soon as the photoelectric sensor detects the contact
between the non-dominant hand index finger and the foam surface, a vibration is generated with the
actuator on the back of the finger. In each trial, a different vibration cue was played, and the user was
asked to tap with the dominant hand on five real materials to determine which one was closest to the
rendered signal. The actuator taped to the dominant hand was used only to keep the weight and conditions
consistent between the right and left hands because that could affect the perception.
105
There were three different methods, and four different vibration signals were generated for each
method. These signals were designed for rendering four different materials when interacting with foam
(why foam was not selected). Each of these 12 signals (4 signals x 3 methods) was repeated five times,
creating a total of 60 trials. All the trials were randomized. Note that we conducted one training trial at
the beginning of the experiment so that the participant learned the process. After that, we recorded all the
responses. This experiment lasted 30 minutes to one hour.
6.4 Results
After generating signals using the three rendering techniques we explored material property commu-
nication through discrimination and matching tasks. With the bubble attached to their fingerpad and a
vibration actuator attached to the dorsal surface of the finger, participants performed both a discrimination
task (to determine whether a vibration cue could make the bubble become imperceptible from the surface
interaction) and a matching task. The matching task requires that participants can not only discriminate
surface hardness changes but also use the available spectral information to identify the material itself.
In the hardness discrimination task, we asked participants to tap on two identical blocks of wood,
one positioned under their right finger and the other under their left finger, with the inflated soft bubble
actuator attached to both index fingers (Figure 6.9-A). Participants indicated which of the two surfaces felt
harder, while we compensated for the high-frequency content that was missing due to the soft bubble by
using one of our three rendering techniques (data-driven, decaying sinusoidal, and Ricker wavelet) The
hardness discrimination result for the finger elasticity compensation is shown in Figure 6.10.
To understand the role of equal energy in simulating material properties and how this external energy
on the skin could be perceived, we also conducted a matching task to determine if the percept resulting
from these rendered signals is perceptually equivalent to the percept arising from the interaction with the
real materials. We rendered the vibration signals with the vibration actuator when participants tapped
106
on a wood block while wearing the bubble on the right index finger. They were asked to tap on five real
materials with their left index finger and choose the best match to the rendered signal. Participants’ ability
to correctly match or classify the material as itself from our rendered cues is shown in Figure 6.11. For
a given material, all rendered signals (data-driven, decaying sine, and Ricker wavelet) were matched in
terms of spectral energy.
Fingerelasticitycompensation: A two-way ANOVA with the rendered material and rendering tech-
nique as factors show that the rendered material significantly affected the perceived material across the
generated signals for all materials (F(4,749) = 177.23, p < 0.001, η 2
= 0.76), but the rendering tech-
nique did not significantly affect the perceived material ( F(2,749) = 2.34,p = 0.158,η 2
= 0.005). The
interaction between signals and rendering techniques was also statistically significant ( F(8,749) = 3.55,
p< 0.001,η 2
= 0.009).A Tukey’s pairwise comparison test shows that participants were able to discrimi-
nate the material properties according to the rendered signal, meaning that five unique categories emerged.
These results highlight the importance of our spectral energy feature for material discrimination behavior.
To identify how generalizable and stable the spectral feature is for communicating material property
information, we compensate for changes in the physical properties of the explored surface. We selected the
Figure 6.9: Experiment setup while compensating for A) finger elasticity, and B) surface properties.
107
foam sample, the softest natural material we tested, as the baseline for compensation. In this experiment,
we compensate for the missing frequency content, as shown in Figure 6.2-B, for the vibration between the
foam and any harder surfaces. We rendered new compensation signals using a data-driven model, which
is shown in Figure 6.2-D.
Participants performed both a discrimination task (to determine if a vibration cue could make the
foam feel harder), and a matching task, as they did in the finger elasticity compensation experiments. In
the hardness discrimination task, participants tapped on two identical foam samples under their right and
left index fingers. A vibration actuator was taped to the nail of each index finger. The actuator rendered
a signal that compensated for the missing vibration cues of different materials, and participants indicated
which side felt harder (Figure 6.9-B). Figure 6.12 shows this experiment’s result, which indicates that this
additional vibration is effective at making the foam feel harder.
The same tapping-based matching task was conducted for the surface compensation condition to de-
termine if the rendered signals can compensate for the foam contact event and generate the percept of a
target material property from our five natural materials. In this condition, participants tapped with their
bare, right index finger on a foam block and compared the sensation to that arising from tapping on five
real materials with their left index finger. They were asked to choose the best match to the signal rendered
to their right index finger. Participants’ ability to use the rendered cues to correctly match or classify the
No signal
Foam
Silicone
Rubber
Wood
Metal
Generated Signal B
No signal
Foam
Silicone
Rubber
Wood
Metal
Generated Signal A
Data-Driven
0
20
0 0
0
0
0
0
0
0
0
0
0
10
100
100
100
100
100
75
100
100
80
100
100
100
75
100
100 90 0
20
40
60
80
100

No signal
Foam
Silicone
Rubber
Wood
Metal
Generated Signal B
No signal
Foam
Silicone
Rubber
Wood
Metal
Generated Signal A
Decaying Sinusoid
0
20
0 0
15
0
0
0
0
0
0
0
0
0
15
100
100
100
100
100
85
100
100
80
100
100
100
100
100 85 0
20
40
60
80
100

No signal
Foam
Silicone
Rubber
Wood
Metal
Generated Signal B
No signal
Foam
Silicone
Rubber
Wood
Metal
Generated Signal A
Ricker Wavelet
10
25
0 0
0
5
0
0
0
5
0
0
0
0
30
90
100
100
100
100
100
100
100
75
95
100
100
95
100 70 0
20
40
60
80
100

My Text
Figure 6.10: Discrimination task: tapping with the bubble actuator attached to the right and left fingers:
Percent of the times signal A selected as harder than B’.)
108
material as itself is shown in Figure 6.13. For a given material, all rendering signals, data-driven, decaying
sine, and Ricker wavelets were matched in terms of spectral energy.
Surface compensation: A two-way ANOVA with signals and rendering techniques as factors also
show that the generated signals created a statistically significant difference in the perceived materials
(F(3,599) = 642.01,p < 0.001,η 2
= 0.55), but that the rendering techniques did not have a significant
effect on the perceived material ( F(2,599) = 2.18,p = 0.194,η 2
= 0.001). The interaction between sig-
nals and rendering techniques was not statistically significant ( F(6,599) = 0.38,p = 0.89,η 2
= 0.0017).
A Tukey’s pairwise comparison test shows that the classification difference aligns with the rendered ma-
terial category.
Foam
Silicone
Rubber
Wood
Metal
Foam Signal
Silicone Signal
Rubber Signal
Wood Signal
Metal Signal
Generated Signal
Data-Driven
0.3636
0.6667
0.8788
0.6364
0.5758 0.6667
1
0.5152
0.3939
1
0.1515
0
0
0
0
0.2727
0.2121
0.1212
0
0
0
0.1212
0
0
0
0
0.2
0.4
0.6
0.8
1
Foam
Silicone
Rubber
Wood
Metal
Foam Signal
Silicone Signal
Rubber Signal
Wood Signal
Metal Signal
Decaying Sine
0.4333
0.8
0.8333
0.8
0.4333
0.4
0.8333 0.3667
1
0.6667
0.5667
1
0.03333
0
0
0
0
0.06667
0.1
0
0
0
0
0
0
0
0.2
0.4
0.6
0.8
1
Foam
Silicone
Rubber
Wood
Metal
Foam Signal
Silicone Signal
Rubber Signal
Wood Signal
Metal Signal
Wavelet
1
0.8621
0.6207
0.7931
0.4828 0.931
0.3448 0.7586
0.9655
0.5517
0.6207
0.03448
0
0
0.06897
0
0.1034
0.06897
0.1379
0
0
0.2759
0
0
0
0
0.2
0.4
0.6
0.8
1
Metal Signal
Wood Signal
Rubber Signal
Silicone Signal
Foam Signal
Generated Signal
Foam
Silicone
Rubber
Wood
Metal
Perceived material
Metal Signal
Wood Signal
Rubber Signal
Silicone Signal
Foam Signal
Foam
Silicone
Rubber
Wood
Metal
Perceived material
Metal Signal
Wood Signal
Rubber Signal
Silicone Signal
Foam Signal
Entropy
Foam
Silicone
Rubber
Wood
Metal
Perceived material
Figure 6.11: Matching task while compensating for fingertip elasticity.
109
6.5 Discussion
For both surface and finger compensation interactions, we observed a near-perfect ability to discriminate
the changes in surface hardness and identify material based on the energy-matched signals. The fact that
all three rendering techniques produce the same behavior, despite differences in the frequency content and
signal duration for a given material, is consistent with our energy hypothesis.
It should be noted that previous work focusing on understanding the role of energy was unable to
determine RMS as the key feature for threshold estimation [7]. However, prior results suggesting fre-
quency content to be the key mechanical information were confounded by issues of actuator grounding
and attachment to the finger [18]. Here we provide evidence for a material property feature based on the
mechanical energy transferred to the skin. Our results shed new light on previous findings focusing on
softness perception [111]and highlight the behavioral relevance of the feature similarly identified by Shao
et al., for material classification based on mechanical signatures [99].
In manipulating the finger and surface characteristics, our goal was to identify the compensation signal
needed to deliver a stable material percept for a range of natural materials. While a primary cue known
to correlate with surface stiffness or softness is the rate of change of the fingertip deformation during
sustained contact normal to an object’s surface, in tapping interactions the transient impact event contains
No signal
Silicone
Rubber
Wood
Metal
Generated Signal B
No signal
Silicone
Rubber
Wood
Metal
Generated Signal A
Data Driven
0 0
5
0
0
10
0
0
0
25
100
100
100
100
95
100
100
90
100 75
0
20
40
60
80
100

No signal
Silicone
Rubber
Wood
Metal
Generated Signal B
No signal
Silicone
Rubber
Wood
Metal
Generated Signal A
Decaying Sine
0 0
5
0
0
15
0
0
0
20
100
100
100
100
95
100
100
85
100 80
0
20
40
60
80
100

No signal
Silicone
Rubber
Wood
Metal
Generated Signal B
No signal
Silicone
Rubber
Wood
Metal
Generated Signal A
Wavelet
0 0
0
0
0
5
0
0
10
30
100
100
100
100
100
100
100
95
90 70
0
20
40
60
80
100

Figure 6.12: Tapping with the bare finger on the foam: Percent of the times signal A selected as harder
than B’)
110
the spectral energy information necessary to perceptually reconstruct surface hardness and material [41].
While our interactions optimize for the primary local energy, mechanical signals in the tactile frequency
range are produced by dynamic touch and propagate throughout the entire hand. These mechanical signals
encode into neural representations. Thus, dynamic tactile inputs can stimulate extensive tactile afferent
populations [91, 73]. These touch-elicited signals help with perceptual discrimination and can be utilized
to infer actions, contact locations, and the characteristics of touched objects [99, 78, 70, 113, 74]. We have
captured the mechanical interaction feature that produces a rich, realistic material sensation for complex
natural material properties.
Critical to successfully rendering realistic rich sensations is the inclusion of spectral content well be-
yond the traditional frequency tuning range of the rapidly adapting mechanoreceptors (Pacinian Corpus-
cles, PCs). As we target these same PCs, responsible for encoding long-range vibration information and
material interaction properties. As is evident in both the finger and surface compensation conditions, the
Foam
Silicone
Rubber
Wood
Metal
Silicone Signal
Rubber Signal
Wood Signal
Metal Signal
Generated Signal
Data-Driven
0.8519 0.6667
0.7778 0.6296
0.963
0.5185 1
0.1111
0
0
0.2963
0.1481
0.3333
0.2593
0.1852
0
0.2963
0
0.03704
0.3333
0
0.2
0.4
0.6
0.8
1
Foam
Silicone
Rubber
Wood
Metal
Silicone Signal
Rubber Signal
Wood Signal
Metal Signal
Decaying Sine
1 0.7037
0.8148 0.5185
0.963
0.9259
0.4074
0.6296
0.1852
0.03704
0
0.2593
0.1111
0.1481
0.1852
0.1852
0
0.1481
0
0.1852
0
0.2
0.4
0.6
0.8
1
Foam
Silicone
Rubber
Wood
Metal
Silicone Signal
Rubber Signal
Wood Signal
Metal Signal
Wavelet
1 0.8519
0.8148
0.3704
0.5926
0.6667
0.6667
0.5185
0.7778
0.07407
0.03704
0 0.2222
0
0.2593
0.1852
0
0.3333
0
0.03704
0
0.2
0.4
0.6
0.8
1
Foam
Silicone
Rubber
Wood
Metal
Perceived material
Metal Signal
Wood Signal
Rubber Signal
Silicone Signal
Generated Signal
Foam
Silicone
Rubber
Wood
Metal
Perceived material
Metal Signal
Wood Signal
Rubber Signal
Silicone Signal
Foam
Silicone
Rubber
Wood
Metal
Perceived material
Metal Signal
Wood Signal
Rubber Signal
Silicone Signal

Entropy
Figure 6.13: Matching task while compensating for surface properties.
111
confusion that arose pertained mainly to limitations in veridically rendering the highest frequency com-
ponents representative of metal. This provides further confirmation for spectral energy over frequency
content.
Furthermore, this has important implications for the engineering of haptic displays, as it emphasizes
the need for broadband actuators that can deliver spectral content up to a bandwidth of 1 kHz to generate
a vibration signal that is matched with human perception, although measuring energy may be more chal-
lenging than measuring force or displacement, this information is useful for both engineering and neural
processing.
6.6 Summary
Our research aimed to investigate the mechanical information that the brain uses to construct a stable
representation of the physical world through the sense of touch. By manipulating the finger and surface
properties during a finger tap event against a physical object, we found that the spectral content of vibra-
tion feedback is crucial in communicating the material properties of the object. Our study provides valuable
insights into the mechanisms underlying tactile perception and could have significant implications for the
development of prosthetics and haptic devices that mimic natural touch sensations. The results suggest
that transformed/filtered vibration information plays a vital role in tactile material perception, highlight-
ing the importance of considering this information when designing haptic systems for use in real-world
applications. This study contributes to our understanding of the complex mechanisms involved in touch
perception and provides a foundation for future research in this area.
112
Chapter7
Conclusion
Stiffness and hardness are fundamental material properties that describe how easily they deform and resist
indentation, respectively. To accurately simulate these properties in haptic technology, it is essential to
understand how humans perceive them and design devices that can replicate the associated tactile or
kinesthetic sensations. This understanding can enhance the realism and immersion of haptic simulations.
This dissertation focused on the use of haptic devices for replicating the human perception of stiffness
and hardness. Our findings emphasized the significance of including hardness rendering in haptic simu-
lations, alongside stiffness, to achieve more realistic outcomes, which can enhance the accuracy of haptic
rendering and create more engaging and immersive virtual experiences. Additionally, our research demon-
strated that the spectral content of vibration feedback is critical in communicating the material properties
of an object during an interaction, highlighting the importance of including this information in the design
of haptic devices for real-world applications.
7.1 Contributions
7.1.1 EffectsofDentalGloveThicknessonTactilePerceptionThroughaTool
Chapter 2 evaluates how tactile perception is affected by external factors such as glove thickness and tool
contact location, which has important practical applications in various fields. The results indicated that
113
thicker gloves lead to higher perceptual thresholds for force, torque, and vibration. This could be due to
an increase in vibration dampening caused by the thicker gloves. However, the location of the tool contact
had a more significant impact on the perceptual threshold of vibration than glove thickness.
The study’s findings can be valuable for clinicians and dentists in selecting gloves that best suit their
needs. They could use this information to balance the tactile sensitivity required for performing delicate
dental procedures with the protection offered by thicker gloves. Furthermore, the study provides insight
into the complex interplay between tactile sensitivity, tool contact location, and glove thickness, which
could inform the design of future haptic devices and haptic rendering algorithms.
7.1.2 Combining Haptic Augmented Reality with a Stylus-Based Encountered-Type
DisplaytoModifyPerceivedHardness
In Chapter 3, we focused on the development and evaluation of a novel haptic rendering method, that can
simultaneously render stiffness and hardness to increase the realism of haptic simulations. Traditionally,
force-feedback haptic devices have only been able to render stiffness, but our results showed that incorpo-
rating hardness can greatly enhance the user’s perception of virtual objects. To achieve this, we modified
a haptic device and designed an encountered-type haptic display (ETHD) system that utilized a physical
plate with a specific hardness on the end-effector to render hardness and the motors of the device to ren-
der stiffness. The modifications of our device allow interacting with the device with an untethered stylus,
which improves our rendering of free space.
Our evaluation showed that our ETHD device had the highest realism ratings and was preferred by
users over previous haptic rendering methods that only rendered stiffness. These results have important
implications for the development of haptic devices that can accurately replicate the tactile feedback of
real-world objects, leading to more immersive and engaging virtual experiences.
114
7.1.3 EffectsofPhysicalHardnessonthePerceptionofRenderedStiffnessinan
Encountered-TypeHapticDisplay
The work in Chapter 4 focused on the exploration of the novel ETHD rendering approach that was pre-
sented in Chapter 3. To evaluate the effectiveness of this approach, two experiments were conducted to
examine how changing the hardness of the plates affects the perception of the interaction regarding the de-
vice’s stiffness. The results show that hardness plays a crucial role in masking the haptic device’s stiffness
and that more attention should be paid to rendering hardness to achieve more realistic haptic feedback.
Based on the findings, it is evident that stiffness masking has a significant impact on the perception
of hardness and stiffness. Specifically, we now understand how hardness can mask stiffness when a high
hardness is presented, which has important implications for rendering harder objects using haptic devices
that may be limited in their ability to apply high forces. By taking these factors into account, we can
improve the realism of haptic simulations for applications that require the rendering of hard objects.
7.1.4 RenderingHardnessandStiffnessUsingaDynamicEnd-Effectorinan
Encountered-TypeHapticDisplay
This chapter improves the ETHD system presented in Chapter 3 by creating a dynamic end-effector that
can render different hardnesses based on the location of virtual objects. The experiments conducted in
this chapter showed that participants were more sensitive to changes in hardness than stiffness when
presented with virtual materials with different properties. This suggests that hardness is a more salient
perceptual feature when it comes to object perception. Additionally, the results indicated the presence of
stiffness masking that was previously found in Chapter 4, which highlights the importance of considering
multiple sensory cues when designing studies or interventions aimed at improving object perception or
haptic discrimination abilities.
115
The chapter also highlighted the impact of the method of touch on the strength of statistical results in
perceptual experiments. The findings suggest that tapping with a stylus and tapping with a bare finger can
lead to significant differences in the way people distinguish between materials’ properties. It is important
to consider these factors when conducting studies or designing interventions that involve virtual materials.
Finally, the contribution of this chapter suggests that our brains use a combination of different types
of information, including vibrations and other sensory cues, to create a perceptual experience of the world
around us. This highlights the complexity of object perception and the importance of considering multiple
factors when designing studies or interventions aimed at improving haptic discrimination abilities.
7.1.5 EnergySpectrumastheKeyFeatureforHardnessPerceptualCue
This chapter has shed light on the importance of vibration feedback in communicating material proper-
ties during a finger tap event against a physical object. Our findings suggest that the spectral content
of vibration feedback is crucial in communicating the material properties of the object, and this trans-
formed/filtered vibration information plays a vital role in tactile material perception. These insights have
important implications for the development of prosthetic and haptic devices that mimic natural touch
sensations.
The study provides a foundation for future research in this area and has the potential to drive innova-
tion in the design of haptic systems. By understanding the mechanisms underlying tactile perception, we
can develop more effective haptic devices that can better replicate the sensations of touch in real-world
applications. This could be particularly useful in medical and rehabilitation settings where haptic devices
can be used to enhance the sensory experience for individuals with impaired tactile abilities. Additionally,
our study contributes to the broader field of sensory perception, which has important implications for
the development of intelligent systems that can interact with the physical world in a more sophisticated
way. By gaining a deeper understanding of the mechanisms underlying touch perception, we can develop
116
more advanced robotic systems that can sense and interact with their environment in a more intuitive and
natural way.
7.2 FutureDirections
There are some limitations to this dissertation that should be acknowledged. First, in our ETHD system
because of the limitation of our workspace, we were only able to render four blocks in chapter 5, and it
is possible that the results could differ with a larger set of blocks. It would be great to use a haptic device
or robotic arm with a larger workspace or even put the robot on a Cartesian rail as a low-cost way to
extend its workspace. Also, the study only looked at how participants sorted the blocks based on hardness
and stiffness and did not explore other factors that may influence how people perceive and interact with
materials, such as texture or temperature.
Future work could focus on further exploring the dynamic rendering of both hardness and stiffness
to create a more versatile system capable of rendering a wide range of materials with varying properties.
Additionally, our ETHD system could be modified to incorporate the rendering of different shapes or
textures on the end-effector or through the tool to enhance the realism of haptic feedback further.
Additionally, future studies could also examine how individual differences in cognitive abilities, such
as visual or audio perception, may influence the ability to differentiate between objects based on their
hardness and stiffness. Future research could explore the underlying mechanisms driving the differences
between tapping with a bare finger versus a stylus and how they might inform design principles for inter-
faces and products.
117
Bibliography
[1] Mustufa H Abidi, Ali Ahmad, Saber Darmoul, and Abdulrahman M Al-Ahmari. “Haptics assisted
virtual assembly”. In: Proc. IFAC Symposium on Information Control Problems in Manufacturing
48.3 (2015), pp. 100–105.
[2] Aesthesio User Manual. DanMic Global LLC. 2013.
[3] JW Andrews, MJ Adams, and TD Montenegro-Johnson. “A universal scaling law of mammalian
touch”. In: Science advances 6.41 (2020), eabb6912.
[4] Bruno Araujo, Ricardo Jota, Varun Perumal, Jia Xian Yao, Karan Singh, and Daniel Wigdor.
“Snake charmer: Physically enabling virtual objects”. In: Proc. Int. Conf. on Tangible, Embedded,
and Embodied Interaction. 2016.
[5] Jeffrey B. Nelson and Anil Mital. “An ergonomic evaluation of dexterity and tactility with
increase in examination/surgical glove thickness”. In: Ergonomics 38 (May 1995), pp. 723–33.
[6] Jonathan Bell, Stanley Bolanowski, and Mark H Holmes. “The structure & function of Pacinian
corpuscles: a review”. In: Prog. in Neurobiology 42-1 (1994).
[7] Sliman Bensmaıa, Mark Hollins, and Jeffrey Yau. “Vibrotactile intensity and frequency
information in the Pacinian system: A psychophysical model”. In: Perception & psychophysics 67
(2005), pp. 828–841.
[8] Wouter M Bergmann Tiest and Astrid ML Kappers. “Physical aspects of softness perception”. In:
Multisensory Softness: Perceived Compliance from Multiple Sources of Information (2014), pp. 3–15.
[9] Wouter M. Bergmann Tiest and Astrid M. L. Kappers. “Cues for Haptic Perception of
Compliance”. In: IEEE Transactions on Haptics 2.4 (2009), pp. 189–199.
[10] Wouter M. Bergmann Tiest and Astrid M. L. Kappers. “Kinaesthetic and Cutaneous Contributions
to the Perception of Compressibility”. In: Haptics: Perception, Devices and Scenarios. Ed. by
Manuel Ferre. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008, pp. 255–264.
[11] S Bolanowski, George Gescheider, R Verrillo, and C Checkosky. “Four channels mediate the
mechanical aspects of touch”. In: The Journal of the Acoustical Society of America 84 (Dec. 1988),
pp. 1680–94.
118
[12] AJ Brisben, SS Hsiao, and KO Johnson. “Detection of vibration transmitted through an object
grasped in the hand”. In: Journal of Neurophysiology 81.4 (1999), pp. 1548–1558.
[13] Grigore C Burdea. “Haptic feedback for virtual reality”. In: Virtual Reality and Prototyping
Workshop. Vol. 2. 1999, pp. 17–29.
[14] William D Callister, David G Rethwisch, et al. Materials science and engineering: an introduction.
Vol. 7. John wiley & sons New York, 2007.
[15] Ilaria Cesini, Jessica Dacleu Ndengue, Eric Chatelet, Jenny Faucheu, and Francesco Massi.
“Correlation between friction-induced vibrations and tactile perception during exploration tasks
of isotropic and periodic textures”. In: Tribology International 120 (2018), pp. 330–339.
[16] Inrak Choi, Yiwei Zhao, Eric J. Gonzalez, and Sean Follmer. “Augmenting Perceived Softness of
Haptic Proxy Objects Through Transient Vibration and Visuo-Haptic Illusion in Virtual Reality”.
In: IEEE Transactions on Visualization and Computer Graphics 27.12 (2021), pp. 4387–4400.
[17] S. Choi and K. J. Kuchenbecker. “Vibrotactile Display: Perception, Technology, and Applications”.
In: Proc. of the IEEE 101.9 (2013).
[18] Jason C Cohen, James C Makous, and Stanley J Bolanowski. “Under which conditions do the skin
and probe decouple during sinusoidal vibrations?” In: Experimental brain research 129.2 (1999),
pp. 211–217.
[19] J. E. Colgate, M. C. Stanley, and J. M. Brown. “Issues in the haptic display of tool use”. In: Proc.
IEEE/RSJ International Conference on Intelligent Robots and Systems, Human Robot Interaction and
Cooperative Robots. Vol. 3. 1995, 140–145 vol.3.
[20] Neil H Cox. “Palpation of the skin—an important issue”. In: Journal of the Royal Society of
Medicine 99.12 (2006), p. 598.
[21] James C Craig and Paul M Evans. “Tactile selective attention and temporal masking”. In: Percep. &
Psychophysics 57.4 (1995), pp. 511–518.
[22] Heather Culbertson and Katherine J. Kuchenbecker. “Importance of Matching Physical Friction,
Hardness, and Texture in Creating Realistic Haptic Virtual Surfaces”. In: IEEE Transactions on
Haptics 10.1 (2017), pp. 63–74.
[23] Heather Culbertson, Juan José López Delgado, and Katherine J. Kuchenbecker. “One hundred
data-driven haptic texture models and open-source methods for rendering on 3D objects”. In:
Proc. IEEE Haptics Symposium. 2014.
[24] Heather Culbertson, Samuel B Schorr, and Allison M Okamura. “Haptics: The present and future
of artificial touch sensation”. In: Annual Review of Control, Robotics, and Auton. Sys. 1 (2018),
pp. 385–409.
[25] N. D’Aurizio, T. Lisini Baldi, A. Villani, K. Minamizawa, Y. Tanaka, and Domenico Prattichizzo.
“Wearable Haptics for Object Compliance Discrimination Through Passive Touch”. In: IEEE World
Haptics Conf. 2021.
119
[26] Benoit Delhaye, Katie Long, and Sliman Bensmaia. “Neural Basis of Touch and Proprioception in
Primate Cortex”. In: Comprehensive Physiology (2018).
[27] Nicola Diolaiti, Günter Niemeyer, F. Barbagli, and J.K. Salisbury. “A Criterion for the Passivity of
Haptic Devices”. In: Proc. IEEE International Conference on Robotics and Automation. May 2005,
pp. 2452–2457.
[28] “The friction and lubrication of human stratum corneum.” In: Thin Films in Tribology. Ed. by
D. Dowson, C.M. Taylor, T.H.C. Childs, M. Godet, and G. Dalmaz. Vol. 25. Tribology Series.
Elsevier, 1993, pp. 663–672.
[29] N. El Mikatti, P. Dillon, and T. E. J. Healy. “Hygienic practices of consultant anaesthetists: a
survey in the North-West region of the UK”. In: Anaesthesia 54.1 (1999), pp. 13–18.
[30] Robert Friedman, Kim Hester, Barry Green, and Robert Lamotte. “Magnitude estimation of
softness”. In: Experimental Brain Research 191 (Sept. 2008), pp. 133–42.
[31] Sarah Frier. Facebook just unveiled a haptic glove for the metaverse. Vox. 2021.url:
https://www.vox.com/recode/2021/11/17/22787191/facebook-meta-haptic-glove-metaverse.
[32] Sivaranjani Gali and Archana Patil. “The technology of haptics in dental education”. In: Journal of
Dental & Orofacial Research 14.2 (2018).
[33] George Gescheider, Stanley Bolanowski, Jennifer Pope, and Ronald Verrillo. “A four-channel
analysis of the tactile sensitivity of the fingertip: Frequency selectivity, spatial summation, and
temporal summation”. In: Somatosensory & motor research 19 (Feb. 2002), pp. 114–24.
[34] Guidelines on Infection Control in Anaesthesia. Australian and New Zealand College of
Anaesthetists (ANZCA). 2005.
[35] Gabjong Han, Seokhee Jeon, and Seungmoon Choi. “Improving Perceived Hardness of Haptic
Rendering via Stiffness Shifting: An Initial Study”. In: Proc. ACM Symposium on Virtual Reality
Software and Technology. New York, NY, USA, 2009, pp. 87–90.
[36] HaptX. https://haptx.com/.
[37] Matthias Harders, Gérald Bianchi, Benjamin Knoerlein, and Gábor Székely. “Calibration,
registration, and synchronization for high precision augmented reality haptics”. In: IEEE
Transactions on Visualization and Computer Graphics 15.1 (2008), pp. 138–149.
[38] Roland Harper and S. S. Stevens. “Subjective hardness of compliant materials”. In: Quarterly
Journal of Experimental Psychology 16.3 (1964), pp. 204–215.
[39] Vincent Hayward, Oliver R Astley, Manuel Cruz-Hernandez, Danny Grant, and
Gabriel Robles-De-La-Torre. “Haptic interfaces and devices”. In: Sensor review 24.1 (2004),
pp. 16–29.
120
[40] K. Higashi, S. Okamoto, H. Nagano, and Y. Yamada. “Effects of Mechanical Parameters on
Hardness Experienced by Damped Natural Vibration Stimulation”. In: Proc. IEEE International
Conference on Systems, Man, and Cybernetics. 2015, pp. 1539–1544.
[41] K. Higashi, S. Okamoto, Y. Yamada, H. Nagano, and M. Konyo. “Hardness perception by tapping:
Effect of dynamic stiffness of objects”. In: Proc. IEEE World Haptics Conference. 2017, pp. 37–41.
[42] Kosuke Higashi, Shogo Okamoto, Yoji Yamada, Hikaru Nagano, and Masashi Konyo. “Hardness
perception based on dynamic stiffness in tapping”. In: Frontiers in psychology 9 (2019), p. 2654.
[43] Jesse D Hwang, Michael D Williams, and Günter Niemeyer. “Toward event-based haptics:
Rendering contact using open-loop force pulses”. In: Proc. IEEE Haptics Symposium. 2004,
pp. 24–31.
[44] Franklin Iheanacho and Anantha Ramana Vellipuram. “Physiology, Mechanoreceptors”. In:
StatPearls (2019).
[45] Yuto Ikeda and Shoichi Hasegawa. “Characteristics of Perception of Stiffness by Varied Tapping
Velocity and Penetration in Using Event-Based Haptics”. In: Proc. Joint Virtual Reality
Eurographics Conference on Virtual Environments. Jan. 2009, pp. 113–116.
[46] Ali Israr, Seungmoon Choi, and Hong Tan. “Detection Threshold and Mechanical Impedance of
the Hand in a Pen-Hold Posture”. In: IEEE/RSJ Int. Conf. on Intelligent Robots and Systems. 2006,
pp. 472–477. (Visited on 02/01/2019).
[47] S. Jeon. “Real Stiffness Augmentation for Haptic Augmented Reality”. In: Presence 20.4 (2011),
pp. 337–370.
[48] Seokhee Jeon, Seungmoon Choi, and Matthias Harders. “Rendering virtual tumors in real tissue
mock-ups using haptic augmented reality”. In: IEEE Transactions on Haptics 5.1 (2011), pp. 77–84.
[49] Roland Johansson and G Westling. “Tactile afferent signals in the control of precision grip”. In:
Attention and Performance: Motor Representation and Control 13 (Jan. 1990), pp. 677–713.
[50] Roland S Johansson and Åke B Vallbo. “Tactile sensory coding in the glabrous skin of the human
hand”. In: Trends in neurosciences 6 (1983), pp. 27–32.
[51] Kenneth O Johnson. “The roles and functions of cutaneous mechanoreceptors”. In: Current
Opinion in Neurobiology 11.4 (2001), pp. 455–461.issn: 0959-4388.
[52] Lynette A Jones and Ian W Hunter. “A perceptual analysis of stiffness”. In: Experimental Brain
Research 79 (1990), pp. 150–156.
[53] Lynette A. Jones. “Kinesthetic Sensing”. In: 2000.
[54] Ji-Hyun Kim, Junsuk Kim, Jiwon Yeon, Jang-Yeon Park, Dongil Chung, and Sung-Phil Kim.
“Neural correlates of tactile hardness intensity perception during active grasping”. In: PeerJ 9
(2021), e11760.
121
[55] Jong-Phil Kim and Jeha Ryu. “Energy bounding algorithm based on passivity theorem for stable
haptic interaction control”. In: Proc. Haptics Symposium. 2004, pp. 351–357.
[56] R.L. Klatzky and S. J. Lederman. “Perceiving texture through a probe”. In: Touch in Virtual Env.
Prentice Hall PTR, 2002, pp. 180–193.
[57] Roberta L. Klatzky and Susan J. Lederman. “Identifying objects from a haptic glance”. In: Perc. &
Psychophysics 57.8 (1995), pp. 1111–1123.
[58] K. J. Kuchenbecker, J. Fiene, and G. Niemeyer. “Improving contact realism through event-based
haptic feedback”. In: IEEE Trans. on Visualization and Computer Graphics 12.2 (2006), pp. 219–230.
[59] Katherine J Kuchenbecker, Jonathan Fiene, and Günter Niemeyer. “Improving contact realism
through event-based haptic feedback”. In: IEEE transactions on visualization and computer
graphics 12.2 (2006), pp. 219–230.
[60] Katherine J. Kuchenbecker. “Haptics and Haptic Interfaces”. In: Encyclopedia of Robotics. Ed. by
Marcelo H Ang, Oussama Khatib, and Bruno Siciliano. Berlin, Heidelberg: Springer Berlin
Heidelberg, 2018, pp. 1–9.
[61] Peter Kudry and Michael Cohen. “Development of a wearable force-feedback mechanism for
free-range immersive experience”. In: Frontiers in Virtual Reality (), p. 141.
[62] Rebecca L Johnson, Hugh Smith, Christopher M Duncan, Laurence Torsher, Darrell R Schroeder,
and James Hebl. “Factors that influence the selection of sterile glove brand: A randomized
controlled trial evaluating the performance and cost of gloves”. In: Canadian Journal of
Anaesthesia 60 (Apr. 2013).
[63] Robert H LaMotte. “Softness discrimination with a tool”. In: Journal of Neurophysiology 83.4
(2000), pp. 1777–1786.
[64] Nils Landin, Joseph M. Romano, William McMahan, and Katherine J. Kuchenbecker.
“Dimensional Reduction of High-Frequency Accelerations for Haptic Rendering”. In: Proc.
EuroHaptics Conference. 2010, pp. 79–86.
[65] D. A. Lawrence, L. Y. Pao, A. M. Dougherty, M. A. Salada, and Y. Pavlou. “Rate-hardness: a new
performance metric for haptic interfaces”. In: IEEE Transactions on Robotics and Automation 16.4
(2000), pp. 357–371.
[66] Susan J Lederman and Roberta L Klatzky. “Hand movements: A window into haptic object
recognition”. In: Cognitive psychology 19.3 (1987), pp. 342–368.
[67] Susan J. Lederman and Roberta L. Klatzky. “Extracting object properties through haptic
exploration”. In: Acta Psychologica 84.1 (1993). Tactile Pattern Recognition, pp. 29–40.issn:
0001-6918.
[68] Yongseok Lee, Somang Lee, and Dongjun Lee. “Wearable Haptic Device for Stiffness Rendering of
Virtual Objects in Augmented Reality”. In: Applied Sciences 11.15 (2021).
122
[69] Min Li, Sina Sareh, Guanghua Xu, Maisarah Binti Ridzuan, Shan Luo, Jun Xie, Helge Wurdemann,
and Kaspar Althoefer. “Evaluation of Pseudo-Haptic Interactions with Soft Objects in Virtual
Environments”. In: PLOS ONE 11 (June 2016), pp. 1–17.
[70] Xavier Libouton, Olivier Barbier, Yorick Berger, Leon Plaghki, and Jean-Louis Thonnard. “Tactile
roughness discrimination of the finger pad relies primarily on vibration sensitive afferents not
necessarily located in the hand”. In: Behavioural brain research 229.1 (2012), pp. 273–279.
[71] Ming C Lin and Miguel Otaduy. Haptic rendering: foundations, algorithms, and applications. CRC
Press, 2008.
[72] Jack M Loomis. “Distal attribution and presence”. In: Presence: Teleoperators & Virtual
Environments 1.1 (1992), pp. 113–119.
[73] Louise R Manfredi, Andrew T Baker, Damian O Elias, John F Dammann III, Mark C Zielinski,
Vicky S Polashock, and Sliman J Bensmaia. “The effect of surface wave propagation on neural
responses to vibration in primate glabrous skin”. In: PloS one 7.2 (2012), e31203.
[74] Louise R Manfredi, Hannes P Saal, Kyler J Brown, Mark C Zielinski, John F Dammann III,
Vicky S Polashock, and Sliman J Bensmaia. “Natural scenes in tactile texture”. In: Journal of
neurophysiology 111.9 (2014), pp. 1792–1802.
[75] Víctor Mercado, Maud Marchal, and Anatole Lécuyer. “ENTROPiA: Towards Infinite Surface
Haptic Displays in Virtual Reality Using Encountered-Type Rotating Props”. In: IEEE Transactions
on Visualization and Computer Graphics 27.3 (2021), pp. 2237–2243.
[76] Victor Rodrigo Mercado, Maud Marchal, and Anatole Lecuyer. “Haptics On-Demand: A Survey
on Encountered-Type Haptic Displays”. In: IEEE Transactions on Haptics (2021), pp. 1–1.
[77] Sean Mills, Miyuki Morioka, and Michael Griffin. “Limitations of Vibrotactile Thresholds”. In: UK
Conf. on Human Responses to Vibration. Sept. 2016.
[78] JW Morley, MJ Hawken, and PD Burge. “Vibratory detection thresholds following a digital nerve
lesion”. In: Experimental brain research 72.1 (1988), pp. 215–218.
[79] Colin Murray, F Burke, and Siobhan Creanor. “An assessment of the incidence of punctures in
latex and non-latex dental examination gloves in routine clinical practice”. In: British dental
journal 190 (May 2001).
[80] Peter Mylon, Matt Carre, Nicolas Martin, and Roger Lewis. “How do gloves affect cutaneous
sensibility in medical practice? Two new applied tests”. In: Proc. of the Institution of Mechanical
Engineers 231 (Nov. 2016).
[81] Peter Mylon, Roger Lewis, Matt Carre, Nicolas Martin, and Steven Brown. “A study of clinicians’
views on medical gloves and their effect on manual performance”. In: American Journal of
Infection Control 42 (2013).
123
[82] Peter Mylon, Roger Lewis, Matt J. Carre, and Nicolas Martin. “Evaluation of the effect of medical
gloves on dexterity and tactile sensibility using simulated clinical practice tests”. In: International
Journal of Industrial Ergonomics 53 (2016), pp. 115–123.
[83] Shogo Okamoto, Hikaru Nagano, and Yoji Yamada. “Psychophysical Dimensions of Tactile
Perception of Textures”. In: IEEE Transactions on Haptics 6.1 (2013), pp. 81–93.
[84] Allison M Okamura. “Haptic feedback in robot-assisted minimally invasive surgery”. In: Current
opinion in urology 19.1 (2009), p. 102.
[85] Allison M Okamura, Robert J Webster, Jason T Nolin, KW Johnson, and H Jafry. “The haptic
scissors: Cutting in virtual environments”. In: 2003 IEEE International Conference on Robotics and
Automation (Cat. No. 03CH37422). Vol. 1. IEEE. 2003, pp. 828–833.
[86] Allison M. Okamura, Matthew W. Hage, Mark R. Cutkosky, and Jack Tigh Dennerlein. “Improving
Reality-Based Models for Vibration Feedback”. In: ASME International Mechanical Engineering
Congress and Exposition Dynamic Systems and Control: Volume 2 (Nov. 2000), pp. 1117–1124.
[87] S. Papetti, H. Jarvelainen, B. L. Giordano, S. Schiesser, and M. Frohlich. “Vibrotactile Sensitivity in
Active Touch: Effect of Pressing Force”. In: IEEE Trans. on Haptics 10.1 (2017), pp. 113–122.
[88] A. Pourkand and J. J. Abbott. “Hybrid Force-Moment Braking Pulse: A Haptic Illusion to Increase
the Perceived Hardness of Virtual Surfaces”. In: IEEE Robotics and Automat. Lett. 5.3 (2020),
pp. 4588–4595.
[89] Ashkan Pourkand, Naghmeh Zamani, and David Grow. “Mechanical model of orthopaedic
drilling for augmented-haptics-based training”. In: Computers in biology and medicine 89 (2017),
pp. 256–263.
[90] Ashkan Pourkand, Naghmeh Zamani, and David Grow. “Mechanical model of orthopaedic
drilling for augmented-haptics-based training”. In: Computers in Biology and Medicine 89(C)
(2017), pp. 256–263.
[91] Benjamin H Pubols. “Effect of mechanical stimulus spread across glabrous skin of raccoon and
squirrel monkey hand on tactile primary afferent fiber discharge”. In: Somatosensory research 4.4
(1987), pp. 273–308.
[92] Markus Rank, Thomas Schauss, Angelika Peer, Sandra Hirche, and Roberta Klatzky. “Masking
effects for damping JND”. In: Int. Conf. on Human Haptic Sensing and Touch Enabled Computer
Applications. June 2012, pp. 145–150.isbn: 978-3-642-31403-2.
[93] Norman Ricker. “The form and laws of propagation of seismic wavelets”. In: Geophysics 18.1 (Jan.
1953), pp. 10–40.issn: 0016-8033.
[94] D Brad Rindal, Valeria V Gordan, Mark S Litaker, James D Bader, Jeffrey L Fellows, Vibeke Qvist,
Martha C Wallace-Dawson, Mary L Anderson, Gregg H Gilbert, and DPBRN Collab. Group.
“Methods dentists use to diagnose primary caries lesions prior to restorative treatment: findings
from The Dental PBRN”. In: Journal of Dentistry 38.12 (2010), pp. 1027–1032.
124
[95] Mikel Sagardia, Thomas Hulin, Katharina Hertkorn, Philipp Kremer, and Simon Schätzle. “A
platform for bimanual virtual assembly training with haptic feedback in large multi-object
environments”. In: Proc. ACM Conference on Virtual Reality Software and Technology. 2016,
pp. 153–162.
[96] K. Salisbury, F. Conti, and F. Barbagli. “Haptic rendering: introductory concepts”. In: IEEE
Computer Graphics and Applications 24.2 (2004), pp. 24–32.
[97] Lucia Seminara, Paolo Gastaldo, Simon J Watt, Kenneth F Valyear, Fernando Zuher, and
Fulvio Mastrogiovanni. “Active haptic perception in robots: a review”. In: Frontiers in
neurorobotics 13 (2019), p. 53.
[98] SenseGlove. SenseGlove Nova. https://www.senseglove.com/product/nova/.
[99] Yitian Shao, Vincent Hayward, and Yon Visell. “Spatial patterns of cutaneous vibration during
whole-hand haptic interactions”. In: Proceedings of the National Academy of Sciences 113.15 (2016),
pp. 4188–4193.
[100] Sung Hun Sim. Spatial-Temporal Characteristics of Multisensory Integration. Arizona State
University, 2017.
[101] H. Singh, A. Jafari, and J. Ryu. “Increasing the rate-hardness of haptic interaction: Successive
force augmentation approach”. In: Proc. IEEE World Haptics Conference. 2017, pp. 653–658.
[102] Harsimran Singh, Aghil Jafari, and Jee-Hwan Ryu. “Successive Stiffness Increment Approach for
High Stiffness Haptic Interaction”. In: Proc. EuroHaptics Conference. 2016, pp. 261–270.isbn:
978-3-319-42321-0.
[103] Mohit Singhala and Jeremy D Brown. “Prefatory study of the effects of exploration dynamics on
stiffness perception”. In: IEEE Haptics Symposium. 2020, pp. 128–133.
[104] Scott Stein. Haptic gloves for Quest 2 are a small step toward VR you can touch. CNET. 2022.url:
%5Curl%7Bhttps://www.cnet.com/tech/computing/haptic-gloves-for-quest-2-are-a-small-step-
towards-vr-you-can-touch/%7D.
[105] Jennifer L. Sullivan, Nathan Dunkelberger, Joshua Bradley, Joseph Young, Ali Israr, Frances Lau,
Keith Klumb, Freddy Abnousi, and Marcia K. O’Malley. “Multi-Sensory Stimuli Improve
Distinguishability of Cutaneous Haptic Cues”. In: IEEE Transactions on Haptics 13.2 (2020),
pp. 286–297.
[106] Ryo Suzuki, Eyal Ofek, Mike Sinclair, Daniel Leithinger, and Mar Gonzalez-Franco.
“Demonstrating HapticBots: Distributed Encountered-Type Haptics for VR with Multiple
Shape-Changing Mobile Robots”. In: Proc. ACM Sym. on User Interface Software and Technology.
2021, pp. 131–133.
[107] “Tactual perception of material properties”. In: Vision Research 50.24 (2010). Perception and
Action: Part I, pp. 2775–2782.issn: 0042-6989.
125
[108] W Tiefenthaler, S Gimpl, Gottfried Wechselberger, and Arnulf Benzer. “Touch sensitivity with
sterile standard surgical gloves and single-use protective gloves”. In: Anaesthesia 61 (Nov. 2006),
pp. 959–61.
[109] F. E. van Beek, D. J. F. Heck, H. Nijmeijer, W. M. Bergmann Tiest, and A. M. L. Kappers. “The
Effect of Global and Local Damping on the Perception of Hardness”. In: IEEE Transactions on
Haptics 9.3 (2016), pp. 409–420.
[110] Yasemin Vardar, Burak Güçlü, and Cagatay Basdogan. “Tactile Masking by Electrovibration”. In:
IEEE Transactions on Haptics PP (July 2018), pp. 1–1.
[111] Yon Visell and Shogo Okamoto. “Vibrotactile sensation and softness perception”. In: Multisensory
Softness: Perceived Compliance from Multiple Sources of Information (2014), pp. 31–47.
[112] Yanghua Wang. “The Ricker wavelet and the Lambert W function”. In: Geophysical Journal
International 200 (Jan. 2015), pp. 111–115.
[113] Alison I Weber, Hannes P Saal, Justin D Lieber, Ju-Wen Cheng, Louise R Manfredi,
John F Dammann III, and Sliman J Bensmaia. “Spatial and temporal codes mediate the tactile
perception of natural textures”. In: Proceedings of the National Academy of Sciences 110.42 (2013),
pp. 17107–17112.
[114] Parris Wellman and Robert D Howe. “Towards realistic vibrotactile display in virtual
environments”. In:Proc.ASMEDynamicSystemsandControlDivision. Vol. 57. 2. 1995, pp. 713–718.
[115] K. Wen, D. Necsulescu, and J. Sasiadek. “HAPTIC FORCE CONTROL BASED ON
IMPEDANCE/ADMITTANCE CONTROL”. In: IFAC Proceedings Volumes 38.1 (2005). 16th IFAC
World Congress, pp. 427–432.issn: 1474-6670.
[116] Naghmeh Zamani and Heather Culbertson. “Combining Haptic Augmented Reality with a
Stylus-Based Encountered-Type Display to Modify Perceived Hardness”. In: IEEE Haptic
Symposium Conference. 2022.
[117] Naghmeh Zamani and Heather Culbertson. “Effects of Dental Glove Thickness on Tactile
Perception Through a Tool”. In: Proc. IEEE World Haptics Conference. 2019, pp. 187–192.
[118] Naghmeh Zamani and Heather Culbertson. “Effects of Physical Hardness on the Perception of
Rendered Stiffness in an Encountered-Type Haptic Display”. In: IEEE Transactions on Haptics 16.1
(2023), pp. 46–56.
[119] Naghmeh Zamani, Ashkan Pourkand, Heather Culbertson, and David Grow. “Plate-and-Cable
(PAC) Haptic Device for Orthopaedic Training”. In: ISMR 2021- International Symposium on
Medical Robotics. 2021.
[120] Naghmeh Zamani, Ashkan Pourkand, Christina Salas, Deana M Mercer, and David Grow. “A
Novel Approach for Assessing and Training the Drilling Skills of Orthopaedic Surgeons”. In: JBJS
101.16 (2019), e82.
126
[121] Craig B Zilles and J Kenneth Salisbury. “A constraint-based god-object method for haptic
display”. In: Proceedings 1995 ieee/rsj international conference on intelligent robots and systems.
Human robot interaction and cooperative robots. Vol. 3. IEEE. 1995, pp. 146–151.
[122] Zane Zook, Joshua Fleck, and Marcia K O’Malley. “Effect of Tactile Masking on Multi-Sensory
Haptic Perception”. In: IEEE Transactions on Haptics (2021).
127

Abstract (if available)

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

SolidSpace: a device for redesigning perception

PDF

Point-based representations for 3D perception and reconstruction

PDF

Acquisition of human tissue elasticity properties using pressure sensors

PDF

Understanding human-building interactions through perceptual decision-making processes

PDF

Multi-scale dynamic capture for high quality digital humans

PDF

Efficiently learning human preferences for proactive robot assistance in assembly tasks

PDF

Hierarchical tactile manipulation on a haptic manipulation platform

PDF

Compositing real and virtual objects with realistic, color-accurate illumination

PDF

Design and use of a biomimetic tactile microvibration sensor with human-like sensitivity and its application in texture discrimination using Bayesian exploration

PDF

Augmented simulation techniques for robotic manipulation

PDF

Rapid creation of photorealistic virtual reality content with consumer depth cameras

PDF

Scalable dynamic digital humans

PDF

Digitizing human performance with robust range image registration

PDF

Anatomically based human hand modeling and simulation

PDF

3D deep learning for perception and modeling

PDF

Coordinating social communication in human-robot task collaborations

PDF

Complete human digitization for sparse inputs

PDF

Resource underutilization exploitation for power efficient and reliable throughput processor

PDF

Developing a playful situational creator for mixed reality: design analysis on Neon City, a city building game

PDF

Enabling human-building communication to promote pro-environmental behavior in office buildings

Asset Metadata

Creator Zamani, Naghmeh (author)

Core Title Perception and haptic interface design for rendering hardness and stiffness

School Viterbi School of Engineering

Degree Doctor of Philosophy

Degree Program Computer Science

Degree Conferral Date 2023-05

Publication Date 05/11/2023

Defense Date 04/20/2023

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag augmented reality,encountered-type haptic display,force-feedback,haptic rendering,haptics,hardness rendering,human perception,human-computer interaction,OAI-PMH Harvest,stiffness rendering,tactile feedback

Format theses (aat)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Culbertson, Heather (committee chair), Bansal, Somil (committee member), Barbic, Jernej (committee member)

Creator Email naghmehz@usc.edu,naghmezd@gmail.com

Permanent Link (DOI) https://doi.org/10.25549/usctheses-oUC113120788

Unique identifier UC113120788

Identifier etd-ZamaniNagh-11828.pdf (filename)

Legacy Identifier etd-ZamaniNagh-11828

Document Type Dissertation

Format theses (aat)

Rights Zamani, Naghmeh

Internet Media Type application/pdf

Type texts

Source 20230511-usctheses-batch-1042 (batch), University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the author, as the original true and official version of the work, but does not grant the reader permission to use the work if the desired use is covered by copyright. It is the author, as rights holder, who must provide use permission if such use is covered by copyright.

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA

Repository Email cisadmin@lib.usc.edu