Heat Map
projecting imaginary heat on human skin

A translation between sight and touch merges human sensing abilities. Projecting imaginary heat on top of human skin and its surrounding to create a virtual thermo reactive surface.
We perceive heat only by using our tactile sense. Thermography, on the other hand, is a camera concept to visualize variations in temperature. Hence, this technology offers the possibility to experience a sensation by sight, which we are only able to recognize by touch. This translation between different types of senses enables a unique way of viewing and perceiving our environment.
This project is motivated by the translation between sight and touch. We combine depth sensing technologies and projector-camera alignment to map real world behavior and their virtual representation. Our system offers the possibility to visualize imaginary heat as we touch a table creating a virtual thermo reactive surface.
The amount of heat is determined by the distance between our touching hand and a table. In order to implement this system we decided to measure this distance by the use of a depth camera. The corresponding temperature is projected by the use of a heat map.

Detecting the touch
In order to measure the distance between a touching hand and the table, we need to separate the camera data retrieved from the table from the data of the hand. Therefore we use a calibration method utilizing our depth cam, to calculate the table its position and orientation in three dimensional space. Once this procedure is finished, we are able to detect the hand as an outlier hovering above the table. The sum of connected pixel which are at least a half centimeter above the table, represent our region of interest and is further defined as the interacting hand. Having calculated the orientation of the table and the hand its region, we are able to calculate the distance of each pixel of the hand in reference to the plane. The heatmap is a data array, which stores a counter value (simulated temperature) for each pixel of the projection area. This counter value increases, if a certain pixel of the hand is above the corresponding pixel of the projection area and closer than a certain threshold. If this condition is not fulfilled the relevant counter decreases. Hence the presence of a pixel that belongs to the hand heats up the spot.

Visualizing Heat
In order to visualize the effect of a heating up hand or object, we need to assign different colors to the corresponding counter value. The color of the projected pixel is read from a gradient image representing the heatmap. The x position of the selected color refers to its counter.
We use OpenCV and lightweight triangulation in order to map the thermographic map of heat on top of our hand. We decided to not use further advanced three dimensional camera and projector calibration, to make sure that the application and its effect is enjoyable with the smallest possible delay.

We presented an application, which allows the user to perceive imaginary heat projected on top of his skin. This concept led us to further ideas and experiments. We were able to investigate interesting effects, as we used different objects on top of the table. Objects that differ from human skin in its geometric and reflective features. Hence we observed beautiful effects which led to different interpretations then the original desired one. Merging different sensing abilities and the way we perceive sensual influences, we like to continue to explore new ways of communicating and perceiving.

Heatmap Flickr Set