Watching me, watching you: making sense of social interaction

» Lire en français: French

Whether it is negotiating, selling, motivating, or even dating, people with access to the latest social sensing technologies and techniques will have a distinct advantage as they engage in a range of activities, business related or otherwise. While they may not be able to read minds, they will be able to monitor the impact of their social interactions on others in real time, and modify their behavior accordingly to achieve their desired outcome.

 4 min read 

SchmidMastMarianne Schmid Mast is a professor of Organizational Behavior. Her research interests include social interactions, verbal and nonverbal communication and power hierarchies.

Imagine that you are in the middle of protracted and difficult commercial negotiation, the outcome of which is vitally important to your business. Talks have ground to a halt. There seems no way forward. You leave the room and check a smartphone app that suggests to you, based on the social interaction between the two negotiating teams, that you need to do more to gain the trust of the other side. Returning, you make some concessions to bridge the trust gap, and seal the deal. It may sound like science fiction, but it is just one example of how advances in the behavioral sciences and the way we study people interacting, are creating exciting new possibilities for business.

In the not so distant past, if psychologists wanted to study social interaction, they would have set up an experiment in a laboratory. More recently, they might have tried to capture that interaction in a real life setting, in the home or workplace, for example. But there have always been frustrating limitations to what is possible. Laboratory settings are artificial. The ability to capture verbal and non-verbal information in real life restricted to environments that can be controlled and monitored, depending on available technology. The number of participants, and the study time frame, similarly limited.

Modern technology is transforming our understanding of how people behave.

Not any longer, though, as Marianne Schmid Mast from HEC Lausanne, University of Lausanne, and her co-authors explain, in their paper “Social Sensing for Psychology: Automated Interpersonal Behavior Assessment”. Modern technology is liberating psychologists to study social interaction in ways simply not possible before, transforming our understanding of how people behave.

Social sensing allows information about verbal and nonverbal social interactions to be captured in the field and analyzed continuously and automatically in real-time. This happens without active input from the participants or human coders. The process is broken down into two main elements: sensing the social interaction via ubiquitous computing; and extracting verbal and nonverbal cues with computational models and algorithms.

Ubiquitous computing is about incorporating sensing devices into everyday human environments like the workplace, and unobtrusively capturing interpersonal behavior. This can be done on the move using smartphones, with their microphone, camera, and GPS capabilities, or with wearable technology such as smartwatches. Using mobile sensors enables the monitoring of spontaneous social interactions in the field. Alternatively, sensing may be stationary using environments equipped with sensor technology, such as smart rooms, for example.

As people interact, the technology automatically captures those interactions and, using machine perception and learning technologies, and complex computer algorithms, extracts relevant data. Thus psychologists can obtain real-time or stored data on behavioral cues such as eye gaze, body posture, head pose, and hand gestures. Mobile devices can monitor conversations and extract information such as speaking rate, turn-taking behavior, pitch, and energy levels.

These capabilities lead to more innovative work as behavioral scientists tackle questions that were impossible to study previously.

Not only will these capabilities lead to more research on interpersonal behavior, but also to more innovative work as behavioral scientists tackle questions that were impossible to study previously. As Schmid Mast points out, for example, it is now possible to track the day-to-day social interactions of large numbers of people in an organization, over lengthy periods of time. This allows researchers to study the links between employee interaction and specific outcomes, such as productivity or job satisfaction, for example, and how these change over time, depending on the context and operational environment.

People make sense of social interactions by observing and interpreting the expressive verbal and nonverbal behavior of the people they interact with. They use this information to make inferences about the other person’s traits, how they are feeling, their intentions, and so on. Social sensing machines can be programmed to make the same kinds of inference, with the information made available to an end user – via a smartphone app, for example. Equally, the extracted data can be related to task performance. In both cases the information provided can be used for a variety of purposes, training and development being one obvious example.

So end users could use social sensing to monitor how persuasive they are being in a presentation, whether they are perceived as trustworthy during a negotiation, or even if someone finds them attractive on a date. This could be done after the event, or even in real time. Automated feedback is available instantly making it theoretically possible for individuals or groups to modify their behavior as they interact, based on the social sensing information they are receiving about that interaction.

The more an employee communicated the less satisfied they were with the job.

Social sensing studies are already taking place, as Schmid Mast and her co-authors detail in the paper. A study of the interaction of 22 employees using wearable devices over a month, for example, revealed that the more an employee communicated the less satisfied they were with the job.

Another study, by Schmid Mast and fellow researchers, identified the nonverbal behavior pattern during a job interview that best predicted whether the job applicant would subsequently be successful in a sales type job. This information could then be used to screen future candidates. Or, in another case, vocal behavior cues of call center workers were analyzed in the context of customer satisfaction levels. The resulting data could be used for employee training purposes, or to select new call center employees.

Social sensing is not without challenges. As people are involved in coding the learning algorithms, care must be taken to avoid bias, for example. Single devices that integrate sensing and extraction capabilities are in short supply. Behavioral scientists and computer sciences experts will need to collaborate effectively.

Ensuring social sensing is used responsibly.

Perhaps the biggest challenge for social sensing, though, will be the ethical aspects and ensuring social sensing is used responsibly. Take privacy, for example. Given the potentially pervasive and unobtrusive nature of social sensing, researchers will need to be acutely sensitive to the privacy issues involved, including areas such as consent and data sharing.

But assuming that these challenges can be overcome, social sensing promises to revolutionize the way that we study human interaction. It offers considerable opportunities for organizations and individuals that want to improve their performance and, in addition, may produce a new generation of behavioral researchers.


Read the original research paper: Schmid Mast, M., Gatica-Perez, D., Frauendorfer, D., Nguyen, L., & Choudhury, T.
Social sensing for psychology: Automated interpersonal behavior assessment. Current Directions in Psychological Science.


Featured image by Antoine Robiez / Flickr CC