Let’s face it, there is a lot to unpack when it comes to Attention data, especially if this type of data is new to you. With growing interest in Attention data in the media industry today, we get all sorts of questions about how Attention data works.
First things first, what is Attention data at Element Human?
Element Human offers an empathy data set which is a combination of 3 types of biometric data: what people focus on when they watch a piece of content (Attention data), what people feel when they watch a piece of content (Emotion data) and how the content impacts people’s brain after watching a piece of content (Implicit test). This data set, along with what people say (Survey data), builds a more holistic view of how people engage overall with digital content, giving our clients the tools to empathise with their audiences.
Attention data is the result of tracking people’s eye movement to help us determine if they are indeed keeping their eyes on-screen and what in particular is holding their gaze whilst watching a video.
From there, our Attention model, developed in-house by our Data Scientists, uses this data to predict where people are looking on the screen. This algorithm powers 3 main features in our dashboards:
the attention series in the time series graph
the attention score
💥 The Challenge
We think our Attention model is amazing, but let's put it to the test! We challenged our data scientists to prove why the Element Human Attention model is that good. Here is what we found…
📓 The Experiment
To test the accuracy of our model, we asked 490 humans to take a custom survey in which we asked them to do one task: follow a pong ball around their screen and count the number of times it changed colour. This task is designed to have participants pay close attention to the ball's position, which we used as a proxy for true gaze coordinates.
The hypothesis is that there is a strong correlation between the accuracy of a participant’s colour-change count and how close the person's predicted gaze is to the ball's position. To put it simply, for a person to keep an accurate count, they should be watching the ball's movement closely.
In terms of demographics, all 490 participants were from the United Kingdom. The gender distribution was 51% Female, 49% Male, and 0.2% Other. 58% of people took the survey using a PC device and 42% used a mobile device.
🚀 The Results
This study resulted in an attention score of 87 out of 100, which means that, on average, people were looking at the video 87% of the time. So we can reliably say that our participants were paying close attention to the ball's movement. Most importantly, the heatmap clearly shows the trend of people watching the pong ball move around the screen.
As the video progresses, the heatmap starts to deviate slightly from the ball's exact position to the area in front of the ball. There could be many reasons for this difference, but our data science team's analysis concludes that this is an example where people began to predict the ball's movement.
After reviewing the data, the original hypothesis appears to hold true as shown by the graph below. The team concluded that there is a significant difference in answer accuracy (answering the question, 'How many times the ball changed colour?’) between people who were paying attention (the 'predicted gaze distance from ball' is small) and those who weren't.
Relationship between answer error and predicted gaze distance from target
✍️ Top 3 Takeaways
Here are the 3 things we learned from this experiment:
Our gaze model accurately predicted participants' gaze location as seen by the resulting heatmap.
It's helpful to dig deeper into the video context and human behavior when there are unexpected heat map results. In this case, the concentrated area of the heatmap began moving in front of the ball instead of surrounding the ball. This can be explained by the fact that people tried tactics to become more efficient at the task at-hand by anticipating the ball's movement.
There will always be an element of human error. In this study, most people with incorrect colour-change counts were actually paying close attention based on their predicted gaze location.