Emoji Likert Scale: How to Master It
In my first post you have learned about how I have developed the Emoji Likert Scale, its purpose and advantages. However, although using the tool is pretty straightforward, there are some aspects you may want to keep in mind:
1. What to do when there is no baseline data?
We are often hired to evaluate initiatives that started before the evaluation itself. When this happens the best approach is to try by all means to reconstruct the stakeholders’ memory. How?
The first step is to register people opinion regarding the present moment. The present is easier to recall than the past. You start by asking the stakeholder how would he/she grade the quality of his/her social relations today. Then you register his/her answer. Than you make the same question about the past.
In order to do this properly it is important to be able to define well the past moment. We can not get people there if they don’t know what period of their lives are we exactly talking about - “Before” is vague. One possibility is to frame a date. Let's assume, a year ago or, for example, at the beginning of 2017. We may ask how was the quality of his/her social relations with friends and family in January 2017.
Another alternative is to focus on a specific event. For example, let’s say that you’re evaluating in 2018 an activity that started in 2015. Many people might find it difficult to return so many years backwards. What you can do is to create anchors: “How were your relations with friends and family in the year before the project started, when the windmills came to the village”
Once you define the past moment will is time to compare the past with the present and create a bid divided in the following parts:
How do you rate your relations with friends and family now? (present)
How were they in January 2017? Were they better or worse?
Was it a lot better (your finger goes to the extreme maximum of the scale) or only a little better (your finger goes advances only one point forward)?
If none of those, how much better then?
2. Does this scale only work for subjective indicators?
No. I have focused on a subjective indicators because they are the most difficult to measure. However, the scale also has numbers, which can be used to measure more tangible data such as the number of times a person goes to the doctor; the number of pieces of fruit someone eats daily or the number of meetings with friends per week.
3. How do you measure impact from here?
So far I have explain why this scale is useful to measure outcomes, i.e., the change that takes place from T0 to T1. Nevertheless, the emojis likert scale is also a powerful tool to estimate the amount of project’s impact. I.e., it helps us measuring the amount of change attributable to our initiative. To do this, simply ask a third question to your stakeholder: "where, would he/she be in the scale if the intervention never happened?" This should give you a grasp of what is the stakeholders perception of impact.
If you think about impact evaluation as rope with two extremes, this method is in the middle. At one extreme are the experimental studies. Quantitative, traditionally random, with a control group and significant samples. At another extreme are the contributions analysis. Qualitative studies that favor the context; strive to understand the process; study meanings and narratives.
This scale offers the possibility of codifying qualitative information into quantitative data and is a very cost-efficient solution because with very little data collection effort (just by introducing an extra question in the survey) you get very rich data. This said, it’s always important to triangulate this data with other sources as it is not a stand alone piece.
4. Extra clues for measuring impact with the emojis likert scale
After asking people how they feel now and how they felt before, say: “ok, let’s imagine that this project never had existed. Where do you think you would be standed? Do you think you would be worst off or better of?” To approach this question you may use the same elicitation method applied early on. If better how much better, if worst how much worst.
Bare in mind that people, at this stage, already made a great effort to visit the past. The question about an imaginary alternative scenario will be seem hard at a first look. You will have to invest sometime creating an extra room in people's mind where this hypothesis actually exists and is well understood. In order to do it, I advise you to first create a dialog, without even showing the scale.
A second option is to ask people a direct question: “how much the project contributed to that quantitative move from T0 to T1. Suppose the person went from 1 (2017) to 3 (2018) a 2 points change. You can ask, how much of that change happened because of the project? Nothing, a small portion, a big portion? According to stakeholder’s reply you can adjust the scale in front of them.
Remember this is the least desired scenario! You have to be extra careful with the way you pose your question in order to avoid influencing the response. Make sure you have a prof that what they are saying is supported by facts. Respondents will tend to please you attributing more value to the project than it actually has. Ask respondents to explain what they are thinking about! If the project has contributed to improve their health in 25%, how did it do it? Was it because they started to drink more potable water as a result of the project? Was it because as a result of the program they have more income and are able to eat better and go to the doctor more often?
Try it, let me know what you think and share it if you liked it!