New Membership Survey

In this project, what we are trying to do, is to design a new way of doing membership survey to more accurately gauge the satisfaction of our volunteers, and use it to make better decisions in regards to volunteer management and engagement.

This project is mainly inspired by this talk by Dave Snowden, especially his knowledge and insight on the fallibility of existing HR engagement survey formats and his recommendations on how it can be improved.

Our existing membership surveys are done based on existing conventions in survey design, especially in regards of HR. The questions we ask to gauge membership engagement & satisfaction is as shown bellow:

As you can see, it is the standard question to gauge Net Promoter Score, overall rating of their AIESEC experience, and open ended questions about what is great and what could be improved.

1. As described by Snowden in the video, it is too easily gamed by the person answering it. As these are standard questions, people know what kind of answers are expected of them. This possibility for the answer being gamed decreases the reliability of the answers we receive. As we have no way of gauging who is being truthful or not, the whole data set is considered unreliable.

2. The answers we gather here, is highly subjective to the person answering it. This is not to say that subjective answers are not valuable. However, in the context of trying to evaluate if people are happy with the culture of the organization, it falls extremely short, as it fails to provide a big picture understanding of current satisfaction trends within the organization.

It is with these reasons, we are using Dave Snowden's video as a guide to design a better membership survey to be able to better capture the reality of membership satisfaction and engagement. There are a couple of objectives we want our surveys to achieve:

1. Questions with no clear "ideal/correct answers" - As explained by Snowden, the only way to get proper feedback is to never put dichotomies, especially on a spectrum. As dichotomies are always one positive and one negative, people tend to gravitate towards the positive answer, which might not be representative of their actual experience.

2. Open ended question to collect actual stories - For open ended question, rather than asking what they think is good, which is always impacted by the Recency Effect, it is better if we just ask them to tell us a story about their experience that stood out to them. The reason for this is explained really well by Snowden in the linked video above.

As a result, the new survey is as follows:

This first segment is a multiple choice grid, where we want to understand what does a member perceive to be the priorities of their team leader. Notice it is not a question about if their team leader is good at their job, but rather about what do they think is the focus of their team leaders. This type of question have no clear "correct" answer which prevents people from gaming it, and allows us to collect answers that are more representative of people's actual experience.

The second segment, is an open ended question which invites people to share the one memory with their team leader that stood out to them. It is also a story that they would share with their friends if they asked. The great thing about this question is, it shines a light on the person's impression of AIESEC, and their satisfaction on their experience with us. In other words, we can see if they associate AIESEC with a positive or negative experience, which is information that can't be collected with a simple NPS question.

A side note, the reason why we ask them specifically on their experience with their team leader, is because a member spend the most time with the person that is directly managing them. As a result, their direct team leader has the biggest influence on their AIESEC experience and their enjoyment of it. Therefore, collection of information on how is a member's experience with their current team leader is a lot more valuable than asking if they are happy with the executive leadership team. Furthermore, this also makes the data a lot more actionable, as to improve a member's experience we just work through their direct team leader, rather than trying to mobilize the whole leadership team. In other words, this survey also allow us to provide idiosyncratic support to our people.

In terms of output, I utilized Google Data Studio to create an aesthetic and interactive dashboard, as shown below:

With the report, what I am trying to do is to draw a correlation between the new information we gather and membership NPS. At its core, what the new survey questions provides us is a picture of the current experience of a member. Therefore, when we separate them based on their reported NPS score, we can see if there is any difference between the experience of a Promoter, Neutral or Detractor.

From the picture above, you can see that in general that most promoters reported that their direct team leader's top priority is in supporting them, as opposed to a neutral and a detractor. What we can infer from this information, is that:

1. What members are looking for is a team leader who is supportive of them. Someone that guides them through the challenging aspects of their work

2. To facilitate better membership experiences, we should get our team leaders to devote more time and energy in supporting their members.

From this, a simple action step is to identify the current team leaders who are doing this well in the network, gather Good Case Practices from them, and downscale this new "management direction" down to the rest of the team leaders, alongside the right education, resources and support.

All in all, these new questions allowed us to not just gain a better insight on the current experiences of our membership, in an unbiased manner, but also enabled us to ask better questions about our current organizational culture, and if we are nurturing it in the right direction.