I think it’s impressive that the Church attempts to improve its programs by regularly surveying its members to gather input on various issues. I say impressive because I’m not sure other Churches are doing the same or attempting to do so, although it’s important to bear in mind that they are not all strictly “top down” leadership either; the people can more readily vote with their feet, their donations or literally vote in elections for leaders in many denominations. As Mormons, we’ve seen surveys on all sorts of things over the years: garment design, doctrines, meeting structure, curriculum, beliefs, etc.
One of the podcasts I listen to regularly is the five-thirty-eight podcast with the statistics guru Nate Silver and his cast of pollsters. They discuss all sorts of polls that have happened or been published in recent news, and one of their segments is to assess whether a specific survey was a “good use of polling or bad use of polling.” There are many reasons they give for things being a bad use of polling. A few of them include:
- Sampling errors. These include things like asking the wrong people, too many people, too few people, only a subset of people, missing an important demographic, sampling only tech savvy people or people who answer the phone during the day.
- Bad wording of questions. There are many versions of this: worded in a way that’s hard to understand, lumping multiple topics into one question so results are conflated, leading the person to answer in a specific way.
- Pointless. Only revealing information that everyone already knows, so it won’t have any impact, or the information being gathered has no way to be implemented in public policy, so it’s not really helpful at all.
A recent Church survey was done to gather input on issues that appear to be either related to women’s concerns, and/or related problems that lead to disaffection. Let’s play Nate Silver’s game and weigh in on whether you think it’s a good use of polling or bad use of polling. Here’s a list of some of the questions with some commentary.
- How likely is it that you will be active in the Church 5 years from now? (Options are “Very likely, somewhat likely, somewhat unlikely, very unlikely, unsure.”)
We start with a doozy of a question that makes me immediately suspicious of the motives in asking it: are they trying to weed out people who plan to leave or possibly to give extra weight to those who are considering leaving? It’s also been my experience that nobody plans to leave the Church five years in advance; if I’m in now, why do I think I will be out in five years? It’s just odd given the really long time frame. I’m going to go with bad use of polling. Why five years? Let’s continue.
How much do you agree or disagree with the following? (This is on a scale of “strongly disagree, disagree, neither agree nor disagree, agree, strongly agree”).
- I believe the Church is led by a prophet of God today.
- I believe Joseph Smith was a prophet of God.
- I am troubled by some Church history, doctrines or practices.
- “The Family: A Proclamation to the World” has been a helpful document in my life.
- The practice of polygamy during the early days of the Church.
These are a mix of belief-based questions, so they still feel “demographic” to me, setting up who is answering the questions, rather than actually gathering the information they seek. Demographic questions are used to divide the other questions into groups, so the intention with these seems to be to assess orthodoxy (belief level) of the respondent. That last one is an obvious mistake in wording / put in the wrong section. The options are a scale of agreement, so what exactly is the respondent supposed to be agreeing with for that one? Polygamy happened in the early Church; it’s a fact, not an opinion. Clear bad use of polling on that one as there’s no answer that makes sense. If it’s gauging comfort level, that scale is in the next section, so “oops.” Parenthetically, I can only assume that the fourth bullet is an IQ test of sorts, as it makes no sense how this document is “helpful” to individual Church members. What help is it giving them exactly? Again, maybe someone can explain to me how this wording makes sense, but I’m not seeing it. 
This next block of questions is rated based on a scale of discomfort rather than disagreement, referring to the “importance” attached to these issues for the respondent (options are: “not at all, very little, somewhat, a fair amount, a great deal”). This type of question is attitudinal, meaning that while you can believe a thing to be true (or false), you can either like it or not like it. This is more gauging the “liking” than the “believing” or agreeing.
- The Church’s teachings about women dressing modestly.
- Not knowing what happens after this life if a man remarries and is sealed to more than one woman.
- The amount of information available about Mother in Heaven.
- The Church’s policies regarding LGBTQ members.
- The number of women speakers in General Conference (other than in the Women’s Session). Clearly this was written before the recent announcement that the Saturday evening sessions are being shelved.
- The way that sexual intimacy is taught or addressed in Church settings.
- Past policies that excluded black members from receiving the priesthood. As usual, they didn’t mention the temple restrictions that impacted both black men & women.
- The Church’s teachings about when to marry, start a family, or how many children to have.
- Any other teachings or practices you find uncomfortable (please specify).
For this next section, we are back to the agreement scale, assessing how strongly the respondent agrees or disagrees with the following. Unfortunately, from where I’m sitting, several of these specific issues are 1) worded weirdly, 2) mixing multiple concepts into single questions, making it difficult to answer accurately:
- Too much emphasis in the Church is placed on men’s roles as priesthood holders, rather than their roles as husbands and fathers. Why lump all of these together into one question? These are two different issues. It’s like when a radio station says “more rock, less talk.” Some people want less talk, but also don’t want more rock. Some people want more rock but they like talk. For me, I’d rather reduce how much gendered talk there is altogether. Jesus taught men and women the same things. The gospel principles aren’t gendered.
- Women with young children at home should only work if it is financially necessary.
- I believe women should be ordained to the priesthood. This one also makes me suspicious. If they are asking this, but it’s completely off the table as an option, then is it just being used to weed people out?
- I believe that women are not currently ordained to the priesthood because of Church policy, not Church doctrine.
- It is appropriate for Priesthood leaders to teach young men (age 12-18) about issues of modesty and sexual morality. Another one where two issues: modesty and sexual morality are lumped together. What if the answers differ for each of these topics? Also, this just seems weird to create gender parity. Do we really think that the Church is teaching boys about modesty on equal footing with how girls are taught about it?
- It is appropriate for Priesthood leaders to teach young women (age 12-18) about issues of modesty and sexual morality. This is even worse than the prior question for lumping things together. Now we have three things: two topics being taught and the fact that men are teaching girls about sexual topics. A respondent might feel that these are appropriate topics but should be taught by women to young women. Or they might feel that sexual morality is a good topic but not modesty.
- Women should have the same opportunities as men in education, business, government and the community.
- Gender should not be considered by those in hiring positions.
- Women should be strong, independent, thinking individuals. As opposed to what? Is the question implying that some people think women should not think? Why are these traits all lumped together in this way? Most feminists would not prescribe how women should be. Like men, they can have their own personalities. Feminists don’t require women to be “warrior princesses” or work in STEM to be valid. Does something think they do? This question feels bizarre. What are they getting at?
- Husbands and wives should have an equal say in decisions in their marriage and family.
- Women should have as much input as men in decisions about Church policies and practices that affect them. The problem with this question is the qualifier “that affect them.” Of course, it’s unfathomable that women would have any input on anything that affects men. As the Apostle Paul would say: God forbid!
- I consider myself a feminist. This is another one that, given the Church’s track record, scares the hell out of me. I can’t help but imagine anyone daring to claim to be a feminist having their survey chucked right in the trash. Even if the survey group is professional about it, will Church leaders want to hear anything from people who identify as feminists?
This survey is published by something called the Correlation Research Division. I’m not familiar with that group, but I thought it was interesting to note. As with all Church surveys, these results will not be made public, so the use is strictly to inform leadership of issues members care about. It’s up to the leaders to decide what to do with that feedback, if anything. From what I understand, these polls are only created when requested by a specific leadership committee in the Church. Given that purpose, is this a good use of polling or bad use of polling?
My initial big picture view of this poll is that it’s well-worn territory in the wake of Jana Reiss’s book The Next Mormons. The polling data she provided already covered many of these topics, but not all of them. Perhaps the Church feels that they need to either confirm or debunk her results by doing their own survey, which makes me wonder about the people they’ve selected to take the survey (I did not receive it directly, so I’m not part of the respondent pool). So here are some possible reasons they want this information:
- To make tweaks to curriculum, councils or leadership.
- To inform or educate church leaders, to get them up to speed on topics that matter to members.
- To identify practices that are driving people out of the Church.
- To comfort themselves that no change is needed because only feminists and apostates dislike how things are.
I’ll let you weigh in on what you think the use of this data will be. Pres. Nelson does like making changes to councils, meetings, and so forth, so you never know, although we don’t know whether he personally will see this data or if it was for some other purpose. Also, Church leaders reserve the right to make decisions about changes based on revelation after discussion within councils. Survey results function as a catalyst to those discussions, giving members a voice in the process.
Let’s see what you think of this survey:
- Overall, good use of polling or bad use of polling?
- What do you think the results will reveal?
- Are there some things you would change about the survey questions?
- Were you asked to complete this survey or others? What’s your general impression?
- Do you think these surveys are resulting in positive changes that improve the Church experience or not? Explain your answer.
- What surveys or poll topics would you like to see in the future?
 I did look up on the Church’s website, and apparently they do think it’s a “useful” document, explaining that families consist of siblings as well as spouses, and that “we preach the ideal, but live reality,” or something like that. I mean, whatever. Their assertion of its “usefulness” is lost on me, and their statement about how it was written are simply false, but I’m sure that many run-of-the-mill Church members don’t know that. I’ve said it many times elsewhere: if the document is descriptive (which is how it reads), what’s the point*, and if it’s prescriptive, it’s destined to fail. *The point is to give the Church grounds to oppose gay marriage legalization in Hawaii in 1995 by putting its teachings on marriage explicitly in writing.
 This rubric describes four options: I believe this thing, and I like it. I believe this thing, and I don’t like it. I don’t believe this thing, but I would like it. I don’t believe this thing, and I wouldn’t like it.