A final look at the Pew Survey

It’s been a couple weeks, but as promised, here’s my comprehensive look at the results of the Pew Religious Knowledge Survey.  This post is very table dependent and I wrestled a bit with the methods I employed to pull out patterns as well as how I selected the table to display.  Constructive critiques and requests for additional tweaks are very welcome.  Feel free to email me or leave a comment if you have a criticism or want to see the data treated in a different way.

In the last post on this topic, I removed Black Protestants and Hispanic Catholics from the table.  In this post, I will have three versions of every table:  one with all groups included, one with super-groups (Christian, Unaffiliated, etc.) removed, and one with Black Protestants and Hispanic Catholics removed.  The second version will be used as the visual material in line with the discussion of the post, with larger versions of each version available at the end of this post along with some more colorblind friendly versions.

The straight performances

To start, here are the average correct response rate for each of the subcategories delineated by Pew in the section of their report discussing group performance by question and category:

Areas of relative strength

Each group showed a different areas of strength and weakness, but looking at the the absolute strength of a group’s performance in a category doesn’t necessarily give a clear picture as to which areas a groups excelled in given their baseline performance.  One way to do that is to take the

I would read this chart as a relative performance of each group on a subject.  While this relative performance is dependent upon the performances of other groups on the subjects, it is not a good reading to suggest that a group with a value <1 on in a category is on average better versed in it than a group with a value of >1.

A prime example of this would be the comparison of Hispanic Catholics to Atheists/ Agnostics on the questions defining atheists and agnostics.  Unsurprisingly, in absolute terms, the Atheist/Agnostic group scored best on these questions and the Hispanic Catholic group scored the worst.  However, as you can see in this chart, the Hispanic Catholic group had the highest value and the Atheist/Agnostic group is below average.  This is because as a relatively easy question, the extent to which the Atheist/Agnostic group could over-perform was significantly reduced and Hispanic Catholics, having scored over 50% on that set of questions and barely more than half as well as the Atheist/Agnostic group overall, were given an insurmountable handicap (the Atheist/Agnostic group would have had a 1.04 if they had answered 100% correctly on those two questions).

Ultimately, I think that this chart shows a pretty strong “home turf” effect.  Christians tended to find their greatest strengths on questions concerning Christian issues whereas the relative strength of  the non-Christians lied in questions about other religions.

Strength relative to general knowledge

Whereas with the last table, I didn’t see a direct comparison between groups to be an appropriate way to read and interpret it, this graph is different.  The “correction” this time is not based on the questions about religion, but rather the general questions that were also included in the survey but not included in the “religious knowledge” results.

Since the questions used to judge the relative performance were not the superset of the questions making up the categories, a group’s performance won’t necessarily look “average” over all, as it did in the previous table.  This is particularly true for Black Protestants and Hispanic Catholics, who did worse on the general knowledge questions than the religious questions, relative to the other groups.

Ultimately, those two groups expose some weaknesses of this method.  The first problem is that the difficulty level of the questions isn’t the same.  In general, survey respondents answered correctly a higher percentage of the time for the general questions than for the religious questions.  Furthermore, there is a spread in how difficult questions in different categories are.  So,  while hardly any of the respondents knew who Maimonides was, the vast majority knew what the definition of an atheist was, so different categories end up more or less comparable to the baseline used and the less comparable they are, poorer this method represents how well a given group did in a category.

This is especially true for Black Protestants and Hispanic Catholics.  They performed poorly pretty much across the board, which means that dividing by their percentage correct on general knowledge (on which their performance was worse relative to the other groups than on religious question) the final ratio is catapulted upward, often beyond what other groups could have possibly reached had they gotten a set of questions completely correct.

Ultimately, if you want to know where a group is strong overall, stick with the absolute percentages.  These two treatments of relative performance can help illustrate a group’s point of emphasis in its knowledge, but it won’t tell you a whole lot about who has better knowledge of a subject area.

Pew’s own analysis

All of these tables are based upon the data that Pew released, not the original data, meaning that there’s a lot less that I can do relative to Pew, so I do want to take note of some of the details that they released in their section on additional factors predicting performance on the survey.

First would be to look at the performance of different groups when Pew controlled for other factors:

Caveat lector: the first two categories probably include black Protestants who fall under either the “Evangelical” or “Mainline” categories in the performances controlled for other factors, while for the uncontrolled data, Pew only distinguished between white Evangelicals and white Mainline Protestants, which are the numbers plugged in above.

Coincidentally, outside of those two groups (and really outside of Mainline Protestants), controlling for other factors lead to less deviation from the average performance.  Obviously, some still remains, but it’s unclear if that’s merely an effect of not choosing the right variables to control or if being an atheist or agnostic really causes one to be more knowledgeable about religion (0r vice versa).

Another point that I missed in my earlier postings is performance based on one’s views of God and the Bible:

Clearly, those who do not believe in God are by definition (or at least my definition…I’ll address that a bit in my next post) atheists, and yet the average score for such people 18.7, is below that of the Atheist/Agnostic category, which really does suggest that the very use of “atheist” and “agnostic” as labels created a selection effect whereby the more religiously informed atheists would identify into one of the categories and the less religiously informed would identify as “nothing in particular” or (perhaps more strangely) one of the other religious categories (particularly Buddhists and perhaps some Jews).

Just to quantify that a bit better, the “Nonbeliever” category (more precisely, those who answered that they did not believe in God) consisted of 233 people, which is 21 more than the 212 in the Atheist/Agnostic category.  Since I’m sure that some of the agnostics either were part of the group of uncertain believers or the group 91 who were not included (likely because of a refusal to answer the particular question) in the statistics behind the above graph.

Regardless, it does seem that taking a literalist stance on the Bible is correlated with poor performance.  The pattern of performance associated with the nature of one’s belief in God segways nicely into a final pattern that is shared between the Pew survey and another survey released recently, this one by OkCupid, an online dating site, of its users.  They compared the sophistication of writing of their users in their profile descriptions by entering it into the Coleman-Liau index, giving the following plot by religious group, separating different levels of “seriousness” of belief:

As you can see, for Protestants, Catholics, Jews, and Buddhists, the lowest profile writing sophistication comes from the “somewhat serious” and the highest comes from the “not serious”.  In Muslims and Hindus, it is the “very serious” who have the lowest writing sophistication and the atheists and agnostics the trend is the reverse of the Muslims and Hindus.  The measure that reminded me of this in the Pew study was the comparison of survey performance by religious commitment of participants

As you can see, this follows the pattern of Protestants, Catholics, Jews, and Buddhists in the OkCupid, which is to be expected as Protestants and Catholics together make up a clear majority of the country.

Now, there is a great deal more interesting material to be found in Pew’s report, so if you find these highlights to be tantalizing and have not yet read the full report, go ahead and do so.


Alternative Tables

Straight performance

Relative strength on religious areas

Strength on religious areas relative to general knowledge

More detailed explanation of methodology

The first set of tables were made by simply averaging the percentages for each question within the subcategories delineated in this page of the Pew Report.  To be more specific, here’s how my labels correspond to Pew’s and to which questions in Appendix B they correspond, with the amount of questions (not always entirely clear from question numbers) in parentheses.

Old Testament – Old Testament – Q39, Q46, Q47a-d – (5)
New Testament – New Testament – Q40, Q41 – (2)
Catholic/Protestant Theology – Catholicism and Protestantism – Q44, Q45 – (2)
Christian Figures – Religious figures – Q48a, Q60, Q63 – (3)
Mormon – Mormonism – Q42, Q43, Q48c – (3)
Judaism – Judaism – Q15, Q48d – (2)
Islam – Islam – Q17, Q36 – (2)
Buddhism – Buddhism – Q37a, Q48b – (2)
Asian Nations – Majority religion in… – Q55a-d – (3)
Other Religions – Other – Q35, Q37b – (2)
Atheist/Agnostic – Atheism and Agnosticism – Q38a-b – (2)
American Law – Religion’s Role in Public Life – Q49, Q50a-c – (4)

The second set of tables were created with two steps: first, each cell in the first table was divided by the total percentage correct for a given group; second, the new results for each group were divided by the new result for the whole sample (the “All” row in the first two variants in the first set of tables and the “Subtotal” row in the third variant).

The third set of tables follows a similar method to the second set, though in the first step, percentages are divided by the percentage each group answered correctly on the general knowledge questions as opposed to the religious knowledge questions.


About Meng Bomin

Real name Benjamin Main, I am a graduate of Grinnell College with a degree in Biological Chemistry.
This entry was posted in Religion and tagged , , , , , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s