A graphical 2023 Report Card breakdown
Jason already makes excellent and informative figures for the annual Six Colors Apple Report Card. Here I’ll just add a few more that, hopefully, will also be informative and also go just a little ways beyond what can easily be accomplished in an application like Numbers or Excel.
For simplicity, let’s say there are three perspectives on the data that we might take. We might be interested in patterns in the answers, patterns amongst the respondents (i.e. the participants), and patterns in answers and respondents considered jointly, or so to speak relationally. I’ll show an example or two in each case.
The Answers
First let’s look at the distribution of answers to the questions. A small-multiple or faceted graph lets us see the distribution of answers with each question and between questions at the same time.
This graph shows something of the variability within questions while also letting us make comparisons between them. Hardware Reliability has high scores and little variation. Meanwhile opinions on Software Quality are not terrible, but the mean score is dragged down by some particularly negative assessments. Opinions on developer relations, meanwhile, tend negative but are spread across the range of possible scores.
A second important thing to know about any survey of this sort is which questions people chose not to answer, and whether there is any structure or pattern in the non-responses. We can see right away that there is:
The question on Developer Relations was skipped the most often by far, followed by the question on Homekit. No-one failed to answer the question on Hardware Reliability. We’ll return to these patterns in non-response below.
The Respondents
We can dig a little deeper on patterns in the respondents’ answers. Just as we looked at the range of scores across questions, we can look at the range of scores awarded across respondents, too. In this next graph, we put the respondents on the y-axis and draw a little circle for every answer they gave. This lets us see how variably generous the survey participants were.
Some respondents use the full range of the scoring scale, others do not.
Questions and Responses Together
For social scientists, the most interesting patterns often arise in the way that kinds of answers are patterned across kinds of respondents. This isn’t the sort of survey where any additional information is collected from the participants, and there aren’t very many of them in any case. But even so we can do a little bit of shuffling of the rows and columns of our table of answers to bring out some features lurking in there.
Let’s look again at the answers that are missing. This time we take our table of responses Imagine a spreadsheet where the questions are in the columns and the survey respondents are in the rows. Each spreadsheet cell is a respondent’s score for a particular question, or it’s blank if they gave no answer. Now imagine we just mark all the cells as “present” or “absent”. Then we spend a bit of time shuffling the rows and columns around so that we get all the “presents” and “absents” as close together as we can, while still keeping each person’s answers together and not scrambling the content of any of the columns. (Actually we tell the computer to do this, but in the dim distant past people did indeed do this by hand on a kind of giant mechanical apparatus.) We get something like this:
This tells us that we have two main sorts of non-response. The big vertical black stripe towards the bottom of the first column shows that a lot of individual respondents just skipped the Developer Relations question, presumably on the grounds that they are not developers and weren’t in a position to give an informed answer. But most of them answered all the other questions. A smaller number of respondents took this attitude to the HomeKit question, too. Meanwhile at the top of the graph, across the columns, we can see that three or four respondents are responsible for most of the missingness across all the other questions. These respondents have views on Developer Relations, Hardware Reliability, Software Quality, and the Mac. But they skipped most of the other questions.
Finally, we can do this shuffling-the-rows-and-columns trick again with the answers as well as the missing data. There are lots of different ways to do this, but let’s follow the lead of Jacques Bertin, a French cartographer and pioneer modern data visualization who was particularly interested in teasing out the structure hidden in tabular data. Here we take everyone’s answers (the scores from 1 to 5) and shuffle our spreadsheet’s rows and columns again, trying to put people with the same pattern of answers across questions as close to one another as possible without actually scrambling the content of their responses. (This shuffling process is sometimes called “seriation”.) Then for each cell we draw a bar indicating the score. If it’s a good score, a 4 or a 5, we color it green. Otherwise we leave it as just an unfilled bar. If there’s no answer, we draw a flat line instead of a bar. Here’s what we get:
This method can be very useful for picking out similarities across respondents (or types of respondent) as well as kinds of question. From the top down you can see the near-unanimity in agreement about Hardware Quality, gradually moving towards more disagreement about the state of Software, the iPhone, and Services, and on down to the most fractured case of Developer Relations, where respondents are split between those who award a reasonably high score, those who are altogether less impressed, and those who have no opinion.
Thanks again to Jason for sharing the data with me. Because it’s a relatively small survey, the challenge it offers to a visualization nerd like myself is finding ways to represent as much of the data as possible in an informative way in a single graph.
[Kieran Healy is a Professor of Sociology at Duke University. He also works on techniques and methods for data visualization.]
If you appreciate articles like this one, support us by becoming a Six Colors subscriber. Subscribers get access to an exclusive podcast, members-only stories, and a special community.