Favorite Works Survey - 2009

Summary

Roger Cutler conducted a survey of Houston Symphony Chorus members in May, 2009 on the subject of "favorite works".   The "bottom line" of this survey is the following overall ranking in two groups, with the order within each group being uncertain:

1. Brahms - German Requiem
2. Verdi - Requiem
3. Orff - Carmina Burana
4. Dvorak - Stabat Mater

5. Mozart Requiem
6. Beethoven - Symphony #9
7. Berlioz - Requiem
8. Beethoven - Missa Solemnis
9. Handel - Messiah
10. Mahler - Symphony #2 ("Resurrection")

A similar, but much smaller, survey was taken at the beginning of 1996, with similar results for the top three but substantially different works thereafter, including Vaughan William's "A Sea Symphony", the Britten War Requiem and Mahler's "Symphony of a Thousand".  Haydn's "The Seasons", Bernstein's "Chichester Psalms", the Bruckner and Schubert masses and the Faure Requiem have also been appreciated by the chorus.

The 1996 Survey

This old survey was based on scraps of paper collected from a number of people on which they recorded their "top 5" favorites from past years.  The analysis then attempted to compensate for the fact that more people were familiar with more recently performed works by including a gentle weighting, resulting in a score between zero and thirty.  The results are summarized in the figure to the left (click on the thumbnail to see a large image). Although these scores are not based on a great deal of data and should not be taken too seriously, one can still clearly recognize three three tiers of "favorites":  the top tier includes the Brahms and Verdi Requiems and Carmina Burana.  The second tier includes the Sea Symphony, War Requiem, Mahler's 8th and the Mozart Requiem, and then the third tier a number of other works, headed by the Haydn Seasons, the Berlioz Requiem and Bernstein's Chichester Psalms.  As we will see shortly most of these works showed up in the new survey with the exception of three major works that we have not performed for a long time and which were highly appreciated by the chorus:

• Vaughan Williams - A Sea Symphony
• Mahler - Symphony #8 ("Symphony of a Thousand")
• Haydn - The Seasons

The 2009 Survey - Methodology

This survey, carried out on the Web, had two parts:  chorus members were presented twice with a list of 41 works, which included most of the works performed in the past five years plus a couple that I thought might be special.  They were asked first to rank their top ten favorites -- that is, the works they would most like to do again, and then to rank the quality of each work on which they had an opinion on a scale from "Excellent" to "Awful".  Responders were also given a free-form comments field in which to list other works of interest, either good or bad.  As it turned out these comments indicated that I really should have included some of the composite performances like "Opera Choruses", "Video Live" and "Pops".  Oh well -- live and learn.

There were 24 responses to the survey, which doesn't sound like a lot, but that's actually a quite a bit of information -- and probably a pretty respectable response rate as surveys go. [From Wikipedia: "Holbrook et al. (2005) assessed whether lower response rates are associated with less ... [accuracy]. By examining the results of 81 national surveys with response rates varying from 5 percent to 54 percent, they found that surveys with much lower response rates were only minimally less accurate."] I reduced rankings and ratings to numerical scores that are shown here in the plots to the left and right.  In both cases the numerical scores are normalized so that a work would score 100 if it were ranked number one by everyone or 100 if it were rated as "Excellent" by everyone.  There was a very mild weighting applied to the rankings of the older works to try to compensate for the fact that less people are familiar with them (and we will see below there is some empirical evidence that this weighting was reasonably effective).  You will note on the ratings plot that there are some negative ratings.  This occurs because the rating categories were assigned both positive and negative values. Specifically, the categories and weightings were: Excellent (100), Very Good (60), Good (20), OK (-20), Not so Good (-60), Awful (-100).  One might argue that "OK" is not negative, but looking at the individual surveys shows that most people rated most works good or better.  When you spend as much time and energy as we do on these works, "OK" is not really what one wants, I think.

Analysis of Ranking and Rating

The first comment I would like to make is that there is a tremendous diversity in taste represented in the chorus.  One way of seeing this is to note that the highest composite ranking is 54 (for Carmina Burana), and it turns out that only two people ranked Carmina Burana #1.  In fact, the work with the highest number of #1's (five) is the Dvorak Stabat Mater, but of course the 2's and the 3's add up, too -- and the overall result has Carmina and Verdi on the top of the rankings.  Looking at the surveys individually bears this out -- many of the surveys looked rather similar, but some of them are real different.  The two components of the survey, ranking and rating, have different character as well.  The forced ranking aspect of the rankings (the Web interface would only allow one 1, one 2, etc) tends to maximize the differentiation between the works.  And, of course, the rankings become essentially meaningless for works that are only mentioned once or twice -- or not at all.  The ratings are more stable and useful for the entire range of works, but possibly less expressive. In the case of ratings there is often pretty good agreement.  For example, of those that ranked the Brahms German Requiem 17 said it was "Excellent", 5 "Very Good", for a total score of 90.  Similarly, the Verdi Requiem had 17 "Excellent", 4 "Very Good" and 1 "Good", for an insignificantly different score of 89. 

So how do the works stack up overall?  Ranking and rating give different pictures, but the most popular works are pretty clear in both:  The "big three" (Brahms, Verdi and Orff) from 1996 are still there, but the Dvorak Stabat Mater (audio clip here) is now right up there with them, indicating a "big four".  The order of the four is different in rankings and ratings, and I have combined them, considering all factors, into the summarized order Brahms, Verdi, Orff, Dvorak.  By any measures, however, all these works scored very high and the exact ordering is by no means obvious.

After the big four, as with the 1996 survey, the situation becomes more complex.  Again the rankings and ratings give different pictures, and I have tentatively combined them, as shown in the summary above, in the order Mozart Requiem, Beethoven's Ninth, Berlioz Requiem, Beethoven Missa Solemnis, Handel's Messiah and Mahler's Resurrection Symphony.  This ordering is even less obvious than that of the top group.  Other works that showed up in one or another lists include the Schubert and Bruckner Masses, Vivaldi Gloria and Bach Magnificat, and the Bernstein Chichester Psalms, all of which we performed quite recently, and Mendelssohn's St. Paul and Vaughan Williams Dona Nobis Pacem.

Finally, works mentioned more than once in the comments (positively unless noted) included: Faure Requiem, Adams Harmonium (2 "Awful", 1 "Excellent"), the Bach B Minor Mass and St. John Passion, Vaughan Williams Sea Symphony and Haydn Seasons, most of which are consistent with the 1996 survey.  The most mentioned of these works is the Faure Requiem, last performed in 1998 (after the old survey) under Eschenbach.

Twentieth Century Music

The survey results for modern music are particularly complex, reflecting strong differences of opinion and perhaps even ambivalence.  The only rankings that were, in total, negative, were for the Sierra Missa Latina and the Adams Transmigrations of Souls, but in both cases there were also a number of people who rated them Excellent or Very Good and ranked them among their favorites. The other modern work that rated quite low, the Shostakovich "Babi Yar" Symphony, however, did not exhibit quite so much range of opinion.  In this case no one rated it either "Awful" or "Excellent", and only one person included it in the rankings as a favorite.  On the other hand, as mentioned above, the Adams Harmonium performed in 1999 was mentioned as both Awful and Excellent (but more the former).  Leaving aside Carmina Burana, which of course is indeed a twentieth century composition, modern music did not fare very well on the whole.  Works that had a substantial level of appreciation (although in all cases there was quite a range of opinions) include:

• Bernstein - Chichester Psalms
• Vaughan Williams - Sea Symphony
• Britten - War Requiem
• Vaughan Williams - Dona Nobis Pacem
• Debussy - Nocturns
• Stravinsky - Symphony of Psalms
• Prokofiev - Alexander Nevsky
• Borodin - Polyvetsian Dances

Note that I have somewhat arbitrarily included the Sea Symphony in this list in approximately the location it held in the 1996 survey because this ranking at the time was so high. 

Comparison of Ranking and Rating

The fact that this survey has two somewhat independent measures of the same thing (ranking and rating) invites comparisons between the two results.  The graph on the left shows such a comparison, obtained by subtracting the rating placement from the ranking.  Let's make that clear by example:  Camina Burana ranked 1 but was rated 4 -- so the comparison is +3.  The German Requiem, on the other hand, was ranked 2 but rated 1 -- so its comparison is -1.  Put in more words, if a work has a negative comparison (like the Berlioz Requiem or the Schubert Mass) it means that a lot of people thought it was a high quality work, but for some reason didn't feel it was one of their favorites.  (That's overstating it, of course, but that's the idea).  Works that seem to fit that discription, from this diagnostic, include the Vivaldi Gloria, Schubert Mass and Bernstein Chichester Psalms.  Why have I not (yet) mentioned the Berlioz Requiem in this list?  Well, it's because it's possible that in the case of the Berlioz Requiem there is another thing happening.  This was a performance from quite a while ago.  It takes a little thinking (and knowledge of how the data analysis actually works), but the ranking is sensitive to how many people are familiar with the work whereas the rating is, at least to first approximation, not.  So if a work was ranked low simply because not enough people remembered it, but those people agreed that it was of high quality, it would show up as negative on this plot. Is that the case with the Berlioz Requiem?  Well, the Berlioz Requiem was last performed in 2002, and this difference is -5, but the Dvorak Stabat Mater was performed even earlier (in 2000), and the difference there is a mere -1.  I conclude from this that the time factor was compensated for reasonably accurately, and that the negative difference for the Berlioz Requiem is not due to that factor.  This is just my personal opinion, but for what it is worth I think it is relevant that the last performance of the Berlioz Requiem was in Mexico City, and the last time we performed it with the Houston Symphony was in 1993, beyond the memory of most chorus members.  I don't want to diss the Mexico City orchestra -- but it's not the Houston Symphony and the Berlioz Requiem is a very symphonic work with many very special, extremely French, effects.  OK -- that's just my opinion, but that's what I think that negative difference is about.  My cherished memory of the Berlioz Requiem is from 1993 with Shaw and Carl Dent (audio clip here), which is the last time the Houston Symphony performed this work.

More generally, looking at this difference plot shows that the differences for the works in the top tier are generally small, and they only get really big down near the bottom of the rankings where very few people were including the works in their list of favorites.  In general the differences are really fairly small, indicating that the two approaches to analysing the opinions of the chorus yielded reasonably consistent results.  The difference that I find most difficult to understand is the -17 for the Vivaldi Gloria.  Most people thought it was a pretty good piece of music, but for some reason it just didn't figure in their favorites.  But the Bach Magnificat - similar era and the same concert series -- scored a negligable -3 value for difference.  I don't get it.

Long Time Between Performances

As you can see from the HSC History, some the works mentioned above as favorites have not been performed in a long time by the Houston Symphony.  This includes

Work	Performed by Houston Symphony	Years
Berlioz Requiem	1993 (Shaw, Carl Dent) [1999 and 2002 not Houston Symphony]	17
Vaughan Williams Sea Symphony	1992 (Seaman)	18
Bernstein Chichester Psalms	2008 (Graf) - 1983 (Comissiona) [1994 not Houston Symphony]	25
Haydn Seasons	1994 (McGegan)	16