Analysis of Rating Experiments
Dprime analysis of ratings is treated by Macmillan & Creelman (1991) in Ch. 3. Recall that a rating experiment uses one stimulus per trial with a response set greater than two (the rating scale). The responses can be either only along the scale, or categorical plus a confidence rating  but the analysis of these is the same.
M&C's example is an experiment with a "yesno" type response plus a 3point confidence rating; these responses are treated as a single 6point scale. So there are 6 response columns rather than 2, and the row total is the total of all 6 columns. (Note how the direction of the confidence scale is reversed across the yesno responses, in order to give a single rating scale.) For each cell, the proportion of responses is calculated from #responses/total#responses just as before. The trick is getting H and FA from 6 columns, and the answer is to get several Hs and FAs. Cumulative probabilities are calculated by summing responses from left to right. That is, the value (proportion) in each cell is replaced by the sum of that cell plus all to its left. The values in the first column are unchanged by this operation, and the values in the last column are now all 1.0. The value in each cell is now a Hit rate for the division of the responses into 2 response categories at that column; the division at each successive column represents a different "decision rule" that the subject could have applied to a 2way choice.
M&C's example (partial):
Raw responses (Table 3.1 top):
responses: 
YES 


NO 


total 

3 
2 
1 
1 
2 
3 

YES 
49 
94 
75 
60 
75 
22 
375 
NO 
8 
37 
45 
60 
113 
113 
375 
Treating only the yes/no responses categorically
(underneath Table 3.1):
responses: 
YES (total) 
NO (total) 
total 
YES 
218 
157 
375 
NO 
90 
285 
375 
from which H and FA can be calculated:
H = 218/375 = .58; FA = 90/375 = .24
and from those, d' can be calculated, as
0.91
Now the proportions across all the columns
(Table 3.2 top):
responses: 
YES 


NO 


total 

3 
2 
1 
1 
2 
3 

YES 
.131 
.251 
.200 
.160 
.200 
.059 
1.00 
NO 
.021 
.099 
.120 
.160 
.301 
.301 
1.00 
And the cumulative proportions (H and
FA) (Table 3.3 top):
responses: 
YES 


NO 



3 
2 
1 
1 
2 
3 
YES = H 
.131 
.382 
.582 
.742 
.942 
1.00 
NO = FA 
.021 
.120 
.240 
.400 
.701 
1.00 
Then d' can be calculated for each
H,FA pair (for each column except the last). M&C show the z scores
and d' (Table 3.4 top):
responses: 
YES 


NO 


3 
2 
1 
1 
2 
yes 
1.121 
0.301 
0.207 
0.649 
1.573 
no 
2.037 
1.175 
0.706 
0.253 
0.527 
d' 
0.916 
0.874 
0.913 
0.902 
1.046 
The d' values are quite similar across decision rules, even though the Hit rates vary.
Rietveld & van Hout (1993), Statistical Techniques for the Study of Language and Language Behaviour, Ch. 5 is more concerned with the validity of ratings, in terms of:
and these can be interrater, or intrarater. So your first step will be to decide what you want to check about your rating responses.
Reliability looks at whether ratings covary, which is independent of whether they use the same absolute values. Rating scores are seen as a combination of the "true" scores plus "error", and the reliability R is the ratio of the "true" variance across scores to the error variance.
Don't just average all the judges as an estimate of the true score! Means = true scores only for infinite judges. Instead, first see how reliable your judges' ratings are.
Example in Rietveld & van Hout, Table 5.1 p. 191, in file rietveld_ex.sav, 10 items rated by 4 judges on a 10point scale. Rows are the rated objects (the stimuli), columns are the raters.
Because we compare the "true" variance, due to the objects, to the variance due to the raters (and potentially, any object x rater interaction), this is like a 2way ANOVA. Indeed, the SPSS Reliability procedure can be used to do an ANOVA.
The SPSS procedure assumes that there are multiple raters and the Raters factor is fixed (more on this below); that said, it works as follows to calculate Cronbach's Alpha:
file rietveld_ex
Analyze  Scale  Reliability Analysis(leave
"model" at "Alpha")  Statistics  ANOVA Table=Ftest
This gives Rietveld & van Hout's Table 5.7 on p. 202. "Alpha" = Cronbach's
Alpha, reliability coefficient for raters as fixed factor.
Crucially, with multiple raters, raters can be treated as either a fixed or a random factor, and these situations require different calculations of the reliability coefficient R and tell you different things about your raters' reliability:
Since the objects (stimuli) are always treated as random, the model is either mixed (raters are fixed) or random (raters are random).
The MS terms in the ANOVA output from Reliability can be used to calculate other versions of R for these other situations. Rietveld & van Hout's Table 5.8 shows 4 reliability coefficients, for: raters fixed vs. random, and a single rater vs. of a group of raters. You can also use the MS terms to do an F test to test if R > 0, where the F ratio is formed from the SPSS MS"between people" over MS"residual", and df for the numerator (n1) and df for the denominator (n1)(k1); see section 5.3.2 for more.
But SPSS Reliability also will compute the "IntraClass Correlation Coefficient". Rietveld & van Hout mentions ICC as another name for R, but does not mention the SPSS procedure. In AnalyzeScaleReliability AnalysisStatisticsIntraclass correlation coefficient, there is a choice of three models (single rater, multiple raters random, multiple raters fixed), so this would seem to be a way to go beyond Cronbach's alpha.
Here's another important design for us:
Comparing reliability coefficients (are
they significantly different?)  e.g. raters from 2 language backgrounds
rate the same materials:
Tested with the M statistic, see section
5.3.3.
¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤
In contrast, Agreement looks at
how similar the absolute levels of the ratings are, rather than whether they
covary. Correlation coefficient is generally not a good tool, as it's
too sensitive to covariance and not sensitive enough to absolute values.
Instead, read Rietveld & van
Hout's section 5.4 and compare T,
Kendall's W, and Cohen's Kappa re your own design.
Prepared by Pat Keating
in Spring 2004
Return to the UCLA
Phonetics Lab statistics page.