In addition to JJJ's answer, the index is usually averaged across a set of votes of interest; for example across a legislative session.
While the Rice-index is calculated for each vote, most often
the average value of this index is of interest.
Cf. Hug (2006).
It's also not actually the case that the Rice index cannot be used if Abstain is an option (except in one extreme case as JJJ correctly notes below in a comment). The paper in question (Hix et al.) notes that:
However, the problem with the Rice index in the European
Parliament is that MEPs have three voting options: Yes, No and Abstain. Attina
consequently developed a cohesion measure specifically for the European Parliament,
where the highest voting option minus the sum of the second and third options was divided
by the sum of all three options. But, the Attina index can produce negative scores on
individual votes, since a party split equally between all three voting options produces a
cohesion score on the Attina index of -0.333.
As a result, by enabling all three voting choices to be taken into account, and by
producing cohesion scores on a scale from 0 to 1, our Agreement Index is an alternative
to the Rice and Attina indices for measuring party cohesion in the European Parliament
(or in any parliament with three voting options). Nevertheless, the cohesion scores
produced by our index can be compared to scores produced by these other two indices.
Our results correlate perfectly with the Attina scores, as our index is simply a rescaling
of the scores from 0 to 1, and correlate at the 0.98 level with the Rice scores for the same
data on the European Parliament. Note, however, that the difference between our scores
and the Rice scores are higher for parties that tend to Abstain as a block (for example, when
parties Abstain strategically).
So in a practical context (EU parliament), the Rice score was usually well-correlated (0.98) with the Abstention-sensitive measures. As for the formula for the latter:
Let M = max{Y, N, A}
and let N = Y+N+A
, then the Hix index is
(M - 1/2 (N - M)) / N = (3M - N) / 2N
which is zero if the votes are equally split (1/3) among Y, N, A.
But all three (Rice, Attina ~ Hix) measures inflate the decohesion score of small parties.
Another limitation is that these (per-party) indexes cannot be computed if the vote is secret, since per-party breakdowns of Y/N/A are not available then. This is actually the case for some types of votes in some European national parliaments. Another criticism related to this latter point is that in systems with mixed open and secret votes
it has often been
questioned if roll call behaviour is an appropriate indicator for the identification
of party cohesion since open votes are often asked for in situations
where party unity is urgently required. In other words, roll call analysis is
probably a better indicator for party discipline than party cohesion. Open
votes actually lead to higher party unity than secret votes because deviant
behaviour is openly manifested. Analytically, this leads to the impression
that party cohesion is higher than it really is. Because of this selection bias,
roll call analysis is not a suitable means for the analysis of party cohesion in
parliamentary systems (Carrubba et al. 2006; Hug 2010).