Clinical cut-offs for hip- and knee arthroplasty outcome - minimal clinically important improvement (MCII) and patient acceptable symptom state (PASS) of patient-reported outcome measures (PROM)

The aims of the present study were twofold. First, to add an extensive report of MCII and PASS estimates following hip- and knee arthroplasty in a setting with specialized orthopedic surgery health care in a community characterized by equal rights and high trust, based on real-life high-quality data. Second, by applying a multitude of anchors and reporting and discussing results from multiple estimation methods, to highlight important considerations to be made when estimating cut-offs and when reviewing results from other studies defining successful surgical outcome following hip- and knee arthroplasty.

We applied six different anchors with 5-point rating scale response categories (Tables 1 and 2). Anchors like these have previously been examined in the same patient population [1] or are recommended used as best practice [26]. Anchors A2 and A4 represent RTAs, where anchor A2 is joint specific and anchor A4 generic, i.e., relates to general health. Anchors A3 and A5 are current state anchors, generic and joint specific (pain), respectively. Anchors A1 and A6 both relate to the operation results (how would you describe the result/how satisfied are you with the operation), and can be viewed as implicit transition anchors, however maybe with a higher influence of current state than questions explicitly asking the patient to compare with before operation. Responses for the anchor questions were collected at 6 weeks and one year postoperative. We found the best suited anchor based on anchor focus and type (corresponding to the PROM subscale; generic/specific, and current state/retrospective transition) while also taking an adequate anchor-PROM correlation into account. Anchor type was our primary concern (current state anchors for PASS and retrospective transition anchors for MCII), then we looked at anchor focus (corresponding domains, generic/specific), monotonicity in the relationship with PROM scores/changes, and correlation. SDs and reliability estimates used in the MDC estimates, as well as mean PROM scores/changes in PROM scores in relation to anchor categories are shown in Tables 3 and 4 and as supplementary Tables S1-S2, respectively.

Change in PROM was calculated by subtracting the preoperative score from the postoperative score, with a higher change score indicating greater improvement. Assessment of the association of the PROM/change in PROM with anchor questions (see Anchors and anchor types above) was performed using Spearman correlation, and with the dichotomized anchor (acceptable (yes/no) state or change; see details later in this section) using the area under Receiver Operating Characteristics (ROC) curve. Furthermore, mean PROM scores/changes in PROM scores within anchor categories were assessed to consider monotonicity in the relationships. PASS and MCII were estimated by comparison with selected anchors, using the following ROC-based methods: (a) the lowest cut-off achieving 80% specificity for classifying cases as acceptable or not acceptable (specificity refers to the probability of being below the PROM cut-off for patients being deemed non-acceptable on the anchor), (b) the point on the ROC curve closest to intersecting with a -45-degree tangent, (c) the cut-off yielding maximum accuracy, defined as the maximum sum of sensitivity and specificity (i.e., the highest Youden index), and (d) the cut-off corresponding to the smallest sum of squares of (1-sensitivity) and (1-specificity), in accordance with Pythagoras’ theorem [27]; and the following non-ROC-based methods: (e) the mean score/change score among those with anchor responses in the lowest category deemed acceptable, and (f) the 75th percentile method, which for PROMs where higher scores indicate better outcomes, corresponds to the 25th percentile score/change score among the same group as under e). The minimum detectable change at a 95% confidence level (MDC₉₅) for each PROM was estimated as MDC₉₅ = √ [2] * z * SEM, where z = 1.96 is the 97.5th percentile of the standard normal distribution, and SEM is the standard error of measurement, calculated as SEM = SD * √(1-r), where SD is the standard deviation and r is Cronbach’s alpha of the preoperative PROM scores [28]. Confidence intervals for PASS and MCII estimates were determined through percentile intervals from non-parametric bootstrapping based on B = 1000 resamples. For MCII estimation using non-ROC approaches, only patients with minimal improvement on the anchor were included, while for PASS estimation, patients with good and better anchor states were included. For ROC approaches, dichotomized anchors including all anchor categories deemed acceptable vs. those deemed non-acceptable were used for both MCII and PASS estimation (see Tables 1 and 2 for details of the classification for each anchor). All data analyses were conducted using Stata (StataSE 17.0, StataCorp, 4905 Lakeway Drive, College Station, Texas 77845 USA), employing functions spearman, roctab, rocmic [29], logistic, lsens, and bootstrap.

All methods were performed in accordance with the relevant guidelines and regulations.

The study was submitted for registration to the Regional Committees for Medical and Health Research Ethics (REC), was there considered a follow-up quality-control study under the jurisdiction of the local data protection officer and has been granted an exemption from requiring ethics approval (REC West 2018/268). Due to the use of no personally identifiable data, i.e., only anonymous data reports from patient journals, the study was exempted informed written consent from patients and was approved by the local data protection officer (journal number 6/2018).

It is not possible to define what is clinically important by using distribution-based methods [40]. To be able to estimate cut-offs based on the patients’ perspective, anchor items are imperative and anchor-based methods should be used, with distribution-based approaches providing supportive information [1, 3]. The mean change approach and the 75th percentile approach are common and simple anchor-based approaches, but by definition the mean change approach misclassifies about half of the patients that are deemed just acceptable on the anchor [34], and the 75th percentile approach misclassifies a quarter of the same group. All ROC approaches, except the 80% specificity approach, value sensitivity and specificity equally [27]. Among methods that were not tested in the present study, item response theory-based methods may represent an improvement being less sensitive to anchor distributions than ROC-based methods, but has currently only been tested in situations with a true gold standard [41].

An adequate anchor-PROM correlation is important, but recommendations vary from at least 0.30, between 0.40 and 0.70, to over 0.50 in absolute value [8, 36, 42]. Using a poorly related anchor can lead to cut-offs that are contaminated with noise and are of little or no use [36]. It is strongly recommended to use multiple anchors [43]. Given this, presentation of anchor-PROM correlations is important as context for evaluating published cut-off estimates [44]. Only one of our correlations fall below 0.30 in absolute value (MCII for HOOS Symptoms) however none of the correlations for change (MCII) are above 0.50. The anchor-PROM correlations are all within 0.40–0.70, for the PASS estimates, except for KOOS Pain (-0.78). It is uncertain if the higher correlations for non-RTAs in Tables S3 and S4 were due to domain (we have no retrospective transition anchor on pain), recall bias/present state bias or due to other reasons.

Our MDC₉₅ estimates were based on preoperative standard deviation and Cronbach alpha, as the preoperative samples were biggest (n = 674–686 for HOOS, n = 434–446 for KOOS). Ideally, it should have been based on test-retest reliability instead of internal consistency, however such data were not available to us. The test-retest reliability may be poorer than the internal consistency, which may lead to a higher MDC. Our MDC for HOOS Pain is identical to previous findings and almost identical (20 vs. 21) for HOOS QoL [1]. The MCII cut-off estimate from the 75th percentile approach was lower than the MDC for HOOS Symptoms, HOOS QoL and all KOOS subscales except KOOS ADL. For these, the difference was highest for KOOS Symptoms – this was also the MCII estimate that was based on an anchor with correlation < 0.30 with the PROM. An MCII lower than the MDC of an instrument may raise questions regarding the PROM’s utility for detecting the smallest clinically relevant changes in individual patients, however in our case it must also be seen in light of uncertainty regarding the MCII estimates and the wide confidence intervals. In clinical practice, where identifying patients who did not improve for further follow-up is more important, using a higher cut-off (> MDC) would tie up more resources but may ensure that fewer patients with significant needs are overlooked.

Recall bias may be a problem with RTAs [8], as well as present state bias [45]. RTAs should have an equal correlation to preoperative- as to postoperative PROMs but may often have a stronger correlation to postoperative PROMs. Repeated current-state anchors may therefore be better than RTAs in this respect. Repeated current-state anchors give information for the specified time point (which can be used for PASS estimation), but also information on absolute change when examining shift in answer category over time, and thus may be used for MCII estimation as well [6]. RTAs on the other hand, may seldom be used for PASS estimation, as these anchors give little information on present status. We were not able to use repeated current-state anchors, due to missing data on preoperative anchors. Using different anchor types and using both RTAs and repeated current-state anchors may pose a solution for minimizing the effect of recall bias.

What constitutes a minimal clinically relevant improvement depends on the nature of the intervention. A small change may be deemed clinically relevant if the intervention has low cost and low risk. The setting of the intervention, i.e., elective or following trauma, as well as patient expectations, may also influence what qualifies as a minimal clinically relevant change. Condition specific outcome and general health outcomes may also be perceived differently. Arthroplasty surgery has high cost and substantial risks and is furthermore well-known to be effective. It is therefore not surprising that only a substantial improvement is considered clinically relevant for these patients. De Vet et al. argue for using «much improved» as a cut-off for acceptable outcome on transition anchors [46]. On the other hand, basing the MCII on large improvements on the anchor may reduce the clinical utility of the cut-off, and it would perhaps no longer be the minimal clinically important change. We opted to use those reporting a small positive change on the anchor as basis for estimation of MCII for the non-ROC approaches. On the other hand, corresponding estimation of PASS, where an acceptable symptom state is the focus (i.e., not “minimal” acceptable symptom state), was performed with all patients reporting an acceptable state on the anchor.

Several methodological limitations should be taken into consideration when interpreting the results of the present study. The distribution was skewed, and the dichotomized anchor questions were unbalanced, which together with the limited sample sizes resulted in high degree of uncertainty in our ROC-based cut-off estimates. We did not have test-retest data, thus MDC estimates may be less accurate. Most of the PROM-anchor correlations for the anchors used for MCII were on the low side. We did not have pre-operative anchors, thus had to use retrospective change anchors for MCII. Few patients (< 100 in each group) answered the preoperative EQ-5D questionnaire, these data as well as MCII estimates for EQ-5D are not presented due to low sample size. Misclassification was assessed in-sample. Strengths: The response rates for all PROMs were high (≥ 80%). High response rates ensure generalizability and minimize selection bias, and a response rate ≥ 80% is usually considered to be adequate and sufficiently representative of the sample studied [47]. We used well-validated PROMs for the patient group, with validated feasibility [48]. Standardized format paper PROMs were used. All returned questionnaire forms were scanned electronically using a validated automated form-processing technique [49]. Although not a limitation specifically for our study, cut-off estimates should not be used for between-group comparisons of mean outcome scores, but for defining individual successful surgical outcome following hip- and knee arthroplasty [50].