Beyond Binary Classification: A Comprehensive Survival Analysis and Calibration Assessment of the Heart Failure Clinical Records Datase

Full text

Text extracted from the article PDF for reading and search. Tables, figures and equations are in the PDF, which remains the authoritative version.

1. Introduction

Cardiovascular diseases (CVDs) remain the leading cause of death worldwide, accounting for an estimated 19.8 million deaths in 2022, approximately 32% of all global deaths[1]. Heart failure (HF), a major contributor to this burden, is a complex clinical syndrome in which the heart cannot pump sufficient blood to meet the body's metabolic demands, and predicting patient survival from routinely collected clinical variables remains a central goal of clinical decision support research. The Heart Failure Clinical Records dataset, originally collected from 299 patients treated at the Institute of Cardiology and Allied Hospital in Faisalabad, Pakistan, in 2015, was first analyzed using Cox regression and a nomogram [2], and was later popularized as a machine learning benchmark, with serum creatinine and ejection fraction alone reported as sufficient to predict survival with reasonable accuracy [3].

Since then, the dataset has been reanalyzed in a very large number of studies, the majority of which frame the problem as static binary classification: the time variable, representing each patient's follow-up duration, is either discarded or used as an ordinary numerical feature, and model performance is reported using accuracy, F1-score, or a single receiver operating characteristic AUC value. Representative examples include SMOTE-based class balancing applied to improve classification metrics [4, 5], a transfer-learning-based feature engineering approach proposed for the same classification task [6], and post hoc explainability techniques applied to both a classification model and, more unusually, a survival gradient boosting model, with the latter reporting a concordance index of 0.714, comparable to the unstratified Cox result obtained in the present study [7]. None of these studies, however, formally tested the proportional hazards assumption or jointly evaluated discrimination and calibration over the follow-up period.

A smaller body of work has revisited the dataset using genuine survival-analytic methods. Most recently, one study examined how follow-up adequacy, quantified via the Person-Time Follow-up Rate, affects the comparative performance of Cox proportional hazards regression and Random Survival Forest (RSF) on this same cohort [8]. However, systematic diagnostic testing of the proportional hazards assumption, joint evaluation of discrimination and calibration over time, and subgroup-level (fairness) performance auditing remain largely absent from the literature on this dataset.

The present study addresses these gaps. We (i) model mortality explicitly as a time-to-event outcome using Kaplan-Meier estimation, Cox PH regression, and RSF; (ii) formally test the proportional hazards assumption and quantify the consequence of its violation; (iii) evaluate model performance jointly through time-dependent discrimination (AUC) and calibration (Brier score); and (iv) assess whether predictive performance is consistent across clinically relevant patient subgroups (sex, age, diabetes status).

2. Materials and Methods

2.1. Dataset

We used the publicly available Heart Failure Clinical Records dataset [2, 3], comprising 299 patients aged 40 years or older with left ventricular systolic dysfunction (NYHA class III–IV), followed for a mean of 130.3 days. The dataset contains 11 clinical/demographic covariates (age, anaemia, creatinine phosphokinase, diabetes, ejection fraction, high blood pressure, platelets, serum creatinine, serum sodium, sex, and smoking status) together with the follow-up duration (time, in days) and a binary event indicator (DEATH_EVENT). Of the 299 patients, 96 (32.1%) died during follow-up and 203 (67.9%) were right-censored at their last observation. As detailed in Section 2.2, ejection fraction was subsequently categorized into three groups (≤30%, 30–45%, and >45%) based on the dataset's observed distribution [9].

2.2. Survival Analysis Framework

Unlike the majority of prior work on this dataset, we treated time and DEATH_EVENT jointly as a censored time-to-event outcome rather than discarding or dichotomizing the follow-up duration. Kaplan-Meier estimation [10] was used to characterize overall and subgroup-specific survival curves, with between-group differences assessed via the log-rank test (and the multivariate log-rank test for the three-level ejection-fraction stratum).

A multivariable Cox proportional hazards model [11] was fitted using all 11 covariates. The proportional hazards assumption was formally tested for each covariate using both Kaplan-Meier-based and rank-based statistics derived from scaled Schoenfeld residuals[12], implemented via lifelines' check_assumptions routine. Where the assumption was violated, the offending covariate was re-expressed as a categorical stratification variable and the model was refit. A Random Survival Forest [13], building on the random forest algorithm[14] (500 trees, minimum split size 10, minimum leaf size 5, sqrt feature subsampling), was trained on a 75/25 train–test split (stratified by event status, random_state = 42) using the same 11 covariates. Model discrimination was quantified using Harrell's concordance index (C-index)[15] on the held-out test set, computed identically for both the Cox and RSF models to ensure a fair comparison. Permutation importance (15 repeats) was used to rank covariate contributions in the RSF model.

2.3. Time-dependent Discrimination and Calibration

Beyond a single aggregate C-index, we evaluated model performance across the follow-up period using cumulative/dynamic time-dependent AUC at fixed time points (30–270 days, 20-day increments), and assessed calibration using the time-dependent Brier score and its integrated summary (Integrated Brier Score, IBS) [16], computed on RSF-predicted survival probabilities at the same time points. Calibration was evaluated as a distinct property from discrimination, following the recommendation that predictive models be assessed on both dimensions rather than discrimination alone [17].

2.4. Subgroup (Fairness) Evaluation

To assess whether model performance was consistent across clinically relevant patient groups, we computed the C-index separately within the held-out test set for sex (female vs. male), age (below vs. at/above the sample median), and diabetes status (absent vs. present).

2.5. Software

All analyses were performed in Python 3 using lifelines [18] for Kaplan-Meier estimation, Cox PH regression, and proportional hazards diagnostics; scikit-survival [18] for Random Survival Forest, the concordance index, time-dependent AUC, and Brier score; and scikit-learn [19] for train/test splitting and permutation importance. Data handling, numerical computation, and visualization relied on pandas [20], NumPy [21], matplotlib [22], and seaborn [23].

3. Results

3.1. Cohort Characteristics

The cohort comprised 299 patients, of whom 96 (32.1%) experienced the primary endpoint (death) during a mean follow-up of 130.3 days, while 203 (67.9%) were right-censored (Figure 1). Deaths were concentrated in the earlier part of the follow-up period, consistent with the well-documented high early-mortality risk in heart failure. Univariate comparisons showed clearly separated distributions for ejection fraction (lower among non-survivors) and serum creatinine (higher among non-survivors), with weaker or negligible separation for platelets and creatinine phosphokinase (Figure 2).

Fig 1: Follow-up time distribution by event status (left) and overall event distribution (right).

Fig 2: Distribution of six continuous clinical variables stratified by death event (0 = survived, 1 = died).

3.2. Kaplan-Meier Survival Analysis

The overall Kaplan-Meier curve showed a steep early decline in survival probability, falling from 100% to approximately 83% within the first 50 days, followed by a more gradual decline and a plateau near 58% after day 250 (Figure 3). Survival did not differ significantly by sex (log-rank p = 0.950; Figure 4), but was significantly lower in patients with hypertension (log-rank p = 0.036; Figure 5) and in older patients (age ≥ median; log-rank p = 0.003).

Fig 3: Kaplan-Meier overall survival curve (shaded region: 95% confidence interval).

Fig 4: Kaplan-Meier survival curves by sex (log-rank p = 0.950).

Fig 5: Kaplan-Meier survival curves by hypertension status (log-rank p = 0.036).

3.3. Cox Proportional Hazards Model and Assumption

Testing In the initial (unstratified) multivariable Cox model fitted on the full dataset, age (HR = 1.05, p < 0.005), ejection fraction (HR = 0.95, p < 0.005), serum creatinine (HR = 1.38, p < 0.005), anaemia (HR = 1.58, p = 0.03), and high blood pressure (HR = 1.61, p = 0.03) were independently associated with mortality (overall concordance = 0.74). Formal proportional hazards testing revealed that ejection fraction violated the assumption (p = 0.013 for both the Kaplan-Meier-based and rank-based tests), while serum creatinine approached the significance threshold (p = 0.07 for the Kaplan-Meier-based test and p = 0.06 for the rank-based test. After stratifying the model by ejection fraction category (≤ 30%, 30–45%, > 45%) to resolve this violation, overall concordance decreased slightly to 0.71 and the covariate significance profile shifted: anaemia was no longer significant (p = 0.13), while high blood pressure (HR = 1.73, p = 0.01) and creatinine phosphokinase (p = 0.01) strengthened. Age and serum creatinine remained robust, significant predictors in both model specifications (Table 1, Figure 6).

3.4. Cox PH vs. Random Survival Forest

On an identical hold-out test set, RSF achieved higher discrimination than the (ejection-fraction-stratified) Cox model (C-index 0.775 vs. 0.716; Table 2), consistent with RSF's ability to capture the non-linear, time-varying effect of ejection fraction that violated the Cox model's core assumption. Permutation importance identified age, ejection fraction, and serum creatinine as the three dominant predictors in the RSF model, mirroring the Cox model's findings and results reported previously for this dataset [24] (Figure 7). Notably, high blood pressure, a significant predictor in the Cox model, ranked low in RSF importance, a discrepancy plausibly attributable to correlation with stronger predictors that reduces its marginal contribution within a tree-based ensemble.

Fig 7: Random Survival Forest permutation importance (mean decrease in C-index over 15 repeats).

3.5. Time-dependent Discrimination and Calibration

RSF achieved a mean time-dependent AUC of 0.820 across the follow-up period, with discrimination weakest around day 90 (AUC = 0.735) and strongest beyond day 250 (AUC > 0.92; Figure 8). However, the integrated Brier score (IBS = 0.185) indicated only moderate overall calibration, and calibration deteriorated markedly at later time points: the Brier score rose from approximately 0.09 at day 30 to 0.53 at day 270 (Figure 9), the same interval in which time-dependent AUC was highest. This divergence indicates that strong rank-based discrimination does not guarantee reliable probability calibration, particularly in regions with a shrinking at-risk population.

Fig 8: Random Survival Forest time-dependent (cumulative/dynamic) AUC.

Fig 9: Random Survival Forest time-dependent Brier score (IBS = 0.185; lower values indicate better calibration).

Table 2: Summary of model discrimination and calibration metrics on the held-out test set. Model / Metric Value Cox PH: C-index (stratified by EF category) 0.716 (test set) Random Survival Forest: C-index 0.775 (test set) RSF: Mean time-dependent AUC 0.820 RSF: Integrated Brier Score (IBS) 0.185

3.6 Subgroup (Fairness) Performance

RSF performance was broadly consistent across sex, age, and diabetes subgroups within the test set (Table 3), with no subgroup showing a marked performance collapse. Combined with the absence of a significant sex-based survival difference (log-rank p = 0.950) and the low permutation importance of sex in the RSF model, these results suggest no strong evidence of systematic sex-based bias in this cohort. Given the small subgroup sample sizes (ranging from n = 24 in the diabetes subgroup to n = 51 in the no-diabetes subgroup; see Table 3), however, these point estimates should be interpreted cautiously, as their precision is limited and confidence intervals were not computed.

Table 3: Random Survival Forest concordance index by patient subgroup, held-out test set. Subgroup n (test set) C-index Female 25 0.801 Male 50 0.765 Younger (< median age) 37 0.727 Older (≥ median age) 38 0.688 No diabetes 51 0.790 Diabetes 24 0.761

4. Discussion

4.1. Principal Findings

This study reframed a dataset that has almost exclusively been analyzed as a binary classification problem as a time-to-event survival problem. Despite this different framing, both Cox PH and RSF converged on the same three dominant predictors (age, ejection fraction, and serum creatinine), corroborating earlier findings[2, 3] through an independent methodological lens. This convergence across three separate analytical approaches (Kaplan-Meier stratification, Cox regression, and RSF) strengthens confidence in these variables as genuine, robust prognostic markers in this cohort, rather than artifacts of a particular modeling choice. The additional risk factors identified (anaemia, hypertension, and impaired renal function as reflected by serum creatinine) are each independently well established in the broader heart failure literature: anaemia has been associated with increased mortality in a meta-analysis of over 150,000 patients [25], hypertension is a recognized precursor in the progression to heart failure [26], and impaired renal function, commonly assessed via serum creatinine, is a well-documented determinant of prognosis in heart failure through the cardiorenal syndrome [27]. The consistency between our data-driven findings and this established clinical literature lends external plausibility to the models beyond their internal statistical performance.

4.2. The Cost of Ignoring Proportional Hazards

Violations A central finding of this study is that ejection fraction violates the proportional hazards assumption, a diagnostic step that is essentially absent from the wider literature on this dataset. Correcting for this violation via stratification altered the model's covariate significance profile, most notably rendering anaemia non-significant after adjustment. This suggests that anaemia's apparent prognostic value in unadjusted Cox analyses may be, at least in part, confounded by the time-varying effect of ejection fraction. More broadly, it demonstrates that an acceptable aggregate concordance index does not guarantee that a Cox model's individual covariate effects are being correctly estimated.

4.3. Discrimination Is Not Calibration

A second key contribution is the demonstration that time-dependent AUC and the Brier score can diverge sharply within the same model. RSF's discrimination improved markedly beyond day 250 (AUC > 0.92), yet its calibration deteriorated in the same window (Brier score rising to 0.53), most plausibly reflecting the shrinking number of patients remaining at risk late in follow-up. This pattern is a concrete illustration of a broader, well-documented concern: calibration has been described as the “Achilles heel” of predictive analytics[28], since models with strong discriminative ability are routinely deployed and reported without any accompanying calibration assessment, even though poor calibration can directly translate into misleading risk estimates for individual patients. Because the vast majority of prior studies on this dataset report only accuracy, F1-score, or a single AUC value, this discrimination-calibration divergence would go entirely undetected under standard reporting practices. For models intended to inform clinical risk communication (where a calibrated probability of survival is arguably more actionable than a rank-ordering of risk), this finding argues for routine joint reporting of discrimination and calibration metrics, evaluated across the follow-up period rather than at a single time point.

4.4. Fairness Considerations

Subgroup analysis found no evidence of a systematic sex-based disparity in either survival (log-rank p = 0.950) or model discrimination, and sex ranked lowest among covariates in RSF permutation importance. The modest differences observed in subgroup C-index (e.g., female 0.801 vs. male 0.765) are more plausibly attributable to sampling variance in small subgroups (n = 24–51 in the test set) than to genuine model bias, though this cannot be confirmed without bootstrapped confidence intervals, an important direction for future fairness auditing. Such auditing is not a purely theoretical concern: a widely deployed healthcare risk-prediction algorithm has been shown to exhibit substantial racial bias that was invisible under its original, aggregate evaluation and only became apparent through explicit subgroup analysis [29]. This precedent underscores the importance of routine subgroup evaluation for any clinical prediction model, particularly as models trained on small, single-center datasets such as this one are increasingly proposed for broader deployment.

4.5. Limitations

This study inherits the well-documented limitations of the underlying dataset: a single-center cohort of 299 patients collected in 2015 at one hospital in Faisalabad, Pakistan, with a documented Person-Time Follow-up Rate well below recommended thresholds[8] [24], which has itself been shown to affect the relative performance of Cox and RSF models. A sample of this size falls well short of the minimum recommended for developing a stable multivariable prediction model with 11 candidate predictors[30], which likely contributes to the wide confidence intervals observed for several covariates and to the instability of subgroup estimates. The relatively small sample size (and the even smaller size of the test-set subgroups analyzed for fairness) limits the precision and generalizability of the time-dependent and subgroup-level estimates reported here. External validation on an independent, multi-center heart failure cohort is required before any of these findings could inform clinical practice, and future reporting on this dataset would benefit from adherence to established reporting guidelines for prediction model studies [31]. Future work could also extend this comparison to deep-learning-based survival models such as DeepSurv [32], which were outside the scope of the present study but may further improve on the non-linear effects captured here by RSF.

5. Conclusion

Beyond confirming ejection fraction, serum creatinine, and age as central predictors of mortality in this frequently reused benchmark dataset, this study shows that standard binary classification pipelines can obscure assumption violations, calibration failures, and covariate confounding that only become visible under an explicit survival-analytic framework. We recommend that future work on the Heart Failure Clinical Records dataset, and on similarly small, single-center clinical datasets more broadly, report proportional hazards diagnostics, time-dependent calibration, and subgroup performance alongside conventional discrimination metrics.