ReXrank is an open-source leaderboard for AI-powered radiology report generation from chest x-ray images. We're setting a new standard in healthcare AI by providing a comprehensive, objective evaluation framework for cutting-edge models. Our mission is to accelerate progress in this critical field by fostering healthy competition and collaboration among researchers, clinicians, and AI enthusiasts. Using diverse datasets like MIMIC-CXR, IU-Xray, and CheXpert Plus, ReXrank offers a robust benchmarking system that evolves with clinical needs and technological advancements. Our leaderboard showcases top-performing models, driving innovation that could transform patient care and streamline medical workflows.
Join us in shaping the future of AI-assisted radiology. Develop your models, submit your results, and see how you stack up against the best in the field. Together, we can push the boundaries of what's possible in medical imaging and report generation.
To evaluate your models, we made available the evaluation script we will use for official evaluation, along with a sample prediction file that the script will take as input.
To run the evaluation, use
python evaluate.py <path_to_data> <path_to_predictions>
.
Once you have a built a model that works to your expectations on the MIMIC-CXR test set, you submit it to get official scores on our Private test set. Here's a tutorial on the submission for a smooth evaluation process.
Submission TutorialPlease cite if you find our leaderboard helpful.
To keep up to date with major changes to the leaderboard and dataset, please subscribe here !
Include top models for different datasets. * denotes model trained on this dataset.
Rank | MIMIC-CXR | IU-Xray | CheXpert Plus |
---|---|---|---|
1 |
MedVersa*
Harvard |
MedVersa
Harvard |
MedVersa
Harvard |
2 |
RaDialog*
TUM |
RGRG
TUM |
RaDialog
TUM |
3 |
RGRG*
TUM |
RadFM
SJTU |
CheXpertPlus-mimic
Stanford |
4 |
CheXpertPlus-mimic*
Stanford |
Cvt2distilgpt2
CSIRO |
RGRG
TUM |
5 |
CheXagent*
Stanford |
RaDialog
TUM |
Cvt2distilgpt2
CSIRO |
6 |
Cvt2distilgpt2*
CSIRO |
CheXpertPlus-mimic
Stanford |
CheXagent
Stanford |
7 |
VLCI*
SYSU |
CheXagent
Stanford |
RadFM
SJTU |
8 |
RadFM*
SJTU |
VLCI
SYSU |
GPT4V
OpenAI |
9 |
GPT4V
OpenAI |
GPT4V
OpenAI |
VLCI
SYSU |
10 |
LLM-CXR*
KAIST |
LLM-CXR
KAIST |
LLM-CXR
KAIST |
MIMIC-CXR contains 377,110 images corresponding to 227,835 radiographic studies performed at the Beth Israel Deaconess Medical Center in Boston, MA. We follow the official split of MIMIC-CXR in the following experiments, and report the score on test set. * denotes the model was trained on MIMIC-CXR.
Rank | Model | FineRadScore ↓ | RadCliQ-v1 ↓ | BLEU ↑ | BertScore ↑ | SembScore ↑ | RadGraph ↑ |
---|---|---|---|---|---|---|---|
1 2024 |
CheXagent*
Stanford |
2.976 | 1.437 | 0.094 | 0.304 | 0.331 | 0.146 |
2 2024 |
CheXpertPlus-mimic*
Stanford |
2.846 | 1.369 | 0.128 | 0.308 | 0.365 | 0.168 |
3 2023 |
Cvt2distilgpt2*
CSIRO |
2.944 | 1.498 | 0.109 | 0.278 | 0.315 | 0.145 |
4 2024 |
MedVersa*
Harvard |
2.839 | 1.088 | 0.193 | 0.43 | 0.315 | 0.273 |
5 2023 |
RadFM*
SJTU |
3.092 | 1.604 | 0.081 | 0.281 | 0.245 | 0.111 |
6 2023 |
RaDialog*
TUM |
2.92 | 1.33 | 0.112 | 0.322 | 0.381 | 0.168 |
7 2023 |
RGRG*
TUM |
3.142 | 1.363 | 0.125 | 0.323 | 0.337 | 0.176 |
8 2023 |
VLCI*
SYSU |
2.99 | 1.573 | 0.124 | 0.252 | 0.293 | 0.136 |
9 2024 |
LLM-CXR*
KAIST |
3.681 | 1.983 | 0.036 | 0.158 | 0.15 | 0.047 |
10 2024 |
GPT4V
OpenAI |
3.345 | 1.819 | 0.065 | 0.204 | 0.19 | 0.085 |
IU Xray contains 7,470 pairs of radiology reports and chest X-rays from Indiana University. We follow the split given by R2Gen. * denotes the model was trained on IU X-ray.
Rank | Model | RadCliQ-v1 ↓ | RadCliQ-v0 ↓ | BLEU ↑ | BertScore ↑ | SembScore ↑ | RadGraph ↑ |
---|---|---|---|---|---|---|---|
1 2024 |
CheXagent
Stanford |
1.137 | 3.272 | 0.102 | 0.38 | 0.494 | 0.157 |
2 2024 |
CheXpertPlus-mimic
Stanford |
1.085 | 3.225 | 0.173 | 0.352 | 0.588 | 0.159 |
3 2023 |
Cvt2distilgpt2
CSIRO |
0.956 | 3.029 | 0.192 | 0.39 | 0.605 | 0.2 |
4 2024 |
MedVersa
Harvard |
0.692 | 2.581 | 0.195 | 0.518 | 0.601 | 0.244 |
5 2023 |
RadFM
SJTU |
0.815 | 2.8 | 0.196 | 0.479 | 0.556 | 0.234 |
6 2023 |
RaDialog
TUM |
0.97 | 3.044 | 0.175 | 0.419 | 0.545 | 0.198 |
7 2023 |
RGRG
TUM |
0.803 | 2.818 | 0.24 | 0.447 | 0.603 | 0.248 |
8 2023 |
VLCI
SYSU |
1.18 | 3.393 | 0.115 | 0.332 | 0.472 | 0.203 |
9 2024 |
LLM-CXR
KAIST |
2.072 | 4.863 | 0.035 | 0.175 | 0.061 | 0.023 |
10 2024 |
GPT4V
OpenAI |
1.462 | 3.856 | 0.079 | 0.235 | 0.403 | 0.16 |
1 2024 |
CheXagent
Stanford |
1.302 | 3.548 | 0.084 | 0.353 | 0.405 | 0.13 |
2 2024 |
CheXpertPlus-mimic
Stanford |
1.161 | 3.361 | 0.169 | 0.342 | 0.534 | 0.156 |
3 2023 |
Cvt2distilgpt2
CSIRO |
1.037 | 3.175 | 0.192 | 0.377 | 0.556 | 0.191 |
4 2024 |
MedVersa
Harvard |
0.804 | 2.768 | 0.184 | 0.497 | 0.546 | 0.226 |
5 2023 |
RadFM
SJTU |
0.898 | 2.944 | 0.187 | 0.463 | 0.507 | 0.227 |
6 2023 |
RaDialog
TUM |
1.034 | 3.151 | 0.16 | 0.403 | 0.509 | 0.197 |
7 2023 |
RGRG
TUM |
0.896 | 2.977 | 0.232 | 0.436 | 0.541 | 0.238 |
8 2023 |
VLCI
SYSU |
1.175 | 3.386 | 0.113 | 0.325 | 0.481 | 0.209 |
9 2024 |
LLM-CXR
KAIST |
2.051 | 4.833 | 0.042 | 0.178 | 0.071 | 0.027 |
10 2024 |
GPT4V
OpenAI |
1.458 | 3.849 | 0.083 | 0.233 | 0.414 | 0.157 |
CheXpert Plus contains 223,228 unique pairs of radiology reports and chest X-rays from 187,711 studies and 64,725 patients. We follow the official split of CheXpert Plus in the following experiments and use the valid set for evaluation. * denotes the model was trained on CheXpert Plus.
Rank | Model | RadCliQ-v1 ↓ | RadCliQ-v0 ↓ | BLEU ↑ | BertScore ↑ | SembScore ↑ | RadGraph ↑ |
---|---|---|---|---|---|---|---|
1 2024 |
CheXagent
Stanford |
2.213 | 5.147 | 0.077 | -0.056 | 0.284 | 0.039 |
2 2024 |
CheXpertPlus-mimic
Stanford |
2.16 | 5.06 | 0.088 | -0.041 | 0.306 | 0.043 |
3 2023 |
Cvt2distilgpt2
CSIRO |
2.202 | 5.132 | 0.083 | -0.052 | 0.288 | 0.038 |
4 2024 |
MedVersa
Harvard |
2.031 | 4.831 | 0.09 | 0.013 | 0.337 | 0.05 |
5 2023 |
RadFM
SJTU |
2.251 | 5.206 | 0.067 | -0.038 | 0.229 | 0.027 |
6 2023 |
RaDialog
TUM |
2.111 | 4.967 | 0.086 | -0.035 | 0.348 | 0.041 |
7 2023 |
RGRG
TUM |
2.194 | 5.139 | 0.103 | -0.039 | 0.263 | 0.047 |
8 2023 |
VLCI
SYSU |
2.317 | 5.34 | 0.084 | -0.092 | 0.248 | 0.032 |
9 2024 |
LLM-CXR
KAIST |
2.369 | 5.388 | 0.032 | -0.082 | 0.201 | 0.013 |
10 2024 |
GPT4V
OpenAI |
2.316 | 5.318 | 0.055 | -0.065 | 0.208 | 0.028 |