Relevance-aware Individual Item Fairness Measures in Recommender Systems ⚖

This repository contains the code for the extra experiments and analyses in our work on "Relevance-aware Individual Item Fairness Measures for Recommender Systems: Limitations and Usage Guidelines" by Theresia Veronika Rampisela, Maria Maistro, Tuukka Ruotsalo, Falk Scholer, and Christina Lioma. The work has been accepted to ACM Transaction on Recommender Systems (ACM TORS).

Link: [ACM]

This work is an extension of the SIGIR'24 full paper "Can We Trust Recommender System Fairness Evaluation? The Role of Fairness and Relevance" by Theresia Veronika Rampisela, Tuukka Ruotsalo, Maria Maistro, and Christina Lioma. The code for the original experiments is available here.

Links to the SIGIR'24 paper: [ACM] [arXiv]

Abstract

Recommender Systems (RSs) aim to provide relevant items to users, with a recent emphasis on improving recommendation fairness. Quantifying fairness of the recommended items can be done with two types of evaluation measures: measures that are purely based on item exposure (exposure-based) and measures that account for both item exposure and item relevance (relevance-aware). While exposure-based measures have been thoroughly analysed, relevance-aware measures have not been examined in such detail yet. We gather all existing relevance-aware individual item fairness measures for RSs and study their theoretical properties. We find that all measures suffer from one or more limitations, which may cause issues in their computation, interpretability, or expressiveness. To address this, we correct the affected measures or explain why a limitation is unresolvable. Further, we empirically investigate the extent of the limitations on the measures and compare the original measures to our reformulations under common and extreme evaluation scenarios across real-world and synthetic data. Our experiments show that our reformulated measures successfully resolve the issues in the original measures. We conclude by providing practical guidelines on how to select measures for a range of use cases.

Citation

@article{10.1145/3765624,
author = {Rampisela, Theresia Veronika and Maistro, Maria and Ruotsalo, Tuukka and Scholer, Falk and Lioma, Christina},
title = {Relevance-aware Individual Item Fairness Measures for Recommender Systems: Limitations and Usage Guidelines},
year = {2025},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3765624},
doi = {10.1145/3765624},
note = {Just Accepted},
journal = {ACM Trans. Recomm. Syst.},
month = sep,
keywords = {relevance-aware fairness, item fairness, individual fairness, evaluation measures, recommender systems}
}

If you use the code for the relevance-aware (joint) fairness measures in metrics.py, please also cite the following work:

@inproceedings{10.1145/3626772.3657832,
author = {Rampisela, Theresia Veronika and Ruotsalo, Tuukka and Maistro, Maria and Lioma, Christina},
title = {Can We Trust Recommender System Fairness Evaluation? The Role of Fairness and Relevance},
year = {2024},
isbn = {9798400704314},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3626772.3657832},
doi = {10.1145/3626772.3657832},,
pages = {271–281},
numpages = {11},
keywords = {fairness and relevance evaluation, recommender systems},
location = {Washington DC, USA},
series = {SIGIR '24}
}

If you use the code for the exposure-based fairness measures in metrics.py (FairWORel), please cite the following work and the original papers proposing the measures.

@article{10.1145/3631943,
author = {Rampisela, Theresia Veronika and Maistro, Maria and Ruotsalo, Tuukka and Lioma, Christina},
title = {Evaluation Measures of Individual Item Fairness for Recommender Systems: A Critical Study},
year = {2024},
issue_date = {June 2025},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
volume = {3},
number = {2},
url = {https://doi.org/10.1145/3631943},
doi = {10.1145/3631943},
journal = {ACM Trans. Recomm. Syst.},
month = nov,
articleno = {18},
numpages = {52},
keywords = {Item fairness, individual fairness, fairness measures, evaluation measures, recommender systems}
}

License and Terms of Usage

The code is usable under the MIT License. Please note that RecBole may have different terms of usage (see their page for updated information).

Datasets, Model Training, and Experiments

Please refer to the code repository of the SIGIR'24 paper to find information on dataset downloads, model training, and the experiment code for the conference paper.

Hyperparameter Search Space & Best Configuration

The hyperparameter search space per model can be found in Recbole/hyperchoice, and the file cluster/bestparam.txt contains the best hyperparameter configurations.