- Computing and Information Systems - Research Publications
Computing and Information Systems - Research Publications
Permanent URI for this collection
4 results
Filters
Reset filtersSettings
Statistics
Citations
Search Results
Now showing
1 - 4 of 4
-
ItemOptimising Equal Opportunity Fairness in Model TrainingShen, A ; Han, X ; Cohn, T ; Baldwin, T ; Frermann, L (ASSOC COMPUTATIONAL LINGUISTICS-ACL, 2022)
-
ItemEvaluating Debiasing Techniques for Intersectional BiasesSubramanian, S ; Han, X ; Baldwin, T ; Cohn, T ; Frermann, L (Association for Computational Linguistics, 2021-01-01)Bias is pervasive in NLP models, motivating the development of automatic debiasing techniques. Evaluation of NLP debiasing methods has largely been limited to binary attributes in isolation, e.g., debiasing with respect to binary gender or race, however many corpora involve multiple such attributes, possibly with higher cardinality. In this paper we argue that a truly fair model must consider 'gerrymandering' groups which comprise not only single attributes, but also intersectional groups. We evaluate a form of bias-constrained model which is new to NLP, as well an extension of the iterative nullspace projection technique which can handle multiple protected attributes.
-
ItemFairness-aware Class Imbalanced LearningSubramanian, S ; Rahimi, A ; Baldwin, T ; Cohn, T ; Frermann, L (Association for Computational Linguistics, 2021-01-01)Class imbalance is a common challenge in many NLP tasks, and has clear connections to bias, in that bias in training data often leads to higher accuracy for majority groups at the expense of minority groups. However there has traditionally been a disconnect between research on class-imbalanced learning and mitigating bias, and only recently have the two been looked at through a common lens. In this work we evaluate long-tail learning methods for tweet sentiment and occupation classification, and extend a margin-loss based approach with methods to enforce fairness. We empirically show through controlled experiments that the proposed approaches help mitigate both class imbalance and demographic biases.
-
ItemTangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation MetricsMathur, N ; Baldwin, T ; Cohn, T (Association for Computational Linguistics, 2020-07)Automatic metrics are fundamental for the development and evaluation of machine translation systems. Judging whether, and to what extent, automatic metrics concur with the gold standard of human evaluation is not a straightforward problem. We show that current methods for judging metrics are highly sensitive to the translations used for assessment, particularly the presence of outliers, which often leads to falsely confident conclusions about a metric’s efficacy. Finally, we turn to pairwise system ranking, developing a method for thresholding performance improvement under an automatic metric against human judgements, which allows quantification of type I versus type II errors incurred, i.e., insignificant human differences in system quality that are accepted, and significant human differences that are rejected. Together, these findings suggest improvements to the protocols for metric evaluation and system performance evaluation in machine translation.