Agreement measures are useful metrics to both compare different evaluations of the same diagnostic outcomes and validate new rating systems or devices. While many of them have been proposed in the literature so far, Cohen's. is still the de facto standard in gauging the agreement.Information Agreement (IA) is a novel two-observers information-theoretic-based metric introduced to overcome all the limitations and alleged pitfalls of Cohen's kappa. It offers an operative meaning to the agreement since it measures - in both dichotomous and multi-value ordered-categorical cases - the information shared between two raters through the virtual diagnostic channel connecting them: the more information exchanged between the raters, the higher their agreement. Unfortunately, this measure is only able to deal with agreement matrices whose values are all strictly positive numbers.This work extends IA by admitting also 0 as a possible value for the entries of an agreement matrix. Moreover, a Python software library to compute the extended version of IA, together with some of the most used agreement measures, is presented and tested.

Extending Information Agreement by Continuity

Alberto Casagrande
;
Rossano Girometti
2020-01-01

Abstract

Agreement measures are useful metrics to both compare different evaluations of the same diagnostic outcomes and validate new rating systems or devices. While many of them have been proposed in the literature so far, Cohen's. is still the de facto standard in gauging the agreement.Information Agreement (IA) is a novel two-observers information-theoretic-based metric introduced to overcome all the limitations and alleged pitfalls of Cohen's kappa. It offers an operative meaning to the agreement since it measures - in both dichotomous and multi-value ordered-categorical cases - the information shared between two raters through the virtual diagnostic channel connecting them: the more information exchanged between the raters, the higher their agreement. Unfortunately, this measure is only able to deal with agreement matrices whose values are all strictly positive numbers.This work extends IA by admitting also 0 as a possible value for the entries of an agreement matrix. Moreover, a Python software library to compute the extended version of IA, together with some of the most used agreement measures, is presented and tested.
2020
978-1-7281-6215-7
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11390/1262204
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? 2
social impact