seamedМорская медицинаMarine Medicine2413-57472587-7828Research Institute of Industrial and Maritime Medicine of the Federal Medical and Biological Agency10.22328/2413-5747-2023-9-3-102-112seamed-683Research ArticleОригинальные исследованияOriginal articlesАнализ надежности оценки дихотомических исходов: размер выборки и расчет каппа-статистикиReliability analysis of binary outcomes: sample size and calculations of kappa statistichttps://orcid.org/0000-0002-5631-5197МитькинаЕкатерина АндреевнаMitkinaEkaterina A.

студентка стоматологического факультета

student of the faculty of dentistry

miekandr@yandex.ruhttps://orcid.org/0009-0003-6496-2719КозловаЮлия ГеннадьевнаKozlovaYulia G.

студентка стоматологического факультета

student of the faculty of dentistry

iuliak0z@yandex.ruhttps://orcid.org/0000-0002-6363-9595ГорбатоваМария АлександровнаGorbatovaMaria A.

кандидат медицинских наук, магистр общественного здоровья, доцент кафедры стоматологии детского возраста

Cand. of Sci. (Med.), MPH, Associate professor at the Department of Pediatric Dentistry

marigora@mail.ruhttps://orcid.org/0000-0002-5464-0498ГржибовскийАндрей МечиславовичGrjibovskiAndrej M.

доктор медицинских наук, начальник управления научно-инновационной работы, заведующий центральной научно-исследовательской лабораторией; профессор кафедры общественного здоровья, здравоохранения, общей гигиены и биоэтики

D-r of Sci. (Med), Master of International Community Health, Head of the Directorate for Research and Innovations, Director of the Central Scientific Research Laboratory; Professor at the Department of Public Health, Public Health, General Hygiene and Bioethics

A.Grjibovski@yandex.ruСеверный государственный медицинский университетРоссияNorthern State Medical UniversityRussian FederationСеверный государственный медицинский университет; Северо-Восточный федеральный университетРоссияNorthern State Medical University; North-Eastern Federal UniversityRussian Federation20232011202393102112Copyright © Митькина Е.А., Козлова Ю.Г., Горбатова М.А., Гржибовский А.М., 20232023Митькина Е.А., Козлова Ю.Г., Горбатова М.А., Гржибовский А.М.Mitkina E.A., Kozlova Y.G., Gorbatova M.A., Grjibovski A.M.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://seamed.elpub.ru/jour/article/view/683

Анализ надежности — это важный методологический инструмент, используемый в медицинских исследованиях для определения степени согласованности измерений, проводимых различными методами или несколькими исследователями. В данной статье авторы доходчиво представляют обзор основных концепций, связанных с анализом надежности, а также статистические критерии, используемые при его применении в медицине. Представлены сходства и отличия анализа валидности (достоверности) от анализа надежности (воспроизводимости). Демонстрируются принципы расчета каппа-статистики для простейшей ситуации с двумя исследователями и бинарными признаками как с помощью формул, так и с помощью пакета статистических программ SPSS, а также ее достоинства и ограничения. Статья предназначена для начинающих исследователей и молодых ученых и будет полезна при планировании исследований и подготовке сборщиков данных.

Reliability analysis is an important methodological tool used in medical research to assess the degree of agreement between measurements taken by different methods or by multiple investigators. In this article, we provide an easy-to-understand overview of the basic concepts associated with reliability analysis, as well as the statistical criteria used in its application in biomedical research. The similarities and differences between the analysis of validity and the analysis of reliability are also presented. The principles of calculating Cohen’s kappa for the simplest situation with two researchers and binary variables are demonstrated both by using the formulas and by applying the SPSS software. Advantages and disadvantages of using kappa statistic are discussed. The article is intended for novice researchers and young scientists and will be useful for planning of research projects and training data collectors.

морская медицинаанализ надежностидостоверностькаппа-статистикастатистическая мощностьSPSSразмер выборкикалибровкаmarine medicinereliability analysisvaliditykappa-statisticstatistical powerSPSSsample sizecalibrationReferencesWhittemore R., Chase S.K., Mandle C.L. Validity in Qualitative Research. Qualitative Health Research, 2001, Vol. 11, № 4, pp. 522–537. doi: 10.1177/104973201129119299.Whittemore R., Chase S.K., Mandle C.L. Validity in Qualitative Research. Qualitative Health Research, 2001, Vol. 11, № 4, pp. 522–537. doi: 10.1177/104973201129119299.Ahmed I., Ishtiaq S. Reliability and validity: Importance in Medical Research. J Pak Med Assoc, 2021, Vol. 71, № 10, pp. 2401–2406. doi: 10.47391/JPMA.06-861.Ahmed I., Ishtiaq S. Reliability and validity: Importance in Medical Research. J Pak Med Assoc, 2021, Vol. 71, № 10, pp. 2401–2406. doi: 10.47391/JPMA.06-861.McHugh Mary L. Interrater reliability: the kappa statistic. Biochem Med (Zagreb), 2012, Vol. 22, № 3, pp. 276–282.McHugh Mary L. Interrater reliability: the kappa statistic. Biochem Med (Zagreb), 2012, Vol. 22, № 3, pp. 276–282.Tang W., Hu J., Zhang H., Wu P., He H. Kappa coefficient: a popular measure of rater agreement. Shanghai Arch Psychiatry, 2015, Vol. 27, № 1, pp. 62–67. doi: 10.11919/j.issn.1002-0829.215010.Tang W., Hu J., Zhang H., Wu P., He H. Kappa coefficient: a popular measure of rater agreement. Shanghai Arch Psychiatry, 2015, Vol. 27, № 1, pp. 62–67. doi: 10.11919/j.issn.1002-0829.215010.Noble H., Smith J. Issue of validity and reliability in quantitative research. Evid Based Nurs, 2015, Vol. 18, № 2, pp. 34–35. doi: 10.1136/eb-2015-102054.Noble H., Smith J. Issue of validity and reliability in quantitative research. Evid Based Nurs, 2015, Vol. 18, № 2, pp. 34–35. doi: 10.1136/eb-2015-102054.Aoki K., Hall T., Takasaki H. Reporting on the level of validity and reliability of questionnaires measuring Katakori severity: A systematic review. SAGE Open Med, 2019, Vol. 7, pp. 1–13. doi: 10.1177/2050312119836617.Aoki K., Hall T., Takasaki H. Reporting on the level of validity and reliability of questionnaires measuring Katakori severity: A systematic review. SAGE Open Med, 2019, Vol. 7, pp. 1–13. doi: 10.1177/2050312119836617.Akturk Z. Reliability and validity in medical research. Dicle Med J, 2012, Vol. 39, № 2, pp. 196–202. doi: 10.5798/diclemedj.0921.2012.02.0150.Akturk Z. Reliability and validity in medical research. Dicle Med J, 2012, Vol. 39, № 2, pp. 196–202. doi: 10.5798/diclemedj.0921.2012.02.0150.Fyffe H.E., Deery C., Nugent Z.J., Nuttall N.M., Pitts N.B. Effect of diagnostic threshold on the validity and reliability of epidemiological caries diagnosis using the Dundee selectable threshold method for caries diagnosis (DSTM). Community Dent Oral Epidemiol, 2000, Vol. 28, № 1, pp. 42–51. doi: 10.1034/j.1600-0528.2000.280106.x.Fyffe H.E., Deery C., Nugent Z.J., Nuttall N.M., Pitts N.B. Effect of diagnostic threshold on the validity and reliability of epidemiological caries diagnosis using the Dundee selectable threshold method for caries diagnosis (DSTM). Community Dent Oral Epidemiol, 2000, Vol. 28, № 1, pp. 42–51. doi: 10.1034/j.1600-0528.2000.280106.x.Рождественская Е.Ю. Надежность качественных методов и качество данных // INTER. 2014. Т. 8. C. 16–28 [Rozhdestvenskaya E.Y. Reliability of qualitative methods and data quality. INTER, 2014, № 8, 16–29 (In Russ.)].Рождественская Е.Ю. Надежность качественных методов и качество данных // INTER. 2014. Т. 8. C. 16–28 [Rozhdestvenskaya E.Y. Reliability of qualitative methods and data quality. INTER, 2014, № 8, 16–29 (In Russ.)].Rechmann P., Jue B., Santo W., Rechmann B.M.T., Featherstone J.D.B. Calibration of dentists for Caries Management by Risk Assessment Research in a Practice Based Research network - cambra pbrn. BMC Oral Health, 2018, Vol. 18, №2. doi: 10.1186/s12903-017-0457-3.10.1186/s12903-017-0457-3Rechmann P., Jue B., Santo W., Rechmann B.M.T., Featherstone J.D.B. Calibration of dentists for Caries Management by Risk Assessment Research in a Practice Based Research network - cambra pbrn. BMC Oral Health, 2018, Vol. 18, №2. doi: 10.1186/s12903-017-0457-3.10.1186/s12903-017-0457-3Tavakol M., Sandars J. Quantitative and qualitative methods in medical education research: AMEE Guide No 90: Part II. Medical Teacher, 2014, Vol. 36, № 10, pp. 838–848. doi: 10.3109/0142159X.2014.915297.Tavakol M., Sandars J. Quantitative and qualitative methods in medical education research: AMEE Guide No 90: Part II. Medical Teacher, 2014, Vol. 36, № 10, pp. 838–848. doi: 10.3109/0142159X.2014.915297.Warren J.J., Weber-Gasparoni K., Tinanoff N., Batliner T.S., Jue B., Santo W., Garcia R.I., Gansky S.A., Early Childhood Caries Collaborating Centers. Examination criteria and calibration procedures for prevention trials of the Early Childhood Caries Collaborating Centers. Public Health Dent, 2015, Vol. 75, № 4, pp. 317–326. doi: 10.1111/jphd.12102.Warren J.J., Weber-Gasparoni K., Tinanoff N., Batliner T.S., Jue B., Santo W., Garcia R.I., Gansky S.A., Early Childhood Caries Collaborating Centers. Examination criteria and calibration procedures for prevention trials of the Early Childhood Caries Collaborating Centers. Public Health Dent, 2015, Vol. 75, № 4, pp. 317–326. doi: 10.1111/jphd.12102.Amarante B.C., Arima L.Y., Pinheiro E., Carvalho P., Michel-Crosato E., Bönecker M. Diagnosis training and calibration for epidemiological studies on primary and permanent teeth with hypomineralization. Eur Arch Paediatr Dent, 2022, Vol. 23, № 1, pp. 169–177. doi: 10.1007/s40368-021-00686-3.Amarante B.C., Arima L.Y., Pinheiro E., Carvalho P., Michel-Crosato E., Bönecker M. Diagnosis training and calibration for epidemiological studies on primary and permanent teeth with hypomineralization. Eur Arch Paediatr Dent, 2022, Vol. 23, № 1, pp. 169–177. doi: 10.1007/s40368-021-00686-3.Shoukri M. Measurement of Agreement. Wiley StatsRef: Statistics Reference Online, 2015, pp. 1–31. doi: 10.1002/9781118445112.stat05301.pub2.Shoukri M. Measurement of Agreement. Wiley StatsRef: Statistics Reference Online, 2015, pp. 1–31. doi: 10.1002/9781118445112.stat05301.pub2.Donner A., Rotondi M.A. Sample Size Requirements for Interval Estimation of the Kappa Statistic for Interobserver Agreement Studies with a Binary Outcome and Multiple Raters. The International Journal of Biostatistics, 2010, Vol. 6, № 1. doi: 10.2202/1557-4679.1275.Donner A., Rotondi M.A. Sample Size Requirements for Interval Estimation of the Kappa Statistic for Interobserver Agreement Studies with a Binary Outcome and Multiple Raters. The International Journal of Biostatistics, 2010, Vol. 6, № 1. doi: 10.2202/1557-4679.1275.Hyunsook H., Yunhee C., Seokyung H., Sue K.P., Byung-Joo P. Nomogram for sample size calculation on a straightforward basis for the kappa statistic. Annals of Epidemiology, 2014, Vol. 24, № 9, pp. 673–680. doi: 10.1016/j.annepidem.2014.06.097.Hyunsook H., Yunhee C., Seokyung H., Sue K.P., Byung-Joo P. Nomogram for sample size calculation on a straightforward basis for the kappa statistic. Annals of Epidemiology, 2014, Vol. 24, № 9, pp. 673–680. doi: 10.1016/j.annepidem.2014.06.097.Guggenmoos-Holzmann I. The meaning of kappa: probabilistic concepts of reliability and validity revisited. Clin Epidemiol, 1996, Vol. 49, № 7, pp. 775–782. doi: 10.1016/0895-4356(96)00011-x.Guggenmoos-Holzmann I. The meaning of kappa: probabilistic concepts of reliability and validity revisited. Clin Epidemiol, 1996, Vol. 49, № 7, pp. 775–782. doi: 10.1016/0895-4356(96)00011-x.Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 1960, Vol. 20, №1, pp. 37–46. doi: 10.1177/001316446002000104.Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 1960, Vol. 20, №1, pp. 37–46. doi: 10.1177/001316446002000104.Sim J., Wright C.C. The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements. Physical Therapy, 2005, Vol. 85, № 3, pp. 257–268. doi: 10.1093/ptj/85.3.257.Sim J., Wright C.C. The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements. Physical Therapy, 2005, Vol. 85, № 3, pp. 257–268. doi: 10.1093/ptj/85.3.257.Кригер Е.А., Гржибовский А.М., Постоев В.А. Оценка распространенности заболеваний с учетом диагностической эффективности тестов на примере использования серологических тестов для диагностики новой коронавирусной инфекции (COVID-19) // Экология человека. 2022. Т. 29, № 5. С. 301–309 [Kriger E.A., Grjibovski A.M., Postoev V.A. Prevalence assessment adjusted for laboratory test performance using an example of the COVID-19 serological tests. Ekologiya cheloveka [Human Ecology], 2022, Vol. 29, № 5, 301–309 (In Russ.)]. doi: 10.17816/humeco108116.Кригер Е.А., Гржибовский А.М., Постоев В.А. Оценка распространенности заболеваний с учетом диагностической эффективности тестов на примере использования серологических тестов для диагностики новой коронавирусной инфекции (COVID-19) // Экология человека. 2022. Т. 29, № 5. С. 301–309 [Kriger E.A., Grjibovski A.M., Postoev V.A. Prevalence assessment adjusted for laboratory test performance using an example of the COVID-19 serological tests. Ekologiya cheloveka [Human Ecology], 2022, Vol. 29, № 5, 301–309 (In Russ.)]. doi: 10.17816/humeco108116.Zec S., Soriani N., Comoretto R., Baldi I. High Agreement and High Prevalence: The Paradox of Cohen’s Kappa. Open Nurs J, 2017, Vol. 11, pp. 211–218. doi: 10.2174/1874434601711010211.Zec S., Soriani N., Comoretto R., Baldi I. High Agreement and High Prevalence: The Paradox of Cohen’s Kappa. Open Nurs J, 2017, Vol. 11, pp. 211–218. doi: 10.2174/1874434601711010211.

The authors declare that there are no conflicts of interest present.