Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Statistical quality of experience analysis: on planning the sample size and statistical significance testing
RISE - Research Institutes of Sweden (2017-2019), ICT, Acreo. Mid Sweden University, Sweden. (Visual Media Quality)ORCID iD: 0000-0001-5060-9402
University of Applied Sciences, Germany.
2018 (English)In: Journal of Electronic Imaging (JEI), ISSN 1017-9909, E-ISSN 1560-229X, Vol. 27, no 5, article id 053013Article in journal (Refereed) Published
Abstract [en]

This paper analyzes how an experimenter can balance errors in subjective video quality tests betweenthe statistical power of finding an effect if it is there and not claiming that an effect is there if the effect is not there,i.e., balancing Type I and Type II errors. The risk of committing Type I errors increases with the number ofcomparisons that are performed in statistical tests. We will show that when controlling for this and at thesame time keeping the power of the experiment at a reasonably high level, it is unlikely that the number oftest subjects that are normally used and recommended by the International Telecommunication Union (ITU),i.e., 15 is sufficient but the number used by the Video Quality Experts Group (VQEG), i.e., 24 is more likelyto be sufficient. Examples will also be given for the influence of Type I error on the statistical significance ofcomparing objective metrics by correlation. We also present a comparison between parametric and nonparametricstatistics. The comparison targets the question whether we would reach different conclusions on the statisticaldifference between the video quality ratings of different video clips in a subjective test, based on thecomparison between the student T-test and the Mann–Whitney U-test. We found that there was hardly a differencewhen few comparisons are compensated for, i.e., then almost the same conclusions are reached. Whenthe number of comparisons is increased, then larger and larger differences between the two methods arerevealed. In these cases, the parametric T-test gives clearly more significant cases, than the nonparametrictest, which makes it more important to investigate whether the assumptions are met for performing a certaintest.

Place, publisher, year, edition, pages
SPIE/IS&T, 2018. Vol. 27, no 5, article id 053013
Keywords [sv]
Type-I error; video quality; statistical significance; quality of experience; Student T-test; Bonferroni; Mann–Whitney U-test; parametric versus nonparametric test.
National Category
Telecommunications Communication Systems Media Engineering
Identifiers
URN: urn:nbn:se:ri:diva-35233DOI: 10.1117/1.JEI.27.5.053013Scopus ID: 2-s2.0-85054069504OAI: oai:DiVA.org:ri-35233DiVA, id: diva2:1252987
Funder
Knowledge Foundation, 20160194
Note

Copyright (2018) Society of Photo-Optical Instrumentation Engineers. One print or electronic copy may be made for personal use only. Systematic reproduction and distribution, duplication of any material in this paper for a fee or for commercial purposes, or modification of the content of the paper are prohibited.

Available from: 2018-10-03 Created: 2018-10-03 Last updated: 2023-05-25Bibliographically approved

Open Access in DiVA

fulltext(1796 kB)511 downloads
File information
File name FULLTEXT01.pdfFile size 1796 kBChecksum SHA-512
38573c0265a662ee8f2b58d1e734f030df692856f05a76cca42d5800cde656c9eed0038471a3d1bc20ee218b11f47bcf08b6c64ecda09c09159e4e1a437cd81f
Type fulltextMimetype application/pdf

Other links

Publisher's full textScopus

Authority records

Brunnström, Kjell

Search in DiVA

By author/editor
Brunnström, Kjell
By organisation
Acreo
In the same journal
Journal of Electronic Imaging (JEI)
TelecommunicationsCommunication SystemsMedia Engineering

Search outside of DiVA

GoogleGoogle Scholar
Total: 511 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 729 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf