Wooden floors usually have worse impact sound insulation (ISI) at low frequencies than concrete floors having the same rating level. Rating level is usually expressed by single-number quantities (SNQs), such as weighted normalized impact sound pressure level Ln,w. Psychoacoustic research among wooden floors is very limited although a controlled laboratory experiment is the strongest method to point out the most adequate SNQs to be declared for the floors. The purpose of our study was to determine how four standardized SNQs of ISO 717-2, Ln,w, Ln,w + CI, Ln,w + CI,50, and LiA,Fmax,V,T, and a recently proposed SNQ, Ln,w + CI,25, are associated with the annoyance of natural impact sounds transmitted through wooden floors. Fifteen floors were built in the laboratory based either on cross-laminated timber (heavy) or open box wood (light) slabs. Different coverings and suspended ceilings were applied on these slabs. The ISI was tested within 25–3150 Hz using both tapping machine and rubber ball. Thereafter, five natural impact sounds were recorded for each floor: rubber ball drops, steel ball drops, walking, jumping, and chair pushing. Fifty-two people rated the annoyance of these 75 recorded natural impact sounds in psychophysics laboratory. Annoyance was best associated with Ln,w for all the five impact sound types. That is, measurement of ISI within 100–3150 Hz is sufficient from subjective point of view. All four SNQs based on tapping machine explained annoyance better than the SNQ based on rubber ball. Our results can significantly guide the future research, development, and regulations of wooden floors.