This paper illustrates the use of psychoacoustic methods and multivariate statistics for perceptual evaluation of auralizations. In four experiments, listeners assessed binaural recordings and auralizations of the same sound sources, using a newly developed tool for real time auralization of room acoustics. The recorded signals were music and speech reproduced by a loudspeaker in a classroom and an auditorium. Listeners assessed (a) perceived realism of recorded versus auralized sounds, (b) perceived similarity of pair wise presented sounds, (c) perceived intelligibility of spoken words, and (d) a set of perceptual attributes of sounds, using eight semantic differential scales. The results showed that listeners could discriminate between recorded and auralized sounds, although they were perceived as equally realistic. Analysis showed that auralized sounds were perceived as less full and more metallic than recorded sounds, and auralized speech were easier to catch (hear, understand) than the recorded speech. The next step of this ongoing research is to conduct psychoacoustic analysis in order to identify acoustic differences between recorded and auralized sounds that may explain their perceptual differences.