Simulation is an essential tool for evaluating the performance of wireless protocols and applications because of its cost effectiveness and its high level of control over the simulated system. As the research community envisions increasingly complex applications in areas like mobile social networks and wireless sensor networks, the need for trustworthy simulation of core communication protocols increases. This paper presents a comparative evaluation of several IEEE 802.11 and IEEE 802.15.4 simulators. The paper defines a set of simple experiments that are intended to highlight handling of backoff and contention. A wide variation in results is observed. Although this is partially explained by differences in wireless communication models, the experiments also reinforce existing concerns about the reliability of simulation results. A long term goal is that this methodology will encourage the adoption of common test and evaluation scenarios. This would lead to better understanding of simulation behavior and help to improve the quality of - and confidence in - simulation results.