In an atomic force microscope AFM , the force is normally sensed by measuring the deflection of a cantilever by an optical lever technique. Experimental results show a nonlinear relationship between the detected signal and the actual deflection of the cantileve, which is widely ignored in literature. In this study we have designed experiments to investigate different possible reasons for this nonlinearity and compared the experimental findings with calculations. It is commonly assumed that this nonlinearity only causes problems for extremely large cantilever deflections. However, our results show that the nonlinear detector response might influence many AFM studies where soft or short cantilevers are used. Based on our analysis we draw conclusions of the main reason for the nonlinearity and suggest a rule of thumb for which cantilevers one should use under different experimental conditions.