We present a systematic investigation of activation annealing of Mg as p-type doping in GaN. The diffusion of Mg and H by rapid thermal processing (RTP) at 700 °C to 975 °C together with the effect of the ambient gas are investigated by SIMS, XRD, AFM, and electrical measurements. The observed diffusion of H to the substrate emphasizes the importance of understanding the diffusion and reactions of ambient N, O, and H in the GaN layers.We conclude that optimization of the resulting hole density, except the Mg concentration and RTP temperature, the surface morphology, the thickness of the Mg-doped GaN and the thickness of any layer covering it must be considered.