The effects of N2 and O2:N2 (1:1) as ambient gases during activation annealing of Mg as p-type doping of GaN have been investigated. The purpose was to understand the mechanisms involved and especially the impact of O2 on the resulting hole concentration and hole mobility. The addition of O2 to the ambient gas during annealing is known to be very effective in reducing the H level of the Mg-doped GaN layer, but the maximum achievable hole concentration and mobility, as determined by Hall characterization, is still higher with pure N2. The difference is explained by an in-diffusion of O to the GaN layer acting as n-dopant and thus giving rise to a compensation effect. It is found that to a large degree only the Mg-H complexes at substitutional (MgGa), i.e., the electrically active acceptor sites that provide free holes, are activated by annealing with N2 only as ambient gas, while annealing with O2:N2 (1:1) also dissociates electrically inactive Mg-H complexes resulting in much less residual H. Thus, the residual H level in relation to the Mg level after activation annealing with N2 only may provide a representative measure of the resulting free hole concentration of the Mg-doped GaN layer.