This study addresses the practical challenges of real-time kinematic (RTK)-based precise relative navigation and integer ambiguity resolution for CubeSat rendezvous missions. We propose a single-frequency, single-station RTK relative navigation system specifically designed for low Earth orbit (LEO) satellites and adapted from ground-based RTK systems. To achieve precise relative navigation, we analyze the statistical characteristics of integer ambiguity in carrier-phase measurements, starting with pseudorange-based differential GPS relative navigation. Instead of using the conventional extended Kalman filter, we introduce a recursive ambiguity filter based on weighted least squares, improving the float solution for ambiguity estimation without requiring complex filter designs. By deriving the statistical characteristics and the spatial metric known as ambiguity dilution of precision, we establish a strategy to achieve a 100 % success-fix rate in integer ambiguity resolution, utilizing the least-squares ambiguity decorrelation adjustment method for RTK relative navigation. The proposed algorithm’s reliability is validated through software-based LEO simulators and ground-based experimental hardware that uses GPS receivers and patch antennas mounted on the SNUGLITE-III CubeSat. This approach enables centimeter-level relative navigation using only commercial off-the-shelf hardware and GPS receivers on CubeSat platforms. Furthermore, the proposed method improves success-fix rates for integer ambiguity resolution and significantly reduces computational complexity.