Holographic imaging provides wavefront information, which is invaluable in various applications, including metrology, industrial inspection, and bioimaging. However, the experimental implementation of holographic imaging remains challenging. Off-axis interferometry has been widely used for its direct phase retrieval capability, but the requirement of a separate reference beam complicates the instrumentation. To address this issue, various reference-free holographic imaging techniques have been developed. It has been recently demonstrated that wavefront information can be directly measured using the speckle-correlation scattering method. The realization of speckle-correlation scattering requires the use of a diffusive layer; however, it poses technical challenges in characterizing the transmission matrix of the diffusive layer or precisely fabricating an engineered random metasurface. Here, liquid crystal geometric phase diffusers are exploited for single-shot reference-free holographic imaging based on a speckle-correlation scattering matrix. The use of a commercial geometric phase diffuser provides precise holographic measurements in a simple imaging setup, with no calibration required. The holographic measurements of various samples are presented, and the proper speckle pattern sampling condition is numerically and experimentally explored. The proposed low-cost, simple, and high-resolution wavefront sensing approach inspired by liquid crystal materials may find direct applications in the area of quantitative phase imaging.