Vortex Solitons in Atomic–Molecular Bose–Einstein Condensates with a Square-Optical-Lattice Potential

Abstract We propose a theoretical framework, based on the two-component Gross–Pitaevskii equation (GPE), for the investigation of vortex solitons (VSs) in hybrid atomic–molecular Bose–Einstein condensates under the action of the stimulated Raman-induced photoassociation and square-optical-lattice potential. Stationary solutions of the coupled GPE system are obtained by means of the imaginary-time integration, while the temporal dynamics are simulated using the fourth-order Runge–Kutta algorithm. The analysis reveals stable rhombus-shaped VS shapes with topological charges m = 1 and 2 of the atomic component. The stability domains and spatial structure of these VSs are governed by three key parameters: the parametric-coupling strength (χ), atomic-molecular interaction strength (g₁₂), and the optical-lattice potential depth (V₀). By varying χ and g₁₂, we demonstrate a structural transition where four-core rhombus-shaped VSs evolve into eight-core square-shaped modes, highlighting the nontrivial nonlinear dynamics of the system. This work establishes a connection between interactions of cold atoms and topologically structured matter waves in hybrid quantum systems.