https://jjcit.org/paper/265 10.5455/jjcit.71-1737031353 Areej J. Alabbadi,Belal H. Sababha Ant colony optimization (ACO),3D dynamic environment,UAV swarm,Hybrid navigation approach,Collision avoidance mechanism 1 3190 774 26-Jan.-2025 13-Apr.-2025 6-May-2025 Unmanned Aerial Vehicles (UAVs) play a crucial role in various operations, especially where human life must be protected. Efficient path planning and autonomous coordination are critical for UAV swarms in dynamic 3D cooperative missions, where real-time adaptability is essential. This work addresses the challenge of optimizing UAV swarm operations by proposing a novel hybrid navigation system based on Ant Colony Optimization (ACO). The system efficiently balances path optimization with dynamic formation control, adapting to mission- specific requirements. A key contribution is the hybrid navigation approach, which prioritizes the desired formation of the swarm or the path length and flight time through a threshold- based mechanism, allowing real- time adaptation to changing environments. The system also introduces a comprehensive cost function that evaluates the quality of the path, time consumption, mission completeness and formation divergence. The experiments show that the system consistently provides high-quality paths, achieving around 97% path quality in most cases and never declines below 90%, even in challenging scenarios. The collision avoidance module ensures the completeness of the 100% mission, successfully navigating drones around obstacles and maintaining an optimal path. Furthermore, the formation conservation mechanism effectively maintained the desired swarm configurations while dynamically adapting to obstacles, with the formation change staying within 30% of the allowable range in most scenarios, highlighting the system’s ability to preserve the desired formation even in dynamic environments. This research advances UAV swarm intelligence, enabling efficient and autonomous operations in complex 3D environments for diverse cooperative missions. The system’s adaptability to formation requirements opens new possibilities for UAV swarm applications, improving navigation efficiency and enhancing formation control.