ON THE RELIABILITY AND SPECTRAL EFFICIENCY OF MULTI-ANTENNA AF RELAY-AIDED NOMA NETWORKS

(Received: 27-Feb.-2026, Revised: 30-Apr.-2026 , Accepted: 4-May-2026)

Authors Hong-Nhu Nguyen, Mui Van Nguyen, Minh Xuan Pham, Sang-Quang Nguyen,

Keywords #Cooperative non-orthogonal multiple access (NOMA) #Multi-antenna AF relaying #Spatial diversity #Nakagami-k fading #Outage probability #Ergodic capacity

Abstract This paper investigates a downlink cooperative non-orthogonal multiple access (NOMA) system assisted by a multi-antenna amplify-and-forward (AF) relay. Unlike conventional single-antenna relay configurations, the considered framework jointly exploits relay diversity and direct transmission links between the base station (BS) and users. Under independent Nakagami-k fading channels, closed-form expressions for the outage probability (OP) and ergodic capacity (EC) of both users are derived for scenarios with and without direct BS-user links. The analytical formulation explicitly captures the effects of the number of relay antennas, fading severity and power allocation coefficients on system performance. For the ergodic capacity analysis, an exact integral representation combined with a Gaussian-Chebyshev quadrature approach is developed to efficiently evaluate the performance under the max-SINR selection criterion. The analytical results are verified through Monte-Carlo simulations and compared with orthogonal multiple access (OMA) benchmarks. Numerical results demonstrate that increasing the number of relay antennas significantly improves reliability due to enhanced spatial diversity. Moreover, the NOMA scheme achieves superior outage performance for the far user compared with OMA, while the ergodic capacity of the near user exhibits a pronounced gain in the moderate-to-high SNR regime. These findings confirm the effectiveness of multi-antenna cooperative relaying in improving both reliability and spectral efficiency.

References

[1] Z. Ding et al., "A Survey on Non-orthogonal Multiple Access for 5G Networks: Research Challenges and Future Trends," IEEE Journal on Selected Areas in Communications, vol. 35, no. 10, pp. 2181-2195, 2017.

[2] L. Dai et al., "A Survey of Non-orthogonal Multiple Access for 5G," IEEE Communications Surveys & Tutorials, vol. 20, no. 3, pp. 2294-2323, 2018.

[3] Z. Wei et al., "A Survey of Downlink Non-orthogonal Multiple Access for 5G Wireless Communication Networks," arXiv preprint, arXiv: 1609.01856, 2016.

[4] S. R. Islam et al., "Power-domain Non-orthogonal Multiple Access (NOMA) in 5G Systems: Potentials and Challenges," IEEE Comm. Surveys & Tutorials, vol. 19, no. 2, pp. 721-742, 2016.

[5] Q.-S. Nguyen, T. N. Nguyen and L.-T. Tu, "On the Security and Reliability Performance of SWIPT-enabled Full-duplex Relaying in the Non-orthogonal Multiple Access Networks," Journal of Information and Telecommunication, vol. 7, no. 4, pp. 462-476, 2023.

[6] M. Zeng et al., "Cooperative NOMA: State of the Art, Key Techniques and Open Challenges," IEEE Network, vol. 34, no. 5, pp. 205-211, 2020.

[7] Y. Yuan et al., "Joint Robust Beamforming and Power-splitting Ratio Design in SWIPT-based Cooperative NOMA Systems with CSI Uncertainty," IEEE Trans. on Vehicular Technology, vol. 68, no. 3, pp. 2386-2400, March 2019.

[8] A.-T. Le, T. D. Hieu, T. N. Nguyen, T.-L. Le, S. Q. Nguyen and M. Voznak, "Physical Layer Security Analysis for RIS-aided NOMA Systems with Non-Colluding Eavesdroppers," Computer Communications, vol. 219, pp. 194-203, Apr. 2024.

[9] T. N. Nguyen, Q.-S. Nguyen, N. M. Quan, T. V. Chien, B. V. Minh and T.-L. Thuong, "Reliability and Security Analysis of Active RIS-assisted IoT NOMA Networks over Nakagami-k fading channels," IEEE Internet of Things Journal, Early Access, DOI:10.1109/JIOT.2025.3650486, 2026.

[10] N. Q. Sang et al, "Performance of RIS-secured Short-packet NOMA Systems with Discrete Phaseshifter to Protect Digital Content and Copyright Against Untrusted User," IEEE Access, vol. 13, pp. 21580-21593, 2025.

[11] V.-D. Le et al., "Enabling D2D Transmission Mode of Reconfigurable Intelligent Surfaces Aided in Wireless NOMA System," Advances in Electrical and Electronic Eng., vol. 23, no. 1, 2025.

[12] T.-H. T. Pham et al., "Performance Analysis in D2D Partial NOMA-assisted Backscatter Communication," Advances in Electrical and Electronic Eng., vol. 23, no. 3, 2025.

[13] S.-Q. Nguyen et al., "Securing Short-packet Transmissions via Partial NOMA: Performance Analysis under Keyhole Fading," Vehicular Communications, vol. 58, p. 100999, 2026.

[14] A. Le-Thi et al., "Power Splitting-based SWIPT in UAV-aided NOMA Systems over Nakagami-m Fading: Performance Analysis and Optimization," Wireless Networks, vol. 32, pp. 315-330, 2026.

[15] M. Tran, M. Bui Vu and S. Nguyen, "On the Performance of Active RIS-enhanced NOMA Systems with Spectrum Sharing Mechanisms," Plos One, vol. 20, no. 11, p. e0336951, 2025.

[16] S.-Q. Nguyen et al., "Securing Wireless Communications with Energy Harvesting and Multi-antenna Diversity," Jordanian Journal of Computers and Information Technology (JJCIT), vol. 11, no. 2, pp. 197-210, DOI: 10.5455/jjcit.71-1732244909, Jun. 2025.

[17] V. S. Nguyen, A. Le-Thi, V. D. Thuan, C.-B. Le, T. H. Nguyen and S.-Q. Nguyen, " Analysis of Ergodic Sum Rate in RSMA with Perfect and Imperfect SIC: A Multiple-antenna Selection Approach for Optimizing UAV Positioning," Physical Communication, vol. 72, p. 102741, 2025.

[18] N. H. Nhu et al., "Covert Communication Performance Evaluation in UAV-assisted Rate-splitting Multiple Access Systems," PLos One, vol. 20, no. 8, p. e0331013, 2025.

[19] Z. Yang, Z. Ding, Y. Wu and P. Fan, "Novel Relay Selection Strategies for Cooperative NOMA," IEEE Trans. on Vehicular Technology, vol. 66, no. 11, pp. 10114-10123, Nov. 2017.

[20] P. Xu, Z. Yang, Z. Ding and Z. Zhang, "Optimal Relay Selection Schemes for Cooperative NOMA," IEEE Trans. on Vehicular Technology, vol. 67, no. 8, pp. 7851-7855, Aug. 2018.

[21] M. Ashraf et al., "Energy Harvesting Non-orthogonal Multiple Access System with Multi-antenna Relay and Base Station," IEEE Access, vol. 5, pp. 17660-17670, 2017.

[22] L. Lv, Q. Ye, Z. Ding, Z. Li, N. Al-Dhahir and J. Chen, "On the Design of NOMA Assisted Multi-Antenna Two-way Relay Systems," Proc. of the IEEE Int. Conf. on Communications (ICC), DOI: 10.1109/ICC40277.2020.9149110, Dublin, Ireland, 2020.

[23] X. Chen et al., "Exploiting Multiple-antenna Techniques for Non-orthogonal Multiple Access," IEEE Journal on Selected Areas in Communications, vol. 35, no. 10, pp. 2207-2220, Oct. 2017.

[24] Y. Cao et al., "Secrecy Analysis for Cooperative NOMA Networks with Multi-antenna Full-duplex Relay," IEEE Trans. on Communications, vol. 67, no. 8, pp. 5574-5587, Aug. 2019.

[25] N. Zaghdoud et al., "Secrecy Performance Analysis of Multi-antenna NOMA System with AF/DF Relaying under External and Internal Eavesdropping Scenarios," Proc. of the 2020 Int. Wireless Communications and Mobile Computing (IWCMC), pp. 1726-1732, Limassol, Cyprus, 2020.

[26] Z. Tang et al., "Reconsidering Design of Multi-antenna NOMA Systems with Limited Feedback," IEEE Trans. on Wireless Communications, vol. 19, no. 3, pp. 1519-1534, March. 2020.

[27] L. Lv, Q. Ye, Z. Ding, Z. Li, N. Al-Dhahir and J. Chen, "Multi-antenna Two-way Relay Based Cooperative NOMA," IEEE Trans. on Wireless Comm., vol. 19, no. 10, p. 64866503, 2020.

[28] T. A. Le and H. Y. Kong, "Energy Harvesting Relay-antenna Selection in Cooperative MIMO/NOMA Network over Rayleigh Fading," Wireless Net., vol. 26, no. 3, pp. 2075-2087, 2020.

[29] A. Hakimi, M. Mohammadi and Z. Mobini, "Outage Probability of Wireless-powered Multi-antenna Cooperative Spectrum Sharing Networks with Full-duplex and NOMA Transmissions," Proc. of the 2018 9th Int. Symposium on Telecommunications (IST), pp. 127-132, Tehran, Iran, 2018.

[30] Z. Mobini et al., "Full-duplex Multi-antenna Relay Assisted Cooperative Non-orthogonal Multiple Access," Proc. of the 2017 IEEE Global Communications Conference, pp. 1-7, Singapore, 2017.

[31] A. Jee and S. Prakriya, "Performance of Energy and Spectrally Efficient AF Relay-aided Incremental CDRT NOMA-based IoT Network with Imperfect SIC for Smart Cities," IEEE Internet of Things Journal, vol. 10, pp. 18766-18781, 2023.

[32] C. Hu, Q. Li, Q. Zhang and J. Qin, "Security Optimization for an AF MIMO Two-way Relay-assisted Cognitive Radio Non-orthogonal Multiple Access Networks with SWIPT," IEEE Trans. on Information Forensics And Security, vol. 17, pp. 1481-1496, 2022.

[33] I. S. Gradshteyn and I. M. Ryzhik, Tables of Integrals, Series and Products, 6th Edn., ISBN-10: 0-12-373637-4, New York: Academic Press, 2000.

[34] Q. Wang, J. Ge, Q. Li and Q. Bu, "Performance Analysis of NOMA for Multiple-antenna Relaying Networks with Energy Harvesting over Nakagami-m Fading Channels," Proc. of the 2017 IEEE/CIC Int. Conf. on Communications in China, pp. 1-5, Qingdao, China, Oct. 2017.

[35] C. C. Hung, C. T. Chiang, S. N. Lin and R. C. Wu, "Outage Capacity Analysis of TAS/MRC Systems over Arbitrary Nakagami-m Fading Channels," IEICE Trans. on Communications, vol. 93, no. 1, pp. 215-218, 2010.