Artificial Intelligence in Malware and Network Intrusion Detection: A Comprehensive Survey of Techniques, Datasets, Challenges, and Future Directions

Abstract

As cyber threats evolve in complexity and scale, traditional detection mechanisms are increasingly inadequate. This paper presents a comprehensive survey of artificial intelligence (AI) applications in malware and network intrusion detection, emphasizing the integration of intelligent techniques for real-time threat mitigation. We categorize state-of-the-art methods across supervised, unsupervised, and reinforcement learning, including deep learning architectures such as CNNs, RNNs, LSTMs, and ensemble models. Static, dynamic, and hybrid analysis techniques are compared, with a focus on feature engineering, behavioral modeling, and real-world deployment constraints. A novel AI-based Malware Detection and Prevention Framework is proposed, combining machine learning classifiers with Network Intrusion Prevention Systems (NIPS) to enhance proactive defense capabilities. The study evaluates publicly available and synthetic datasets, addressing challenges such as class imbalance, adversarial evasion, and data scarcity. We also highlight ethical considerations including bias, privacy, and accountability in AI-enabled cybersecurity systems. Case studies from mobile and IoT ecosystems demonstrate the practicality and limitations of AI-based defenses in dynamic threat landscapes. Finally, the paper outlines future research directions in explainable AI, automated model generation (AutoML), and adaptive, context-aware intrusion prevention systems. This work serves as a critical resource for developing resilient, scalable, and intelligent cybersecurity infrastructures aligned with modern digital ecosystems