Paper Details

Name: Vaibhav Bajaj, Taniya Mukhija, Azhar Asroof & Harshita Dhingra
Country: India

Year: 2026
Volume: Volume No: 13, January, Year: 2026 (Special Issue)
Page Number: 240-247
DOI: https://doi.org/10.5281/zenodo.18976678

ABSTRACT—
With cyber threats evolving constantly, we needed something beyond traditional rule-based detection, which struggles against polymorphic attacks. We tested machine learning models, starting with Random Forest (RF) and XGBoost on the CICIDS2017 dataset, and while they gave a solid baseline, they missed sequential attack patterns. That’s when we moved to Deep Learning (DL), specifically Long Short-Term Memory (LSTM) networks, which handle time-series network traffic way better. But then we ran into another issue—class imbalance, where rare attacks were barely represented. So, we used GANs and SMOTE to fix that, generating synthetic attack traffic to train the model better. We evaluated everything with accuracy, precision, recall, F1-score, and AUC-ROC, and the pattern was clear—LSTM outperformed RF and XGBoost, improving malware detection by capturing sequential dependencies in network traffic. Our results highlight the tradeoff between accuracy and computational cost, showing that while LSTM is powerful, hybrid approaches may work even better in balancing detection efficiency and real-time processing.

Keywords— Malware Traffic Analysis, Machine Learning, Deep Learning, LSTM, XGBoost, Random Forest, GANs, SMOTE, Network Security