NYC Gazette

Researchers at the Katz School have proposed a new approach to enhance feature engineering in machine learning models, aiming to improve accuracy and efficiency. The study, titled “Mutual Information Reduction Techniques and its Applications in Feature Engineering,” was presented at the 2025 IEEE International Conference on Consumer Electronics.

Feature engineering is crucial for building effective machine learning models as it involves selecting and transforming raw data into features that aid in predictions. Traditionally, mutual information (MI) has been used to identify relevant features by measuring how much one piece of information reveals about another. However, this method often overlooks redundancy among features.

Dr. David Li, senior author of the study and program director of the M.S. in Data Analytics and Visualization, highlighted the importance of minimizing redundancy. “We introduce a new way of thinking: instead of only looking for features with the highest MI scores, our study also focuses on reducing mutual information between the features themselves,” he stated. This approach aims to create a set of features that each add unique value to the model.

The researchers began with an MI matrix to assess shared information among features and applied reduction techniques to eliminate overlaps. They also incorporated Weight of Evidence (WOE), which enhances predictive power in binary classification tasks by capturing subtle data nuances.

Testing their method on a loan default dataset, they successfully reduced redundancy using a brute-force method for parameter tuning. Adding WOE transformation further improved performance, particularly for logistic regression models used in risk management.

This dual approach resulted in a more efficient model with better insights into factors driving loan defaults. The benefits include faster processing due to less redundant data, improved predictive accuracy, and enhanced interpretability—making such models valuable across various fields like finance and healthcare.

Ruixin Chen, lead author and student in the M.S. in Artificial Intelligence program, noted future research possibilities: “Future research could explore automated ways to optimize mutual information reduction, apply the technique to more complex datasets or expand its use in unsupervised learning tasks.”