Mutual Information for Feature Selection

Mutual Information for Feature Selection
Name
Institutional Affiliation
Mutual Information for Feature Selection
Introduction
While the model selection is critical in learning signals from the provided data, providing the right variables or data is of utmost importance. In machine learning, the model building requires the construction of relevant features or variables through feature engineering, and the resulting data set can then be employed as a statistical input to train a model. While these models are often assumed to be sophisticated and smart algorithms, they are easily fooled by unnecessary clutter and dependencies in the data. Data scientists often make the signals to be easily identifiable by performing feature selection, which is a necessary step in data pre-processing (Huijskens, 2017). According to Zhou, Wang, and Zhu (2022), feature selection is a fundamental pre-processing step in machine learning as it selects only the crucial features by eliminating redundant or irrelevant features from the primary dataset. Battiti (1994) recognizes this pre-processing stage as a critical step where the required number of appropriate features are selected from raw data to impact on learning phase complexity and the achievable generalization performance. While using mutual information (MI) for selecting features in supervised neural net learning, Battiti (1994) notes that although it is important that information in the input vector is sufficient in determining the output class, excess input results in the burdening of the training process and thus lead to the production of neural networks with excess connection weights compared to those needed by the problem at hand. Based on an application-oriented perspective, excessive features lengthen the duration for pre-processing and recognition, regardless of a satisfactory performance in learning and recognition (Battiti, 1994). Data scientists use algorithms that maximize only relevant information while minimizing unnecessary or redundant information.
One of these techniques that have been adopted by machine learning experts and data scientists is mutual information feature selection. In this algorithm, the in-filter feature selection approach is used in assessing the relevancy of a subset of features to predict the largest variable as well as the redundancy based on other variables. Nevertheless, Beraha et al. (2019) note that the existing algorithms are often heuristic and fail to provide any guarantee that they will resolve a proposed problem. This limitation has motivated the authors to propose a novel way of observing the theoretical results that indicate conditional mutual information may occur naturally when handling the ideal regression or classification errors that are achieved by various features or a subset of features. One thing to do before selecting is to remove words that appear only infrequently in one category because they are destined to have high mutual information with one category and low mutual information with the others. Ones show that low word frequency has a great influence on mutual information. If a word is not frequent enough but mainly appears in a certain category, it will have a high level of mutual information, which will bring noise to scree...