Overcoming the Main Challenges of Knowledge Discovery through Tendency to the Intelligent Data Analysis
Samaher Al-Janabi
Abstract
Intelligent Data Analysis (IDA) approach proves its ability to deal with small and huge data sets; this ability can be done by generating and developing new methodologies. While; Knowledge Discovery in Database (KDD) aiming to automatic interpretation of large data sets. As a result, we can conclude a strong relationship between two concepts. This study discusses five challenges related to data analytical to making right decision. These challenges include: Missing values; Data scarcity, Data dimensionality reduction, Black box; and Mathematical model. First: One of the important trends in KDD will be the growing importance of data processing. But this point faces problems similar to those of data mining (High dimensional data, missing values imputation and data integration). As explained before, one of the open still problems in estimation missing values methods are how to select the optimal number of nearest neighbors of those values. Second: Data scarcity (insufficient size of dataset) problem is one of the challenges in KDD. Insufficient size of data is very often responsible for the poor performances in learning. Third: The goal of dimension reduction methods is using the correlation structure among the predicator variables to reduction the three main dimensions (features, samples and value of features). But, how we can combination among these three dimensions without lose any important inform is still as one of the challenges in KDD system. Fourth: As we know data mining algorithm are “black box character” for many reasons explained in chapter two. But how we can convert it to system will behaviors and conclusion that can be explained and analyzed in a comprehensible manner (i.e., white box) is remained one of the main challenges of KDD system.