Data reduction

Data Reduction

1. Data Mining Data Reduction Dr.J.Kalavathi . M.Sc., P.hD ., Assistant Professor, Department of Information Technology, V.V.Vanniaperumal College for Women, Virudhunagar.

2. Data Reduction A database or date warehouse may store terabytes of data.So it may take very long to perform data analysis and mining on such huge amounts of data. Data reduction techniques can be applied to obtain a reduced representation of the data set that is much smaller in volume but still contain critical information.

3. Data Reduction Strategies:- 1 Data Cube Aggregation Aggregation operations are applied to the data in the construction of a data cube. 2 Dimensionality Reduction In dimensionality reduction redundant attributes are detected and removed which reduce the data set size. 3 Data Compression Encoding mechanisms are used to reduce the data set size. 4 Numerosity Reduction In numerosity reduction where the data are replaced or estimated by alternative. 5 Discretisation and concept hierarchy generation Where raw data values for attributes are replaced by ranges or higher conceptual levels.

4. Data Cube Aggregation : This technique is used to aggregate data in a simpler form. For example, imagine that information you gathered for your analysis for the years 2012 to 2014, that data includes the revenue of your company every three months . They involve you in the annual sales, rather than the quarterly average,  So we can summarize the data in such a way that the resulting data summarizes the total sales per year instead of per quarter. It summarizes the data.

5. Attribute Subset Selection : Attribute subset Selection is a technique which is used for data reduction in data mining process. Data reduction reduces the size of data so that it can be used for analysis purposes more efficiently . The data set may have a large number of attributes. But some of those attributes can be irrelevant or redundant. The goal of attribute subset selection is to find a minimum set of attributes such that dropping of those irrelevant attributes does not much affect the utility of data and the cost of data analysis could be reduced. 

6. Methods of Attribute Subset Selection- 1. Stepwise Forward Selection. 2. Stepwise Backward Elimination. 3. Combination of Forward Selection and Backward Elimination. 4. Decision Tree Induction.

7. Stepwise Forward Selection This procedure start with an empty set of attributes as the minimal set. The most relevant attributes are chosen(having minimum p-value) and are added to the minimal set. In each iteration, one attribute is added to a reduced set . Initial attribute Set: {X1, X2, X3, X4, X5, X6 } Initial reduced attribute set: { } Step-1 : {X1} Step-2 : {X1, X2} Step-3 : {X1, X2, X5} Final reduced attribute set: {X1, X2, X5}

8. Stepwise Backward Elimination: Here all the attributes are considered in the initial set of attributes. In each iteration, one attribute is eliminated from the set of attributes whose p-value is higher than significance level. Initial attribute Set: {X1, X2, X3, X4, X5, X6} Initial reduced attribute set: {X1, X2, X3, X4, X5, X6 } Step-1 : {X1, X2, X3, X4, X5} Step-2 : {X1, X2, X3, X5} Step-3 : {X1, X2, X5} Final reduced attribute set: {X1, X2, X5}

9. Combination of Forward Selection and Backward Elimination:  The stepwise forward selection and backward elimination are combined so as to select the relevant attributes most efficiently. This is the most common technique which is generally used for attribute selection. Decision Tree Induction:  This approach uses decision tree for attribute selection. It constructs a flow chart like structure having nodes denoting a test on an attribute. Each branch corresponds to the outcome of test and leaf nodes is a class prediction. The attribute that is not the part of tree is considered irrelevant and hence discarded.

10. Thank You