Knowledge discovery process

Knowledge discovery process

1. SUBMITTED BY: SHUVRA GHOSH ROLL NO: 07 COURSE: MLIS GUIDED BY: PROF. UDAYAN BHATTACHARYA   DEPARTMENT OF LIBRARY AND INFORMATION SCIENCE JADAVPUR UNIVERSITY Knowledge Discovery Process

2. What is Knowledge discovery? Process of discovering valuable information from a collection of data, or it is the process of converting raw data into useful information . Knowledge discovery is an activity that produces knowledge by discovering it or deriving it from existing information . Knowledge Discovery refers to the overall process of discovering useful knowledge from data, and data mining refers to a particular step in this process.

3. This is information age, day to day creates new data. Data overload creates various problems to us to search proper information. Deductively, Knowledge discovery process helps us to find accurate information . There is an urgent need for a new generation of computational theories and tools to assist humans in extracting useful information (knowledge) from the rapid growing volumes of digital data. Use in various fields such as science and business like Marketing, investment, Fraud Detection, Telecommunication Why do we need knowledge discovery process?

4. Example

5. Database data Data Warehouse Transactional data Other kinds of Data- Time related data Sequence data (historical data records, Stock Exchange) Data streams (Video surveillance, Sensor data) Spatial data (Maps) Hypertext and Multimedia data (Text, Video, Audio) Graph and networked data Engineering design data (auto CAD) Web What kinds of data can be processed?

6. Interactive Iterative Procedure to extract knowledge from data Knowledge being searched for is – implicit previously unknown potentially useful Characteristics

7. Diagram

8. Data Cleaning − in this step, the noise and inconsistent data is removed . Example Parsing the Data. Cleaning is performed for detection Of syntax error. Parser decides the given string of Data is acceptable within data Specification. Steps of Knowledge discovery Process

9. Data Integration − in this step, multiple data sources are combined Example: Retail loan application, commercial loan application, demand deposit application are combined in bank data warehouse. . Data Integration

10. Data Selection − in this step, data relevant to the analysis task are retrieved from the database. Data Selection

11. Data Transformation − in this step, data is transformed or consolidated into forms appropriate for mining by performing summary or aggregation operations . The aggregation operators perform mathematical operations like Average, Aggregate, Count, Max, Min and Sum, on the numeric property of the elements in the collection . Data Transformation

12. Data Mining − in this step, intelligent methods are applied in order to extract data patterns . intelligent methods are – Association Classification Decision tree Clustering Regression Data Mining

13. Example: Bread=>Milk

14. Pattern Evaluation − in this step, data patterns are evaluated. Pattern Evaluation

15. Knowledge Presentation − in this step, knowledge is represented by various visualize tools. Table Chart Graph Knowledge Presentation

16. Knowledge discovery process has three parts Academic Research Models Industrial Models Hybrid Models Knowledge Discovery Process Models

17. The efforts to establish a KDP model were initiated in academia, in the mid-1990s. when the DM field was being shaped, researchers started defining multistep procedures to guide users of DM tools in the complex knowledge discovery world. The two process models developed in 1996 and 1998 are the nine-step model by Fayyad et al. and the eight-step model by Anand and Buchner. Academic Research Models

18. 1.Developing and understanding the application domain . This step includes learning the relevant prior knowledge and the goals of the end user of the discovered knowledge. 2. Creating a target data set . Here the data miner selects a subset of variables (attributes) and data points (examples) that will be used to perform discovery tasks. This step usually includes querying the existing data to select the desired subset. 3. Data cleaning and pre-processing . This step consists of removing outliers, dealing with noise and missing values in the data, and accounting for time sequence information and known changes. 4. Data reduction and projection . This step consists of finding useful attributes by applying dimension reduction and transformation methods, and finding invariant representation of the data. 5. Choosing the data mining task . Here the data miner matches the goals defined in Step 1 with a particular DM method, such as classification, regression, clustering, etc . The Fayyad et al. nine steps KDP model

19. 6 . Choosing the data mining algorithm . The data miner selects methods to search for patterns in the data and decides which models and parameters of the methods used may be appropriate . 7. Data mining . This step generates patterns in a particular representational form, such as classification rules, decision trees, regression models, trends, etc . 8. Interpreting mined patterns . Here the analyst performs visualization of the extracted patterns and models, and visualization of the data based on the extracted models . 9. Consolidating discovered knowledge . The final step consists of incorporating the discovered knowledge into the performance system, and documenting and reporting it to the interested parties. This step may also include checking and resolving potential conflicts with previously believed knowledge.

20. Two representative industrial models are the five-step model by Cabena et al., with support from IBM and the industrial six-step CRISP-DM model, developed by a large consortium of European companies. Industrial Models

21. The CRISP-DM (Cross-Industry Standard Process for Data Mining) was first established in the late 1990s by four companies: Integral Solutions Ltd. (a provider of commercial data mining solutions), NCR (a database provider), DaimlerChrysler (an automobile manufacturer), and OHRA (an insurance company). CRISP-DM Model

22. The CRISP-DM KDP model consists of six steps, which are summarized below: Business understanding : This is the first phase of CRISP-DM process which focuses on and uncovers important factors including success criteria, business and data mining objectives and requirements as well as business terminologies and technical terms. Data understanding : This is the second phase of CRISP-DM process which focuses on data collection, checking quality and exploring of data to get insight of data to form hypotheses for hidden information. Data preparation . This phase focuses on selection and preparation of final data set. This phase may include many tasks records, tables and attributes selection as well as cleaning and transforming of data.  

23. Modeling : This is the fourth phase of CRISP-DM process selection and application of various modeling techniques. Different patterns are set and different models are built for same data mining problem . Evaluation : The processes which focus on evaluation of obtained models and deciding of how to use the results. Interpretation of the model depends upon the algorithm and model can be evaluated to review whether achieves the objectives properly or not . Deployment : This phase focuses on determining the use of obtain knowledge and results. It also focuses on organizing, reporting, presenting the gained knowledge when needed.

24. The development of academic and industrial models has led to the development of hybrid models, i.e., models that combine aspects of both. One such model is a six-step KDP model developed by Cios et al. The main differences and extensions include providing more general, research-oriented description of the steps, introducing a data mining step instead of the modeling step , introducing several new explicit feedback mechanisms, (the CRISP-DM model has only three major feedback sources, while the hybrid model has more detailed feedback mechanisms) and Modification of the last step, since in the hybrid model, the knowledge discovered for a particular domain may be applied in other domains. Hybrid Models

25. Diagram

26. 1. Understanding of the problem domain. This initial step involves working closely with domain experts to define the problem and determine the project goals, identifying key people, and learning about current solutions to the problem. It also involves learning domain-specific terminology. A description of the problem, including its restrictions, is prepared. Finally, project goals are translated into DM goals, and the initial selection of DM tools to be used later in the process is performed. 2. Understanding of the data. This step includes collecting sample data and deciding which data, including format and size, will be needed. Background knowledge can be used to guide these efforts. Data are checked for completeness, redundancy, missing values, plausibility of attribute values, etc. Finally, the step includes verification of the usefulness of the data with respect to the DM goals. D escription of the six steps

27. 3. Preparation of the data. This step concerns deciding which data will be used as input for DM methods in the subsequent step. It involves sampling, running correlation and significance tests, and data cleaning, which includes checking the completeness of data records, removing or correcting for noise and missing values, etc. The cleaned data may be further processed by feature selection and extraction algorithms (to reduce dimensionality), by derivation of new attributes (say, by discretization), and by summarization of data (data granularization). The end results are data that meet the specific input requirements for the DM tools selected in Step 1 . 4. Data mining. Here the data miner uses various DM methods to derive knowledge from pre-processed data.

28. 5. Evaluation of the discovered knowledge. Evaluation includes understanding the results, checking whether the discovered knowledge is novel and interesting, interpretation of the results by domain experts, and checking the impact of the discovered knowledge. Only approved models are retained, and the entire process is revisited to identify which alternative actions could have been taken to improve the results. A list of errors made in the process is prepared . 6. Use of the discovered knowledge. This final step consists of planning where and how to use the discovered knowledge. The application area in the current domain may be extended to other domains. A plan to monitor the implementation of the discovered knowledge is created and the entire project documented. Finally, the discovered knowledge is deployed.

29. Knowledge Discovery in Databases is the process by which a task is identified and performed upon a database in order to extract information about the elements of the database. This process involves first collecting the data to be analysed, cleaning up the data, and reducing it to those features of interest to the process. At which time the tool or tools to be used upon the data are identified. These tools are then used to mine the data for information. Once the information has been created, it must be evaluated as to it efficacy to the process. Any knowledge thereupon gained is then re-incorporated into the process as well as used for purposes outside the scope of the process. This is a very complex process, but it is one that lends itself to a fair degree of automation. As such, it enters into the field of artificial intelligence, not just for the tools it employs, but for the fact that the process tries to re-incorporate the knowledge it has created. Conclusion

30. Thank you