1.CSV stands for Comma-Separated Values.

CSV stands for Comma-Separated Values.�It’s a plain text file used to store tabular data (like spreadsheets

1. UNIT - I I Data Extraction Fundamentals Introduction to Tabular Formats Parsing CSV Parsing XLS with XLRD Parsing XML Introductin to JSON Getting Data into MongoDB MongoDB - CURD Database Creation, Update, Delete, Read using MongoDB Operators - $gt, $lt, $exists, $regex Querying Arrays and using $in and $all operators Changing Entries $update, $set, $unset

2. Data Extraction Fundamentals - ETL Extraction  gathers data from one or more sources. The process of extracting data includes locating and identifying the relevant data, then preparing to be transformed and loaded. Transformation  is where data is sorted and organized. Cleansing — such as removing missing values — also happens during this step. Depending on the destination you choose, data transformation could include data typing, JSON structures, object names, and time zones to ensure compatibility with the data destination. Loading  is the last step, where the transformed data is delivered to a central repository for immediate or future analysis.

3. Data Extraction Fundamentals Introduction to Tabular Format One of the simplest methods used to analyze the data and to display the data is in tabular form. In the tabular form, you get a systematic arrangement of rows and columns. The first column is used to indicate the titles and the first row is also used to indicate the same. It is very accurate as well as an easy method to display the data.

4. Data Extraction Fundamentals XML - D esigned to be both human- and machine-readable. The design goals of XML emphasize simplicity, generality, and usability across the Internet. Parse XML File

5. Data Extraction Fundamentals Parse XSL with XLRD Excel files are spreadsheet documents in which data is stored in form of tables. These files can be read in python by using the XLRD module. The retrieved data can also be filtered according to users choice. import xlrd workbook = xlrd.open_workbook (‘filename.xlsx’) worksheet = workbook.sheet_by_index ( sheet_index ) worksheet.cell_value ( row,column )

6. Data Extraction Fundamentals

7. CURD - MongoDB The  Create  operation is used to insert new documents in the MongoDB database. The  Read  operation is used to query a document in the database. The  Update  operation is used to modify existing documents in the database. The  Delete  operation is used to remove documents in the database.

8. CURD - MongoDB Create  operation db.collection.insertOne() db.collection.insertMany()

9. CURD - MongoDB Create operation db.RecordsDB.insertOne ({ name: "Marsh", age: "6 years", species: "Dog", ownerAddress : "380 W. Fir Ave", chipped: true })

10. CURD - MongoDB

11. CURD - MongoDB Create operation db.RecordsDB.insertMany ([{ name: "Marsh", age: "6 years", species: "Dog", ownerAddress : "380 W. Fir Ave", chipped: true}, { name: " Kitana ", age: "4 years", species: "Cat", ownerAddress : "521 E. Cortland", chipped: true}])

12. CURD - MongoDB

13. CURD - MongoDB Read   operation Operations allow you to supply special query filters and criteria that let you specify which documents you want.  db.collection.find() db.collection.findOne()

14. CURD - MongoDB Read  operation db.RecordsDB.find () db.RecordsDB.find ({"species":"Cat"}) db.{collection}. findOne ({query}, {projection})

15. CURD - MongoDB Update   operation  Operations operate on a single collection, and they are atomic at a single document level. An update operation takes filters and criteria to select the documents you want to update. db.collection.updateOne() - update a currently existing record and change a single document with an update operation. db.collection.updateMany() - update multiple items by passing in a list of items db.collection.replaceOne() - replace a single document in the specified collection.

16. CURD - MongoDB Update   operation   db.RecordsDB.updateOne ({name: "Marsh"}, {$set:{ ownerAddress : "451 W. Coffee St. A204"}}) db.RecordsDB.updateMany ({species:"Dog"}, {$set: {age: "5"}}) db.RecordsDB.replaceOne ({name: "Kevin"}, {name: "Maki"})

17. CURD - MongoDB Delete   operation   db.collection.deleteOne() -  remove a document from a specified collection on the MongoDB server db.collection.deleteMany() - method used to delete multiple documents from a desired collection with a single delete operation. EG: db.RecordsDB.deleteOne ({name:"Maki"}) db.RecordsDB.deleteMany ({species:"Dog"})

18. MongoDB - OPERATORS Offers different types of operators that can be used to interact with the database. Operators are special symbols or keywords that inform a compiler or an interpreter to carry out mathematical or logical operations. The query operators enhance the functionality of MongoDB by allowing developers to create complex queries to interact with data sets that match their applications.

19. MongoDB - OPERATORS Operators add Conditions { <fieldName1> : { <operator1> : <value1> }, …. } EG: {“ favfruit ” : {“$ne”: “apple”}} {“age” : {“$ gt ”: 25}} {“ eyeColor ” : {“$in”: [“blue”, “green”]}}

20. Comparison- OPERATORS $ gt Matches if values are greater than to the given value. $ lt Matches if values are lesser than to the given value. $ gte Matches if values are greater or equal to the given value. $ lte Matches if values are less or equal to the given value. $in Matches any of the values in an array. $ nin Matches none of the values specified in an array. $ eq Matches values that are equal to the given value. $ne Matches values that are not equal to the given value.

21. Comparison- OPERATORS $ gt Matches if values are greater than the given value. Syntax:  { field: { $ gt : value } } $ lt Matches if values are less than the given value. Syntax:  { field: { $ lt : value } }

22. MongoDB - OPERATORS db.inventory.insertMany ( [ { "item" : "nuts" , "quantity" : 30, "carrier" : { "name" : " Shipit " , "fee" : 3 } }, { "item" : "bolts" , "quantity" : 50, "carrier" : { "name" : " Shipit " , "fee" : 4 } }, { "item" : "washers" , "quantity" : 10, "carrier" : { "name" : " Shipit " , "fee" : 1 } } ] )

23. MongoDB - OPERATORS db.inventory.find ( { quantity: { $ gt : 20 } } ) Example output: { _id: ObjectId ("61ba25cbfe687fce2f042414"), item: 'nuts' , quantity: 30, carrier: { name: ' Shipit ' , fee: 3 } }, { _id: ObjectId ("61ba25cbfe687fce2f042415"), item: 'bolts' , quantity: 50, carrier: { name: ' Shipit ' , fee: 4 } }

24. MongoDB - OPERATORS db.inventory.find ( { quantity: { $ lt : 20 } } ) Example output: { _id: ObjectId ("61ba634dfe687fce2f04241f"), item: 'washers' , quantity: 10, carrier: { name: ' Shipit ' , fee: 1 } }

25. ELEMENT- OPERATORS Element Operators The element query operators are used to identify documents using the fields of the document.  $exists Matches documents that have the specified field. Syntax :  { field: { $exists: < boolean > } } When < boolean > is true,  $exists  matches the documents that contain the field, including documents where the field value is null. If < boolean > is false, the query returns only the documents that do not contain the field .

26. MongoDB - OPERATORS name:{$exists: true} name:{$exists: false} db.inventory.find ( { qty: { $exists: true}}) .count() db.inventory.find ( { qty: { $exists: true, $ nin : [ 5, 15 ] } } ) .count()

27. MongoDB - OPERATORS { a: 5, b: 5, c: null } { a: 3, b: null, c: 8 } { a: null, b: 3, c: 9 } { a: 1, b: 2, c: 3 } { a: 2, c: 5 }     { a: 3, b: 2 } { a: 4 } { b: 2, c: 4 } { b: 2 } { c: 6 } db.records.find ( { a: { $exists: true } } db.records.find ( { b: { $exists: false } } )

28. MongoDB - OPERATORS Output 1: { a: 5, b: 5, c: null } { a: 3, b: null, c: 8 } { a: null, b: 3, c: 9 } { a: 1, b: 2, c: 3 } { a: 2, c: 5 } { a: 3, b: 2 } { a: 4 }    Output 2: { a: 2, c: 5 } { a: 4 } { c: 6 }

29. EVALUATION - OPERATORS The MongoDB evaluation operators can evaluate the overall data structure or individual field in a document. $ regex Select documents that match the given regular expression.  SYNTAX: {<fieldname>: {“$ regex ”: /pattern/<options>}} This operator provides regular expression capabilities for pattern matching strings in the queries. Operator is used to search for the given string in the specified collection.

30. EVALUATION- OPERATORS Collection: employee { name: "Tony", position: "Backend developer" } { name: "Bruce", position: "frontend developer“ } { name: "Nick", position: "HR Manager“ } Query db.employee.find ({ position : {$ regex :"developer"}})

31. EVALUATION- OPERATORS Collection: employee { name: "Tony", position: "Backend developer" } { name: "Bruce", position: "frontend developer“ } { name: "Nick", position: "HR Manager“ } db.employee.find ({ position : {$ regex :/ ger / i } }) db.employee.find ({ name : {$ regex :/ Bru /}})

32. MongoDB - OPERATORS

33. Querying Arrays - Using $in and $all operators MongoDB’s   flexible structure makes it a popular choice for  storing  and  managing  diverse data. In  MongoDB , data is stored in collections and collections have documents that support data types like  strings ,  numbers ,  objects , and most important arrays. Arrays in MongoDB allow the users to store data in an ordered form. Efficiently querying array elements is crucial for developers to extract meaningful information from the databases.

34. $in operators The  $in  operator selects the documents where the value of a field equals any value in the specified array. { field: { $in: [<value1>, <value2>, ... < valueN > ] } } If the field holds an array, then the  $in  operator selects the documents whose field holds an array that contains at least one element that matches a value in the specified array (for example, <value1>, <value2>, and so on).

35. $in operators db.inventory.insertMany ( [ { "item" : "Pens" , "quantity" : 350, "tags" : [ "school" , "office" ] }, { "item" : "Erasers" , "quantity" : 15, "tags" : [ "school" , "home" ] }, { "item" : "Maps" , "tags" : [ "office" , "storage" ] }, { "item" : "Books" , "quantity" : 5, "tags" : [ "school" , "storage" , "home" ] }] ) db.inventory.find ( { quantity: { $in: [ 5, 15 ] } })

36. $all operators The  $all  operator selects the documents where the value of a field is an array that contains all the specified elements. Syntax: { <field>: { $all: [ <value1> , <value2> ... ] } } { arrayField : {$all: [element1, element2]} }

37. $all operators db.products.insertMany ([ { "_id" : 1, "name" : " xPhone ", "price" : 799, "color" : [ "white", "black" ]} , { "_id" : 2, "name" : " xTablet ", "price" : 899, "color" : [ "white", "black", "purple" ]}, { "_id" : 3, "name" : " SmartTablet ", "price" : 899, "color" : [ "blue" ]}, { "_id" : 4, "name" : " SmartPad ", "price" : 699, "color" : [ "white", "orange", "gold", "gray" ]}, { "_id" : 5, "name" : " SmartPhone ", "price" : 599, "color" : [ "white", "orange", "gold", "gray" ]} db.products.find ({ color: { $all: ["black", "white"] }

38. $all operators { "_id" : 1, "name" : " xPhone ", "price" : 799, "color" : [ "white", "black" ]} , { "_id" : 2, "name" : " xTablet ", "price" : 899, "color" : [ "white", "black", "purple" ]}, db.products.find ({ color: { $all: ["black", "white"] }

39. Changing Entries Update(), $set & $unset operators The $set operator sets the value of a specified field(s) in a document that matches the search criteria. Syntax {   $set: {     <field1>: <value1>,     ...   } } The $unset operator performs the inverse operation of the  $set operator. That is, it removes the specified field(s) from the document that matches the search criteria. Syntax {   $unset: {     <field1>: <value1>,     ...   } }

40. $set operators EG db.RecordsDB.updateOne ({Designation:“Architect"}, {$set: {salary: 76000}}) db.RecordsDB.updateMany ({Designation:“ FullStack Developer"}, {$set: {salary: 88000}})

41. $unset operators db.products.insertMany ( [ { "item" : "chisel" , “ pid " : "C001" , "quantity" : 4, " instock " : true }, { "item" : "hammer" , “ pid " : "unknown" , "quantity" : 3, " instock " : true }, { "item" : "nails" , “ pid " : "unknown" , "quantity" : 100, " instock " : true }] ) db.products.updateOne ( { pid : "unknown" }, { $unset: { quantity: "", instock : "" } }) db.products.updateMany ( { pid : "unknown" }, { $unset: { quantity: "", instock : "" } })

42. $unset operators - output { item: 'chisel' , pid : 'C001' , quantity: 4, instock : true}, { item: 'hammer' , pid : 'unknown' }, { item: 'nails' , pid : 'unknown' , quantity: 100, instock : true}