7_Files_python_fileprocessing_program.pdf

File processing in python

1. Python Programming
Using Problem Solving Approach
Reema Thareja
1
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.
1

2. 2
CHAPTER 7
File Handling
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.

3. File
3
Afileisacollectionofdatastoredonasecondarystoragedevicelikeharddisk.
Afileisbasicallyusedbecausereal-lifeapplicationsinvolvelargeamountsofdataandinsuchsituationsthe
consoleorientedI/Ooperationsposetwomajorproblems:
•First,itbecomescumbersomeandtimeconsumingtohandlehugeamountofdatathroughterminals.
•Second,whendoingI/Ousingterminal,theentiredataislostwheneithertheprogramisterminatedor
computeristurnedoff.Therefore,itbecomesnecessarytostoredataonapermanentstorage(thedisks)and
readwhenevernecessary,withoutdestroyingthedata.
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.

4. File Path
4
Filesthatweusearestoredonastoragemediumliketheharddiskinsuchaway
thattheycanbeeasilyretrievedasandwhenrequired.
Everyfileisidentifiedbyitspaththatbeginsfromtherootnodeortherootfolder.In
Windows,C:\(alsoknownasCdrive)istherootfolderbutyoucanalsohaveapath
thatstartsfromotherdriveslikeD:\,E:\,etc.Thefilepathisalsoknownaspathname.
RelativePathandAbsolutePath
Afilepathcanbeeitherrelativeorabsolute.Whileanabsolutepathalwayscontains
therootandthecompletedirectorylisttospecifytheexactlocationthefile,relative
pathneedstobecombinedwithanotherpathinordertoaccessafile.Itstartswith
respecttothecurrentworkingdirectoryandthereforelackstheleadingslashes.For
example, C:\Students\Under Graduate\BTech_CS.docx but Under
Graduate\BTech_CS.docxisarelativepathasonlyapartofthecompletepathis
specified.
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.

5. ASCII Text Files
5
Atextfileisastreamofcharactersthatcanbesequentiallyprocessedbyacomputerinforwarddirection.Forthis
reasonatextfileisusuallyopenedforonlyonekindofoperation(reading,writing,orappending)atanygiventime.
Becausetextfilescanprocesscharacters,theycanonlyreadorwritedataonecharacteratatime.InPython,atext
streamistreatedasaspecialkindoffile.
Dependingontherequirementsoftheoperatingsystemandontheoperationthathastobeperformed(read/write
operation)onthefile,thenewlinecharactersmaybeconvertedtoorfromcarriage-return/linefeedcombinations.
Besidesthis,othercharacterconversionsmayalsobedonetosatisfythestoragerequirementsoftheoperating
system.However,theseconversionsoccurtransparentlytoprocessatextfile.Inatextfile,eachlinecontainszeroor
morecharactersandendswithoneormorecharacters
Anotherimportantthingisthatwhenatextfileisused,thereareactuallytworepresentationsofdata-internalor
external.Forexample,anintegervaluewillberepresentedasanumberthatoccupies2or4bytesofmemory
internallybutexternallytheintegervaluewillberepresentedasastringofcharactersrepresentingitsdecimalor
hexadecimalvalue. © OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.

6. Binary Files
6
Abinaryfileisafilewhichmaycontainanytypeofdata,encodedinbinaryformforcomputerstorageand
processingpurposes.Itincludesfilessuchaswordprocessingdocuments,PDFs,images,spreadsheets,videos,
zipfilesandotherexecutableprograms.Likeatextfile,abinaryfileisacollectionofbytes.Abinaryfileis
alsoreferredtoasacharacterstreamwithfollowingtwoessentialdifferences.
•Abinaryfiledoesnotrequireanyspecialprocessingofthedataandeachbyteofdataistransferredtoor
fromthediskunprocessed.
•Pythonplacesnoconstructsonthefile,anditmaybereadfrom,orwrittento,inanymannerthe
programmerwants.
Whiletextfilescanbeprocessedsequentially,binaryfiles,ontheotherhand,canbeeitherprocessed
sequentiallyorrandomlydependingontheneedsoftheapplication.
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.

7. The Open() Function
7
Beforereadingfromorwritingtoafile,youmustfirstopenitusingPython’sbuilt-inopen()function.This
functioncreatesafileobject,whichwillbeusedtoinvokemethodsassociatedwithit.Thesyntaxofopen()is:
fileObj=open(file_name[,access_mode])
Here,
file_nameisastringvaluethatspecifiesnameofthefilethatyouwanttoaccess.
access_modeindicatesthemodeinwhichthefilehastobeopened,i.e.,read,write,append,etc.
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.
Example:

8. The open() Function –Access Modes
8
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.

9. The File Object Attributes
9
Onceafileissuccessfullyopened,afileobjectisreturned.Usingthisfileobject,youcaneasilyaccessdifferent
typeofinformationrelatedtothatfile.Thisinformationcanbeobtainedbyreadingvaluesofspecific
attributesofthefile.
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.
Example:

10. The close () Method
10
Theclose()methodisusedtoclosethefileobject.Onceafileobjectisclosed,youcannotfurtherreadfrom
orwriteintothefileassociatedwiththefileobject.Whileclosingthefileobjecttheclose()flushesany
unwritteninformation.Although,Pythonautomaticallyclosesafilewhenthereferenceobjectofafileis
reassignedtoanotherfile,butasagoodprogramminghabityoushouldalwaysexplicitlyusetheclose()
methodtocloseafile.Thesyntaxofclose()isfileObj.close()
Theclose()methodfreesupanysystemresourcessuchasfiledescriptors,filelocks,etc.thatare
associatedwiththefile.Moreover,thereisanupperlimittothenumberoffilesaprogramcanopen.Ifthat
limitisexceededthentheprogrammayevencrashorworkinunexpectedmanner.Thus,youcanwaste
lotsofmemoryifyoukeepmanyfilesopenunnecessarilyandalsorememberthatopenfilesalwaysstand
achanceofcorruptionanddataloss.
Oncethefileisclosedusingtheclose()method,anyattempttousethefileobjectwillresultinanerror.
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.

11. The write() and writelines() Methods
11
Thewrite()methodisusedtowriteastringtoanalreadyopenedfile.Ofcoursethisstringmayinclude
numbers,specialcharactersorothersymbols.Whilewritingdatatoafile,youmustrememberthatthe
write()methoddoesnotaddanewlinecharacter('\n')totheendofthestring.Thesyntaxofwrite()method
is:fileObj.write(string)
Thewritelines()methodisusedtowritealistofstrings.
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.
Examples:

12. append() Method
12
Onceyouhavestoredsomedatainafile,youcanalwaysopenthatfileagaintowritemoredataorappend
datatoit.Toappendafile,youmustopenitusing'a'or'ab'modedependingonwhetheritisatextfileora
binaryfile.Notethatifyouopenafilein'w'or'wb'modeandthenstartwritingdataintoit,thenits
existingcontentswouldbeoverwritten.Soalwaysopenthefilein'a'or'ab'modetoaddmoredatato
existingdatastoredinthefile.
Appendingdataisespeciallyessentialwhencreatingalogofeventsorcombiningalargesetofdatainto
onefile.
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.
Example:

13. The read() and readline() Methods
13
Theread()methodisusedtoreadastringfromanalreadyopenedfile.Assaidbefore,thestringcaninclude,
alphabets,numbers,charactersorothersymbols.Thesyntaxofread()methodisfileObj.read([count])
Intheabovesyntax,countisanoptionalparameterwhichifpassedtotheread()methodspecifiesthe
numberofbytestobereadfromtheopenedfile.Theread()methodstartsreadingfromthebeginningofthe
fileandifcountismissingorhasanegativevaluethen,itreadstheentirecontentsofthefile(i.e.,tilltheend
offile).
Thereadlines()methodisusedtoreadallthelinesinthefile
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.
Example:

14. Opening Files using “with” Keyword
14
Itisgoodprogramminghabittousethewithkeywordwhenworkingwithfileobjects.Thishastheadvantage
thatthefileisproperlyclosedafteritisusedevenifanerroroccursduringreadorwriteoperationoreven
whenyouforgettoexplicitlyclosethefile.
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.
Examples:

15. Splitting Words
15
Pythonallowsyoutoreadline(s)fromafileandsplitstheline(treatedasastring)basedonacharacter.By
default,thischaracterisspacebutyoucanevenspecifyanyothercharactertosplitwordsinthestring.
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.
Example:

16. Some Other Useful File Methods
16
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.

17. File Positions
17
Witheveryfile,thefilemanagementsystemassociatesapointeroftenknownasfilepointerthatfacilitatethe
movementacrossthefileforreadingand/orwritingdata.Thefilepointerspecifiesalocationfromwherethe
currentreadorwriteoperationisinitiated.Oncetheread/writeoperationiscompleted,thepointeris
automaticallyupdated.
Pythonhasvariousmethodsthattellsorsetsthepositionofthefilepointer.Forexample,thetell()method
tellsthecurrentpositionwithinthefileatwhichthenextreadorwriteoperationwilloccur.Itisspecifiedas
numberofbytesfromthebeginningofthefile.Whenyoujustopenafileforreading,thefilepointeris
positionedatlocation0,whichisthebeginningofthefile.
Theseek(offset[,whence])methodisusedtosetthepositionofthefilepointerorinsimplerterms,move
thefilepointertoanewlocation.Theoffsetargumentindicatesthenumberofbytestobemovedandthe
fromargumentspecifiesthereferencepositionfromwherethebytesaretobemoved.
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.

18. File Positions
18
Thereferencepointisselectedbythewhenceargument.Itacceptsthreevalues:
0:setsthereferencepointatthebeginningofthefile
1:setsthereferencepointatthecurrentfileposition
2:setsthereferencepointattheendofthefile
•Seeking from the current position (whence=1)or end (whence=2)is not allowedunless the file is
opened in binary mode("rb", "wb", etc.).
•Only seek(offset, 0)(absolute positioning) is allowed in text mode.
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.

19. File Positions -Example
19
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.

20. Reading data files
So far, we've learned how to work with regular text files. However, sometimes data comes in other
formats like CSV, JSON etc.,
Its common for data professionals to retrieve required information and manipulate the content of CSV
and other files.
20
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.

21. HANDLING CSV FILES
Python offers several libraries to facilitate this task, such as csv, pandas, and built-in functions like open()
for general file handling.
#Reading CSV file
# Open the CSV file
with open('data.csv', 'r') as file:
# Read all lines from the file
lines = file.readlines()
# Process each line
for line in lines:
# Remove newline character and split the line by comma
values = line.strip().split(',')
# Print each value
print(values)
21
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.

22. HANDLING CSV FILES
We can also readCSV files using the csvmodule:
import csv
# Open the CSV file
with open('data.csv', newline='') as csvfile:
# Create a CSV reader object
csvreader= csv.reader(csvfile)
# Read each row in the CSV file
for row in csvreader:
print(row) # Each row is a list of values
22
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.

23. Renaming and Deleting Files
23
TheosmoduleinPythonhasvariousmethodsthatcanbeusedtoperformfile-processingoperationslike
renaminganddeletingfiles.Tousethemethodsdefinedintheosmodule,youshouldfirstimportitinyour
programthencallanyrelatedfunctions.
Therename()Method:Therename()methodtakestwoarguments,thecurrentfilenameandthenew
filename.Itssyntaxis:os.rename(old_file_name,new_file_name)
The remove() Method: This method can be used to delete file(s). The method takes a filename (name of the
file to be deleted) as an argument and deletes that file. Its syntax is: os.remove(file_name)
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.
Examples:

24. Directory Methods
24
Themkdir()Method:Themkdir()methodoftheOSmoduleisusedtocreatedirectoriesinthecurrent
directory.Themethodtakesthenameofthedirectory(theonetobecreated)asanargument.Thesyntaxof
mkdir()is,os.mkdir("new_dir_name")
Thegetcwd()Method:Thegetcwd()methodisusedtodisplaythecurrentworkingdirectory(cwd).
os.getcwd()
Thechdir()Method:Thechdir()methodisusedtochangethecurrentdirectory.Themethodtakesthename
ofthedirectorywhichyouwanttomakethecurrentdirectoryasanargument.Itssyntaxis
os.chdir("dir_name")
Thermdir()Method:Thermdir()methodisusedtoremoveordeleteadirectory.Forthis,itacceptsnameof
thedirectorytobedeletedasanargument.However,beforeremovingadirectory,itshouldbeabsolutely
emptyandallthecontentsinitshouldberemoved.Thesyntaxofremove()methodisos.rmdir(("dir_name")
Themakedirs()method:Themethodmkdirs()isusedtocreatemorethanonefolder.
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.

25. Directory Methods -Examples
25
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.

26. Methods from the osModule
26
Theos.path.abspath()methodusesthestringvaluepassedtoittoformanabsolutepath.Thus,itisanother
waytoconvertarelativepathtoanabsolutepath
Theos.path.isabs(path)methodacceptsafilepathasanargumentandreturnsTrueifthepathisanabsolute
pathandFalseotherwise.
Theos.path.relpath(path,start)methodacceptsafilepathandastartstringasanargumentandreturnsa
relativepaththatbeginsfromthestart.Ifstartisnotgiven,thecurrentdirectoryistakenasstart.
Theos.path.dirname(path)Methodreturnsastringthatincludeseverythingspecifiedinthepath(passedas
argumenttothemethod)thatcomesbeforethelastslash.
Theos.path.basename(path)Methodreturnsastringthatincludeseverythingspecifiedinthepath(passedas
argumenttothemethod)thatcomesafterthelastslash.
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.

27. Methods from the osModule
27
Theos.path.split(path)Method:Thismethodacceptsafilepathandreturnsitsdirectorynameaswellasthe.
Soitisequivalenttousingtwoseparatemethodsos.path.dirname()andos.path.basename()
Theos.path.getsize(path)Method:Thismethodreturnsthesizeofthefilespecifiedinthepathargument.
Theos.listdir(path)Method:Themethodreturnsalistoffilenamesinthespecifiedpath.
Theos.path.exists(path)Method:Themethodasthenamesuggestsacceptsapathasanargumentand
returnsTrueifthefileorfolderspecifiedinthepathexistsandFalseotherwise.
Theos.path.isfile(path)Method:Themethodasthenamesuggestsacceptsapathasanargumentand
returnsTrueifthepathspecifiesafileandFalseotherwise.
Theos.path.isdir(path)Method:Themethodasthenamesuggestsacceptsapathasanargumentandreturns
TrueifthepathspecifiesafolderandFalseotherwise.
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.

28. Methods from the osModule —Examples
28
© OXFORD UNIVERSITY PRESS 2017. ALL RIGHTS RESERVED.