Parsing various file formats
Created for
Iva E. Popova, 2018,
Data Serialization
Data Serialization
Overview
- Data serialization is the concept of converting structured data (lists, dictionaries, objects, etc) into a format that can be stored (for example, in a file) or transmitted and reconstructed later.
- The native data serialization module for Python is called Pickle.
- pickle — Python object serialization
Example
import pickle
# let's serialize a simple dict
prices = { 'apples': 2.50, 'oranges': 1.90, 'bananas': 2.40 }
#convert the object to a serialized string
serialized_prices = pickle.dumps( prices )
print(serialized_prices)
#de-serialize (unpickle) an object
received_prices = pickle.loads( serialized_prices )
print(received_prices)
Notes
- For storing data in databases and archival storage, you’re probably better off using a more standard data encoding, such as JSON, XML, CSV
Parsing JSON
Parsing JSON
Overview
- JSONJavaScriptObjectNotation
JSON is an open-standard file format that uses human-readable text to transmit data objects consisting of attribute–value pairs and array data types (or any other serializable value)JSON @wikipedia
Overview - Examples
examples of JSON data: Examples @JSON Schema
{
"title": "Person",
"type": "object",
"properties": {
"firstName": {
"type": "string"
},
"lastName": {
"type": "string"
},
"age": {
"description": "Age in years",
"type": "integer",
"minimum": 0
}
},
"required": ["firstName", "lastName"]
}
Overview - JSON in Python
- Python standard library provides the
jsonmodule - json — JSON encoder and decoder
Parse JSON string to Python objects.
json.loads() function- Deserialize s (a str, bytes or bytearray instance containing a JSON document) to a Python object using next conversion table.
json.loads() example
import json
from operator import itemgetter
json_str = """
[
{
"name": "apple",
"price": 1.80
},
{
"name": "orange",
"price": 2.10
},
{
"name": "bananas",
"price": 1.60
}
]
"""
#read json from string
json_data = json.loads(json_str)
# print the list sorted by "price"
for i in sorted(json_data,key=itemgetter("price")):
print(i)
Parse JSON file to Python objects.
json.load() function- Deserialize fp (a .read()-supporting file-like object containing a JSON document) to a Python object using next conversion table.
json.load() sample file
[
{
"name": "apple",
"price": 1.80
},
{
"name": "orange",
"price": 2.10
},
{
"name": "bananas",
"price": 1.60
}
]
json.load() example
import json
from operator import itemgetter
json_file = "sample.json"
#read json from file
with open(json_file) as f:
json_data = json.load(f)
for i in sorted(json_data,key=itemgetter("price")):
print(i)
Convert Python objects to JSON
json.dump() function- Serialize obj as a JSON formatted stream to fp (a .write()-supporting file-like object) using next conversion table.
json.dumps() function- Same as
dump(), but serialize to string
json.dumps() - list example
import json
mylist = [1,2,3]
matrix = [
[1,2,3],
[4,5,6],
[7,8,9],
]
print('List :', json.dumps(mylist))
print('Matrix :', json.dumps(matrix))
json.dumps() - indented list example
import json
mylist = [1,2,3]
matrix = [
[1,2,3],
[4,5,6],
[7,8,9],
]
print('List :', json.dumps(mylist,indent=2))
print('Matrix :', json.dumps(matrix,indent=2))
Parsing CSV
Parsing CSV
Overview
- CSVCommaSeparatedValues
- the most common import and export format for spreadsheets and databases.
- For CSV parsing Python provides the built-in csv module
- csv — CSV File Reading and Writing
Parsing CSV - example 1 - sample_data.csv
Symbol,Price,Date,Time,Change,Volume
"AA",39.48,"6/11/2007","9:36am",-0.18,181800
"AIG",71.38,"6/11/2007","9:36am",-0.15,195500
"AXP",62.58,"6/11/2007","9:36am",-0.46,935000
"BA",98.31,"6/11/2007","9:36am",+0.12,104800
"C",53.08,"6/11/2007","9:36am",-0.25,360900
"CAT",78.29,"6/11/2007","9:36am",-0.23,225400
Parsing CSV data - example 1 - code
import csv
with open('sample_data.csv') as f:
# read CSV file:
f_csv = csv.reader(f)
# skip header row:
headers = next(f_csv)
# print CSV data, sorted by "Price"
for row in sorted(f_csv, key=lambda a:a[1]):
print(row)
#OUTPUT
#['AA', '39.48', '6/11/2007', '9:36am', '-0.18', '181800']
#['C', '53.08', '6/11/2007', '9:36am', '-0.25', '360900']
#['AXP', '62.58', '6/11/2007', '9:36am', '-0.46', '935000']
#['AIG', '71.38', '6/11/2007', '9:36am', '-0.15', '195500']
#['CAT', '78.29', '6/11/2007', '9:36am', '-0.23', '225400']
#['BA', '98.31', '6/11/2007', '9:36am', '+0.12', '104800']
Parsing CSV data - example 2 - TheCure_Discography.csv
Given is next CSV file: The Cure - Discography
Album Title;Label;Year;
Three Imaginary Boys;Fiction Records;1979
Boys Don't Cry;PVC Records, Fiction Records;1980
Seventeen Seconds;Fiction Records, Polydor;1980
Faith And 'Carnage Visors';Fiction Records;1981
Faith;Fiction Records;1981
Pornography;Fiction Records;1982
The Walk;Sire;1983
The Top;Fiction Records;1984
Concert (The Cure Live) And Curiosity (Cure Anomalies 1977 -1984);Fiction Records;1984
Concert - The Cure Live;Fiction Records;1984
In Concert - 371 (LP, Transcription);BBC Transcription Services;1985
The Head On The Door;Fiction Records;1985
In Concert-393 (LP, Transcription);BBC Transcription Services;1986
The Cure / The Divinyls* - The King Biscuit Flower Hour (LP, Transcription + LP, S/Sided);DIR Broadcasting;1986
The Call / The Cure - King Biscuit Flower Hour (Transcription + LP + LP, S/Sided);DIR Broadcasting;1986
The Cure, The Outfield - King Biscuit Flower Hour (2xLP, Transcription);King Biscuit Flower Hour;1987
Kiss Me Kiss Me Kiss Me;Fiction Records;1987
The Cure / Europe (2) - In Concert (2xLP, Transcription);Westwood One;1987
Kiss Me Kiss Me Kiss Me Interview (LP, Promo);Fiction Records, Polydor;1987
In Concert-427 (LP, Transcription);BBC Transcription Services;1988
Disintegration;Fiction Records;1989
Entreat;Fiction Records, Polydor;1989
In Concert-500 (LP, Transcription);BBC Transcription Services;1990
Mixed Up;Fiction Records;1990
An Interview (CD, Promo);Polydor;1990
Parsing CSV data - example 2 - code
Print only albums, produced by "Fiction Records" during the period 1980 - 1989 (both included)
import csv
with open('TheCure_Discography.csv') as f:
f_csv = csv.reader(f, delimiter=';')
headers = next(f_csv)
for row in filter(
lambda l: "Fiction Records" in l[1] and 1980 <= int(l[2]) <= 1989,
f_csv
):
print(row)
#OUTPUT
#["Boys Don't Cry", 'PVC Records, Fiction Records', '1980']
#['Seventeen Seconds', 'Fiction Records, Polydor', '1980']
#["Faith And 'Carnage Visors'", 'Fiction Records', '1981']
#['Faith', 'Fiction Records', '1981']
#['Pornography', 'Fiction Records', '1982']
#['The Top', 'Fiction Records', '1984']
#['Concert (The Cure Live) And Curiosity (Cure Anomalies 1977 -1984)', 'Fiction Records', '1984']
#['Concert - The Cure Live', 'Fiction Records', '1984']
#['The Head On The Door', 'Fiction Records', '1985']
#['Kiss Me Kiss Me Kiss Me', 'Fiction Records', '1987']
#['Kiss Me Kiss Me Kiss Me Interview \u200e(LP, Promo)', 'Fiction Records, Polydor', '1987']
#['Disintegration', 'Fiction Records', '1989']
#['Entreat', 'Fiction Records, Polydor', '1989']
Parsing XML
Parsing XML
Overview
- XMLeXtensibleMarkupLanguage
(XML) is a markup language that defines a set of rules for encoding documents in a format that is both human-readable and machine-readable.XML @wikipedia
Overview
Processing XML in Python:
Parsing XML data - example
<data>
<items>
<item atr1="value1">item1 data</item>
<item atr2="value2">item2 data</item>
</items>
</data>
Parsing XML data - example
import xml.etree.ElementTree as ET
tree = ET.parse('sample_data.xml')
root = tree.getroot()
# print items attributes
print('\nAll attributes:')
for elem in root:
for subelem in elem:
print(subelem.attrib)
# print items data
print('\nAll item data:')
for elem in root:
for subelem in elem:
print(subelem.text)
Parsing XML data - output
All attributes:
{'atr1': 'value1'}
{'atr2': 'value2'}
All item data:
item1 data
item2 data
Exercises
Task1: print_books_by_category.py
The Task
- Given is next JSON file: pythonbooks.revolunet.com.issues.json, which contains information about the books listed in pythonbooks.revolunet.com
- Make a program, that will extract from these data only the "title", "author" and "url" fields for the books labeled as "Advanced", and will print the extracted information, as shown in sample output
Task2: rearange_books_by_category
The task
- Given is the same JSON file: pythonbooks.revolunet.com.issues.json
- You have to perform the same filtering actions, as in Task1, but instead of to print, now you have to create a new, filtered JSON file, like next:rearanged_books_by_category:
{
"Advanced": [
{
"author": "Ian Ozsvald",
"title": "High Performance Python tutorial",
"url": "http://ianozsvald.com/HighPerformancePythonfromTrainingatEuroPython2011_v0.2.pdf"
},
{
"author": "Jan Erik Solem",
"title": "Programming Computer Vision with Python",
"url": "http://programmingcomputervision.com/downloads/ProgrammingComputerVision_CCdraft.pdf"
},
{
"author": "S. Bird, E. Klein...",
"title": "Natural Language Processing with Python",
"url": "http://www.nltk.org/book/"
},
{
"author": "Allen B. Downey",
"title": "Think Complexity",
"url": "http://www.greenteapress.com/complexity/thinkcomplexity.pdf"
},
{
"author": "Allen B. Downey",
"title": "Think Stats",
"url": "http://greenteapress.com/thinkstats/thinkstats.pdf"
},
{
"author": "Allen B. Downey",
"title": "Think Stats 2e",
"url": "http://greenteapress.com/thinkstats2/thinkstats2.pdf"
},
{
"author": "Massimo Di Pierro",
"title": "Annotated Algorithms In Python",
"url": "https://raw.githubusercontent.com/mdipierro/nlib/master/docs/book_numerical.pdf"
},
{
"author": "John Hearty",
"title": "Advanced Machine Learning with Python",
"url": "https://www.packtpub.com/packt/free-ebook/advanced-python-machine-learning"
}
],
"Beginner": [
{
"author": "David Mertz",
"title": "Picking a Python Version: A Manifesto",
"url": "http://www.oreilly.com/programming/free/files/from-future-import-python.pdf"
},
{
"author": "B. Miller & D. Ranum",
"title": "How to Think Like a Computer Scientist: Second Interactive Edition",
"url": "http://interactivepython.org/runestone/static/thinkcspy/index.html"
},
{
"author": "Jeffrey Elkner...",
"title": "How to Think Like a Computer Scientist: Learning with Python 2nd Ed.",
"url": "http://www.openbookproject.net/thinkcs/python/english2e/"
},
{
"author": "Zed A. Shaw",
"title": "Learn Python The Hard Way",
"url": "http://learnpythonthehardway.org/book"
},
{
"author": "Allen B. Downey",
"title": "Think Python",
"url": "http://www.greenteapress.com/thinkpython/thinkpython.pdf"
},
{
"author": "Swaroop C H",
"title": "A byte of Python",
"url": "http://files.swaroopch.com/python/byte_of_python.pdf"
},
{
"author": "Dave Kuhlman",
"title": "Python 101 - Introduction to Python",
"url": "http://www.davekuhlman.org/python_101.html"
},
{
"author": "Jason R. Briggs",
"title": "Snake Wrangling for Kids",
"url": "http://www.briggs.net.nz/snake-wrangling-for-kids.html"
},
{
"author": "John C. Lusth",
"title": "An introduction to Python",
"url": "http://beastie.cs.ua.edu/cs150/book/index.html"
},
{
"author": "Steven F. Lot",
"title": "Building skills in Programming",
"url": "http://www.itmaybeahack.com/homepage/books/nonprog/html/index.html"
},
{
"author": "Steven F. Lot ",
"title": "Building skills in Python",
"url": "http://www.itmaybeahack.com/book/python-2.6/html/index.html"
},
{
"author": "G\u00e9rard Swinnen",
"title": "Programmez avec Python 2",
"url": "http://inforef.be/swi/download/apprendre_python.pdf"
},
{
"author": "G\u00e9rard Swinnen",
"title": "Programmez avec Python 3",
"url": "http://inforef.be/swi/download/apprendre_python3_5.pdf"
},
{
"author": "Kushal Das",
"title": "Python for you and me",
"url": "http://pymbook.readthedocs.io/en/latest/"
},
{
"author": "Patrick Fuchs",
"title": "Python course",
"url": "http://www.dsimb.inserm.fr/~fuchs/python/cours_python.pdf"
},
{
"author": "Jesse Noller",
"title": "A bit of Python & other things.",
"url": "http://jessenoller.com/good-to-great-python-reads/"
},
{
"author": "Google",
"title": "Python Course",
"url": "https://developers.google.com/edu/python/"
},
{
"author": "Josh Cogliati and others",
"title": "Non-Programmer's Tutorial for Python 3",
"url": "http://en.wikibooks.org/wiki/Non-Programmer%27s_Tutorial_for_Python_3"
},
{
"author": "Kenneth Reitz",
"title": "The Hitchhiker\u2019s Guide to Python!",
"url": "https://media.readthedocs.org/pdf/python-guide/latest/python-guide.pdf"
},
{
"author": "Al Sweigart",
"title": "Hacking Secret Ciphers with Python",
"url": "http://inventwithpython.com/hackingciphers.pdf"
},
{
"author": "Kenneth Love",
"title": "Getting Started with Django",
"url": "http://gettingstartedwithdjango.com/"
},
{
"author": "Jody S. Ginther",
"title": "Python 3x Programming (sample)",
"url": " http://www.toonzcat.com/media/pythonfree.pdf"
},
{
"author": "Ra\u00fal Gonz\u00e1lez Duque",
"title": "Python para todos",
"url": "http://edge.launchpad.net/improve-python-spanish-doc/0.4/0.4.0/+download/Python%20para%20todos.pdf"
},
{
"author": "Sat Kumar Tomer",
"title": "Python in Hydrology",
"url": "http://www.greenteapress.com/pythonhydro/pythonhydro.pdf"
},
{
"author": "Leif Azzopardi",
"title": "How to Tango with Django",
"url": "http://www.tangowithdjango.com/book17/"
},
{
"author": "Anand Chitipothu.",
"title": "Python Practice Book",
"url": "http://anandology.com/python-practice-book/index.html"
},
{
"author": "Community",
"title": "Django Girls Tutorial",
"url": "https://www.gitbook.com/download/pdf/book/djangogirls/djangogirls-tutorial?lang=en"
},
{
"author": "John B. Schneider...",
"title": "Algorithmic Problem Solving with Python",
"url": "http://www.eecs.wsu.edu/~schneidj/PyBook/swan.pdf"
},
{
"author": "Sean M. Tracey",
"title": "Make Games with Python",
"url": "https://www.raspberrypi.org/magpi-issues/Essentials_Games_v1.pdf"
},
{
"author": "Pierluigi Riti",
"title": "What You Need to Know about Python",
"url": "https://www.packtpub.com/packt/free-ebook/what-you-need-know-about-python2"
},
{
"author": "Charles R. Severance",
"title": "Python for Everybody",
"url": "https://www.py4e.com/book"
}
],
"Intermediate": [
{
"author": "Mike Pirnat",
"title": "How to Make Mistakes in Python",
"url": "http://www.oreilly.com/programming/free/files/how-to-make-mistakes-in-python.pdf"
},
{
"author": "David Mertz",
"title": "Functional Programming in Python",
"url": "http://www.oreilly.com/programming/free/files/functional-programming-python.pdf"
},
{
"author": "Luiz Eduardo Borges",
"title": "Python para Desenvolvedores (2nd Edition)",
"url": "http://ark4n.files.wordpress.com/2010/01/python_para_desenvolvedores_2ed.pdf"
},
{
"author": "Muhammad Yasoob",
"title": "Intermediate Python",
"url": "https://media.readthedocs.org/pdf/intermediatepythongithubio/latest/intermediatepythongithubio.pdf"
},
{
"author": "B. Miller & D. Ranum",
"title": "Problem Solving with Algorithms and Data Structures Using Python",
"url": "http://interactivepython.org/runestone/static/pythonds/index.html"
},
{
"author": "Mark Pilgrim",
"title": "Dive into Python (2004)",
"url": "http://www.diveintopython.net/download/diveintopython-pdf-5.4.zip"
},
{
"author": "Mark Pilgrim",
"title": "Dive into Python 3",
"url": "https://github.com/downloads/diveintomark/diveintopython3/dive-into-python3.pdf"
},
{
"author": "Kivy",
"title": "Kivy programming Guide",
"url": "https://readthedocs.org/projects/kivy/downloads/"
},
{
"author": "Community",
"title": "Django Tutorial",
"url": "https://media.readthedocs.org/pdf/django/latest/django.pdf"
},
{
"author": "Community",
"title": "Pyramid for Humans",
"url": "docs.pylonsproject.org/projects/pyramid_tutorials/en/latest/index.html"
},
{
"author": "Armin Ronacher",
"title": "Flask microframework",
"url": "http://flask.pocoo.org/docs/tutorial/"
},
{
"author": "Al Sweigart",
"title": "Making games with Python and Pygame",
"url": "http://inventwithpython.com/pygame/chapters/"
},
{
"author": "Fredrik Lundh",
"title": "The Standard Python Library",
"url": "http://effbot.org/zone/librarybook-index.htm"
},
{
"author": "Doug Hellman",
"title": "Python Module of the week",
"url": "https://pymotw.com/2/contents.html"
},
{
"author": "Steven F. Lot ",
"title": "Building skills in OOP",
"url": "http://www.itmaybeahack.com/book/oodesign-python-2.1/html/index.html"
},
{
"author": "By The Community",
"title": "Python Scientific lecture notes",
"url": "http://scipy-lectures.github.com/"
},
{
"author": "Bruno R. Preiss",
"title": "Data Structures and Algorithms with Object-Oriented Design Patterns in Python",
"url": "http://www.brpreiss.com/books/opus7/html/book.html"
},
{
"author": "Lennart Regebro",
"title": "Porting to Python 3: An in-depth guide",
"url": "http://python3porting.com/pdfs/SupportingPython3-screen-1.0-latest.pdf"
},
{
"author": "Cam Davidson-Pilon",
"title": "Probabilistic Programming and Bayesian Methods for Hackers: Using Python and PyMC",
"url": "http://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/"
},
{
"author": "Massimo Di Pierro",
"title": "web2py Complete Manual",
"url": "https://dl.dropboxusercontent.com/u/18065445/web2py/web2py_manual_5th.pdf"
},
{
"author": "Tom De Smedt",
"title": "Modeling Creativity",
"url": "http://www.clips.ua.ac.be/sites/default/files/modeling-creativity.pdf"
},
{
"author": "Harry Percival",
"title": "Test-Driven Development with Python",
"url": "http://chimera.labs.oreilly.com/books/1234000000754/index.html"
},
{
"author": "Agiliq",
"title": "Djen of Django",
"url": "http://agiliq.com/books/djenofdjango/"
},
{
"author": "Ron Zacharski",
"title": "A Programmer's Guide to Data Mining",
"url": "http://guidetodatamining.com/assets/guideChapters/Guide2DataMining.pdf"
},
{
"author": "Al Sweigart",
"title": "Invent Your Own Computer Games with Python",
"url": "http://inventwithpython.com/chapters/"
},
{
"author": "Various authors",
"title": "Biopython",
"url": "http://biopython.org/DIST/docs/tutorial/Tutorial.pdf"
},
{
"author": "David Mertz",
"title": "Text Processing in Python",
"url": "http://gnosis.cx/TPiP/"
},
{
"author": "Various authors",
"title": "Python Cookbook, Third Edition",
"url": "http://chimera.labs.oreilly.com/books/1230000000393/index.html"
},
{
"author": "Robert Picard.",
"title": "Explore Flask",
"url": "http://exploreflask.com"
},
{
"author": "Al Sweigart",
"title": "Automate the Boring Stuff with Python ",
"url": "https://automatetheboringstuff.com/"
},
{
"author": "Ian Ozsvald",
"title": "High Performance Python",
"url": "http://ianozsvald.com/HighPerformancePythonfromTrainingatEuroPython2011_v0.2.pdf"
},
{
"author": "Matt Makai",
"title": "Full Stack Python",
"url": "http://www.fullstackpython.com/table-of-contents.html"
},
{
"author": "Fabrizio Romano",
"title": "Learning Python",
"url": "https://www.packtpub.com/packt/free-ebook/learning-python"
},
{
"author": "Gabriel A. C\u00e1nepa",
"title": "What You Need to Know about Machine Learning",
"url": "https://www.packtpub.com/packt/free-ebook/what-you-need-know-about-machine-learning2"
},
{
"author": "Willi Richert, Luis Pedro Coelho",
"title": "Building Machine Learning Systems with Python",
"url": "https://www.packtpub.com/packt/free-ebook/python-machine-learning-algorithms"
},
{
"author": "Hector Cuesta",
"title": "Practical Data Analysis",
"url": "https://www.packtpub.com/packt/free-ebook/practical-data-analysis"
},
{
"author": "Tim Cox",
"title": "Raspberry Pi Cookbook for Python Programmers",
"url": "https://www.packtpub.com/packt/free-ebook/python-raspberry-pi-cookbook"
},
{
"author": "Caleb Hattingh",
"title": "20 Python libraries You Aren't Using (But Should)",
"url": "http://www.oreilly.com/programming/free/20-python-libraries-you-arent-using-but-should.csp"
}
]
}
Task2 -Template: rearange_books_by_category
- You can use the following template file for your code: rearange_books_by_category.py
import json
def rearange_books_by_category(books_json):
"""rearange_books_by_category.
Returns JSON with following structure:
{
"Beginner":[
{
"title":
"author":
"url"
}
],
"Intermediate":[
{
"title":
"author":
"url"
}
],
}
"""
rearanged_books = {}
#####################################
# YOUR CODE IS HERE #
#####################################
return rearanged_books
def main():
json_file = "pythonbooks.revolunet.com.issues.json"
#read json data from file
with open(json_file) as f:
json_data = json.load(f)
rearanged_books = rearange_books_by_category(json_data)
# print json data:
print(json.dumps(rearanged_books,indent=4,sort_keys=True))
if __name__ == '__main__':
main()
Task2 - A possible, pure Python, Solution
def rearange_books_by_category(books_json):
"""re-arange books by category.
returns:
books =
{
"Beginner":[
{
"title":
"author":
"url"
}
],
"Intermediate":[
{
"title":
"author":
"url"
}
],
}
"""
rearanged_books = {}
for book in books_json['books']:
category = book['level']
book_data = {
"title": book['title'],
"author":book['author'],
"url":book['url']
}
if category in rearanged_books.keys():
rearanged_books[category].append(book_data)
else:
rearanged_books[category] = [book_data]
return rearanged_books
Submission
- PLease, prefix your filenames/archive with your name initials, before sending.
- For instance: iep_task1.py or iep_tasks.rar
- Send files to progressbg.python.course@gmail.com
These slides are based on
customised version of
framework