Keyboard shortcuts:
N/СпейсNext Slide
PPrevious Slide
OSlides Overview
ctrl+left clickZoom Element
If you want print version => add '
?print-pdf' at the end of slides URL (remove '#' fragment) and then print.
Like: https://progressbg-python-course.github.io/...CourseIntro.html?print-pdf
Working with various data-exchange formats
Created for
Iva E. Popova, 2016-2025,
Data Serialization
Overview
- Data serialization in Python refers to the process of converting complex data structures, such as objects or dictionaries, into a format that can be easily stored or transmitted and then reconstructed back into their original form when needed.
- Serialization is essential for tasks such as saving data to files, transmitting data over networks, or storing data in databases.
- There are several reasons why data serialization is necessary:
- Persistence: Serialized data can be saved to disk and retrieved later, allowing programs to store and retrieve their state across multiple executions.
- Interoperability: Serialized data can be transmitted between different systems or platforms, allowing communication between applications written in different programming languages or running on different platforms.
- Efficiency: Serialized data is typically more compact than its in-memory representation, making it more efficient to store and transmit.
Serialization in Python
- Python provides several built-in modules for data serialization, including:
- CSV (Comma-Separated Values): a common format for data serialization, particularly for tabular data.
- JSON (JavaScript Object Notation): a lightweight data interchange format commonly used for transmitting data between a server and a web application. Python provides the json module for encoding and decoding JSON data.
- Pickle: a Python-specific serialization format that can serialize and deserialize virtually any Python object. It is highly efficient but Python-specific, meaning that pickled data may not be compatible with other programming languages.
Handling JSON Data in Python
Handling JSON Data in Python
JSON format overview
- JSONJavaScriptObjectNotation
JSON is an open-standard file format that uses human-readable text to transmit data objects consisting of attribute-value pairs and array data types (or any other serializable value)JSON @wikipedia
JSON - Examples
- Example JSON data:
- Note, that JSON data are just string, not a Python's dictionaries, lists, etc.
{
"title": "Person",
"type": "object",
"properties": {
"firstName": {
"type": "string"
},
"lastName": {
"type": "string"
},
"age": {
"description": "Age in years",
"type": "integer",
"minimum": 0
}
},
"required": ["firstName", "lastName"]
}
Python to JSON Conversion table
| Python | JSON | |
|---|---|---|
| 0 | dict | object |
| 1 | list, tuple | array |
| 2 | str | string |
| 3 | int, float, int- & float-derived Enums | number |
| 4 | True | true |
| 5 | False | false |
| 6 | None | null |
Overview - JSON in Python
- Python standard library provides the
jsonmodule (json - JSON encoder and decoder), which provides functions for: - Parsing (Deserializing) JSON into Python objects
- json.load(): parse JSON data from a file-like object and returns a Python object
- json.loads(): parses a JSON string and returns a Python object
- Writing (Serializing) Python objects to JSON
- json.dump(): serializes Python objects to a file-like object in JSON format
- json.dumps(): serializes Python objects to a JSON formatted string
Parse JSON string to Python objects.
json.loads(): De-serialize a string containing a JSON document into a Python objects using next conversion table- the s in loads() stands for String, not plural.
- Example:
import json
json_str = """
[
{
"name": "apple",
"price": 1.80
},
{
"name": "orange",
"price": 2.10
},
{
"name": "bananas",
"price": 1.60
}
]
"""
#read json from string
data = json.loads(json_str)
print(type(data))
# <class 'list'>
Parse JSON file to Python objects.
json.load(): Deserialize а readable (text or binary) file-like object containing a JSON document to a Python object using next conversion table.- Example:
- Let's have next sample json file:
- We can read the data into a Python data structure with:
[
{
"name": "apple",
"price": 1.80
},
{
"name": "orange",
"price": 2.10
},
{
"name": "bananas",
"price": 1.60
}
]
import json
from operator import itemgetter
json_file = "sample.json"
#read json from file
with open(json_file) as f:
json_data = json.load(f)
# use the data, stored in json_data:
for i in sorted(json_data,key=itemgetter("price")):
print(i)
Writing (Serializing) Python objects to JSON string
json.dumps()- serializes Python objects to a JSON formatted string
import json
mylist = [1,2,3]
matrix = [
[1,2,3],
[4,5,6],
[7,8,9],
]
print('List :', json.dumps(mylist))
print('Matrix :', json.dumps(matrix))
Writing (Serializing) Python objects to JSON file
json.dump()- Serialize an object as a JSON formatted file-like object using next conversion table.
- Example: write Python dictionary in file, indented
import json
prices = {
"apples":2.50,
"bananas":1.80,
"strawberry": 3.20
}
with open('json_dump.out', 'w+') as fh:
json.dump(prices, fh, indent=4)
Handling CSV Data in Python
Reading and Writing CSV Files
CSV format overview
- CSV (Comma-Separated Values) format is a widely used data file format that stores tabular data (numbers and text) in plain text.
- Characteristics:
- Simple Text: CSV files are basic text files, making them easy to create, edit, and share.
- Rows: Each line of text in the CSV represents a single row of the table.
- Columns: Within each line, columns are typically separated by commas (but can be other symbol, like ';' or tab).
- Headers: The first line often contains headers, which are the names of the columns.
- Data: Subsequent lines contain data for each row, following the same order as the headers.
- Example of CSV format
Name,Age,Occupation
Alice,30,Engineer
Bob,25,Designer
Reading and Writing CSV Files
- Using the csv Module
- The csv module in Python provides functionality to read and write CSV files.
- We can use the
csv.readerobject to iterate over rows in a CSV file, and thecsv.writerobject to write data to a CSV file. - Example:
import csv
# Reading CSV file
with open('data.csv', 'r') as file:
csv_reader = csv.reader(file)
for row in csv_reader:
print(row)
# Writing CSV file
data = [
['Name', 'Age', 'City'],
['John', 30, 'New York'],
['Alice', 25, 'San Francisco']
]
with open('output.csv', 'w', newline='') as file:
csv_writer = csv.writer(file)
csv_writer.writerows(data)
Parsing CSV: Example
- Let's have next data in "sample_data.csv" file:
- we can parse them with:
Symbol,Price,Date,Time,Change,Volume
"AA",39.48,"6/11/2007","9:36am",-0.18,181800
"AIG",71.38,"6/11/2007","9:36am",-0.15,195500
"AXP",62.58,"6/11/2007","9:36am",-0.46,935000
"BA",98.31,"6/11/2007","9:36am",+0.12,104800
"C",53.08,"6/11/2007","9:36am",-0.25,360900
"CAT",78.29,"6/11/2007","9:36am",-0.23,225400
import csv
# Open the file
with open('sample_data.csv',newline='') as f:
# Create a CSV reader:
csv_reader = csv.reader(f)
# skip header row:
headers = next(csv_reader)
# print CSV data, sorted by "Price"
for row in sorted(csv_reader, key=lambda a:a[1]):
print(row)
Reading and Writing CSV Files with Dictionaries
Reading and Writing CSV Files with Dictionaries
- Using the csv.DictReader and csv.DictWriter Objects
- The csv.DictReader and csv.DictWriter objects provide a way to read and write CSV files using dictionaries.
- This allows accessing columns by their header names, making it more convenient for data manipulation.
import csv
# Reading CSV file with dictionaries
with open('data.csv', 'r') as file:
csv_reader = csv.DictReader(file)
for row in csv_reader:
print(row)
# Writing CSV file with dictionaries
fieldnames = ['Name', 'Age', 'City']
data = [
{'Name': 'John', 'Age': 30, 'City': 'New York'},
{'Name': 'Alice', 'Age': 25, 'City': 'San Francisco'}
]
with open('output_dict.csv', 'w', newline='') as file:
csv_writer = csv.DictWriter(file, fieldnames=fieldnames)
csv_writer.writeheader()
csv_writer.writerows(data)
CSV vs JSON
CSV vs JSON
CSV pros and cons
- Advantages
- Simplicity: CSV is incredibly simple and easy to create, edit, and process, which is perfect for tabular data.
- Efficiency: CSV files are typically smaller and faster to read and write compared to JSON because they don't contain keys or metadata.
- Data Handling: Ideal for large datasets like those used in machine learning and data analysis, where data is primarily numeric and tabular.
- Software Compatibility: Widely supported by spreadsheet software, databases, and data analysis tools.
- Disadvantages
- Limited Data Structure: CSV only supports simple, flat data structures. It cannot represent hierarchies or relationships between data.
- Lack of Type Information: All data in CSV is stored as plain text, which means there is no distinction between types (e.g., string, number, date), potentially leading to type conversion issues.
- No Standard Encoding: Handling of special characters (like commas or newlines within entries) can vary by implementation, which may lead to compatibility issues.
JSON pros and cons
- Advantages
- Structured Data: JSON supports nested and hierarchical data structures, allowing for the representation of complex and interrelated data in an organized way.
- Flexible Schema: You can represent different data types and structures in the same file without conforming to a rigid schema.
- Wide Compatibility: Extensively supported across web technologies and programming languages, making it ideal for web APIs and configurations.
- Readability: JSON format is easy to read and understand, especially for developers.
- Disadvantages
- Size: JSON files can be larger than CSV because they include keys in each data object, which increases redundancy.
Choosing Between JSON and CSV
- Use JSON if:
- You need to store or transmit nested or structured data.
- Your data includes a variety of data types.
- You are developing web applications or APIs where interoperability across different systems is crucial.
- Use CSV if:
- You are dealing with large but simple and flat tabular data.
- Your primary concern is the efficiency of storage and speed of access.
- You need to import or export data from spreadsheet tools or databases frequently.
Exercises
Task1: print_books_by_category.py
The Task
- Given is next JSON file: pythonbooks.revolunet.com.issues.json, which contains information about the books listed in pythonbooks.revolunet.com
- Make a program, that will extract from these data only the "title", "author" and "url" fields for the books labeled as "Advanced", and will print the extracted information, as shown in sample output
Task2: rearange_books_by_category
The task
- Given is the same JSON file: pythonbooks.revolunet.com.issues.json
- You have to perform the same filtering actions, as in Task1, but instead of to print, now you have to create a new, filtered JSON file, like next:rearanged_books_by_category:
{
"Advanced": [
{
"author": "Ian Ozsvald",
"title": "High Performance Python tutorial",
"url": "http://ianozsvald.com/HighPerformancePythonfromTrainingatEuroPython2011_v0.2.pdf"
},
{
"author": "Jan Erik Solem",
"title": "Programming Computer Vision with Python",
"url": "http://programmingcomputervision.com/downloads/ProgrammingComputerVision_CCdraft.pdf"
},
{
"author": "S. Bird, E. Klein...",
"title": "Natural Language Processing with Python",
"url": "http://www.nltk.org/book/"
},
{
"author": "Allen B. Downey",
"title": "Think Complexity",
"url": "http://www.greenteapress.com/complexity/thinkcomplexity.pdf"
},
{
"author": "Allen B. Downey",
"title": "Think Stats",
"url": "http://greenteapress.com/thinkstats/thinkstats.pdf"
},
{
"author": "Allen B. Downey",
"title": "Think Stats 2e",
"url": "http://greenteapress.com/thinkstats2/thinkstats2.pdf"
},
{
"author": "Massimo Di Pierro",
"title": "Annotated Algorithms In Python",
"url": "https://raw.githubusercontent.com/mdipierro/nlib/master/docs/book_numerical.pdf"
},
{
"author": "John Hearty",
"title": "Advanced Machine Learning with Python",
"url": "https://www.packtpub.com/packt/free-ebook/advanced-python-machine-learning"
}
],
"Beginner": [
{
"author": "David Mertz",
"title": "Picking a Python Version: A Manifesto",
"url": "http://www.oreilly.com/programming/free/files/from-future-import-python.pdf"
},
{
"author": "B. Miller & D. Ranum",
"title": "How to Think Like a Computer Scientist: Second Interactive Edition",
"url": "http://interactivepython.org/runestone/static/thinkcspy/index.html"
},
{
"author": "Jeffrey Elkner...",
"title": "How to Think Like a Computer Scientist: Learning with Python 2nd Ed.",
"url": "http://www.openbookproject.net/thinkcs/python/english2e/"
},
{
"author": "Zed A. Shaw",
"title": "Learn Python The Hard Way",
"url": "http://learnpythonthehardway.org/book"
},
{
"author": "Allen B. Downey",
"title": "Think Python",
"url": "http://www.greenteapress.com/thinkpython/thinkpython.pdf"
},
{
"author": "Swaroop C H",
"title": "A byte of Python",
"url": "http://files.swaroopch.com/python/byte_of_python.pdf"
},
{
"author": "Dave Kuhlman",
"title": "Python 101 - Introduction to Python",
"url": "http://www.davekuhlman.org/python_101.html"
},
{
"author": "Jason R. Briggs",
"title": "Snake Wrangling for Kids",
"url": "http://www.briggs.net.nz/snake-wrangling-for-kids.html"
},
{
"author": "John C. Lusth",
"title": "An introduction to Python",
"url": "http://beastie.cs.ua.edu/cs150/book/index.html"
},
{
"author": "Steven F. Lot",
"title": "Building skills in Programming",
"url": "http://www.itmaybeahack.com/homepage/books/nonprog/html/index.html"
},
{
"author": "Steven F. Lot ",
"title": "Building skills in Python",
"url": "http://www.itmaybeahack.com/book/python-2.6/html/index.html"
},
{
"author": "G\u00e9rard Swinnen",
"title": "Programmez avec Python 2",
"url": "http://inforef.be/swi/download/apprendre_python.pdf"
},
{
"author": "G\u00e9rard Swinnen",
"title": "Programmez avec Python 3",
"url": "http://inforef.be/swi/download/apprendre_python3_5.pdf"
},
{
"author": "Kushal Das",
"title": "Python for you and me",
"url": "http://pymbook.readthedocs.io/en/latest/"
},
{
"author": "Patrick Fuchs",
"title": "Python course",
"url": "http://www.dsimb.inserm.fr/~fuchs/python/cours_python.pdf"
},
{
"author": "Jesse Noller",
"title": "A bit of Python & other things.",
"url": "http://jessenoller.com/good-to-great-python-reads/"
},
{
"author": "Google",
"title": "Python Course",
"url": "https://developers.google.com/edu/python/"
},
{
"author": "Josh Cogliati and others",
"title": "Non-Programmer's Tutorial for Python 3",
"url": "http://en.wikibooks.org/wiki/Non-Programmer%27s_Tutorial_for_Python_3"
},
{
"author": "Kenneth Reitz",
"title": "The Hitchhiker\u2019s Guide to Python!",
"url": "https://media.readthedocs.org/pdf/python-guide/latest/python-guide.pdf"
},
{
"author": "Al Sweigart",
"title": "Hacking Secret Ciphers with Python",
"url": "http://inventwithpython.com/hackingciphers.pdf"
},
{
"author": "Kenneth Love",
"title": "Getting Started with Django",
"url": "http://gettingstartedwithdjango.com/"
},
{
"author": "Jody S. Ginther",
"title": "Python 3x Programming (sample)",
"url": " http://www.toonzcat.com/media/pythonfree.pdf"
},
{
"author": "Ra\u00fal Gonz\u00e1lez Duque",
"title": "Python para todos",
"url": "http://edge.launchpad.net/improve-python-spanish-doc/0.4/0.4.0/+download/Python%20para%20todos.pdf"
},
{
"author": "Sat Kumar Tomer",
"title": "Python in Hydrology",
"url": "http://www.greenteapress.com/pythonhydro/pythonhydro.pdf"
},
{
"author": "Leif Azzopardi",
"title": "How to Tango with Django",
"url": "http://www.tangowithdjango.com/book17/"
},
{
"author": "Anand Chitipothu.",
"title": "Python Practice Book",
"url": "http://anandology.com/python-practice-book/index.html"
},
{
"author": "Community",
"title": "Django Girls Tutorial",
"url": "https://www.gitbook.com/download/pdf/book/djangogirls/djangogirls-tutorial?lang=en"
},
{
"author": "John B. Schneider...",
"title": "Algorithmic Problem Solving with Python",
"url": "http://www.eecs.wsu.edu/~schneidj/PyBook/swan.pdf"
},
{
"author": "Sean M. Tracey",
"title": "Make Games with Python",
"url": "https://www.raspberrypi.org/magpi-issues/Essentials_Games_v1.pdf"
},
{
"author": "Pierluigi Riti",
"title": "What You Need to Know about Python",
"url": "https://www.packtpub.com/packt/free-ebook/what-you-need-know-about-python2"
},
{
"author": "Charles R. Severance",
"title": "Python for Everybody",
"url": "https://www.py4e.com/book"
}
],
"Intermediate": [
{
"author": "Mike Pirnat",
"title": "How to Make Mistakes in Python",
"url": "http://www.oreilly.com/programming/free/files/how-to-make-mistakes-in-python.pdf"
},
{
"author": "David Mertz",
"title": "Functional Programming in Python",
"url": "http://www.oreilly.com/programming/free/files/functional-programming-python.pdf"
},
{
"author": "Luiz Eduardo Borges",
"title": "Python para Desenvolvedores (2nd Edition)",
"url": "http://ark4n.files.wordpress.com/2010/01/python_para_desenvolvedores_2ed.pdf"
},
{
"author": "Muhammad Yasoob",
"title": "Intermediate Python",
"url": "https://media.readthedocs.org/pdf/intermediatepythongithubio/latest/intermediatepythongithubio.pdf"
},
{
"author": "B. Miller & D. Ranum",
"title": "Problem Solving with Algorithms and Data Structures Using Python",
"url": "http://interactivepython.org/runestone/static/pythonds/index.html"
},
{
"author": "Mark Pilgrim",
"title": "Dive into Python (2004)",
"url": "http://www.diveintopython.net/download/diveintopython-pdf-5.4.zip"
},
{
"author": "Mark Pilgrim",
"title": "Dive into Python 3",
"url": "https://github.com/downloads/diveintomark/diveintopython3/dive-into-python3.pdf"
},
{
"author": "Kivy",
"title": "Kivy programming Guide",
"url": "https://readthedocs.org/projects/kivy/downloads/"
},
{
"author": "Community",
"title": "Django Tutorial",
"url": "https://media.readthedocs.org/pdf/django/latest/django.pdf"
},
{
"author": "Community",
"title": "Pyramid for Humans",
"url": "docs.pylonsproject.org/projects/pyramid_tutorials/en/latest/index.html"
},
{
"author": "Armin Ronacher",
"title": "Flask microframework",
"url": "http://flask.pocoo.org/docs/tutorial/"
},
{
"author": "Al Sweigart",
"title": "Making games with Python and Pygame",
"url": "http://inventwithpython.com/pygame/chapters/"
},
{
"author": "Fredrik Lundh",
"title": "The Standard Python Library",
"url": "http://effbot.org/zone/librarybook-index.htm"
},
{
"author": "Doug Hellman",
"title": "Python Module of the week",
"url": "https://pymotw.com/2/contents.html"
},
{
"author": "Steven F. Lot ",
"title": "Building skills in OOP",
"url": "http://www.itmaybeahack.com/book/oodesign-python-2.1/html/index.html"
},
{
"author": "By The Community",
"title": "Python Scientific lecture notes",
"url": "http://scipy-lectures.github.com/"
},
{
"author": "Bruno R. Preiss",
"title": "Data Structures and Algorithms with Object-Oriented Design Patterns in Python",
"url": "http://www.brpreiss.com/books/opus7/html/book.html"
},
{
"author": "Lennart Regebro",
"title": "Porting to Python 3: An in-depth guide",
"url": "http://python3porting.com/pdfs/SupportingPython3-screen-1.0-latest.pdf"
},
{
"author": "Cam Davidson-Pilon",
"title": "Probabilistic Programming and Bayesian Methods for Hackers: Using Python and PyMC",
"url": "http://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/"
},
{
"author": "Massimo Di Pierro",
"title": "web2py Complete Manual",
"url": "https://dl.dropboxusercontent.com/u/18065445/web2py/web2py_manual_5th.pdf"
},
{
"author": "Tom De Smedt",
"title": "Modeling Creativity",
"url": "http://www.clips.ua.ac.be/sites/default/files/modeling-creativity.pdf"
},
{
"author": "Harry Percival",
"title": "Test-Driven Development with Python",
"url": "http://chimera.labs.oreilly.com/books/1234000000754/index.html"
},
{
"author": "Agiliq",
"title": "Djen of Django",
"url": "http://agiliq.com/books/djenofdjango/"
},
{
"author": "Ron Zacharski",
"title": "A Programmer's Guide to Data Mining",
"url": "http://guidetodatamining.com/assets/guideChapters/Guide2DataMining.pdf"
},
{
"author": "Al Sweigart",
"title": "Invent Your Own Computer Games with Python",
"url": "http://inventwithpython.com/chapters/"
},
{
"author": "Various authors",
"title": "Biopython",
"url": "http://biopython.org/DIST/docs/tutorial/Tutorial.pdf"
},
{
"author": "David Mertz",
"title": "Text Processing in Python",
"url": "http://gnosis.cx/TPiP/"
},
{
"author": "Various authors",
"title": "Python Cookbook, Third Edition",
"url": "http://chimera.labs.oreilly.com/books/1230000000393/index.html"
},
{
"author": "Robert Picard.",
"title": "Explore Flask",
"url": "http://exploreflask.com"
},
{
"author": "Al Sweigart",
"title": "Automate the Boring Stuff with Python ",
"url": "https://automatetheboringstuff.com/"
},
{
"author": "Ian Ozsvald",
"title": "High Performance Python",
"url": "http://ianozsvald.com/HighPerformancePythonfromTrainingatEuroPython2011_v0.2.pdf"
},
{
"author": "Matt Makai",
"title": "Full Stack Python",
"url": "http://www.fullstackpython.com/table-of-contents.html"
},
{
"author": "Fabrizio Romano",
"title": "Learning Python",
"url": "https://www.packtpub.com/packt/free-ebook/learning-python"
},
{
"author": "Gabriel A. C\u00e1nepa",
"title": "What You Need to Know about Machine Learning",
"url": "https://www.packtpub.com/packt/free-ebook/what-you-need-know-about-machine-learning2"
},
{
"author": "Willi Richert, Luis Pedro Coelho",
"title": "Building Machine Learning Systems with Python",
"url": "https://www.packtpub.com/packt/free-ebook/python-machine-learning-algorithms"
},
{
"author": "Hector Cuesta",
"title": "Practical Data Analysis",
"url": "https://www.packtpub.com/packt/free-ebook/practical-data-analysis"
},
{
"author": "Tim Cox",
"title": "Raspberry Pi Cookbook for Python Programmers",
"url": "https://www.packtpub.com/packt/free-ebook/python-raspberry-pi-cookbook"
},
{
"author": "Caleb Hattingh",
"title": "20 Python libraries You Aren't Using (But Should)",
"url": "http://www.oreilly.com/programming/free/20-python-libraries-you-arent-using-but-should.csp"
}
]
}
Task2 -Template: rearange_books_by_category
- You can use the following template file for your code: rearange_books_by_category.py
import json
def rearange_books_by_category(books_json):
"""rearange_books_by_category.
Returns JSON with following structure:
{
"Beginner":[
{
"title":
"author":
"url"
}
],
"Intermediate":[
{
"title":
"author":
"url"
}
],
}
"""
rearanged_books = {}
#####################################
# YOUR CODE IS HERE #
#####################################
return rearanged_books
def main():
json_file = "pythonbooks.revolunet.com.issues.json"
#read json data from file
with open(json_file) as f:
json_data = json.load(f)
rearanged_books = rearange_books_by_category(json_data)
# print json data:
print(json.dumps(rearanged_books,indent=4,sort_keys=True))
if __name__ == '__main__':
main()
Task2 - A possible, pure Python, Solution
def rearange_books_by_category(books_json):
"""re-arange books by category.
returns:
books =
{
"Beginner":[
{
"title":
"author":
"url"
}
],
"Intermediate":[
{
"title":
"author":
"url"
}
],
}
"""
rearanged_books = {}
for book in books_json['books']:
category = book['level']
book_data = {
"title": book['title'],
"author":book['author'],
"url":book['url']
}
if category in rearanged_books.keys():
rearanged_books[category].append(book_data)
else:
rearanged_books[category] = [book_data]
return rearanged_books
These slides are based on
customised version of
framework