![Python for Data Science. Copyright © 2022 by Yuli Vasiliev.
Printed in the United States of America
First printing
26 25 24 23 22 1 2 3 4 5
ISBN-13: 978-1-7185-0220-8 (print)
ISBN-13: 978-1-7185-0221-5 (ebook)
Publisher: William Pollock
Managing Editor: Jill Franklin
Production Manager: Rachel Monaghan
Production Editor: Jennifer Kepler
Developmental Editor: Nathan Heidelberger
Cover Illustrator: Gina Redman
Interior Design: Octopod Studios
Technical Reviewer: Daniel Zingaro
Copyeditor: Rachel Head
Compositor: Jeff Lytle, Happenstance Type-O-Rama
Proofreader: Jamie Lauer
For information on distribution, bulk sales, corporate sales, or translations, please contact No Starch
Press, Inc. directly at info@nostarch.com or:
No Starch Press, Inc.
245 8th Street, San Francisco, CA 94103
phone: 1.415.863.9900
www.nostarch.com
Library of Congress Cataloging-in-Publication Data
Names: Vasiliev, Yuli, author.
Title: Python for data science : a hands-on introduction / Yuli Vasiliev.
Description: San Francisco : No Starch Press, [2022] | Includes index.
Identifiers: LCCN 2022002116 (print) | LCCN 2022002117 (ebook) | ISBN
9781718502208 (print) | ISBN 9781718502215 (ebook)
Subjects: LCSH: Python (Computer program language) | Electronic data
processing. | Data mining.
Classification: LCC QA76.73.P98 V37 2022 (print) | LCC QA76.73.P98
(ebook) | DDC 005.13/3--dc23/eng/20220325
LC record available at https://lccn.loc.gov/2022002116
LC ebook record available at https://lccn.loc.gov/2022002117
No Starch Press and the No Starch Press logo are registered trademarks of No Starch Press, Inc.
Other product and company names mentioned herein may be the trademarks of their respective
owners. Rather than use a trademark symbol with every occurrence of a trademarked name, we are
using the names only in an editorial fashion and to the benefit of the trademark owner, with no
intention of infringement of the trademark.
The information in this book is distributed on an “As Is” basis, without warranty. While every
precaution has been taken in the preparation of this work, neither the author nor No Starch Press, Inc.](https://image.slidesharecdn.com/81471-250322184221-0c8b4f14/85/Python-for-Data-Science-1-converted-Edition-Yuli-Vasiliev-15-320.jpg)
![degree of structure, usually in the form of self-describing tags or other
markers.
The most common semistructured data formats include XML and JSON.
This is what our financial statement might look like in JSON format:
{
"Company": "GoodComp",
"Date": "2021-01-07",
"Stock": 8.2,
"Details": "the company announced positive early-stage trial results for its
vaccine."
}
Here you can recognize the key information that we previously extracted
from the statement. Each piece of information is paired with a descriptive
tag, such as "Company" or "Date". Thanks to the tags, the information is
organized similarly to how it appeared in the previous section, but now we
have a fourth tag, "Details", paired with an entire fragment of the original
statement, which looks unstructured. This example shows how
semistructured data formats can accommodate both structured and
unstructured pieces of data within a single record.
Moreover, you can put multiple records of unequal structure into the
same container. Here, we store the two different records derived from our
example financial statement in the same JSON document:
[
{
"Company": "GoodComp",
"Date": "2021-01-07",
"Stock": 8.2
},
{
"Company": "GoodComp",
"Date": "2021-01-07",
"Product": "vaccine",
"Stage": "early-stage trial"
}
]](https://image.slidesharecdn.com/81471-250322184221-0c8b4f14/85/Python-for-Data-Science-1-converted-Edition-Yuli-Vasiliev-27-320.jpg)
![Recall from the discussion in the previous section that a relational
database, being a rigidly structured data repository, cannot accommodate
records of varying structures in the same table.
Time Series Data
A time series is a set of data points indexed or listed in time order. Many
financial datasets are stored as a time series due to the fact that financial
data typically consists of observations at a specific time.
Time series data can be either structured or semistructured. Imagine
you’re receiving location data in records from a taxi’s GPS tracking device
at regular time intervals. The data might arrive in the following format:
[
{
"cab": "cab_238",
"coord": (43.602508,39.715685),
"tm": "14:47",
"state": "available"
},
{
"cab": "cab_238",
"coord": (43.613744,39.705718),
"tm": "14:48",
"state": "available"
}
...
]
A new data record arrives every minute that includes the latest location
coordinates (latitude/longitude) from cab_238. Each record has the same
sequence of fields, and each field has a consistent structure from one record
to the next, allowing you to store this time series data in a relational
database table as regular structured data.
Now suppose the data comes at unequal intervals, which is often the case
in practice, and that you receive more than one set of coordinates in one
minute. The incoming structure might look like this:](https://image.slidesharecdn.com/81471-250322184221-0c8b4f14/85/Python-for-Data-Science-1-converted-Edition-Yuli-Vasiliev-28-320.jpg)
![[
{
"cab": "cab_238",
"coord": [(43.602508,39.715685),(43.602402,39.709672)],
"tm": "14:47",
"state": "available"
},
{
"cab": "cab_238",
"coord": (43.613744,39.705718),
"tm": "14:48",
"state": "available"
}
]
Note that the first coord field includes two sets of coordinates and is thus
not consistent with the second coord field. This data is semistructured.
Sources of Data
Now that you know what the main categories of data are, what are the
sources from which you might receive such data? Generally speaking, data
may come from many different sources, including texts, videos, images, and
device sensors, among others. From the standpoint of Python scripts that
you’ll write, however, the most common data sources are:
An application programming interface (API)
A web page
A database
A file
This list isn’t intended to be comprehensive or restrictive; there are many
other sources of data. In Chapter 9, for example, you’ll see how to use a
smartphone as a GPS data provider for your data processing pipeline,
specifically by using a bot application as a go-between connecting the
smartphone and your Python script.](https://image.slidesharecdn.com/81471-250322184221-0c8b4f14/85/Python-for-Data-Science-1-converted-Edition-Yuli-Vasiliev-29-320.jpg)
![extract data from it using Pythonic idioms. For example, the following
fragment of a parse tree:
[<td title="03/01/2020 00:00:00"><a href="Download.aspx?ID=630751"
id="lnkDownload630751"
target="_blank">03/01/2020</a></td>,
<td title="03/01/2020 00:00:00"><a href="Download.aspx?ID=630753"
id="lnkDownload630753"
target="_blank">03/01/2020</a></td>,
<td title="03/01/2020 00:00:00"><a href="Download.aspx?ID=630755"
id="lnkDownload630755"
target="_blank">03/01/2020</a></td>]
can be easily transformed into the following list of items within a for loop in
your Python script:
[
{'Document_Reference': '630751', 'Document_Date': '03/01/2020',
'link': 'http://www.dummy.com/Download.aspx?ID=630751'}
{'Document_Reference': '630753', 'Document_Date': '03/01/2020',
'link': 'http://www.dummy.com/Download.aspx?ID=630753'}
{'Document_Reference': '630755', 'Document_Date': '03/01/2020',
'link': 'http://www.dummy.com/Download.aspx?ID=630755'}
]
This is an example of transforming semistructured data into structured
data.
Databases
Another common source of data is a relational database, a structure that
provides a mechanism to efficiently store, access, and manipulate your
structured data. You fetch from or send a portion of data to tables in the
database using a Structured Query Language (SQL) request. For instance,
the following request issued to an employees table in the database retrieves the
list of only those programmers who work in the IT department, making it
unnecessary to fetch the entire table:
SELECT first_name, last_name FROM employees WHERE department = 'IT' and title =
'programmer'](https://image.slidesharecdn.com/81471-250322184221-0c8b4f14/85/Python-for-Data-Science-1-converted-Edition-Yuli-Vasiliev-32-320.jpg)
![money or percent phrase. This filter can be considered a data cleansing
operation because the goal is to remove unnecessary data and prepare for
the data transformation and data analysis operations.
Transformation
Data transformation is the process of changing the format or structure of
data in preparation for analysis. For example, to extract the information
from our GoodComp unstructured text data as we did in “Structured Data,”
you might shred it into individual words or tokens so that a named entity
recognition (NER) tool can look for the desired information. In information
extraction, a named entity typically represents a real-world object, such as a
person, an organization, or a product, that can be identified by a proper
noun. There are also named entities that represent dates, percentages,
financial terms, and more.
Many NLP tools can handle this kind of transformation for you
automatically. After such a transformation, the shredded GoodComp data
would look like this:
['GoodComp', 'shares', 'soared', 'as', 'much', 'as', '8.2%', 'on',
'2021-01-07', 'after', 'the', 'company', 'announced', 'positive',
'early-stage', 'trial', 'results', 'for', 'its', 'vaccine']
Other forms of data transformation are deeper, with text data being
converted into numerical data. For example, if we’ve gathered a collection
of news articles, we might transform them by performing sentiment
analysis, a text processing technique that generates a number representing
the emotions expressed within a text.
Sentiment analysis can be implemented with tools like
SentimentAnalyzer, which can be found in the nltk.sentiment package. A
typical analysis output might look like this:
Sentiment URL
--------- ----------------------------------------------------------------
0.9313 https://mashable.com/uk/shopping/amazon-face-mask-store-july-28/
0.9387 https://skillet.lifehacker.com/save-those-crustacean-shells-to
-make-a-sauce-base-1844520024](https://image.slidesharecdn.com/81471-250322184221-0c8b4f14/85/Python-for-Data-Science-1-converted-Edition-Yuli-Vasiliev-36-320.jpg)
![practice. Say you need to process this multisentence fragment of text:
txt = ''' Eight dollars a week or a million a year - what is the difference? A
mathematician or a wit would give you the wrong answer. The magi brought valuable
gifts, but that was not among them. - The Gift of the Magi, O'Henry'''
Specifically, you need to split the text by sentences, creating a list of
individual words for each sentence, excluding punctuation symbols. Thanks
to Python’s list comprehension feature, all of this can be implemented in a
single line of code, a so-called one-liner:
word_lists = [[w.replace(',','') ❶ for w in line.split() if w not in ['-']]
❷ for line in txt.replace('?','.').split('.')]
The for line in txt loop ❷ splits the text into sentences and stores those
sentences in a list. Then the for w in line loop ❶ splits each sentence into
individuals words and stores the words in a list within the larger list. As a
result, you get the following list of lists:
[['Eight', 'dollars', 'a', 'week', 'or', 'a', 'million', 'a', 'year', 'what',
'is', 'the', 'difference'], ['A', 'mathematician', 'or', 'a', 'wit',
'would', 'give', 'you', 'the', 'wrong', 'answer'], ['The', 'magi',
'brought', 'valuable', 'gifts', 'but', 'that', 'was', 'not', 'among',
'them'], ['The', 'Gift', 'of', 'the', 'Magi', "O'Henry"]]
Here you’ve managed to accomplish two steps of the data processing
pipeline within a single line of code: cleansing and transformation. You’ve
cleansed the data by removing punctuation symbols from the text, and
you’ve transformed it by separating the words from each other to form a
word list for each sentence.
If you’ve come to Python from another programming language, try
implementing this task with that language. How many lines of code does it
take?
Summary
After reading this chapter, you should have a cursory understanding of the
main categories of data, where data comes from, and how a typical data](https://image.slidesharecdn.com/81471-250322184221-0c8b4f14/85/Python-for-Data-Science-1-converted-Edition-Yuli-Vasiliev-39-320.jpg)
![[2,4,7]
['Bob', 'John', 'Will']
Lists are mutable, meaning you can add, remove, and modify a list’s
elements. Unlike sets, which we’ll discuss later in the chapter, lists can have
duplicate elements.
Lists contain elements that represent series of usually related, similar
things that can be logically grouped together. A typical list contains only
elements belonging to a single category (that is, homogeneous data, such as
people’s names, article titles, or participant numbers). Understanding this
point is essential when it comes to choosing the right tool for the task at
hand. If you need a structure to include objects with different properties,
consider using a tuple or a dictionary.
NOTE
Although lists are generally understood to be homogeneous, Python
does allow you to have elements of different data types in the same
list. This list, for example, includes both strings and integers:
['Ford', 'Mustang', 1964]
Creating a List
To create a basic list, simply place a sequence of elements inside square
brackets and assign the sequence to a variable name:
regions = ['Asia', 'America', 'Europe']
In practice, however, lists are commonly populated dynamically from
scratch, often using a loop that calculates one item per iteration. In such
cases, your first step is to create an empty list, as illustrated here:
regions = []
Once you’ve created a list, you can add, remove, and sort items in that
list as needed. You can perform these and other tasks using Python’s various](https://image.slidesharecdn.com/81471-250322184221-0c8b4f14/85/Python-for-Data-Science-1-converted-Edition-Yuli-Vasiliev-42-320.jpg)
![list object methods.
Using Common List Object Methods
List object methods are functions that implement particular behaviors
within lists. In this section we’ll look at some common list object methods,
including append(), index(), insert(), and count(). To practice using them, start
by creating a blank list. You’ll build it into a to-do list as you go along,
filling it with chores and organizing it:
my_list = []
Perhaps the most common list object method is append(). It adds an item to
the end of the list. You can use append() to add some chores to your to-do
list, as shown here:
my_list.append('Pay bills')
my_list.append('Tidy up')
my_list.append('Walk the dog')
my_list.append('Cook dinner')
The list now contains four items, in the order in which they were
appended:
['Pay bills', 'Tidy up', 'Walk the dog', 'Cook dinner']
Each item in a list has a numeric key known as an index. This feature
enables a list to keep its items in a specified order. Python uses zero-based
indexing, meaning the initial item of a sequence is assigned the index 0.
To access an individual item from a list, specify the name of the list,
followed by the index of the desired item in square brackets. For example,
here’s how to print just the first item of your to-do list:
print(my_list[0])
The print() function yields the following output:
Pay bills](https://image.slidesharecdn.com/81471-250322184221-0c8b4f14/85/Python-for-Data-Science-1-converted-Edition-Yuli-Vasiliev-43-320.jpg)
![You can use a list’s indices not only to access a required item but also to
insert a new item at a certain position in the list. Say you want to add a new
chore to your to-do list between walking the dog and cooking dinner. To
make this insertion, you first use the index() method to determine the index
of the item before which you want to insert the new item. Here, you’ll store
it in the variable i:
i = my_list.index('Cook dinner')
This will become the index for the new chore, which you can now add
using the insert() method, as shown here:
my_list.insert(i, 'Go to the pharmacy')
The new chore is added to the list at the specified index, shifting all
subsequent chores down by one. This is what the updated list looks like:
['Pay bills', 'Tidy up', 'Walk the dog', 'Go to the pharmacy', 'Cook dinner']
Because lists allow for duplicate items, you may need to check how
many times a certain item appears in a list. This can be done with the count()
method, as illustrated in the following example:
print(my_list.count('Tidy up'))
The print() function reveals only one instance of 'Tidy up' in the list, but it
might be a good idea to include this item in your daily list more than once!
NOTE
The entire list of list object methods can be found in the Python
documentation at
https://docs.python.org/3/tutorial/datastructures.html.
Using Slice Notation
It’s possible to access a range of items from a sequential data type such as a
list by using slice notation. To take a slice of a list, specify the index of the
starting position and the index of the ending position plus 1. Separate the](https://image.slidesharecdn.com/81471-250322184221-0c8b4f14/85/Python-for-Data-Science-1-converted-Edition-Yuli-Vasiliev-44-320.jpg)
![[Medicalstudyzone.com] 1. AIIMS NOV EMBER 2015 SOLVED PAPER.pdf](https://cdn.slidesharecdn.com/ss_thumbnails/medicalstudyzone-250904212000-ef4cbf83-thumbnail.jpg?width=560&fit=bounds)
Python for Data Science 1 / converted Edition Yuli Vasiliev
- 1. Read Anytime AnywhereEasy Ebook Downloads at ebookmeta.com Python for Data Science 1 / converted Edition Yuli Vasiliev https://ebookmeta.com/product/python-for-data- science-1-converted-edition-yuli-vasiliev/ OR CLICK HERE DOWLOAD EBOOK Visit and Get More Ebook Downloads Instantly at https://ebookmeta.com
- 2. Recommended digital products(PDF, EPUB, MOBI) that you can download immediately if you are interested. Practical Linear Algebra for Data Science 1 / converted Edition Mike. Cohen https://ebookmeta.com/product/practical-linear-algebra-for-data- science-1-converted-edition-mike-cohen/ ebookmeta.com Python Unit Test Automation: Practical Techniques for Python Developers and Testers 1 / converted Edition Ashwin Pajankar https://ebookmeta.com/product/python-unit-test-automation-practical- techniques-for-python-developers-and-testers-1-converted-edition- ashwin-pajankar/ ebookmeta.com Python Data Science Chaolemen Borjigin https://ebookmeta.com/product/python-data-science-chaolemen-borjigin/ ebookmeta.com American Spy Wry Reflections on My Life in the CIA H K Roy https://ebookmeta.com/product/american-spy-wry-reflections-on-my-life- in-the-cia-h-k-roy/ ebookmeta.com
- 3. The International Companionto Scottish Literature 1400 1650 1st Edition Nicola Royan https://ebookmeta.com/product/the-international-companion-to-scottish- literature-1400-1650-1st-edition-nicola-royan/ ebookmeta.com Manual of Percutaneous Coronary Interventions A Step by Step Approach 1st Edition Emmanouil Brilakis https://ebookmeta.com/product/manual-of-percutaneous-coronary- interventions-a-step-by-step-approach-1st-edition-emmanouil-brilakis/ ebookmeta.com Utopias in Nonfiction Film 1st Edition Simon Spiegel https://ebookmeta.com/product/utopias-in-nonfiction-film-1st-edition- simon-spiegel/ ebookmeta.com The Rebirth of African Orthodoxy 3rd Edition Thomas C. Oden https://ebookmeta.com/product/the-rebirth-of-african-orthodoxy-3rd- edition-thomas-c-oden/ ebookmeta.com History of Science Technology Environment and Medicine in India 1st Edition Edited By Suvobrata Sarkar https://ebookmeta.com/product/history-of-science-technology- environment-and-medicine-in-india-1st-edition-edited-by-suvobrata- sarkar/ ebookmeta.com
- 4. On the Natureof Ecological Paradox 1st Edition Michael Charles Tobias https://ebookmeta.com/product/on-the-nature-of-ecological-paradox-1st- edition-michael-charles-tobias/ ebookmeta.com
- 6. CONTENTS IN DETAIL TITLEPAGE COPYRIGHT ABOUT THE AUTHOR INTRODUCTION Using Python for Data Science Who Should Read This Book? What’s in the Book? CHAPTER 1: THE BASICS OF DATA Categories of Data Unstructured Data Structured Data Semistructured Data Time Series Data Sources of Data APIs Web Pages Databases Files The Data Processing Pipeline Acquisition Cleansing Transformation
- 7. Analysis Storage The Pythonic Way Summary CHAPTER2: PYTHON DATA STRUCTURES Lists Creating a List Using Common List Object Methods Using Slice Notation Using a List as a Queue Using a List as a Stack Using Lists and Stacks for Natural Language Processing Making Improvements with List Comprehensions Tuples A List of Tuples Immutability Dictionaries A List of Dictionaries Adding to a Dictionary with setdefault() Loading JSON into a Dictionary Sets Removing Duplicates from Sequences Performing Common Set Operations Exercise #1: Improved Photo Tag Analysis Summary CHAPTER 3: PYTHON DATA SCIENCE LIBRARIES NumPy Installing NumPy
- 8. Creating a NumPyArray Performing Element-Wise Operations Using NumPy Statistical Functions Exercise #2: Using NumPy Statistical Functions pandas pandas Installation pandas Series Exercise #3: Combining Three Series pandas DataFrames Exercise #4: Using Different Joins scikit-learn Installing scikit-learn Obtaining a Sample Dataset Loading the Sample Dataset into a pandas DataFrame Splitting the Sample Dataset into a Training Set and a Test Set Transforming Text into Numerical Feature Vectors Training and Evaluating the Model Making Predictions on New Data Summary CHAPTER 4: ACCESSING DATA FROM FILES AND APIS Importing Data Using Python’s open() Function Text Files Tabular Data Files Exercise #5: Opening JSON Files Binary Files Exporting Data to Files Accessing Remote Files and APIs How HTTP Requests Work
- 9. The urllib3 Library TheRequests Library Exercise #6: Accessing an API with Requests Moving Data to and from a DataFrame Importing Nested JSON Structures Converting a DataFrame to JSON Exercise #7: Manipulating Complex JSON Structures Loading Online Data into a DataFrame with pandas- datareader Summary CHAPTER 5: WORKING WITH DATABASES Relational Databases Understanding SQL Statements Getting Started with MySQL Defining the Database Structure Inserting Data into the Database Querying Database Data Exercise #8: Performing a One-to-Many Join Using Database Analytics Tools NoSQL Databases Key-Value Stores Document-Oriented Databases Exercise #9: Inserting and Querying Multiple Documents Summary CHAPTER 6: AGGREGATING DATA Data to Aggregate Combining DataFrames Grouping and Aggregating the Data
- 10. Viewing Specific Aggregationsby MultiIndex Slicing a Range of Aggregated Values Slicing Within Aggregation Levels Adding a Grand Total Adding Subtotals Exercise #10: Excluding Total Rows from the DataFrame Selecting All Rows in a Group Summary CHAPTER 7: COMBINING DATASETS Combining Built-in Data Structures Combining Lists and Tuples with + Combining Dictionaries with ** Combining Corresponding Rows from Two Structures Implementing Different Types of Joins for Lists Concatenating NumPy Arrays Exercise #11: Adding New Rows/Columns to a NumPy Array Combining pandas Data Structures Concatenating DataFrames Joining Two DataFrames Summary CHAPTER 8: CREATING VISUALIZATIONS Common Visualizations Line Graphs Bar Graphs Pie Charts Histograms Plotting with Matplotlib
- 11. Installing Matplotlib Using matplotlib.pyplot Workingwith Figure and Axes Objects Exercise #12: Combining Bins into an “Other” Slice Using Other Libraries with Matplotlib Plotting pandas Data Plotting Geospatial Data with Cartopy Exercise #13: Drawing a Map with Cartopy and Matplotlib Summary CHAPTER 9: ANALYZING LOCATION DATA Obtaining Location Data Turning a Human-Readable Address into Geo Coordinates Getting the Geo Coordinates of a Moving Object Spatial Data Analysis with geopy and Shapely Finding the Closest Object Finding Objects in a Certain Area Exercise #14: Defining Two or More Polygons Combining Both Approaches Exercise #15: Further Improving the Pick-Up Algorithm Combining Spatial and Nonspatial Data Deriving Nonspatial Attributes Exercise #16: Filtering Data with a List Comprehension Joining Spatial and Nonspatial Datasets Summary CHAPTER 10: ANALYZING TIME SERIES DATA Regular vs. Irregular Time Series
- 12. Common Time SeriesAnalysis Techniques Calculating Percentage Changes Rolling Window Calculations Calculating the Percentage Change of a Rolling Average Multivariate Time Series Processing Multivariate Time Series Analyzing Dependencies Between Variables Exercise #17: Adding More Metrics to Analyze Dependencies Summary CHAPTER 11: GAINING INSIGHTS FROM DATA Association Rules Support Confidence Lift The Apriori Algorithm Creating a Transaction Dataset Identifying Frequent Itemsets Generating Association Rules Visualizing Association Rules Gaining Actionable Insights from Association Rules Generating Recommendations Planning Discounts Based on Association Rules Exercise #18: Mining Real Transaction Data Summary CHAPTER 12: MACHINE LEARNING FOR DATA ANALYSIS Why Machine Learning? Types of Machine Learning
- 13. Supervised Learning Unsupervised Learning HowMachine Learning Works Data to Learn From A Statistical Model Previously Unseen Data A Sentiment Analysis Example: Classifying Product Reviews Obtaining Product Reviews Cleansing the Data Splitting and Transforming the Data Training the Model Evaluating the Model Exercise #19: Expanding the Example Set Predicting Stock Trends Getting Data Deriving Features from Continuous Data Generating the Output Variable Training and Evaluating the Model Exercise #20: Experimenting with Different Stocks and New Metrics Summary INDEX
- 14. PYTHON FOR DATASCIENCE A Hands-On Introduction by Yuli Vasiliev
- 15. Python for DataScience. Copyright © 2022 by Yuli Vasiliev. Printed in the United States of America First printing 26 25 24 23 22 1 2 3 4 5 ISBN-13: 978-1-7185-0220-8 (print) ISBN-13: 978-1-7185-0221-5 (ebook) Publisher: William Pollock Managing Editor: Jill Franklin Production Manager: Rachel Monaghan Production Editor: Jennifer Kepler Developmental Editor: Nathan Heidelberger Cover Illustrator: Gina Redman Interior Design: Octopod Studios Technical Reviewer: Daniel Zingaro Copyeditor: Rachel Head Compositor: Jeff Lytle, Happenstance Type-O-Rama Proofreader: Jamie Lauer For information on distribution, bulk sales, corporate sales, or translations, please contact No Starch Press, Inc. directly at [email protected] or: No Starch Press, Inc. 245 8th Street, San Francisco, CA 94103 phone: 1.415.863.9900 www.nostarch.com Library of Congress Cataloging-in-Publication Data Names: Vasiliev, Yuli, author. Title: Python for data science : a hands-on introduction / Yuli Vasiliev. Description: San Francisco : No Starch Press, [2022] | Includes index. Identifiers: LCCN 2022002116 (print) | LCCN 2022002117 (ebook) | ISBN 9781718502208 (print) | ISBN 9781718502215 (ebook) Subjects: LCSH: Python (Computer program language) | Electronic data processing. | Data mining. Classification: LCC QA76.73.P98 V37 2022 (print) | LCC QA76.73.P98 (ebook) | DDC 005.13/3--dc23/eng/20220325 LC record available at https://lccn.loc.gov/2022002116 LC ebook record available at https://lccn.loc.gov/2022002117 No Starch Press and the No Starch Press logo are registered trademarks of No Starch Press, Inc. Other product and company names mentioned herein may be the trademarks of their respective owners. Rather than use a trademark symbol with every occurrence of a trademarked name, we are using the names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The information in this book is distributed on an “As Is” basis, without warranty. While every precaution has been taken in the preparation of this work, neither the author nor No Starch Press, Inc.
- 16. shall have anyliability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in it.
- 17. About the Author YuliVasiliev is a programmer, writer, and consultant specializing in open source development, building data structures and models, and implementing database backends. He is the author of Natural Language Processing with Python and spaCy (No Starch Press, 2020). About the Technical Reviewer Dr. Daniel Zingaro is an associate teaching professor of computer science and award-winning teacher at the University of Toronto. His research focuses on understanding and enhancing student learning of computer science. He is the author of two recent No Starch Press books: Algorithmic Thinking (2020), a no-nonsense, no-math guide to algorithms and data structures; and Learn to Code by Solving Problems (2021), a primer for learning Python and computational thinking.
- 18. INTRODUCTION We live ina world of information technology (IT), where computer systems collect enormous quantities of data, process it, and extract useful information from it. This data-driven reality affects not only the way modern businesses operate but our daily lives too. Without the numerous devices and systems that employ data-focused technologies, it would be hard for many of us to maintain contact with society. Mobile maps and navigation, online shopping, and smart home devices are some common examples of data-focused technology for everyday life. In the business world, companies often use IT systems to make decisions by extracting actionable information from large volumes of data. The data may arrive from various sources, in different formats, and may require transformation before it’s ready for analysis. For example, many companies that do business online use data analytics to drive customer acquisition and retention, collecting and measuring everything they can to model and understand their users’ behavior. They often combine and analyze both quantitative and qualitative user data from many different sources, such as user profiles, social media, and company websites. And in many cases, they accomplish all these tasks using the Python programming language.
- 19. This book willintroduce you to the Pythonic world of working with data, without the taint of academic jargon or excessive complexity. You’ll learn to use Python for data-oriented applications, writing code to power a ride- sharing service, generate product recommendations, predict stock market trends, and more. Through real-world examples such as these, you’ll gain practical, hands-on experience with the key Python data science libraries. Using Python for Data Science The easy-on-the-brain Python programming language is an ideal choice for accessing, manipulating, and gaining insight from data of any kind. It has both a rich set of built-in data structures for basic operations and a robust ecosystem of open source libraries for data analysis and manipulation of any level of complexity. We’ll explore many such libraries in this book, including NumPy, pandas, scikit-learn, Matplotlib, and more. With Python, you can write concise and intuitive code with minimal time and effort, expressing most concepts in just a few lines of code. In fact, Python’s agile syntax allows you to implement several data operations with a single line of code. For example, you can write a one-liner that filters, transforms, and aggregates data all at once. As a general-purpose language, Python is suitable for a wide variety of tasks. When you work with Python, you can seamlessly integrate data science with other tasks to create fully functional, well-rounded applications. For example, you could build a bot application that makes stock market predictions in response to natural language requests from users. To create such an application, you’d need a bot API, a machine learning model to make predictions, and a natural language processing (NLP) tool to interact with users. There are powerful Python libraries for all of these. Who Should Read This Book? This book is for developers looking to gain a better understanding of Python’s data processing and analysis capabilities. Perhaps you work for a company that wants to use data to improve business processes, make better
- 20. decisions, and targetmore customers. Or maybe you want to develop your own data-driven applications, or simply expand your knowledge of Python into the realm of data science. The book assumes you have some basic experience with Python and that you’re comfortable following instructions to perform tasks such as installing a database or obtaining an API key. However, the book covers Python data science concepts from the bottom up, through hands-on examples that are all thoroughly explained. You’ll learn by doing, with no prior data experience necessary. What’s in the Book? The book begins with a conceptual introduction to data processing and analysis, explaining a typical data processing pipeline. Then we’ll cover Python’s built-in data structures and some of the third-party Python libraries that are widely used for data science applications. Next, we’ll explore increasingly sophisticated techniques for obtaining, combining, aggregating, grouping, analyzing, and visualizing datasets of different sizes and data types. As the book goes on, we’ll apply Python data science techniques to real use cases from the world of business management, marketing, and finance. Along the way, each chapter contains “Exercise” sections so you can practice and reinforce what you’ve just learned. Here’s an overview of what you’ll find in each chapter: Chapter 1: The Basics of Data Provides the necessary background for understanding the essentials of working with data. You’ll learn that there are different categories of data, including structured, unstructured, and semistructured data. Then you’ll walk through the steps involved in a typical data analysis process. Chapter 2: Python Data Structures Introduces four data structures that are built into Python: lists, dictionaries, tuples, and sets. You’ll see how to use each structure and how to combine them into more complex structures that can represent real-world objects. Chapter 3: Python Data Science Libraries Discusses Python’s robust ecosystem of third-party libraries for data analysis and manipulation.
- 21. You’ll meet thepandas library and its primary data structures, the Series and DataFrame, which have become the de facto standard for data- oriented Python applications. You’ll also learn about NumPy and scikit- learn, two other libraries often used for data science. Chapter 4: Accessing Data from Files and APIs Dives into the details of obtaining data and loading it into your scripts. You’ll learn to load data from different sources, such as files and APIs, into data structures in your Python scripts for further processing. Chapter 5: Working with Databases Continues the discussion of importing data into Python, covering how to work with database data. You’ll look at examples of accessing and manipulating data stored in databases of different types, including relational databases like MySQL and NoSQL databases like MongoDB. Chapter 6: Aggregating Data Approaches the problem of summarizing data by sorting it into groups and performing aggregate calculations. You’ll learn to use pandas to group data and produce subtotals, totals, and other aggregations. Chapter 7: Combining Datasets Covers how to combine data from different sources into a single dataset. You’ll learn techniques that SQL developers use to join database tables and apply them to built-in Python data structures, NumPy arrays, and pandas DataFrames. Chapter 8: Creating Visualizations Discusses visualizations as the most natural way to bring to light hidden patterns in data. You’ll learn about different types of visualizations, such as line graphs, bar graphs, and histograms, and you’ll see how to create them with Matplotlib, the leading Python library for plotting. You’ll also use the Cartopy library to generate maps. Chapter 9: Analyzing Location Data Explains how to work with location data using the geopy and Shapely libraries. You’ll learn ways to get and use GPS coordinates for both stationary and moving objects, and you’ll explore the real-world example of how a ride-sharing service can identify the best car for a given pick-up. Chapter 10: Analyzing Time Series Data Presents some analysis techniques that you can apply to time series data to extract meaningful
- 22. statistics from it.In particular, the examples in this chapter illustrate how time series data analysis can be applied to stock market data. Chapter 11: Gaining Insights from Data Explores strategies for gaining insight from data in order to make informed decisions. As an example, you’ll learn how to discover associations between products sold at a supermarket so you can determine what groups of items are frequently bought together in a single transaction (useful for recommendations and promotions). Chapter 12: Machine Learning for Data Analysis Covers the use of scikit-learn for advanced data analysis tasks. You’ll train machine learning models to classify product reviews according to their star ratings and to predict trends in a stock’s price.
- 23. 1 THE BASICS OFDATA Data means different things to different people: a stock trader might think of data as real-time stock quotes, while a NASA engineer might associate data with signals coming from a Mars rover. When it comes to data processing and analysis, however, the same or similar approaches and techniques can be applied to a variety of datasets, regardless of their origin. All that matters is how the data is structured. This chapter provides a conceptual introduction to data processing and analysis. We’ll first look at the main categories of data you may have to deal with, then touch on common data sources. Next, we’ll consider the steps in a typical data processing pipeline (that is, the actual process of obtaining, preparing, and analyzing data). Finally, we’ll examine Python’s unique advantages as a data science tool. Categories of Data Programmers divide data into three main categories: unstructured, structured, and semistructured. In a data processing pipeline, the source data is typically unstructured; from this, you form structured or semistructured datasets for further processing. Some pipelines, however, use structured
- 24. data from thestart. For example, an application processing geographical locations might receive structured data directly from GPS sensors. The following sections explore the three main categories of data as well as time series data, a special type of data that can be structured or semistructured. Unstructured Data Unstructured data is data with no predefined organizational system, or schema. This is the most widespread form of data, with common examples including images, videos, audio, and natural language text. To illustrate, consider the following financial statement from a pharmaceutical company: GoodComp shares soared as much as 8.2% on 2021-01-07 after the company announced positive early-stage trial results for its vaccine. This text is considered unstructured data because the information found in it isn’t organized with a predefined schema. Instead, the information is randomly scattered within the statement. You could rewrite this statement in any number of ways while still conveying the same information. For example: Following the January 7, 2021, release of positive results from its vaccine trial, which is still in its early stages, shares in GoodComp rose by 8.2%. Despite its lack of structure, unstructured data may contain important information, which you can extract and convert to structured or semistructured data through appropriate transformation and analysis steps. For example, image recognition tools first convert the collection of pixels within an image into a dataset of a predefined format and then analyze this data to identify content in the image. Similarly, the following section will show a few ways in which the data extracted from our financial statement could be structured. Structured Data Structured data has a predefined format that specifies how the data is organized. Such data is usually stored in a repository like a relational database or just a .csv (comma-separated values) file. The data fed into such
- 25. a repository iscalled a record, and the information in it is organized in fields that must arrive in a sequence matching the expected structure. Within a database, records of the same structure are logically grouped in a container called a table. A database may contain various tables, with each table having a set structure of fields. There are two basic types of structured data: numerical and categorical. Categorical data is that which can be categorized on the basis of similar characteristics; cars, for example, might be categorized by make and model. Numerical data, on the other hand, expresses information in numerical form, allowing you to perform mathematical operations on it. Keep in mind that categorical data can sometimes take on numerical values. For example, consider ZIP codes or phone numbers. Although they are expressed with numbers, it wouldn’t make any sense to perform math operations on them, such as finding the median ZIP code or average phone number. How can we organize the text sample introduced in the previous section into structured data? We’re interested in specific information in this text, such as company names, dates, and stock prices. We want to present that information in fields in the following format, ready for insertion into a database: Company: ABC Date: yyyy-mm-dd Stock: nnnnn Using techniques of natural language processing (NLP), a discipline that trains machines to understand human-readable text, we can extract information appropriate for these fields. For example, we look for a company name by recognizing a categorical data variable that can only be one of many preset values, such as Google, Apple, or GoodComp. Likewise, we can recognize a date by matching its explicit ordering to one of a set of explicit ordering formats, such as yyyy-mm-dd. In our example, we recognize, extract, and present our data in the predefined format like this: Company: GoodComp Date: 2021-01-07 Stock: +8.2%
- 26. To store thisrecord in a database, it’s better to present it as a row-like sequence of fields. We therefore might reorganize the record as a rectangular data object, or a 2D matrix: Company | Date | Stock --------------------------- GoodComp |2021-01-07 | +8.2% The information you choose to extract from the same unstructured data source depends on your requirements. Our example statement not only contains the change in GoodComp’s stock value for a certain date but also indicates the reason for that change, in the phrase “the company announced positive early-stage trial results for its vaccine.” Taking the statement from this angle, you might create a record with these fields: Company: GoodComp Date: 2021-01-07 Product: vaccine Stage: early-stage trial Compare this to the first record we extracted: Company: GoodComp Date: 2021-01-07 Stock: +8.2% Notice that these two records contain different fields and therefore have different structures. As a result, they must be stored in two different database tables. Semistructured Data In cases where the structural identity of the information doesn’t conform to stringent formatting requirements, we may need to process semistructured data formats, which let us have records of different structures within the same container (database table or document). Like unstructured data, semistructured data isn’t tied to a predefined organizational schema; unlike unstructured data, however, samples of semistructured data do exhibit some
- 27. degree of structure,usually in the form of self-describing tags or other markers. The most common semistructured data formats include XML and JSON. This is what our financial statement might look like in JSON format: { "Company": "GoodComp", "Date": "2021-01-07", "Stock": 8.2, "Details": "the company announced positive early-stage trial results for its vaccine." } Here you can recognize the key information that we previously extracted from the statement. Each piece of information is paired with a descriptive tag, such as "Company" or "Date". Thanks to the tags, the information is organized similarly to how it appeared in the previous section, but now we have a fourth tag, "Details", paired with an entire fragment of the original statement, which looks unstructured. This example shows how semistructured data formats can accommodate both structured and unstructured pieces of data within a single record. Moreover, you can put multiple records of unequal structure into the same container. Here, we store the two different records derived from our example financial statement in the same JSON document: [ { "Company": "GoodComp", "Date": "2021-01-07", "Stock": 8.2 }, { "Company": "GoodComp", "Date": "2021-01-07", "Product": "vaccine", "Stage": "early-stage trial" } ]
- 28. Recall from thediscussion in the previous section that a relational database, being a rigidly structured data repository, cannot accommodate records of varying structures in the same table. Time Series Data A time series is a set of data points indexed or listed in time order. Many financial datasets are stored as a time series due to the fact that financial data typically consists of observations at a specific time. Time series data can be either structured or semistructured. Imagine you’re receiving location data in records from a taxi’s GPS tracking device at regular time intervals. The data might arrive in the following format: [ { "cab": "cab_238", "coord": (43.602508,39.715685), "tm": "14:47", "state": "available" }, { "cab": "cab_238", "coord": (43.613744,39.705718), "tm": "14:48", "state": "available" } ... ] A new data record arrives every minute that includes the latest location coordinates (latitude/longitude) from cab_238. Each record has the same sequence of fields, and each field has a consistent structure from one record to the next, allowing you to store this time series data in a relational database table as regular structured data. Now suppose the data comes at unequal intervals, which is often the case in practice, and that you receive more than one set of coordinates in one minute. The incoming structure might look like this:
- 29. [ { "cab": "cab_238", "coord": [(43.602508,39.715685),(43.602402,39.709672)], "tm":"14:47", "state": "available" }, { "cab": "cab_238", "coord": (43.613744,39.705718), "tm": "14:48", "state": "available" } ] Note that the first coord field includes two sets of coordinates and is thus not consistent with the second coord field. This data is semistructured. Sources of Data Now that you know what the main categories of data are, what are the sources from which you might receive such data? Generally speaking, data may come from many different sources, including texts, videos, images, and device sensors, among others. From the standpoint of Python scripts that you’ll write, however, the most common data sources are: An application programming interface (API) A web page A database A file This list isn’t intended to be comprehensive or restrictive; there are many other sources of data. In Chapter 9, for example, you’ll see how to use a smartphone as a GPS data provider for your data processing pipeline, specifically by using a bot application as a go-between connecting the smartphone and your Python script.
- 30. Technically, all ofthe options listed here require you to use a corresponding Python library. For example, before you can obtain data from an API, you’ll need to install a Python wrapper for the API or use the Requests Python library to make HTTP requests to the API directly. Likewise, in order to access data from a database, you’ll need to install a connector from within your Python code that enables you to access databases of that particular type. While many of these libraries must be downloaded and installed, some libraries used to load data are distributed with Python by default. For example, to load data from a JSON file, you can take advantage of Python’s built-in json package. In Chapters 4 and 5, we’ll take up the data sourcing discussion in greater detail. In particular, you’ll learn how to load specific data from different sources into data structures in your Python script for further processing. For now, we’ll take a brief look at each of the common source types mentioned in the preceding list. APIs Perhaps the most common way of acquiring data today is via an API (a software intermediary that enables two applications to interact with each other). As mentioned, to take advantage of an API in Python, you may need to install a wrapper for that API in the form of a Python library. The most common way to do this nowadays is via the pip command. Not all APIs have their own Python wrapper, but this doesn’t necessarily mean you can’t make calls to them from Python. If an API serves HTTP requests, you can interact with that API from Python using the Requests library. This opens you up to thousands of APIs that you can use in your Python code to request datasets for further processing. When choosing an API for a particular task, you should take the following into account: Functionality Many APIs provide similar functionalities, so you need to understand your precise requirements. For example, many APIs let you conduct a web search from within your Python script, but only
- 31. some allow youto narrow down your search results by date of publication. Cost Many APIs allow you to use a so-called developer key, which is usually provided for free but with certain limitations, such as a limited number of calls per day. Stability Thanks to the Python Package Index (PyPI) repository (https://pypi.org), anyone can pack an API into a pip package and make it publicly available. As a result, there’s an API (or several) for virtually any task you can imagine, but not all of these are completely reliable. Fortunately, the PyPI repository tracks the performance and usage of packages. Documentation Popular APIs usually have a corresponding documentation website, allowing you to see all of the API commands with sample usages. As a good model, look at the documentation page for the Nasdaq Data Link (aka Quandl) API (https://docs.data.nasdaq.com/docs/python-time-series), where you’ll find examples of making different time series calls. Many APIs return results in one of the following three formats: JSON, XML, or CSV. Data in any of these formats can easily be translated into data structures that are either built into or commonly used with Python. For example, the Yahoo Finance API retrieves and analyzes stock data, then returns the information already translated into a pandas DataFrame, a widely used structure we’ll discuss in Chapter 3. Web Pages Web pages can be static or generated on the fly in response to a user’s interaction, in which case they may contain information from many different sources. In either case, a program can read a web page and extract parts of it. Called web scraping, this is quite legal as long as the page is publicly available. A typical scraping scenario in Python involves two libraries: Requests and BeautifulSoup. Requests fetches the source code of the page, and then BeautifulSoup creates a parse tree for the page, which is a hierarchical representation of the page’s content. You can search the parse tree and
- 32. extract data fromit using Pythonic idioms. For example, the following fragment of a parse tree: [<td title="03/01/2020 00:00:00"><a href="Download.aspx?ID=630751" id="lnkDownload630751" target="_blank">03/01/2020</a></td>, <td title="03/01/2020 00:00:00"><a href="Download.aspx?ID=630753" id="lnkDownload630753" target="_blank">03/01/2020</a></td>, <td title="03/01/2020 00:00:00"><a href="Download.aspx?ID=630755" id="lnkDownload630755" target="_blank">03/01/2020</a></td>] can be easily transformed into the following list of items within a for loop in your Python script: [ {'Document_Reference': '630751', 'Document_Date': '03/01/2020', 'link': 'http://www.dummy.com/Download.aspx?ID=630751'} {'Document_Reference': '630753', 'Document_Date': '03/01/2020', 'link': 'http://www.dummy.com/Download.aspx?ID=630753'} {'Document_Reference': '630755', 'Document_Date': '03/01/2020', 'link': 'http://www.dummy.com/Download.aspx?ID=630755'} ] This is an example of transforming semistructured data into structured data. Databases Another common source of data is a relational database, a structure that provides a mechanism to efficiently store, access, and manipulate your structured data. You fetch from or send a portion of data to tables in the database using a Structured Query Language (SQL) request. For instance, the following request issued to an employees table in the database retrieves the list of only those programmers who work in the IT department, making it unnecessary to fetch the entire table: SELECT first_name, last_name FROM employees WHERE department = 'IT' and title = 'programmer'
- 33. Python has abuilt-in database engine, SQLite. Alternatively, you can employ any other available database. Before you can access a database, you’ll need to install the database client software in your environment. In addition to the conventional rigidly structured databases, there’s been an ever-increasing need in recent years for the ability to store heterogeneous and unstructured data in database-like containers. This has led to the rise of so-called NoSQL (non-SQL or not only SQL) databases. NoSQL databases use flexible data models, allowing you to store large volumes of unstructured data using the key-value method, where each piece of data can be accessed using an associated key. Here’s what our earlier sample financial statement might look like if stored in a NoSQL database: key value --- ----- ... 26 GoodComp shares soared as much as 8.2% on 2021-01-07 after the company announced ... The entire statement is paired with an identifying key, 26. It might seem odd to store the entire statement in a database. Recall, however, that several possible records can be extracted from a single statement. Storing the whole statement gives us the flexibility to extract different pieces of information at a later time. Files Files may contain structured, semistructured, and unstructured data. Python’s built-in open() function allows you to open a file so you can use its data within your script. However, depending on the format of the data (for example, CSV, JSON, or XML), you may need to import a corresponding library to be able to perform read, write, and/or append operations on it. Plaintext files don’t require a library to be further processed and are simply considered as sequences of lines in Python. As an example, look at the following message that a Cisco router might send to a logfile: dat= 'Jul 19 10:30:37' host='sm1-prt-highw157' syslogtag='%SYS-1-CPURISINGTHRESHOLD:'
- 34. msg=' Threshold: TotalCPU Utilization(Total/Intr): 17%/1%, Top 3 processes(Pid/Util): 85/9%, 146/4%, 80/1%' You’ll be able to read this line by line, looking for the required information. Thus, if your task is to find messages that include information about CPU utilization and extract particular figures from it, your script should recognize the last line in the snippet as a message to be selected. In Chapter 2, you’ll see an example of how to extract specific information from text data using text processing techniques. The Data Processing Pipeline In this section, we’ll take a conceptual look at the steps involved in data processing, also known as the data processing pipeline. The usual steps applied to the data are: 1. Acquisition 2. Cleansing 3. Transformation 4. Analysis 5. Storage As you’ll see, these steps aren’t always clear-cut. In some applications you’ll be able to combine multiple steps into one or omit some steps altogether. Acquisition Before you can do anything with data, you need to acquire it. That’s why data acquisition is the first step in any data processing pipeline. In the previous section, you learned about the most common types of data sources. Some of those sources allow you to load only the required portion of the data in accordance with your request. For example, a request to the Yahoo Finance API requires you to specify the ticker of a company and a period of time over which to retrieve stock
- 35. prices for thatcompany. Similarly, the News API, which allows you to retrieve news articles, can process a number of parameters to narrow down the list of articles being requested, including the source and date of publication. Despite these qualifying parameters, however, the retrieved list may still need to be filtered further. That is, the data may require cleansing. Cleansing Data cleansing is the process of detecting and correcting corrupt or inaccurate data, or removing unnecessary data. In some cases, this step isn’t required, and the data being obtained is immediately ready for analysis. For example, the yfinance library (a Python wrapper for Yahoo Finance API) returns stock data as a readily usable pandas DataFrame object. This usually allows you to skip the cleansing and transformation steps and move straight to data analysis. However, if your acquisition tool is a web scraper, the data certainly will need cleansing because fragments of HTML markup will probably be included along with the payload data, as shown here: 6.tThe development shall comply with the requirements of DCCâx80x99s Drainage Division as followsrnrn After cleansing, this text fragment should look like this: 6. The development shall comply with the requirements of DCC's Drainage Division as follows Besides the HTML markup, the scraped text may include other unwanted text, as in the following example, where the phrase A View full text is simply hyperlink text. You might need to open this link to access the text within it: Permission for proposed amendments to planning permission received on the 30th A View full text You can also use a data cleansing step to filter out specific entities. After requesting a set of articles from the News API, for example, you may need to select only those articles in the specified period where the titles include a
- 36. money or percentphrase. This filter can be considered a data cleansing operation because the goal is to remove unnecessary data and prepare for the data transformation and data analysis operations. Transformation Data transformation is the process of changing the format or structure of data in preparation for analysis. For example, to extract the information from our GoodComp unstructured text data as we did in “Structured Data,” you might shred it into individual words or tokens so that a named entity recognition (NER) tool can look for the desired information. In information extraction, a named entity typically represents a real-world object, such as a person, an organization, or a product, that can be identified by a proper noun. There are also named entities that represent dates, percentages, financial terms, and more. Many NLP tools can handle this kind of transformation for you automatically. After such a transformation, the shredded GoodComp data would look like this: ['GoodComp', 'shares', 'soared', 'as', 'much', 'as', '8.2%', 'on', '2021-01-07', 'after', 'the', 'company', 'announced', 'positive', 'early-stage', 'trial', 'results', 'for', 'its', 'vaccine'] Other forms of data transformation are deeper, with text data being converted into numerical data. For example, if we’ve gathered a collection of news articles, we might transform them by performing sentiment analysis, a text processing technique that generates a number representing the emotions expressed within a text. Sentiment analysis can be implemented with tools like SentimentAnalyzer, which can be found in the nltk.sentiment package. A typical analysis output might look like this: Sentiment URL --------- ---------------------------------------------------------------- 0.9313 https://mashable.com/uk/shopping/amazon-face-mask-store-july-28/ 0.9387 https://skillet.lifehacker.com/save-those-crustacean-shells-to -make-a-sauce-base-1844520024
- 37. Each entry inour dataset now includes a number, such as 0.9313, representing the sentiment expressed within the corresponding article. With the sentiment of each article expressed numerically, we can calculate the average sentiment of the entire dataset, allowing us to determine the overall sentiment toward an object of interest, such as a certain company or product. Analysis Analysis is the key step in the data processing pipeline. Here you interpret the raw data, enabling you to draw conclusions that aren’t immediately apparent. Continuing with our sentiment analysis example, you might want to study the sentiment toward a company over a specified period in relation to that company’s stock price. Or you might compare stock market index figures, such as those on the S&P 500, with the sentiment expressed in a broad sampling of news articles for this same period. The following fragment illustrates what the dataset might look like, with S&P 500 data shown alongside the overall sentiment of that day’s news: Date News_sentiment S&P_500 --------------------------------------- 2021-04-16 0.281074 4185.47 2021-04-19 0.284052 4163.26 2021-04-20 0.262421 4134.94 Since both the sentiment figures and stock figures are expressed in numbers, you might plot two corresponding graphs on the same plot for visual analysis, as illustrated in Figure 1-1.
- 38. Figure 1-1: Anexample of visual data analysis Visual analysis is one of the most commonly used and efficient methods for interpreting data. We’ll discuss visual analysis in greater detail in Chapter 8. Storage In most cases, you’ll need to store the results generated during the data analysis process to make them available for later use. Your storage options typically include files and databases. The latter is preferable if you anticipate frequent reuse of your data. The Pythonic Way When doing data science with Python, your code is expected to be written in a Pythonic way, meaning it should be concise and efficient. Pythonic code is often associated with the use of list comprehensions, which are ways to implement useful data processing functionality with a single line of code. We’ll cover list comprehensions in more detail in Chapter 2, but for now, the following quick example illustrates how the Pythonic concept works in
- 39. practice. Say youneed to process this multisentence fragment of text: txt = ''' Eight dollars a week or a million a year - what is the difference? A mathematician or a wit would give you the wrong answer. The magi brought valuable gifts, but that was not among them. - The Gift of the Magi, O'Henry''' Specifically, you need to split the text by sentences, creating a list of individual words for each sentence, excluding punctuation symbols. Thanks to Python’s list comprehension feature, all of this can be implemented in a single line of code, a so-called one-liner: word_lists = [[w.replace(',','') ❶ for w in line.split() if w not in ['-']] ❷ for line in txt.replace('?','.').split('.')] The for line in txt loop ❷ splits the text into sentences and stores those sentences in a list. Then the for w in line loop ❶ splits each sentence into individuals words and stores the words in a list within the larger list. As a result, you get the following list of lists: [['Eight', 'dollars', 'a', 'week', 'or', 'a', 'million', 'a', 'year', 'what', 'is', 'the', 'difference'], ['A', 'mathematician', 'or', 'a', 'wit', 'would', 'give', 'you', 'the', 'wrong', 'answer'], ['The', 'magi', 'brought', 'valuable', 'gifts', 'but', 'that', 'was', 'not', 'among', 'them'], ['The', 'Gift', 'of', 'the', 'Magi', "O'Henry"]] Here you’ve managed to accomplish two steps of the data processing pipeline within a single line of code: cleansing and transformation. You’ve cleansed the data by removing punctuation symbols from the text, and you’ve transformed it by separating the words from each other to form a word list for each sentence. If you’ve come to Python from another programming language, try implementing this task with that language. How many lines of code does it take? Summary After reading this chapter, you should have a cursory understanding of the main categories of data, where data comes from, and how a typical data
- 40. processing pipeline isorganized. As you’ve seen, there are three major categories of data: unstructured, structured, and semistructured. The raw input material in a data processing pipeline is typically unstructured data, which is passed through cleansing and transformation steps to turn it into structured or semistructured data that is ready for analysis. You also learned about data processing pipelines that use structured or semistructured data from the start, acquired from an API or a relational database.
- 41. 2 PYTHON DATA STRUCTURES Datastructures organize and store data, making it easier to access the data. Python ships with four data structures: lists, tuples, dictionaries, and sets. These structures are easy to work with, yet they can be used to tackle complex data operations, making Python one of the most popular languages for data analysis. This chapter will cover Python’s four built-in data structures, with an emphasis on the features that allow you to easily build functional data- centric applications with minimal coding. You’ll also learn how to combine the basic structures into more complex structures, such as a list of dictionaries, to more accurately represent real-world objects. You’ll apply this knowledge to the field of natural language processing and in a short introduction to processing photographs. Lists A Python list is an ordered collection of objects. The elements in a list are separated by commas, and the entire list is enclosed in square brackets, as shown here:
- 42. [2,4,7] ['Bob', 'John', 'Will'] Listsare mutable, meaning you can add, remove, and modify a list’s elements. Unlike sets, which we’ll discuss later in the chapter, lists can have duplicate elements. Lists contain elements that represent series of usually related, similar things that can be logically grouped together. A typical list contains only elements belonging to a single category (that is, homogeneous data, such as people’s names, article titles, or participant numbers). Understanding this point is essential when it comes to choosing the right tool for the task at hand. If you need a structure to include objects with different properties, consider using a tuple or a dictionary. NOTE Although lists are generally understood to be homogeneous, Python does allow you to have elements of different data types in the same list. This list, for example, includes both strings and integers: ['Ford', 'Mustang', 1964] Creating a List To create a basic list, simply place a sequence of elements inside square brackets and assign the sequence to a variable name: regions = ['Asia', 'America', 'Europe'] In practice, however, lists are commonly populated dynamically from scratch, often using a loop that calculates one item per iteration. In such cases, your first step is to create an empty list, as illustrated here: regions = [] Once you’ve created a list, you can add, remove, and sort items in that list as needed. You can perform these and other tasks using Python’s various
- 43. list object methods. UsingCommon List Object Methods List object methods are functions that implement particular behaviors within lists. In this section we’ll look at some common list object methods, including append(), index(), insert(), and count(). To practice using them, start by creating a blank list. You’ll build it into a to-do list as you go along, filling it with chores and organizing it: my_list = [] Perhaps the most common list object method is append(). It adds an item to the end of the list. You can use append() to add some chores to your to-do list, as shown here: my_list.append('Pay bills') my_list.append('Tidy up') my_list.append('Walk the dog') my_list.append('Cook dinner') The list now contains four items, in the order in which they were appended: ['Pay bills', 'Tidy up', 'Walk the dog', 'Cook dinner'] Each item in a list has a numeric key known as an index. This feature enables a list to keep its items in a specified order. Python uses zero-based indexing, meaning the initial item of a sequence is assigned the index 0. To access an individual item from a list, specify the name of the list, followed by the index of the desired item in square brackets. For example, here’s how to print just the first item of your to-do list: print(my_list[0]) The print() function yields the following output: Pay bills
- 44. You can usea list’s indices not only to access a required item but also to insert a new item at a certain position in the list. Say you want to add a new chore to your to-do list between walking the dog and cooking dinner. To make this insertion, you first use the index() method to determine the index of the item before which you want to insert the new item. Here, you’ll store it in the variable i: i = my_list.index('Cook dinner') This will become the index for the new chore, which you can now add using the insert() method, as shown here: my_list.insert(i, 'Go to the pharmacy') The new chore is added to the list at the specified index, shifting all subsequent chores down by one. This is what the updated list looks like: ['Pay bills', 'Tidy up', 'Walk the dog', 'Go to the pharmacy', 'Cook dinner'] Because lists allow for duplicate items, you may need to check how many times a certain item appears in a list. This can be done with the count() method, as illustrated in the following example: print(my_list.count('Tidy up')) The print() function reveals only one instance of 'Tidy up' in the list, but it might be a good idea to include this item in your daily list more than once! NOTE The entire list of list object methods can be found in the Python documentation at https://docs.python.org/3/tutorial/datastructures.html. Using Slice Notation It’s possible to access a range of items from a sequential data type such as a list by using slice notation. To take a slice of a list, specify the index of the starting position and the index of the ending position plus 1. Separate the
- 45. Another Random ScribdDocument with Unrelated Content
- 46. elektrischen Laterne aufder Generalstabskarte zu einem Granitblock am Weg. Unheimlich war das... Besser der Tag als die Nacht. Aber der Unteroffizier auf dem Bock drehte sich um und sagte: „Die Nacht ist gut! Niemand schießt. Man kommt leicht nach vorn...“ Und Schjelting dachte sich: ‚Wer bist Du eigentlich da oben, der meine Gedanken errät?... Du hast so eine sonderbare Stimme...‘ Der Schlitten hielt an. Denn nun kam ihm auch von links ein Zug von Planwagen entgegen. Sie fuhren ganz langsam. Ihre Laternen schaukelten. „Was ist in den Karren, Unteroffizier?“ „Verwundete, Euer Hochwohlgeboren!“ Hundert Karren ... zweihundert ... dreihundert... Schjelting sagte sich: Still seid Ihr da drinnen ... seltsam still... Weiter ... weiter... Im Lichtstreifen der Laterne Grabkreuze auf den Feldern. Immer mehr und mehr. Rehwild huschte zwischen den verwitterten Pickelhauben, den vergilbten Tannenkränzen auf den Russengräbern. Jetzt ganze Reihen da, wo man die Gefallenen früherer Kämpfe in zugeschütteten Schützengräben beerdigt hatte. Die ganze Leere umher schien auf einmal ein weiter Kirchhof. Die Straße war öde geworden. Nur ein Licht. Ein Leiterwagen wankte heran. Vorn, auf dem Stroh, ein Pope und eine Frauengestalt. Dahinter ein Sarg. Auf ihm ein Säbel und eine russische Generalsmütze. Vorbei. Der Schlitten schwenkte plötzlich wie erschrocken von der Hauptstraße ab und glitt seitwärts auf einen schmalen Weg in den Tannenwald hinein. Nun war es stockdunkel. Kein Laut umher. Dann ein Ruck. Ein Halt. „Belieben Sie, die Leiter hinunterzusteigen!“ Eine Luke in dem beschneiten Boden öffnete sich. Ein vorsündflutliches Ungetüm wohnte in dem unterirdischen, elektrisch
- 47. beleuchteten Raum. EinRiesenmörser mit seiner dreifachen Auswölbung und seinen Schaufelrädern, das glotzende Maul steil aus seinem Versteck nach der Decke von verschneiten Fichtenzweigen gerichtet, die den Lindwurm vor Feindesaugen schützte und zurückgeschoben wurde, wenn er den Inhalt der halbmannslangen Geschoßkörbe neben ihm brüllend über die Wipfel des Tannenwalds in den Himmel hinaufspie. Ein Offizier saß auf einem Schemel, an die Lafette angelehnt. Schjelting hielt ihn für den General Schiraj und trat auf ihn zu. Aber das Gesicht, das sich langsam nach ihm wandte, kannte er nicht. Es hatte den länglichen Schnitt der Ukräne, war abgezehrt bis auf die Knochen, mit in den Höhlen eingesunkenen Augen. Schjelting dachte sich: ‚Der sieht ja aus wie der Tod!‘ Er reichte dem bleichen, unbekannten Offizier die Hand. Die des Anderen war kalt und bleiern. „Der General Schiraj erwartet Sie vorne in der Stellung...“ Der Offizier sagte es dumpf und teilnahmlos. „Ist es weit bis dahin?“ „Sie sind hier dicht hinter der Front. Eine halbe Stunde zu Fuß.“ „Kann ich nicht fahren?“ „Man wird sie umwerfen! Es sind überall frische Granatlöcher im Schnee. Man muß den Fußstapfen dazwischen folgen.“ „Und wie kommt man bei Tag zurück? Bald graut der Morgen...“ „Zurück? Nun ... irgendwie ... macht nichts...“ Der Artillerie-Offizier sagte es mit tiefer Gleichgiltigkeit und schaute, den Rücken an die eine Auswulstung des Mörserschlunds gelehnt, geistesabwesend vor sich hin. Schjelting fröstelte. „Man kann dabei fallen...“ „Man fällt ... man lebt ... einerlei...“ „... wie denn einerlei?... Nun ja ... wenn man nur siegt...“
- 48. „Man siegt ...man siegt nicht... Gott allein weiß es...“ Schjelting dachte sich: ‚Nun, Du Bruder mit dem Totenkopf ... hast Du Furcht...?‘ Da sah er auf dessen Brust das Georgskreuz, die Auszeichnung für Tapferkeit. „Wie lange sind Sie im Felde?...“ „Seit Mitte Juli alten Stils!“ sagte der fahle Artillerist und blickte stumpf nach dem stählernen Götzen neben ihm, der, einem kleinen Elefanten an Größe nah, fast bis an die Decke aus beschneitem Tannenreisig reichte. „Wir haben viel zusammen durchgemacht ... der Spitzbube da und ich...“ Er horchte jäh mit seinem gespannten Geistergesicht in die Nacht und machte dann eine matte Handbewegung. „Noch ist es ja dunkel ... ich höre immer die Flieger brummen... Macht nichts...“ Und Schjelting sagte sich:... ‚Der Krieg..! In Petersburg ... in den Salons meiner Freundinnen ... hatten wir ihn seit Jahren auf den Lippen... c’est ma guerre... der Krieg... Das war der Einzug in Berlin ... Mütterchen Moskau in Fahnenpracht... Glockenklang von der Isaak-Kathedrale... Das waren Orden ... Gelder ... Exzellenzentitel... Nun ist dies hier der Krieg ... dieses unbestimmte Schwarz ... diese Grabkreuze ... diese weite Leere ... diese furchtbare, erwartungsvolle Stille wie vor etwas Ungeheurem ... dieser Mann da mit den niedergebrochenen Nerven...‘ Er mußte sich zusammennehmen, um einen Schauer zu unterdrücken. „Nun denn ... ich gehe...“ „Mit Gott!“ Der Unteroffizier schritt im Finstern voraus, schwer, bärtig, im Pelz, aufrecht wie ein Bär durch den Schnee. Schjelting folgte ihm. Er sagte sich: Wir führen Rußland nicht mehr! Ich folge diesem Stück russischer Erde da vor mir, die wir aufstehen und wandeln hießen, — folge ihr in das dunkle Land vor mir hinein... Ein Aufstöhnen des Winterwinds. Er hielt die Pelzärmel schützend vor Mund und Nase. Ihm wurde beinahe übel. Das war wieder der
- 49. belgische Geruch... „Unteroffizier ...liegen hier irgendwo Leichen?“ „Überall, Euer Hochwohlgeboren!“ „Warum bergt Ihr sie nicht?“ „Man findet sie nicht im Wald und Schnee, Euer Hochwohlgeboren...“ In der Ferne, über den deutschen Stellungen, stieg eine Rakete auf. Eine märchenhafte, zauberweiße Lichtkugel stand am Himmel, erhellte mild die ganze Gegend... „Unteroffizier ... halt ... halt...“ „Was denn, Euer Hochwohlgeboren?“ „Der Hochwald da neben uns ist ja voll Menschen ... da ... sie sitzen um das Loch im Schnee ... ein Hauptmann vorne...“ „Die Unsern, Euer Hochwohlgeboren!“ „Erfrieren die denn nicht?“ „Wie sollten sie frieren? Sie sind doch alle tot. Die Granate... Man wird sie morgen holen...“ Ah... c’est ma guerre... Schjelting dachte sich: Dein Leben war dieser Nachtwanderung durch den Schnee geweiht. Durch ganz Europa bist Du gefahren, in allen Zungen hast Du gesprochen, an Menschen aller Art hast Du Rubel verteilt, Du kauftest die Seelen, die Druckerschwärze und die Telegrafendrähte ... prägtest, was da kommen sollte, in Formeln und Methoden. Gut... Aber wer findet sie jetzt wieder, die große Rechenmaschine — wer bedient sie in dieser furchtbaren, bleiernen Nacht über Europa? Er blieb stehen. Durch das Dunkel kam ein Laut, der ihm die Haare sträuben machte. Nicht von Menschenstimmen... „Unteroffizier... Was ist das?... Nie vernahm ich es...“ „Der Schrei eines sterbenden Pferds, Euer Hochwohlgeboren!“
- 50. Der Riese imPelz vor ihm ging weiter, bückte sich plötzlich an einer dunkleren Stelle des Bodens, bekreuzigte sich, las etwas auf und legte es vorsichtig zur Seite. Es war ein abgerissener Menschenarm... „Unteroffizier...“ „Vorwärts, Euer Hochwohlgeboren! Es wird schon hell!“ „Unteroffizier ... ich will lieber umkehren...“ „Nein, Euer Hochwohlgeboren...“ „Wie denn...“ „Man befahl mir, Euer Hochwohlgeboren nach vorn zu bringen...“ „Doch nicht mit Gewalt...“ „Man befahl mir, Euer Hochwohlgeboren!“ Im ersten Morgengrauen stand der bärtige, finstere Riese vor ihm wie das heilige Rußland selber. An Stelle der Axt, die er sonst als Waldarbeiter im weißen, heimischen Birkensumpf über dem roten Hemd getragen, hatte er jetzt das vollgeladene Magazingewehr über den Pelz gehängt. Er sah düster und drohend aus. Schjelting dachte sich: Wenn ich umdrehe, ist er im Stande und schickt mir, dem Zivilisten, den er vor den General führen soll, aus mißverstandenem Diensteifer eine Kugel nach. Auf einmal begriff er, daß er in Lebensgefahr war — nach hinten sowohl, wo der Tag aufstieg, wie nach vorne, wo der Deutsche war. Er spürte kalten Schweiß unter dem Rand seiner Pelzmütze. Er merkte, daß seine Nerven ihn verließen. Er hatte keinen Willen mehr. Er tat, was dieser Bauer in Feldbraun von ihm wollte. Er ging weiter und sagte sich: Die Welt verkehrt sich. Ich dachte, den Muschik gegen den Feind zu schicken. Statt dessen führt der Muschik jetzt mich an die Front... Man sah nun schon weithin die verschneite, leichtgewellte Ebene. Sie lag völlig tot und leer. Eine ausgestorbene Öde wie die Tundren Sibiriens. Ein paar Krähen das Einzige, was sich regte. Ihr Krächzen der einzige Laut. Schjelting dachte sich: dabei hausen da, soweit das
- 51. Auge reicht, tausendeund zehntausende menschliche Maulwürfe in ihren unterirdischen Gängen, huschen geschäftig hin und her, graben, wühlen, scharren sich immer tiefer gegeneinander ein ... leben in Löchern ... das leise Rauchgekräusel aus den gemauerten Kaminen ihrer Unterstände allein verrät das Dasein der Höhlenbewohner... „Heute sind sie ganz still ... die Deutschen ...,“ sagte der Unteroffizier in seinem rauhen Brummbaß durch das Todesschweigen. „Werden sie nicht noch schießen?“ „Warum schießen? Es ist Winter. Sie schlafen. Wie wir...“ Sie gingen durch den Schlammpfuhl des russischen Labyrinths von Schützengräben, immer weiter im Zickzack, ganz nach vorn. Der Tag wollte nicht recht kommen. Nebelschwaden strichen wieder über die unterirdische Stadt hin und hüllten sie in zähes Grau. „Wo ist der General?“ „Bald, Euer Hochwohlgeboren!“ Schjelting biß die Zähne zusammen. Er dachte sich: Was ist das Alles? Wohin geh’ ich?... Hier hat nun doch die Welt ein Ende... Da, wo der Sanitätssoldat mit dem Genfer Kreuz am Pelzärmel im Schutz des äußersten Grabens steht... Was hast Du mich an der Schulter zu fassen?... Packe Dich, Kerl... Er sah zwei fanatische blaue Augen auf sich gerichtet und erkannte im Nebel den Professor Korsakoff. Der zog ihn zwei Schritte zur Seite. „Sie suchen Schiraj, Nicolai Wassiljewitsch?“ „Ja. Ihn.“ „Sie wollen auch ihn für Eure Wilnaer Pläne gewinnen?“ „Jeden, der noch in letzter Stunde auf mich hört!“ „Kehren Sie um, Nicolai Wassiljewitsch...“
- 52. „Wie?“ „Man wird Siegeleiten!... Gehen Sie auf Ihre Güter! Warten Sie dort innen in Rußland den Gang der Dinge ab!“ Über Schjelting kam der Zorn. Er richtete sich in seinem früheren Hochmut auf. „Habt Ihr ein Recht, mich zu verschicken — he?“ „Man warnt Sie!... Sie sind uns hier im Wege...“ „Euch frage ich nicht! Wo ist der General?“ „Sie wollen trotzdem zu ihm?“ „Ja!“ „Nun denn, mit Gott! Ich begleite Sie!“ sagte Korsakoff ruhig. „Kommen Sie! Wir steigen hier herauf...!“ „Wie das? Vor den Schützengraben...?“ „Man sieht ja nicht zehn Fuß weit im Nebel! Wie sollte der Feind uns bemerken! Vorwärts...“ „Belieben Euer Hochwohlgeboren gut Acht zu geben. Der Weg durch den Drahtverhau ist eng...“ Der riesenhafte Unteroffizier stapfte voraus. Es ging quer wie durch einen schmalen, tief verschneiten Weinberg, dessen Pfähle kahl aus dem Schnee ragten. Schnee hing auch an den Drähten, die sie kreuz und quer verbanden. Dies seltsame, verstrickte Band verlor sich zu beiden Seiten ins Wesenlose des Nebels. Unwillkürlich dachte sich Schjelting: Es reicht vom Njemen bis zu den Karpathen. Es spannt sich von der Schweiz bis zur Nordsee. Nie sah die Welt etwas Ähnliches... Und dann ein Schrecken in ihm: Was tue ich außerhalb von ihm ... da draußen ... im unbetretenen Land ... im schweigenden Reich des Todes zwischen Freund und Feind...? Und ist da der Platz für einen General...? Sie waren einen Abhang hinabgestiegen. Hinter ihnen, auf der Höhe des Kammes, zeichnete sich noch unbestimmt der Rand des
- 53. Schützengrabens ab. Einpaar Pelzzipfel von hohen Mützen bewegten sich unruhig dahinter, als lauerten da Wölfe. Vor ihnen endete die Böschung jäh, in einem senkrecht an der Rückwand einer verlassenen Kiesgrube zehn Fuß tief abstürzenden toten Winkel. Schjelting stand allein mit dem Panslawisten und dem bewaffneten, finsteren Muschik hart an dem Abgrund in dem dicken, totenstillen Nebel. Es schien ihm, als seien die drei die letzten Menschen auf der vom Krieg in Nichts verwandelten Welt. Seine Stimme war plötzlich heiser vor Schrecken. „Wo ist Schiraj?“ „Geduld! Blicken Sie nach vorn, Euer Hochwohlgeboren!...“ „Er kann doch nicht vom Feind her kommen...“ „Er ist überall...“ Schjelting hob das verzerrte Gesicht. „Ich höre seine Stimme ganz deutlich da hinten ... im Schützengraben...“ „Sie täuschen sich, Nicolai Wassiljewitsch...“ „Und Morskois Baß!... Wie kommt er hierher?“ „Er ist es nicht...“ „Er muß im Automobil an mir vorbeigefahren sein ... laßt mich zurück...“ „Still... Erbarmen Sie sich ... der Feind hört uns ja...“ „Zurück...“ „Beruhigen sich Euer Hochwohlgeboren...“ „Warum machst Du Dich schußfertig ... um Gotteswillen ...?“ „Um den General zu schützen!... Da kommt er ja auf uns zu...“ „Wo denn ... wo?“
- 54. „Da vor uns... vom Feinde her... Er ist ein Falke... Ihm tun die Kugeln nichts...“ „Ich kann ihn nicht sehen...“ „Der Nebel ist zu dicht... Beugen sich Euer Hochwohlgeboren nur noch etwas mehr vor...“ „Da ist kein General... Ihr lügt...“ Nicolai von Schjelting wollte sich umwenden. Vor sich sah er Korsakoffs entschlossenes und beinahe leidendes Fanatikergesicht. Ein jäher, schlenkernder Handwink des Panslawisten zu dem Unteroffizier hin, so, als scheuche er eine Fliege: „Nun, Bruder: Mit Gott!“ In den Schützengräben hinten drehten sich einen Augenblick horchend bärtige Köpfe. Das kurze, scharfe ‚Peng‘ vor dem Drahtverhau war der erste Schuß dieses Morgens. Die Posten auf dem Auftritt spähten durch den Spalt der Schutzschilder: „Wer schießt denn da vorn?“ „Kommen die Deutschen?“ „Nein. Es ist nichts!“ „Es ist ja alles dick voll Nebel, Brüder...“ „Noch ist Nebel. Aber er löst sich. Bald haben wir hellen Tag.“ Das Licht kam. Ein grauer Winterhimmel wölbte sich über der Welt. Nicolai von Schjelting schaute noch einmal zu ihm auf, allein lang hingestreckt am Fuß des Kieshangs, wo ihn Niemand sah, einsam im leeren Todesland zwischen den beiden Linien, und das war seine letzte Erkenntnis: Der Krieg ... mein Krieg ... ich habe ihn gerufen ... da ist er ... geht über mich hinweg ... und all das hinter mir... Im russischen Schützengraben, vierzig, fünfzig Schritte entfernt, raunte es: Er glaubte, den Baß Schirajs zu unterscheiden, die Stimmen der Anderen, während seine Augen sich in dem fahlen
- 55. Nichts über ihmerlöschend verloren. Durch diese Leere senkte sich ein pfeilschnelles Heulen wie ein Raubvogel auf die Russenschanze dahinten, krallte sich ein, schleuderte mit einem Donnerschlag Schnee, Erde, Gasqualm, Bretter, Draht und Menschen kirchturmhoch in die Luft, spielte da oben mit dem Kopf des Generals Schiraj, mit dem Rumpf des Hofmeisters, den Gliedern des Unteroffiziers, den Fetzen des Panslawisten und hüllte vergrollend den Greuel in schmutzigen Rauch. Aber schon raste der nächste der stählernen Stoßvögel heran. Schwärme von ihnen schwirrten unsichtbar aus unbekannter deutscher Ferne. Es waren die Donner des jüngsten Gerichts, unter denen an diesem Februarmorgen Nicolai von Schjelting beim Beginn der Winterschlacht von Masuren in das Nichts hinüberging. Ungeheure Leiterwagen rasselten am Himmel, Walfische durchzischten das Luftmeer, Riesen gurgelten sich und wieherten in den Wolken, Schiffssirenen heulten, Gassenjungen pfiffen schrill durch die Finger, Teufel johlten, Zyklopen hämmerten in wildem Takt auf dröhnendem Ambos und drüben, in den Russenlinien, verwandelte sich jäh das Toben des unsichtbaren wütenden Heers in aufspritzenden weißen Schnee und auffliegenden schwarzen Qualm und aufschießende rote Feuerzungen und stille feldbraune Hügel von Menschen, in einstürzende Erdhöhlen, klaffende Krater, betrunken umkippende Mammuth-Kanonen, sich langsam verneigende Kirchtürme, in der Luft tanzende Bäume, rasch, wie beim Untergang von Pompeji, zu Steinbrüchen sich wandelnde Städtchen. Tagelang und tageweit, von Lasdehnen bis hinter Lyck, donnerte die Winterschlacht. Schjelting hörte es nicht mehr. Er hörte nicht mehr das Hurrah, mit dem sie Alle durch den Schnee herankamen, die er in Nord und Süd in Deutschland geschaut, grau, unermeßlich, unaufhaltsam anschwellend wie das graue Meer zur Stunde der Flut, und über ihn und den Schanzenbrei dahinter wegwogten, gen Osten, den Russen nach. Dann, viel später, sagte eine Stimme:
- 56. „Die Kiesgrube dakommt uns gerade zu paß! Da liegt ohnedies schon Einer drin!“ Es waren deutsche Landsturm-Männer. Sie hatten ihre Gewehre in Pyramiden gestellt und die Pfeifen im Bart. Sie trugen die steifgefrorenen toten Russen aus den Schützengräben und senkten sie auf den Grund des Abhangs, auf dem Schjelting lag. Die Muschiks kamen zu ihm herab und leisteten ihm, der sie in den Krieg geführt, im Tode Gesellschaft. Einer nach dem andern legte sich auf ihn. Sie bedeckten ihn, türmten sich über ihn mit ihren feierlichen und starren, groben Gesichtern. Die menschgewordene und wieder erstorbene breite russische Erde wurde sein Grab. Die Unzähligen machten auch ihn zur Zahl. Die Namenlosen löschten seinen Namen, schieden ihn aus der Erinnerung aus, als einen Verschollenen, von dem Niemand wußte, wo und wie er sein Ende gefunden. „...’Morgen, Leute!“ „Guten Morgen, Exzellenz!“ Der weißhaarige, blitzäugige Preußengeneral, der, die Zigarre im Mund, die Autobrille über dem Mützenrand, die Hände in den Manteltaschen, von der Chaussee her mit seinem Stab über das Feld kam, war sehr guter Dinge. In seinem Abschnitt hatte die Geschichte geklappt. Überall. Fern, gegen Süden hin, tönten noch in langen Abständen die letzten Donnerschläge des verhallenden Wintergewitters. „Ach, hören Sie ’mal, lieber Isebrink...“ „Exzellenz...“ Der Generalstabs-Major Isebrink trat, die Rechte an dem Helm, an den Kommandierenden heran. „Haben Sie die Tagebücher bei sich, die man da oben in dem Mörsertrichter bei dem russischen General ohne Kopf gefunden hat?“ „Jawohl, Exzellenz!“
- 57. „Das Zeug scheintmir doch sehr wichtig. Namentlich auch in politischer Hinsicht. Am besten ist es, Sie fahren ’mal selbst rasch zurück und bringen es dem A. O. K. In ein paar Stunden sind Sie ja wieder da!“ „Zu Befehl, Exzellenz!“ Der Major Isebrink jagte in offenem, feldgrauen Rennwagen gegen Westen. Leichtverwundete Offiziere fuhren, soviel Platz war, dichtgedrängt mit verbundenen Köpfen und Armen mit. Der Chauffeur vorn in seinem langen Mantel von chinesischem Ziegenhaar blies auf dem Horn unaufhörlich das Oberkommando- Signal: Straße frei. Nur so kam man, rechts den nicht endenden Kolonnen entgegen, links an dem ebenso endlosen Strom der Gefangenen vorbei. Stund’ um Stunde, über Hügel und Täler floß die fahlbraune, pelzmützige Flut. Sie wanderte auf allen Wegen, soweit nur rechts und links das Auge reichte. Fern am Horizont noch spann sich, in unwahrscheinlicher Länge, der entwaffnete, russische Heerwurm. Die Wälder lebten und entsandten aus ihrem Dickicht tausendköpfige, hohläugige, von fern schon die Hände hochhaltende Herden brauner Hungerleider. Armeestäbe stiegen unvermutet aus den Sümpfen und kamen, die Säbel in der Hand, heran, die Einen düster den Blick am Boden, die Andern froh lachend und schwatzend. Batterienweise standen mit zerschmetterten Schutzschilden und verbeulten Lafetten die genommenen Geschütze am Weg. Nahe der Stadt wurde das Gewimmel der Gefangenen zu einem braunen Meer. Die Offiziere fuhren fast als die einzigen Deutschen durch diese Tausende von struppigen Köpfen und Kerlen, von denen Keiner an Widerstand dachte. Nur ein stilles Gewinsel: „Bissele Brot ... bissele Brot...“ „Maul halten! Pratzen weg...“ Am Weg stand eine von den Russen in die Luft gesprengte Kirche. Aufgebrochene Särge davor, deren sterblichen Inhalt sie in den Teich des sinnlos eingeäscherten Dorfs geworfen hatten.
- 58. Ostpreußischer Landsturm zogvorbei, sah die Greuel der Kosacken. Frisch und grimmig klang sein Gesang: „Oh Hindenburg, oh Hindenburg, wie schön sind Deine Hiebe...“ und ferne noch, in der Eile des Marsches hinter dem Sieg her: „Dein Lorbeer grünt zu jeder Zeit, im Winter auch, wenn’s friert und schneit...“ In der Stadt hingen zwischen zerschossenen Häusern und verkohltem Gebälk die schwarz-weiß-roten und die schwarz-weißen Siegesfahnen. Von allen Türmen läuteten die Glocken die letzte Befreiung Ostpreußens ein. Der Quartiermeister mußte lauter als gewöhnlich sprechen, während er die russischen Schriftstücke in Empfang nahm. „Danke sehr! Sonst noch Etwas?“ „Nein, Herr General!“ „Fahren Sie gleich zurück?“ „Zu Befehl!“ Der Major Isebrink sprang wieder in das Auto, sah nach der Uhr: „Na los, Mann Gottes! Warum nicht über den Marktplatz?“ Aber da sah er selbst: da war kein Durchkommen. Den füllte an den Häusern rechts und links ein brauner, stumpfer, stiller Sumpf von gefangenen Russen, in der Mitte ein grauer, brausender, jubelnder Wildbach von deutschen Kriegern. Sie umstrudelten Etwas, sie hoben die Hände, sie sangen, sie jauchzten... Isebrink sprang im Wagen auf und auch über sein feldgebräuntes Gesicht glitt plötzlich eine wilde, strahlende Freude. Die Autoreihe da vorn wies am Kühler nicht die vier schwarz-weißen rotgerahmten Würfel des Oberkommando-Fähnchens, nicht einmal die Anfangsbuchstaben des Oberbefehlshaber Ost, sonst hier, bei der Wacht an der Weichsel,
- 59. das Sinnbild höchsterKommandogewalt. Eine Purpurstandarte flatterte über den grauen Helmen, dem tausendstimmigen Hurrah. Kaiser Wilhelm stand inmitten der Seinen, der Kriegsherr inmitten des herrlichsten Heers aller Völker und Zeiten. Lange schaute der Major Isebrink hinüber. Dann besann er sich, daß es für ihn höchste Zeit war, weiterzukommen. Er fuhr durch eine Nebengasse. Aber auch da standen die Menschen und schwenkten die Hüte und drängten sich, um von ferne den Kaiser zu sehen. Er beugte sich stehend in dem Wagen vor. Und da die Ostpreußen in ihrem Jubel der Befreiung nicht auf ihn achteten, sprach er, ohne es zu wissen, das Geheimnis seiner Zeit und seines Volks und seiner Siege aus: „Bitte, lassen Sie mich durch: Ich muß in den Dienst!“ E n d e
- 60. Von R ud o l p h S t r a t z erschienen ferner im Verlage Ullstein & Co,Berlin: A r m e T h e a Roman. 221.-230. Tausend P r e i s 1 M a r k Das Schicksal der armen Mädchen, die schutz- und haltlos im Leben stehen, bis es ihnen gelingt, sich zur Höhe emporzuschwingen und in der Liebe und in der Arbeit ihr volles Glück zu finden, hat hier Rudolph Stratz seinem feinsinnigen Roman zugrunde gelegt. Ein froher und gesunder Optimismus spricht aus dem prächtigen und fesselnden Werke. L i e b V a t e r l a n d Roman. 226.-235. Tausend
- 61. P r ei s 1 M a r k (31. Tausend, Preis 3 Mark, vergriffen) Die Erzählung führt in die unruhigen Jahre des Marokkokonfliktes, in die Zeit, die den Gegensatz Deutschlands zu Frankreich zum ersten Male verstärkte. Ein Buch von Deutschlands Kraft und Größe ist dieser Roman mit dem Titel des ehernen Schlachtliedes, ein Buch, das die deutsche Tüchtigkeit verherrlicht.
- 62. Von R ud o l p h S t r a t z erschienen in anderen Verlagen:
- 64. brosch. Mark geb. Mark Der weiße Tod. Romanaus der Gletscherwelt. 25. Auflage 3.— 4.— Herzblut. Roman. 23. Auflage 4.— 5.— Für dich. Roman. 25. Auflage 4.— 5.— Liebestrank. Roman. 20. Auflage 4.— 5.— Du bist die Ruh. Roman aus dem deutschen Leben Moskaus. 10. Auflage 3.50 4.50 Gib mir die Hand. Odessaer Roman. 14. Auflage 4.— 5.— Du Schwert zu meiner Linken Ein
- 65. Linken. Ein Roman aus derdeutschen Armee. 45. Aufl. 4.50 5.50 Seine englische Frau. Roman. 40. Aufl. 4.50 5.50 Stark wie die Mark. Roman. 30. Aufl. 5.— 6.— Ich harr’ des Glücks. Novellen. 6. Aufl. 3.50 4.50 Es war ein Traum. Berliner Novellen. 5. Auflage 3.50 4.50 Die zwölfte Stunde. Novellen. 5. Aufl. 2.— 3.— König und Kärrner. Roman 4.— 5.—
- 66. M ä nn e r u n d V ö l k e r Bismarcks Erbe Von Professor Hans Delbrück • Das englische Gesicht • Die Welt des Islam
- 67. Von Professor Friedrich Delitzsch • Moltke VonA. v. Janson, General d. I. z. D. • Afrikanische Köpfe Von Dr. Carl Peters • Ägypten Von Professor Georg Steindorff • Die Träger des
- 68. deutschen Idealismus Von Professor RudolfEucken • J e d e r B a n d 1 M a r k V e r l a g U l l s t e i n & C o , B e r l i n
- 69. M ä nn e r u n d V ö l k e r Weltpolitik und Weltkatastro phe 1890– 1915 Von Professor Paul Herre • Französische Staatsmänne r Von Max Nordau
- 70. • Die Kolonialreich e der Großmächte Von LegationsratDr. Zimmermann • Russische Köpfe Von Professor Dr. Theodor Schiemann • Deutschlands wirtschaftlich e Widerstandsk raft
- 71. Von Professor GustafCassel • Englische Staatsmänne r Von Sil-Vara • Amerika Von C. A. Bratter • J e d e r B a n d 1 M a r k V e r l a g U l l s t e i n & C o , B e r l i n
- 72. Ullstein & Co BerlinSW 68
- 73. *** END OFTHE PROJECT GUTENBERG EBOOK DAS DEUTSCHE WUNDER: ROMAN *** Updated editions will replace the previous one—the old editions will be renamed. Creating the works from print editions not protected by U.S. copyright law means that no one owns a United States copyright in these works, so the Foundation (and you!) can copy and distribute it in the United States without permission and without paying copyright royalties. Special rules, set forth in the General Terms of Use part of this license, apply to copying and distributing Project Gutenberg™ electronic works to protect the PROJECT GUTENBERG™ concept and trademark. Project Gutenberg is a registered trademark, and may not be used if you charge for an eBook, except by following the terms of the trademark license, including paying royalties for use of the Project Gutenberg trademark. If you do not charge anything for copies of this eBook, complying with the trademark license is very easy. You may use this eBook for nearly any purpose such as creation of derivative works, reports, performances and research. Project Gutenberg eBooks may be modified and printed and given away—you may do practically ANYTHING in the United States with eBooks not protected by U.S. copyright law. Redistribution is subject to the trademark license, especially commercial redistribution. START: FULL LICENSE
- 74. THE FULL PROJECTGUTENBERG LICENSE
- 75. PLEASE READ THISBEFORE YOU DISTRIBUTE OR USE THIS WORK To protect the Project Gutenberg™ mission of promoting the free distribution of electronic works, by using or distributing this work (or any other work associated in any way with the phrase “Project Gutenberg”), you agree to comply with all the terms of the Full Project Gutenberg™ License available with this file or online at www.gutenberg.org/license. Section 1. General Terms of Use and Redistributing Project Gutenberg™ electronic works 1.A. By reading or using any part of this Project Gutenberg™ electronic work, you indicate that you have read, understand, agree to and accept all the terms of this license and intellectual property (trademark/copyright) agreement. If you do not agree to abide by all the terms of this agreement, you must cease using and return or destroy all copies of Project Gutenberg™ electronic works in your possession. If you paid a fee for obtaining a copy of or access to a Project Gutenberg™ electronic work and you do not agree to be bound by the terms of this agreement, you may obtain a refund from the person or entity to whom you paid the fee as set forth in paragraph 1.E.8. 1.B. “Project Gutenberg” is a registered trademark. It may only be used on or associated in any way with an electronic work by people who agree to be bound by the terms of this agreement. There are a few things that you can do with most Project Gutenberg™ electronic works even without complying with the full terms of this agreement. See paragraph 1.C below. There are a lot of things you can do with Project Gutenberg™ electronic works if you follow the terms of this agreement and help preserve free future access to Project Gutenberg™ electronic works. See paragraph 1.E below.