Python and Tidyverse

Introduction

One of the great things about the R world has been a collection of R packages called tidyverse that are easy for beginners to learn and provide a consistent data manipulation and visualisation space. The value of these tools has been so great that many of them have been ported to Python. That’s why we thought we should provide an introduction to tidyverse for Python blog post.

What is tidyverse?

Tidyverse is an opinionated collection of R packages designed for data science. All packages share an underlying design philosophy, grammar, and data structures. The core R tidyverse packages are: ggplot2, dplyr, tidyr, readr, purrr, tibble, stringr and forcats.

Python implementation of dplyr

The tidyverse package dplyr is a grammar of data manipulation, providing a consistent set of verbs that help you solve the most common data manipulation challenges. Here are some of the functions dplyr provides that are commonly used:

  • mutate() - adds new variables that are functions of existing variables
  • select() - picks variables based on their names.
  • filter() - picks cases based on their values.
  • summarise() - reduces multiple values down to a single summary.
  • arrange() - changes the ordering of the rows.

Dplython is a Python implementation of dplyr which can be installed using pip and the following command:

pip install dplython

Instructions on how to use pip to install python packages can be found here.

The Dplython README provides some clear examples of how the package can be used. Below is an summary of the common functions:

  • select() - used to get specific columns of the data-frame.
  • sift() - used to filter out rows based on the value of a variable in that row.
  • sample_n() and sample_frac() - used to provide a random sample of rows from the data-frame.
  • arrange() - used to sort results.
  • mutate() - used to create new columns based on existing columns.

For more functions and example code visit the Dplython README page.

At the bottom of the README a comparison is provided to pandas-ply which is another python implementation of dplyr.

Dplython comes with a sample data-set called ‘diamonds’. Here are some basic examples of how to use Dplython.

Import Python packages and the ‘diamonds’ data-frame:

import pandas
from dplython import (DplyFrame, X, diamonds, select, sift, sample_n,
    sample_frac, head, arrange, mutate, group_by, summarize, DelayFunction)

Create a new data-frame by selecting columns of the ‘diamonds’ data-frame:

diamondsSmall = diamonds >> select(X.carat, X.cut, X.price, X.color, X.clarity  , X.depth  , X.table)

Display the top 4 rows of the ‘diamondsSmall’ data-frame:

print(diamondsSmall >> head(4)) 

##    carat      cut  price color clarity  depth  table
## 0   0.23    Ideal    326     E     SI2   61.5   55.0
## 1   0.21  Premium    326     E     SI1   59.8   61.0
## 2   0.23     Good    327     E     VS1   56.9   65.0
## 3   0.29  Premium    334     I     VS2   62.4   58.0

Filter the data-frame for rows where the price is higher than 18,000 and the carat less than 1.2 and sort them by depth:

print((diamondsSmall >> sift(X.price > 18000, X.carat < 1.2) >> arrange(X.depth)))

##        carat        cut  price color clarity  depth  table
## 27455   1.14  Very Good  18112     D      IF   59.1   58.0
## 27457   1.07  Very Good  18114     D      IF   60.9   58.0
## 27530   1.07    Premium  18279     D      IF   60.9   58.0
## 27635   1.04  Very Good  18542     D      IF   61.3   56.0
## 27507   1.09  Very Good  18231     D      IF   61.7   58.0

Provide a random sample of 5 rows from the data-frame

print(diamondsSmall >> sample_n(5))

##        carat        cut  price color clarity  depth  table
## 320     0.71       Good   2801     F     VS2   57.8   60.0
## 9813    0.91    Premium   4670     H     VS1   61.8   54.0
## 11795   1.18  Very Good   5088     E     SI2   62.5   60.0
## 11845   0.95  Very Good   5101     D     SI1   63.7   55.0
## 11552   1.17      Ideal   5032     F     SI1   63.0   54.0

Add a column to the data-frame containing the rounded value of ‘carat’

print((diamondsSmall >> mutate(carat_bin=X.carat.round()) >>  sample_n(5)))

##        carat        cut  price color clarity  depth  table  carat_bin
## 11883   0.99  Very Good   5112     F     SI1   62.5   58.0        1.0
## 45123   0.77       Fair   1651     D     SI2   65.1   63.0        1.0
## 51630   0.31    Premium    544     E     SI1   59.2   60.0        0.0
## 49382   0.51  Very Good   2102     G      IF   62.6   56.0        1.0
## 18296   1.54  Very Good   7437     I     SI2   63.3   60.0        2.0

Python implementation of ggplot2

The tidyverse package ggplot2 is a system for declaratively creating graphics, based on The Grammar of Graphics. You provide the data, tell ggplot2 how to map variables to aesthetics, what graphical primitives to use, and it takes care of the details.

A Python port of ggplot2 has long been requested and there are now a few Python implementations of it; Plotnine is the one we will explore here. Plotting with a grammar is powerful, it makes custom (and otherwise complex) plots easy to think about and create, while the plots remain simple.

Plotnine can be installed using pip:

pip install plotnine

Plotnine splits plotting into three distinct parts which are data, aesthetics and layers. The data step adds the data to the graph, the aesthetics (aes) step adds visual attributes and the layers step creates the objects on a plot. Multiple aesthetics and layers functions can be added to a Plotnine graph.

If you are a python user used to Matplotlib it can take some getting used to a Grammar of Graphics plotting tool which is partly due to the difference in philosophy. Plotnine provides some tutorials to help with getting to grips with the package and there is also the Plotnine README. However if you are new to Grammar of Graphics plotting then this highly recommended kaggle notebook for Plotnine is probably the best place to start.

Here are some examples of how to use plotnine to visualize data from the ‘diamonds’ data-frame that comes with Dplython.

Import Python packages, the ‘diamonds’ data-frame and create a sample data-frame:

import warnings; warnings.filterwarnings("ignore") # hide Python warnings 
import pandas
import dplython as dplython
from plotnine import *
diamondsSample = dplython.diamonds >> dplython.sample_n(5000)

Create a scatter plot of ‘carat’ vs ‘price’:

print(ggplot(diamondsSample) # diamondsSample is the data  
 + aes('carat', 'price') # plot 'carat' vs 'price'
 + geom_point() # display the results as a scatter plot
 )

## <ggplot: (41012744)>

Add additional layers e.g. a line of best fit:

print(ggplot(diamondsSample)  
 + aes('carat', 'price') 
 + stat_smooth() # add a line of best fit
 + geom_point()) 

## <ggplot: (-9223372036813567705)>

Add another aesthetic, here the data is coloured by the ‘cut’ variable:

print(ggplot(diamondsSample)
 + aes('carat', 'price')
 + aes(color='cut') # colour the data by the variable cut and create a ledgend 
 + geom_point())

## <ggplot: (-9223372036816020904)>

Add a layer which separates the data into graphs based on ‘colour’

print(ggplot(diamondsSample)
 + aes('carat', 'price')
 + aes(color='cut')
 + facet_wrap('color') # seperate the data by 'colour' and graph seperately  
 + geom_point())

## <ggplot: (64014519)>

This article compares a variety of alternative plotting packages for Python.

Next steps

  • Read the documents that are linked in this blog post.
  • Learn the basics of Pandas.
  • Use Dplython and Plotnine to practice data manipulation & visualization. For example complete some of the exercises at kaggle.

Do you know of other good Python implementations of tidyverse? If so let us know about them!

Search