Locke Data Blog

Gift ideas for the R lovers

Fri, 14 Dec 2018 00:00:00 +0000

Are you looking for gift ideas for the R addicts in your life or the next guest speaker at your user group? Here are a few ideas, in four categories! Not all of them feature our company, promised. ;-)

Learning and using R

Is your dear one just starting to learn R? You might help them by gifting them a book, or even a couple of them! We suggest the R fundamentals series by Locke Data’s Steph Locke, Volume 1, Working with R and Volume 2, Data Manipulation in R, both of them available as paperback and Kindle versions. Other useful books for R users of all levels include R books by O’Reilly.

Got all my #rstats cheat sheets printed and laminated to pass around during R-Lab to help the students this semester. #tidyverse pic.twitter.com/AcxIDfj0i9
— Dylan McDowell (@dylanjm_ds) September 14, 2018

If the R user in your life uses RStudio packages, they might appreciate using RStudio cheatsheets about data manipulation, dates and times, Keras, etc. The cheatsheets are available for free, and you could print and laminate a few of them as a gift!

Becoming a legit package developer

Is the person you want to surprise a package developer? How about increasing their confidence and joy in their package by gifting them a voucher for a hex logo creation by Locke Creatives? Oz Locke, who has designed the hex logos for RDogLadies, visdat and naniar, among others. Reach out to Oz on Twitter if you’d like to know more!

As a complement, you could then offer to print the created hex logos as stickers.

Decorating one’s office

Why not gift R art to hang on the walls? We like

personifyr::sf [be a space wrangler]. #rstats pic.twitter.com/HehHARwM3E
— Allison Horst (@allison_horst) December 8, 2018

Drawings by the very talented Allison Horst, that you could have print and have framed. She does not seem to have a shop yet, but you could contact her!
This nifty magrittr cross-stich found on Etsy!

Getting out of one’s office

Loving R does not mean working at one’s desk all the time.

The R community likes hex shaped things. You could gift a hex shaped baking tin, or other hex shaped objects, with material to personalize them, or you could personalize them yourself. E.g. see these gorgeous coasters!

I recently acquired some cork and a soldering iron, and learned how to make coasters by burning patterns into them. So even though I don't have any actual hex stickers, I now have these! pic.twitter.com/g8ddXvAH3S
— Kim Cressman (@swmpkim) November 21, 2018

Have you heard of the SatRday conferences? They’re community-lead, low-cost one day R conferences. You could buy a ticket to your dear one’s local satRday conference, or even make a trip out of it if the conference is not that local. Check out SatRdays calendar. There’s one planned in Paris on February the 23d!

That’s it! It might be a little bit late for the holidays depending on the ones you celebrate, but occasions to give presents will surely happen again soon! Please share your other gift ideas for R lovers in the comments!

covrpage, more information on unit testing

Mon, 10 Dec 2018 00:00:00 +0000

In this post, we shall explore the first R package that received Locke Data’s new support, covrpage by Jonathan Sidi! With this nifty package you can better communicate the unit testing completeness and goodness of your package!

What’s `covrpage`?

Trust is earned not ~~inherited~~ importedFrom. Now that you’ve built a cool package, you want potential users to trust it so that they might adopt it. So how can you build trust in your software? Unit testing is one of the components building trustworthiness of your package. Imagine you’re at the point where you’ve tested most lines of your code with thorough assertions, including checks of edge cases. Proof of that hard work will be a high test coverage, that potential users of your package might notice thanks to a bright green coverage badge.

Green coverage badge for the rfm package

But how would they know your tests are thorough? That’s what covrpage helps you with, by creating a summary report of your tests that goes beyond the coverage percentage.

Tests summary report for the rfm package

This way, potential users can see at a glance how good the unit testing of your package is. This report can be used as a README for the tests folder, as well as a vignette in the pkgdown website and/or CRAN page of a package.

Is the `covrpage` report only for users?

No, it can also inform your work on your package, by helping you track progress of the unit tests you’re working on, and it can show to potential contributors where help is needed.

How do I publish the `covrpage` report?

There are two places where you can keep the covrpage report, and it’s advised to use both since they will get seen by different readers:

A README for the tests/ folder, which is the original report location. covrpage::covrpage() sets it up. Target audience: users or collaborators browsing the GitHub repo of your package, possibly guided there by a badge in the main README.
A vignette, that’ll get inserted into the pkgdown website of your package, and the CRAN page if/when your package is released on CRAN. covrpage::use_covrpage_vignette() sets it up. Target audience: users reading the rendered documentation.

In both cases, you can also ensure the report stays up-to-date by having it deployed from Travis every time you push to your repository.

Can I get a `covrpage` report for any package?

See a new package on GitHub that you would like to use, but need to feel more comfortable before committing to it? You can run covrpage_snapshot() to create a report in a sterile environment, without affecting your .libPaths, and make a more informed decision wether to install the package.

For more information see the snapshots vignette.

How do I learn more about `covrpage`?

To read more about getting started with covrpage in your own package in a few lines of code only, we recommend checking out the “get started” vignette. It explains more how to setup the Travis deploy, mentions which functions power the covrpage report, and gives more motivation for using covrpage.

And to learn how the information provided by covrpage should be read, read the “How to read the covrpage report” vignette.

What did `covrpage` gain from Locke Data’s help?

covrpage logo

As announced recently, we at Locke Data will help a package a month get more widely adopted. In the case of covrpage, our main input was:

the creation of a nice logo by Oz Locke. The logo highlights that covrpage extends the covr and testthat packages by helping the user inspect their results.
the improvement of documentation, in particular vignettes and the README, to make it clearer why potential users should care about covrpage.
exploration of Travis deploy of the report via the tic package, to make the setup smoother.

Thanks Jonathan for letting us get involved for your package, and good luck with future development! And if you think your broadly applicable package could benefit from our support getting your package into shape then apply now.

Project planning with plotly

Mon, 26 Nov 2018 00:00:00 +0000

Something a little different today for a quick chat about my latest project and why I’m finding the plotly package so helpful!

Are you like me and physically can’t function unless you’ve got a to do list in front of you? Well even if you’re not, imagine my pain while I’m wearing my non - Locke Data hat and trying to plan out the final year of my PhD thesis!

I needed something that updated easily, something visual and something to keep my supervisors in the know. I’ve previously made gantt charts using LaTeX but found it ridiculously clunky to get working and decided there had to be a better way. And if I could include interactivity then all the better, which is how I discovered plotly.

Plotly

The plotly package makes interactive, publication quality graphs online. It uses the JavaScript graphing library and is really versatile! You can make graphs, maps, 3D plots and, as I’m about to explain, gantt charts.

require(plotly)

Step by Step Gantt Chart

Install the packages you need for generating the gantt chart

library(RColorBrewer)
library(readxl)
library(widgetframe)

Setting fonts isn’t necessary as there are defaults, but it’s always nice to know how to do it should you want to.

f <- list(
  family = "Courier New, monospace",
  size = 18,
  color = "#7f7f7f"
)

Set a title for the X axis. Uncomment the Y axis if you’d like to label that too :)

x <- list(
  title = "Date",
  titlefont = f
)
# y <- list(
#   title = "Task Number",
#   titlefont = f
# )

Read in your data

To make this gantt chart work, you need a column with a date you intend to start the task - we’ll make sure this is formatted in a minute. You also need a column with a duration to generate the length of the bar for each task (this is in days in my case!) and if you want to group and colour code your tasks, it’s the Chapter column which does this.

The Task and Progress columns aren’t necessary, but they are good for labelling and keeping yourself accountable, and an easy way to share updates with people who are checking up on your progress!

df <- read_xlsx(path = "../../../../Book_thesis/00_Time_Plan/timeplan.xlsx", sheet = 1) 
head(df)

## # A tibble: 6 x 5
##   Task                       start               Duration Chapter Progress
##   <chr>                      <dttm>                 <dbl> <chr>   <chr>   
## 1 Review literature review … 2018-07-03 00:00:00        2 2       Complet…
## 2 2.1 Energy on a larger sc… 2018-07-05 00:00:00        5 2       Complet…
## 3 2.1 Energy in a UK context 2018-07-12 00:00:00        7 2       Complet…
## 4 2.2 Traditional collectio… 2018-07-23 00:00:00        3 2       Complet…
## 5 2.2 New technologies and … 2018-07-26 00:00:00        3 2       Complet…
## 6 2.3 Data Bias and Represe… 2018-07-31 00:00:00        3 2       Complet…

Make sure your date column is correctly formatted

df$start <- as.Date(df$start, format="%Y-%m-%d")

Next generate a list of colours. This corresponds to my Chapter column, so change this accordingly. It creates a list assigning a colour to each group of tasks

cols <- brewer.pal(length(unique(factor(df$Chapter))), name = "Set3") # Generate a list of colours that are as long as your group of tasks
# This won't work if column has blanks

df$color <- factor(df$Chapter, labels = cols) # Attach these colours as factors to each group of tasks

The next chunk of code generates a date line updated automatically based on your computer’s system date.

# annotation
a <- list(text = "Today's date",
          x = Sys.Date(),
          y = 1.02,
          xref = 'x',
          yref = 'paper',
          xanchor = 'left',
          showarrow = FALSE
)

# use shapes to create a line
l <- list(type = line,
          x0 = Sys.Date(),
          x1 = Sys.Date(),
          y0 = 0,
          y1 = 1,
          xref = 'x',
          yref = 'paper',
          line = list(color = 'black',
                      width = 0.7)
)

Build your plot combining all the elements you generated earlier. Change the text section to personalise your chart

p <- plot_ly()
for(i in 1:(nrow(df) - 1)){
  p <- add_trace(p,
                 x = c(df$start[i], df$start[i] + df$Duration[i]), 
                 y = c(i, i), 
                 mode = "lines",
                 line = list(color = df$color[i], width = 20),
                 showlegend = F,
                 hoverinfo = "text",
                 text = paste("Task: ", df$Task[i], "<br>",
                              "Duration: ", df$Duration[i], "days<br>",
                              "Chapter: ", df$Chapter[i], "<br>",
                              "Status: ", df$Progress[i]), "<br>",
                 evaluate = T
  )
}

Now simply plot your chart and add axis labels, your title and ‘Today’s date’ - The plotly package will generate an interactive chart that you can zoom, hover and share!

p %>%
  layout(xaxis = x, 
         # yaxis = y,
         title = "Thesis Schedule",
         annotations = a,
         shapes = l)

Ta-Dah! Now all I have to do is stick to it…

namer, Automatic Labelling of R Markdown Chunks

Wed, 31 Oct 2018 00:00:00 +0000

We’ve just released a sweet package to save you stress from the hassle of unnamed chunks in R Markdown! namer will name all your chunks, so you can quickly debug in future. More details in this post!

Why name your R Markdown chunks?

When writing R Markdown documents, be it a single report or a whole book based on dozens of documents, it’s crucial to name your R Markdown chunks. Informative names can help when navigating your files and will be used as informative filenames for figures generated in chunks. But even when not informative, chunk labels are very important! Imagine you’re compiling a whole book, and oops, a bug appears in… unnamed-chunk-566. How do you find the culprit?! That name you get in the error message is not written in any file!

Sad chibi Stef being told by R that there is an error in unnamed-chunk-12

Or, since caches are named after chunks, if you delete one chunk in a document full of unnamed chunks, all your caches become useless. The horror!

One good piece of advice would be to always name your chunks. That’s what we say when teaching R Markdown, and what’s promoted in this blog post of mine featuring puppy photographs! But, sadly, we don’t always follow best practice, do we?

What does `namer` do?

Luckily, thanks to a brilliant idea of Steph’s, namer is now here to save your (vegan) bacon! This nifty package will name your chunks for you when you’ve (willingly or unwillingly) forgotten to do so! For any R Markdown document, namer creates and saves chunk labels based on the filename stripped from its extension. Therefore, all chunks of “my-fantastic-report.Rmd” get named “my-fantastic-report-1”, “my-fantastic-report-2”. Voilà!

Happy chibi Stef being told by R that there is an error in the chunk labelled test-12

In practice, namer provides this service for a single document, or a whole folder at once, and an RStudio add-in for naming the chunks of the current active document.

The screenshot below is a real-life example result of running namer::name_dir_chunks(“pres”). In each of the files in the dir “pres”, it labelled chunks using the filename and numbers.

Difference between two R Markdown reports before and after using namer to label chunks

How to use `namer`?

First, install namer from CRAN.

install.packages("namer")

Then, say you want to name the chunks of a report saved under “reports/my-report.md”, type

namer::name_chunks("reports/my-report.md")

Here comes a warning from us… When using namer, please check the edits before pushing them to your code base. Such automatic chunk labelling is best paired with version control!

If you want to label the chunks of all the R Markdown documents of your “reports” folder, use namer::name_dir_chunks():

namer::name_dir_chunks("reports")

Now, as life is sometimes a bit more complicated, you might sometimes need to unname all chunks of a report before re-naming it, if you e.g. used namer once, then added many chunks, or if you don’t like the naming scheme you had been using. We have a function for that too! It will unname all chunks except the setup chunk!

namer::unname_all_chunks("reports/my-report.md")

Last but not least, we have a minimal RStudio addin shipped with the package, to name chunks of any R Markdown file. It allows you to select that file thanks to our very first Hacktoberfest contributor, Ellis Valentiner! Read more about Hacktoberfest at Locke Data here.

Future plans for even more nifty automatic naming

Like all our packages, namer is developed in the open on GitHub so you can go read what we’re planning for the future. In particular, we’re aiming to extend the RStudio addin to make it possible to select any file or folder, not just the active document.

Furthermore, if you’re keen to get involved in open-source development, we’d be glad to mentor you, check out the issues that’d welcome external help! As a side-note, if you’re into programmatic change or assessment of chunk options, you might be interested in the tinkr package, that provides a function to transform R Markdown files into XML documents, and another one to save them back as R Markdown.

In the meantime, go write awesome R Markdown documents, now that the fear of the unnamed chunks no longer dawns on you!

Packages for Testing your R Package

Mon, 22 Oct 2018 00:00:00 +0000

Testing your R package is crucial, and thankfully it only gets easier with time, thanks to experience… and awesome packages helping you setup and improve tests! In this post, we shall offer a roundup of packages for testing R packages, first in a section about general testing setup, and then in a section about testing “peculiar” stuff.

General package testing infrastructure

Create tests

If you’re brand-new to unit testing your R package, I’d recommend reading this chapter from Hadley Wickham’s book about R packages.

There’s an R package called RUnit for unit testing, but in the whole post we’ll mention resources around the testthat package since it’s the one we use in our packages, and arguably the most popular one. testthat is great! Don’t hesitate to reads its docs again if you started using it a while ago, since the latest major release added the setup() and teardown() functions to run code before and after all tests, very handy.

To setup testing in an existing package i.e. creating the test folder and adding testthat as a dependency, run usethis::use_testthat(). In our WIP pRojects package, we set up the tests directory for you so you don’t forget. Then, in any case, add new tests for a function using usethis::use_test().

The testthis package might help make your testing workflow even smoother. In particular, test_this() “reloads the package and runs tests associated with the currently open R script file.”, and there’s also a function for opening the test file associated with the current R script. Edit: as of version 2.0.0 devtools itself features functions for testing single files.

Assess your tests

To get a sense of how good your tests are, check these out:

covr computes the test coverage i.e. the percentage of lines of code that are covered by tests. covr allows skipping some lines. If run on Travis or Appveyor, for instance, it can send a report to online coverage tools such as CodeCov or Coveralls, allowing you to visualize the coverage. At Locke Data we mostly don’t run covr locally but instead have it run on Travis. To set that up, usethis::use_coverage(). Here is Coveralls report for HIBPwned and the corresponding badge. See a CodeCov report for comparison.
covrpage creates a detailed coverage report that can serve as a README for your test folder. We’ve done that for HIBPwned so now without clicking in a coverage report, thanks to “detailed test results”, you can see the tests associated with each context.

Test all the things

Now, sometimes you might encounter cases of things that you don’t quite know how to test. Here’s a small list, but please comment about anything we’ve forgotten!

Mocking

Sometimes you need to test whether your package works as expected “if something happens”, “if a thing has this value” and can’t rely on arguments. E.g. what happens if the environment variable GITHUB_PAT doesn’t exist, or if a dependency isn’t installed? In such cases, what you might be after is mocking. The testthat package itself has a with_mock() function, but it’s now recommended to rather use the mockery or mockr packages.

Webmocking

If the mocking you need to perform is e.g. mimicking a 404 result from an API, or saving a web API response and replay it to not have to re-query the API at each test, you can use:

webmockr like we did for HIBPwned
vcr and webmockr together, see vcr docs. It works for both crul and httr. vcr docs include a list of packages using vcr for testing in the wild.
httptest for httr only.

Test plot outputs

You can test your plot outputs haven’t changed by using vdiffr. To set things up you need to run vdiffr::manage_cases().

Test Shiny apps

Fear not, there’s a whole package dedicated to help you test Shiny apps! Check out shinytest.

Test RStudio add-ins

The remedy package by ThinkR has tests for its add-ins. See in particular the scratch_file() function. The tests need to be run only when RStudio is available so a helper defines a skip_if_not_rstudio() function. Thanks to Colin Fay and Jonathan Sidi for showing it to me.

Test htmlwidgets?

I’ll be honest, I haven’t seen examples of this in the wild, which is not surprising given I don’t use htmlwidgets a lot. Still, worth mentioning are:

viztest tests htmlwidgets based on screenshots.
the idea to test the content of the html before or after interactions. rdom can be a part of such a workflow: rdom + xml2 to scrape the result + testthat of course. Thanks to David Gohel for telling me this!

Test interactive behavior?

This is another topic I haven’t totally figured out, but I like using usethis tests as a reference, be it the manual/ folder or the testthat/ folder. testthis tests might also be an inspiration.

Conclusion

The ecosystem for R package testing is getting more and more complete and automatic, which is very exciting. To deploy your tests, you’ll need to learn about continuous integration, which thankfully is also an area with exciting developments. Maybe a subject for another post… In the meantime, feel free to tell us about your favourite resources for testing packages!

Package support offer

Mon, 15 Oct 2018 00:00:00 +0000

The R community and the package ecosystem are awesome but it can be difficult to sustain your R packages when you only have so much free time. To make a stellar package you’ve got to keep on top of the issues, make great documentation, have that all important hex sticker, and generally have good quality code. This all takes time, time you don’t always have. We would like to help.

Once per month, we’ll select (from applicants) one package to setup for success. These packages will have general utility to a broad part of the community but could benefit from some decidated time to help give it the boost it needs to be more widely adopted.

For the selected packages we will then work with the package author to ensure:

The GitHub repo is setup well and go over all existing issues to make sure they’re well formed, tagged, and prioritised. (3 hours of Locke Data time)
The package implements best practices in terms of the code, tests, and documentation. (5 hours of Locke Data time)
The package is easy to market with a pkgdown site and a logo. (Up to a day of Locke Creatives time)
The package starts getting coverage by showcasing the package in our blog. (Up to 3 hours of Locke Data time)

In return you’ll list us as sponsors in the DESCRIPTION using the fnd tag and list our support on the README (and therefore the pkgdown). We’ll also tell folks we helped you.

If you think your package fits the bill (i.e. broadly applicable) and you could benefit from our support getting your package into shape then apply now.

PS We can also offer this service commercially, and provide CRAN submission and ongoing maintainence options too if you’re a company who just wants to get stuff live.

cransays - Follow your R Package Journey to CRANterbury with our Dashboard!

Thu, 11 Oct 2018 00:00:00 +0000

We at Locke Data maintain a few R packages that we’ve submitted to CRAN to help increase their userbase. After running devtools::release(), clicking in a confirmation email… what remains is waiting. Inspired by our experience, we’ve created a dashboard to help other package maintainers follow their package’s journey to CRANterbury. Read more about its making in this post!

Why create the cransays dashboard?

Sometimes, depending on the workload of CRAN volunteers, it can last a while before a package ends up on their way to CRAN. The number of submissions has increased a lot within the last few years! As per CRAN policies, “You can check that the submission was received by looking at ftp://CRAN.R-project.org/incoming/.". The content of that ftp server is divided into various subfolders, making the research of one’s package pretty tricky.

Part of the content of CRAN incoming ftp server

After having to do that a few days in a row for checking on one of our submissions, we decided a dashboard showing all packages and their submission stage in one go would be quite useful, so we made it happen!

How we made the cransays dashboard?

In this section you’ll discover how we create a snapshot of the CRAN incoming state, format it, and then deploy it to the interwebs every hour!

Creating a snapshot of CRAN incoming

First of all, we wrote a function take_snapshot() to create a snapshot of the ftp server. Technically, we query the different ftp folders using curl and utils::read.delim(), munging strings and datetimes along the way. Our original code was adapted from an insightful repo by Edgar Ruiz at RStudio, that is a good read about the different stages of submission.

The take_snapshot() function was improved thanks to the contributions of Mitchell O’Hara-Wild and Hugo Gruson. Receiving PRs so rapidly after opening hacktoberfest issues was really appreciated!

Presenting the snapshot

cransays dashboard

Then, the dashboard itself is the data.frame returned by take_snapshot() formatted by the DT package. The formatting of that datatable was improved thanks to external contributions again! Thank you, Hugo Gruson and Jim Hester!

Packaging it all together

Find the source for our dashboard here. cransays is structured as a package, because

Packages are cool, duh!
It helps with managing dependencies thanks to DESCRIPTION.
It allowed us to rip off the benefits of pkgdown and travis + tic. We wrote the DT code inside a vignette – usethis::use_vignette() for the win. Then, with the two lines below, we were able to deploy a website for the package from Travis! Every push to the master branch of our repo meant pkgdown::build_site() is run on Travis and the result is pushed to a gh-pages branch!

# install.packages("usethis")
# remotes::install_github("ropenscilabs/tic")

usethis::use_pkgdown() # creates the pkgdown config file
travis::use_tic() # creates boilerplate deploy code, lets you browse GitHub and Travis to create tokens.

We added a few lines to and _pkgdown.yml and travis.yml to be able to use our own pkgdown style. Steph wrote posts about deploying docs from Travis in case you want to know more about the process but when using rOpenSci’s packages tic and travis, you don’t even need to think about it much.

Thanks to this pkgdown-Travis setup, you can find our dashboard here with a homepage with more information.

Updating it regularly!

Steph took care of the deploy via Netlify to a subdomain of itsalocke.com, otherwise we could have used the GitHub pages hosting from the gh-pages branch. Steph also created a Zapier zap so that there’d be a Travis build triggered every hour. It means that our dashboard is updated at least every hour, which seemed lagom.

Sit back and enjoy, or improve the project?

Now, we’re able to follow our submissions using our nifty dashboard, and you can do that too! We’ve also been thinking about:

getting metadata about each submission by unpacking them and using the desc package and pkgload::parse_ns_file(),
and saving the snapshots over time, very similar to what the RStudio folks did in the spring.

Stay tuned via cransays’ issue tracker! In the meantime, happy R package development, and good luck with your CRAN submissions!

Processing complicated package outputs

Tue, 09 Oct 2018 00:00:00 +0000

Sometimes packages have functions that don’t do the things the way you want them to do them and you have to either re-build the function, or work with it as-is and add code around it to solve your issue.

I’ve had to do this recently with the googleway package and it’s google_distance() function so I wanted to take you through step by step how I wrote code to go from a single value function to a function that handles many inputs and returns 4 rows per input. I won’t be dwelling on how to write a function specifically, just showing you the workflow I often go through.

Requirements

Key functionality we’ll need today is:

googleway for providing the base function
the tidyverse, namely purrr and dplyr, for lots of the data manipulation
memoise for caching requests so we spend less cash

library(tidyverse)
library(memoise)
library(googleway)

Google Distance

To calculate distances we can use the google distance API.

This needs a key in order to use it. Note that this service does not have a free tier to use, however it is ~$5 per 1,000 requests and a trial of Google Cloud is available.

key="AIzaSyBIeuWMWWweyv1SAoAxcY1IZ-2nuErFQY8"

Then we need to prep our desired information.

office = "E14 5EU"
monday_9am = as.POSIXct("2018-12-03 09:00")

Handling `google_distance()`

The API is used to working with just a single address at a time so we need to do a bit of prep here to make it work with lots of accounts.

For starters, we can use the memoise package to cache results so if we send the same address multiple times it doesn’t need to go back to the API. Phew, since that API costs money to call!

google_distance_deduped = memoise::memoise(google_distance)

Giving this a go with a single example, let’s see what google gives us:

from = "SE3 8UQ"

example_1 = google_distance_deduped(from,
                        office,
                        mode = "transit",
                        arrival_time = monday_9am,
                        key=key)

example_1

## $destination_addresses
## [1] "Canary Wharf, London E14 5EU, UK"
## 
## $origin_addresses
## [1] "Shooters Hill Rd, London SE3 8UQ, UK"
## 
## $rows
##                          elements
## 1 5.7 km, 5653, 37 mins, 2221, OK
## 
## $status
## [1] "OK"

The possibly() function will mean that if there’s an error for a call that it doesn’t break everything and we won’t have to start all over again.

google_distance_try = purrr::possibly(google_distance_deduped, "Fail")

from = c(NA)

example_2 = google_distance_try(from,
                        office,
                        mode = "transit",
                        arrival_time = monday_9am,
                        key=key)

example_2

## [1] "Fail"

Then to make the function work over multiple addresses, we need to change it slightly. The map() function will iterate over all the addresses.

google_distance_loop = function(x,...){
    purrr::map(x, google_distance_try,...)
}

from = rep("SE3 8UQ",2)

example_3 = google_distance_loop(from,
                        office,
                        mode = "transit",
                        arrival_time = monday_9am,
                        key=key)

example_3

## [[1]]
## [[1]]$destination_addresses
## [1] "Canary Wharf, London E14 5EU, UK"
## 
## [[1]]$origin_addresses
## [1] "Shooters Hill Rd, London SE3 8UQ, UK"
## 
## [[1]]$rows
##                          elements
## 1 5.7 km, 5653, 37 mins, 2221, OK
## 
## [[1]]$status
## [1] "OK"
## 
## 
## [[2]]
## [[2]]$destination_addresses
## [1] "Canary Wharf, London E14 5EU, UK"
## 
## [[2]]$origin_addresses
## [1] "Shooters Hill Rd, London SE3 8UQ, UK"
## 
## [[2]]$rows
##                          elements
## 1 5.7 km, 5653, 37 mins, 2221, OK
## 
## [[2]]$status
## [1] "OK"

So our code is working over multiple cases and handling bad inputs pretty well, but how do we get some meaningful stuff out of it. Looking at the data, we get back a part of a table that contains a response.

example_1 %>% 
  map("rows") %>% 
  map("elements")

## $destination_addresses
## NULL
## 
## $origin_addresses
## NULL
## 
## $rows
## NULL
## 
## $status
## NULL

We see that there seems to be no way someone can use public transport between the two locations. Perhaps another way of getting there will return a result?

from = "SE3 8UQ"
example_4 = google_distance_loop(from,
                        office,
                        mode = "walking",
                        arrival_time = monday_9am,
                        key=key)

example_4 %>% 
  map("rows") %>% 
  map("elements") %>% 
  flatten() %>% 
  map("duration")

## [[1]]
##             text value
## 1 1 hour 20 mins  4777

When a commute is possible, we get a response back that includes the number of seconds it might take someone to travel to work for 9am on a Monday.

First of all, we’ll need to reliably extract this information from a batch of repsonses. This takes multiple steps due to the way the API gives us info.

from = c("SE3 8UQ", NA, "SE3 8UQ")

google_distance_tbl = function(x, ...) {
  google_distance_loop(x,...) %>% 
  map("rows") %>% 
  map("elements") %>% 
  flatten() %>% 
  map(unclass) %>% 
  map_df(flatten) %>% 
  cbind(x)
}
  
example_5 = google_distance_tbl(from,
                        office,
                        mode = "walking",
                        arrival_time = monday_9am,
                        key=key)
example_5

##             text value    status       x
## 1 1 hour 20 mins  4777        OK SE3 8UQ
## 2           <NA>    NA NOT_FOUND    <NA>
## 3 1 hour 20 mins  4777        OK SE3 8UQ

So now we’re going to need to ask about the different transit options for each address to find out the range of values in order to cope with “ZERO_RETURN” records. Once we have this information, we can then use the google_distance_all function to find out how long it’ll take someone to drive, walk, cycle, or use public transport to travel between two points.

from = c("SE3 8UQ", NA, "SE3 8UQ")

google_distance_all = function(x, office, arrival_time, key, ...) {
  
  interested_in = expand.grid(from=x, 
     mode=c("driving", "walking", "bicycling", "transit"), 
      stringsAsFactors = FALSE)

map2_df(interested_in$from,interested_in$mode, 
     ~mutate(
       google_distance_tbl(.x, office, mode=.y,
                        arrival_time = arrival_time,
                        key=key),
       from=.x, mode=.y)
)
}

example_6 = google_distance_all(from, office, monday_9am, key)

example_6

##              text value    status       x    from      mode
## 1         12 mins   739        OK SE3 8UQ SE3 8UQ   driving
## 2            <NA>    NA NOT_FOUND    <NA>    <NA>   driving
## 3         12 mins   739        OK SE3 8UQ SE3 8UQ   driving
## 4  1 hour 20 mins  4777        OK SE3 8UQ SE3 8UQ   walking
## 5            <NA>    NA NOT_FOUND    <NA>    <NA>   walking
## 6  1 hour 20 mins  4777        OK SE3 8UQ SE3 8UQ   walking
## 7         32 mins  1930        OK SE3 8UQ SE3 8UQ bicycling
## 8            <NA>    NA NOT_FOUND    <NA>    <NA> bicycling
## 9         32 mins  1930        OK SE3 8UQ SE3 8UQ bicycling
## 10        37 mins  2221        OK SE3 8UQ SE3 8UQ   transit
## 11           <NA>    NA NOT_FOUND    <NA>    <NA>   transit
## 12        37 mins  2221        OK SE3 8UQ SE3 8UQ   transit

Having this many functions though clutters things up and makes it difficult to refactor and improve things. We should unpack all the functionality into one big function.

#' Get distance data between two points based on all the travel mode options. Works for many origin points.
#'
#' @param x A vector of origins in address or postcode format
#' @param dest A single destinationin address or postocde format
#' @param arrival_time A POSIXct datetime that folks need to arrive by
#' @param key A google distance API key
#' @param ... Additional options to pass to `google_distance()`
#'
#' @return Data.frame containing (typically) 4 rows per input element

google_distance_all =  function(x, dest, arrival_time, key, ...){
  
  # simple hygeine stuff
  gd = purrr::possibly(
    memoise::memoise(
      google_distance)
    , "Fail"
  )
  
  # Prep dataset
   interested_in = expand.grid(from=x, 
     mode=c("driving", "walking", "bicycling", "transit"), 
      stringsAsFactors = FALSE)
   # Perform google_distance calls for all combos
  purrr::map2(interested_in$from,interested_in$mode, 
     ~gd(.x, dest, mode=.y,
                        arrival_time = arrival_time,
                        key=key)
  ) %>% 
    # Extract relevant section
    purrr::map("rows") %>% 
    purrr::map("elements") %>% 
    purrr::flatten() %>% 
    # Simplify the data.frames
    purrr::map(unclass) %>% 
    purrr::map_df(purrr::flatten) %>% 
    # Add original lookup values
    cbind(interested_in)
}

results = google_distance_all(
  from,
  office,
  arrival_time = monday_9am,
  key = key
)

results

##              text value    status    from      mode
## 1         12 mins   739        OK SE3 8UQ   driving
## 2            <NA>    NA NOT_FOUND    <NA>   driving
## 3         12 mins   739        OK SE3 8UQ   driving
## 4  1 hour 20 mins  4777        OK SE3 8UQ   walking
## 5            <NA>    NA NOT_FOUND    <NA>   walking
## 6  1 hour 20 mins  4777        OK SE3 8UQ   walking
## 7         32 mins  1930        OK SE3 8UQ bicycling
## 8            <NA>    NA NOT_FOUND    <NA> bicycling
## 9         32 mins  1930        OK SE3 8UQ bicycling
## 10        37 mins  2221        OK SE3 8UQ   transit
## 11           <NA>    NA NOT_FOUND    <NA>   transit
## 12        37 mins  2221        OK SE3 8UQ   transit

I will undoubtedly want to do some cleaning after this and there’s certainly room for improvement on the function but this is a good starting point for getting some data to work with. The iterative way I build functions means I can try to solve a bit at a time – hopefully this will help you when you’re faced with needing to build your own functions.

Tidyverse 'Starts_with' in M/Power Query

Mon, 08 Oct 2018 15:13:39 +0100

As a heavy R and Tidyverse user, I’ve been playing with Microsofts m/Power Query language included in Excel and PowerBI from that perspective, looking for the functions to make my life easier, developing small code pipelines for my processing and trying to get a smooth, clear and maintainable data manipulation process in place.

The Problem

In PowerBI I have data generated from an API call to HubSpot, which deliveres a json which is flattened as the first step of the process into a table with hundreds of columns. These columns have a pretty regular naming convention, in a form similar to this:

client_notified_timestamp
client_notified_source
client_notified_sourceid
client_notified_value
client_responded_timestamp
client_responded_source
client_responded_sourceid
client_responded_value

The general rule is that the variable is encoded in the first part of the column name string, and that the columns with [variable]_value hold the actual value while the other three columns ([variable]_source, [variable]_sourceid and [variable]_timestamp) contain metadata we don’t really need here.

The Target

If I was using R to do this job (which technically I could, but was not possible because of the context the PowerBI file is going to be used in), I could use tidyverse to do this pretty simply:

dataset %>%
    select(c(-ends_with("_source"),-ends_with("_sourceid")))

Anything that ends with "_source" or "_sourceid" gets dropped, everything else remains. A nice compact, maintainable and clear expression of a ‘rule’ of processing.

The Solution

This is the solution I used:

let
    Source = ...,
    rawData = Source{[tableId="myData"]}[Data],
    removeSources = Table.RemoveColumns(rawData, List.Select(Table.ColumnNames(rawData), each Text.EndsWith(_, "Source ID") or Text.EndsWith(_, "Source")))
in
    removeSources

This code block sources rawData and ‘lists’ the columns matching my requirements ("_source" and "_sourceid") using the logical condition each Text.EndsWith(_, "Source ID") or Text.EndsWith(_, "Source") on the column names returned from Table.ColumnNames(rawData) feeding into List.Select(...). This list is the second argument to the function Table.RemoveColumns(...), which is operating on the rawData again, to finally return only the columns I want.

The Observations

This generally suits the requirements: relatively readable functions, multiple logical conditions operating on the column names that ‘select’ which I want returned in the next step.

It is admittedly a little more verbose than the R I had in mind, and right now I’m not sure if that’s me or just the language. There is some repetition in specifying rawData in multiple places, which I haven’t found a shorthand for if there is one. Parts of it seem only ‘functional-ish’? The construction of each Text.EndsWith(_, "Source ID") or Text.EndsWith(_, "Source")) is pretty object-oriented. Without wanting to sound insulting maybe m is only ‘semi-functional’ in the technical definition of the term?

The Caveat

This is the first m code I’ve really written and my knee-jerk first impressions. I’m sure there is a lot more to this language that I have yet to understand and maybe even come to appreciate.

The Conclusion

Despite these observations I wouldn’t discount the potential of m/Power Query. While many Microsoft tools let you use R baked in, it’s only baked in to the point where you can guarantee R is installed on the machine, and it’s an undeniable fact of data that we have to work with Excel and Power BI in many situations. I’m actually quite looking forward to working with this not-quite-familar ‘functional-ish data language’ in the future. When it’s the tool for the job at least :)

Speed Up With Microsoft

Thu, 04 Oct 2018 13:19:59 +0100

People use R for lots of reasons: “It’s great for the models I need”, “I like the functional approach”, “It’s the tool I’m most comfortable with”.

People don’t use R for these reasons: “I have a favourite processor core, I don’t want to use the others”, “I love how my memory needs to fit all my data”.

What if I told you that you didn’t need to worry about that any more?

Multi-threaded R

Microsoft have their own version of R called Microsoft R Client. It has a bunch of high-tech, whiz-bang features, but we’re going to focus on one: multi-threading calculations. Vanilla R is single threaded.This means any calculations are done sequentially, in order, one at a time. However, this isn’t using the ability of most modern, domestic laptops. To get setup with Microsoft R Client, follow the install instructions. Once you’ve got Microsoft R Client installed, you will need to make sure it’s the version of R that is active in your session. If you are using RStudio this is easy to do by going to Tools > Global Options > General > R Version.

You will probably be asked to Restart R Studio, so close it and open back up, then we can run the code.

R version 3.4.3 (2017-11-30) -- "Kite-Eating Tree"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-w64-mingw32/x64 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

Microsoft R Open 3.4.3
The enhanced R distribution from Microsoft
Microsoft packages Copyright (C) 2018 Microsoft

Loading Microsoft R Client packages, version 3.4.3.0097. 
Microsoft R Client limits some functions to available memory.
See: https://go.microsoft.com/fwlink/?linkid=799476 for information
about additional features.

Type 'readme()' for release notes, privacy() for privacy policy, or
'RevoLicense()' for licensing information.

Using the Intel MKL for parallel mathematical computing(using 2 cores).
Default CRAN mirror snapshot taken on 2018-01-01.
See: https://mran.microsoft.com/.

This message will pop up, and it’s worth noting as it’s got some information in it that you might need to think about:

It’s worth noting that right now Microsoft r Client is lagging behind the current R version, and is based on version 3.4 of R, not 3.5. This will mean your default package libraries will not be shared between the installations if you are running R 3.5.
It’s using a snapshot of CRAN called MRAN to source packages by default. 90% of the time it will operate just as you expect, but because it takes a ‘snapshot’ of packages, newer features and changes that have hit CRAN may not be in the version of the package you are grabbing.
- RevoScaleR and probably the ggplot2 and dplyr packages will likely be installed for you already as default in Microsoft R Client. The other two you will probably have to install yourself.
Intel MKL will have scanned your system on install and attempted to work out how many cores your processor has. Here it’s identified 2 on my old Lenovo Yoga. This is where the speed boost will come from.

knitr::opts_chunk$set(echo = TRUE)
library(microbenchmark)
library(RevoScaleR)
library(ggplot2)
library(lockeutils)
theme_set(theme_ld() + theme(axis.title.x = element_text(vjust = -1)))
library(dplyr)

Test Data

Here we make a set of example data, 3 data frame of random numbers with various ‘normal’ distributions. The data frames are of different lengths: 500,000 , 1,000,000 and 5,000,000.

set.seed(9000)
sample_df = function(n){
  data.frame(
    col_1 = c(rnorm(n, mean = 11, sd = 0.5)), 
    col_2 = c(rnorm(n, mean = 6, sd = 1)), 
    col_3 = c(rnorm(n, mean = 3, sd = 0.75))
  )
}

df_500k <- sample_df(500000)
df_1m <- sample_df(1000000)
df_5m <- sample_df(5000000)

Benchmarking

Running benchmarks in R is easy with the microbenchmark package. The package tries to be as accurate as possible in measuring the time for each of it’s runs, and also allows you to easily compare different approaches and specify the amount of repeats. I’ve decided to test each of the 3 data sets in each of 2 different linear modelling functions, giving 6 different groups of results. Each group will be run 10 times for a total set of 60 runs.

res <- microbenchmark(
  lm_50k = lm(col_1 ~ col_2 + col_3, data = df_500k),
  rxLM_50k = rxLinMod(col_1 ~ col_2 + col_3, data = df_500k, reportProgress = 0),
  lm_1m = lm(col_1 ~ col_2 + col_3, data = df_1m),
  rxLM_1m = rxLinMod(col_1 ~ col_2 + col_3, data = df_1m, reportProgress = 0),
  lm_5m = lm(col_1 ~ col_2 + col_3, data = df_5m),
  rxLM_5m = rxLinMod(col_1 ~ col_2 + col_3, data = df_5m, reportProgress = 0),
  times = 10
  )

Results

The res object is of class microbenchmark, and has a plotting method that can be used via ggplot2::autoplot().

autoplot(res) +  
  labs(title = "Violin plot of model run durations", 
       subtitle = "`rxLinMod()` vs `lm()`",
       caption = "Microsoft R Client 3.4.3, 2 cores")

In each of these tests, we can see that the RevoScaleR::rxLinMod() functions outperforms the base lm() by a large margin. Note the log scale for Time [milliseconds]!

knitr::kable(summary(res))

expr	min	lq	mean	median	uq	max	neval
lm_50k	1106.2070	1141.3168	1740.0966	1340.5601	1474.8239	4679.1277	10
rxLM_50k	115.3501	120.7544	137.2811	136.9788	147.7357	171.9592	10
lm_1m	2203.7005	2890.8738	3018.6052	3072.3086	3340.6342	3934.8914	10
rxLM_1m	184.8220	199.1018	223.0184	214.2320	223.2292	307.1239	10
lm_5m	14897.7473	15450.7170	16295.3558	16183.4236	16720.8177	18837.9140	10
rxLM_5m	704.0234	805.8069	1108.2574	965.9696	1044.1400	2084.2737	10

But how?

It does this because the Microsoft R Client uses Intel MKL for parallel mathematical computing. This allows the RevoScaleR package to implement a ‘parallelised algorithm’ to solve the linear regression using the BLAS and LAPACK FORTRAN libraries. More technical details are avilable in the docs.

What about the out of memory part?

RevoScaleR and Microsoft R Client can help with that, though the solution requires a little more involvement. The short version is that it can leverage distributed data sources as a backend, such as spark, hadoop and sqlserver. this allows the data to be spread across many compute nodes that are managed by software called ‘Microsoft Machine Learning Server`. This is a good overview of the ideas on the main site, which also holds all the documentation.

Would you like to know more?

Locke Data have are developing course going into depth with this technology from an R context and are releasing it at the end of October. Please come and join us (virtually!) for some hands-on learning and detailed tutorials.

Up your open source game with Hacktoberfest at Locke Data!

Mon, 01 Oct 2018 00:00:00 +0000

How awesome is open source software? Quite awesome in our opinion! Locke Data maintains several open source repos on GitHub, in particular of R packages, and we’d like you to join in the fun! This month, we’re taking part in Hacktoberfest and will do our best to mentor you through your first open source contributions if you wish!

Hacktoberfest is a month-long operation celebrating open source software. As an open source newbie, it’s the occasion to start participating in open source development! All you need to do is to sign up at the Hacktoberfest website, and then to look for “hacktoberfest”-labelled issues on GitHub to see where your help is needed. More general details in GitHub blog post.

We at Locke Data have labelled a few issues (follow the link) in our package repos and would be glad to have you as a contributor. All our repos are nice places with a code of conduct!

If you choose to contribute to one of our repositories, we’ll offer pointers as needed. Ellen is the team member in charge of managing our Hacktoberfest efforts, but we’ll all cheer! Have fun! If you need to reach out to us for extra help, pointers or just want to know more, feel free to reach out to us on twitter @LockeData and give us a follow while you’re over there - we’ll be talking about Hacktoberfest for the whole month!

Functions and Packages

Sat, 29 Sep 2018 00:00:00 +0000

We’re done with the basics of handling data in R. Now we want to know how to make sense of it. We know what kind of data it is, we know how to look at column names, dimensions and the like. If you’re trying to add value to this data however, that very often isn’t enough, so here’s a look at using the tools available to you to start figuring out how to do what you want.

R packages

An R package is a bundle of functions and/or datasets. It extends the capabilities that the “base” and “recommended” R packages have. By using packages we can do data manipulation in a variety of ways, produce all sorts of awesome charts, generate books like this, use other languages like Python and JavaScript, and of course, do all sorts of data analysis.

Installing packages

Once you’ve identified a package that contains functions or data you’re interested in using, we need to get the package onto our machine.

To get the package, you can use an R function or you can use the Install button on the Packages tab.

install.packages("datasauRus")

If you need to install a number of packages, install.packages() takes a vector of package names.

install.packages(c("datasauRus","tidyverse"))

Updating packages involves re-running install.packages() and it’s usually easier to trigger this by using the Update button on the Packages tab and selecting all the packages you want to update.

Installing from GitHub and other sources

The install.packages() function works with CRAN, CRAN mirrors, and CRAN-like repositories

If you want to install BioConductor packages, there are some helper scripts available from the BioConductor website, bioconductor.org.

Other package sources, such as GitHub, will involve building packages before they can be installed. If you’re on Windows, this means you need an additional piece of software called Rtools. The other handy thing you’ll need is the package devtools (available from CRAN). devtools provides a number of functions designed to make it easier to install from GitHub, BitBucket, and other sources.

library(devtools)
install_github("lockedata/pRojects")

Recommended packages

Here are my recommended packages – look out for books and blogposts on these in the future!

tidyverse

The tidyverse is a suite of packages designed to make your life easier. It’s well worth installing and many of the packages in this recommendations section are part of the tidyverse.

install.packages("tidyverse")

Getting data in and out of R

The following packages can be used to get data into, and out of R:

Working with databases, you can use the DBI package and it’s companion odbc to connect to most databases
To get data from web pages, you can use rvest
To work with APIs, you use httr
To work with CSVs, you can use readr or data.table.[6]
To work with SPSS, SAS, and Stata files, use readr and haven

Data manipulation

The tidyverse contains great packages for data manipulation including dplyr and purrr.

Additionally, a favourite data manipulation package of mine is data.table. data.table tends to have a bit of a steeper learning curve than the tidyverse but it’s phenomenal for brevity and performance.

Data visualisation

For static graphics ggplot2 is fantastic - it adds a sensible vocabulary to help you construct charts with ease
plotly helps you build interactive charts from scratch or make ggplot2 charts interactive
leaflet is a great maps package
ggraph helps you build effective network diagrams

Data science

caret is an interface package to many model algorithms and has a raft of insanely useful features itself
broom takes outputs from model functions and makes them into nice data.frames
modelr helps build samples and supplement result sets
reticulate is a package for talking to Python and, therefore, enables you to work with any deep learning framework that is based in Python. tensorflow is a package based on reticulate and allows you to work with tensorflow in R
sparklyr allows you to run and work with Spark processes on your R data
h2o is a package for working with H2O, a super nifty machine learning platform

Presenting results

rmarkdown is the core package for combining text and code and being able to produce outputs like HTML pages, PDFs, and Word documents
bookdown facilitates books like this
revealjs allows you to make slide decks using rmarkdown
flexdashboard and shiny allow you to make interactive, reactive dashboards and other analytical apps

Finding packages

As well as using online search facilities like CRAN and rdrr.io for packages, there are some handy packages that help you find other packages!

ctv allows you to get all the packages in a given CRAN task view, which are maintained lists of package for various tasks
sos allows you to search for packages and functions that match a keyword

Loading packages

To make functions and data from a package available to use, we need to run the library() function.

library("utils")

The library() function accepts a vector of length 1, so you need to perform multiple calls to the function to load up multiple packages.

library("utils")
library("stats")

Once a package is loaded, you can then use any of it’s functions.

You can find what functions are available in a package by looking at it’s help page.

Alternatively, you can type the package’s name and hit Tab. This auto-completes the package’s name, adds two colons (::) and then shows the list of available functions for that package. The double colon trick is very helpful for when you want to browse package functionality, e.g. utils::find().

Any function in R can be prefixed with it’s package name and the double colon (::) - this is great for telling people where functions are coming from and for tracking dependencies in long scripts. It is also really useful when you have two packages loaded that might have a function of the same name. This is because the order the packages are loaded in dictates which one gets overridden.

Learning how to use a package

R documentation is some of the best out there.

Yes, I will complain about the impenetrable statistical jargon some package authors use, but the CRAN gatekeepers require that packages generally have a really high standard of documentation.

Every function you use will have a help page associated with it. This page usually contains a description, shows what parameters the function has, what those parameters are, and most importantly, there’s usually examples.

To navigate to the help page of an individual function in an R package you:

Hit F1 on a function name in a script
Type ??fnName and send to the console

??mean

Search in the Help tab
Use the help() function to open up the packages index page and navigate to the relevant function

help(package="utils")

Find the relevant package in the Packages tab and click on it. Scroll through the index that opens up on the Help page to find the right function

As well as the function level documentation, good packages also provide a higher level of documentation that covers workflows using the packages, how to extend package functionality, or outlines any methodologies or research that led to the package.

These pieces of documentation are called vignettes. They are accessible on the package’s index page or you can use the function vignette() to read them.

vignette("multi")

Using functions in packages

In previous sections we’ve seen R functions that are used on objects to perform some activity. Functions seen so far include:

class() and is.*() functions for checking datatypes
as.* for converting to datatypes
length() and names() for metadata
head() and tail() for getting a small amount of elements from an object
ncol(), nrow(), colnames(), and rownames() for getting data.frame metadata
Sys.Date() and Sys.time() for getting current date-time values

There are a huge range of functions out there, whether available in R straight away, or from adding extra functionality.

Understanding how functions work and being able to use them correctly will help you learn, and use R effectively.

Using a function

A function does some computation on an object. The use of a function consists of:

A function’s name
Parentheses
0 or more inputs

Each input is provided to an argument or parameter within a function.

These arguments have names, although you don’t often need to provide the names.

You can find out what arguments a function takes by using the code completion and it’s help snippet, or by searching for the function in the Rstudio Help tab.

When you’re inside the brackets of a function you can get the list of available arguments and auto-complete them.

Examining functions

One of the niftiest things about R is being able to see the code for a function. You can examine how many functions work by just typing their name without any parentheses.

You can find out what arguments a function takes by using the code completion and it’s help snippet, or by searching for the function in the Rstudio Help tab.

When you’re inside the brackets of a function you can get the list of available arguments and auto-complete them.

Examining functions

One of the niftiest things about R is being able to see the code for a function. You can examine how many functions work by just typing their name without any parentheses.

Sys.Date

## function () 
## as.Date(as.POSIXlt(Sys.time()))
## <bytecode: 0x10df91748>
## <environment: namespace:base>

The first line(s) show how the arguments are specified. Subsequent lines show the code and the final lines starting with < can be mostly ignored.

Function input patterns

Functions tend to conform to certain patterns of inputs.

No inputs

Some functions don’t require the user to provide info and so they don’t have any arguments. Sys.Date() and similar functions do not need user input because the functions provide information about the system.

Looking at the function definition above, we can see that there are no arguments specified in the first line.

Single inputs

Other functions only have a single allowed input. length() returns the length of an object so it only allows you to provide it with an object.

length

## function (x)  .Primitive("length")

We can see in this definition that the function takes the argument x.

Many inputs

Some functions have multiple inputs, although not all of them are necessarily mandatory. head() and tail() have been used so far with only a single input but they take an optional argument as to how many elements should be returned.

head(letters)

## [1] "a" "b" "c" "d" "e" "f"

head(letters, 2)

## [1] "a" "b"

The rnorm() function allows us to generate a vector of values from a normal distribution. We can tell it how many values we need (n), and we can optionally provide the mean (mean) and standard deviation (sd) to describe the Normal curve that values should be selected from.

rnorm

## function (n, mean = 0, sd = 1) 
## .Call(C_rnorm, n, mean, sd)
## <bytecode: 0x10ded12a8>
## <environment: namespace:stats>

Looking at how rnorm is specified we can see that we’re expected to provide n, but mean and sd are given values of 0 and 1 respectively by default.

rnorm(n=5)

## [1] -1.4818734 -1.0309718  1.4056332  0.4328255 -0.6992250

rnorm(n = 5, mean = 10, sd = 2)

## [1]  6.759442  8.453144  8.035506 12.549855 11.781241

Unlimited inputs

Other functions can take an unlimited amount of input values. Functions like sum() will sum the values from a number of objects.

sum

## function (..., na.rm = FALSE)  .Primitive("sum")

The ellipsis (...) is used to denote when the user can provide any number of values.

sum(1:3, 1:9, pi)

## [1] 54.14159

Naming arguments

Every input provided to a function is associated with an argument.

Each argument must have a name. Even functions that allow unlimited inputs assign these inputs to a name. Behind the scenes, they get put into a list object and the list gets called ... (or ellipsis).

There are some typical names for arguments that take your data object. These include:

x
data
.data
df

You don’t usually have to provide the argument names, just put things in the relevant places in the function. Sometimes though, you will need to use argument names.

Here are my rules of thumb for knowing when you need to name names:

You’re using the arguments in an order that is different from the function author’s intended order (you might be skipping some arguments as the default values are fine or you might just prefer a different order)
The arguments you want to specify show up after the ... in a function’s argument list
You want to give a specific name to a value in a ... argument

We can provide names for clarity or so we can use arguments out of order if we prefer to.

rnorm(n = 5, mean = 10, sd = 2)

## [1] 10.775754 10.104590  7.055771 14.544010  9.135413

rnorm(mean = 10, sd = 2, n = 5)

## [1]  9.625624  8.345576  8.794756  8.914196 10.417960

A common behaviour change that you’ll need to work with is how missing (NA) values get handled. Functions that allow you change this behaviour, usually have an argument called things like na.rm, na.omit, and na.action.

sum(1:5, NA)

## [1] NA

sum(1:5, NA, na.rm = TRUE)

## [1] 15

In the sum() example, I used the na.rm argument’s name. This is because otherwise the TRUE would be considered part of the values being passed for summing. Without the name, the value gets considered as part of the ....

sum(1:5, NA, TRUE)

## [1] NA

A function will sometimes have ... at the end of it’s list of arguments when it utilises other functions and those have optional / default values.

For instance the predict() function allows us to take a model we’ve built and apply it to some new data.

It works for many different types of model and these different models expect different types of inputs. Some models expect data.frames, others expect time series data, etc.

There’s lots of potential variations, the only thing that is mandatory is the model object.

predict

## function (object, ...) 
## UseMethod("predict")
## <bytecode: 0x103e21638>
## <environment: namespace:stats>

The predict() function then determines what type of model object you’ve provided it and passes the model, and any other values you provided, to the relevant function, returning back the results.

linearMod<-lm(Sepal.Length~., data=iris)

predict(linearMod, iris[1,])

##        1 
## 5.004788

And so very quickly before I summarise all that for you, just a note to say that that’s the very basics of R covered, but look out soon for a couple of posts on making R work for you - R projects (A very good habit) and a Github 101 coming up!

Summary

R uses functions as the means of performing operations.

Functions can take 0 or more arguments. All arguments may be mandatory, but some can be optional or even undefined.

You can use argument names to provide arguments in different orders to that defined by the function author or to provide them in the case where an ellipsis (...) is used in a function.

R packages bundle functionality and/or data.

You can install packages from the central public repository (CRAN) via install.packages() or install them from GitHub with the package devtools. R packages contain documentation that helps you understand how functions work and how the package overall works.

When you want to make use of functionality from a package you can either load all of a package’s functionality by using the library() function or refer to a specific function by prefixing the function with the package name and two colons (::) e.g. utils::help("mean").

There are many packages out there for different activities and domain-specific types of analysis. Use online search facilities like rdrr.io or CRAN task views to find ones specific to your requirements.

As usual, all the code from this installments video is included below in one fell swoop.

Video code

install.packages("tidyverse")

library("tidyverse")

help(package="dplyr")

and then

?bind_rows

one <- mtcars[1:4, ]
two <- mtcars[11:14, ]

# You can supply data frames as arguments:
bind_rows(one, two) -> THREE

Sys.Date

length

length(THREE)

head

head(THREE, 5)

rnorm

rnorm(n=10)

rnorm(n=5, mean = 2, sd = 7)

#Have a read of the blog post here to find out why we sometimes use the argument names inside the brackets!

sum

sum(1:9, 7, pi)

predict

linearMod<-lm(Sepal.Length~., data=iris)
# logisticMod<-glm(Species~., data=iris, family=binomial)

predict(linearMod, iris[1,])

Happy coding :) Ellen!

Cosmos DB for Data Science

Fri, 07 Sep 2018 10:54:45 +0100

Cosmos DB is a snazzy new(ish) Microsoft Azure product. I was able to go to Microsoft Office in London for three days of training on the database service, which was really well structured and well run, with a lot of knowledgeable Microsoft bods around to pass on their considerable knowledge. This post will extract out some key features and benefits of the service, and then discuss how this fit’s into a data scientists role.

What is CosmosDB?

From a pure and simple database perspective cosmos is a ‘Document Database’. This means it is a non-relational store of JSON documents. MongoDB and CouchDB are examples of other Document Databases, but CosmosDB opens a whole new universe of potential.

Two quotes from Microsoft demonstrate this pretty effectively.

“A globally distributed, massively scalable, multi-model database service”

“A fully-managed globally distributed database service built to guarantee extremely low latency and massive scale for modern apps”

Microsoft seems to me to be targeting a few specific markets with this product. Many of the case studies revolved around IoT, specifically the increasing importance of data from car telemetry systems, web and mobile apps, gaming and retail.

These applications are highly suited to the technology for a number of reasons

Semi-structured data

Imagine you have a website where you sell phones, which consumers may want to compare against in relevant ways. For example, your phone data may look like this:

{
    "phone1":{
        "brand":"Nokia",
        "model":"3310",
        "year":"2000",
        "cpu":"MAD2WD1"
    },
    "phone2":{
        "brand":"Nokia",
        "model":"3310",
        "year":"2017",
        "soc":"MT6260",
        "camera":"2mp"
    }
}

Two Nokia 3310s, separated by 17 years. Some properties are identical: brand and model, some are different: year, and some are present in one, but not the other: cpuand soc, and the two documents have a different length, the phone2 has a camera resolution value.

Imagine a representation of this in a relational database. If both phones were a record, each would have a few NULL values. Further, if you were making a schema for this database in 2000, could you have predicted the ubiquity of phone cameras? I doubt it, which will have meant a potentially awkward schema update later.

The retail market hits this all the time as products are developed, but so does the car industry, as new models are released with improved features: hybrid power > electric power > self-driving > ??hover cars??

It’s 2018. I was robbed. via GIPHY

CosmosDB’s solution is to make the product ‘schemaless’. Writes won’t fail because the schema isn’t met, there isn’t one to validate against. Data is automatically parsed and indexed. The default is for all data to be indexed.

Write optimised

Internet of Things telemetry data is a growing source of data, and automobile telemetry is of particularly high value, and high scale. CosmosDB is a ‘write-optimised’ data store. It has been engineered to write high volumes, fast, every time.

All writes are always durably quorum committed in any region where you write while providing performance guarantees.

Some other document databases are similarly noted for their ‘fast’ writes, however these tend to rely on in-memory caching. In the event that someone starts driving their car you have multiple thousands of documents (records) hitting your data base really fast. In-memory caching solutions in other databases rely that that stream of data will eventually end and until that point the data is held in memory, before it is written to disk. If the car is being driven for five minutes, maybe the memory caching will hold all the data, which can then be written back to disk in the future. If it’s a cross country, 3 hour trip though, it’s either going to be really expensive to hold that data, or some data will be lost due to buffer overflow. Literally the data just can’t fit into the memory. CosmosDB doesn’t rely on this (risky) business, but still maintains supremely fast performance, which is backed by some pretty big SLAs.

“Oh my God, it’s full of documents!” via GIPHY

Turn-key distribution

A distributed database is one that is run across multiple machines, nearly always via a cloud provider. This enables data to be closer to it’s users. In the event of a global audience this becomes really important at scale, because the distance the data needs to travel from device to database is one of the factors in determining write speed. Imagine your video game has just launched new DLC, and you are expecting very bursty traffic for a few days as all your users dive into the new maps. Initialise some more clones of your data sets to decrease latency and serve the increase in traffic volume.

If that sounds a little complicated, the ‘turn-key’ nature should help calm your mind.

Turn-key geo-distribution in cosmosDB is really this easy

Data storage compliance is another consideration. Some data (such as UK medical data) might not be inherently suited to this architecture, however, cosmosDB is a ‘Foundational (Ring 0)’ Azure service, so it is available in all Azure regions, including their ‘Sovereign’ and ‘Government’ regions. This allows compliance control on where, and in what way, the data is stored.

A database army. And I must say, one of the finest we’ve ever created via GIPHY

Multi-model

And so how do you communicate with this weird alien database? In the creation step for a CosmosDB account you can select an api (and resultant data model) from the following list:

SQL
MongoDB
Cassandra
Azure Table
Gremlin (graph)

CosmosDB’s data model is then set relative to this selection, however the query methods can be modified without impacting the underlying data structure.

The data models include key:value, column:family, document and graph via it’s use of an atom-record-sequence type system in the database engine. In practical terms this means that you can read in data in a graph form from a data source, but still query it using SQL in a ‘relational’ way. Your social network app may be suited to write in data in a directed network way, but maybe your business intelligence team are highly invested in a SQL toolset. CosmosDB should help to bridge this gap.

SQL and Graph at Tenegra! via GIPHY

Gotchas

`JOIN`

In relational DBs JOIN works in a familiar and intuitive way, but with this non-relational data structure what does a JOIN mean, and what can we use it for?

Firstly, JSON allows for arrays to be included in a document.

[
  {
    "droid": "BB8",
    "affiliation": "New Republic",
    "series": "BB",
    "films": [
      "The Force Awakens",
      "The Last Jedi"
    ]
  },
  {
    "droid": "BB9E",
    "affiliation": "First Order",
    "series": "BB",
    "films": [
      "The Last Jedi"
    ]
  }
]

This can be queried through SQL like this:

SELECT
    droids.droid,
    droids.affiliation,
    films
FROM
    droids
JOIN
    films in droids.films

Which will return an output like this:

[
    {
        "droid":"BB8",
        "affiliation":"New Republic",
        "films":"The Force Awakens"
    },
    {
        "droid":"BB8",
        "affiliation":"New Republic",
        "films":"The Last Jedi"
    },
    {
        "droid":"BB9E",
        "affiliation":"First Order",
        "films":"The Last Jedi"
    }
]

The query has merged multiple documents, into a single flattened results set. Here there is no INNER, LEFT, or OUTER to do subsetting. This is achieved with the WHERE clause.

SELECT
    droids.droid,
    droids.affiliation,
    films
FROM
    droids
JOIN
    films in droids.films
WHERE
    films IN ("The Last Jedi")

[
    {
        "droid":"BB8",
        "affiliation":"New Republic",
        "films":"The Last Jedi"
    },
    {
        "droid":"BB9E",
        "affiliation":"First Order",
        "films":"The Last Jedi"
    }
]

JOIN does just, and only that. Filtering is done in the WHERE clause.

Queries and Indexing

CosmosDB automatically indexes all data as it comes in by default. For high volume data this might be problematic leading to longer write times, so how can this be mitigated? The index can be selectively applied through an ‘index policy’. However the trade off for this means that value will no longer be queryable.

For instance if your data has information like this:

[
    {
        "affiliation":"Jedi",
        "character_name":"Luke Skywalker"
    },
    {
        "affiliation":"Sith",
        "character_name":"Snoke"
    }
]

And character_name was excluded by your indexing policy, then

SELECT
    force_users.affiliation,
    force_users.character_name
FROM
    force_users
WHERE
    force_users.character_name == "Luke Skywalker"

would error. It wouldn’t know where Luke was!

How should a data scientist use CosmosDB?

Connection

To make a simple connection to the data base from an R session you can use the ODBC driver supplied by Microsoft. Once this has been setup you can use the odbc and DBI packages to run queries against the connection in R, which is even easier using RStudio’s tools. However, the connection can be only part of the challenge if you are interested in using a large scale data source.

Computing

With such effortless scaling of data collection and storage, data scientists (& Business Analysts and & Management Information Specialists, etc.) might see this as a simple solution to scale issues. However, remember that this is a ‘write optimised, data storage layer’, frequently for analytics workstreams what we should be looking for is data computation layers. Our workflows can be very memory intensive, though luckily there are solutions.

HDInsights

Microsoft have a number of implementations of Apache Spark tied into their Azure platform. HD Insights is a fully managed analytics service, built on Apache Spark (and others) and is easy to plumb into an R interface.

Databricks

During the training many examples of scaled out analysis workflow in practice used the Databricks service, a proprietary fork of Apache Spark. This can then be used in an R session via SparkR and sparklyr.

Normal Apache Spark

Of course, as spark is so highly supported, there is nothing to stop you setting up your own standard spark cluster and connecting to it in the traditional way.

Cosmic Power

CosmosDB is an exciting piece of technology. Distributed, scalable, schema-less, multi-model and multi-lingual, for many applications I see this as a real Cosmic Power. It’s not for everything, but for where it works it’s not only a powerful tool, but one that seems more straightforward to implement than many of it’s competitors, and one that’s also got Microsoft’s focus on enterprise application baked into the foundations. Enjoy your exploration of the stars.

via GIPHY

If this article has sparked a curiosity in this astronomical technology, make sure to have a look at our upcoming online training course

R Objects

Fri, 24 Aug 2018 00:00:00 +0000

R objects

To quickly recap, so far we’ve just worked with some single values to get to grips with how some of the various operations work. Of course, we rarely work with a single value! If we did, we could just use a calculator.

This instalment you’ll get to grips with some different ways of storing data and how to manipulate your datasets in the “traditional” way. This will help you understand a lot of code written in the past, and will equip you to understand data manipulation of tabular data.

Get a cuppa and settle in to this sessions video, then have a play with the code yourself and read along with the blog! As always, get in touch with any questions on twitter using @LockeData!

Storing values

When we were performing operations, we got some values output to the console. One of the key principles in writing code is Don’t Repeat Yourself (DRY) so we need to know how we can avoid repeating ourselves in R. One of the ways you can do that is to store a value for use later.

In R, we can store values by assigning them a name. This makes a variable or object. We can do this with a few different operators, but the traditional operator is a <-[1]. The format for assigning a value is nameofthing <- value.

my_variable <- 5 + 3
my_variable*2

## [1] 16

Valid names for a variable include upper-case letters, lower case letters, numbers anywhere but the beginning, periods (.), and hyphens (_).

There are a number of different competing conventions for how you name variables. The most common conventions are shown below. I have no strong feelings for any system and only ask that you pick one and stick with it within a single script. Whatever you do, don’t forget names are case sensitive!

myfirstvariable <- 1
myFirstVariable <- 1
MyFirstVariable <- 1
my_first_variable <- 1
my.first.variable <- 1

You can create names breaking the rules governing valid names by placing the rule breaking name between two back-ticks (`). I don’t recommend you do this with variables you’ll create, but you’ll often end up with names that break conventions when importing data, especially when you import from spreadsheets.

Vectors

A vector is a collection of values that hold the same datatype. It is one-dimensional in that none of the elements in the collection correspond to other values like they might in a table of values.

A single value is actually a vector of length 1.

When I introduced the colon (:) as a means of generating a sequence, we were in fact generating a vector where each element was a number in the sequence. The vector has a length which is as long as the number of values generated by the sequence.

-1:1

## [1] -1  0  1

Another way of producing a vector is to use the combine function (c()). This is great for combining a number of disparate character strings into a vector.

c("red","yellow","blue")

## [1] "red"    "yellow" "blue"

A single value is a still a vector. What we see when we use the c() function is that we’re combining vectors. As a result we can also use it on longer vectors too.

c(1:3, 2:1, 5:8)

## [1] 1 2 3 2 1 5 6 7 8

When we combine values into a single vector, R will change everything to the same datatype using some conversions.

Getting information about vectors

Our class() function will still work with a vector with a length greater than 1 to get you it’s datatype.

Let’s look at a sequence of numbers and one of the built-in vectors that contains the alphabet.

class(1:10)

## [1] "integer"

LETTERS

##  [1] "A" "B" "C" "D" "E" "F" "G" "H" "I" "J" "K" "L" "M" "N" "O" "P" "Q"
## [18] "R" "S" "T" "U" "V" "W" "X" "Y" "Z"

class(LETTERS)

## [1] "character"

We can use the length() function to find out the number of elements in a vector.

length(pi)

## [1] 1

length(LETTERS)

## [1] 26

To extract the names of values in a vector, we can use the names() function.

steph<-c(Steph="forename", Locke="surname")
names(steph)

## [1] "Steph" "Locke"

Calculations on multiple vectors

When we perform calculations on two vectors, R will try to perform the operation for each set of elements. This is an element-wise or pair-wise calculation methodology.

In SQL, it’s equivalent to where you might write colA*colB and you’ll get the answer calculated for every row in the table. In Excel, it’s equivalent to a Fill Down of multiplying two values on the same row.

Let’s looks at how this works in practice in R.

We have two vectors, each containing two elements.

vecA <- 1:2
vecB <- 2:3

## [1] 1 2

## [1] 2 3

If we want to multiply the two vectors by each other, R will match each element in the first vector with it’s counterpart in the second and multiply the two values together to make a new element.

You can also use this functionality of making a vector the same length as another, known as recycling, work for other mis-matched vector sizes. The only rule is that one of the vector lengths must divide cleanly by the other.

Two vectors of the same length divide by the other’s length exactly one time and won’t need to recycle
A vector of length one always cleanly divides any other vector’s length and so will be recycled
A vector of length 2, will divide any vector with an even length and so will be recycled in those cases, but it cannot recycle cleanly for odd length vectors

1:10 * 2
1:10 * 2:3
1:10 * 2:4

##  [1]  2  4  6  8 10 12 14 16 18 20

##  [1]  2  6  6 12 10 18 14 24 18 30

## [1] "longer object length is not a multiple of shorter object length"

Vector recycling is useful and dangerous – it can help you make elegant code or give you unexpected results. Especially when starting out, I recommend you make your vectors either the same length or length 1.

Proceed with caution

Bitwise

Our logical operators that we covered earlier, work in a pairwise fashion. They’ll return a vector of the same length as the longest one used in your logical statement.

a<-1:2>1
b<-2:3>1

## [1] FALSE  TRUE

## [1] TRUE TRUE

Making logical statements returns vectors with a logical datatype.

a&b

## [1] FALSE  TRUE

a|b

## [1] TRUE TRUE

Occasionally, you expect to only be operating on a single pair of values and want to enforce that R should only do the calculation on the first pair. In R, this called a bitwise AND (&&) or OR (||).

A bitwise logical statement will only do the check for the first elements in the vectors and ignore all the others.

a&&b

## [1] FALSE

a||b

## [1] TRUE

Use bitwise operators with extreme care!

And there we have it - It’s fair to say that you’ve pretty much covered the basics of data handling in R now, so next time we’ll have a look at some packages and functions - these are the bits where somebody else has done the hard work for you!

Happy coding!

Ellen :)

As always, here’s the video code to take away and play.

my_first_var <- 5+3

my_first_var * 2

my.other.variable = 7*4

9-5 -> YetAnotherVariable

rm(my_first_var)

dont.delete.me <- "please don't!"

A glass shattering book draw with gganimate

Wed, 01 Aug 2018 00:00:00 +0000

It’s time for a Twitter book draw again: every month, a random Locke Data Twitter follower wins an excellent data science book! This month’s book was Weapons of Math Destruction : How Big Data Increases Inequality and Threatens Democracy. The animation I chose to create was inspired by the idea of destruction and by my wanting to try out the fantastic new API of the gganimate package, and a very fast new gif encoder, gifski.

Choosing an animation concept

I am not a designer nor an artist, but I like imagining new animations to announce the book winner, that I want to be fun, and useful by illustrating the use of some nifty R tools. That’s a good way for me and you to learn new R skills! I’ve reported on my efforts with magick to create a crystal ball for chibi Steph, R package for image manipulation, and with particles, R package for simulating, well, particles, to move followers’ names around!.

This month I thought about what destruction meant to me, and got the rather simple idea to have glass shatter, thus revealing the winner’s name.

Planning the my animation implementation

Feel free to skip over this section and go to the code directly, unless you want to know more about my search and process to write said code!

I wondered how I could shatter glass, i.e. cut a blue square in small pieces and make these pieces fall and somehow ended up thinking that I could draw random points and then use their Voronoi tiles as fragments. I’m not sure glass shattering really looks like this but I decided against experimenting with my windows.

I wasn’t too sure how to make and use the fragments though. I re-read Thomas Lin Pedersen’s excellent post about making a Christmas card using Voronoi tesselation but as nice and well-explained as it was, it wasn’t exactly what I was after. I set out to find a way to create Voronoi tiles as polygons and then to shift at least some of them downwards to create the animation.

I tried using the deldir package a bit to create Voronoi tesselation out of random points but its output didn’t make me too happy since it was segments rather than polygons. Re-creating polygons from them didn’t sound too easy. I ditched this idea and googled keywords such as Voronoi and polygons and R and then sf the geospatial package since the first results indicated Voronoi tesselation was well supported for mapping stuff, and found this very useful Stack Overflow post that made me adapt an example from sf documentation.

Once I had the polygons, the rest was animation as usual, well not really, since gganimate new API by Thomas Lin Pedersen is different, and so powerful! I haven’t watched Thomas’ useR! keynote talk about The Grammar of animation yet, but I look forward to it since I both admire his packages, and his blog writing. I only use very basic gganimate stuff here. A good intro to its grammar can be found at https://github.com/thomasp85/gganimate/tree/master#gganimate- .

I was also glad to read that gifski is now on CRAN for all your gif making needs (it’s actually the default gif renderer of gganimate). This package by Jeroen Ooms is not only a wrapper to the fastest gif renderer around which is cool enough in itself, but also the first CRAN package that interfaces a Rust library! Read more about gifski in this tech note.

Writing the actual animation code

The first part of the code consisted of drawing random points and using them as a basis for a Voronoi tesselation inside a box.

# adapted from https://r-spatial.github.io/sf/reference/geos_unary.html
lims <- c(0, 10)
# sample 200 points inside an environment
# with a fix seed
with(set.seed(42),
     points <- sf::st_multipoint(matrix(runif(n = 400, lims[1],
                                              lims[2]),,2)))

# get a square box that'll be the area of our animation 
box <- sf::st_polygon(list(rbind(c(lims[1], lims[1]),
                             c(lims[2], lims[1]),
                             c(lims[2], lims[2]),
                             c(lims[1], lims[2]),
                             c(lims[1], lims[1]))))

# Get the Voronoi polygons of the points
v <- sf::st_sfc(sf::st_voronoi(points, sf::st_sfc(box)))
# Keep only the part of them inside the box
voronoi <- sf::st_intersection(sf::st_cast(v), box)

# Get a data.frame with the polygons
get_df_from_polygon <- function(polygon, index){
  mat <- as.matrix(polygon)
  tibble::tibble(x = mat[,1],
                 y = mat[,2],
                 tile = index)
}

df <- purrr::map2_df(voronoi, 1:length(voronoi),
                     get_df_from_polygon)

The animation will consist of two parts:

The fragments appearing by drawing their borders with different shades from blue like the background to white.
The fragments falling.

The code below creates the PNGs corresponding to the first part of the animation.

fs::dir_create("august_frames")

plot_one <- function(step, df){
  cols <- colorRampPalette(c("#2165B6", "white"))(10)
  ggplot(df)  +
    geom_polygon(aes(x = x, y = y, group = tile),
                 fill = "#2165B6", col = cols[step]) +
    theme_void()

  outfil <- file.path("august_frames",
                      sprintf("plot1_%02d.png", step))
  ggsave(outfil, width = 6.7, height = 6.7)
  magick::image_read(outfil) %>%
    magick::image_resize("480x480") %>%
    magick::image_write(outfil)


}


purrr::walk(1:10, plot_one, df)

Then I made the data.frame a bit bigger by adding a second state with most tiles fallen at the bottom. I first create a variable border that indicates if the tile pertains to the sides or top because magically these tiles won’t fall.

df <- dplyr::group_by(df, tile) %>%
  dplyr::mutate(border = any(x %in% lims) |
                  any(y == lims[2]))

# Define the second state with tiles fallen
df2  <- df
# a bit random but it does look ok!

df2 <- dplyr::group_by(df2, tile) %>%
  dplyr::mutate(y = ifelse(!border,
                           -2 - min(y) + y,
                           y)) %>%
 dplyr::ungroup()

df$frame <- 1
df2$frame <- 2
dfall <- dplyr::bind_rows(df, df2)

This was followed by the animating of the second part with gganimate, here after creating a fake winner (sorry Steph, you won’t get to gift yourself the book). Because we’ll need to use PNGs from the first part, we don’t use gganimate’s built-in gif or video renderers, but instead save PNGs.

Important note: gganimate latest version isn’t on CRAN yet, follow the instructions in its GitHub repository or download a built source version of the package from Appveyor.

winner <- list(name = c("Steph Locke"))
p <- ggplot(dfall) +
    annotate("text", label = winner$name,
             x = 5, y = 5,
             size = 12, family = "Roboto",
             col = "#E8830C") +
    geom_polygon(aes(x = x, y = y, group = tile),
                 fill = "#2165B6", col = "white") +
    # Here comes the gganimate code
    # create transition, with the variable frame
    transition_states(
      frame,
      transition_length = 1,
      state_length = 1,
      wrap = FALSE
    ) +
    # use an ease that makes tiles fall faster
    # at the end of the transition
    ease_aes('exponential-in') +
    theme_void() +
    coord_cartesian(xlim = lims, ylim = lims)

  # create and save frames

   animate(p,
   renderer = file_renderer(dir = "august_frames",
                            prefix = "plot2_") )

Now we can use all the PNGs that are in the “august_frames” folder, with gifski!

images <- sort(as.character(
   fs::dir_ls("august_frames")))

gifski::gifski(images,
                gif_file = path,
                delay = 0.1,
                width = 480,
                height = 480)

# clean
 fs::dir_delete("august_frames")

And voilà, a glass shattering animation!

Glass shattering book winner reveal

Conclusion and resources round-up

In this post I explained how I created an animation of glass shattering.

R packages that are important to know for producing animations are:

gganimate that now implements the Grammar of animation, like ggplot2 implements the Grammar of graphics! I’d recommend to watch Thomas Lin Pedersen’s useR! keynote talk about The Grammar of animation and to follow the impressive activity of the gganimate GitHub repo.
gifski for gif rendering. It’s used under the hood by gganimate but you might need it to combine PNGs yourself if you tweak animations a bit more like we did here. You can read this tech note about gifski by Jeroen Ooms, and to check out the r-rust Github organization.
Likewise, if you want to tweak some frames, it’ll be useful to know a bit about magick, wrapper to ImageMagick, an R package for image manipulation developed at rOpenSci by Jeroen Ooms. This package has a good vignette.

Have fun animating R visualizations! And don’t forget that next month, again a random Locke Data follower will win a great book: you should follow Locke Data on Twitter to be in with a chance of winning!

Learn to R blog series - Operators and Objects

Thu, 19 Jul 2018 00:00:00 +0000

Basic operations

Now that we have some datatypes, we can start learning what we can do with them.

This weeks video whisks over the basic operators - you know what plus and minus do, right? Then we look at some other less common operators and recap it all below.

Pay special attention to all.equal(), there’s a reason I bang on about it!

Maths

In R, we have our common operators that you’re probably used to if you’ve performed calculations on computers before.

Action	Operator	Example
Subtract	-	`5 - 4` = 1
Add	+	`5 + 4` = 9
Multiply	*	`5 * 4` = 20
Divide	/	`5 / 4` = 1.25
Raise to the power	^	`5 ^ 4` = 625

R adheres to BODMAS[1] so you can construct safe calculations that combine operators in reliable ways.

Action	Operator	Example
Basic sequence	:	`1:3` = 1, 2, 3
Integer division	%/%	`9 %/% 4` = 2
Modulus	%%	`9 %% 4` = 1

The colon (:) is a snazzy way of generating a sequence of numbers that step by 1. You specify a beginning number and an end number and R will produce all the whole numbers including and between the two numbers. This even works for negative numbers or producing descending values.

Comparison

The next important thing to know about is how to write comparisons; ways of looking at two or more things and finding out if they’re the same, or different.

Common operators

The less thans and greater thans are symbols that are in pretty much every language for comparisons, but the test to see if two values are the same or not can often vary across languages.

Summary

Action	Operator	Example
Less than (lt)	<	`5 < 5` = FALSE
lt or equal to	<=	`5 <= 5` = TRUE
Greater than (gt)	>	`5 > 5` = FALSE
gt or equal to	>=	`5 >= 5` = TRUE
Exactly equal	==	`(0.5 - 0.3) == (0.3 - 0.1)` is FALSE
Exactly equal	==	2 == 2 is TRUE
Not equal	!=	`(0.5 - 0.3) != (0.3 - 0.1)` is TRUE
Not equal	!=	2 != 2 is FALSE
Equal	all.equal()	`all.equal(0.5-0.3,0.3-0.1)` is TRUE
In	%in%	`"Red" %in% c("Blue","Red")` is TRUE

Logic

Once we can do a single check, we inevitably want to do multiple checks at the same time.

To combine multiple checks, we can use logical operators.

Common operators

The ampersand (&) allows us to combine two checks to do an AND check, which is “are both things true?”.

The pipe, or bar (|)[2] allows us to do an OR check, which is “are either of these things true?”.

The exclamation point (!) allows us to a perform a NOT check, by negating or swapping a check’s result. This allows you say things like “is this check true and that check not true?”.

Other operators

Less commonly, there other logical checks you might to perform.

We can do an XOR, where one and only one of two values being checked is true.

Summary

We can produce sophisticated checks from a few simple building blocks. This will come in very handy down the line when doing things like filtering datasets or creating new fields in your data.

Action	Operator	Example
Not	!	`!TRUE` is FALSE
And	&	`TRUE & FALSE` is FALSE
And	&	`c(TRUE,TRUE) & c(FALSE,TRUE)` is FALSE, TRUE
Or	`	`
Xor	xor()	`xor(TRUE,FALSE)` is TRUE

Play time!

This basic operations section has hopefully taught you how to manipulate values and construct comparisons. These are important building blocks in data analysis, and whilst we’ve been working with only a single value at a time, in the next section we’ll see how it works with more data.

Now take the code below and run wild…or let your turtle run wild at least! Use what you’ve learned here and see what you can do with operators - I’ve given you some blog posts and guides with example code in - take this and alter it so that the numbers are sums, or even better come up with your own!

Four simple turtle graphs.
A guide to the TurtleGraphic package for R This one has some great tricky examples at the end!

install.packages("TurtleGraphics")

library(TurtleGraphics)

## Loading required package: grid

turtle_init() # This starts off the turtle

turtle_down() # This means that when you feed instructions to your turtle, it leaves a mark.

Use “turtle_left()” and place the number of an angle between the brackets to turn your turtle. The same applies for “turtle_right()”

Use “turtle_forward()” and “turtle_backward()” to move him around and draw something. Inbetween the brackets should be a number specifying the number of “steps” the turtle should take.

Here’s the twist. Use what you’ve learned today with the basic maths operations and create your drawing using sums instead of numbers. We’d love to see your efforts so tweet us a picture to @LockeData!

Ellen :)

P.S. My video code is below if you want to take it away with you!

3 + 3

## [1] 6

10 - 5

## [1] 5

100 / 4

## [1] 25

25 * 9

## [1] 225

6 ^ 2

## [1] 36

2 + 3 * 10 - 7

## [1] 25

(2 + 3) * 10 - 7

## [1] 43

3 < 5

## [1] TRUE

6 > 4

## [1] TRUE

3.5 == 3.5

## [1] TRUE

9 <= 12

## [1] TRUE

12 != 9

## [1] TRUE

24 >= 27

## [1] FALSE

[1] Brackets, Other, Division, Multiplication, Addition, Subtraction. Note that in some countries it’s BEDMAS, where the E stands for Exponents, which is a special Other

[2] Getting this symbol can be painful as it varies substantially by keyboard, so apologies if it takes you a while to hunt this symbol down.

Harmonizing and emojifying our GitHub issue trackers

Thu, 12 Jul 2018 00:00:00 +0000

A part of Locke Data’s mission is sharing R knowledge and tooling with the world for free. If you have a look at our GitHub account, you’ll see we’ve pinned six of our package repos. Furthermore, to make it easier to find all the R stuff we’ve packaged up, we’ve added an “r-package” repo topic to all packages: find them all via this URL. Adding such repo topics isn’t the only harmonization effort we’ve done to make it easier to maintain and promote our packages suite. In this post, at the request of Irene Steves, we shall explain why and how we semi-automatically harmonized and emojified our issue trackers with the help of GitHub’s V3 API and of the gh package!

Emojify what? A short intro to GitHub issue tracker organization

If you’re a bit familiar with software development on GitHub, either because you develop your own packages or sometimes browse the repos of other packages, you’ll know each repository has an issue tracker. That’s the place where bugs and feature requests are gathered. Both repo members and external contributors can add stuff to the list! Often, the issue tracker is your personal to-do list because no one else cares yet, and you can get righteously very excited over any new issue!

Yaaayy!! I built a thing and somebody is using it :)

I guess this is what popular OSS library authors feel when they get an issue, right?https://t.co/2WyqcOkElW
— El esposo de Daniela 💉💉💉 (@g3rv4) July 10, 2018

But as your repo gets a bit more popular, or as you yourself take notes of a lot of stuff, your to-do list gets very long and hard to make sense of! GitHub provides tool for organizing it. Of particular interest are milestones that are collections of issues corresponding to say how you imagine each future release to be, and labels that indicate what each issue is about; so you can have issues related to docs. Labels can also indicate the level needed to resolve an issue, and whether you’d welcome external contributors, cf this GitHub article. Labels are great, for instance you can decide to work on only docs on one day and filter issues that are related to that.

Why harmonizing sets of issue labels over an organization?

As mentioned above, Locke Data maintains a few packages now. Each of them used to have its own set of issue labels. We decided to make them all similar for two reasons. First, it’s nice to speak the same language over different projects within our GitHub organization, making it easier to transition from one repo to the other. Then, we wanted to be able to eventually make use of tools such as Jim Hester’s tidyversedashboard package, and thought the issue tab would be handier if having similar labels.

Therefore, we set off to decide on a set of issue labels!

Why emojifying issue labels?

Maybe one reason we decided to add emojis in our set of issue labels is simply that we could! You actually only can do that since February the 22d this year! Other reasons include that it’s fun, and that emojis make it easier to differentiate issue labels at a glance. Colours are good, more on that later, but not everyone can differentiate them anyway.

How to update issue label sets in practice?

We updated the issue labels more or less automatically, making the most of GitHub’s APIs and of related R packages! The two actual scripts live here and here. Here, we write up some general lessons, updated to make your life easier since the API evolved a bit since we worked on our labels.

We first got the list of all package repos using the experimental ghrecipes package that interacts with GitHub V4 API and is developped in rOpenSci’s ropenscilabs GitHub organization. ghrecipes::is_package returns TRUE if a repo has DESCRIPTION and NAMESPACE files and R and man folders. Maybe you already have such a list of your repos handy!

repos <- ghrecipes::get_repos("lockedata")
repos <- dplyr::mutate(repos, 
                       name = stringr::str_remove(name,
                                                  "lockedata\\/"))
# are packages
repos$is_pkg <- purrr::map_lgl(repos$name,
                               ghrecipes::is_package_repo,
                               owner = "lockedata")

repos <- dplyr::filter(repos, is_pkg, !is_fork)

Once we had that list, we switched to using GitHub’s V3 API via the gh package developed in RStudio’s r-lib organization.

At the time we updated labels, we couldn’t directly update them but in any case now you can, no need to delete and re-create labels at the risk of losing your labelling (we didn’t loose anything though). Note that in general, when messing with your issue trackers automatically, you should definitely test your code on a single repo first, since your power could, for instance, make you unlabel all issues at once, which would be quite bad!

With the current endpoints here’s what you should do:

Take care of your authentication taking into account your rights over the repos and choosing the scope of the token wisely.
List all labels over the repos. It’d look like this.

gh::gh("GET /repos/:owner/:repo/labels", owner = "lockedata", repo = "optiRum")

After doing that, define what each old label is going to become. When defining our new labels, we used this cheatsheet of emojis for GitHub.

# dput(sort(unique(labels$name)))
new_labels <- tibble::tibble(old = c("bug", "duplicate", "enhancement", "first-timers-only", "good first issue", 
                                     "help wanted", "invalid", "question", "up-for-grabs", "wontfix"
),
                             new = c("bug :bug:",
                                     "duplicate :dancers:",
                                     "enhancement :sparkles:",
                                     "good first issue :hatching_chick:",
                                     "good first issue :hatching_chick:",
                                     "help wanted :raised_hand:",
                                     "wontfix :see_no_evil:",
                                     "question :question:",
                                     "help wanted :raised_hand:",
"wontfix :see_no_evil:"))

You’ll also need to choose colours. We first used random colours sampled thanks to charlatan::ch_hex_color() but also chose to use official Locke Data colours for a few of the labels.
Then, using this correspondance table and your favorite functional programming / looping method (purrr in our case, get started with this slidedeck), use that fantastic endpoint that didn’t exist when we did that (or that we missed?). Here’s how it’d look for a single label, identified by its current name, and repo.

gh::gh("PATCH /repos/:owner/:repo/labels/:current_name",
       owner = "ropenscilabs", repo = "ghrecipes",
       current_name = "enhancement",
       name = "feature :sparkles:",
       color = "ffa07a",
       description = "stuff that'd be nice to add",
       .send_headers = c(Accept = "application/vnd.github.symmetra-preview+json"))

Notice that the color has to be a hex code, minus the “#”, and also notice the .send_headers parameter. I’ve tested the above chunk for real and it worked. Depending on the current state of your repos, you might want to create labels from scratch after deleting labels. But updating labels is good because it means that if the labels are already used, the labelled issues won’t loose their labelling.

Now, enjoy labels by working on issues!

Admire the set of issues we can now use in each of our package repos:

Locke Data issue labels

Here’s for instance the datasauRus issue tracker whose pretty labels inspired Irene Steves.

datasauRus issue tracker

Now, it’s time for us to get cracking on some of our repo issue trackers! Have fun with your own labelling!

Edited to add: usethis::use_github_labels might be of interest, although it doesn’t update the names of existing labels. It is probably particularly useful for setup. Also follow this issue thread including a prototype of an issue label set by Mara Averick, with emojis.

SatRdays Cardiff

Wed, 04 Jul 2018 00:00:00 +0000

Hey again lovely readers! This blog is a very special one indeed, you get to hear about our great day out at SatRdays in Cardiff recently not once, not twice, but five times, from each of our team members perspectives! I think it’s fair to say that it was a very different experience for each of us - from seasoned conference attendees like Steph and Maëlle, Amy who had never presented before, sponsorship newbie Oz and then Ellen somewhere inbetween, we all had very different (but great) take aways from the day! Read on for laughs, nerves, proud moments and a quick cuddle too!

Ellen

Friday afternoon began with a hot and sticky drive, a 2 hour delay on the M6 and a rumbling tummy, but I made it to Cardiff just in time to catch a drink with a few of the Cardiff R user group! I was welcomed warmly by all, but most warmly of all by Steph’s dogs, Obi and Leia, with whom I was spending the night to make an early start on Saturday morning.

We arrived at the venue at Cardiff University so early that even the coffee shop wasn’t open yet - but whilst we were setting up the Locke Data stall, complete with life size chibi (head not to scale!) and freebies - some pastries and coffee arrived to start/save the day. The morning workshops got off to a great start, and I went and sat in on Steph’s Tidy Data Science workshop. I wanted to see Steph present in person for the first time, and I just knew there would be some great nuggets of info in there for me to take away. I felt like I was being let in on a naughty secret when she told us about the skimr and dataexplorer packages! Everybody should have those in their back pocket!

Lunch arrived and all too soon it was time for me to present. After watching Amy present and handle her questions like a practised pro I was up! This was my longest presentation to date, but I was looking forward to it and unusually for me, didn’t get any last minute jitters. I was so chuffed with how well received it was - a few people had seen me present the problem in Manchester a while back, and said how pleased they were that I had found a way around it. The questions I received were tricky but engaging and provided plenty of food for thought for future work - so maybe watch this space and see if I end up going along to SatRdays Amsterdam in September to present anything I come up with between now and then?

Ellen :)

Amy

Saturday began like any other day for me, other than the nerves of speaking at my first SatRdays Cardiff event! I woke up, got ready (kids included) and bid farewell to my family. Racked with nerves about doing my first talk, I strolled through Cardiff trying to enjoy the lovely weather. I arrived at the event to see some familiar and friendly faces from Team Locke, and some newer faces for people making the journey. The morning was very enjoyable, sitting at our booth while the workshops took place, allowing me some time to mentally prepare to give my afternoon talk.

As lunch time arrived, so did my family! Steph is very keen on making speaking easier and more accessible to women (especially us mothers)! We had a quick bite to eat, and the kids loved playing with life size Steph Chibi! Then it was time! We all strolled over to the rooms where talks were being held, I could feel the nerves building as we got closer. The room was quite intimidating for a first time speaker, looking up and seeing row after row of seats - hoping they wouldn’t be filled! After a few technical issues in getting the laptops and presentation all setup, while listening to chants from my 3 year old calling “Mummy, Mummy!” cheering me on with his dad. Just as I was about to begin my talk, my daughter thought this would be a perfect time to toddle on over for a quick cuddle. After I got the motherly duties out of the way, my talk began.

The room wasn’t overfilled however this didn’t stop me being nervous, I forgot my words a little and had to skip a slide. However once I got into the swing of it, things came a little more naturally (even though nerves were still on high). Overall it was a very nerve racking but enjoyable experience, and I got to talk about one of my favourite tools (Airtable), and received plenty of encouragement from my family and co-workers. A big thank you to Dave for asking questions, to try and prompt more information on the topic, but also for organising this amazing event (not forgetting the bottle of wine for helping with the organisation)!

P.S If you weren’t there on the day, I wrote a blog post on this topic too, which inspired my talk!

Amy :/

Steph

The Cardiff satRday was an incredibly proud moment for me on many fronts:

I’ve handed the reins of the Cardiff R UG over to Dave and Paul and they ran their first conference. They did this with panache and everyone I’ve talked to about it has gushed over the awesome job they did! They had over 75% of speakers coming from under-represented groups, they had a mentoring program for new speakers, and were able to make T&E available for many speakers – in short they aced it!
This was the first UK satRday event and we’re really starting to gain traction on these events with 8 more already in the pipeline. As I head up the central team on satRdays it’s incredibly gratifying to see folks delivering and participating in satRdays.
This was the first event Locke Data made a big commitment to sponsor. We invested a huge amount of time and money getting all the things made. Oz, in particular, did a vast amount of preparation and then stayed on the booth when we were all moving about doing talks and catching up with folks. Four of us spoke at the event and everyone did a great job – especially Amy who did her first ever talk! Seeing our core team in the flesh, sharing their knowledge and making the world a better place, really made me feel proud of what we’re growing here.

I’m so thankful that people have chosen to take up some of my banners and do things way better than I ever conceived was possible. Thank you all for making me feel on top of the world this week!

Steph :D

Maëlle

I’m so glad I could attend satRday Cardiff! It was my second satRday this year, since I got to keynote at satRday Cape Town in March. I came all the way from France, well, almost: I actually used the conference as an opportunity to visit family and friends in London with my husband and baby, and only travelled to Cardiff for the day. I was able to assess how welcoming the organisers were: my train tickets from London to Cardiff and back were reimbursed, and a lactation room was set at my disposal. Steph co-founded satRdays (and maintains a list of awesome organiser resources), and one feels her touch in such efforts.

Unsurprisingly, a big highlight of the conference was meeting my remote co-workers! I was able to pose for a photo with Steph…

Maëlle and Chibi Steph

Or was it her chibi? Hard to tell, honestly. In any case, it was great to spend time with Team Locke Data! I’m also thankful to have met other people I only interact with online in different spheres of the R community (R-Ladies, rOpenSci, #rstats Twitter). Everyone was as nice as online!

As regards talks, I gave one about rOpenSci onboarding system of packages, more info in the slidedeck. Nujcharee Haswell gave an enthusiastic and informative intro to tidytext just before my talk. Later, I attended the session where both Ellen and Amy were speakers!

Amy

Amy was a bit nervous, but even so, her passion for Airtable was obvious! I can only be impressed when thinking that she started learning R a few months ago, and is now able to explain to other R users how to leverage Airtable from R.

Ellen

Ellen’s talk included both really interesting info about energy consumption and its analysis and real talk about the research process that made me laugh (although it probably wasn’t very funny at the time!).

I live-tweeted all day long and made a storrry out of my tweets on my own blog. I went “home” to London very happy, and can only recommend attending a satRday near you if you can!

Maëlle :o) (why am I the only one with a nose?)

Oz

I’ve never run a sponsorship stall before, and needless to say I was very nervous, but the attendees at SatRdays made the experience a breeze and a delight. I got to chat with no end of lovely people, and spread the word about Locke Data, the community work Steph does, and even satRdays itself. A lot of work went into making all the banners, signs, and that cardboard cuttout chibi, but it was worth it to see so many people getting a laugh from them (and Amy’s kids running around with stickers and booklets!)

The event was a hit from where I was standing. Though small in scope, it was perfectly formed, and handled with a level of skill and organisation I’ve seen seasoned conference organisers fail to match. The food looked lovely, and the catering staff were very friendly and helpful. All in all it was a wonderful day, and I know Locke Data will be back again next time.

Oz ( ⋂‿⋂’)

And so there we have it! A day to remember for the whole team, for all the right reasons, The next SatRday is already in the diary in Amsterdam on September 1st. Have we inspired you to get involved? The Call for Speakers is open now!

A crystal clear book draw

Fri, 01 Jun 2018 00:00:00 +0000

As you might know, every month, a random Locke Data Twitter follower wins an excellent data science book! This month’s gift was “An Introduction to Statistical Learning: with Applications in R”, a classic and useful textbook. In this post I’ll give you some magick-al tips from behind-the-scenes of this month’s winner announcement. It’ll feature learning from my mistakes, and reading from a crystal ball… or more seriously, image manipulation in R!

An Introduction to Learning from my Mistakes

In this month’s instalment, I corrected two of my previous mistakes, here is what I learnt…

Package your code right away!

This book draw was the fourth one I was in charge of. The first three times I simply (or so I thought) put an R script and pictures in a GitHub repo, merely listing the dependencies in a DESCRIPTION file… how suboptimal! This month I transformed the repo into a package, moving the old scripts to a subfolder after naming them correctly in order to no longer have one called… “code.R”, and creating a function to draw the winner.

Now the repo looks good which is nice but I can’t but regret not having organized it right away. I’d have saved time and energy and my pride. Steph actually maintains a nifty package helping with proper project setup, pRojects, that’ll soon get some love and will probably be featured on this blog. Stay tuned!

Automate what can be automated

As I confessed last month, in the last announcement tweet the winner’s Twitter handle was different in the tweet text and the gif. Now it won’t happen ever again because I wrote an announce_winner function so that the worfklow now is

Draw the winner via twitterbookdraw::draw_winner()
Automatically create tweet text using that winner twitterbookdraw::announce_winner()
Automatically create visualization using that winner via twitterbookdraw::show_winner()

You can witness this in twitterbookdraw README.

Writing announce_winner was quite pleasing thanks to using the nice glue package by RStudio Jim Hester:

announce_winner <- function(winner, book, book_url){
  glue::glue('This month\'s winner is {winner$name} (@{winner$screen_name})! DM us to receive "{book}"!

             \n{book_url}

             \n#datascience')
}

An Introduction to Crystal Ball Reading

I try to vary the viz used each month to keep it interesting for Twitter readers but also for anyone interested in playing with the code. Now that the GitHub repo is tidier, finding previous month’s scripts for inspiration has gotten easier! But still, blogging a few tips is probably the best way to digest that information.

This month, the challenge was to create a crystall ball that’d progressively reveal the Twitter avatar of the winner, as if chibi Steph were reading it to announce the winner.

I prepared this gif using the magick package for image manipulation, developed by Jeroen Ooms at rOpenSci. It wraps the famous image manipulation library ImageMagick, allowing the user to use it in pipelines within R. Using magick is in my opinion a great geometry and logic training: what size and form should each piece have, which one should you put below/above the other etc. So on top of scripting your image manipulation to make it reproducible, you get to play!

Step 1: create a face-in-hole picture prop

I designed the gif skeleton as a face-in-hole photo prop, where the hole was the crystal ball.

june_background <- function(){
  # create a blue rectangle as background
  background <-  magick::image_blank(400, 300, '#2165B6')

  # activate it as a plot background
  img <- magick::image_draw(background)
  # add stars using base plot!
  points(runif(n = 42, min = 0, max = 400),
         runif(n = 42, min = 0, max = 300),
         cex = runif(42, min = 2, max = 4),
         col = "white",
         pch = 8)

  # add ball support using base plot as well
  rect(250, 250, 350, 150,
       border = "white", lwd = 5,
       col = "white")

  # add crystal ball, its background is a random color
  symbols(300, 150, circles = 50,
          fg = "white", inches = 1, add = TRUE,
          bg = "#B657A3", lwd = 2)

  dev.off()

  # get and resize wizard chibi
  chibi <- magick::image_read(system.file("assets/wizard_steph.png",
                                          package="twitterbookdraw")) %>%
    magick::image_resize("200x200")


  # add chibi on background
  img <- magick::image_composite(img, chibi, offset = "+10+50",
                                 operator = "Over")

  # make the crystal ball transparent by replacing its color
  # by transparency. the hole is done!
  img <- magick::image_transparent(img, "#B657A3")

  # return face-in-hole photo prop
  img
}

Face in hole photo prop

Step 2: add the winner’s face behind the hole and an evolving opacity veil

Now, I needed to get the winner’s face behind that photo prop! For this I first downloaded the winner’s avatar from Twitter, using the winner’s information previously drawn.

# winner <- twitterbookdraw::draw_winner()
winner_face <- magick::image_read(winner$profile_image_url) %>%
    magick::image_resize("150x150")

Then I wrote a helper function placing the winner’s face behind the photo prop and behind a veil of a given opacity.

june_colorized_frame <- function(opacity, winner_face, background){

  # add opacity veil in front of the winner's avatar
  winner_face <- winner_face %>%
    magick::image_colorize(opacity = opacity,
                           color = "black")

  # put the winner's avatar on a white rectangle
  # which will help put the avatar right behind the hole
  winner_face <- magick::image_blank(400, 300, "white") %>%
    magick::image_composite(winner_face, offset = "+220+80")

  # put the carefully placed and veiled winner's avatar
  # behind the photo prop
  magick::image_composite(winner_face, background)
}

After this, I created a sequence of evolving opacity (from very dark to lighter and to very dark again, before the big reveal of the avatar, because crystal balls blink, don’t they?).

show_june_winner <- function(winner, path = "june.gif"){
  # get winner's avatar
  winner_face <- magick::image_read(winner$profile_image_url) %>%
    magick::image_resize("150x150")

  # create a sequence of varying opacities
  # and for each opacity create a frame
  # then join and animate frames
  frames <- purrr::map(c(seq(from = 100, to = 50, by = -5),
                         seq(from = 50, to = 100, by = 5),
                         seq(from = 100, to = 50, by = -5),
               rep(0, 5)),
             june_colorized_frame,
             winner_face = winner_face,
             background = june_background())

  frames %>%
    magick::image_join() %>%
    magick::image_animate() %>%
    magick::image_write(path)

}

Imagine the winner were Amy, Locke Data’s operation manager (post written before the actual draw… Locke Data team members actually can’t win, we’d re-draw!):

winner <- rtweet::search_users("AmyMcDougall96", n = 1)
twitterbookdraw::show_june_winner(winner, path = "2018-06-01-winner.gif")

Winner gif

Not too shabby, good crystal ball reading skills Wizard Steph!

Future plans?

Next month, again a random Locke Data follower will win a great book: you should follow Locke Data on Twitter to get a chance! Besides, do not hesitate to ping us there if this post inspired you to play with magick

Python and Tidyverse

Fri, 01 Jun 2018 00:00:00 +0000

Introduction

One of the great things about the R world has been a collection of R packages called tidyverse that are easy for beginners to learn and provide a consistent data manipulation and visualisation space. The value of these tools has been so great that many of them have been ported to Python. That’s why we thought we should provide an introduction to tidyverse for Python blog post.

What is tidyverse?

Tidyverse is an opinionated collection of R packages designed for data science. All packages share an underlying design philosophy, grammar, and data structures. The core R tidyverse packages are: ggplot2, dplyr, tidyr, readr, purrr, tibble, stringr and forcats.

Python implementation of dplyr

The tidyverse package dplyr is a grammar of data manipulation, providing a consistent set of verbs that help you solve the most common data manipulation challenges. Here are some of the functions dplyr provides that are commonly used:

mutate() - adds new variables that are functions of existing variables
select() - picks variables based on their names.
filter() - picks cases based on their values.
summarise() - reduces multiple values down to a single summary.
arrange() - changes the ordering of the rows.

Dplython is a Python implementation of dplyr which can be installed using pip and the following command:

pip install dplython

Instructions on how to use pip to install python packages can be found here.

The Dplython README provides some clear examples of how the package can be used. Below is an summary of the common functions:

select() - used to get specific columns of the data-frame.
sift() - used to filter out rows based on the value of a variable in that row.
sample_n() and sample_frac() - used to provide a random sample of rows from the data-frame.
arrange() - used to sort results.
mutate() - used to create new columns based on existing columns.

For more functions and example code visit the Dplython README page.

At the bottom of the README a comparison is provided to pandas-ply which is another python implementation of dplyr.

Dplython comes with a sample data-set called ‘diamonds’. Here are some basic examples of how to use Dplython.

Import Python packages and the ‘diamonds’ data-frame:

import pandas
from dplython import (DplyFrame, X, diamonds, select, sift, sample_n,
    sample_frac, head, arrange, mutate, group_by, summarize, DelayFunction)

Create a new data-frame by selecting columns of the ‘diamonds’ data-frame:

diamondsSmall = diamonds >> select(X.carat, X.cut, X.price, X.color, X.clarity  , X.depth  , X.table)

Display the top 4 rows of the ‘diamondsSmall’ data-frame:

print(diamondsSmall >> head(4)) 

##    carat      cut  price color clarity  depth  table
## 0   0.23    Ideal    326     E     SI2   61.5   55.0
## 1   0.21  Premium    326     E     SI1   59.8   61.0
## 2   0.23     Good    327     E     VS1   56.9   65.0
## 3   0.29  Premium    334     I     VS2   62.4   58.0

Filter the data-frame for rows where the price is higher than 18,000 and the carat less than 1.2 and sort them by depth:

print((diamondsSmall >> sift(X.price > 18000, X.carat < 1.2) >> arrange(X.depth)))

##        carat        cut  price color clarity  depth  table
## 27455   1.14  Very Good  18112     D      IF   59.1   58.0
## 27457   1.07  Very Good  18114     D      IF   60.9   58.0
## 27530   1.07    Premium  18279     D      IF   60.9   58.0
## 27635   1.04  Very Good  18542     D      IF   61.3   56.0
## 27507   1.09  Very Good  18231     D      IF   61.7   58.0

Provide a random sample of 5 rows from the data-frame

print(diamondsSmall >> sample_n(5))

##        carat        cut  price color clarity  depth  table
## 320     0.71       Good   2801     F     VS2   57.8   60.0
## 9813    0.91    Premium   4670     H     VS1   61.8   54.0
## 11795   1.18  Very Good   5088     E     SI2   62.5   60.0
## 11845   0.95  Very Good   5101     D     SI1   63.7   55.0
## 11552   1.17      Ideal   5032     F     SI1   63.0   54.0

Add a column to the data-frame containing the rounded value of ‘carat’

print((diamondsSmall >> mutate(carat_bin=X.carat.round()) >>  sample_n(5)))

##        carat        cut  price color clarity  depth  table  carat_bin
## 11883   0.99  Very Good   5112     F     SI1   62.5   58.0        1.0
## 45123   0.77       Fair   1651     D     SI2   65.1   63.0        1.0
## 51630   0.31    Premium    544     E     SI1   59.2   60.0        0.0
## 49382   0.51  Very Good   2102     G      IF   62.6   56.0        1.0
## 18296   1.54  Very Good   7437     I     SI2   63.3   60.0        2.0

Python implementation of ggplot2

The tidyverse package ggplot2 is a system for declaratively creating graphics, based on The Grammar of Graphics. You provide the data, tell ggplot2 how to map variables to aesthetics, what graphical primitives to use, and it takes care of the details.

A Python port of ggplot2 has long been requested and there are now a few Python implementations of it; Plotnine is the one we will explore here. Plotting with a grammar is powerful, it makes custom (and otherwise complex) plots easy to think about and create, while the plots remain simple.

Plotnine can be installed using pip:

pip install plotnine

Plotnine splits plotting into three distinct parts which are data, aesthetics and layers. The data step adds the data to the graph, the aesthetics (aes) step adds visual attributes and the layers step creates the objects on a plot. Multiple aesthetics and layers functions can be added to a Plotnine graph.

If you are a python user used to Matplotlib it can take some getting used to a Grammar of Graphics plotting tool which is partly due to the difference in philosophy. Plotnine provides some tutorials to help with getting to grips with the package and there is also the Plotnine README. However if you are new to Grammar of Graphics plotting then this highly recommended kaggle notebook for Plotnine is probably the best place to start.

Here are some examples of how to use plotnine to visualize data from the ‘diamonds’ data-frame that comes with Dplython.

Import Python packages, the ‘diamonds’ data-frame and create a sample data-frame:

import warnings; warnings.filterwarnings("ignore") # hide Python warnings 
import pandas
import dplython as dplython
from plotnine import *
diamondsSample = dplython.diamonds >> dplython.sample_n(5000)

Create a scatter plot of ‘carat’ vs ‘price’:

print(ggplot(diamondsSample) # diamondsSample is the data  
 + aes('carat', 'price') # plot 'carat' vs 'price'
 + geom_point() # display the results as a scatter plot
 )

## <ggplot: (41012744)>

Add additional layers e.g. a line of best fit:

print(ggplot(diamondsSample)  
 + aes('carat', 'price') 
 + stat_smooth() # add a line of best fit
 + geom_point()) 

## <ggplot: (-9223372036813567705)>

Add another aesthetic, here the data is coloured by the ‘cut’ variable:

print(ggplot(diamondsSample)
 + aes('carat', 'price')
 + aes(color='cut') # colour the data by the variable cut and create a ledgend 
 + geom_point())

## <ggplot: (-9223372036816020904)>

Add a layer which separates the data into graphs based on ‘colour’

print(ggplot(diamondsSample)
 + aes('carat', 'price')
 + aes(color='cut')
 + facet_wrap('color') # seperate the data by 'colour' and graph seperately  
 + geom_point())

## <ggplot: (64014519)>

This article compares a variety of alternative plotting packages for Python.

Next steps

Read the documents that are linked in this blog post.
Learn the basics of Pandas.
Use Dplython and Plotnine to practice data manipulation & visualization. For example complete some of the exercises at kaggle.

Do you know of other good Python implementations of tidyverse? If so let us know about them!

Locke Data Blog

Gift ideas for the R lovers

Learning and using R

Becoming a legit package developer

Decorating one’s office

Getting out of one’s office

covrpage, more information on unit testing

What’s covrpage?

Green coverage badge for the rfm package

Tests summary report for the rfm package

Is the covrpage report only for users?

How do I publish the covrpage report?

Can I get a covrpage report for any package?

How do I learn more about covrpage?

What did covrpage gain from Locke Data’s help?

covrpage logo

Project planning with plotly

Plotly

Step by Step Gantt Chart

Read in your data

namer, Automatic Labelling of R Markdown Chunks

Why name your R Markdown chunks?

Sad chibi Stef being told by R that there is an error in unnamed-chunk-12

What does namer do?

Happy chibi Stef being told by R that there is an error in the chunk labelled test-12

Difference between two R Markdown reports before and after using namer to label chunks

How to use namer?

Future plans for even more nifty automatic naming

Packages for Testing your R Package

General package testing infrastructure

Create tests

Assess your tests

Test all the things

Mocking

Webmocking

Test plot outputs

Test Shiny apps

Test RStudio add-ins

Test htmlwidgets?

Test interactive behavior?

Conclusion

Package support offer

cransays - Follow your R Package Journey to CRANterbury with our Dashboard!

Why create the cransays dashboard?

Part of the content of CRAN incoming ftp server

How we made the cransays dashboard?

Creating a snapshot of CRAN incoming

Presenting the snapshot

cransays dashboard

Packaging it all together

Updating it regularly!

Sit back and enjoy, or improve the project?

Processing complicated package outputs

Requirements

Google Distance

Handling google_distance()

Tidyverse 'Starts_with' in M/Power Query

The Problem

The Target

The Solution

The Observations

The Caveat

The Conclusion

Speed Up With Microsoft

Multi-threaded R

Test Data

Benchmarking

Results

But how?

What about the out of memory part?

Would you like to know more?

Up your open source game with Hacktoberfest at Locke Data!

Functions and Packages

R packages

Installing packages

Installing from GitHub and other sources

Recommended packages

tidyverse

Getting data in and out of R

Data manipulation

What’s `covrpage`?

Is the `covrpage` report only for users?

How do I publish the `covrpage` report?

Can I get a `covrpage` report for any package?

How do I learn more about `covrpage`?

What did `covrpage` gain from Locke Data’s help?

What does `namer` do?

How to use `namer`?

Handling `google_distance()`

`JOIN`