#209: GitHub and Google on Public Datasets and BigQuery - Changelog

This week’s show was produced in collaboration with GitHub and Google to announce a big expansion to GitHub’s public dataset on BigQuery.

Download: MP3 Audio

We talked with Arfon Smith (GitHub), Felipe Hoffa (Google), and Will Curran (Google) about BigQuery, the big picture behind Google Cloud’s push to host public datasets, the collaboration between the two companies to expand GitHub’s public dataset, adding query capabilities that have never been possible before, example queries, and more!

Special thanks to Brandon Keepers for helping us put this show together!

Show sponsors

Toptal – Join the Toptal as an Engineer or Designer, or hire the best Engineers and Designers! Email Adam (adam@changelog.com) for a personal introduction to our friends at Toptal.

Linode – Our cloud server of choice! This is what we’re building our new CMS on. Use the code changelog20 to get 2 months free!

Full Stack Fest 2016 – Early Bird tickets available until July 15. Use the code THECHANGELOG after July 15 to save 75 EUR (before taxes).

Show notes and links

Arfon Smith on GitHub

Felipe Hoffa on Twitter

Will Curran on LinkedIn

#144: GitHub Archive and Changelog Nightly with Ilya Grigorik

GitHub announcement

Google Cloud Blog announcement

Google Open Source Blog announcement

Felipe Hoffa – GitHub on BigQuery: Analyze all the code

GitHub public dataset – This 3TB+ dataset comprises the largest released source of GitHub activity to date. It contains a full snapshot of the content of more than 2.8 million open source GitHub repositories including more than 145 million unique commits, over 2 billion different file paths, and the contents of the latest revision for 163 million files, all of which are searchable with regular expressions.

NOAA Global Surface Summary of the Day Weather Data

USA Name Data

Google BigQuery

Gist: BigQuery Examples from Arfon Smith

Shawn Pearce (Google) – The unsung hero at Google who did all the hard work getting the data pipeline working for this new dataset

Email bq-public-data@google.com to talk with Will and BigQuery’s public dataset team

Have comments? Send a tweet to @Changelog on Twitter.Subscribe to Changelog Weekly – our weekly email covering everything that hits our open source radar.