index.knit

My name is Scott Spencer. I’ll be your teacher, one of your guides, in learning Storytelling with data this semester.

When I’m not teaching, I provide data science consulting on projects that I believe are good for us all, like using analysis to change people’s minds about mitigating adverse climate change. I also consult in professional sports, and am working on a super-secret project to simulate entire baseball games — don’t tell anyone!

To perform these analyses, I typically use Bayesian, and ideally generative, modeling implemented in Stan, which is a probabilistic programming language. Because I think I can be helpful, I give back to the Stan community by helping to develop interfaces, answer questions, squash bugs, and so forth. In the past, I have also provided data science analyses in engineering and legal fields.

As our course material makes some of my projects relevant, I will try to bring those into the discussion and share them with you. In fact, I’ll briefly do a little of that tonight.

🗓 Click to schedule an appointment

First off, congrats on finding our classroom! We’ll continue to meet on Tuesdays from 6:10 to 8pm, as we’re doing tonight. We’ll see how things play out for meeting in-person on campus, either during our scheduled time and / or when spring comes and we have grass and warmer weather, maybe we can meet together on the lawn for a coffee chat or something.

Along with holding a weekly discussion or lecture, I will be available to schedule office hours. It’s pretty easy, just click the links here or on our Course website.

I’ll show you where in the menu bar now. The link will open our calendars and you can schedule whatever open slot works for you.

Let me click the links and show you.

[DEMONSTRATION]

Of course, we’re first going to encourage you to discuss your question and help to answer other student’s questions on our online discussion forum. Posting your question there will ensure you get the quickest response.

In fact, to help encourage you all to make it a habit in this class to contribute with your questions and answers to this online discussion, that will be part of your grade. The discussion tracks whether you ask or answer questions, and how many. Each week, part of your participation grade will reflect whether you’ve done that. You’ll get credit for at least minimally engaging here, but I encourage you to go beyond the minimum because a big part of a class here at Columbia is learning from each other.

Speaking of which, you are all off to a great start by introducing yourselves in the discussion! Let’s remember to tag our questions and discussions to help everyone find them again later. For questions and discussions for this lecture, then, tag them under lecture 01. I’ll show you now. Make sense?

I’ll go over the details of how we’ll grade in a few minutes.

The most fundamental idea is not specific to analytics: it applies to all communications we’re interested in here. And, by the way, I’m pulling this idea from a chapter from Jean-Luc Doumont’s book that I’ll have you read in a few weeks, which I’ve cited here.

Communications have a goal. A purpose. The components of whether our communication succeeds in its goal is.

Get our audiences to pay attention to, understand, and be able to act upon a maximum of messages, given constraints.

This single concept, what I’m also showing you on this word diagram, is so fundamental, you’ve going to grow tired of hearing me repeat it throughout this course. But I hope that by the end of this course, you will automatically begin to think about how each of your communications meet each of the components I’m showing you here.

Along with guiding and measuring our communications by this, we’ll also need to apply a range of skills. I’ve categorized a few of what various industry experts most commonly identify. Let’s take a look.

If you think about what we just considered from Doumont, I think you’ll see that those goals in communication show up here. First off, our typical goal is to drive change in some way.

We see that right at the intersection of this Venn diagram I’m showing you here — our first visual — to help us think about this course. We’re in an applied analytics graduate program. So, perhaps obviously, our class should relate to that. And we represent data analyses in the first, or top, circle.

But there are many more useful skills in applied analytics. These include skills around writing narratives and creating visuals, and then ultimately bringing all three into a single communication with the goal to enabling change.

To help us think about these three overriding aspects, let’s relate them to quotes from a few characters.

To help give personality to each of these areas, I’ve added a quote to each. To start, we’ll have Sherlock Holmes, the quote in the upper right here, representing data analyses, who’s character shouted,

“Data! Data! Data!”, “I can’t make bricks without clay.”

Data was essential to assist the detective uncover insights to the mysteries he investigated. And it will be to you, too!

We’ll also study ideas from Daniel Kahneman later in the course. He explains, and I quote him in the lower left:

“No one ever made a decision because of a number. They need a story.”

That’s the “storytelling” part of our course title. And when we think deeply about narrative and writing, our communications about data analyses will improve dramatically. It’s common that students focus so hard on learning to use software tools that they lose site of this. Don’t lose site of this very important aspect.

Speaking of visuals, both narrative and data analyses are complimented well through the use of visuals in various forms. And the best part of visuals is captured well in a quote from John Tukey — he is a famous mathematician — , and he says,

“The greatest value of a picture is when it forces us to notice what we never expected to see.”

As we learn more about creating graphics, and we will start this course sort of getting you up to speed on creating visuals so that we can use them in communications. As we gain experience with visuals, we’ll come to appreciate what John Tukey means.

Our aim in this course, as I mentioned, is to bring each of these ideas together in an effort to enable decisions and change minds. It is to make an impact. To make an impact with various audiences, which we’ll come to learn about, and practice communicating with.

So let’s look at what kinds of deliverables we’ll use to give us practice with these three aspects of communication.

I’m representing all your assignments here as a graphic. The the pink represents individual assignments; the blue indicates group work. The weighting of these assignments are 40/50 individual and group. Within individual assignments, each one is 10 percent of your grade, and participation overall is 10 percent.

As I mentioned, the four homeworks are for practice, and will be focused on particular skills.

The first and second assignments give you practice with creating basic data visuals.

The third assignment shifts to communication. You’ll write a memo about an analytics project. Now this memo really bridges the work between individual and group activities.

What I mean is each of your group members will have written a memo about an analytics project they might take on for an organization, and then the group can decide to tackle that project together, or the group may decide to do something entirely new and different from any of the memos. But the idea is that the individual memos helps you bring ideas to your group to get started.

Now this is a small class. So I’m considering the best approach for group work this summer.

Now the last, the fourth, individual assignment, gives you more practice in creating explanatory visuals, so thinking about how visuals help us communicate with others.

I’ll give you more details as we move forward in the next few weeks. Does that make sense?

Now another part of how we’ll learn is by working on a case study all together, as a class. Let’s turn to that now.

In the real world, we’re usually confronted with a problem or opportunity, and then we try to gain insight into it in some way that enables decision making. We can use questions as tools to guide how we approach any given project. I’ve listed a few important questions here.

Let’s read these together:

What problem is to be solved? Is the problem important? Be specific. Could an answer have impact? How? Do data have a role in solving the problem? Are the right data available? If so, where? If not, can we generate it somehow, like with an experiment? In what contexts may the data be generated? Finally, is the organization ready to tackle the problem and take actions from insights?

The process is, in the real world, and in this class, an iterative process of matching data and analysis to possible goals and actions. Does that make sense?

Ok, so let’s go through a short example together.

From reading your hellos on discussion, quite a few of you are living in NYC and have some experiment living here.

Has anyone used Citi Bike, the New York bike sharing program? Cool.

For those of you that haven’t had an opportunity in New York, have you tried renting bikes from bike shares elsewhere?

[SHARE SOME EXPERIENCES]

The New York Citi Bike bike share originated in 2013. And imbalances at these docking stations have been an issue since there have been stations and bikes. Three years after it was setup, and continuing to grow, the Citi Bike spokeswoman explained that, in fact, I’m quoting her here: Let’s read together.

“Rebalancing is one of the biggest challenges of any bike share system, especially in New York where residents don’t all work a traditional 9-5 schedule, and people work in a variety of other neighborhoods.”

For those of you who are not familiar with New York City yet, there are lots of neighborhoods within it to explore, and before the pandemic, work and lifestyle commute patterns were complex, and on top of that, the City plays host to probably more tourists than any other city in the United States, at least. Those tourists also use the bike shares.

So what might the natural flow of bikes depend on? Ideas?

[DISCUSSION]

Since the pandemic, as is the case in other major cities, work and lifestyle patterns have changed, and continues change, dramatically. Right?

Would someone familiar with NYC help give us a sense in how our lives have changed?

The ideas in social distancing can certainly help us stay healthy. But as people return to the city, and continue to socially distance, this places an even greater burden on commuting within the city. In this New York Times article this year, which we’ll look more closely at in a later discussion, and in fact in your first homework, the author explains how biking and bike shares are becoming even more critical for this city.

With that context, how might we begin to help Citi Bike with rebalancing today? How might we approach this?

To start our conversation, would it make sense to first consider what Citi Bike does now?

From the cited article, I’m showing you four of the ways that Citi Bike has attempted to rebalance the bike share system. Anyone heard of Citi Bike’s Bike Angels program? You can join it, and you get biking credits by riding bikes from full stations to empty ones, woohoo!

They also use electric bikes and vans to move bikes from full stations to empty ones, and even in some locations with high traffic the use valet service that can help you.

Even with all this, rebalancing, as we saw a few minutes ago, remains a major problem. Last spring, teaching in-person near Columbia, the only time I was able to rent a bike, either to go to campus from my UWS apartment or to leave campus, was when the weather was quite cold. In warmer days, there were never any bikes!

So we will begin to explore this rebalancing issue together as an applied way to learn how to accomplish our communication goals in this course. And this leads us to our next in-class collective exercise.

I’m going to suggest that before thinking about data, start by thinking about the real world. How we experience it. Try to identify events and user behaviors associated with the NYC bike share.

What events may be correlated with, or cause, empty or full docking stations? Ideas?

For riders, are there certain behaviors or preferences that lead to the problem?

And, by the way, these images I’m showing you aren’t just for decoration. I never show you anything that I do not believe gives you information. And we’ll talk about that idea in this course, too. So these images, I’ve picked them to give you hints or ideas.

What analogous or related things can we draw comparisons from that give us more context?

Now that we’ve identified some possible events and behaviors, what information, what data, might we find available that relates to those. What measurements may be recorded? How can we find out? Might these data lead to insights that help inform decisions or goals? How will we find them? Google searching is probably a solid first approach.

Let’s first just read two definitions together. First, datum, which is the single form of data.

“A datum is an abstraction of a real-world entity (person, object, or event). The terms variable, feature, and attribute are often used interchangeably to denote an individual abstraction.”

Now we can probably guess what a data set is. Let’s read this definition.

“It consists of the data relating to a collection of entities, with each entity described in terms of a set of attributes. In its most basic form, a data set is organized in an n by m data matrix called the analytics record, where n is the number of entities (rows) and m is the number of attributes (columns).”

Hang with me a moment, I want us to consider just a few more definitions to differentiate types of variables, then we’ll start to apply them to our example.

So, it may seem obvious, data may be of different types. These types include nominal, ordinal, and numeric, and the numeric type may be on two scales.

First nominal. Nominal is another way of saying we’re giving attributes names, like apple or banana. Notice that it is not clear how we would order apples or bananas, is one greater than the other?

When we can naturally order them, we call that type ordinal.

Then, we have numeric types. Numeric types can be represented using integer or real values. Within these types, we can think of some on an interval scale, others on a ratio scale.

An easy example for understanding the difference in the two scale types is temperature. More specifically, consider the difference between Celsius and Kelvin.

We cannot say that 20 degrees Celsius is double or twice that of 10 degrees Celsius, right?

But we can make such a comparison with Kelvin. That’s because Kelvin is on a absolute scale that begins with 0. We can say that 20 Kelvin is double 10 Kelvin. Does that difference make sense?

Data and information may also be found as structured or unstructured. Let’s consider this next.

Bike and dock: data are recorded of each bike unlocked and docked, along with remaining dock capacities at the locations, dates, and times of each event: https://www.citibikenyc.com/system-data

Taxi pickup and drop-off locations and times: https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page

Subway lines entrance/exit locations: https://data.cityofnewyork.us/Transportation/Subway-Stations/arq3-7z49

MTA usage data (and change since coronavirus): https://new.mta.info/coronavirus/ridership

Historical weather: National Weather Service. API Web Service. https://www.weather.gov/documentation/services-web-api; Yip, Stan. “Weatherr: Tools for Handling and Scraping Instant Weather Forecast Feeds.” Manual, 2020. https://CRAN.R-project.org/package=weatherr.

Geography (elevation): USGS. Elevation Point Query Service. https://ned.usgs.gov/epqs/; Hollister, Jeffrey, et al. “Elevatr: Access Elevation Data from Various APIs.” Manual, 2021. https://doi.org/10.5281/zenodo.5119662.

Traffic data and more: http://www.nyc.gov/html/dot/html/about/datafeeds.shtml#realtime

Twitter: https://twitter.com/CitiBikeNYC; Gentry, Jeff. “TwitteR: R Based Twitter Client.” Manual, 2015. https://CRAN.R-project.org/package=twitteR.

I’ve collected a few sources of data for us to explore that may be related to Citi Bike. And I encourage you to explore these links to data after our class. We’ll look at one of these together in class and you’ll get to look at another for your first homework.

First, the City of New York has made public its data recording every time a bike is unlocked and used. Second, I’ve included a link to Taxi usage in the city.

Similarly, I’ve included data on subway and MTA, that’s the Metra North Trainlines, usage.

Along with alternative transportation information, I’ve included a link to weather and traffic.

Finally, you’ll notice I’ve included land elevation data. Why might that relate to rebalancing?

So we’ve found some data. What do we do with it? Let’s spend a few minutes talking about components of a process for data analysis.

And we can think of data exploration and analysis as a workflow. We import the data into our software analysis tools. We tidy and transform it. Meaning, we decide how to handle miscoded or missing data, we create new variables and summaries from what we’ve collected, we visualize it in its various ways, and model it.

Typically we repeat this process iteratively many times until we understand what insights the information provides. Then, we consider how to communicate those insights to whomever needs it: our audiences.

None of these steps are trivial. And as a diagram for workflow, this one is sort of very general and basic.

Now you’ll notice the entire diagram is labelled as program. Ideally, we want our entire workflow to be reproducible, for us and others, and coding is currently the best approach to accomplish these goals.

And two popular software choices for data analyses are R and Python, though their histories differ. R was developed specifically for data analysis, and has been evolving ever since.

Python was developed as a general programming language for teaching, and it too has been evolving ever since. Both languages are extended in their functionality through language libraries that others develop. I’ve created R packages for data visualization, for example.

I’ll be demonstrating ideas in this class using R for a few reasons. But tonight I’m also going to enable you to mimic the same materials I present in Python, if you run into that need. We will start a bit abstractly at first, and then apply these tools to our new Citi Bike case study.

[POLL]

Before we discuss them, let’s take a quick poll to see how many of you have used R before? How many of you have used Python before? How many have used both? How many of you have not used either?

Thank you. This is helpful to know. Now this course is not intended as a learn to program course in either language, but we will need to use software to create our visualizations for communication. So I’ll show you how we set things up to do just that, and show you the best software for its purposes along the way.

So in R, we can load libraries by calling a function called library and give it the library package name as the parameter.

We can load this into an R session as I have generically coded it here on the left.

Similarly, people have written libraries in Python and two that mimic the functionality I’ll be showing you from the R’s tidyverse in Python are called datar and plotnine. These Python libraries were ported from the R tidyverse packages for a reason.

I’ll later explain some of the things in all these libraries. But to access any library installed on your computer, you simply type them in like I’ve shown for the correct language environment.

Later, when we do this on the examples, we’ll actually run this code using RStudio. So we’ll switch back and forth between these visuals and coding it in my RStudio IDE.

We manipulate these data frames using functions in either language.

Before getting into any specific function, let’s review the basics of how functions work. And I’m showing a couple of general ways they are used in both languages, in a way that will become more useful soon.

Basically, we use a function by coding its name, and giving it anything it needs to do its job. The things it needs are called parameters. In the gray boxes, I’m showing you generic versions in both languages, for your reference, but let’s look at the colored example.

In the colored example here, the name of the function is “roll2”, and the thing it needs, the parameter, is called “bones”. The function gives you back, or returns, the sum of two die that are rolled. Now, our main course examples will not be about gambling, but this is a simple example for us to just examine a function.

Let’s look at a list of functions that we will commonly use for data analysis.

As I mentioned, we sometimes want to transform our data in several ways. And these functions are common things we do.

Finally, once we have the data frame in R, we usually need to modify it in various ways before we can visualize it or gain insights. So here, I’m just listing some important functions for manipulating data and showing you that the names pretty much map straight into Python’s libraries, too. And these functions are named, using the English language, as verbs telling you what they do. So think of them as English verbs too. Use the function “select” to select a variable. Use the function “rename” to rename a variable in your data frame. Use the function “filter” to filter rows out of your data frame.

Now a couple I’ve listed here take a little more practice to use. Those are pivot_longer and pivot_wider. These two important functions reshape your data.frame. And we’ll talk more about this. Are part of preparing data for analysis and communications, you’ll become very familiar with these functions as they are common whatever tool you use in the future.

Now, we can execute or run a single function and store the results in an object. But frequently we want to apply several functions to our data in one or more data frames. Let’s consider how we approach that.

In R, we can use something we call the pipe operator, which we code as the percent sign, followed by the greater than sign, and another percent sign. Here’s what it does. The pipe operator takes the thing on it’s left, typically your data frame, and gives it to the first parameter of the function to the right. And the first parameter of all the functions I just showed you expect you to give it a data frame. N ow Python generally works a bit differently and until some authors created Python libraries for this, the closest idea was to chain methods together. We won’t cover method chaining here, but you can google it to see how it works. I also have a short paper demonstrating python method chaining on my website if you’re interested.

The package I’m showing you here, datar, does let you mimic almost exactly what we will be doing using R with tidyverse. And I’m generically showing you how this looks in both languages.

So I’m hoping that as we develop software coding skills in this course using one language, R and its packages, that you’ll be able to look back on this information, whenever you need to, and be able to translate your R code to Python. I’m hoping you see that if you use these Python packages, it has a few style quirks but is otherwise almost identical.

Questions so far?

Now that we have our libraries loaded, we need to read in our csv file. Let’s look at that code now. As we review the code and import the data, we’ll also look at a few rows of the data as examples of the definitions we just discussed.

Is this data structured or unstructured and what makes it so?

Can we identify any nominal data in this data frame? Do you see any variables you might consider ordinal? Do you see examples of numeric data?

[DEMO]

Now since we’re thinking about rebalancing, let’s keep that in mind as we review the variables in these data. I do not see anything that indicates whether a bike was rebalanced. Do you?

But let’s think a little harder about what that means.

We do see variables that may help us. These data, each observation, represents a single bike trip. It includes a bike id. A start station id. An end station id. And a trip start time. Now these data represent bike trips, not removal for rebalancing or repairs. That means, normally, when someone takes a bike and then parks it, and then someone else takes the bike, the first ride’s end station should match the next ride’s start station, right? Does that make sense?

Cool. So I think we can try to make our own rebalanced variable using these data we have. But we will need to transform these data. To do that, we’ll go through essentially the same logic that we just discussed. Let’s see this in action using code.