13 Working with data sources
Do data play a role in solving the problem? Before a project can move forward, data must be both accessible and relevant to the problem. Consider what variables each data source contributes. While some data are publicly available, other data are privately owned and permission becomes a prerequisite. And to the extent data are unavailable, we may need to setup experiments to generate it ourselves. To be sure, obtaining the right data is usually a top challenge: sometimes the variable is unmeasured or not recorded.
13.1 Identifying accessible data
In cataloging the data, be specific. Identify where data are stored and in what form. Are data recorded on paper or electronically, such as in a database or on a website? Are the data structured — such as a CSV file — or unstructured, like comments on a twitter feed? Provenance is important (Moreau et al. 2008): how were the data recorded? By a human or by an instrument?
13.2 Public and private data sources
13.2.1 Example: NYC transportation data
To illustrate working with public data sources, consider two datasets we use throughout this text for analyzing urban mobility patterns in New York City.
Citi Bike System Data (Citi Bike NYC 2019)
The Citi Bike program in New York City provides publicly accessible system data through their open data portal at citibikenyc.com/system-data. This dataset contains real-time station status information showing bike and dock availability at each station. The bikeshare_nyc_raw201901.csv file includes variables such as:
- Station identifiers and locations (dock_id, dock_name, _lat, _long)
- Temporal information (date, hour, minute)
- Bike availability (avail_bikes, avail_docks, tot_docks)
- Service status (in_service, status_key)
These data enable analysis of usage patterns, station balancing, and temporal variations in demand. Because Citi Bike operates as a public-private partnership with NYC DOT, the data are released under open data policies that permit research and analysis. When working with such data, consider what is measured (bikes present at stations) versus what is not measured (actual trips taken, user demographics, failed rental attempts).
MTA Ridership Data (Metropolitan Transportation Authority 2022)
The Metropolitan Transportation Authority publishes ridership data across its various transit modes—subways, buses, commuter rail, and bridges/tunnels. The MTA_recent_ridership_data_20220127.csv file was used in a New York Times analysis examining how COVID-19 affected transit usage patterns in New York City. This dataset includes:
- Daily ridership counts by transit mode
- Comparison to pre-pandemic baselines (percentage of comparable pre-pandemic day)
- Separate tracking for subways, buses, LIRR, Metro-North, and Access-A-Ride
- Bridge and tunnel traffic data
The data reveal how different transportation modes recovered at different rates during the pandemic. Subway ridership, for instance, remained at roughly 48% of pre-pandemic levels in early 2022, while bridges and tunnels saw traffic exceeding pre-pandemic volumes (100.8%). Such data demonstrate the value of longitudinal comparison—without the pre-pandemic baseline, we could not assess the magnitude of change.
Both datasets exemplify publicly available government data that can inform policy decisions, operational improvements, and academic research. They also illustrate an important principle: the most relevant data for a problem often reside outside the organization experiencing the problem. Citi Bike’s operational challenges cannot be understood through trip data alone—weather, transit competition, and broader mobility patterns (captured in MTA data) all provide essential context.