Schedule

Suggested Readings:

• Lessons from measurement [only need to read measurement section]
• When You Hear the Margin of Error Is Plus or Minus 3 Percent, Think 7 Instead

Suggested Readings:

• Pew - comparing weighting techniques
• High-frequency polling with non-representative data
• We Gave Four Good Pollsters the Same Raw Data. They Had Four Different Results.
• How One 19-Year-Old Illinois Man Is Distorting National Polling Averages

Optional Readings:

• Designing Informative Rating Systems: Evidence from an Online Labor Market
• Reputation Inflation
• Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?

Suggested Readings:

• Mining Massive Datasets, Ch 9, Recommendations
• Stitchfix Algorithms tour

Suggested Readings:

• The Engagement-Diversity Connection: Evidence from a Field Experiment on Spotify
• Algorithmic Effects on the Diversity of Consumption on Spotify

Suggested Readings:

• The Algorithm behind Plane Ticket Prices and How to Get the Best Deal
• Book chapter on Dynamic Programming

Notes:

• Handwritten notes backward induction 1
• Handwritten notes backward induction 2

Suggested Readings:

• Controlled experiments on the web: survey and practical guide
• A/B testing: A step-by-step guide in Python
• The Power of Experiments: Decision Making in a Data-Driven World
• Harvard Business Review on Experimentation

Suggested Readings:

• Lyft – experimentation in marketplaces (Part 1), (Part 2), (Part 3)
• Toward Causal Inference With Interference

Suggested Readings:

• Fairness as a Constraint on Profit Seeking: Entitlements in the Market

Title: Equitable Decision-Making in Public Resource Allocation

Abstract: Algorithmically guided decisions are becoming increasingly prevalent and, if left unchecked, can amplify pre-existing societal biases. In this talk, I audit the equity of decision-making in two public resource allocation settings. First, I present a methodological framework for online advertisers to determine a demographically equitable allocation of individuals being shown ads for SNAP (food stamp) benefits – specifically, considering budget-constrained trade-offs between ad conversions for English-speaking and Spanish-speaking SNAP applicants. Second, I discuss sensitivity analyses on public funding allocation algorithms such as CalEnviroScreen, an algorithm used to promote environmental justice by aiding disadvantaged census tracts – which we find to encode bias against tracts with high immigrant populations. In both case studies, we will discuss methods to mitigate allocative harm and to foster equitable outcomes using accountability mechanisms.

Required reading: CalEnviroScreen

Optional reading: SNAP,

Bio: Allison Koenecke is an Assistant Professor of Information Science at Cornell University. Her research on algorithmic fairness applies computational methods, such as machine learning and causal inference, to study societal inequities in domains from online services to public health. Koenecke is regularly quoted as an expert on disparities in automated speech-to-text systems. She previously held a postdoctoral researcher role at Microsoft Research and received her PhD from Stanford’s Institute for Computational and Mathematical Engineering. Awards won include the NSF Graduate Research Fellowship and Forbes 30 under 30 in Science.

Suggested Readings:

• Switchback Tests and Randomized Experimentation Under Network Effects at DoorDash
• A/B testing at marketplaces
• Universal Holdout Groups at Disney Streaming
• Large scale experimentation – Stitch Fix Technology

Suggested Readings:

• Synthetic control book chapter
• Universal Holdout Groups at Disney Streaming
• Large scale experimentation – Stitch Fix Technology

Title: On human and algorithmic decisions in healthcare

Abstract:

Machine learning systems in healthcare often fall short of their promises. I argue that a central reason is our inability to precisely characterize how humans generate clinical data and how models behave in the real world. Without such behavioral models, we can misinterpret both human and algorithmic decision-making, making it difficult to reason about how to introduce predictive systems into patient care.

In this talk, I will introduce two methods to reason about human and algorithmic decision-making in healthcare. The first studies how human behavior shapes healthcare data, focusing on underdiagnosis – a pervasive source of corrupted labels in which diseases remain undocumented. I introduce a method to quantify underdiagnosis, revealing systematic gaps in recorded diagnoses and enabling new ways to study how social factors distort observed data. The second addresses the challenge of evaluating algorithms with limited labeled data. Rather than treating algorithmic disagreements as flaws, I show how they can be leveraged to estimate real-world performance efficiently, reducing dependence on labeled data and expanding the contexts in which we can reliably evaluate models. Together, these works illustrate a simple principle: through new methods to carefully characterize the data and models at hand, we can build systems appropriate for deployment in high-stakes domains.

Suggested Readings:

• Algorithmic Foundations of Differential Privacy
• How to perform these attacks Part 2
• Simple Demographics Often Identify People Uniquely
• Differential privacy: A survey of results

Suggested Readings:

• Limits to Prediction class, Princeton, (Pre-class summary reading)
• Measuring the predictability of life outcomes with a scientific mass collaboration
• Manipulation of Social Program Eligibility