The downstream consequences of platform design and market entry on consumers and firms

PDF

Download

Open document

Flip pages

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content The Downstream Consequences of Platform Design and Market Entry on
Consumers and Firms
by
Poet Larsen
A Dissertation Presented to the
FACULTY OF THE GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF PHILOSOPHY
(BUSINESS ADMINISTRATION)
May 2025
Copyright 2025 Poet Larsen

Acknowledgments
To Davide, thank you for all of your guidance and patience. I would not be the researcher
and adult I am today without your supervision.
To my committee members—Dina Mayzlin, Lan Luo, Nikhil Malik, and Odilon Camara—
thank you for your time, insight, and encouragement. To Paul, Sean, Srinivas, and Srini,
thank you for pulling me into this profession. To my labmates, thank you for all the good
times.
Thank you to my family for supporting and encouraging me throughout all these years.
Finally, to the USC men’s rowing team—thank you for being a constant I could always rely
on. I am a better researcher and person because of crew.
ii

Table of Contents
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Chapter 1:
Information Signals in Sponsored Search: Evidence from Google’s BERT 1
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3 Empirical Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.3.1 Sponsored Search Advertising . . . . . . . . . . . . . . . . . . . . . . 9
1.3.2 BERT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.4 Modeling the Market . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.4.1 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.4.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.5 Empirical Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
1.5.1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
1.5.2 Empirical Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
1.5.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
1.5.4 Robustness Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
1.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Chapter 2:
The Financial Consequences of Legalized Sports Gambling . . . . . . . . . 44
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.2 Literature: Sports Gambling and Financial Health . . . . . . . . . . . . . . . 47
2.3 Background and Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
2.3.1 Background on Legal Gambling . . . . . . . . . . . . . . . . . . . . . 49
2.3.2 Consumer Credit Data . . . . . . . . . . . . . . . . . . . . . . . . . . 50
2.3.3 Types of Gambling Access . . . . . . . . . . . . . . . . . . . . . . . . 51
2.3.4 Primary Outcomes of Interest . . . . . . . . . . . . . . . . . . . . . . 52
2.4 Empirical Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.4.1 Identification Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
iii

2.5 Aggregate Effects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
2.5.1 Overall Consumers’ Financial Health . . . . . . . . . . . . . . . . . . 56
2.5.2 Indicators of Excessive Debt . . . . . . . . . . . . . . . . . . . . . . . 57
2.5.3 Overall ATTs and Summary of Results . . . . . . . . . . . . . . . . . 60
2.5.4 Robustness Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
2.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Appendix A:Information Signals in Sponsored Search: Evidence from Google’s
BERT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
A.1 Theory Model Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
A.1.1 Full Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
A.1.2 No Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
A.1.3 Partial Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
A.1.4 Region Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
A.2 Sampling Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
A.3 Alternative Identification Strategy . . . . . . . . . . . . . . . . . . . . . . . . 114
A.3.1 Defining Query Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . 114
A.3.2 Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
A.3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
A.3.4 Robustness Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
Appendix B:The Financial Consequences of Legalized Sports Gambling . . 131
B.1 Treatment Type and Handles . . . . . . . . . . . . . . . . . . . . . . . . . . 131
iv

List of Tables
1.1 Summary statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
1.2 The effect of BERT on Has Competition for non-competitive queries. . . . . 31
1.3 The effect of BERT on log(CS) by query length. . . . . . . . . . . . . . . . . 32
1.4 The effect of BERT on log(CP C) by query length. . . . . . . . . . . . . . . 34
1.5 The effect of BERT on log(CS) by query length. . . . . . . . . . . . . . . . . 35
1.6 The effect of BERT on log(CP C) by query length. . . . . . . . . . . . . . . 36
1.7 Changes in organic results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
1.8 Changes in log(CS) by query length, controlling for organic rank changes. . 38
1.9 Changes in log(CP C) by query length, controlling for organic rank changes. 39
1.10 The effect of BERT on log(CS) by query length controlling for search interest
and search volume. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
1.11 The effect of BERT on log(CP C) by query length controlling for search interest and search volume. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.1 Pre-treatment summary statistics. . . . . . . . . . . . . . . . . . . . . . . . . 54
2.2 Fiscal policies of treated and control states. . . . . . . . . . . . . . . . . . . 56
2.3 Hazard model test of treatment likelihood with all states. . . . . . . . . . . . 65
2.4 Overall ATT estimates. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
2.5 Overall ATT estimates excluding iGaming States. . . . . . . . . . . . . . . . 67
v

2.6 General Access: 2015-2017 Observable Differences Before/After Matching. . 67
2.7 Online Access: 2015-2017 Observable Differences Before/After Matching. . . 68
2.8 Observable Differences Between All Counties Just Before Treatment. . . . . 68
2.9 Observable Differences Between Matched Counties Just Before Treatment. . 69
2.10 Overall ATT estimates with matched counties. . . . . . . . . . . . . . . . . . 69
A.1 Average Log(Cosine+1). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
A.2 Average Linguistic Complexity. . . . . . . . . . . . . . . . . . . . . . . . . . 118
A.3 Alternative Strategy: log(CS) . . . . . . . . . . . . . . . . . . . . . . . . . . 120
A.4 Predicted log(CS) estimates by model specification. . . . . . . . . . . . . . . 121
A.5 Alternative Identification: log(CP C) . . . . . . . . . . . . . . . . . . . . . . 121
A.6 Predicted log(CP C) estimates by model specification. . . . . . . . . . . . . . 122
A.7 Event Study: log(CS). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
A.8 Event Study: log(CP C). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
A.9 Placebo Test: log(CS). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
A.10 Placebo test: log(CP C). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
A.11 Controlling for organic rank changes: log(CS) . . . . . . . . . . . . . . . . . 127
A.12 Controlling for organic rank changes: log(CP C) . . . . . . . . . . . . . . . . 128
A.13 Controlling for search interests and search volume: log(CS) . . . . . . . . . 129
A.14 Controlling for search interests and search volume: log(CP C) . . . . . . . . 130
B.1 Treatment start dates in our dataset. . . . . . . . . . . . . . . . . . . . . . . 132
B.2 Average handle by channel per state. The data does not include handles for
states where tribal lands run the offline sports gambling market. . . . . . . . 133
vi

List of Figures
1.1 Regions that determine how auction evolves with changes to a. C is the
cost the platform incurs for showing a topically irrelevant ad and Bt
is the
importance of topic alignment for a query’s CTR. . . . . . . . . . . . . . . . 19
1.2 Potential market changes in Region 1. . . . . . . . . . . . . . . . . . . . . . . 22
1.3 The effect of BERT on Has Competition for non-competitive queries. Error
bars represent 95% confidence intervals. . . . . . . . . . . . . . . . . . . . . . 30
1.4 Percent change in Competition Score by query length. Error bars represent
95% confidence intervals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
1.5 Percent change in CPC by query length. Error bars represent 95% confidence
intervals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.1 Monthly sports handle in billions and the number of legalized states. The left
axis is the sports handle, and the right axis is the number of legalized states.
Our data does not contain handles for states where tribal lands run the offline
sports gambling market. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
2.2 The effect of sports gambling legalization on consumer credit score. . . . . . 57
2.3 Changes in bankruptcies, collection on account, credit card delinquency likelihood, auto loan delinquency likelihood, and debt consolidation usage. . . . 59
A.1 Regions that determine how auction evolves with changes to a. C is the
cost the platform incurs for showing a topically irrelevant ad and Bt
is the
importance of topic alignment for a query’s CTR. . . . . . . . . . . . . . . . 99
A.2 Region 2: Threshold functions for γq and γt that impact whether prices can
increase or decrease. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
vii

A.3 Region 3: Threshold functions for γq and γt that impact whether prices can
increase or decrease. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
A.4 Region 4: Threshold functions for γq and γt that impact whether prices can
increase or decrease. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
viii

Abstract
In this thesis, we study the downstream consequences of digital platform design and market
entry on consumers and firms. We utilize various methods, such as causal inference, machine
learning, econometrics, and analytical economic modeling to generate novel insights that
speak to firms, consumers, and policymakers.
Digital platforms, such as search engines, social media platforms, websites, and mobile
apps, face many design decisions that directly impact the consumers and businesses they
interact with. In this stream of research, we study how platform design changes affect
businesses, particularly advertisers, in the context of sponsored search.
In the first chapter, we study how improvements to search engine interpretation algorithms and the information signals they generate affect sponsored search markets. We focus
on two outcomes: the number of advertisers bidding for a query (i.e., competition) and
cost-per-click (CPC). We start by developing a theoretical auction model. We find that as
the quality of a search engine’s interpretation algorithm improves, the number of bidders
allocated to auctions generally increases for all queries by improving the platform’s ability
to identify relevant advertisers more often. Despite this, prices may decline in some markets.
Specifically, we find that shifts in CPC depend on the prevalence of context in a query.
For queries lacking context (e.g., shorter queries), prices generally increase. However, for
queries with more contextual information (e.g., longer queries), prices may decrease. This
can occur when a platform’s new algorithm significantly improves contextual interpretation capabilities, leading to more precise advertiser relevancy score estimates and therefore
weaker competition among bidders. We then test the model predictions using a monthly
dataset of competition scores and CPC for 12,000 queries, leveraging Google’s October 2019
rollout of Bidirectional Encoder Representations from Transformers (BERT) as a natural experiment. Employing two Difference-in-Differences identification strategies, we find results
ix

consistent with the theoretical model. Our results offer insight into the economic impact
of AI and Large Language Models on advertising markets and help advertisers prepare for
future algorithm updates.
In the second chapter of this thesis, we study how the development of new digital platforms can impact consumers and focus in on the context of sports gambling.
Following a 2018 ruling of the U.S. Supreme Court, 38 states have legalized sports gambling. We study how this policy has impacted consumer financial health using a large and
comprehensive dataset on consumer financial outcomes. We use data from the University
of California Consumer Credit Panel, containing credit rating agency data for a representative sample of roughly 7 million U.S. consumers. We exploit the staggered rollout of legal
sports betting across U.S. states and evaluate two treatment effects: the presence of any
legal sports betting in a state and the specific presence of online or mobile access to betting. Our main finding is that overall consumers’ financial health is modestly deteriorating
as the average credit score in states with legalized sports gambling decreases by roughly
0.8 points. When states introduce access to online sports gambling, average credit scores
decline by nearly three times as much (2.75 point decline). The decline in credit score is
associated with changes in indicators of excessive debt. We find a substantial increase in
average bankruptcy rates, debt sent to collections, use of debt consolidation loans, and auto
loan delinquencies. Together, these results indicate that the ease of access to sports gambling
is harming consumer financial health by increasing their level of debt.
x

Chapter 1
Information Signals in Sponsored Search:
Evidence from Google’s BERT
1.1 Introduction
Sponsored search continues to be a dominant advertising channel for firms to reach consumers. It also remains the primary source of revenue for many search engines. In 2023
alone, US search advertising revenue reached nearly $90 billion.1
When a consumer submits a search query, a search engine faces the fundamental problem
of interpreting the query to generate a response. Search engines such as Google and Bing
increasingly rely on Natural Language Processing (NLP) algorithms to interpret consumer
search queries and generate search intent signals. These signals help identify and rank
relevant advertisers for sponsored search auction opportunities.
At times, search engines update their interpretation algorithms, which can impact the
quality of the search intent signal received by the search engine and subsequently influence
the sponsored search auction market. These changes are becoming more prevalent due to
recent advancements in NLP research. In June 2017, researchers at Google Brain introduced
the transformer model architecture [1]. Notable for its ability to dynamically accommo1See: https://www.iab.com/news/2023-u-s-digital-advertising-industry-hits-new-record-a
ccording-to-iabs-annual-internet-advertising-revenue-report/
1

date contextual language information through its “self-attention” mechanism, transformers
significantly improved the ability to interpret (encode) and generate (decode) text. This
technology has since become the backbone of modern-day Large Language Models (LLMs)
and has proclaimed a new era of programmatic interpretation capabilities.
We aim to understand how improvements to the quality of a search engine’s interpretation
algorithm impact short-term query-level cost-per-click (CPC) and the number of bidders
present in auctions (i.e., competition).
The answer to this question is not trivial. In our context, the platform (Google) faces
a fundamental trade-off between market thickness and allocation efficiency [2]. On the one
hand, new information generated by an improved interpretation algorithm could help the
platform better interpret search queries and identify more relevant advertisers, leading to
more bidders per auction. If the platform is cautious about allocating irrelevant advertisers,
then more informative signals may help it identify relevant bidders more often and lead to
potentially higher prices and thicker markets.
On the other hand, new information may help the platform segment markets and remove
irrelevant advertisers from auctions, leading to fewer bidders and potentially lower prices per
auction [3]. The winning advertisements may be more relevant, but prices could decline due
to increased market sparsity [4].
The effect on prices (CPC) is ambiguous due to potential changes in the number of bidders
competing for a search query, the types of bidders competing, and advertiser relevancy scores.
In sponsored search, query interpretation algorithms provide search intent signals to the
seller (Google) to help estimate advertiser relevancy scores. Relevancy scores are used by
search engines to measure the match quality (i.e., relevance) between an advertiser and a
search query and impact both auction eligibility and final paid prices.2 Within an auction,
advertisers are ranked based on ad ranks, which depend on both the bid they submit and the
platform estimated relevancy score. Higher relevancy scores will boost ad ranks, increasing
2See https://support.google.com/google-ads/answer/1722122?hl=en for discussion of Google ad
rank and use of relevancy scores.
2

an advertiser’s likelihood of being matched to an auction and winning an ad slot. Higher
scores also generally reward advertisers with lower final prices. More relevant advertisers
being matched may lead to higher relevancy scores and lower prices. Additionally, a higher
(lower) number of bidders will drive prices up (down). Advertisers could also adjust bids,
but in the short term, they are not receiving any new direct information from the platform,
making it unclear how or in which ways they could adjust their bidding strategies.
Closely related to our setting, existing literature on information disclosure in auction
markets highlights under what conditions prices may go up or down when buyers receive
new information. Information revelation specifically to bidders can lead to better matches,
improved allocation efficiency, and higher prices [5, 6, 7]. But, it can also lead to potentially
thinner markets and lower prices [8, 3, 4]. Whether prices increase or decrease ultimately
depends on whether the decline in market thickness lowers the number of competing bidders
and drives prices down or allocation efficiency raises bids and drives prices up [9]. However,
in our setting, the platform, not the bidders, is the one directly receiving the interpretation information generated by the language model. Given the platform has an incentive to
maximize its own profits, it remains unclear how the market might change and what the
underlying mechanism could be causing such changes.
In this paper, we reason that the first-order, short-run impact of a change in a platform’s
interpretation algorithm on sponsored auctions is increased precision in relevancy score estimates that lead to improvements in the platform’s matching process. To this end, we develop
(and test with search ads data) a theoretical auction model to better understand how new
information (learned via better interpretation algorithms) affects the number of auction competitors and CPC in the short run.3 Our model considers a platform (e.g., Google) that faces
a two-sided matching problem: matching relevant advertisers to incoming consumer search
queries. The platform uses an algorithm to learn about the search intent of a query and
3The focus on the short run is due to data limitations that do not allow us to study and test long-run
effects. Specifically, in March 2020, the COVID-19 pandemic started and likely affected consumer, advertiser,
and platform behavior in ways that could directly affect our empirical estimates (e.g., more people spending
more time at home and online).
3

subsequently uses this information to identify and rank relevant advertisers via relevancy
scores. Selected advertisers then compete in the auction for query advertising space. Advertiser relevancy scores impact auction eligibility and final paid prices. Advertisements are
sold in a second-price pay-per-click (PPC) auction.
The model analysis focuses on the comparative statics of this partial equilibrium shortrun model to understand what immediate impacts the introduction of a new interpretation
algorithm may have on the market. Several interesting results stem from the model. First,
our model predicts that queries will generally see thicker, not thinner, markets. This translates to more queries with auctions and more bidders within existing auctions and is in
contrast to other work that finds that new targeting technology segments and thins markets [3, 4]. The driver of this result is the negative externality cost the platform incurs if it
shows irrelevant ads. When this cost is high, the platform errs on the side of not allocating
advertisers (akin to avoiding a false positive). More informative signals generated by a new
algorithm increase the platform’s confidence that it understands the query, which leads to
relevant bidders being identified more often, driving the average number of bidders up, not
down.
Second, the model predicts that the effects on CPC depend on the amount of contextual
information (i.e., linguistic modifier information) contained within a query. Context is a
distinct type of information that is present in only certain queries.
For queries that contain substantial contextual information (e.g., longer queries such as
“running shoes with no laces”), our model predicts that short-run prices may fall despite an
increase in the average number of bidders competing for advertising space. This result is
driven by the presence of context within a query and an algorithm’s ability to understand
context. When an algorithm better understands the context in a query, it enables the
platform to estimate more precise relevancy scores and prioritize highly relevant advertisers
(i.e., those with high expected click-through rate (CTR)) within the auction. This can
lead to softer competition among bidders and lower final prices despite there also being
4

an increase in the average number of bidders. Our findings complement existing work on
information acquisition in auctions and suggest that market allocation efficiency and market
thickness might not always be in conflict [8, 3, 4]. In the context of a matchmaking seller
with relevancy scores, auctions can experience thicker markets, lower prices, and improved
advertising relevancy.
We rely on a recent event in which Google updated its search interpretation algorithm to
test these theoretical predictions. In October 2018, Google researchers used the transformer
architecture to build Bidirectional Encoder Representations from Transformers (BERT), one
of the first LLMs. BERT was notable for its dramatic improvement at predictive tasks over
previous state-of-the-art models [10]. In October 2019, Google introduced BERT as one of
its main interpretation algorithms to parse and understand user-generated search queries.4
To study BERT’s effect, we collect monthly average CPC and Competition Score (CS, a
measure of the number of bidders participating in an auction) from SEMRush—a company
that tracks search query performance—for a sample of roughly 12,000 search queries over
two years (2018-2020).
To estimate the effect of BERT on CS and CPC, we exploit the panel nature of our data
and employ a difference-in-differences (DD) approach akin to those employed in [11], [12],
and [13]. Specifically, we compare changes in CPC and CS before and after the introduction
of BERT, with a baseline of changes over the same months and queries but in the previous
year. This strategy exploits variation within queries and across time to identify BERT’s
effect.
To test for market expansion effects, we focus on queries that did not have competition
prior to BERT. Among these queries (about 20% in our dataset), roughly 11.5% of them
experienced an increase in CS after the introduction of BERT. These results are consistent
4See https://blog.google/products/search/search-language-understanding-bert/ for the
official BERT announcement. See https://searchengineland.com/welcome-bert-google-artificia
l-intelligence-for-understanding-search-queries-323976 for industry announcement. See https:
//twitter.com/searchliaison/status/1204152378292867074 for the announcement of international
BERT stating that US English BERT was introduced in October 2019.
5

with our theoretical model and reinforce the idea that new algorithms help the cautious
platform expand markets and allocate relevant advertisers to more advertising opportunities.
To test how the amount of contextual information present in a query affects CS and CPC,
we group queries based on their length (in general, the amount of contextual information
present within queries increases with query length). Consistent with the theoretical model,
we find that CS increases by 1.3% for both short and long queries, while CPC increases for
short queries by 3.8% and declines by -3.8% for longer queries.
These results withstand several robustness checks, including using additional years as
control, and accounting for changes in organic rank results and changes in consumer search
behavior. In addition, we replicate these results using an alternative identification strategy.
The idea behind this second strategy is to use query linguistic properties to identify queries
more or less likely to be affected by BERT. In doing so, we estimate the impact of BERT using
an identification strategy similar to a traditional DD with continuous treatment variables.5
.
Combined, our theoretical model and empirical findings help advertisers understand how
improved query interpretation algorithms affect sponsored search markets. In particular, our
results highlight the role of contextual information present within search queries. For advertisers, the lack of context in short, simple queries inherently limits the ability to differentiate
bidders, suggesting that future algorithms will cause these queries to become increasingly
more competitive and costly. At the same time, future algorithms will reward advertisers
with more precise relevancy score estimates in long-query markets, leading to greater bidder
differentiation within auctions. Whether this translates to lower prices depends on the new
algorithm’s signal improvements.
For academics, our empirical findings contribute to the sponsored search literature and
the growing literature on the economic impact of AI and LLMs. Additionally, our theoretical
model offers testable predictions for future algorithm updates and improves our theoretical
understanding of LLMs’ benefits.
5Several papers relied on this type of identification strategy in the past, including [14, 15, 16, 17, 18].
6

1.2 Related Work
Our paper relates to the growing literature on sponsored search, the economics of AI and
LLMs, information disclosure in auction markets, and targeted advertising.
Sponsored Search Sponsored search continues to be a prevalent advertising channel for
firms. It also remains an active area of research in both marketing and economics [19, 20, 21,
22, 23, 24, 25, 26, 4, 27]. Motivated by the auction structure of sponsored search, [19] and [28]
study equilibrium bidding strategies in generalized second price auctions. Empirical work
has also analyzed ad effectiveness [20, 24], sponsored and organic complementarities [21],
keyword spillovers [22], and advertising competition [26]. One stream of research focuses
on the downstream consequences of search engine platform design decisions, including the
impact of a search engine’s services on click behavior [25], result page features [29], and
the interaction between search engine optimization (SEO) and sponsored links [23]. We
contribute to this literature by empirically and theoretically studying how improved search
engine interpretation algorithms affect sponsored search markets.
Economic Impact of AI and LLMs Recent advancements in LLM technology have
spurred academic interest in understanding the potential economic effect of these models.
More recently, the development of ChatGPT has motivated researchers to study how generative LLMs impact areas such as labor markets [30, 31], information markets such as
Stack Overflow and Reddit [32], and marketing practices [33, 34, 35]. We contribute to this
growing literature by studying how LLMs used to interpret consumer search queries impact
sponsored search auction markets.
Information Disclosure in Auction Markets Research on information disclosure in
auction markets has primarily focused on settings where sellers can endogenously hide or
reveal information to buyers. [36] and [6] find that revealing information to buyers about
7

object features when markets are thick generally leads to increasing profits. However, this
may not occur in sparse markets due to what [6] calls the allocation effect. The allocation
effect occurs when information causes the rank ordering of bidder types to swap, leading to
weakly decreasing prices.
Related to our setting, [4] empirically studies how disclosing new information to advertisers in the sponsored search market affects prices, profits, and CTR. The authors find that
information disclosure generally leads to thinner markets and lower prices but higher overall
profits due to improved CTR. These market adjustments are due to advertisers improving
their self-selection into preferred markets, leading to better query-advertiser matching.
[7] also studies how information disclosure affects buyer selection into auction opportunities. In this paper, the authors find that information disclosure in the automobile resale
market can help quality-differentiated buyers self-select into preferred auction opportunities,
leading to higher market clearance rates and higher profits across all quality types.
When considering seller trade-offs between strategically revealing and hiding information, the theoretical and empirical literature has argued that there is a fundamental tradeoff between keeping auctions dense and improving buyer pricing accuracy [2]. Revealing
information may help extract value from buyers [7], but can also lead to thinner markets [4].
Google faces the same theoretical trade-offs between keeping auctions dense and improving buyer pricing accuracy in our market setting. However, prior literature has focused on
settings where buyers receive information [5, 7, 9, 4, 37]. In our context, buyers do not
receive any information to help with market selection or bid adjustment. Instead, the seller
uses the information to pick buyers for auction opportunities and maximize their expected
profits. This incentive and market structure may lead to theoretical and empirical results
that differ from previous work.
Targeted Advertising Our paper also relates to a stream of literature on matching and
targeting technology improvements in advertising markets [3, 37]. [38] studies how improve8

ments to sponsored search broad match technology affect market entry and seller profits.
The authors find that better broad match bidding algorithms can lower market entry costs
and induce greater auction participation, leading to prices.
Empirical and theoretical work has also documented the benefits and market effects of
better-targeted advertising technology. [39] empirically studies the newspaper market and
finds that better targeting can lead to higher prices due to improved advertiser-audience
alignment. [40] theoretically studies how targeting technology can lead to an increase in
the supply of advertising opportunities, potentially putting downward pressure on prices.
Similarly, [3] studies how improved targeting technologies given to advertisers can improve
matching efficiency, increase advertiser bids, but lower prices and market competition. We
contribute to this literature by studying how more informative search query intent signals
given to a platform and not advertisers impact matching in sponsored search markets.
1.3 Empirical Context
1.3.1 Sponsored Search Advertising
When a consumer types a query into a search engine (e.g., “how to cook chicken”), a search
engine must interpret the query and decide what domains (URLs) to present on the Search
Engine Results Page (SERP). At the top of the SERP, search engines may offer sponsored
search links. These links are bid for in a real-time auction before the SERP loads on the
consumer’s web browser.
An advertiser must create an ad campaign to begin bidding for sponsored search ad space.
A sponsored search campaign consists of seven main components: 1) The ad creatives (i.e.,
what the sponsored search ads look like), 2) The budget of the campaign, 3) The type of
individuals the advertiser wants to target (e.g., target Chicago and Los Angeles residents),
4) The keywords the advertiser wants to target, 5) How much the advertiser is willing to bid
for each keyword, 6) how to optimize bidding, and 7) The matching strategy. With these
9

ingredients, an advertiser can begin bidding for sponsored search positions.
Sponsored search auctions follow a pay-per-click (PPC) model. Under PPC, advertisers
pay only when a consumer clicks on an ad. Because search engines allow multiple sponsored
search links to appear at the top of result pages, most rely on a Generalized Second-Price
Auction (GSPA) to sell ad space and rank buyers [19, 28]. Under a GSPA, advertisers
are ranked by bids, and the top N winners for N ad slots show advertisements. If an
advertisement receives a click, the advertiser pays the minimum price needed to out-price
the next highest bidder (usually by $0.01).
In practice, advertisers are ranked with Ad Ranks. Ad Rank scores depend on many
factors, including the advertiser’s bid and their ad relevancy score. The relevancy score
measures how related an advertiser is to a particular query. Higher scores mean the advertiser
is more relevant and predicted to receive a click. Ad relevancy matters because it impacts
auction eligibility and CPC, with higher relevancy scores increasing the likelihood of auction
participation and lowering final paid prices. 6
The organic rank results are below the sponsored links on the SERP. These are links
that the search engine deems relevant to the given search query. Unlike sponsored links,
advertisers do not purchase organic links. Instead, the search engine uses an internal ranking
system to decide positions. A firm can appear in both sponsored and organic rank slots. For
example, even if Nike bids (and wins) the top ad slot in the sponsored search position for
the query “women’s running shoes”, it can still appear in the organic rank results.
1.3.2 BERT
Bidirectional Encoder Representations from Transformers (BERT) is a large-scale neural
network-based language model developed by Google in 2018. It is a pre-trained model
capable of understanding the context and nuances of natural language text, making it a
powerful tool for a wide range of NLP tasks [10]. Google introduced BERT in October of
6See https://support.google.com/google-ads/answer/1722122?hl=en.
10

2019.
BERT’s novelty comes from its ability to understand the context in language objects [10,
41, 42].7 Effectively capturing contextual information was a significant challenge for previous
state-of-the-art NLP algorithms such as Word2Vec [43] and was considered a breakthrough
in NLP research. With contextual knowledge, BERT can better understand language semantics, e.g., when “bank” indicates a financial institution or land alongside a river based on the
other words in the sentence. It can also lead to a better understanding of complex syntactic language structures, such as adjectives or adverbs, and non-linear relationships between
words. To capture this information, BERT generates a vector for each word based on the
other words surrounding the focal word, i.e., its context, using the transformer-based selfattention mechanism [1]. As such, the same word can have a different vector representation
depending on its surrounding words.
BERT can impact the interpretation of queries both with and without explicit contextual information. For longer queries with explicit contextual information, BERT will improve Google’s ability to understand how the words within the query interact and connect.
BERT’s contextual understanding can also change the interpretation and relationships of
short queries. Consider the simple query “bank”. By understanding context, BERT will
know that “bank” can potentially relate to financial institutions or rivers. Even though the
query itself does not explicitly contain context, the underlying training process allows BERT
to learn the implicit associations between words and pack that information within the word’s
vector representation. This can change the underlying categorization and interpretation of
shorter queries that don’t explicitly contain contextual information.
Google introduced BERT in its search engine in October 2019. In its release notes,8
Google stated that “by applying BERT models to both ranking and featured snippets in
Search, we’re able to do a much better job helping you find useful information”. In other
7
In simple terms, this means that BERT can differentiate when the word “mouse” refers to a computer
mouse or rodent, depending on the words around “mouse”.
8https://blog.google/products/search/search-language-understanding-bert/
11

words, Google uses BERT to identify which websites are related to a specific search query
(likely by computing similarity scores using vector representations of queries and websites).
While Google does not explicitly mention in its press release how this could affect search ads,
discussions with current and former employees suggest that signals generated from models
like BERT would be readily accessible by the sponsored search team to help rank advertisers.
1.4 Modeling the Market
To approach our research question, we consider a two-sided matching market where a platform uses text to match potentially relevant advertisers to incoming query advertising opportunities. To facilitate matching, the platform relies on an interpretation algorithm to
generate signals about the search intent of the search query typed by a consumer. The platform then uses these signals to estimate relevancy scores for advertisers.9 Using these scores,
the platform selects relevant bidders to compete in the auction. Selected bidders submit bids
and are ranked within the auction based on their ad rank, which is a combination of their
bid and relevancy score. The winning ad (that is, the highest ad rank) is displayed and a
click is received depending on the true match quality between the winning advertiser and
the query. The winning advertiser pays only if a click is received.
The model has two key components. First, we hypothesize that language is inherently
multidimensional. More specifically, a query is defined by its topic (i.e., the focus of the
query) and its context (i.e., linguistic modifier information). Each dimension is distinct,
and interpretation algorithms generate signals across each dimension. This multidimensional data structure is motivated by extensive literature in multiple disciplines, including
linguistics, computer science, and neuroscience, which commonly models language as multidimensional [44, 45, 46, 47]. Second, we assume the platform aims to maximize its own
profit and exclusively receives interpretation signals. Advertisers do not observe the inter9See https://support.google.com/google-ads/answer/1722122?hl=en for discussion of Google ad
rank and use of relevancy scores.
12

pretation signals generated by the platform’s language model and also do not observe their
relevancy scores. This is consistent with the fact that advertisers on Google’s platform do
not observing their relevancy scores at bid time or know the exact information that is being
used to estimate their scores. We highlight this component as it differentiates our empirical
setting from other information disclosure in auction settings where bidders are the ones that
receive information [7, 4, 37].
A query can be deconstructed into the topic (focus) of the query and its modifier information that conveys additional meaning within the topic. For example, the query “running
shoes with no laces” focuses on “shoes” and has modifier information in the form of “running” and “with no laces”. “Running shoes with laces” has the same focus (“shoes”) but
conveys different modifier information, particularly “with laces” instead of “with no laces”.
Both of these queries are categorically similar and have the same topic, but each contains
distinct modifier information that conveys different intents within the given topic category.
This modifier information is “context”.
In NLP, topic modeling is a prevalent area of research. Even prior to BERT, NLP
algorithms like Latent Dirichlet Allocation (LDA), Word2Vec (W2V), or Doc2Vec (D2V)
organized textual documents into latent “topics”. Ultimately, topic modeling focuses on
categorizing and clustering similar documents [48, 43, 49]. Topic categorization and differentiation across queries is no different.
Contextual modifier information is distinct from topical information and plays a fundamentally different role within a query. It conveys additional meaning that differentiates
queries within a topic category. Although both “running shoes with no laces” and “running shoes with laces” topically relate to “shoes”, they have different types of contextual
information that lead to fundamentally different meanings. The context dimension leads to
differentiation within a given topic category.
Yet, not all queries explicitly contain context. The short query “shoes” topically relates to
“running shoes with no laces” and “running shoes with laces” but lacks additional contextual
13

information to differentiate its search intent within the topic category. Without context, its
difficult to further differentiate the query within the given topic. The inherent difference
in information conveyed within shorter search queries fundamentally differentiates shorter
queries from longer queries.
To summarize, these examples highlight that queries are multidimensional. Queries can
be bucketed and differentiated across topics and further differentiated within topics based on
context, but only when context is explicitly present. We capture this structure in our model
by defining a query as having a topic dimension that defines the focus of the query and a
context dimension that defines the modifier information present within the query. A topic
and a context dimension will also define an advertiser’s text-based advertisement. The CTR
function for a query will depend on the topic and context alignment between the winning
advertiser and the query. A platform’s interpretation algorithm generates both topic and
context signals.
While not novel to our setting, it is essential to note that advertisers are horizontally
differentiated (e.g., Nike and Geico are relevant to different queries). Additionally, we assume
there are negative externality costs paid by the platform when irrelevant advertising is shown.
Irrelevant ads can annoy consumers, increase consumer search costs, and harm the reputation
of the platform [50, 23, 51]. We now describe our model.
1.4.1 Setup
* Advertiser An advertiser Ai
is defined by a topic θi ∈ {0, 1} and a context zi ∈ {0, 1}.
We assume four bidders are in the market, one for each combination of topic and context
(< 0, 0 >, < 0, 1 >, < 1, 0 >, and < 1, 1 >). Advertisers maintain a private value vi ∼ U[0, 1]
for clicks.10
10We assume valuations vi ∼ U[0, 1] because advertisers do not observe match quality prior to the auction
and therefore cannot change short-term valuations. A two-period model aimed at studying long-term effects
could relax this assumption and allow advertisers to learn over time what the new match quality is, leading
to changes in valuations that depend on the topic and context dimensions. We discuss this idea further in
the Robustness Checks section.
14

Query Queries are also defined by a topic t ∈ {0, 1}, a context q ∈ {0, 1}. θ and z directly
map to t and q types for queries, respectively. There are four possible query types (< 0, 0 >,
< 0, 1 >, < 1, 0 >, and < 1, 1 >). t captures the underlying topic of the query (e.g., is
it about “shoes” or “insurance”) while q captures contextual modifier information that can
differentiate the query type within a topic. Each query has an equally likely probability of
being drawn ( 1
4
).
Denote the advertiser who wins the advertising opportunity for query Q by Aw, its type
by θw, and its context by zw. The CTR for Q depends on the match quality between the
winning advertiser and the query. Define Bt ∈ [
1
2
, 1] as the relative importance of topic
alignment to the query’s CTR. We model query CTR in Equation 1.1.
CT R(A, Q) = BtI{θ=t} + (1 − Bt)I{z=q}, (1.1)
Bt measures the relative importance of topic alignment compared to context alignment.
For a query where topic alignment matters more relative to context, Bt
is large. This will
be the case for shorter queries where context is not present. Bt will be smaller for queries
where context alignment is meaningful relative to topic alignment. This will be the case for
longer queries. Therefore, we assume Bt
is decreasing with a query’s length. This captures
the intuition that for a short query like “shoes”, topic alignment dictates CTR as context
is not present, while for a query like “running shoes with no laces”, both topic and context
alignment matters to the query CTR.
Platform Our market considers a platform (Google) that sells query ad space to advertisers. Ad space for a query gets sold via a second-price auction (SPA) where the platform
endogenously selects who participates in the auction. The advertising space is sold in a
pay-per-click (PPC) structure, so the winning advertiser does not pay unless their ad is
clicked.
15

The platform’s profit function is:
π(A, Qˆ ) = I{Click}
Adr ˜
rw
− I{θw̸=t}C, (1.2)
where Adr ˜
rw
is the second highest Ad Rank in the auction scaled by the relevancy score of
the winner bidder, I{Click} indicates a click, I{θw̸=t} indicates when there is a mismatch
between the query topic and winning advertiser topic, and C is the negative externality cost
associated with displaying an irrelevant advertisement to the consumer (i.e., wrong topic).
Adr ˜
rw
is structured such that winning advertisers pay the minimum bid price needed to win
the position [23, 52].
Denote the set of chosen advertisers for an auction opportunity by Aˆ. We assume the
platform observes advertiser types (θ and z) before selection. However, the platform needs
to use an algorithm to interpret and identify the context and topic of a query. When a query
arrives, the platform uses an algorithm a ∈ R to interpret it. For a query with topic t and
context q, an algorithm a takes the query as input and outputs a signal Qˆ with components
tˆ and qˆ to the platform.
For a query Q and algorithm a, P r(tˆ= t) = γt(a) and P r(ˆq = q) = γq(a). The platform
learns the true value of t with probability γt(a), and it receives an inconclusive signal with
probability 1 − γt(a). Similarly, the platform learns the true value of the context q with
probability γq(a) and it receives an inconclusive signal with probability 1−γq(a). We assume
the inconclusive signal is equivalent to the platform’s prior: P r(t = 1) = P r(q = 1) = 1
2
.
Therefore, tˆ ∈ {0,
1
2
, 1} and qˆ ∈ {0,
1
2
, 1}. We also assume that γt
∂a > 0 and γq
∂a > 0, i.e.,
a better algorithm always improves the probability of learning the true value of t and q.
Finally, we assume advertisers and the platform know γt(a) and γq(a).
11
After receiving the estimates tˆ and qˆ, the platform picks which advertisers, if any, to
allocate to the auction opportunity in order to maximize expected profits. Conditional on
11This assumption stems from the fact that Google announces the algorithm update and releases press
articles that describe how new algorithms work.
16

being allocated to an auction, it is a weakly dominant strategy for advertisers to bid their
valuations vi (the proof is in Appendix A.1). We model the expected CPC for a query with
advertiser set Aˆ by E[CP Ck(Aˆ)] = E[CT Rk] ∗ E[
Adr ˜
rw
].
Ad ranks determine the order of bidders. We model ad rank for an advertiser i as
Adri = biri(Qˆ), where bi
is the submitted bid and ri(Qˆ) is the relevancy score advertiser i
receives from the platform given the information set Qˆ. The relevancy score is modeled as the
expected CTR for advertiser Ai given signal Qˆ: ri(Qˆ) = P r(θi = t|tˆ)Bt+P r(zi = q|qˆ)(1−Bt).
We assume that the platform knows the CTR coefficient Bt
.
Timing The order of the game is as follows. A query Q is randomly drawn by Nature,
producing unobservable components q and t. The platform uses algorithm a to generate
signals qˆ and tˆ. The platform then estimates relevancy scores for each advertiser and picks
some number of auction participants. If the seller runs an auction, chosen advertisers submit
bids, the winning advertiser displays an ad, the consumer decides whether to click given the
type alignment, and the platform receives a profit. An algorithm a affects profits, CPC, the
average number of query bidders, and CTR by changing the seller’s q and t signals.
1.4.2 Results
Our results will focus on auction outcomes at the query level and show how improving the
algorithm a can impact CPC and the Number of Bidders (NOBs) competing for advertising
opportunities in the sponsored search auction market. Interestingly, we will show that under
reasonable assumptions, query-level CPC may decline despite an increase in the average
number of bidders. We now walk through how this can occur in this partial equilibrium
model. Formal proofs are in Appendix A.1.
Average auction market CPC and the number of bidders depend on the cost of showing
topically irrelevant advertisements (C) and the underlying importance of topic alignment
(Bt) to a query’s CTR. If C is small, the platform is willing to allocate less relevant advertisers
17

to auctions. Even if it shows an irrelevant advertisement, the cost of doing so is low, and its
overall expected profit is better off because the auctions have more bidders. If C is large,
the platform is unwilling to allocate potentially irrelevant advertisers to auctions. Here,
the platform errs on the side of caution when allocating bidders because it doesn’t want to
incur the cost C. When C is moderate, the average CPC and number of bidders largely
depends on the underlying importance of topic alignment (Bt). In combination, C and Bt
create six distinct regions that lead to different platform selection strategies and ultimately
different expected NOBs and CPC. Figure 1.1 visualizes these regions based on Bt and C,
and Lemma 1 presents the expected NOBs and CPC by region.
Lemma 1. The average CPC and NOBs for a query Q depend on C and Bt
. Let J =
40−60Bt+45B2
t −16B3
t
30(2−Bt)
2
, X =
9+18Bt−3B2
t +16B3
t
30(1+Bt)
2
, Z = J + X −
B(3−B)
6
, P =
−5+20Bt−25B2
t +15B3
t −3B4
t
6Bt
,
and L = X + J − P.
1. Region 1: C ≥
4
(2−Bt)
2
[
2
3 − Bt + B2
t −
4
15B3
t
]
• E[NOB] = 2γt
• E[CP C] = γt
6
[1 + Bt − γq(Bt − 1)2
]
2. Region 2: 4+11Bt
15 ≤ C ≤
4
(2−Bt)
2
[
2
3 − Bt + B2
t −
4
15B3
t
]
• E[NOB] = 2γt + 4γq − 4γtγq
• E[CP C] = γt
1+Bt
6 + Jγq − γtγq(J +
(Bt−1)2
6
)
3. Region 3: Bt ≤ C ≤
4+11Bt
15
• E[NOB] = 4(γt + γq) − 6γtγq
• E[CP C] = Jγq + Xγt − Zγtγq
4. Region 4: 3
5 ≤ C ≤ Bt
• E[NOB] = 4(γt + γq) − 5γtγq
18

• E[CP C] = Xγt + Jγq − Lγtγq
5. Region 5: Bt ≤ C ≤
3
5
• E[NOB] = 4 − 2γtγq
• E[CP C] = (J −
3
10 )γq + (X −
3
10 )γt − (Z −
3
10 )γtγq
6. Region 6: C ≤
3
5
and C ≤ Bt
• E[NOB] = 4 − γtγq
• E[CP C] = (X −
3
10 )γt + (J −
3
10 )γq − (L −
3
10 )γtγq
Figure 1.1: Regions that determine how auction evolves with changes to a. C is the cost
the platform incurs for showing a topically irrelevant ad and Bt
is the importance of topic
alignment for a query’s CTR.
19

Region 1 in Lemma 1 describes the average CPC and NOB when C is generally large,
while Regions 5 and 6 dictate average CPC and NOB when C is small. Regions 2 through
4 describe average query CPC and NOB when C is generally moderate. When C > 8
5
, all
queries live in Region 1.
One way to interpret C is the cost of a false positive advertising allocation. Notice that
when C is low (Regions 5 and 6), NOB is declining with there are any gains in topic or
context signals (γq and γt). The decline in the NOBs is consistent with existing literature [3,
4]. Here, we would observe market segmentation outcomes with improvements in the quality
of the interpretation algorithm (higher a) due to the platform removing irrelevant advertisers. When C is high (Region 1), the platform is extremely conservative regarding which
advertisers it allocates to auctions, so the number of bidders actually increases with topic
signal improvements. Here, the platform errs on the side of not allocating under uncertainty, and signal improvements overcome this uncertainty, leading to market expansion not
segmentation.
For brevity, we assume C is large (C > 8
5
) and focus on Region 1 for the rest of the
discussion. We refer the reader to Appendix A.1.4 for a discussion of the other regions.12
In Region 1, the platform is extremely conservative about allocating advertisers to auctions because the cost of showing topically irrelevant advertisements is high. This leads to an
average number of bidders for a query Q of 2γt(a). Intuitively, the platform will only allocate
advertisers when they know they’re in the right ballpark of the query’s focus. If the platform
knows an incoming query is about “shoes”, it is happy to allocate “shoe” advertisers. But,
if the platform is unsure whether a query is about “shoes” or “insurance”, it’s not willing
to allocate any bidders for fear of incorrectly showing an “insurance” advertiser on a “shoe”
query, or vice versa.
When the platform improves the quality of its interpretation algorithm (increasing value
12Anecdotal evidence from talking with former employees at Google suggests that Google is extremely
cautious about bidder selection and worries about platform quality and reputation, suggesting that C is
indeed large.
20

of a), this improves its ability to generate informative topic signals. In turn, it improves
the platform’s ability to categorize queries and ultimately allocate relevant advertisers to
relevant auctions. This leads to Proposition 1.
Proposition 1 (Platform Selection of Advertisers). Under sufficiently high cost C (C ≥
8
5
),
the average number of bidders for query ad space increases with a (
∂
∂a 2γt(a) > 0).
Proposition 1 contrasts [3, 4] and suggests that improvements to targeting technologies
will increase, not decrease, market density. This is a byproduct of the platform’s market
selection process, which aims to maximize profits and avoid C.
Average CPC for a query in Region 1 is γt
6
[1 + Bt − γq(Bt − 1)2
] and depends on both
γt and γq. Improvements to topic signals help the platform get relevant advertisers into
the auctions and puts upward pressure on average CPC. Context signals help the platform
further differentiate bidders and prioritize the high CTR advertisers. Doing so separates
bidder overlap within the auction and puts downwards pressure on CPC.
Consider the case when there are two bidders that the platform knows are topically
relevant. They have the same relevancy score r and valuations are both drawn from U[0, 1],
so they have the same ad rank distribution U[0, r]. Competition is intense because their ad
rank distributions are identical and each has a 50% chance of winning. Now consider when
the platform identifies the context of the query and separates the two bidders. One bidder
is of high relevance and gets a high relevancy score, r < rH, the other is of low relevance
and gets a low relevancy score, rL < r. Now the bidder with rH has an ad rank distribution
of U[0, rh] while the low relevancy bidder has a distribution U[0, rL] with rL < rH. The rH
bidder is more likely to win and is the bidder that has a higher likelihood of receiving a click,
but the second highest ad rank will generally be lower than before because rL < r and the
scaling relevancy score (rH) is now higher, so average CPC will decrease.
When the platform improves its algorithm a, both its topic and context signals change.
If context signal improvements dominate topic signal improvements, and context alignment
matters to a query’s CTR, average CPC may decline. This leads us to Proposition 2.
21

Proposition 2 (Average CPC). Average CPC for a query in Region 1 will decline if 1+Bt
6(Bt−1)2 <
γt
γ
′
t
γ
′
q + γq, where γ
′
q and γ
′
t are the partial derivatives of each with respect to a.
If context alignment matters to the CTR of a query (as is the case for longer queries
where Bt
is small), and contextual signal improvements are large, CPC will likely decline.
However, if contextual alignment is not important for the CTR of a query (as is the case for
short queries where Bt
is large), then average CPC is unlikely to decline.
In Figure 1.2 we present the threshold value log(
1+Bt
6(Bt−1)2 ) by Bt
.
13 The gray region of the
figure indicates when the inequality would hold (i.e., log(
1+Bt
6(Bt−1)2 ) < log(
γ
′
q
γ
′
t
γt+γq)) and CPC
would decline despite an average increase in the number of bidders. The blue region indicates
where the inequality would not hold and CPC would increase with the rise in the number of
bidders. The line is where there is equality and there would be no changes to average CPC.
As Bt decreases, the potential for lower CPC increases. In other words, longer queries have
greater potential to see declining CPC due to the increasing prevalence of context.
Figure 1.2: Potential market changes in Region 1.
13We log scale the inequality for visualization and indicate what regions of Bt short, medium, and long
query lengths would live.
22

Summary and Predictions To summarize, our model states that improvements to a
platform’s interpretation algorithm help the platform estimate more precise relevancy scores
for advertisers. This leads to thicker markets for all queries and expands queries eligible for
auctions due to topic signal improvements helping the platform identify relevant advertisers
more often. For context rich queries, when context signal improvements are large, platforms
may see lower CPC due to significant improvements in bidder differentiation. For context
poor queries, there is a lack of contextual information to further differentiate bidders, making
it difficult to see declines in CPC.
Given our model’s results, we make the following predictions about BERT’s effect. First,
incremental improvements in topic signals will lead to expanded markets, both in terms of
more advertisers for existing auctions and more queries with competitive auctions. This is
driven by BERT’s ability to better understand the focus of a query and this will occur for all
queries. Second, we expect BERT to substantially improve Google’s context signals. This
will predominantly benefit existing long query auctions (low Bt) and may lead to lower CPC
despite an average increase in the number of bidders. Existing short query auctions (high
Bt) will not benefit from context signal improvements and will see higher CPC associated
with the increase in the number of bidders.
1.5 Empirical Analysis
1.5.1 Data
Sampling search queries To empirically test the predictions of our model, we collect data
from SEMRush, a leading provider of sponsored search rank and keyword data. We rely on
a survey that we administered to Amazon Mturkers to generate a sample of queries for the
analyses presented in this paper. After selecting 32 different topics (see Appendix A.2 for
the list of topics), we asked survey respondents the following question: “Please write a search
query related to the topic of ‘Topic’ that you would search for on Google”. 100 Amazon
23

Mturk participants took the survey and each participant was asked about five randomly
selected categories, giving us 500 responses and roughly 16 responses per topic category.
Not all MTurk submissions were queries. To focus on submissions that were search
queries, we removed 210 irrelevant answers. Irrelevant responses included answers where
participants typed in specific URLs, website content, descriptions of how to type up a search
query, and quotes from Google “Help Pages” describing how to search. Removing these
responses left us with 290 search query answers.
Using the remaining 290 sampled queries, we turned to the broad match database at
SEMRush.14 For a given query, SEMRush returns a set of similar queries for which it has
data for the current month.15 We limited our match to queries with an average search volume
of at least 500 from 2021-2022, generating a dataset of roughly 120, 000 queries related to
the original 290 queries. However, many of the queries contained explicit terms (e.g., adult
content such as pornographic search queries) unsuitable for analysis (advertisers don’t target
adult content queries). To filter these out, we generated a list of adult content terms and
fuzzy-matched each query with the list of terms. We removed the query from our data if
there was significant similarity between the query and any of the terms. After completing
this process, we retained roughly 40, 000 queries.
CPC and Competition Score SEMRush provides two outcomes that are relevant to
answering our research question: CPC and Competition Score. CPC measures the average
price advertisers pay for an ad click. Competition Score is a proprietary normalized score
(between 0 and 1) that measures the relative number of advertisers bidding for ad space for
the given query.16 It is worth noting that SEMRush collects data for many queries, even
if they have little to no advertising, meaning CPC and Competition Score can take on the
14See: https://www.semrush.com/features/keyword-research/
15For example, for the query “shoes”, the database will show monthly data for “shoes” as well as similar
queries, such as “running shoes”, “women’s shoes”, “best shoes for hiking”, etc.
16From discussions with SEMRush, Competition Score is linearly comparable and can be viewed similarly
to the Google Trends information commonly used in research to measure search volume. While we can’t
estimate the precise number of bidders, we can estimate relative changes.
24

value of zero.
We collected historical monthly information about CPC and Competition Score for each
of these queries from January 2018 to February 2020. We then filtered out queries that did
not have persistent historical information (more than a year of missing data) and those with
limited historical search volumes (less than 100 average searches/month during the time
frame).
To empirically validate Competition Score and CPC, we collect SEMRush and Google
Keyword Planner (GKP) tool data from July 2021 to February 2022 for 10,000 queries in
our dataset.17 From GKP we have two measures of competition: (1) “Competition”, which
is a measure of the number of bidders competing in the query’s auction using three values:
low, mid, and high; and (2) “Competition (index value)” which measures the total number
of ad slots filled divided by the total number of ad slots available on a scale from 0 to
100. The correlation between these two measures is 0.98. We then compute the Pearson
correlation between the Competition Score of SEMRush and the two competition measures
from Google. We find that the Competition Score of SEMRush has a correlation of 0.65
with the Competition measure obtained from Google 18 and a correlation of 0.87 with the
Competition (index value) obtained from Google, suggesting SEMRush data is reasonably
correlated with data obtained directly from Google.
To validate CPC, we use we use the only CPC-related measure from GKP: the maximum
submitted bid. While average CPC and maximum bid are different measures, they should
reasonably correlate if SEMRush data is reliable. The Spearman correlation between the
average CPC of SEMRush and the maximum submitted bid from Google is 0.57, again
suggesting a high level of agreement.
Our final dataset contains 11, 949 unique queries and 284, 146 monthly observations. In
the top part of Table 1.1, we present overall summary statistics for our dependent variables,
17Unfortunately, GKP data goes back in time to at most three years, and we did not collect data in time
to replicate some of the analysis reported in this paper with GKP data.
18We encode Google’s Competition measure as low (0), medium (0.5), and high (1) to calculate this
correlation.
25

Competition Score and CPC, across all queries from January 2018 to February 2020.
Table 1.1: Summary statistics.
Mean St. Dev. Min Max Median
All queries
CPC 1.25 3.343 0.00 584.73 0.54
Competition score 0.244 0.356 0.00 1.00 0.05
Short queries
CPC 0.911 1.918 0.00 64.110 0.400
Competition Score 0.248 0.364 0.00 1.00 0.050
Medium queries
CPC 1.339 3.947 0.00 584.730 0.610
Competition Score 0.278 0.378 0.00 1.00 0.060
Long queries
CPC 1.533 3.078 0.00 83.860 0.720
Competition Score 0.126 0.216 0.00 1.00 0.030
Table 1.1 show that, in the aggregate, 1) many queries are not competitive (low median
and mean Competition Score) and 2) CPC is often relatively low, but the maximum price
can be extremely high. We also break down queries by word length, an important moderator
we focus on in our analysis. We group queries into three categories for ease of presenting
results. A short query has two or fewer words, a medium query has three to five words,
and a long query has six or more words. We present summary statistics by query length
in the bottom part of Table 1.1. The table shows that as query length increases, demand
generally decreases while CPC increases. These results generally align with the trend that,
as query length increases, the queries become more niche and are harder to match to but
also potentially more valuable (higher CTR) to advertisers.
1.5.2 Empirical Strategy
To study how BERT affects CS and CPC, we implement two identification strategies that rely
on different underlying assumptions. In this section, we describe our preferred identification
26

strategy. We discuss the second strategy in Appendix A.3. Results are consistent across
methods.
Our main identification strategy exploits the panel nature of our data and compares
changes in each query’s CS and CPC before and after the introduction of BERT with changes
in the same query’s CS and CPC in the year before the introduction of BERT.19 In doing
so, we implement a strategy akin to a difference-in-differences (DD) where the treated and
control queries are the same, but outcomes are observed across years.20 Our identification
strategy is similar to those employed in recent papers, such as [11], [12], and [13].
This strategy aims to use variation across time within queries to identify BERT. An
advantage of this strategy is that selection into the treatment is not an issue since treated
and control queries are the same. However, time-varying unobservables that vary at the yearmonth level may bias our results. For example, our results could be positively biased if some
market factor correlated with CS and/or CPC and motivated Google to launch BERT in
October 2019. Alternatively, queries may not be comparable across years if a query correlates
with year-month-specific unobservable events, e.g., large concerts or sporting events. Both
issues are classic DD challenges akin to unit-time-specific events tampering with causal effect
estimates.
We operationalize this identification strategy using the following model:
Yijt = β1T reatedij × P ostt + δi + γj + ψt + ϵijt, (1.3)
where Yi,j,t is the outcome of interests (either log(CP C)ijt or log(CS)ijt) for query i, year
group j and month t.
21 T reatedij is a binary indicator that takes on the value of one when
query i is observed during the treated year-group window (July 2019–February 2020), zero
19Essentially, we are comparing outcomes for each query in the year in which BERT was introduced with
outcomes in the year prior to BERT introduction.
20In standard DD settings, outcomes are observed over the same period, but treated and control units are
different.
21We add 1 to both CPC and CS to avoid taking the log of zero.
27

otherwise (July 2018–February 2019).22 P ostt
is a dummy that takes on the value one if
the month t is after the month Google introduced BERT (November–February), zero otherwise (July–October). We include query fixed effects (δi) to control for time-invariant query
unobservables, year-group fixed effects (γj ) to control for group-specific shocks impacting
all queries in the same year-group (e.g., in 2019–20, demand is higher for search ads), and
month-of-year fixed effects (ψt) to account for monthly shocks common to all queries (e.g.,
holidays and advertiser monthly spend). We cluster standard errors at the query level to
account for potential serial correlations in the dependent variable and estimate the model
using OLS. The parameter of interest is β1, representing the incremental impact of BERT
on the sponsored search auction outcomes of interest.
We estimate Equation 1.3 from July to February so that we have four pre-BERT months
(July–October) and four post-BERT months (November–February).23 In addition, we focus
on two periods, July 2018–February 2019 (control period) and July 2019–February 2020
(treated period). In Section 1.5.4, we show that results hold with additional control year
groups, specifically, July 2016–February 2017 and July 2017–February 2018 months.
Identification Checks The key assumption behind our DD identification strategy is that
no unobserved time-variant, group-specific shocks correlate with the entry of BERT and
auction market outcomes.
In our setting, we worry about changes in advertiser or consumer search behavior due
to time-varying unobservable events that correlate with BERT’s release. It is possible that
consumer search demand characteristics differ across years and that consumer behavior or
advertiser behavior in, say, the holiday season in 2018 is different than that in 2019. If this
is the case, the results we observe may be due to these unobservable time-varying factors.
This concern is analogous to worrying about unit-specific time-varying unobservable events
22We refer to j as a group rather than a year because our treated (control) periods fall across multiple
years. For example, the treated period includes the years 2019 and 2020 because the treated period spans
July 2019 to February 2020.
23We limit the post-BERT period to February because we want to avoid picking up any COVID-19 effect,
which started impacting the US in March 2020.
28

correlated with the treatment event in a DD across states or cities.
To reduce concerns about these issues, we do two things. First, to alleviate demand-side
consumer shocks, our sampling procedure removes queries that have intermittent or volatile
search volumes that may correlate with unobservable time-varying events. For example, a
query about a political event (e.g., November 2018 elections) that may see a spike in search
in a particular month and then quickly disappear will not be part of our sample. Doing
this restricts our analysis and results to persistently searched queries in order to minimize
empirical exposure to volatile, time-varying search queries.
Second, as is common in DD analyses, we show that treated and control units behaved
similarly before the introduction of BERT (parallel trends assumption), suggesting that the
year prior to BERT is a good counterfactual. We do so by implementing an event study
design.
To obtain event study parameter estimates, we estimate the following specifications:
Yijt = β1T reatedij × Montht + δi + γj + ψt + ϵijt, (1.4)
Everything is as in Equation 1.3, but we replace our binary P ost variable with eight monthly
dummies that indicate the eight-month window from July to February. We estimate Equation 1.4 for all the outcomes we study, setting the baseline level for the monthly dummies to
be October (the month prior to the introduction of BERT). In the next section, we present
these event studies along with the main estimates.
1.5.3 Results
Topical information effect The theoretical model predicts that topic signal gains due
to BERT will help Google expand auction markets. This will lead to Google identifying
relevant bidders more often for already competitive queries and turn non-competitive queries
into competitive ones. To test this prediction, we start by splitting the data into competitive
29

and non-competitive queries. We deem a query non-competitive if the mode Competition
Score during the July-October pre-period in 2018 and 2019 is zero. We find that roughly
20% of our queries are deemed non-competitive.
We then take these queries deemed non-competitive and create a binary variable called
Has Competitionijt that takes on the value one if query i’s Competition Score is greater
than zero for a given year-group j month t, zero otherwise. Using this variable, we can
measure what proportion of non-competitive auctions become competitive in the post-BERT
periods. We then estimate Equations 1.3 and 1.4, using Has Competitionijt as the dependent
variable. In Figure 1.3, we visualize the event study parameter estimates. We find that prior
to the treatment, estimates are indistinguishable from zero, with no apparent trends. In the
post-treatment period, we observe that the estimates become positive, an effect consistent
with BERT increasing ad space supply by identifying more relevant bidders for auction
opportunities.
0%
5%
10%
15%
Jul Aug Sep Oct Nov Dec Jan Feb
Percent Change in Has Comp.
Pre Post
Figure 1.3: The effect of BERT on Has Competition for non-competitive queries. Error bars
represent 95% confidence intervals.
We present the estimates of Equation 1.3 in Table 1.2. We find that BERT converts
roughly 11.5% of non-competitive queries into competitive markets.24 This suggests that
24As a sanity check, we estimated Equation 1.3 using log(CP C) as the dependent variable for the queries
deemed non-competitive before BERT. We find a positive and statistically significant coefficient (0.03 with
30

topical signal improvements due to BERT help the platform identify meaningfully relevant
advertisers for more query auctions.
Table 1.2: The effect of BERT on Has
Competition for non-competitive queries.
(1)
Post × Treated 0.114∗∗∗
(0.009)
Observations 30,474
R
2 0.380
Significance Levels: ∗p<0.1; ∗∗p<0.05;
∗∗∗p<0.01.
Note: Regressions include year-group,
month, and query fixed effects. Standard errors clustered at the query level are reported
in parentheses.
Next, we turn to the sample of competitive queries. Our theoretical model predicts that
under sufficiently large C, the average number of bidders should generally increase across Bt
.
To test variation in Bt
, we group queries based on their length. Query length correlates with
the degree of contextual information present within the query (high Bt associated with short
queries, longer queries associated with a smaller Bt) and is often used as a rule-of-thumb
measure by advertisers when defining targeting strategies. We then estimate Equation 1.3
and Equation 1.4 by query length and using log(CS) as the dependent variable.
We start by presenting the event study estimates in Figure 1.4. Before the treatment,
we find estimates close to zero, partially validating the parallel trends assumption. In the
post-treatment period, estimates become positive and significant for short, medium, and long
queries, suggesting that competition increases for all queries that were competitive prior to
BERT’s introduction.
In Table 1.3, we report the main effect estimates from Equation 1.3. We find CS increases
p = 0.0005), suggesting that CPC also rises with the increase in the number of competitive auctions for the
non-competitive queries following the introduction of BERT.
31

0%
1%
2%
Jul Aug Sep Oct Nov Dec Jan Feb
Percent Change in CS
Pre Post
(a) Short queries
0.0%
0.5%
1.0%
1.5%
Jul Aug Sep Oct Nov Dec Jan Feb
Percent Change in CS
Pre Post
(b) Medium queries
0%
1%
2%
Jul Aug Sep Oct Nov Dec Jan Feb
Percent Change in CS
Pre Post
(c) Long queries
Figure 1.4: Percent change in Competition Score by query length. Error bars represent 95%
confidence intervals.
between 1% and 1.3% for all queries.
Table 1.3: The effect of BERT on log(CS) by query
length.
(1) (2) (3)
Short Medium Long
Post × Treated 0.013∗∗∗ 0.010∗∗∗ 0.013∗∗∗
(0.001) (0.001) (0.001)
Observations 47,912 88,888 23,887
R
2 0.951 0.971 0.920
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Note: Estimated with competitive queries. Regressions include year-group, month, and query fixed effects. Standard errors clustered at the query level are
reported in parentheses.
Consistent with our model’s predictions, the empirical evidence suggests that BERT’s
introduction helped Google uniformly expand auction markets and improve advertiser allocation to auctions. This is because BERT’s topic signal improvements helped Google
overcome the negative cost C and do a better job identifying topically relevant advertisers
for queries.
Contextual information effects Our model predicts that CPC can increase or decrease
depending on whether the query contains contextual information. In particular, CPC should
increase for queries with limited to no contextual information; however, CPC may decrease
32

for queries with contextual information.
Again, we group queries based on their length to operationalize and test these hypotheses.
We then estimate Equation 1.3 and Equation 1.4 on the set of competitive queries, by query
length, and using log(CP C) as the dependent variable.25 In Figure 1.5, we present the event
study estimates by query length. First, as before, in all cases, we find parallel pre-treatment
trends, supporting the validity of our identification strategy. Second, we see that in the
post-treatment period, CPC increases for short queries but decreases for medium and long
queries.
0%
2%
4%
6%
Jul Aug Sep Oct Nov Dec Jan Feb
Percent Change in CPC
Pre Post
(a) Short queries.
−4%
−2%
0%
2%
Jul Aug Sep Oct Nov Dec Jan Feb
Percent Change in CPC
Pre Post
(b) Medium queries.
−7.5%
−5.0%
−2.5%
0.0%
Jul Aug Sep Oct Nov Dec Jan Feb
Percent Change in CPC
Pre Post
(c) Long query.
Figure 1.5: Percent change in CPC by query length. Error bars represent 95% confidence
intervals.
In Tables 1.4, we present the estimates of Equation 1.3 by query length. Consistent with
our model predictions, query length moderates how BERT’s introduction affects existing
auction market CPC. We find a 3.8% increase in CPC for short queries. However, as query
length increases, CPC quickly decreases. CPC decreases by 1.5% for medium queries and
3.7% for long queries. These results imply that BERT’s introduction significantly improves
Google’s ability to estimate more precise relevancy scores for queries that contain contextual
information, leading to lower CPC but ultimately more relevant ads.
25We again remove non-competitive queries because, for these queries, prices can only increase since they
were zero when they were non-competitive.
33

Table 1.4: The effect of BERT on log(CP C) by query
length.
(1) (2) (3)
Short Medium Long
Post × Treated 0.038∗∗∗ −0.015∗∗ −0.038∗∗∗
(0.008) (0.006) (0.013)
Observations 47,912 88,888 23,887
R
2 0.686 0.728 0.671
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Note: Estimated with competitive queries. Regressions
include year-group, month, and query fixed effects. Standard errors clustered at the query level are reported in
parentheses.
1.5.4 Robustness Checks
In this section, we discuss three tests aimed at reinforcing our main results. Specifically,
we show that our results are robust to the inclusion of additional years as controls, changes
in organic search ranking, and changes in consumer and advertiser search behavior. In
Appendix A.3, we also present an alternative identification strategy aimed at identifying
BERT’s effect using variation in query linguistic properties. This strategy relaxes the yearmonth unconfoundness assumption made with our primary identification strategy. We find
results consistent with our year-over-year DD. For the sake of brevity, we focus on replicating
the effects for competitive queries, i.e., an increase in CS across all queries and heterogeneous
adjustments to CPC depending on the length of the query.
Additional Years as Controls Our primary DD analysis uses query data from 2018 to
2020, with July 2018 to February 2019 as control units for July 2019 to February 2020. The
downside of using only one year-month set (2018-2019) as a control window is we don’t know
whether it accurately represents the behavior of the auction market during those months.
(This is equivalent to worrying about using only two states in a traditional state-level DD).
34

As a robustness check, we re-estimate Equations 1.3 and 1.4 using 2016-2017 and 2017-2018
July-February months as additional control groups.
To identify competitive queries with this additional data, we take the mode Competition
Score across all July–October months and years (2016–2019, generating 16 observations per
query) and deem a query non-competitive if the mode is 0. In Figure ??, we present event
analysis market changes for log(CS) and log(CP C) for queries deemed competitive before
BERT.
We repeat our regressions by query length for log(CS) and log(CP C). We report these
results in Tables 1.5 and 1.6. The results are directionally consistent and similar in magnitude
to those estimated in Tables 1.3 and 1.4. The fact that using additional control years leads
to broadly similar results suggests we are reasonably controlling for within-unit seasonal
patterns and that the 2018-2019 auction market outcomes are fairly representative of the
average Google market in the absence of BERT.
Table 1.5: The effect of BERT on log(CS) by query
length.
(1) (2) (3)
Short Medium Long
Post × Treated 0.012∗∗∗ 0.009∗∗∗ 0.017∗∗∗
(0.001) (0.001) (0.001)
Observations 83,765 154,872 36,927
R
2 0.917 0.956 0.905
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Note: Estimated with competitive queries. Regressions include year-group, month, and query fixed effects. Standard errors clustered at the query level are
reported in parentheses. Data from 2016 to 2020.
Changes in Organic Rank Results One potential explanation for our results could be
that advertisers respond to changes in the organic ranking results or to new features added
35

Table 1.6: The effect of BERT on log(CP C) by query
length.
(1) (2) (3)
Short Medium Long
Post × Treated 0.032∗∗∗ −0.021∗∗∗ −0.058∗∗∗
(0.009) (0.006) (0.013)
Observations 83,765 154,872 36,927
R
2 0.629 0.702 0.627
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Note: Estimated with competitive queries. Regressions
include year-group, month, and query fixed effects. Standard errors clustered at the query level are reported in
parentheses. Data from 2016 to 2020.
to the results page that are a function of BERT. For example, in its release note, Google
mentions that BERT will help with featured snippets. Adding more featured snippets at
the top of the page may take away premium space and increase demand for sponsored
search. This could cause an increase in the number of bidders and, potentially, prices.
Moreover, if the top organic result of a search query changes, this may suggest that BERT
has substantially altered the interpretation and identified a new website as more relevant to
a given query. A switch or change in the top URL could lead to a potential allocation effect
and could lead to increasing or decreasing prices [6].
To address this concern, we collect monthly organic rankings data from SEMRush for
the queries in our data. Using this data, we test for changes in the likelihood of a query
resulting in a featured snippet, the number of Search Engine Results Pages (SERP) features
shown on a results page, and whether the top domain has changed. Feature Snippet is
an indicator variable that takes on a value of 1 if the query has a featured snippet at the
top of the page, 0 otherwise. SERP Features Count is the sum of the unique number of
SERP features present on the given query’s results page, including featured snippets. Other
36

SERP features include carousels, top stories, reviews, videos, and knowledge panels.26 In
general, SERP features take up top-of-page space, crowding out advertising, thus potentially
affecting auctions. Finally, Domain Change is an indicator that takes on the value of 1 when
a month’s top URL is different than the previous month’s top URL, 0 otherwise.
Using our primary identification strategy, we test for changes in these front-end organic
rank changes. We report these results in Table 1.7. We find that there is a roughly 11.5%
increase in the likelihood a query contains a featured snippet, the number of SERP features
is generally increasing by about 3.7%, and there is a 4.3% increase in the likelihood that the
top domain link changes after BERT. Google said that BERT would meaningfully change
the result pages of about 10% of queries and increase the presence of featured snippets.
Our estimates seem to capture changes in organic search rankings that are consistent with
Google’s statement.
Table 1.7: Changes in organic results.
Dependent variable:
Feature Snippet log(SERP Features Count) Domain Change
(1) (2) (3)
Post × Treated 0.115∗∗∗ 0.037∗∗∗ 0.043∗∗∗
(0.004) (0.004) (0.005)
Observations 104,228 104,228 104,176
R
2 0.565 0.655 0.207
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Note: Estimated with competitive queries. Regressions include year-group, month, and
query fixed effects. Standard errors clustered at the query level are reported in parentheses.
To test for the possibility that these changes drive our empirical results, we estimate reestimate Equation 1.3 while controlling for organic result changes. We present these results
in Tables 1.8 and 1.9 for CS and CPC, respectively.
26For a complete list of all features, see https://developer.semrush.com/api/v3/analytics/basic-d
ocs/
37

Table 1.8: Changes in log(CS) by query length, controlling
for organic rank changes.
(1) (2) (3)
Short Medium Long
Post × Treated 0.013∗∗∗ 0.009∗∗∗ 0.014∗∗∗
(0.001) (0.001) (0.002)
Feature Snippet 0.001 −0.001 0.005∗∗∗
(0.003) (0.002) (0.002)
Top Domain Change 0.001 −0.00003 0.001
(0.001) (0.001) (0.001)
SERP Features Count 0.0001 0.0002 −0.002∗∗
(0.001) (0.0005) (0.001)
Observations 30,546 57,048 16,582
R
2 0.949 0.969 0.934
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Note: Estimated with competitive queries. Regressions include
year-group, month, and query fixed effects. Standard errors clustered at the query level are reported in parentheses.
We find that the CS estimates are almost the same as those reported in Table 1.3. For
CPC, we find that controlling for organic rank variables slightly attenuates the estimates
reported in Table 1.4. These results suggest that BERT likely did affect organic rank changes,
but these changes don’t appear to be the primary mechanism driving our empirical results.
Changes in Consumer Behavior Another potential explanation is that BERT affects
consumer search behavior, which, in turn, affects CPC and CS. For example, consider the case
in which consumers learn that results are getting better for longer/complex queries because
of BERT, and this changes the types of queries consumers are searching for. Advertises,
then, may start targeting and bidding for these newly popular queries, which can lead to
changes in competition and CPC.
First, it is worth noting that our analysis is over a fixed set of competitive queries.
38

Table 1.9: Changes in log(CP C) by query length, controlling
for organic rank changes.
(1) (2) (3)
Short Medium Long
Post × Treated 0.024∗∗ −0.018∗∗ −0.030∗∗
(0.010) (0.008) (0.015)
Feature Snippet −0.008 −0.018∗ −0.017
(0.015) (0.010) (0.017)
Domain Change −0.014∗∗ 0.005 −0.009
(0.006) (0.005) (0.010)
SERP Features Count 0.014∗∗∗ 0.008∗∗ 0.029∗∗∗
(0.004) (0.003) (0.007)
Observations 30,546 57,048 16,582
R
2 0.696 0.736 0.690
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Note: Estimated with competitive queries. Regressions include
year-group, month, and query fixed effects. Standard errors clustered at the query level are reported in parentheses.
Therefore, it is unlikely that our estimates directly capture this type of effect involving new
search queries. However, it could be the case that the competitive queries in our dataset
are indirectly affected by different search patterns through spillovers. If this was the case,
however, we argue that competition for our queries should have decreased, not increased;
and in response to competition changes, prices would have decreased for all queries.
Second, recall that our empirical analysis focus on short-term effects, which reduces
the likelihood of consumers changing search behavior and advertisers responding to these
changes.27
Third, we perform a test to partially address this concern. We collect data on Google
27In line with the hypothesis that advertiser changes are unlikely to be happening in the short term, [53]
find that it takes months for major advertisers to change bidding strategies following a shift from a second
to a first-price auction format. In addition, even studies from [54] also find that advertisers appear to take
several months to adjust advertising budgets following data privacy regulation shifts.
39

Trends search interest and search volume from SEMRush for each query in our data set. We
then use these variables to account for each query’s popularity.
We re-estimate Equation 1.3, including these variables as control. We report these results
in Tables 1.10 and 1.11. We find that controlling for Google trends and estimated search
volumes does not affect the estimates.
Table 1.10: The effect of BERT on log(CS) by query
length controlling for search interest and search volume.
(1) (2) (3)
Short Medium Long
Post × Treated 0.012∗∗∗ 0.010∗∗∗ 0.013∗∗∗
(0.001) (0.001) (0.002)
Interest 0.0001 0.00002 0.0001
(0.0001) (0.00005) (0.0001)
log(SV) 0.011∗∗∗ 0.003 0.003
(0.004) (0.003) (0.003)
Observations 47,864 88,840 23,807
R
2 0.951 0.970 0.919
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Note: Estimated with competitive queries. Regressions
include year-group, month, and query fixed effects. Standard errors clustered at the query level are reported in
parentheses.
Advertiser Strategies Our model hypothesizes that improvements to a platform’s interpretation algorithm give the platform more informative signals about the search intent of
a query, enabling the platform to estimate more precise relevancy scores for advertisers. In
this model, we assume that advertisers do not adjust in the short-run. This is driven by the
fact that advertisers do not observe the output of BERT and they receive limited information regarding the relevancy of their ads to queries. However, advertisers could still respond
40

Table 1.11: The effect of BERT on log(CP C) by query
length controlling for search interest and search volume.
(1) (2) (3)
Short Medium Long
Post × Treated 0.032∗∗∗ −0.017∗∗∗ −0.041∗∗∗
(0.008) (0.006) (0.013)
Interest −0.0002 0.0001 0.0005
(0.0003) (0.0003) (0.0005)
log(SV) 0.065∗∗∗ 0.041∗∗∗ 0.102∗∗∗
(0.013) (0.009) (0.018)
Observations 47,864 88,840 23,807
R
2 0.686 0.728 0.670
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Note: Estimated with competitive queries. Regressions
include year-group, month, and query fixed effects. Standard errors clustered at the query level are reported in
parentheses.
to changes in match quality or to their beliefs that match quality will improve. Here, we
describe several potential advertiser-focused mechanisms that could be considered but are
ultimately not supported by our empirical findings.
One hypothesis could be that BERT induces more advertisers to enter the sponsored
search market, increases the set of queries advertisers target, or increases advertiser budgets.
Theory would suggest that an advertiser market entry mechanism would lead to marginal
advertisers entering and generally increasing both market size and prices [38]. This could
explain the increase in CPC for short queries, but not for longer queries. It is unclear how an
increase in marginal bidders entering the market would lead to lower CPC in longer query
auctions.
An alternative hypothesis could be that advertisers change their bids in response to
41

the introduction of BERT. Extensive work finds that improved targeting technology for
advertisers generally leads to higher bids because advertisers are better matched [3, 7, 4].
Better targeting has empirically led to higher bids and more relevant advertising (higher
CTR), but ultimately lower CPC due to market thinning reducing auction competition [4].
In this mechanism, the decline in the number of bidders is what drives prices down [3].
However, empirically, we observe an increase, not a decrease, in the number of bidders. This
observation would hamper the main explanation (declining number of bidders) for why prices
decline in this targeting mechanism. In fact, it would suggest that advertiser bids should
further increase due to more bidders in the market [28]. Theoretical results combined with
our empirical observations that the number of bidders is uniformly increasing would suggest
that bids should also increase. Yet, we observe that CPC decreases as the length of the
query increases.
Improved relevancy scores are the primary mechanism within the sponsored search market
that would allow for lower CPC and still make the platform better off. This would enable the
platform to prioritize highly relevant advertisers and increase overall profits even when CPC
declines. If advertisers were lowering their bids for long query auctions and there weren’t
improvements in relevancy, the platform would leave money on the table. This is unlikely.
To conclude, while it is likely that advertisers will adjust their strategy over time due
to better interpretation algorithms, existing theories do not seem to explain the short-term
empirical observations we document.
1.6 Conclusion
Advancements in NLP research have significantly improved search engine programmatic
interpretation abilities. Yet, little is known about how changes to query interpretation
algorithms can affect sponsored search markets. In this paper, we develop a theoretical
auction model to address this question. In our model, a seller uses an algorithm to interpret
42

queries. Queries are uniquely defined by both a topic and context. Context is a distinct set of
linguistic information that is fundamentally different from a query’s topic and is only present
in certain queries. Signals across these dimensions are used to estimate advertiser relevancy
scores. Our model finds that when a platform improves the quality of its interpretation
algorithm 1) queries generally see more bidders competing and 2) when improvements to
contextual understanding are large, and context alignment matters to a query’s CTR, CPC
may decline due to more precise relevancy scores differentiating bidders within the auction.
Ultimately, the platform can do a better job of identifying how relevant advertisers are to
query auctions.
We then test our model predictions by studying how Google’s October 2019 rollout of
BERT affected sponsored search auction competition and prices. We find significant market
changes: the average number of bidders uniformly increases across queries, the supply of advertising space increases, CPC decline for context-rich queries and increase for context-poor
queries. These findings are supported by two identification strategies and several robustness
checks and are consistent with our model’s predictions.
Advertisers are frequently left in the dark when search engines like Google release significant platform updates. Our paper provides a rigorous theoretical and empirical study
to help practitioners and academics better understand how changes to query interpretation
algorithms impact short-term sponsored search market prices and competition. Our paper
also contributes to the growing literature on the economic consequences of AI and LLMs
in the search engine ecosystem. As LLMs continue to develop and increase in sophistication, our paper sheds light on the types of information these models are learning and what
implications this has for markets.
43

Chapter 2
The Financial Consequences of Legalized Sports Gambling
2.1 Introduction
In 2018, the U.S. Supreme Court ruled that the Professional and Amateur Sports Protection
Act (PASPA), which prohibited states from authorizing and regulating sports gambling, was
unconstitutional. Since the ruling, 38 states have legalized some form of sports gambling.
Before this, almost all legal gambling in the U.S. came in the form of tribal casinos with
limited gaming options, commercial casinos in a small number of jurisdictions, and state
lotteries [55]. In this environment, survey data suggested that roughly 75-80% of Americans engaged in some gambling over a year, with roughly 10% gambling twice per week or
more [56]. The new availability of legal sports betting and growth in mobile accessibility
represent a substantial increase in gambling accessibility. Between 2018 and 2023, nearly
$300 billion has been wagered via newly legalized sports gambling markets, with most bets
flowing through online channels.1
While for many, gambling is a relatively inexpensive and generally harmless form of recreation, there is a fraction of so-called “problem gamblers,” for whom gambling is associated
1See: https://www.legalsportsreport.com/sports-betting/revenue/
44

with a range of serious harms [57]. These include financial stress, disruption of family life
and relationships, health problems, worsening of job performance, criminal activity, and even
suicide [58, 59, 60]. The bulk of prior research into the factors associated with problem gambling comes from the period before legalized sports gambling and, therefore, has focused on
either commercial casino gambling or illegal gambling [59]. In addition, it’s unclear whether
to view negative correlations between gambling and health from prior research as causal, as
unobserved underlying factors, such as psychological or environmental factors, could drive
both.
This paper studies the causal impact of legalized sports gambling (LSG) on consumer
financial health using the variation in legalization across states and time following the stateby-state legalization of sports gambling during the period 2018–2023. To do so, we leverage
data from the University of California Consumer Credit Panel (UC CCP), which contains
detailed financial information from a nationwide credit bureau for a sample of roughly 7
million U.S. adults. This data includes credit scores, credit card balances, loan delinquency
information, and many other measures of financial health.
We study the impact of sports gambling on a set of key financial health indicators. We
first test for consumer credit score changes, an overall summary indicator of a person’s
financial health or creditworthiness. Next, we measure changes in indicators associated with
consumers taking on problematic levels of debt: bankruptcies, total debt collections, use of
debt consolidation loans, credit card delinquencies, and auto loan delinquencies.
We consider two definitions of treatment. First, we focus on all states that implemented
LSG, with the treatment date being the first month in which any type of sports gambling
became legal (online or offline). Next, we differentiate between sports gambling that occurs
offline, at specified retail locations such as casinos, and sports gambling that occurs online,
typically via mobile apps. In doing so, we define an additional treatment focused on online
accessibility and consider states that legalized online gambling at some point (some time in
addition to offline gambling) and use the first date when betting was available online as the
45

treatment start date.
Our empirical strategy leverages the staggered state-by-state rollout of legal sports gambling and compares how financial outcomes evolve in treated states compared to states that
did not implement legal sports gambling or did so at a later date. The primary challenge
in isolating the causal effect on consumer financial outcomes is the possibility that the decision by state policymakers to legalize sports betting is correlated with unrelated state-level
trends in economic conditions, budgetary conditions, or other policies.2 We use fixed effects
to control for state-level time-invariant features and national time trends. Because treatment
is staggered and treatment effects are potentially heterogeneous in time and across groups,
we follow best practices in the estimation by employing the estimator proposed in [61].
This estimator aggregates comparisons of treated and not-yet-treated states and allows us
to easily estimate dynamic treatment effects and test for parallel trends across states in the
pre-treatment data.3
We then separately estimate each treatment’s average treatment effect for the full population. We find that for all states that implemented LSG, we observe a small but significant
decrease in the average credit score. In states that allow online/mobile gambling, the decrease
is roughly three times larger, suggesting that legal sports gambling does worsen consumer
financial health, especially so when mobile access is allowed. Next, we turn to signs of problematic debt loads. For the full set of states, we find that only one of our measures (auto loan
delinquencies) increases by a statistically significant amount. By contrast, when we focus
on states with online access to gambling, we also find a roughly 10% increase in bankruptcy
likelihood and an 8% increase in debt collection amounts, both of which are statistically
significant. These effects generally appear roughly two years after when gambling became
2For example, a state may implement LSG because of revenue shortfalls and a need for the additional
tax revenue that LSG may generate, and these states may also be more susceptible to economic shocks.
3
In addition, we test for differences between treatment states and control states for whether they offered
different levels of financial assistance or social insurance programs like unemployment insurance before,
during, or after the 2020 COVID pandemic. We find no differences except that legalized gambling states
offer persistently more generous unemployment insurance. We also show that local trends in economic
conditions are not significantly related to the timing of gambling legalization.
46

legal.
Next, we examine the heterogeneous impact of LSG. We use the highly granular consumer
credit data to examine effects separately for men and women, old and young male panelists,
high- vs. low-income male panelists. We find relatively few significant differences in outcomes
across these groups, although we find a pattern suggesting that effects are strongest for
low-income younger men. Finally, we test for heterogeneous treatment effects across pretreatment credit score categories (sub prime, prime, and super prime) for changes in credit
scores and bankruptcies. We find that the subprime category is where we the most significant
financial stress signals occur (higher bankruptcy rates, greater declines in credit scores).
While many consumers get real enjoyment from legal gambling, and states benefit in the
form of additional tax revenue, there is a corresponding concern that the introduction of
sports gambling and the ease at which consumers can now bet online are negatively harming
consumer financial health. Our paper provides evidence that this concern is well founded
by quantifying the extent to which the recent aggressive expansion of gambling accessibility
impacts consumer financial health.
2.2 Literature: Sports Gambling and Financial Health
There is a large literature studying gambling and its effect on consumers. Past research on
gambling has highlighted the potential negative consequences that it can have on individuals’ financial health. These studies have shown that excessive gambling is associated with
financial difficulties, debt accumulation, rising mortgage delinquencies, and bankruptcy [62,
63, 60].
Prior research finds that the associated financial harms of gambling vary across consumers. Demographic groups, such as young adults and individuals with lower socioeconomic
status, are often more vulnerable to the adverse effects of gambling on financial well-being [64,
65, 66]. In addition to demographics, factors like impulsivity and psychological distress have
47

also been identified as risk factors for financial harm associated with betting [67, 68]. In our
paper, we use these findings to study the heterogeneous impact of sports gambling on groups
of consumers who are more likely to be affected.
The legalization of sports gambling led to the rise of online sports betting platforms and
mobile gambling apps, which made gambling more accessible. Past research has found that
ease of access may exacerbate gambling-related financial harm, as individuals can place bets
anytime and anywhere, leading to increased gambling frequency and expenditure [67, 69, 70,
71].
Prior economic research on gambling has placed an emphasis on studying the relationship between gambling and personal bankruptcies. Using county-level bankruptcy results,
existing economic literature has generally found that access to casinos and state lotteries
lead to increasing bankruptcy rates [72, 73, 74, 75].
To mitigate the negative effects of sports gambling, regulatory interventions, such as
mandatory self-exclusion programs, pre-commitment tools, and advertising restrictions, have
been implemented [69, 76]. Responsible gambling initiatives, including education and awareness campaigns, aim to promote informed decision-making and prevent excessive gambling
behavior [77, 78].
A recent working paper [79] attempts to estimate the causal effect of sports gambling
legalization on tax revenue, irresponsible gambling behavior, problem gambling hotline calls,
and suicides. Using an individual-level credit card panel dataset, they find that legalization
increases gambling and irresponsible gambling behavior. The authors also find evidence that
online sports betting legalization significantly increases problem gambling hotline calls but
find inconclusive evidence that sports betting increases suicide rates.
We add to this recent literature by studying the causal effect of sports betting legalization on consumers’ financial health. We do so by using a dataset tracking a wide range
of individual-level financial measurements for a representative sample of U.S. consumers.
Our data allows us to study changes in consumer financial stress beyond credit card data,
48

therefore complementing the findings discussed in [79].
2.3 Background and Data
This section provides an overview and history of state-level legal sports gambling regimes.
We then introduce our primary data source, the University of California Consumer Credit
Panel (UC CCP), and provide some high-level summary statistics for this data.
2.3.1 Background on Legal Gambling
In May 2018, the Supreme Court overturned the Professional and Amateur Sports Protection
Act (PAPSA), deeming it unconstitutional and infringing on states’ rights. This opened the
door for individual states to legalize and regulate sports betting. Before this ruling, only
Nevada continued to offer legal sports betting. Within just one month of this ruling, Delaware
and New Jersey launched retail sports betting at casinos and racetracks, with many states
following in the years since. As of February 25, 2025, 39 states and the District of Columbia
have legalized some form of sports betting.4
There are a wide variety of different state-level regulations and tax structures for sports
betting. Perhaps most notable is the decision of whether to allow online (typically mobile)
betting or whether to require bets to be placed in person at a qualified location. Currently,
33 states and DC choose to allow some form of online betting accessibility, while the other
eight states with LSG only allow retail betting, i.e., betting at a physical location.5 As
shown in Supplementary table B.1 in Appendix B.1, many states in our data legalized retail
betting before mobile betting, though time lags between the two types of legalization are
often small. Other policy choices states make include whether advertising is allowed and
how, what types of entities are licensed to offer sports betting, what tax rate is levied, and
4See: https://www.americangaming.org/research/state-gaming-map/.
5A few states (Tennessee and Wyoming) exclusively offer online access. Several tribal lands in Oregon
offer offline sports betting access, but these do not fall directly under the Oregon state government.
49

on what tax base. In Figure 2.1, we show how sports betting handles (dark series) have
grown over time along with the number of states with LSG (grey series).6
In Supplementary
table B.2 in Appendix B.1, we report handles by state and for both online and retail channels.
0
3
6
9
0
10
20
30
Jan 2019 Jan 2020 Jan 2021 Jan 2022 Jan 2023
Handle (in Billions)
Number of Legalized States
Number of Legalized States Handle (in Billions)
Figure 2.1: Monthly sports handle in billions and the number of legalized states. The left
axis is the sports handle, and the right axis is the number of legalized states. Our data does
not contain handles for states where tribal lands run the offline sports gambling market.
2.3.2 Consumer Credit Data
Our primary dataset is the University of California Consumer Credit Panel (UC-CCP). It
contains anonymized individual-level records of a nationally representative 2% sample of
U.S. adults with a credit report (i.e., roughly 7 million panelists). Data is tracked from 2004
to the present day. For each year, we observe records from March, June, September, and
December.7 We observe demographic characteristics for nearly all individuals. This includes
information such as age, gender, and ethnicity. The panel also contains modeled and/or
self-reported information such as occupation, if the individual owns a home, marital status,
and if the individual has children.8
6We obtained sports betting handles data in June 2023 from https://www.legalsportsreport.com/sp
orts-betting/revenue/.
7We refer to these observations as quarterly observations or quarters.
8See https://www.capolicylab.org/data-resources/university-of-california-consumer-credi
t-panel/ for additional discussion of data.
50

We observe account information across all open and closed accounts for each individualmonth combination. This includes mortgages, student loans, auto loans, credit cards, secured
and unsecured loans, debt consolidation loans, debt buyer accounts, and collections. Information includes when the account was opened, most recent account balance, most recent
payment amount, amount past due, if the account is delinquent, what type of business the
account is associated with, and, in the case of loans, various loan categories such as personal
or medical.
We restrict our panel to individuals who maintain at least one active account and are
not deceased. We also remove any individual who moved across states to prevent treatmentcontrol spillovers and any individual whose location or gender information is not present in
the data. Our final dataset contains observations for 4, 382, 529 unique individuals and 90
million quarterly observations over seven years, from March 2016 to June 2023.
2.3.3 Types of Gambling Access
We study the causal impact of gambling access on financial health using the treatment effects
framework, and consider two types of treatment definitions. The first is meant to capture
the overall effect of any type of gambling legalization and defines a state as treated after the
first month a state begins reporting state tax revenue from any sports gambling operations.
In our analysis, we call this group “General Access.” Note that states may offer online,
offline, or both gambling channels. The rollout of channels may occur at different times. For
example, in Pennsylvania, casinos began accepting offline wagers in November 2018, with
online channels beginning in May 2019. In this case, we define Pennsylvania’s treatment
status to begin in January 2019 (the first month in our dataset after November 2018).
Our second treatment status is meant to capture the specific effects of the legalization of
online gambling. For this, we consider only states that eventually legalize online gambling. In
our analysis, we call this group “Online Access.” In our data, 23 states and DC legalized some
51

form of online sports betting as of June 2023.9 Treatment begins in the first month after the
state implements online betting. Additionally, we removed states that exclusively offer offline
gambling venues. This removes nine states (Delaware, Mississippi, Montana, New Mexico,
North Carolina, South Dakota, Washington, and Wisconsin), leaving us with 40 states and
DC that are either eventually treated with online sports gambling access or are never treated.
In some cases, states introduce offline gambling before online gambling (10 states). The lags
between offline and online rollout are small for most states, excluding Arkansas and New
York. No states with both online and offline access implemented online access before offline
access. Lastly, three states in our data only offered online access (Tennessee, Wyoming, and
Virginia).
A full list of treated states and their legalization timing can be found in Supplementary
table B.1 in Appendix B.1. Start dates are calculated based on the first month the state
began collecting tax revenue.10
2.3.4 Primary Outcomes of Interest
We focus our analysis on six outcomes designed to capture overall financial health and the
presence of excessive debt.
2.3.4.0.1 Overall financial health A credit score is a numerical expression based on
a level analysis of a person’s credit files, representing the creditworthiness of an individual.
Essentially, it is used by lenders to evaluate the risk of lending money to consumers and
to mitigate losses due to bad debt. Decreases in consumer credit scores represent lower
consumer creditworthiness. Our data observes a consumer’s credit score for a given quarter.
9Since beginning the project, ten more states have introduced online betting access. For example,
Delaware introduced online betting at the end of 2023. However, in our data, we treat it as retail-only
because our data does not go that far.
10We do not include Nevada in our analysis because it offered sports betting prior to 2018.
52

2.3.4.0.2 Indicators of excessive debt Next, we consider five measures of excessive
debt. The first is bankruptcy, which captures instances where consumers do not think they
can reasonably repay outstanding debts and need to manage or restructure their finances to
pay off debts over time. Filing for bankruptcy is a serious financial decision that requires a
consumer to go to bankruptcy court. It seriously harms a consumer’s credit score and is a
significant indicator of financial stress.
The second is the total amount of debt on an account that has been sent to collections.
This is a measure of how much unpaid debt that the consumer’s creditors have assigned to
collection agencies. When a consumer misses payments, or a lender does not think it will
receive payment on a debt, the lender may coordinate with a collections agency to manage
the debt collection process or sell the debt to a collections agency. Any missed debt can be
sent to collections. A debt going to collections can seriously harm a consumer’s credit score.
In our data, we observe each consumer’s collection amounts on file. Unfortunately, we do
not observe which specific debts the collections come from. We only know how much the
collection amount is for and whether it is present on the consumer’s account.
The second is the use of debt consolidation loans, a financial strategy for managing
and reducing debt by combining multiple debts into a single, more manageable loan. This
approach is often used by individuals with high debt levels with various creditors, particularly
if they face high interest rates from loans or credit cards. Prior survey and observation work
finds that gamblers with high levels of debt may use debt consolidation loans [80]. Given
their low usage rate and the association between the loan product and problem gambling,
we focus on changes in the likelihood individuals take on debt consolidation loans.
Finally, we study credit card and auto loan delinquencies, which indicate missed payments
and are a strong sign of financial distress. Delinquencies for credit cards and auto loans will
typically be reported if a consumer has missed 1-2 monthly payments. We analyze changes
to the number of actively delinquent credit card accounts and auto loans on file to measure
failing payments.
53

In Table 2.1, we present summary statistics from the pre-legalization period for our six
dependent variables.
Table 2.1: Pre-treatment summary statistics.
Dependent Variable Min. 1st Qu. Median Mean 3rd Qu. Max.
Credit Score 300 649 731 714.657 798 850
Pr(Bankruptcy) 0 0 0 0.00072 0 1
Collections 0 0 0 365.071 0 2, 084, 548
Pr(CC Delinquency) 0 0 0 0.0185 0 1
Pr(Auto Loan Delinq.) 0 0 0 0.014 0 1
Pr(Consol. Loan) 0 0 0 0.00066 0 1
2.4 Empirical Strategy
We exploit the staggered legalization of sports gambling across states to measure its impact
on consumer financial health. We do so by implementing a difference-in-differences (DD)
identification strategy that compares changes in average outcomes before and after legalization relative to the changes in these outcomes for states that did not yet legalize sports
gambling or that never legalized it over the same period. While DD is typically implemented
using a Two-Way Fixed Effect (TWFE)—county and year-quarter in our case—recent literature has pointed out some shortcomings of this estimator [81]. In particular, in cases
where there is treatment heterogeneity by treatment groups or time, TWFE can generate
biased estimates. To avoid this issue, econometricians have developed a set of alternative
estimators [81, 61, 82]. In this paper, we rely on the proposed method by [61], which we
refer to as CS estimator. We aggregate our data to the county-level and weight county-level
observations by the average number of individuals present in the data in 2015. This is done
for computational efficiency purposes using the CS estimator. The estimated Average Treatment on the Treated (ATT) can be interpreted as the average change in treated individuals’
financial outcomes.
54

Since states decide whether to legalize sports gambling, the main concern with our identification strategy is that unrelated trends in consumer financial outcomes correlate with
state-level decisions to implement legalization.
2.4.1 Identification Checks
Since states decide whether to legalize sports gambling, the primary concern is that unrelated trends in consumer financial outcomes correlate with state-level decisions to implement
legalization. Of particular concern would be if states that choose to legalize sports betting
to generate revenue do so because they have budgetary problems and, consequently, when
economic shocks such as the COVID pandemic arise, are less able to provide social assistance.
We test for this possibility in two ways. First, we test for cross-sectional differences
between treated and control states across various social assistance programs and COVID-19
fiscal responses. We compare states across 13 policies, as shown in Table 2.2. We find no
significant differences in these policies except for the duration of unemployment insurance,
which is consistently higher among treated states both pre- and post-pandemic. There is little
time variation in Unemployment Insurance duration across the periods studied among treated
states. Nevertheless, any declines in consumer financial health observed among treated states
could be understated due to those states’ more generous unemployment policies.
The second test we perform relates to the timing of gambling legalization. We estimate
the relationship between this timing and local economic indicators, namely weekly wages,
the quarterly unemployment rate, and the number of COVID cases. We do so using a Cox
Hazard model with all states in our data. We present these results in Table 2.3. We find no
significant relationships between these variables and the timing of legalization, alleviating
concerns that such factors may plausibly correlate with treatment timing and our dependent
variables.
55

Table 2.2: Fiscal policies of treated and control states.
Policy Treated Control t
2020 UI maximum amount ($) 471.4 490.85 .467
COVID Expanded eligibility for UI (high-risk individuals) .2333 .1905 -.359
COVID Expanded eligibility for UI (lost childcare/school) .4333 .2857 -1.064
COVID Expanded eligibility for UI (quarantined or caregiver) .8333 .8095 -.215
COVID Extended UI duration .0667 .0476 -.279
2021 UI maximum duration (weeks) 25.73 23.85 -2.348
January 2020 UI maximum duration (weeks) 25.2 22.67 -2.238
July 2020 UI maximum duration (weeks) 25.73 23.52 -2.255
Reinstated one week waiting period for UI .5 .6190 .829
Reinstated work search requirement for UI .4667 .6667 1.413
Stopped Participating in Federal UI Programs .4 .4762 .531
Waived work search requirement for UI .9333 .9524 .279
Weekly UI maximum amount with extra stimulus ($) 1071.4 1090.9 .467
2.5 Aggregate Effects
This section presents aggregated (across all consumers) event study estimates covering eight
quarters before and 16 quarters after the treatment.By doing so, we can validate the parallel
trends assumption and observe the evolution of the treatment over time. At the end of the
section, we present ATT estimates for our treatment conditions and all consumers.
2.5.1 Overall Consumers’ Financial Health
Credit score The first outcome we study is the average consumer credit score. As we
discussed in Section 2.3, a credit score is a measure of the overall financial health of a
consumer. It is designed to summarize a consumer’s creditworthiness by predicting their
future default risk based on all the data observed in credit reports. In Figure 2.2, we present
the event study estimates for changes in the average credit score by treatment status.
First, it is worth noting that we observe largely zero pre-treatment period estimates. This
suggests that before the treatment, treated and control states’ average credit scores evolved
56

similarly, supporting the validity of our identification strategy.11
Figure 2.2: The effect of sports gambling legalization on consumer credit score.
In the post-treatment period, we see that general access to sports betting decreases the
average credit score. After about four years, the average credit score declines by about 0.8
points. This negative effect is stronger for Online Access treatment. During the same time
window, the average credit score drops by roughly 2.74 points with access to online gambling,
or close to three times the decline we observe for general access to sports gambling.
2.5.2 Indicators of Excessive Debt
Next, we analyze changes in indicators of excessive debt. This analysis can help us better
understand the reasons behind the decrease in the average credit score we observe.
Bankruptcies In Figure 2.3a, we present the event study estimates for bankruptcies by
treatment conditions. We find that while the general accessibility to sports betting leads to
insignificant changes to bankruptcy filing, online access significantly increases the likelihood
of bankruptcy filing. We also see that the increase in bankruptcy filings occurs only after
a lag of roughly two years. This is expected given that bankruptcies are often a lastresort option for consumers, and they would likely occur only after consumers experience
significant financial stress. Three to four years after the legalization of online sports gambling,
11This is the case for all our study outcomes.
57

we observe that the likelihood of bankruptcy filing increases by as much as 25-30% when
compared to pre-treatment levels.
Collections In Figure 2.3b, we present the event study estimates for changes in the amount
of debt in collection on account. We observe a significant increase in collections when focusing
on online accessibility, translating to a roughly 7.5% average increase. Given that the pretreatment period average collection amounts were about $360, our estimate translates to a
roughly $30 increase in the average amount of debt in collections per consumer due to sports
betting.
A natural question is whether the increase in collections on account with online accessibility is coming from extensive or intensive margins. It could be the case that more unique
individuals are generating collections (extensive), or individuals with collections and generating more collections (intensive). To test for extensive margins, we analyze the likelihood
an individual has a collection on file. We find that there is a statistically significant increase
(p < 0.01) of about 1.3% in the likelihood an individual has a collection on file, suggesting
that collection rates are increasing across individuals in states with online sports gambling
access relative to states that don’t. To analyze intensive margins, we then conditioned
our dataset to individuals that had any collections on file in the pre-supreme court ruling
period (January 2016 to March 2018) and re-estimated changes in collection amounts on
account. We find a statistically significant increase of close to 15% among these individuals
(p < 0.001). These findings suggest that sports gambling is leading to both extensive and
intensive margin changes in collections on account.
Credit card delinquency In Figure 2.3c, we present event study estimates for changes
in the probability of an individual having a credit card delinquency on file. While initially
no effect is present, we find that starting roughly three years after LSG implementation,
credit card delinquencies appear to be declining, particularly in states with online access
to sports betting. These findings suggest that sports gambling does not appear to affect
58

(a) The effect of sports gambling legalization on
bankruptcy filing likelihood.
(b) The effect of sports gambling legalization on
collections on account amount.
(c) The effect of sports gambling legalization on
the likelihood an individual has a credit card
delinquency.
(d) The effect of sports gambling legalization on
the likelihood of having an auto loan delinquency
on file.
(e) The effect of sports gambling legalization on
the likelihood of having an open debt consolidation loan.
Figure 2.3: Changes in bankruptcies, collection on account, credit card delinquency likelihood, auto loan delinquency likelihood, and debt consolidation usage.
59

consumers’ financial health through credit card debt directly but through harder forms of
loan accessibility.
Auto loan delinquency In Figure 2.3d, we present event study estimates for changes in
the probability of an individual having an auto loan delinquency on file. For both forms of
treatment, we see that auto loan delinquency likelihoods are significantly increasing. Compared to pre-treatment averages, this leads to a roughly 20% (6%) increase in delinquency
likelihood with General Access (Online Access) treatment.
Average cumulative credit card payment in our data is $123 across our data time window.
In contrast, cumulative auto loan payments are roughly $530. Auto loans tend to be fixed
with lower interests and less flexibility relative to credit cards. Banks also have greater
flexibility to negotiate over missed credit card payments to avoid credit card delinquencies
relative to auto loans. Auto loan interests are generally lower.
Debt consolidation Problem gamblers may use debt consolidation loans to consolidate
and manage high-interest loans (e.g., credit cards). Given that these types of loans are lastresort measures to manage debt, similar to bankruptcies, we expect to see a delayed effect
post-introduction of sports gambling. In Figure 2.3e, we present the event study estimates
for the likelihood of an individual having an open debt consolidation loan. We see rates
increase after roughly two years (eight quarters), translating to a statistically insignificant
average ATT of roughly 0.01%. This translates to about a 7.5% increase in the likelihood of
an individual opening a consolidation loan compared to pre-treatment period average rates.
2.5.3 Overall ATTs and Summary of Results
In Table 2.4, we report ATT estimates for all our dependent variables across the event study
time windows (eight periods pre-treatment and 16 periods post treatment). Given that we
analyzed several dependent variables, we use the Benjamini-Hochberg procedure to maintain
a 5% false discovery rate and account for multiple hypothesis testing [83].
60

While sports betting accessibility appears to be financially harming consumers, online
access drives most of the effect we observe. Furthermore, the effect of sports betting does
not appear to be driven by higher credit card delinquencies but by increased exposure and
use of hard debts such as consolidation loans, secured loans, and bankruptcies. The fact
that credit card delinquencies are unaffected or lower is likely due to financial institutions
trying to mitigate their exposure to risk by lowering credit limits. Despite this, we observe
consumers missing payments for other loans and products, leading to increased collections
and auto loan delinquencies.
2.5.4 Robustness Checks
iGaming One potential concern could be that our results are driven by the introduction of
iGaming, not sports betting. iGaming is other forms of online gambling, including lotteries,
slots, and casino games like poker. In our period six states had some form of legalized
iGaming accessibility (Pennslyvania, West Virginia, New Jersey, Michigan, Delaware, and
Connecticut). The market for iGaming is smaller but still generates substantial revenue. In
November 2024, iGaming company revenue was over $800 million. 12 To test the robustness
of our Online Access results, we drop these states and re-estimate the same models used to
generate results in Table 2.4. In Table 2.5 we present ATT estimates over the time period.
We find that our results continue to hold, which suggest these findings are not driven by
iGaming accessbility, but by sports gambling.
Coarsened Exact Matching While our hazard model in Table 2.3 does not show that
these various economic and demographic measures correlate with the timing of sports betting
legalization, we may still worry that they could correlate with some confound that impacts
both adoption and our dependent variables. For example, if Covid-19 affects states and
counties at particular income thresholds, and this impacts both sports betting adoption and
12https://www.americangaming.org/resources/aga-commercial-gaming-revenue-tracker/
61

our outcome variables, then our results could be confounded by the interaction between
income and Covid-19. To strengthen the causal interpretation of our results, we consider
matching counties on the economic and demographic characteristics captured in our hazard
model.
Because of the inherent staggered setting and the lack of a post period for never-treated
units, we elect to match on pre-period observations that exist for all counties. Specifically, we
take 2015-2017 (inclusive) observations for each of our economic and demographic variables
and take the average for each county. A county is defined as treated if it is eventually
treated, otherwise it is deemed a control unit (i.e., never treated). We then use coarsened
exact matching [84] with one-to-one restriction within bins to match control counties to
(eventually) treated counties based on 2015-2017 average observable characteristics. Since
we drop states with only retail access to sports betting for the Online Access treatment,
we run two separate matching procedures for each treatment status (General Access and
Online Access). In Table 2.6, we present differences in observable characteristics between
treated (General Access) and control counties before and after matching. Matching leaves us
with 696 eventually treated counties and 696 never treated control counties. (1141 treated
counties and 571 control counties are dropped).
Similarly, in Table 2.7, we present differences in observable characteristics between treated
(Online Access) and control units. This leaves us with 509 treated and 509 control counties.
(758 control counties are dropped and 830 treated counties are dropped).
The challenge with matching using only 2015-2017 observable characteristics is that it
does not necessarily guarantee similar observable characteristics in the periods just before
treatment. This is done due to the staggered setting varying the underlying post-period for
treated and control units, making traditional pre-treatment matching infeasible. One can
view our matching as a “light” matching.
This strategy works if county observable characteristics evolve similarly and do not “drift”
apart after the 2015-2017 periods. If this holds, then we should observe that the matched
62

counties share similar observable characteristics just before treatment occurs. This is testable
with our data. Taking the period just before treatment (-2), we can look at the differences
in observable characteristics between the matched counties. In Table 2.8 we show that the
full panel set of counties show observable differences right before treatment. However, once
matched, these differences are insignificant, as shown in Table 2.9. These results suggest that
counties do not “drift” apart from one another on observables with our matching strategy.
After matching, we re-run our primary CS estimator using this subset of counties. Results
are presented in Table 2.10. We find results that are consistent with our main, unmatched
specification.
2.6 Conclusion
In this paper, we estimate the causal effect of sports gambling accessibility on consumer
financial health by exploiting the recent legalization of sports gambling across U.S. states.
We focus on changes to consumer credit risk and the composition of loans taken out by
consumers across general sports betting accessibility and online accessibility.
Overall, we find that the legalization of sports gambling decreased consumer financial
health. These results seem to be particularly pronounced when states legalize online betting,
suggesting that the ease of access to gambling increases the problems associated with it.
Our paper provides a better understanding of how the legalization of sports gambling
negatively affects consumer financial health. While many states may have opted for legalization with the hope of increasing tax revenue, the negative effect we document can partially
offset tax revenue benefits as more consumers’ financial health deteriorates.
While many consumers get real enjoyment from legal gambling, and states benefit in the
form of additional tax revenue, there is a corresponding concern that the introduction of
sports gambling and the ease at which consumers can now bet online are negatively harming
consumer financial health. Our paper provides evidence that this concern is well founded
63

by quantifying the extent to which the recent aggressive expansion of gambling accessibility
impacts consumer financial health.
64

Table 2.3: Hazard model test of treatment likelihood with all states.
Dependent variable:
time = start
Log(COVID Cases) −0.476
(1.121)
Percentage Higher Educated 4.668
(9.896)
Percentage Low Income 5.323
(24.743)
Percentage Unemployed 63.689
(67.313)
Log(Median Income) −0.666
(4.965)
Log(Total Population) −0.348
(0.452)
Men Under 45 (Percent of Population) −17.293
(31.159)
Previous Year State Budget, Scaled 0.0002
(0.006)
Log(Cumulative State Date) 0.113
(0.563)
3-Year Rolling State Budget Deficit, Scaled −0.002
(0.004)
Observations 1,070
R
2 0.005
Max. Possible R2 0.179
Log Likelihood −102.493
Wald Test 5.250 (df = 10)
LR Test 5.465 (df = 10)
Score (Logrank) Test 5.414 (df = 10)
Note: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01
65

Table 2.4: Overall ATT estimates.
(1) (2)
General Access Online Access
Overall Financial Health:
Credit Score -0.804∗∗∗ -2.74092∗∗∗
(0.1793) (0.28174)
Excessive Debt Indicators:
Pr(Bankruptcy) -0.000004 0.00009∗∗∗
(0.00003) (0.00002)
Collections 0.00767 0.0748∗∗∗
(0.01315) (0.01293)
Pr(Auto Loan Delinquency) 0.00323∗∗∗ 0.00089∗∗
(0.00065) (0.00038)
Pr(CC Delinquency) -0.00008 -0.00013
(0.00021) (0.0003)
Pr(Cons. Loan) 0.00005 0.00005
(0.00003) (0.00003)
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Note: Each row shows the coefficients from a separate [61] estimation
for the dependent variable shown on the left. Column (1) defines treatment based on any form of legal sports gambling and column (2) defines
treatment based on access to mobile betting. Data is aggregated at the
county level, and therefore, standard errors are clustered at the county
level. All p-values are adjusted using the Benjamini-Hochberg procedure
to account for multiple hypothesis testing.
66

Table 2.5: Overall ATT estimates excluding
iGaming States.
(1)
Online Access
Overall Financial Health:
Credit Score -2.23041∗∗∗
(0.45237)
Excessive Debt Indicators:
Pr(Bankruptcy) 0.00008∗∗
(0.00002)
Collections 0.05729∗∗
(0.0165)
Pr(Auto Loan Delinquency) 0.00092∗∗
(0.00039)
Pr(CC Delinquency) -0.00014
(0.00037)
Pr(Cons. Loan) 0.00004
(0.00004)
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Note: Each row shows the coefficients from a separate [61] estimation for the dependent variable shown
on the left. Data is aggregated at the county level,
and therefore, standard errors are clustered at the
county level.
Table 2.6: General Access: 2015-2017 Observable Differences Before/After Matching.
(1) (2)
Original Sample Matched Sample
log(Population) -0.557∗∗∗ 0.051
(0.058) (0.065)
Young Men Rate -0.007∗∗∗ -0.0007
(0.0009) (0.0012)
Unemployment Rate -0.0002 0.00017
(0.0003) (0.0005)
Poverty Rate -0.0132∗∗∗ -0.002
(0.002) (0.003)
log(Median HH Income) 0.022∗∗ -0.004
(0.009) (0.011)
College Share 0.018∗∗∗ 0.001
(0.003) (0.002)
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
67

Table 2.7: Online Access: 2015-2017 Observable Differences Before/After Matching.
(1) (2)
Original sample Matched sample
log(Population) -0.449∗∗∗ -0.121
(0.063) (0.073)
Young Men Rate -0.007∗∗∗ -0.001
(0.0009) (0.0012)
Unemployment Rate 0.00009 0.0004
(0.0003) (0.0004)
Poverty Rate -0.015∗∗∗ -0.002
(0.002) (0.002)
log(Median HH Income) 0.037∗∗∗ 0.009
(0.01) (0.013)
College Share 0.022∗∗∗ 0.0002
(0.003) (0.003)
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Table 2.8: Observable Differences Between All Counties Just
Before Treatment.
(1) (2)
General Access Online Access
log(Population) 0.182∗∗∗ 0.387∗∗∗
(0.054) (0.052)
Young Men Rate -0.0041∗∗∗ -0.0006
(0.0012) (0.0012)
Unemployment Rate 0.00045 0.00054
(0.000437) (0.00035)
Poverty Rate -0.0129∗∗∗ -0.018∗∗∗
(0.0021) (0.0028)
log(Median HH Income) 0.0455∗∗∗ 0.0661∗∗∗
(0.0091) (0.0089)
College share 0.0189∗∗∗ 0.0153∗∗∗
(0.0023) (0.0024)
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
68

Table 2.9: Observable Differences Between Matched Counties
Just Before Treatment.
(1) (2)
General Access Online Access
log(Population) 0.038 0.033
(0.067) (0.073)
Young Men Rate -0.0009 -0.0006
(0.0011) (0.0012)
Unemployment Rate 0.00088∗ 0.0009
(0.00046) (0.0005)
Poverty Rate -0.0008 -0.0015
(0.0027) (0.0028)
log(Median HH Income) -0.009 -0.018
(0.011) (0.012)
College Share 0.0016 -0.0009
(0.0026) (0.0028)
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Table 2.10: Overall ATT estimates with matched counties.
(1) (2)
General Access Online Access
Overall Financial Health:
Credit Score -0.81785∗∗∗ -2.12829∗∗∗
(0.22235) (0.30198)
Excessive Debt Indicators:
Pr(Bankruptcy) 0.00004 0.0001∗∗
(0.00005) (0.00004)
Collections -0.00009 0.06222∗∗∗
(0.03195) (0.01689)
Pr(Auto Loan Delinquency) 0.00271∗∗ -0.00086
(0.00082) (0.00051)
Pr(CC Delinquency) 0.00022 -0.00009
(0.00036) (0.00041)
Pr(Cons. Loan) 0.00012∗ 0.0001∗∗
(0.00006) (0.00005)
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Note: Each row shows the coefficients from a separate [61] estimation
for the dependent variable shown on the left. Data is aggregated at the
county level, and therefore, standard errors are clustered at the county
level.
69

Bibliography
[1] Ashish Vaswani et al. “Attention is all you need”. In: Advances in neural information
processing systems 30 (2017).
[2] Dirk Bergemann, Tibor Heumann, and Stephen Morris. “Selling Impressions: Efficiency
vs. Competition”. In: (2021).
[3] Dirk Bergemann and Alessandro Bonatti. “Targeting in advertising markets: implications for offline versus online media”. In: The RAND Journal of Economics 42.3 (2011),
pp. 417–443.
[4] Bo Cowgill and Cosmina Dorobantu. “Competition and Specificity in Market Design:
Evidence from Geotargeted Advertising”. In: Columbia Business School Research Paper
101 (2020).
[5] Paul R Milgrom and Robert J Weber. “A theory of auctions and competitive bidding”.
In: Econometrica: Journal of the Econometric Society (1982), pp. 1089–1122.
[6] Simon Board. “Revealing information in auctions: the allocation effect”. In: Economic
Theory 38.1 (2009), pp. 125–135.
[7] Steven Tadelis and Florian Zettelmeyer. “Information disclosure as a matching mechanism: Theory and evidence from a field experiment”. In: American Economic Review
105.2 (2015), pp. 886–905.
[8] Jonathan Levin and Paul Milgrom. “Online advertising: Heterogeneity and conflation
in market design”. In: American Economic Review 100.2 (2010), pp. 603–07.
[9] Patrick Hummel and R Preston McAfee. “When does improved targeting increase
revenue?” In: ACM Transactions on Economics and Computation (TEAC) 5.1 (2016),
pp. 1–29.
[10] Jacob Devlin et al. “Bert: Pre-training of deep bidirectional transformers for language
understanding”. In: arXiv preprint arXiv:1810.04805 (2018).
[11] Martin S Eichenbaum et al. Expectations, Infections, and Economic Activity. Tech. rep.
National Bureau of Economic Research, 2020.
[12] Bryan Bollinger et al. “Educational campaigns for product labels: Evidence from onshelf nutritional labeling”. In: Journal of Marketing Research 59.1 (2022), pp. 153–
172.
70

[13] Jura Liaukonyte, Anna Tuchman, and Xinrong Zhu. “Spilling the Beans on Political Consumerism: Do Social Media Boycotts and Buycotts Translate to Real Sales
Impact?” In: Available at SSRN 4006546 (2022).
[14] Georgios Zervas, Davide Proserpio, and John W Byers. “The rise of the sharing economy: Estimating the impact of Airbnb on the hotel industry”. In: Journal of marketing
research 54.5 (2017), pp. 687–705.
[15] Chiara Farronato and Andrey Fradkin. “The welfare effects of peer entry: the case
of Airbnb and the accommodation industry”. In: American Economic Review 112.6
(2022), pp. 1782–1817.
[16] David Card and Alan B Krueger. “Minimum wages and employment: a case study of
the fast-food industry in New Jersey and Pennsylvania: reply”. In: American Economic
Review 90.5 (2000), pp. 1397–1420.
[17] Esther Duflo. “Schooling and labor market consequences of school construction in
Indonesia: Evidence from an unusual policy experiment”. In: American economic review
91.4 (2001), pp. 795–813.
[18] Anders Akerman, Ingvil Gaarder, and Magne Mogstad. “The skill complementarity of
broadband internet”. In: The Quarterly Journal of Economics 130.4 (2015), pp. 1781–
1824.
[19] Benjamin Edelman, Michael Ostrovsky, and Michael Schwarz. “Internet advertising
and the generalized second-price auction: Selling billions of dollars worth of keywords”.
In: American economic review 97.1 (2007), pp. 242–259.
[20] Anindya Ghose and Sha Yang. “An empirical analysis of search engine advertising:
Sponsored search in electronic markets”. In: Management science 55.10 (2009), pp. 1605–
1622.
[21] Sha Yang and Anindya Ghose. “Analyzing the relationship between organic and sponsored search advertising: Positive, negative, or zero interdependence?” In: Marketing
science 29.4 (2010), pp. 602–623.
[22] Oliver J Rutz and Randolph E Bucklin. “From generic to branded: A model of spillover
in paid search advertising”. In: Journal of Marketing Research 48.1 (2011), pp. 87–102.
[23] Ron Berman and Zsolt Katona. “The role of search engine optimization in search
marketing”. In: Marketing Science 32.4 (2013), pp. 644–651.
[24] Thomas Blake, Chris Nosko, and Steven Tadelis. “Consumer heterogeneity and paid
search effectiveness: A large-scale field experiment”. In: Econometrica 83.1 (2015),
pp. 155–174.
71

[25] Benjamin Edelman and Zhenyu Lai. “Design of search engine services: Channel interdependence in search engine results”. In: Journal of Marketing Research 53.6 (2016),
pp. 881–900.
[26] Andrey Simonov, Chris Nosko, and Justin M Rao. “Competition and crowd-out for
brand keywords in sponsored search”. In: Marketing Science 37.2 (2018), pp. 200–215.
[27] Navdeep S Sahni and Charles Zhang. “Are consumers averse to sponsored messages?
The role of search advertising in information discovery”. In: Quantitative Marketing
and Economics 22.1 (2024), pp. 63–114.
[28] Hal R Varian. “Position auctions”. In: international Journal of industrial Organization
25.6 (2007), pp. 1163–1178.
[29] Jeffrey Gleason et al. “Google the gatekeeper: How search components affect clicks and
attention”. In: Proceedings of the International AAAI Conference on Web and Social
Media. Vol. 17. 2023, pp. 245–256.
[30] Tyna Eloundou et al. “Gpts are gpts: An early look at the labor market impact potential of large language models”. In: arXiv preprint arXiv:2303.10130 (2023).
[31] Ali Zarifhonarvar. “Economics of chatgpt: A labor market view on the occupational impact of artificial intelligence”. In: Journal of Electronic Business & Digital Economics
(2023).
[32] Gordon Burtch, Dokyun Lee, and Zhichen Chen. “The consequences of generative ai
for ugc and online community engagement”. In: Available at SSRN 4521754 (2023).
[33] Amit Kumar Kushwaha and Arpan Kumar Kar. “MarkBot–a language model-driven
chatbot for interactive marketing in post-modern world”. In: Information systems frontiers (2021), pp. 1–18.
[34] Martin Reisenbichler et al. “Frontiers: Supporting content marketing with natural language generation”. In: Marketing Science 41.3 (2022), pp. 441–452.
[35] Ali Goli and Amandeep Singh. “Frontiers: Can Large Language Models Capture Human Preferences?” In: Marketing Science (2024).
[36] Juan-José Ganuza. “Ignorance promotes competition: an auction model with endogenous private valuations”. In: RAND Journal of Economics (2004), pp. 583–598.
[37] Sıla Ada, Nadia Abou Nabout, and Elea McDonnell Feit. “Context information can
increase revenue in online display advertising auctions: Evidence from a policy change”.
In: Journal of Marketing Research 59.5 (2022), pp. 1040–1058.
72

[38] Wilfred Amaldoss, Kinshuk Jerath, and Amin Sayedi. “Keyword management costs
and “broad match” in sponsored search advertising”. In: Marketing Science 35.2 (2016),
pp. 259–274.
[39] Ambarish Chandra. “Targeted advertising: The role of subscriber characteristics in
media markets”. In: The Journal of Industrial Economics 57.1 (2009), pp. 58–84.
[40] Susan Athey and Joshua S Gans. “The impact of targeting technology on advertising markets and media competition”. In: American Economic Review 100.2 (2010),
pp. 608–613.
[41] Ian Tenney et al. “What do you learn from context? probing for sentence structure in
contextualized word representations”. In: arXiv preprint arXiv:1905.06316 (2019).
[42] Anna Rogers, Olga Kovaleva, and Anna Rumshisky. “A primer in BERTology: What we
know about how BERT works”. In: Transactions of the Association for Computational
Linguistics 8 (2021), pp. 842–866.
[43] Tomas Mikolov et al. “Efficient estimation of word representations in vector space”.
In: arXiv preprint arXiv:1301.3781 (2013).
[44] Andriy Mnih and Geoffrey E Hinton. “A scalable hierarchical distributed language
model”. In: Advances in neural information processing systems 21 (2008).
[45] Gerhard Jäger and James Rogers. “Formal language theory: refining the Chomsky
hierarchy”. In: Philosophical Transactions of the Royal Society B: Biological Sciences
367.1598 (2012), pp. 1956–1970.
[46] Shigeru Miyagawa, Robert C Berwick, and Kazuo Okanoya. “The emergence of hierarchical structure in human language”. In: Frontiers in psychology 4 (2013), p. 40804.
[47] Cas W Coopmans, Karthikeya Kaushik, and Andrea E Martin. “Hierarchical structure
in language and action: A formal comparison.” In: Psychological Review 130.4 (2023),
p. 935.
[48] David M Blei, Andrew Y Ng, and Michael I Jordan. “Latent dirichlet allocation”. In:
Journal of machine Learning research 3.Jan (2003), pp. 993–1022.
[49] Petros Karvelis et al. “Topic recommendation using Doc2Vec”. In: 2018 International
Joint Conference on Neural Networks (IJCNN). IEEE. 2018, pp. 1–6.
[50] Andrei Broder et al. “Online expansion of rare queries for sponsored search”. In: Proceedings of the 18th international conference on World wide web. 2009, pp. 511–520.
73

[51] Yan Lu, Michael Chau, and Patrick YK Chau. “Are sponsored links effective? Investigating the impact of trust in search engine advertising”. In: ACM Transactions on
Management Information Systems (TMIS) 7.4 (2017), pp. 1–33.
[52] Wilfred Amaldoss, Preyas S Desai, and Woochoel Shin. “Keyword search advertising
and first-page bid estimates: A strategic analysis”. In: Management Science 61.3 (2015),
pp. 507–519.
[53] Miguel Alcobendas and Robert Zeithammer. “Adjustment of Bidding Strategies After
a Switch to First-Price Rules”. In: Available at SSRN 4036006 (2021).
[54] Guy Aridor et al. “Evaluating The Impact of Privacy Regulation on E-Commerce
Firms: Evidence from Apples App Tracking Transparency”. In: (2024).
[55] Melissa Schettini Kearney. “The Economic Winners and Losers of Legalized Gambling”. In: National Tax Journal 58.2 (2005), pp. 281–302. doi: 10.17310/ntj.2005
.2.08. url: https://ideas.repec.org/a/ntj/journl/v58y2005i2p281-302.html.
[56] J. Welte et al. “Gambling and problem gambling in the United States: Changes between
1999 and 2013”. In: Journal of Gambling Studies 31 (2015), 695–715. doi: 10.1007/s
10899-014-9471-4.
[57] Gerhard Meyer, Tobias Hayer, and Mark Griffiths. Problem gambling in Europe: Challenges, prevention, and interventions. Jan. 2009. doi: 10.1007/978-0-387-09486-1.
[58] Howard J Shaffer and David A Korn. “Gambling and related mental disorders: A public
health analysis”. In: Annual review of public health 23.1 (2002), pp. 171–212.
[59] E. Gabellini, F. Lucchini, and M.E. Gattoni. “Prevalence of Problem Gambling: A
Meta-analysis of Recent Empirical Research (2016–2022)”. In: Journal of Gambling
Studies 39 (2023), 1027–1057. doi: 10.1007/s10899-022-10180-0.
[60] Matthew Q Clarida. “An Empirical Analysis of the Impact of Legal Sports Betting On
Consumer Credit Health”. In: Colum. Bus. L. Rev. (2020), p. 1000.
[61] Brantly Callaway and Pedro H.C. Sant’Anna. “Difference-in-Differences with multiple
time periods”. In: Journal of Econometrics 225.2 (2021), pp. 200–230. issn: 0304-4076.
doi: https://doi.org/10.1016/j.jeconom.2020.12.001. url: https://www.scie
ncedirect.com/science/article/pii/S0304407620303948.
[62] Mark Griffiths. Problem gambling in Europe: An overview. Nottingham Trent University Nottingham, 2009.
74

[63] Heather Wardle et al. “Defining the online gambler and patterns of behaviour integration: Evidence from the British Gambling Prevalence Survey 2010”. In: International
Gambling Studies 11.3 (2011), pp. 339–356.
[64] Per Binde. “Exploring the impact of gambling advertising: An interview study of
problem gamblers”. In: International journal of mental health and addiction 7 (2009),
pp. 541–554.
[65] Nerilee Hing et al. “Sports-embedded gambling promotions: A study of exposure, sports
betting intention and problem gambling amongst adults”. In: International Journal of
Mental Health and Addiction 13 (2015), pp. 115–135.
[66] Tara Hahmann et al. “Problem gambling within the context of poverty: A scoping
review”. In: International Gambling Studies 21.2 (2021), pp. 183–219.
[67] Robert T Wood and Robert J Williams. “Problem gambling on the Internet: Implications for Internet gambling policy in North America”. In: New Media & Society 9.3
(2007), pp. 520–542.
[68] Nerilee Hing et al. “Maintaining and losing control during internet gambling: A qualitative study of gamblers’ experiences”. In: New Media & Society 17.7 (2015), pp. 1075–
1095.
[69] Sally M Gainsbury et al. “The interaction between gambling activities and modes of
access: A comparison of Internet-only, land-based only, and mixed-mode gamblers”.
In: Addictive Behaviors 41 (2015), pp. 34–40.
[70] Debi A LaPlante et al. “Disordered gambling, type of gambling and gambling involvement in the British Gambling Prevalence Survey 2007”. In: The European Journal of
Public Health 21.4 (2011), pp. 532–537.
[71] Johanna Nordmyr et al. “Associations between problem gambling, socio-demographics,
mental health factors and gambling type: Sex differences among Finnish gamblers”. In:
International Gambling Studies 14.1 (2014), pp. 39–52.
[72] John M Barron, Michael E Staten, and Stephanie M Wilshusen. “The impact of casino
gambling on personal bankruptcy filing rates”. In: Contemporary Economic Policy 20.4
(2002), pp. 440–455.
[73] Ernie Goss, Edward A Morse, and John Deskins. “Have casinos contributed to rising bankruptcy rates?” In: International Advances in Economic Research 15 (2009),
pp. 456–469.
75

[74] Bogdan Daraban and Clifford F Thies. “Estimating the effects of casinos and of lotteries
on bankruptcy: A panel data set approach”. In: Journal of Gambling Studies 27 (2011),
pp. 145–154.
[75] Kent R Grote and Victor A Matheson. “The impact of state lotteries and casinos on
state bankruptcy filings”. In: Growth and Change 45.1 (2014), pp. 121–135.
[76] Alex MT Russell, Nerilee Hing, and Matthew Browne. “Risk factors for gambling
problems specifically associated with sports betting”. In: Journal of Gambling Studies
35.4 (2019), pp. 1211–1228.
[77] Nerilee Hing et al. “Does the uptake of wagering inducements predict impulse betting
on sport?” In: Journal of Behavioral Addictions 7.1 (2018), pp. 146–157.
[78] Philip WS Newall et al. “Gambling marketing from 2014 to 2018: A literature review”.
In: Current Addiction Reports 6 (2019), pp. 49–56.
[79] Wayne Taylor, Daniel McCarthy, and Kenneth C Wilbur. “Online Gambling Policy
Effects on Tax Revenue and Irresponsible Gambling”. In: Available at SSRN 4856684
(2024).
[80] CM Downs and Ryan Woolrych. “Gambling and debt pathfinder study”. In: (2009).
[81] Kirill Borusyak, Xavier Jaravel, and Jann Spiess. “Revisiting event study designs:
Robust and efficient estimation”. In: Review of Economic Studies (2024), rdae007.
[82] John Gardner. “Two-stage differences in differences”. In: arXiv preprint arXiv:2207.05943
(2022).
[83] Yoav Benjamini and Yosef Hochberg. “Controlling the false discovery rate: a practical
and powerful approach to multiple testing”. In: Journal of the Royal statistical society:
series B (Methodological) 57.1 (1995), pp. 289–300.
[84] Stefano M Iacus, Gary King, and Giuseppe Porro. “Causal inference without balance
checking: Coarsened exact matching”. In: Political analysis 20.1 (2012), pp. 1–24.
[85] Yongjie Lin, Yi Chern Tan, and Robert Frank. “Open sesame: Getting inside BERT’s
linguistic knowledge”. In: arXiv preprint arXiv:1906.01698 (2019).
[86] Ian Tenney, Dipanjan Das, and Ellie Pavlick. “BERT rediscovers the classical NLP
pipeline”. In: arXiv preprint arXiv:1905.05950 (2019).
[87] Nelson F Liu et al. “Linguistic knowledge and transferability of contextual representations”. In: arXiv preprint arXiv:1903.08855 (2019).
76

[88] John Hewitt and Christopher D Manning. “A structural probe for finding syntax in
word representations”. In: Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019, pp. 4129–4138.
77

Appendix A
Information Signals in Sponsored Search:
Evidence from Google’s BERT
A.1 Theory Model Proofs
A.1.0.0.1 Bidding Strategies Advertiser private valuations for a click are drawn vi ∼
U[0, 1]. Conditional on being allocated to an auction, their objective is to maximize E[v −
˜br˜
r
|b
∗
r ≥ ˜br˜], where b
∗
is the optimal bid, r is their relevancy score for the given auction, ˜b is
the second highest bid value, and r˜ is the second highest relevancy score. (˜br˜ is the second
highest ad rank). They aim to maximize the following:
Z b
∗r
r˜
0
(v −
Adr ˜
r
)f(
˜b)d
˜b
Taking the derivative w.r.t b
∗ and rearranging, we arrive to b
∗ = v
A.1.0.0.2 Platform Objective The platform wishes to maximize Equation A.1.
π = I{Click}
Adr ˜
rw
− I{θw̸=t}C (A.1)
It will do so by selecting the optimal bidders given the information set Qˆ
k it has available
to it. We now go through the possible information sets the platform can receive and identify
the optimal selection strategy within each set.
78

A.1.1 Full Information
With full information of both topic and context, the platform has two options: Allocate only
topically relevant advertisers, or allocate all advertisers with non-zero relevancy scores. Let
a = 1 − Bt and b = Bt
.
A.1.1.1 Only Topically Relevant Allocation
With only topically relevant advertisers, there are two advertisers in the auction. One
advertiser has a relevancy score of 1, the other has a relevancy score of Bt
.
(1 − Bt)
Bt
2
+ Bt
[
1
2
Bt
3
+
1
2
Bt
3
]
Bt(1 − Bt)
2
+ Bt
[
Bt
3
]
=
ab
2
+
b
2
3
In this scenario, the platform is allocating 2 bidders to the auction.
A.1.1.2 Contextually and Topically Relevant Allocation
With this scenario, the platform allocates 3 bidders to the auction. Consider the case when
the Q = (0, 0). The possible advertisers to allocate are (0, 0), (0, 1), and (1, 0). (A type
(1, 1) advertiser has a relevancy score of 0 and will be ignored). The subsequent relevancy
scores for each advertiser is 1, Bt
, and 1−Bt
. In this environment there are 5 possible cases.
Define advertiser (0, 0) Ad Rank as variable X, (0, 1) Ad Rank as Z, and (1, 0) Ad Rank as
Y. X ∼ U[0, 1], Z ∼ U[0, b], and Y ∼ U[0, a].
With these bidders, there are five possible cases.
1. a ≤ X ≤ b, a ≤ Z ≤ b, Y ≤ a. Occurs w.p. (b−a)
2
b
2. X, Y, Z ≤ a. Occurs w.p. a
2
b
79

3. X ≥ b. Occurs w.p. 1 − b
4. X, Y ≤ a, a ≤ Z ≤ b. Occurs w.p. a(b−a)
b
5. Y, Z ≤ a, a ≤ X ≤ b. Occurs w.p. a(b−a)
b
A.1.1.2.1 Case 1 a ≤ X ≤ b, a ≤ Z ≤ b, Y ≤ a. Simply a 50/50 chance either X or Z
wins. If X wins, they pay Z, if Z wins, they pay X. For two uniform random variables, the
expected second highest value takes on the form d + (m − d)
n−1
n+1 .
(b − a)
2
b
(a + (b − a)
1
3
)
=
a(b − a)
2
b
+
(b − a)
3
3b
A.1.1.2.2 Case 2 X, Y, Z ≤ a. Each bidder has a 1
3
chance of winning within this region.
If the (1, 0) advertiser, Y, wins, the platform pays a cost of −C.
a
2
b
(
1
3
1
E[max(Z, Y )]
1
+
1
3
b
E[max(X, Y )]
b
+
1
3
(a
E[max(X, Z)]
a
− C))
=
a
2
3b
(
a
2
+
a
2
+
a
2
− C)
=
a
2
3b
(
3a
2
− C)
A.1.1.2.3 Case 3 X ≥ b. Occurs w.p. 1 − b = a. In this case, X advertiser (0, 0)
wins with certainty. The expected price to pay will be max(Z,Y )
1
. In this scenario, Z and Y
follow different distributions (U[0, a] and U[0, b]). We need distribution of the max of the
two variables to then calculate the expected value. Let W = max(Z, Y )
F(w) = P r(Z ≤ w)P r(Y ≤ w)
80

F(w) =



w2
ab w ≤ a
w
b
a ≤ w ≤ b
1 w ≥ b
f(w) =



2w
ab w ≤ a
1
b
a ≤ w ≤ b
0 w ≥ b
E[W] = Z b
0
wf(w)dw =
Z a
0
2w
2
ab dw +
Z b
a
w
b
dw
=
2a
3
3ab +
b
2
−
a
2
2b
=
2a
2
3b
+
b
2
−
a
2
2b
Now the expected probability of this event is a or 1 − Bt
. In this case, X wins and has a
CTR and relevancy score of 1. Final expected profit in this scenario is
a(3b
2 − a
2
)
6b
Which simplifies to
ab
2
−
a
3
6b
A.1.1.2.4 Case 4 X, Y ≤ a, a ≤ Z ≤ b. Occurs w.p. a(b−a)
b
. In this case Z advertiser
(0, 1) wins. Expected profits in this scenario are
a(b − a)
b
E[max(X, Y )]
81

Since X and Y are both conditionally under value a, the expected max is 2a
3
. This gives
us a final value of
2a
2
(b − a)
3b
A.1.1.2.5 Case 5 This case is identical to Case 4. Final expected profits in this scenario
are
2a
2
(b − a)
3b
A.1.1.2.6 Expected Profit Summing across these scenarios we reach an expected profit
of
a
2
b + b
3 − a
3
3b
−
Ca2
3b
+
ab
2
A.1.1.2.7 Comparison of Profits When is it better to “misallocate” only contextually
relevant bidders? When the expected profits are greater compared to only allocating relevant
bidders.
a
2
b + b
3 − a
3
3b
−
Ca2
3b
+
ab
2
≥
ab
2
+
b
2
3
Rearranging and simplifying, we find a simple solution that its more profitable for the
platform to allocate both topically and contextually relevant bidders when Bt ≥ C.
A.1.2 No Information
Now consider the case when the platform learns no information about the query. The
question is whether to run an auction, and if so, how many bidders should be allocated?
Note that because the platform has no information, all relevancy scores for all advertisers
are equivalent and equal to 1
2
.
82

Consider the scenario in which the platform allocates all four advertisers and lets the
market decide for itself.
E[π] = 1
4
(
3
5
+
3Bt
5
+
3(1 − Bt)
5
− 2C)
1
4
(
6
5
− 2C)
When is this greater than 0?
1
4
(
6
5
− 2C) ≥ 0
3
5
≥ C
When C ≤
3
5
, the platform is willing to run an auction with all bidders. The platform
has no incentive to allocate fewer advertisers as it doesn’t know who is better or worse.
Randomly choosing 3 advertisers from the set of 4 leads to expected profits of 5
24 −
C
2 which
is always less than profit with 4 bidders.
A.1.3 Partial Information
A.1.3.1 Topical Understanding, no Contextual Understanding
Consider now the case where the platform learns the query’s topic, but not the context of the
query. Here tˆ= t and qˆ =
1
2
. Consider a query Q = (0, 0) and all four advertiser X = (0, 0),
Y = (0, 1), Z = (1, 0), and U = (1, 1). The relevancy scores for X and Y advertisers are
1+Bt
2
and 1−Bt
2
for Z and U advertisers. Topically relevant advertisers are X and Y.
A.1.3.1.1 Allocate Only Topically Relevant If the platform allocates only topically
relevant advertisers X and Y, it creates a simple second-price auction outcome. (Relevancy
scores cancel out, so the model simplifies to a simple SPA). There is a 1
2
chance a given
83

advertiser wins. If X wins, CTR is 1. If Y wins, CTR is Bt
. Expected second highest value
is 1
3
.
E[π] = 1
6
+
Bt
6
=
1 + Bt
6
A.1.3.1.2 Allocate All Advertisers Now, the platform could allocate all advertisers
with non-zero relevancy scores. (If Bt = 1, Z and U advertisers are essentially removed).
Define a =
1−Bt
2
and b =
1+Bt
2
. There are four possible cases that could occur with all four
advertisers allocated to the auction.
1. X, Y, Z, U ≤ a. Occurs w.p. a
2
b
2
2. X, Y ∈ (a, b] and Z, U ≤ a. Occurs w.p. (b−a)
2
b
2
3. X ≥ a and Y, Z, U ≤ a. Occurs w.p. a(b−a)
b
2
4. Y ≥ a and X, Z, U ≤ a. Occurs w.p. a(b−a)
b
2
A.1.3.1.3 Case 1: The first case considers when all four bidder ad ranks fall under a. In
this scenario, each bidder has an equal chance of winning. Define W as the expected second
highest ad rank.
a
2
b
2
[
1
4
X +
1
4
Y +
1
4
Z +
1
4
U]
a
2
b
2
[
1
4
W
b
+
1
4
W
b
Bt +
1
4
((1 − Bt)
W
a
− C) −
1
4
C]
a
2
b
2
[W −
C
2
]
84

Now W is the expected second highest value among 4 draws from U[0, a]. This translates
to 3a
5
.
a
2
b
2
[
3a
5
−
C
2
]
A.1.3.1.4 Case 2: In this case, X and Y are in the range of a and b. This occurs w.p.
(b−a)
2
b
2
. Define W as now the second highest value among X and Y, conditional on X and Y
being greater than a. Both X and Y have a relevancy score of b.
(b − a)
2
b
2
[
1
2
X +
1
2
Y ]
(b − a)
2
b
2
[
W
2b
+
W
2b
Bt
]
(b − a)
2
b
2
[
W
2b
(1 + Bt)]
Since 2b = 1 + Bt
(b − a)
2
b
2
[W]
Now W is the second highest value among two draws from U[a, b]. This translates to
1
3
(b − a) + a.
(b − a)
2
b
2
(
b − a
3
+ a)
A.1.3.1.5 Case 3: This case occurs when X ≥ a and all other ad ranks are less than or
equal to a. This occurs w.p. a(b−a)
b
2
. In this case, X wins and pays the max among Y, Z, and
U scaled by its relevancy score. Define W as the expected max among Y, Z, U. It is equal
to 3a
4
85

a(b − a)
b
2
[
W
b
]
a(b − a)
b
2
[
3a
4b
]
3a
2
(b − a)
4b
3
A.1.3.1.6 Case 4: Case 4 is similar to case 3, except expected CTR for Y is Bt
instead
of 1 like it is for X. Therefore, we have
a(b − a)
b
2
[
3a
4b
Bt
]
A.1.3.1.7 Expected Profits This covers all the possible cases. Expected profit is now
the summation of these cases. First, consider that Case 3 and 4 joined are:
(1 + Bt)
3a
2
(b − a)
4b
3
Since 2b = (1 + Bt), we can simplify this to
3a
2
(b − a)
2b
2
Summing all cases now:
E[π] = a
2
b
2
[
3a
5
−
C
2
] + (b − a)
2
b
2
[
b − a
3
+ a] + 3a
2
(b − a)
2b
2
What we want to know is when is this more profitable in expectation than allocating just
the topically relevant advertisers.
a
2
b
2
[
3a
5
−
C
2
] + (b − a)
2
b
2
[
b − a
3
+ a] + 3a
2
(b − a)
2b
2
≥
b
3
Simplifying, we get:
86

b −
7
15
a ≥ C
Since b =
1+Bt
2
and a =
1−Bt
2
, we can further simplify
4 + 11Bt
15
≥ C
When C ≤
4+11Bt
15 , it is more profitable for the company to allocate all advertisers to the
auction than to run the auction with only topically relevant advertisers.
A.1.3.1.8 Allocate Topically Relevant and a Topically Irrelevant Now, there
could be cases where the platform only wants to allocate topically relevant and one topically irrelevant advertiser. The platform may benefit from this if it can weakly boost the
final price paid. Here, they can allocate X and Y and either Z or U. If they allocate Z or U,
there’s a chance they’re contextually relevant. Again define a =
1−Bt
2
and b =
1+Bt
2
.
1. X, Y ≥ a, Z/U ≤ a. Occurs w.p. (b−a)
2
b
2
2. X ≥ a, Y, Z/U ≤ a. Occurs w.p. a(b−a)
b
2
3. Y ≥ a, X, Z/U ≤ a. Occurs w.p. a(b−a)
b
2
4. X, Y, Z/Y ≤ a. Occurs w.p. a
2
b
2
A.1.3.1.9 Case 1: In this scenario, the effect of Z/U on price is irrelevant. Expected
profits are the same as Case 2 in the previous option when all advertisers are allocated.
(b − a)
2
b
2
(
b − a
3
+ a)
A.1.3.1.10 Case 2: Here, X wins and pays the second highest price, or the max between
Y and the other advertiser Z/U. Define W as the max of the two variables Y and Z/U.
87

a(b − a)
b
2
[1W
b
]
a(b − a)
b
2
[
2a
3b
]
2a
2
(b − a)
3b
3
A.1.3.1.11 Case 3: This is the same as Case 2, except expected CTR is Bt
instead of
1.
2a
2
(b − a)
3b
3
Bt
A.1.3.1.12 Case 4: The final case considers when X, Y, and Z/U are all less than or
equal to a. Define W as the expected second highest draw among the three variables between
0 and a.
a
2
b
2
(
1
3
X +
1
3
Y +
1
3
Z/U)
a
2
b
2
(
1
3
W
b
+
1
3
W
b
Bt +
1
3
[
1
2
(
W
a
(1 − Bt) − C) −
1
2
C])
a
2
b
2
(
W
3b
(1 + Bt) + 1
6
(
W
a
(1 − Bt) − 2C))
Since 2a = (1 − Bt) and 2b = (1 + Bt), we can further simplify.
a
2
b
2
(
2W
3
+
1
6
(2W − 2C))
a
2
b
2
(W −
C
3
)
88

Now W is the second highest draw of 3 draws from the distribution U[0, a]. This translates
to a
2
. This gives us:
a
2
b
2
(
a
2
−
C
3
)
A.1.3.1.13 Summing Up Cases
(b − a)
2
b
2
(
b − a
3
+ a) + 2a
2
(b − a)
3b
3
+
2a
2
(b − a)
3b
3
Bt +
a
2
b
2
(
a
2
−
C
3
)
(b − a)
2
b
2
(
b − a
3
+ a) + 4a
2
(b − a)
3b
2
+
a
2
b
2
(
a
2
−
C
3
)
A.1.3.1.14 Comparison Across Selection Choices We want to know when this decision is more profitable than allocating only topically relevant or allocating all advertisers.
First, consider case for when expected profit is greater than allocating only topically relevant
advertisers.
(b − a)
2
b
2
(
b − a
3
+ a) + 4a
2
(b − a)
3b
2
+
a
2
b
2
(
a
2
−
C
3
) ≥
b
3
Simplifying, we get the following:
1 + 3Bt
4
≥ C
So, when C ≤
1+3Bt
4
, its more profitable to allocate topically relevant and one topically
irrelevant advertiser to the auction compared to allocating only topically relevant.
Now, we also need to compare to allocating all advertisers.
(b − a)
2
b
2
(
b − a
3
+a)+ 4a
2
(b − a)
3b
2
+
a
2
b
2
(
a
2
−
C
3
) ≥
a
2
2b
2
[
6a
5
−C]+ (b − a)
2
b
2
[
b − a
3
+a]+ 3a
2
(b − a)
2b
2
89

Simplifying this down, we get
3 + 7Bt
10
≤ C
So when 3+7Bt
10 ≤ C, its better to allocate 3 advertisers instead of all 4.
A.1.3.1.15 Allocating Two Topically Irrelevant and One Topically Relevant Advertiser The platform could allocate two topically irrelevant advertisers and one topically
relevant advertisers. This would lead to an expected profit of:
E[π] = (b − a)
b
2a
3
+
a
b
[
a
2
−
2C
3
]
When simplifying, this translates to
1 − Bt
1 + Bt
(
2
3
Bt +
1 − Bt
4
−
2
3
C)
Compare it to when we allocate 3 bidders where 2 are topically relevant and 1 is topically
irrelevant.
(b − a)
b
2a
3
+
a
b
[
a
2
−
2C
3
] ≥
(b − a)
2
b
2
(
b − a
3
+ a) + 4a
2
(b − a)
3b
2
+
a
2
b
2
(
a
2
−
C
3
)
0 ≥ 12a
2
b − 4ab2 − 5a
3 − 3ab + 2C(2ab − a
2
) + 2(b − a)
3 + 6a(b − a)
2
This eventually simplifies to
(1 − 3B
2
t + 2Bt)C ≤
Bt + B2
t − 5B3
t − 5
4
Note that between 0 and 1, Bt+B2
t −5B3
t −5
4
is always negative. Since C ≥ 0 and 1 −
3B2
t + 2Bt ≥ 0, the left hand side is always positive, so this inequality never holds. Since
allocating two topically relevant and one topically irrelevant is already trumped by alternative
90

selections, the choice to select two topically irrelevant and one topically relevant is not
relevant.
A.1.3.1.16 What Selection is Optimal Under these conditions, we now have 3 cutoff
values: when 3+7Bt
10 ≤ C 3 bidders is better than 4, when 1+3Bt
4 ≥ C 3 is better than 2, and
when 4+11Bt
15 ≥ C 4 is better than 2.
1 + 3Bt
4
≤
4 + 11Bt
15
≤
3 + 7Bt
10
,
1
2
≤ Bt ≤ 1
When 3+7Bt
10 ≤ C, its better to allocate 2 bidders over 3 and 4 bidders. When 4+11Bt
15 ≤
C ≤
3+7Bt
10 its optimally to still only allocate 2 bidders. When 1+3Bt
4 ≤ C ≤
4+11Bt
15 , it’s
optimal to allocate all 4 bidders. And finally, when C ≤
1+3Bt
4
its optimal to also allocate
all 4 bidders.
1. 3+7Bt
10 ≤ C Two bidders.
2. 4+11Bt
15 ≤ C ≤
3+7Bt
10 Two bidders.
3. 1+3Bt
4 ≤ C ≤
4+11Bt
15 Four bidders.
4. C ≤
1+3Bt
4
Four bidders.
So the primary cutoff value is 4+11Bt
15 . If C ≥
4+11Bt
15 , the platform allocates only the
topically relevant bidders, otherwise the platform allocates all bidders.
A.1.3.2 Contextual Understanding, no Topical Understanding
The final partial information case to consider is when the platform learns the context but
not the topic of the query. Consider a query Q = (0, 0) and advertisers X, Y, Z, and U
where X, Y ∼ U[0, b] and Z, U ∼ U[0, a]. Advertisers X and Y have relevancy scores equal
to 1 −
Bt
2 while Z and U advertisers have relevancy scores of Bt
2
.
91

A.1.3.2.1 Allocate Only Contextually Relevant Consider the case when the platform only allocates known contextually relevant advertisers (X and Y). Both advertisers
have the same relevancy score. Let W equal the second highest value from the two random
variables X and Y.
E[π] = [1
2
X +
1
2
Y ]
1
2
W
b
+
1
2
((1 − Bt)
W
b
− C)
W
2b
(2 − Bt) −
C
2
Note that 2b = 2 − Bt
W −
C
2
Now, W =
b
3
, where b = 1 −
Bt
2
. We therefore have:
E[π] = b
3
−
C
2
This is the expected profit allocating only the advertisers with known context. If C ≥
2−Bt
3
, it is not profitable in expectation for the firm to run an auction with only contextually
relevant bidders.
A.1.3.2.2 Allocate All Bidders An alternative option for the platform is to allocate
all bidders. Let a =
Bt
2
and b = 1 −
Bt
2
. This creates four cases.
1. X, Y ∈ (a, b] and Z, U ≤ a. Occurs with probability (b−a)
2
b
2
2. X ≥ a and Y, Z, U ≤ a. Occurs w.p. a(b−a)
b
2
3. Y ≥ a and X, Z, U ≤ a. Occurs w.p. a(b−a)
b
2
92

4. X, Y, Z, U ≤ a. Occurs w.p. a
2
b
2
We now consider these cases.
A.1.3.2.3 Case 1: X, Y ∈ (a, b]. This occurs w.p. (b−a)
2
b
2
. The expected profit is the
same structure as when the platform allocates only contextually relevant bidders, except
now the lower bound of values is a and not 0. Define W as the expected second highest Ad
Rank among X and Y given they’re greater than or equal to a.
(b − a)
2
b
2
[
1
2
X +
1
2
Y ]
(b − a)
2
b
2
[
1
2
W
b
+
1
2
((1 − Bt)
W
b
− C)]
(b − a)
2
b
2
[W −
C
2
]
Note that W =
1
3
(b − a) + a. We find that expected profit is:
(b − a)
2
b
2
[
(b − a)
3
+ a −
C
2
]
A.1.3.2.4 Case 2: This occurs with X ≥ a and Y ≤ a. This occurs w.p. a(b−a)
b
2
. Define
W as the expected max value among the three variables Y, Z, and U, given they’re all less
than or equal to a. Expected profit is:
a(b − a)
b
2
[
1
2
W
b
+
1
2
((1 − Bt)
W
b
− C)]
a(b − a)
b
2
[W −
C
2
]
Here, W =
3a
4
. Final expected profit in this case is
a(b − a)
b
2
[
3a
4
−
C
2
]
93

A.1.3.2.5 Case 3: Case 3 is symmetrical to case 2, so final expected profit in this scenario
is:
a(b − a)
b
2
[
3a
4
−
C
2
]
A.1.3.2.6 Case 4: The final case is when all four ad ranks are below a. This occurs w.p.
a
2
b
2
. Each bidder has an equal likelihood of winning.
a
2
b
2
[
1
4
X +
1
4
Y +
1
4
Z +
1
4
U]
a
2
b
2
[
W
b
+ (W
b
(1 − Bt) − C) + W
a
Bt − C]
Here, W is defined as the expected second highest ad rank among the four bidders,
conditional on their values being less than or equal to a. This again simplifies down to
W −
C
2
a
2
b
2
[W −
C
2
]
Where W =
3
5
a
a
2
b
2
[
3a
5
−
C
2
]
A.1.3.2.7 Total Expected Profit Total profit is going to be the summation of each of
these events.
(b − a)
2
b
2
[
(b − a)
3
+ a −
C
2
] + a(b − a)
b
2
[
3a
4
−
C
2
] + a(b − a)
b
2
[
3a
4
−
C
2
] + a
2
b
2
[
3a
5
−
C
2
]
The question is then, when is this more profitable than allocating only contextually
relevant bidders or compared to not running an auction.
94

(b − a)
2
b
2
[
(b − a)
3
+ a −
C
2
] + a(b − a)
b
2
[
3a
4
−
C
2
] + a(b − a)
b
2
[
3a
4
−
C
2
] + a
2
b
2
[
3a
5
−
C
2
] ≥
b
3
−
C
2
Notice that C
2
is common among all outcomes and so the left and right side each have a
C
2
, so we can cancel it out.
(b − a)
2
b
2
[
(b − a)
3
+ a] + a(b − a)
b
2
[
3a
4
] + a(b − a)
b
2
[
3a
4
] + a
2
b
2
[
3a
5
] ≥
b
3
(b − a)
2
b
2
[
(b − a)
3
+ a] + a(b − a)
b
2
[
3a
2
] + a
2
b
2
[
3a
5
] ≥
b
3
(1 + a)(b − a)
2
3b
2
+
3a
2
(b − a)
2b
2
+
3a
3
5b
2
≥
b
3
Simplifying this inequality, we find that
Bt ≤
15
11
In other words, it is more profitable to allocate all advertisers that to only allocate
contextually relevant advertisers when Bt ≤
15
11 . Since Bt ≤ 1, this always holds.
A.1.3.2.8 Comparing to no Auction
(b − a)
2
b
2
(
1 + a
3
) + a(b − a)
b
2
3a
2
+
a
2
b
2
3a
5
≥
C
2
We can simplify this down to
C ≤
10 − 30a + 45a
2 − 32a
3
15(1 − a)
2
Which can be further simplified to
95

C ≤
4
(2 − Bt)
2
[
2
3
− Bt + B
2
t −
4
15
B
3
t
]
When C is less than or equal to 4
(2−Bt)
2
[
2
3 − Bt + B2
t −
4
15B3
t
], it is more profitable for the
company to run an auction with all bidders than to not run any auction.
A.1.3.2.9 Allocate Contextually Relevant and one Contextually Irrelevant Bidder Let a =
Bt
2
and b = 1 −
Bt
2
. This creates four cases.
1. X, Y ∈ (a, b] and Z/U ≤ a. Occurs with probability (b−a)
2
b
2
2. X ≥ a and Y, Z/U ≤ a. Occurs w.p. a(b−a)
b
2
3. Y ≥ a and X, Z/U ≤ a. Occurs w.p. a(b−a)
b
2
4. X, Y, Z/U ≤ a. Occurs w.p. a
2
b
2
A.1.3.2.10 Case 1:
E[π] = (b − a)
2
b
2
(
1 + a
3
−
C
2
)
A.1.3.2.11 Cases 2 and 3:
E[π] = a(b − a)
b
2
[
2a
3
−
C
2
]
A.1.3.2.12 Case 4:
E[π] = a
2
b
2
[
a
2
−
C
2
]
A.1.3.2.13 Summing Up Cases Summing up the cases we have
E[π] = (b − a)
2
b
2
(
1 + a
3
) + a(b − a)
b
2
(
4a
3
) + a
2
b
2
(
a
2
) −
C
2
96

Now we want to consider when this expected profit is greater than allocating only contextually relevant bidders or allocating all bidders. Since allocating all bidders dominates
allocating only contextually relevant bidders, we will compare to allocating all bidders
A.1.3.2.14 Comparing to Allocating All Bidders
(b − a)
2
b
2
(
1 + a
3
) + a(b − a)
b
2
(
4a
3
) + a
2
b
2
(
a
2
) −
C
2
≥
(1 + a)(b − a)
2
3b
2
+
3a
2
(b − a)
2b
2
+
3a
3
5b
2
−
C
2
Here, both C
2
and (b−a)
2
b
2 (
1+a
3
) cancel out. This leaves us with:
4a
2
(b − a)
3b
2
+
a
3
2b
2
≥
3a
2
(b − a)
2b
2
+
3a
3
5b
2
−
a
2
(b − a)
6
≥
a
3
10
10b ≤ 4a
10(1 − a) ≤ 4a
5
7
≤ a
10
7
≤ Bt
Therefore, When Bt ≥
10
7
, the platform is better off running the auction with 3 bidders
and not all 4. However, Bt ≤ 1, so this cannot occur.
A.1.3.2.15 Allocate 2 Non Contextually relevant and 1 Contextually Relevant
The platform could consider allocating 2 non contextually relevant advertisers and only 1
97

contextually relevant advertiser to the auction. Expected profit in this case is:
E[π] = 2a(b − a)
3b
+
a
2
2b
−
C
2
Comparing to allocating all four bidders:
2a(b − a)
3b
+
a
2
2b
−
C
2
≥
(1 + a)(b − a)
2
3b
2
+
3a
2
(b − a)
2b
2
+
3a
3
5b
2
−
C
2
It is only better to allocate this combination of three bidders compared to all four when
0 ≥ 10 − 25Bt +
45
2
B
2
t −
57
8
B
3
t
However, this inequality does not hold for all 1
2 ≤ Bt ≤ 1. Therefore, it is not a considered
platform option.
A.1.4 Region Analysis
We now have our selection process criteria. Under full information, when C ≤ Bt
, it is better
to allocate all topically and contextually relevant bidders. When C ≥ Bt
, the platform only
wants to allocate topically relevant bidders. Under no information, the platform allocates
all bidders if C ≤
3
5
, otherwise none are allocated. When the platform knows the topic
dimension, but not the context dimension, the platform allocates only topically relevant
bidders when C ≥
4+11Bt
15 . Otherwise, it allocates all bidders. When context is understood,
the platform allocates all bidders only when C ≤
4
(2−Bt)
2
[
2
3 − Bt + B2
t −
4
15B3
t
], otherwise no
bidders are allocated. These inequalities create six unique regions that depend on C and Bt
.
In Figure A.1 we plot the regions created by these thresholds.
Recall, that w.p. γt the platform learns the topic dimension of the query and w.p. γq,
the platform learns the context dimension of the query. We now describe expected number
of bidders and expected CPC based on region.
98

Figure A.1: Regions that determine how auction evolves with changes to a. C is the cost
the platform incurs for showing a topically irrelevant ad and Bt
is the importance of topic
alignment for a query’s CTR.
1. Region 1: C ≥
4
(2−Bt)
2
[
2
3 − Bt + B2
t −
4
15B3
t
]. The platform only allocates 2 bidders
when topics are known. This leads to expected number of bidders of 2γt
.
2. Region 2: C ≤
4
(2−Bt)
2
[
2
3 − Bt + B2
t −
4
15B3
t
] and C ≥
4+11Bt
15 . Here the platform
allocates only topically relevant bidders when the topic dimension is understood and
all four bidders when only the context dimension is understood. This leads to an
expected number of bidders of 2γt + 4γq − 4γtγq.
3. Region 3: C ≤
4+11Bt
15 and C ≥ Bt
. In this region, the platform allocates 2 bidders
with full information, and all four bidders when either the topic or context dimension
is known. This leads to an expected number of bidders of 4(γt + γq) − 6γtγq.
99

4. Region 4: C ≤ Bt and C ≥
3
5
. Here, the platform is willing to allocate all three
bidders with non-zero relevancy scores with full information and allocate all bidders
when either topic or context is known. This translates to an expected number of
bidders of 4(γt + γq) − 5γtγq.
5. Region 5: C ≤
3
5
and C ≥ Bt
. Here, the platform only wants to allocate topically
relevant bidders with full information, but is willing to allocate all bidders in all other
cases. This translates to an expected number of bidders of 4 − 2γtγq.
6. Region 6: C ≤
3
5
and C ≤ Bt
. Here, the platform allocates every bidder with
non-zero relevancy scores. With full information, we assume the platform allocates all
3 bidders with non-zero relevancy scores. This translates to an expected number of
bidders 4 − γtγq.
Note that Google states it does not run auctions when it isn’t confident it understands
the query. Therefore, we can assume that the cost C ≥
3
5
and the platform is unwilling to
run auctions when there is no signal about the query’s type. We will focus on regions 1
through 4 for analysis and exclude 5 and 6 for this reason.
We can notice several things from the plot and the results. First, for queries where topic
alignment is the priority (i.e., short queries where Bt
is large), Regions 2 and 4 dominate
most of the possible spaces. For queries where context matters more, Region 1 takes up more
and more of the potential space. If C is greater than 1, then all queries with Bt roughly
greater than 0.7852 will be in Region 2, with all other queries in Region 1. As C lowers,
we start to enter other regions. Consider when C = 0.8. All queries with Bt ≤ 0.647 will
live in Region 1, all queries with 0.6471 ≤ Bt ≤
8
11 will be in Region 2, all queries where
8
11 ≤ Bt ≤ 0.8 will be in Region 3, and all queries with Bt ≥ 0.8 will live in Region 4.
These regions define the auction environment in which a given query lives in and determines how the market will change with the introduction of a new algorithm. Note that CPC
is calculated as P r(Click)
Adr ˜
rw
where Adr ˜ is the second highest ad rank and rw is the winning
100

relevancy score. If no click is received, CPC is 0, if a click is received, its the second highest
ad rank scaled by the winning ad rank score. We now calculate expected CPC and number
of bidders for each region.
A.1.4.1 Region 1
A.1.4.1.1 Number of Bidders In Region 1, the platform is conservative and only
allocates topically relevant bidders. Here the expected number of bidders is 2γt and so the
number of bidders is increasing with a since ∂γt(a)
∂a > 0.
A.1.4.1.2 Cost-Per-Click (CPC) In Region 1, the platform only allocates topically
relevant bidders. Expected CPC is then
γqγt(
(1 − Bt)Bt
2
+
B2
3
) + γt(1 − γq)
1 + Bt
6
We can simplify to
γt
6
[1 + Bt − γq(Bt − 1)2
]
Taking the derivative w.r.t. to a and setting less than 0, we find that CPC declines when
1 + Bt
6(Bt − 1)2
<
γ
′
q
γ
′
t
γt + γq
Notice that as Bt closes in on 1, the left-hand value approaches infinity, meaning it is
unlikely that prices can decline. This matches to our intuition that short queries, which
lack context and therefore have a higher Bt value are unlikely to see prices decline because
context does not exist to further differentiate bidders within auctions. But, when Bt
is
moderate and gains to contextual understanding are large relative to topical gains, there is
an opportunity for prices to decline. The intuition for the result is that when the platform’s
interpretation algorithm substantially improves context signals it leads to higher frequency
of full information auction events and improved bidder differentiation. This can lead to
101

lower prices but more relevant advertisements and higher expected profit. (Lower prices but
higher CTR).
A.1.4.2 Region 2
A.1.4.2.1 Number of Bidders In Region 2, the platform allows for some misallocation
and we observe the number of bidders equal to 2γt + 4γq −4γtγq. Taking the derivative w.r.t.
γq equates to 4(1−γt), which is always non-negative. Taking the derivative w.r.t. γt equates
to 2 − 4γq, which is positive when γq <
1
2
, 0 when γq =
1
2
, and negative when γq >
1
2
.
Taking the derivative of this w.r.t a, we find that the number of bidders will decline
when:
(1 − γt)
γ
′
q
γ
′
t
< γq −
1
2
When the inequality flips, the number of bidders is increasing and when there is equality
the number of bidders stays the same with a. Note that γ
′
q
γ
′
t
is always positive and that γq −
1
2
only begins to be positive when γq >
1
2
. Therefore, when the context signal is weak (γq <
1
2
),
it’s not possible to see a decline in the number of bidders.
When γq >
1
2
, it is possible to see a decline in the number of bidders. Assuming the
improvements to contextual signals are small (small γ
′
q
), a decline could occur when the
topic signal is also strong (large γt) or improvements to topic signals are large (large γ
′
t
).
When the platform learns the topic dimension, the market is small (two bidders). So,
as this signal improves (larger γt), the left-hand side term shrinks due to (1 − γt). Whether
this leads to a decline in the average number of bidders depends on the strength of the
context signal. If the context signal is large and getting all advertisers through the door
in certain situations, then we can potentially see the average number of bidders decline.
Intuitively, this is capturing the market segmentation effects we might imagine could occur.
The platform would be, on average, removing some irrelevant advertisers and thinning the
markets to more relevant advertisers.
102

A.1.4.2.2 Cost-Per-Click (CPC) Region 2 considers when the platform allocates two
topically relevant when the topic is known and all four bidders when the context is known
but the topic is not known. Define J =
40−60Bt+45B2
t −16B3
t
30(2−Bt)
2
. Expected CPC in Region 2 for a
query is defined as
γt
6
[1 + Bt − γq(Bt − 1)2
] + γq(1 − γt)A
To simplify notation, let D =
1+Bt
6
and let M = J +
(Bt−1)2
6
. Average CPC in Region 2
becomes
γtD + Jγq − γtγqM
Taking the derivative w.r.t. a and finding when its less than 0, we get
γ
′
q
γ
′
t
(J − M γt) < M γq − D
We now are interested in which cases cause this inequality to hold or not. When it does
not hold, CPC can either increase or stay the same.
1. γq <
D
M
and γt ≤
J
M
. Inequality cannot hold. CPC must increase.
2. γq <
D
M
and γt >
J
M
. Inequality can hold. CPC may decline if γ
′
q
γ
′
t
is large.
3. γq ≥
D
M
and γt <
J
M
. Inequality can hold. CPC may decline if γ
′
q
γ
′
t
is small.
4. γq >
D
M
and γt =
J
M
. Inequality always holds and CPC must decline. When γq =
D
M
and γt =
J
M
, the sides equal and implies that there is no change to CPC.
5. γq ≥
D
M
and γt >
J
M
. Inequality always holds. CPC drops with certainty.
Note that the minimum possible value of J
M
is roughly 0.971, so the topic signal for the
platform must be extremely strong when the context signal is less than or equal to D
M
for
103

prices to potentially decline. Note also that D
M
is a declining function in Bt
, so higher values
of Bt
lead to lower thresholds D
M
.
Predictions for short queries in Region 2 with introduction of BERT: γq ≥
D
M
and γt <
J
M
.
This means it’s possible for CPC to decline if γ
′
q
γ
′
t
. But, since we expect γ
′
q
γ
′
t
to be large, we
expect CPC to rise for short queries in Region 2.
For long queries, γq could either be greater than or less D
M
. It is unlikely γt >
J
M
. More
likely γt ≤
J
M
. In either case, it is likely that prices increase.
In Figure A.2 we plot the behavior of the thresholds D
M
and J
M
.
Figure A.2: Region 2: Threshold functions for γq and γt that impact whether prices can
increase or decrease.
104

A.1.4.2.3 Region 2: Interesting Market Outcomes We could empirically observe a
decline in CPC with more bidders if the following inequality holds:
γq −
1
2
(1 − γt)
<
γ
′
q
γ
′
t
<
M γq − D
(J − M γt)
Alternatively, we could empirically observe an increase in CPC despite fewer bidders if
the inequality flips:
γq −
1
2
(1 − γt)
>
γ
′
q
γ
′
t
>
M γq − D
(J − M γt)
A.1.4.3 Region 3
A.1.4.3.1 Number of Bidders The possibility for the number of bidders to decline
when in Region 3 depends on if γq ≤
2
3
and γt >
2
3
or γq >
2
3
. This can be seen with the
following inequality:
(1 −
3γt
2
)
γ
′
q
γ
′
t
<
3
2
γq − 1
The intuition for the result can be seen by looking at the expected number of bidders
4(γt + γq) − 6γtγq. When one signal is strong (topic or context), while the other is weak,
the platform is generally running auctions with all bidders. However, when both signals
are strong, the platform is predominantly allocating only the two topically relevant bidders.
The number of bidders can decline when signal gains for one dimension are large while the
other dimension also has a strong signal. In a sense, the platform is moving towards a more
segmented market once it has strong signals of both.
A.1.4.3.2 Cost-Per-Click (CPC) In Region 3, the platform allocates only topically
relevant advertisers when it knows the topic and context and all advertisers when it knows
either just the topic or just the context. No advertiser is allocated when there is no infor105

mation. Define J =
40−60Bt+45B2
t −16B3
t
30(2−Bt)
2
, X =
9+18Bt−3B2
t +16B3
t
30(1+Bt)
2
, and Z = J + X −
B(3−B)
6
. X is
the expected CPC when allocating all four bidders when the platform knows the topic and
J is the expected CPC when the platform allocates all four bidders and it only knows the
context dimension. Expected CPC is therefore
Jγq + Xγt − Zγtγq
Taking the derivative w.r.t. a and setting it less than 0, we reach the following inequality:
γ
′
q
γ
′
t
(J − Zγt) < Zγq − X
This inequality follows a similar structure to Region 2, except Z is different than the M
we defined for Region 2.
Z is going to be equal to:
Z =
9 + 3Bt − 28B2
t + 11B3
t + 5B4
t
30(1 + Bt)
2
+
40 − 60Bt + 45B2
t − 16B3
t
30(2 − Bt)
2
which simplifies to:
Z =
76 − 4Bt − 150B2
t + 173B3
t − 39B4
t − 25B5
t + 5B6
t
30(2 − Bt)
2
(1 + Bt)
2
The ratio X
Z will be equal to
X
Z
=
(9 + 18Bt − 3B2
t + 16B3
t
)(2 − Bt)
2
76 − 4Bt − 150B2
t + 173B3
t − 39B4
t − 25B5
t + 5B6
t
The ratio J
Z will be equal to
J
Z
=
(40 − 60Bt + 45B2
t − 16B3
t
)(1 + Bt)
2
76 − 4Bt − 150B2
t + 173B3
t − 39B4
t − 25B5
t + 5B6
t
Under what conditions will the inequality γ
′
q
γ
′
t
(J − Zγt) < Zγq − X hold
106

Figure A.3: Region 3: Threshold functions for γq and γt that impact whether prices can
increase or decrease.
1. γq <
X
Z
and γt ≤
J
Z
. Inequality cannot hold and CPC must increase.
2. γq <
X
Z
and γt >
J
Z
. Inequality can hold and prices decline if γ
′
q
γ
′
t
is large.
3. γq >
X
Z
and γt <
J
Z
. Inequality can hold. CPC can decline if γ
′
q
γ
′
t
is small.
4. γq >
X
Z
and γt =
J
Z
. Inequality holds and CPC must decline. When γq =
X
Z
and
γt =
J
Z
, the sides equal and implies there is no change to CPC.
5. γq ≥
X
Z
and γt >
J
Z
.Inequality always holds and CPC will drop.
The behavior of CPC in Region 3 is similar to Region 2, except the thresholds are
different. In Figure A.3 we present the behavior of X
Z
and J
Z
.
107

Here, the region in which γq and γt are under X
Z
and J
Z
respectively is much larger than
in Region 2. The minimum value for both functions is 308
391 or roughly 0.7877. If γq and γt are
both less than that value, then prices cannot decline in Region 3. If we believe γt >
J
Z
and
γq <
X
Z
and γ
′
q
γ
′
t
is large then prices are likely to decline. This may occur for longer queries
with smaller Bt
.
A.1.4.3.3 Region 3: Interesting Market Outcomes We could empirically observe a
decline in CPC with more bidders if the following inequality holds:
3
2
γq − 1
1 −
3γt
2
<
γ
′
q
γ
′
t
<
Zγq − X
(J − Zγt)
Alternatively, we could empirically observe an increase in CPC despite fewer bidders if
the inequality flips:
3
2
γq − 1
1 −
3γt
2
>
γ
′
q
γ
′
t
>
Zγq − X
(J − Zγt)
A.1.4.4 Region 4
A.1.4.4.1 Number of Bidders Region 4 behaves similarly to Region 3, except the
threshold values are higher. Now, γq ≤
4
5
and γt >
4
5
or γq >
4
5
can lead to a decline in the
number of bidders. Otherwise, the number of bidders is increasing when the inequality is
flipped and 0 at equality.
(1 −
5γt
4
)
γ
′
q
γ
′
t
<
5
4
γq − 1
In this region, the likelihood of seeing a decline in the number of bidders is smaller
because the number of bidders is generally higher due to three bidders being allocated with
full information instead of only two.
108

A.1.4.4.2 Cost-Per-Click (CPC) Region 4 differs from Region 3 because the platform
is willing to allocate the three advertisers with non-zero relevancy scores to the auction, not
just the topically relevant advertisers. Like Region 3, define J =
40−60Bt+45B2
t −16B3
t
30(2−Bt)
2 and
X =
9+18Bt−3B2
t +16B3
t
30(1+Bt)
2
. X is the expected CPC when allocating all four bidders when the
platform knows the topic and J is the expected CPC when the platform allocates all four
bidders and it only knows the context dimension. Finally, define P =
−5+20Bt−25b
2
t +15B3
t −3b
4
t
6Bt
and L = X + J −P. P is the expected CPC when three bidders are allocated to the auction
when the platform has full information. CPC falls when:
γ
′
q
γ
′
t
(J − Lγt) < Lγq − X
This is the same structure as Region 3, except L replaces Z.
Under what conditions will the inequality γ
′
q
γ
′
t
(J − Lγt) < Lγq − X hold
1. γq <
X
L
and γt ≤
J
L
. Inequality cannot hold and CPC must increase.
2. γq <
X
L
and γt >
J
L
. Inequality can hold and prices decline if γ
′
q
γ
′
t
is large.
3. γq >
X
L
and γt <
J
L
. Inequality can hold. CPC can decline if γ
′
q
γ
′
t
is small.
4. γq >
X
L
and γt =
J
L
. Inequality holds and CPC must decline. When γq =
X
L
and
γt =
J
L
, the sides equal and implies there is no change to CPC.
5. γq ≥
X
L
and γt >
J
L
.Inequality always holds and CPC will drop.
L is equal to:
L =
100 − 224Bt + 81B2
t + 340B3
t − 282B4
t − 119B5
t + 230B6
t − 105B7
t + 15B8
t
30(2 − Bt)
2Bt(1 + Bt)
2
The ratio J
L
is equal to
109

Figure A.4: Region 4: Threshold functions for γq and γt that impact whether prices can
increase or decrease.
J
L
=
(Bt(1 + Bt)
2
)(40 − 60Bt + 45B2
t − 16B3
t
)
100 − 224Bt + 81B2
t + 340B3
t − 282B4
t − 119B5
t + 230B6
t − 105B7
t + 15B8
t
and the ratio X
L
is equal to
X
L
=
(9 + 18Bt − 3B2
t + 16B3
t
)(Bt(2 − Bt)
2
)
100 − 224Bt + 81B2
t + 340B3
t − 282B4
t − 119B5
t + 230B6
t − 105B7
t + 15B8
t
In Figure A.4 we plot X
L
and J
L
. Note that the gray area is slightly largely compared to
Region 3.
110

A.1.4.4.3 Region 4: Interesting Market Outcomes We could empirically observe a
decline in CPC with more bidders if the following inequality holds:
5
4
γq − 1
1 −
5γt
4
<
γ
′
q
γ
′
t
<
Lγq − X
(J − Lγt)
Alternatively, we could empirically observe an increase in CPC despite fewer bidders if
the inequality flips:
5
4
γq − 1
1 −
5γt
4
>
γ
′
q
γ
′
t
>
Lγq − X
(J − Lγt)
A.1.4.5 Regions 5 and 6
A.1.4.5.1 Number of Bidders These regions see the number of bidders declining due
to the platform allocating fewer bidders when it has full information. The number of bidders
declines with a. The decline is faster in Region 5 compared to Region 6.
A.1.4.5.2 Cost-Per-Click (CPC): Region 5 Under Region 5, it is optimal for the
platform to allocate all bidders except when it has full information. With full information,
it allocates only the topically relevant bidders (2). Region 5 behaves similarly to Region
3. Define F =
3
10 . This is the average CPC when the platform allocates all bidders when
it has no information. Again, define J =
40−60Bt+45B2
t −16B3
t
30(2−Bt)
2
, X =
9+18Bt−3B2
t +16B3
t
30(1+Bt)
2
, and
Z = J + X −
B(3−B)
6
. Prices can decline if the following inequality holds:
γ
′
q
γ
′
t
(J − Zγt − F(1 + γt)) < Zγq − X + F(1 − γq)
This is similar to the inequality in Region 3, except we have this extra −F(1 +γt) on the
left-hand side and F(1−γq) on the right-hand side. Ultimately, we find that critical inequality
thresholds are now J−F
Z−F
for γt and X−F
Z−F
for γq. These are the same critical thresholds in
Region 3, except both the numerator and denominator are reduced by F.
111

Note that the max value of J is 3
10 when Bt = 1. Therefore J−F
Z−F ≤ 0, meaning γt ≥
J−F
Z−F
will always hold. When Bt ≤ 0.75, γq ≥
X−F
Z−F
because X−F
Z−F ≤ 0. Once Bt > 0.75, γq can
potentially be less than X−F
Z−F
, though it must be extremely small.
In general, prices are going to just decline in Region 5 with the decline in the number of
bidders. This is driven by the market segmentation effects of context.
A.1.4.5.3 Cost-Per-Click (CPC): Region 6 The relevant threshold inequality for
Region 6 is similar to Region 4 and Region 5. Again, define J =
40−60Bt+45B2
t −16B3
t
30(2−Bt)
2
, X =
9+18Bt−3B2
t +16B3
t
30(1+Bt)
2
, P =
−5+20Bt−25b
2
t +15B3
t −3b
4
t
6Bt
, and L = X + J − P.
γ
′
t
γ
′
q
(J − Lγt − F(1 + γt)) < Lγq − X + F(1 − γq)
Now, the relevant thresholds are J−F
L−F
for γt and X−F
L−F
for γq. Whether prices decline or
increase follow the same constraints as Region 4, except the thresholds are now J−F
L−F
and
X−F
L−F
. Like Region 5, γt ≥
J−F
Z−F will always hold and when Bt ≤ 0.75, γq ≥
X−F
Z−F
because
X−F
Z−F ≤ 0. Once Bt > 0.75, γq can potentially be less than X−F
Z−F
, though it must be extremely
small.
112

A.2 Sampling Procedure
Below is the list of search topics we asked for in our Amazon MTurk survey.
Insurance, Travel, Home Renovation, Cars, Restaurants, How-to’s, Things To Do, Company Research, Product Research, Financial Resources, Financial Products, News, Politics,
Animals, Books, Movies, Shows, Clothes, Electronics, Pest Control, Birthday Gifts, Kitchen
Shopping, Moving Services, Social Media, Entertainment, Cooking, Transportation, Home
Decor, Health, Medicine, Cosmetics.
Each participant was randomly asked about only five of these categories.
113

A.3 Alternative Identification Strategy
The main identification strategy discussed in the paper will be biased if there exists timevarying unobservables correlated with the introduction of BERT and auction outcomes.
Given our findings, such a confound would need to be a significant event that uniformly
drives up the number of bidders across queries and differentially affects CPC across query
lengths. While such a confounder seems unlikely, as in any observational study, we cannot
completely rule it out.
To strengthen the validity of our results, we present an alternative identification strategy
that exploits the fact that BERT is likely to affect queries differently due to their inherent
linguistic properties. In doing so, we compare queries more likely to be affected by BERT
with queries less likely to be affected by BERT before and after the introduction of BERT.
Under this identification strategy, a confound must be year-month specific and correlate with
query linguistic properties.
A.3.1 Defining Query Metrics
The Computer Science literature has documented that BERT better understands complex
syntactic language structures and semantic relationships between words and sentences [10,
85, 86, 41, 42]. A complex query will benefit from BERT’s ability to capture syntactic information, while all queries, including those with simple syntactic structures, will experience
changes to semantic relationships and understanding.
Motivated by these observations, we create two measurement variables, Linguistic Complexity and Cosine, to capture query syntactic complexity and semantics. Under the assumption that the interpretation and use of these linguistic properties changed with the introduction of BERT, we can identify variation in our dependent variables caused by BERT’s
implementation interacting with our measurements.
114

A.3.1.0.1 Linguistic Complexity BERT can better understand complex syntactic language structures [85, 86]. Therefore, BERT’s introduction should change how Google handles
complex syntactic language. Examples of complex syntactic information include query syntax tree structures [85], parts of speech [86], and sentence dependency features [86, 87]. [88]
finds that syntax trees, i.e., hierarchical characterizations of grammatical language structures, are embedded in BERT vector spaces and [41] finds that Parts of Speech (POS) tags
are also present in BERT vector spaces. These findings help us identify the information
BERT will interact with and drive the design of our first measurement: Linguistic Complexity (LC).
For each query, we measure the depth of the syntax tree and count the unique number
of Parts Of Speech (POS). LC is defined as a dummy variable and takes on a value of 1 if
either a query’s syntax tree depth is greater than the median depth in our dataset (median
= 2) or the unique POS count is greater than the median county in our dataset (median =
2). Otherwise, it’s 0. LC captures the syntactic complexity of the query.
A.3.1.0.2 Cosine BERT understands semantics and how words and concepts relate to
each other [41]. It knows that socks relate to shoes, banks can relate to bodies of water or
financial institutions, and computers can sometimes relate to mice. This semantic knowledge
improves Google’s ability to interpret and categorize search queries, ultimately affecting
the relationships between queries. This latter observation is critical as changes to query
relations likely impact Google’s matching process and identification of relevant advertisers.
Understanding that a query relates to more (fewer) queries will likely lead to more (fewer)
advertisers being deemed relevant.These observations motivate the design of our second
linguistic property: Cosine.
At a high level, we measure semantic changes (Cosine) using the difference in the number
of queries related to a given query before and after BERT.
To define Cosine, we must first identify the primary interpretation algorithm used by
115

Google before BERT (RankBrain). While we cannot be certain, RankBrain is likely Doc2Vec
(D2V) or Word2Vec (W2V), the former being a more generalized form of W2V.1
For a query i, we generate a vector representation with BERT and D2V models. We
then calculate the cosine similarity score between query i and all other queries in the dataset
within a given model vector space. This generates two N × N cosine similarity matrices,
one for the BERT vector space and one for the D2V vector space, where N is the number
of queries in our dataset and matrix element i, j is the cosine similarity score between query
i and query j.
We use these matrices to measure changes in query semantic relationships across D2V
and BERT vector spaces. Specifically, we measure the set of queries that pass “relatedness”
thresholds within vector spaces and then compare differences in these sets across vector
spaces. We first calculate each cosine matrix’s 25th, 50th, and 75th percentile cosine scores.
These are our “relatedness” thresholds.2 Then, for each query i, we identify the set of other
queries with a cosine similarity score greater than or equal to the chosen threshold (i.e., 25,
50, or 75) in the respective vector spaces. For example, if the 75th threshold in the D2V
space is 0.5 and the query “shoes” has a cosine similarity score of 0.8 with the query “socks”
and 0.2 for the query “hat”, then the “socks” query will be in the relevant set for the D2V
space. If, in the BERT space, the 75th threshold is 0.8 and the cosine score between “shoes”
and “hats” is 0.7, then “hat” would now appear in the relevancy set for “shoes” in the BERT
space. This gives us two sets of relevant queries per query, one for each vector space.
Then, we calculate the differences between the relevancy sets across BERT and D2V for
a given query. We define “added” queries as those that did not appear in the D2V relevancy
set but did appear in the BERT set. Alternatively, we define “removed” queries as those
1Google filed patents: https://patents.google.com/patent/US9740680B1/en. See also: https:
//opensource.googleblog.com/2013/08/learning-meaning-behind-words.html for technology before
RankBrain that looks similar to Word2Vec, and the researchers for both Word2Vec and Doc2Vec were
employed at Google.
2We vary the thresholds to generalize the measurement. We focus on thresholds because we reason that
Google likely has some cut-off requirement determining whether an advertiser is relevant enough to a given
query.
116

that appeared in the D2V set but did not appear in the BERT set. For a given query i, we
sum up the total number of “added” and “removed” queries and take the ratio between the
two sums. We define Cosine as this ratio.
A ratio value equal to one tells us that just as many queries were added as removed.
Ratio values less than one indicate that more queries were removed than added, indicating
that BERT believes the query is more specific and related to fewer other queries. Finally,
ratio values greater than 1 tell us that more queries were added than removed, suggesting
that BERT believes the query is related to more queries. Because the ratio of added to
removed queries is skewed to the right (median = 1.094, mean = 175.605), we use the log of
1 + Cosine in our model.3
It is worth noting that we use this process to generate Cosine because the D2V and
BERT vector spaces and their cosine scores are not directly comparable without making
strong assumptions about what the dimensions of each model’s vector space represent. D2V
and BERT capture potentially different sets of information, making vector comparison and
projection methods infeasible. Additionally, the scales of these vector spaces are potentially different, meaning we cannot directly compare query cosine scores across algorithms.
Therefore, to effectively measure query semantic changes, we must adequately standardize measurements within vector spaces such that they can then be compared across vector
spaces.
A.3.1.0.3 Average Query Metrics Table A.1 and A.2 present average Cosine and LC
measures, respectively, for all queries (“All”) and by keyword length. Table A.2 also presents
the proportion of queries with POS counts and syntax tree depths greater than the median.
(A value of 1 means above the median for the particular measurement). We identify several
patterns. First, LC measures correlate with query length due to the presence of within-query
contextual information, inherently making it longer. Second, short queries have, on average,
a large Cosine score, while long queries have a low Cosine score. The average long query
3
In Appendix ??, we show that our results are robust to how we define Cosine.
117

ratio between “added” and “removed” is 0.78, indicating that BERT finds these queries more
specific than D2V.
Table A.1: Average Log(Cosine+1).
Keyword Length 75th Ratio 50th Ratio 25th Ratio
All 1.761 1.324 1.049
Short 3.107 2.348 1.645
Medium 1.393 1.043 0.914
Long 0.650 0.481 0.451
These values are the average log(Cosine + 1), where Cosine is
the ratio between added and removed sets.
Table A.2: Average Linguistic Complexity.
Keyword Length Linguistic Complexity Tree Height POS
All 0.613 0.566 0.400
Short 0.004 0.001 0.004
Medium 0.811 0.730 0.439
Long 1.000 0.998 0.986
A.3.2 Specification
Given the two linguistic metrics defined above, we estimate the following model:
Yi,t = β1P ostt ×LCi +β2P ostt ×log(Cosi)+β3P ostt ×LCi ×log(Cosi)+δi +γt +ϵi,t, (A.2)
where P ost takes on a value of 1 for the months post-BERT (November to February), and
0 otherwise. LCi
is Language Complexity of query i and Cosi
is the the Cosine of query
i. δi are query fixed effects and γt year-month fixed effects. β1 captures the variation in Y
that is explained by the effect of LC after BERT gets introduced, β2 captures the variation
in Y explained by the effect of changes to Cosine after the introduction of BERT, and β3
captures the interaction between LC and Cos after the introduction of BERT.
118

We include query and year-month fixed effects, δi and γt
, respectively. We estimate
Equation A.2 using data from July 2019 to February 2020 and cluster standard errors at
the query level. To replicate the findings in the previous section, we restrict this analysis to
competitive queries, i.e., those queries with non-zero mode CS scores from July to October
2019.
A.3.2.0.1 Identification Assumptions This identification strategy relies on the assumptions that: (1) the linguistic properties we defined affect CPC and CS only through
their interaction with the query interpretation system; (2) BERT interacts with our linguistic
measurements differently than the previous algorithm; (3) queries with high and low values
of Linguistic Complexity and Cosine are comparable; (4) there are no time-varying-query
linguistic type-specific confounders.4
To partially validate these assumptions, we perform three tests. To test assumption
three, in Appendix A.3.4, we use an event study-like model to show that we estimate null
pre-trends. We then perform a placebo test that shows that our estimates are likely due to
the introduction of BERT (partially validating assumptions one and two). To account for
consumer search behavior and changes to organic rank, we also show that results hold when
we control search volume or for organic rank changes.
A.3.3 Results
We present the estimates of Equation A.2 using CS as the dependent variable in Table A.3.
P ostt × LCi
is positive and significant, suggesting that when Linguistic Complexity is above
the median, CS increases by 0.33 − 0.4%. These findings suggest that when syntactic information acquisition opportunities are significant, BERT’s information increases Google’s
confidence that it correctly interprets the query and increases auction density. Second, con4For example, changes in search behavior that are query-type specific and unrelated to BERT can be
problematic. Consider the case in which wealthier consumers begin submitting search queries that are more
linguistically complex because of something other than BERT. If advertisers change their targeting strategies
and increase their bids in response to this behavior, our results will be upward biased.
119

sistent with our expectations, P ostt × Cosi
is also positive and significant. The coefficient
estimates suggest that a 1% increase Cosine translates to about a 0.1% increase in CS. Finally, as expected, the interaction term between LC and Cosine is negative. Linguistically
complex queries are generally more specific, and BERT learns this, leading to partial market
filtering and some removal of advertisers.
Table A.3: Alternative Strategy: log(CS)
(1) (2) (3)
75th 50th 25th
Post × Linguistic Complexity 0.0040∗∗ 0.0033∗ 0.0033∗
(0.002) (0.002) (0.002)
Post × Log(Cosine) 0.0012∗∗ 0.0009∗ 0.0007
(0.001) (0.001) (0.001)
Post × Linguistic Complexity × Log(Cosine) −0.0015∗∗ −0.0020∗∗ −0.0030∗∗
(0.001) (0.001) (0.001)
Observations 63,474 63,474 63,474
R
2 0.9786 0.9785 0.9785
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Note: Regressions include year-month and query fixed effects. Standard errors clustered at
the query level are reported in parentheses. Data from July 2019 to Feb 2020.
To put these results into context, we can use the average keyword length values in Table A.1 and Table A.2 to bootstrap predicted changes to the average number of bidders. We
find that aggregate CS increases by a conservative 0.1 − 0.3%. When we predict average
values by keyword length, we find that, across lengths, CS is increasing by about 0.1 − 0.4%
(See Table A.4). The uniform increase in CS is consistent with our empirical observations
using the main DD setup. All estimates are more conservative, likely due to measurement
errors in our treatment variables.
In Table A.5, we present the results for log(CP C). Consistent with our expectations, a 1%
increase in Cosine positively increases CPC by about 0.6%. Inconsistent with our predictions,
120

Table A.4: Predicted log(CS) estimates by model
specification.
Keyword Length 75th 50th 25th
All 0.003∗∗∗ 0.002∗∗∗ 0.001∗∗∗
Short 0.004∗∗∗ 0.002∗∗∗ 0.001∗∗∗
Medium 0.003∗∗∗ 0.002∗∗∗ 0.001∗∗∗
Long 0.004∗∗∗ 0.003∗∗∗ 0.002∗∗∗
Note: 99% confidence intervals are estimated by bootstrapping predictions (1000 iterations). ∗∗∗p<0.01.
we find that P ostt × LCi
is negative and significant. Finally, the triple interaction is close
to zero and not significant.
Table A.5: Alternative Identification: log(CP C)
(1) (2) (3)
75th 50th 25th
Post × Linguistic Complexity −0.0270∗ −0.0289∗∗ −0.0307∗∗
(0.015) (0.014) (0.015)
Post × Log(Cosine) 0.0056∗ 0.0065∗ 0.0075
(0.003) (0.004) (0.005)
Post × Linguistic Complexity × Log(Cosine) −0.00003 0.0009 0.0006
(0.005) (0.006) (0.009)
Observations 63,474 63,474 63,474
R
2 0.795 0.795 0.795
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Note: Regressions include year-month and query fixed effects. Standard errors clustered at
the query level are reported in parentheses. Data from July 2019 to Feb 2020.
We again use the average values in Tables A.1 and A.2 to put these results into context.
We present the estimated effect on CPC in Table A.6. We find that the average CPC
decreases by roughly 0.8%. When we break it down by keyword length, we find that CPC
increases for short queries by roughly 1.5%, medium queries see CPC decrease by about
1.6%, and long queries see CPC decrease by approximately 2.5%. These average values are
121

broadly consistent with our primary DD analysis, though we note that average CPC declines
under this specification and is null in our main DD result.
Table A.6: Predicted log(CP C) estimates by model specification.
Keyword Length 75th 50th 25th
All −0.007∗∗∗ −0.008∗∗∗ −0.011∗∗∗
Short 0.017∗∗∗ 0.015∗∗∗ 0.012∗∗∗
Medium −0.014∗∗∗ −0.016∗∗∗ −0.018∗∗∗
Long −0.023∗∗∗ −0.025∗∗∗ −0.027∗∗∗
Note: 99% confidence intervals are estimated by bootstrapping predictions (1000 iterations). ∗∗∗p<0.01.
A.3.4 Robustness Checks
A.3.4.0.1 Event Study Results Our identification hinges on the idea that these linguistic properties should have not correlate with changes in our outcomes until BERT is
introduced. To partially validate this assumption, we can estimate an event study where we
replace P ostt
in Equation A.2 with a month of year indicator. In Table A.7 and Table A.8
we present event study estimates for CS and CPC for the 75th cosine percentile threshold.
Across both results, we see that our measures are largely statistically significant until the
post BERT months (November through February), suggesting that the variation over time
that we are picking up comes from BERT’s introduction.
A.3.4.0.2 Placebo Test Our identification rests on the assumption that the effects we
observe are due to the linguistic variables interacting with a change in the interpretation
algorithm (i.e., BERT). We support this assumption by performing a placebo test. We
estimate A.2 using the same period but a year before (July 2018 to February 2019) and create
a placebo post-BERT variable that takes on a value of 1 for November through February, 0
otherwise. Since no known change exists that we suspect interacts with these measurements
122

Table A.7: Event Study: log(CS).
Month LC Cosine LC × Cosine
July 0.0001 −0.00002 −0.0001
0.0003 0.00005 0.0001
August −0.001∗∗ −0.0001∗∗ 0.0002
0.0002 0.00004 0.0001
September −0.0002 −0.00005 0.0001
0.0001 0.00003 0.00005
November 0.003∗ 0.0004 −0.001∗∗
0.001 0.0003 0.0004
December 0.005∗∗ 0.001∗ −0.002∗
0.002 0.001 0.001
January 0.006∗∗ 0.001∗∗ −0.002∗
0.002 0.001 0.001
February 0.002 0.002∗∗ −0.002∗
0.003 0.001 0.001
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Note: Regressions include year-month and query fixed
effects. Standard errors clustered at the query level are
reported in parentheses. Data from July 2019 to Feb 2020.
during this time frame, we expect to estimate insignificant results. In Tables A.9 and A.10
we present null results.
A.3.4.0.3 Changes in Organic Rank Results As we did for the main identification
strategy in the paper, we also test whether our results hold when controlling for organic
rank changes. We re-estimate Equation A.2 including Featured Snippet usage, the Number
of SERP features, and whether the top domain has changed compared to the previous month.
We present these results in Table A.11 and Table A.12 for CS and CPC, respectively. We
find results that are directionally consistent but attenuated and more imprecise, particularly
123

Table A.8: Event Study: log(CP C).
Month LC Cosine LC × Cosine
July 0.0003 0.0003 −0.001
(0.002) (0.0003) (0.001)
August 0.002 0.0004 −0.001
(0.002) (0.0003) (0.001)
September −0.001 −0.0003 −0.0002
(0.001) (0.0002) (0.0005)
November −0.019∗ −0.001 0.003
(0.011) (0.002) (0.004)
December −0.029 0.008∗ −0.002
(0.018) (0.004) (0.006)
January −0.034∗ 0.008∗∗ 0.00002
(0.018) (0.004) (0.006)
February −0.024 0.008∗∗ −0.002
(0.018) (0.004) (0.006)
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Note: Regressions include year-month and query fixed
effects. Standard errors clustered at the query level are
reported in parentheses. Data from July 2019 to Feb
2020.
for CPC, where they become insignificant.5
A.3.4.0.4 Changes in Consumer and Advertiser Behavior Similar to what we did
for the main identification strategy, here we re-estimate our alternative identification regression models controlling for Google Search trends Interest and SEMRush’s search volume. In
Table A.13, we present changes to CS, controlling for both.
Similarly, in Table A.14, we present percent changes to CPC controlling for Interest
5The loss in precision is because we do not have organic rank information for all queries (only for 41, 555
out of 63, 394). Indeed, we find that using the same subset of queries for which we have organic rank
information and not including controls, we obtain very similar results to those reported in Table A.11 and
Table A.12
124

Table A.9: Placebo Test: log(CS).
(1) (2) (3)
75th 50th 25th
Post × Linguistic Complexity 0.001 0.001 0.0004
(0.001) (0.001) (0.001)
Post × Log(Cosine) 0.0003 0.0003 0.0004
(0.0002) (0.0002) (0.0004)
Post × Linguistic Complexity × Log(Cosine) −0.0002 −0.0002 −0.00001
(0.0002) (0.0003) (0.001)
Observations 63,468 63,468 63,468
R
2 0.997 0.997 0.997
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Note: Regressions include year-month and query fixed effects. Standard errors clustered
at the query level are reported in parentheses. Data from July 2018 to Feb 2019.
and search volume. We find that estimates remain largely consistent with our primary
specification.
125

Table A.10: Placebo test: log(CP C).
(1) (2) (3)
75th 50th 25th
Post × Linguistic Complexity 0.001 0.002 0.001
(0.002) (0.002) (0.002)
Post × Log(Cosine) 0.0004 0.0005 0.0004
(0.001) (0.001) (0.001)
Post × Linguistic Complexity × Log(Cosine) 0.001 −0.0002 0.001
(0.002) (0.001) (0.002)
Observations 63,468 63,468 63,468
R
2 0.979 0.979 0.979
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Note: Regressions include year-month and query fixed effects. Standard errors clustered at the query level are reported in parentheses. Data from July 2018 to Feb
2019.
126

Table A.11: Controlling for organic rank changes: log(CS)
(1) (2) (3)
75th 50th 25th
Post × Linguistic Complexity 0.0046∗ 0.0039∗ 0.0033
(0.0024) (0.0023) (0.0024)
Post × Cosine 0.0011∗ 0.001 0.007
(0.0006) (0.0007) (0.0010)
Post × Linguistic Complexity × Cosine −0.0011 −0.0013 −0.0016
(0.0008) (0.0010) (0.0014)
log(SERP Features Count) −0.0003 −0.0003 −0.0003
(0.0004) (0.0004) (0.0004)
Feature Snippet 0.0069∗∗∗ 0.0069∗∗∗ 0.0069∗∗∗
(0.0022) (0.0022) (0.0022)
Domain Change −0.0008 −0.0008 −0.0008
(0.0008) (0.0008) (0.0008)
Observations 41,555 41,555 41,555
R
2 0.9772 0.9772 0.9772
Note: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Note: All regressions include year-month and query fixed effects. Standard errors clustered
at the query level are reported in parentheses. Data is from July 2019 to February 2020.
127

Table A.12: Controlling for organic rank changes: log(CP C)
(1) (2) (3)
75th 50th 25th
Post × Linguistic Complexity −0.0158 −0.0211 −0.0206
(0.0178) (0.0169) (0.0176)
Post × Cosine 0.0052 0.0050 0.0065
(0.0039) (0.0044) (0.0063)
Post × Linguistic Complexity × Cosine −0.0043 −0.0022 −0.0047
(0.0061) (0.0075) (0.0104)
log(SERP Features Count) 0.0113∗∗∗ 0.0113∗∗∗ 0.0113∗∗∗
(0.0032) (0.0032) (0.0032)
Feature Snippet −0.0168 −0.0167 −0.0167
(0.0133) (0.0133) (0.0133)
Domain Change −0.0078 −0.0077 −0.0077
(0.0057) (0.0057) (0.0057)
Observations 41,555 41,555 41,555
R
2 0.8038 0.8038 0.8038
Note: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Note: All regressions include year-month and query fixed effects. Standard errors clustered
at the query level are reported in parentheses. Data is from July 2019 to February 2020.
128

Table A.13: Controlling for search interests and search volume: log(CS)
(1) (2) (3)
75th 50th 25th
Post × Linguistic Complexity 0.0042∗∗ 0.0036∗ 0.0036∗
(0.002) (0.002) (0.002)
Post × Cosine 0.0012∗∗ 0.0010∗ 0.0007
(0.0005) (0.0006) (0.0008)
Post × Linguistic Complexity × Cosine −0.0015∗∗ −0.002∗∗ −0.003∗∗∗
(0.0007) (0.0008) (0.0012)
Interest 0.00004 0.00004 0.00004
(0.00004) (0.00004) (0.00004)
log(SV) 0.0062∗∗∗ 0.0062∗∗∗ 0.0062∗∗∗
(0.0023) (0.0023) (0.0023)
Observations 63,394 63,394 63,394
Note: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Note: All regressions include year-month and query fixed effects. Standard Errors
clustered at the query level are reported in parentheses. Data is from July 2019 to
February 2020.
129

Table A.14: Controlling for search interests and search volume: log(CP C)
(1) (2) (3)
75th 50th 25th
Post × Linguistic Complexity −0.0254∗ −0.0274∗∗ −0.0290∗∗
(0.0147) (0.0139) (0.0145)
Post × Cosine 0.0057∗ 0.0066∗ 0.0077
(0.0032) (0.0037) (0.0052)
Post × Linguistic Complexity × Cosine −0.0003 0.0006 0.00001
(0.0051) (0.0064) (0.0092)
Interest 0.0001 0.0001 0.0001
(0.0003) (0.0003) (0.0003)
log(SV) 0.0142 0.0142 0.0141
(0.0129) (0.0129) (0.0129)
Observations 63,394 63,394 63,394
R
2 0.7951 0.7951 0.7950
Note: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01
Significance Levels: ∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01.
Note: All regressions include year-month and query fixed effects. Standard Errors
clustered at the query level are reported in parentheses. Data is from July 2019 to
February 2020.
130

Appendix B
The Financial Consequences of Legalized Sports Gambling
B.1 Treatment Type and Handles
In Table B.2, we present cumulative handle amounts (total amount wagered) by state and
channel. On average, we observe that roughly 91% of betting is done online in our data.
In Table B.1, we report legalization dates by state.
131

Table B.1: Treatment start dates in our dataset.
State First Start Online Offline
1 Delaware Jun 2018 Jun 2018
2 New Jersey Jun 2018 Aug 2018 Jun 2018
3 Mississippi Aug 2018 Aug 2018
4 West Virginia Aug 2018 Aug 2018 Aug 2018
5 New Mexico Oct 2018 Oct 2018
6 Pennsylvania Nov 2018 May 2019 Nov 2018
7 Rhode Island Nov 2018 Sep 2019 Nov 2018
8 Arkansas Jul 2019 Mar 2022 Jul 2019
9 New York Jul 2019 Jan 2022 Jul 2019
10 Iowa Aug 2019 Aug 2019 Aug 2019
11 Indiana Sep 2019 Oct 2019 Sep 2019
12 Oregon Oct 2019 Oct 2019
13 New Hampshire Dec 2019 Dec 2019 Aug 2020
14 Illinois Mar 2020 Jun 2020 Mar 2020
15 Michigan Mar 2020 Jan 2021 Mar 2020
16 Montana Mar 2020 Mar 2020
17 Colorado May 2020 May 2020 May 2020
18 District of Columbia May 2020 May 2020 Jul 2020
19 Tennessee Nov 2020 Nov 2020
20 Virginia Jan 2021 Jan 2021
21 North Carolina Mar 2021 Mar 2021
22 North Dakota Jun 2021 Jun 2021
23 Arizona Sep 2021 Sep 2021 Sep 2021
24 South Dakota Sep 2021 Sep 2021
25 Washington Sep 2021 Sep 2021
26 Wyoming Sep 2021 Sep 2021
27 Connecticut Oct 2021 Oct 2021 Oct 2021
28 Louisiana Nov 2021 Jan 2022 Nov 2021
29 Wisconsin Nov 2021 Nov 2021
30 Maryland Dec 2021 Nov 2022 Dec 2021
31 Kansas Sep 2022 Sep 2022 Sep 2022
32 Massachusetts Jan 2023 Mar 2023 Jan 2023
33 Ohio Jan 2023 Jan 2023 Jan 2023
132

Table B.2: Average handle by channel per state. The data does not include handles for
states where tribal lands run the offline sports gambling market.
State Online Retail Pct. Online Cum. Handle
1 New Jersey 34, 569, 741, 984 3, 713, 428, 792 0.90 38, 283, 170, 776
2 New York 22, 913, 323, 803 494, 149, 829 0.98 23, 407, 473, 632
3 Illinois 21, 675, 191, 603 897, 187, 635 0.96 22, 572, 379, 238
4 Pennsylvania 20, 038, 138, 857 2, 017, 121, 329 0.91 22, 055, 260, 187
5 Colorado 11, 950, 981, 364 148, 645, 515 0.99 12, 099, 626, 879
6 Indiana 10, 981, 313, 615 1, 342, 513, 286 0.89 12, 323, 826, 902
7 Michigan 10, 045, 093, 194 778, 801, 244 0.93 10, 823, 894, 438
8 Virginia 10, 019, 131, 704 0 1 10, 019, 131, 704
9 Arizona 9, 538, 088, 892 87, 268, 559 0.99 9, 625, 357, 452
10 Tennessee 8, 622, 329, 752 0 1 8, 622, 329, 752
11 Iowa 5, 334, 919, 136 834, 747, 402 0.86 6, 169, 666, 538
12 Louisiana 2, 974, 460, 677 531, 448, 026 0.85 3, 505, 908, 703
13 Ohio 2, 932, 320, 051 81, 894, 000 0.97 3, 014, 214, 051
14 Maryland 2, 400, 918, 372 410, 743, 480 0.85 2, 811, 661, 852
15 Connecticut 2, 400, 899, 080 155, 720, 243 0.94 2, 556, 619, 323
16 New Hampshire 1, 848, 595, 721 444, 716, 787 0.81 2, 293, 312, 508
17 Massachusetts 1, 571, 946, 198 70, 122, 644 0.96 1, 642, 068, 843
18 Kansas 1, 506, 528, 875 71, 935, 725 0.95 1, 578, 464, 600
19 West Virginia 1, 412, 507, 612 593, 207, 201 0.70 2, 005, 714, 813
20 Oregon 1, 254, 314, 057 0 1 1, 254, 314, 057
21 Rhode Island 817, 111, 648 835, 880, 558 0.49 1, 652, 992, 206
22 Arkansas 247, 307, 519 198, 623, 807 0.55 445, 931, 327
23 Wyoming 238, 202, 106 0 1 238, 202, 106
24 DC 159, 734, 511 419, 702, 471 0.28 579, 436, 981
25 Mississippi 0 2, 211, 473, 311 0 2, 211, 473, 311
26 Delaware 0 562, 446, 621 0 562, 446, 621
27 Montana 0 143, 854, 952 0 143, 854, 952
28 South Dakota 0 12, 888, 714 0 12, 888, 714
29 New Mexico NA NA NA NA
30 North Carolina NA NA NA NA
31 North Dakota NA NA NA NA
32 Washington NA NA NA NA
33 Wisconsin NA NA NA NA
133

Asset Metadata

Creator Larsen, Poet Ossian (author)

Core Title The downstream consequences of platform design and market entry on consumers and firms

Contributor Electronically uploaded by the author (provenance)

School Marshall School of Business

Degree Doctor of Philosophy

Degree Program Business Administration

Degree Conferral Date 2025-05

Publication Date 04/23/2025

Defense Date 04/14/2025

Publisher University of Southern California (original), Los Angeles, California (original), University of Southern California. Libraries (digital)

Tag advertising,Auctions,digital platforms,sponsored search,sports gambling

Format theses (aat)

Language English

Advisor Proserpio, Davide (committee chair), Mayzlin, Dina (committee member), Malik, Nikhil (committee member), Camara, Odilon (committee member), Luo, Lan (committee member)

Creator Email poet.larsen@marshall.usc.edu

Unique identifier UC11399KEUT

Identifier etd-LarsenPoet-13971.pdf (filename)

Legacy Identifier etd-LarsenPoet-13971

Document Type Dissertation

Format theses (aat)

Rights Larsen, Poet Ossian

Internet Media Type application/pdf

Type texts

Source 20250428-usctheses-batch-1256 (batch), University of Southern California Dissertations and Theses (collection), University of Southern California (contributing entity)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the author, as the original true and official version of the work, but does not grant the reader permission to use the work if the desired use is covered by copyright. It is the author, as rights holder, who must provide use permission if such use is covered by copyright.

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA

Repository Email uscdl@usc.edu

Abstract (if available)

Abstract In this thesis, we study the downstream consequences of digital platform design and market entry on consumers and firms. We utilize various methods, such as causal inference, machine learning, econometrics, and analytical economic modeling to generate novel insights that speak to firms, consumers, and policymakers.

Digital platforms, such as search engines, social media platforms, websites, and mobile apps, face many design decisions that directly impact the consumers and businesses they interact with. In this stream of research, we study how platform design changes affect businesses, particularly advertisers, in the context of sponsored search.

In the first chapter, we study how improvements to search engine interpretation algorithms and the information signals they generate affect sponsored search markets. We focus on two outcomes: the number of advertisers bidding for a query (i.e., competition) and cost-per-click (CPC). We start by developing a theoretical auction model. We find that as the quality of a search engine's interpretation algorithm improves, the number of bidders allocated to auctions generally increases for all queries by improving the platform's ability to identify relevant advertisers more often. Despite this, prices may decline in some markets. Specifically, we find that shifts in CPC depend on the prevalence of context in a query. For queries lacking context (e.g., shorter queries), prices generally increase. However, for queries with more contextual information (e.g., longer queries), prices may decrease. This can occur when a platform's new algorithm significantly improves contextual interpretation capabilities, leading to more precise advertiser relevancy score estimates and therefore weaker competition among bidders. We then test the model predictions using a monthly dataset of competition scores and CPC for 12,000 queries, leveraging Google's October 2019 rollout of Bidirectional Encoder Representations from Transformers (BERT) as a natural experiment. Employing two Difference-in-Differences identification strategies, we find results consistent with the theoretical model.
Our results offer insight into the economic impact of AI and Large Language Models on advertising markets and help advertisers prepare for future algorithm updates.

In the second chapter of this thesis, we study how the development of new digital platforms can impact consumers and focus in on the context of sports gambling.

Following a 2018 ruling of the U.S. Supreme Court, 38 states have legalized sports gambling. We study how this policy has impacted consumer financial health using a large and comprehensive dataset on consumer financial outcomes. We use data from the University of California Consumer Credit Panel, containing credit rating agency data for a representative sample of roughly 7 million U.S. consumers. We exploit the staggered rollout of legal sports betting across U.S. states and evaluate two treatment effects: the presence of any legal sports betting in a state and the specific presence of online or mobile access to betting.
Our main finding is that overall consumers' financial health is modestly deteriorating as the average credit score in states with legalized sports gambling decreases by roughly $0.8$ points. When states introduce access to online sports gambling, average credit scores decline by nearly three times as much ($2.75$ point decline). The decline in credit score is associated with changes in indicators of excessive debt. We find a substantial increase in average bankruptcy rates, debt sent to collections, use of debt consolidation loans, and auto loan delinquencies.
Together, these results indicate that the ease of access to sports gambling is harming consumer financial health by increasing their level of debt.