University of Southern California Dissertations and Theses

Towards combating coordinated manipulation to online public opinions on social media

(USC Thesis Other)

Towards combating coordinated manipulation to online public opinions on social media

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Towards Combating Coordinated Manipulation to Online Public Opinions on Social Media
by
Yizhou Zhang
A Dissertation Presented to the
FACULTY OF THE USC GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF PHILOSOPHY
(COMPUTER SCIENCE)
December 2024
Copyright 2024 Yizhou Zhang

Dedication
This work is dedicated to my advisor, Prof. Yan Liu, my defense committee members, Prof. Jieyu Zhao and
Prof. Kimon Drakopoulos, my friend, labmates and collaborators, and my beloved family.
ii

Acknowledgements
First of all, I would like to express my sincere gratitude to my Ph.D advisor, Prof. Yan Liu. During the
five years that I have been pursuing my Ph.D degree, she gave me a lot of insightful ideas in research
exploration, priceless suggestions on problem resolutions, and quite useful experience in drafting papers. I
am fortunate to learn from her on how to do meaningful and impactful research.
I would also like to extend my sincere thanks to my thesis committee members, Prof. Jieyu Zhao and
Prof. Kimon Drakopoulos. Their constructive comments and insightful suggestions have significantly
enriched my research. I am grateful for their time and support in helping me refine my work.
I am also grateful for Prof. Emilio Ferrara, Prof. Yue Zhao, Prof. Xiang Ren, Prof. Greg Ver Steeg, Prof.
Muhao Chen. In my long journey working on my thesis proposal and qualification exam, they provide
me with invaluable suggestions, which guides my work and helps me navigate through challenges. I am
fortunate to have had the opportunity to learn from such distinguished scholars, whose insights have
shaped my research in meaningful ways.
During my research, I have been fortunate to collaborate with outstanding scholars Dr. Karishma
Sharma, Dr. Nitin Kamra, Dr. Sirisha Rambhatla, Dr. Chuizheng Meng, Dr. Ram Nevatia, Dr. Zijun Cui, Loc
Trinh, Defu Cao, Wen Ye, Wei Yang, Lumingyuan Tang, Jie Cai. Their intellect and teamwork improved the
depth and breadth of my work. Additionally, I am thankful to my labmates, James Enouen, Sam Griesemer,
Emily Nguyen, Sajjad Shahabi and Yongchan Hong, Bryan Ramirez-Gonzalez who have been not only my
peers in research but also friends in everyday life. I am also fortunate to collaborate with Dr. Wei Cheng,
iii

Dr. Jingchao Ni, Dr. Liang Tong, Dr. Zhengzhang Chen, and Dr. Haifeng Chen, Dr. Qiang Fu and Lun Du
during my internships. The collaboration, discussions, and companionship in the lab helped me overcome
challenges and made this journey fulfilling.
Finally and mostly, I want to express my heartfelt thanks to my friends and family, who have supported
and encouraged me every step of the way. Your understanding and companionship have been essential to
my progress, and I am forever grateful for your unwavering support.
iv

Table of Contents
Dedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Chapter 1: Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Summary of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 Thesis Outlines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.4 Related Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Chapter 2: Existing Works in Misinformation Mitigation . . . . . . . . . . . . . . . . . . . . . . . . 9
2.1 Influence of Misinformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2 Social Media Manipulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2.1 Classical Feature based Social Manipulator Detection . . . . . . . . . . . . . . . . . 12
2.2.2 Graph based Social Manipulator Detection . . . . . . . . . . . . . . . . . . . . . . . 13
2.2.2.1 Graph-Structured Data Generation . . . . . . . . . . . . . . . . . . . . . 13
2.2.2.2 Graph-based Detection Models . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2.3 Activity Representation Learning based Social Manipulator Detection . . . . . . . 15
2.3 Large Language Model and Social Manipulators . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3.1 Opportunities of LLM based Manipulator Detection . . . . . . . . . . . . . . . . . . 16
2.3.2 Challenges in Detecting LLM based Manipulator . . . . . . . . . . . . . . . . . . . 17
2.4 Causal Inference on Social Media . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.4.1 Correlation vs Causality on Social Media . . . . . . . . . . . . . . . . . . . . . . . . 18
2.4.2 Causal Analysis on Social Media . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.5 Misinformation Detection in Large Language Model Era . . . . . . . . . . . . . . . . . . . 20
2.5.1 Paradigm of Applying LLM in Misinformation Detection . . . . . . . . . . . . . . . 20
2.5.2 LLM Prompting Strategies in Misinformation Detection . . . . . . . . . . . . . . . 21
2.6 New Challenges and Opportunities in Foundation Model Era . . . . . . . . . . . . . . . . . 22
2.6.1 LLM-boosted Social Manipulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.6.2 LLM-boosted Fake News Generation . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Chapter 3: Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
v

3.1 Coordination Detection Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.1.1 U.S. 2016 Election . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.1.2 COVID-19 Pandemic Twitter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.1.3 COVID-19 Vaccine Twitter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.2 Causal Analysis Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.2.1 Synthetic Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3 Fact Verification Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.3.1 Hallucination Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.3.2 Misinformation Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Chapter 4: Data-Driven Detection of Coordinated Social Manipulation . . . . . . . . . . . . . . . . 32
4.1 Preliminary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.1.1 Task Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.1.2 Marked Temporal Point Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.2 AMDN-HAGE: Attentive Mixture Density Networks with Hidden Account Group Estimation 38
4.2.1 Attentive Mixture Density Networks: Modeling Activity Traces . . . . . . . . . . . 39
4.2.1.1 Encoder: Masked Attentive History Encoder . . . . . . . . . . . . . . . . 41
4.2.1.2 Decoder: Conditional Probability Density Function . . . . . . . . . . . . 42
4.2.2 Hidden Account Group Estimation: Modeling Account Groups . . . . . . . . . . . 44
4.2.3 Joint Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.3 Detecting Coordinated Accounts in U.S. 2016 Election . . . . . . . . . . . . . . . . . . . . . 48
4.3.1 Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.3.2 Evaluation Metrics, Baselines and Model Variants . . . . . . . . . . . . . . . . . . . 49
4.3.3 Results of Coordination Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.3.3.1 Comparison with Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.3.3.2 Ablation Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.3.4 Analysis of Coordination Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.4 Uncovering Suspicious Coordinated Efforts on Social Media during the COVID-19 Pandemic 55
4.4.1 Experiment Set-up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.4.2 Coordination Detection Results on COVID-19 Dataset . . . . . . . . . . . . . . . . 56
Chapter 5: Knowledge-Informed Detection of Coordinated Social Manipulation . . . . . . . . . . . 59
5.1 Task Definition: Coordinated Group Detection on Social Media . . . . . . . . . . . . . . . . 61
5.2 Proposed Method: VigDet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
5.2.1 Prior Knowledge-based Graph Construction . . . . . . . . . . . . . . . . . . . . . . 62
5.2.2 Integrate Prior Knowledge and Neural Temporal Point Process . . . . . . . . . . . 63
5.2.2.1 E-step: Inference Procedure. . . . . . . . . . . . . . . . . . . . . . . . . . 65
5.2.2.2 M-step: Learning Procedure. . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.2.2.3 Joint Training: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
5.2.3 Semi-supervised extension . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
5.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.3.1 Coordination Detection on IRA Dataset . . . . . . . . . . . . . . . . . . . . . . . . 71
5.3.1.1 Evaluation Metrics and Baselines . . . . . . . . . . . . . . . . . . . . . . 72
5.3.1.2 Implementation details on IRA dataset . . . . . . . . . . . . . . . . . . . 73
5.3.1.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5.3.2 Analysis on COVID-19 Vaccines Twitter Data . . . . . . . . . . . . . . . . . . . . . 75
5.3.2.1 Implementation details on COVID-19 Vaccine Tweets dataset . . . . . . 75
5.3.2.2 Results on COVID-19 Vaccine Tweets dataset . . . . . . . . . . . . . . . 76
vi

Chapter 6: Revealing the Causal Influence of Misinformation on Social Media Users . . . . . . . . 79
6.1 Preliminary and Related Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
6.1.1 Temporal Point Process with Event Features . . . . . . . . . . . . . . . . . . . . . . 82
6.1.2 Counterfactual Analysis on Temporal Point Process and Continuous Time Series . 82
6.2 Proposed Causal Structure Model and Treatment Effect . . . . . . . . . . . . . . . . . . . . 83
6.2.1 Causal Structure Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
6.2.2 Treatment Effect Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
6.2.3 Treatment Effect Calculation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
6.3 Neural Estimation of Treatment Effects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
6.3.1 Learning Conditional Intensity Function via Maximum Likelihood Estimation . . . 89
6.3.2 Adversarial Balanced Neural Temporal Point Process . . . . . . . . . . . . . . . . . 91
6.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
6.4.1 Experiments on Synthetic Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
6.4.2 Experiments on Real World Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
6.4.2.1 Identifiability between misinformation and information in influencing
people’s narratives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
6.4.2.2 Misinformation is hurting people’s subjective emotion related to COVID
vaccine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
Chapter 7: Making LLM Discerning to Misinformation with Divide-and-Conquer Prompt . . . . . 101
7.1 Preliminary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
7.1.1 Expressive Power of Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
7.1.2 Prompting Strategies of LLM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
7.2 Divide-and-Conquer Prompting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
7.3 Main Theoretic Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
7.3.1 Divide-and-Conquer vs. IO Prompting . . . . . . . . . . . . . . . . . . . . . . . . . 110
7.3.2 DaC vs. CoT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
7.3.3 Advantages of DaC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
7.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
7.4.1 Hallucination Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
7.4.2 Misinformation Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
7.4.3 Discussions and Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Chapter 8: Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
Appendix A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
Experiment Details on COVID-19 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
vii

List of Tables
1.1 Summary of the presented works. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 Notation Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4.1 Results on detection of coordinated accounts (IRA dataset) on Twitter in 2016 U.S. Election 53
4.2 Overlap between suspended Twitter accounts, and identified coordinated groups/ overall
accounts in collected COVID-19 data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.3 Representative tweets in disinformation topic clusters in identified COVID-19 coordinated
accounts groups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
5.1 Results on unsupervised coordination detection (IRA) on Twitter in 2016 U.S. Election . . . 74
5.2 Results on semi-supervised coordination detection (IRA) on Twitter in 2016 U.S. Election . 75
5.3 Representative tweets from topic clusters in tweets of identified coordinated accounts. . . 78
6.1 Estimation Error to the ground-truth ITE . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
6.2 Causal Effect Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
6.3 Comparison of (normalized) Average Sum of Distances with different methods on the
real-world dataset. This metric reflect how well the a group of data points is clustered. . . 97
7.1 Performance of different prompting methods on HaluEval dataset. . . . . . . . . . . . . . 123
7.2 Performance of different prompting methods on SciFact dataset. . . . . . . . . . . . . . . . 123
viii

List of Figures
1.1 Summary of research thrusts in the thesis. . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.1 This figure describe a basic case of correlation vs causality. Due to the confounding factor
(political stance), we can not tell whether the the anti-vaccine attitudes is caused by the
conspiracy or not. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2 The common causal models describing misinformation spreading on social media. . . . . . 19
3.1 The general idea of generating the synthetic dataset. . . . . . . . . . . . . . . . . . . . . . . 28
4.1 Coordinated accounts suspended by Twitter officially (also detected by our proposed
method). Example tweets in the figure spread political conspiracies collaboratively. The
time differences between the tweets with same contents vary from less than 6 hours to
more than half a week. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.2 Architecture of proposed (AMDN-HAGE) to model conditional density function of account
activities and hidden groups on social media. . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.3 The silhouette score with different cluster numbers on IRA and COVID-19 datasets. . . . . 49
4.4 Comparison between our proposed optimization algorithm and standard Adam. . . . . . . 53
4.5 Analysis to the learnt influence and account embedding of AMDN-HAGE. . . . . . . . . . 54
4.6 Analysis to the learnt timely influence of AMDN-HAGE. . . . . . . . . . . . . . . . . . . . 54
4.7 Top-35 frequent unique tweet hashtags of the normal users and the suspicious coordinated
group (the larger one). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
5.1 The figure (a) is an example of coordinated accounts detected by our method on Twitter.
They retweet similar anti-vaccine contents about COVID-19 Vaccines from same or
different sources. The figure (b) is the frequency statistic of accounts in IRA dataset about
the U.S. 2016 Election. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
ix

5.2 The overview of VigDet. In this framework, we aim at learning a knowledge informed
data-driven model. To this end, based on prior knowledge we construct a graph describing
the potential of account pairs to be coordinated. Then we alternately enhance the prediction
of the data-driven model with the prior knowledge based graph and further update the
model to fit the enhanced prediction as well as the observed data. . . . . . . . . . . . . . . 64
5.3 The silhouette scores of different group number. . . . . . . . . . . . . . . . . . . . . . . . . 76
5.4 Top-30 hashtags in tweets of suspicious coordinated group and normal group . . . . . . . 77
6.1 The proposed causal structured model describing the impact of a piece of information on user. 84
6.2 The proposed neural model to estimate the impact of misinformation. . . . . . . . . . . . . 89
6.3 Analysis on real world social media data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
6.4 Visualization of the results from baselines on real world. . . . . . . . . . . . . . . . . . . . 99
7.1 An illustrative example of hallucination detection with entangled problem solving (i.e.,
directly forward all inputs into the LLM) and divide-and-conquer problem solving (i.e.,
divide the problem inputs to parallel sub-tasks and tackle them parallelly). The sentence
marked with red back font in the material is the evidence that contradicts the first claim in
the summary (marked with red font). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
7.2 The comparison between DaC and the existing methods for prompting. The ellipse marks
represent sub-tasks, the right-angled rectangles represent sub-task solutions, and the
rounded rectangles represent intermediate steps that entangle sub-tasks and sub-solutions.
The different shades in Tree of Thoughts (subfigure D) indicate the rates of different search
directions. In CoT (Chain-of-Thoughts), CoT-SC, and ToT, the Large Language Models
must simultaneously generate and resolve sub-tasks. Least-to-Most (also Decomposed
Prompting) disentangles sub-task generation and resolution. However, its sub-task
resolution and resolution assembly process are intertwined as it sequentially attaches new
sub-tasks to the previous resolution. Different from them, DaC totally disentangles the
sub-task generation, sub-task resolution, and resolution assembly process. . . . . . . . . . 105
7.3 Comparison of Least-to-Most (LtM) Prompting and Decomposed Prompting (DeP). . . . . 107
A.1 The hashtag distribution in the smaller anomalous group (C1). . . . . . . . . . . . . . . . . 145
x

Abstract
Over the recent years, public opinion and online credibility have been suffering from the manipulation of
campaigns that control malicious accounts to document and spread misinformation with specific narratives
such as fake news and conspiracy theories. Such campaigns, also known as misinformation campaigns, are
increasingly threatening various areas related to public opinions and decisions, such as politics and public
health. Such threats, prominent in highly scrutinized societal events like the U.S. Presidential Elections and
the COVID-19 pandemic, have significantly undermined societal trust and public interests.
My thesis will discuss how to exploit machine learning to discover knowledge and skills that help combat
this aforementioned social manipulation. More specifically, my thesis will present my research attempts
to apply machine learning algorithms in three directions: Manipulation Source Identification, Susceptible
Population Recognition, and Automated Authenticity Verification. To identify the online manipulation
from misinformation campaigns, my collaborators and I developed a series of neural temporal point process
models that can recognize patterns of coordinated manipulators with data-driven learning and domain
knowledge. To identify users susceptible to specific misinformation, we developed a counterfactual neural
network that can estimate the causal effect of a piece of misinformation on an individual user or a group of
populations. To complete our target of automated authenticity verification, we use the advances of large
language models (LLM), which can clarify misinformation and provide a reference for accurate information.
To achieve this goal, more work is conducted on developing robust prompting engineering strategies to
prevent the LLM from being deceived by misinformation when verifying the genuineness of a given text.
xi

Chapter 1
Introduction
Recent years have witnessed the rapid growth of misinformation campaigns, which control malicious
accounts such associal-bots (i.e., accounts operated by automatic scripts) and trolls (i.e., accounts with human
operators)[97, 20, 59] to manipulate the public opinions on social media. For example, according to the
investigation by the U.S. Congress, the Internet Research Agency (IRA), a Russian organization, registered
and controlled a large amount of social media accounts to intervene in the U.S. 2016 election by spreading
misinformation posts with specific narratives [97]. Similarly, during the COVID-19 epidemic, research works
from various resources revealed the existence of efforts that coordinated malicious accounts to increase the
visibility of misinformation about the COVID-19 epidemic and vaccines, such as conspiracies that scare
people to avoid getting COVID-19 vaccines[135]. To effectively and efficiently address these challenges
and risks, I have been studying and developing machine learning models to understand misinformation
campaigns and mitigate their negative influence.
Machine learning, as a subarea of artificial intelligence, presents a potential to automatically discover
knowledge and skills as a supplement to human experts in communication studies and social media service
[134]. For example, rumors and fake news threaten the credibility of online information and social media
platforms [119, 98]. As a supplement to human verifiers, machine learning can distinguish potential rumors
1

Thrust 2: Estimate Causal
Impact of Misinformation to
communities and individual
users.
Thrust 1: Manipulation
Detection with Machine
Learning Models and
Domain Knowledge
Results Acquired:
Counterfactual Neural Network
for Estimation of Misinformation
Impact
Thrust 3: Automated
Generation of Clarification
using Large Language Models
Results Acquired:
Prompting and Inference
Framework for Large Language and
Multimodal Models on Deceptive
Contents
Results Acquired:
Domain-Knowledge-guided
data-driven coordination
detection with latent influence
and group anomaly estimation
Clarification/Mitigation to
Misinformation
Combating Coordinated Manipulation to Online Public Opinions on Social Media
Identification of
Manipulator
Objective
Challenges
Research
Thrusts
Outputs
Recognition of
Susceptible Population
Figure 1.1: Summary of research thrusts in the thesis.
based on linguistic patterns, such as deceptive wording and opinions of intimidation. My thesis will focus
on applying machine learning to combat coordinated online manipulation.
Specifically, my work follows a paradigm inspired by epidemic control and prevention. In epidemiology,
successful pandemic control relies on three measures: localizing infectious sources, searching susceptible
populations, and producing and distributing protective measures (e.g., drugs, vaccines, and masks). Similarly,
in combating social manipulation based on misinformation, we also face three challenging tasks: Identifying Manipulation Source, Recognizing Susceptible Population, and Verification/Clarification of
Misinformation. To address these challenges, I have worked with my collaborators to develop algorithms
and models incorporated with advanced machine-learning techniques and domain knowledge. In detail,
my accomplished research thrusts for this thesis include:
Machine Learning Models for Identifying Misinformation Campaigns: A critical challenge in
understanding and combating misinformation campaigns is distinguishing their coordinated accounts from
normal users deceived by misinformation. Existing research and investigations have indicated that modern
misinformation campaigns have various strategies to bypass detection, such as recruiting professional
human operators and incorporating their social bots with advanced Large Language Models (LLM) and
AI-Generate Content (AIGC). To address this challenge, my collaborators and I propose that the activities of
2

malicious accounts belonging to the same misinformation campaigns are anomalously concerted compared
to normal users. Therefore, we can detect them by capturing such correlations with deep-learning models in
an unsupervised manner. Our first result is AMDN-HAGE [138], a data-driven model that jointly models the
account behaviors and collective influence in an embedding space where the high correlation of coordinated
accounts is represented as an anomalously concerted cluster. To better benefit detectors from human
expertise, we further developed VigDet [182], a knowledge-informed coordination detector that learns from
both activity data and human prior knowledge. We further applied our models to analyze the social media
posts about the COVID-19 Vaccine on Twitter. Our analysis reveals the existence of coordinated efforts to
manipulate public opinions on COVID-19 vaccines.
Counterfactual Analysis Models for Estimating Causal Affect of Misinformation: Similar to
epidemic control and prevention, mitigating misinformation manipulation also requires us to identify
communities that are susceptible to specific deceptive contents and narratives. This requirement naturally
motivates us to develop machine-learning models that can estimate the causal effect of misinformation on
individual users and communities. To this end, my collaborators and I developed a Counterfactual Neural
Temporal Point Process model [180] to estimate the causal effect of misinformation on social media users’
behavior. It applies adversarial training to reduce information cocoon bias. We applied it to the Twitter
posts about COVID-19 vaccines and discovered that the misinformation is causally responsible for vaccine
hesitancy.
LLM-based Automated Verification and Clarification to Misinformation: Existing works have
achieved success in detecting misinformation to a certain extent by applying advanced deep-learning
algorithms [119, 126, 144]. However, their success relies on the powerful hidden representation spaces
learned by artificial neural networks, which are not interpretable and understandable to the public. As a
result, social media platforms usually need to additionally recruit human verifiers to document authenticity
reports that are easy to understand from a human’s perspective. To alleviate the load of verifiers and
3

Table 1.1: Summary of the presented works.
Focuses Work Contributions
Identifying Manipulators of
Misinformation Campaigns
on Social Media Platforms.
[138]
Developing a data-driven machine learning model to detect accounts
coordinated by manipulators.
[182]
Incorporating data-driven machine learning models with domain knowledge
to acquire better performance in identifying coordinated manipulators.
Estimating Causal Affect of
Misinformation to Social
Media Users.
[180]
Using counterfactual neural networks to analysis the causal effect of
misinformation on individual users and communities.
Prompting Strategies for
Misinformation Detection
to Generate Authenticity
Evaluation.
[181]
Developing a theoretic guarantee for Divide-and-Conquer prompting
in recognizing misinformation.
platforms, this thesis studies applying machine learning models to draft authenticity reports for news
articles automatically. To achieve this goal, we plan to exploit the advances of Large Language Models
in generating text [181]. A critical challenge in this direction is preventing LLM from being deceived
by the misinformation. To this end, we explore how to design robust prompts guided by a program or
symbolic procedure to prevent the LLM from being deceived or interrupted by the deceptive contents of
the misinformation.
1.1 Summary of the Thesis
In this thesis, we present the challenges of Coordinated Manipulations on Social Media and introduce the
methodologies that contribute to combatting them. Misinformation campaigns are increasingly manipulating online public opinion through fake news, especially during times of uncertainty. For example, during
the Presidential Elections of 2016 and 2020, fake news widely spread on online social media platforms
4

became a severe threat to the presidential election, such as the fake news about election cheating. According
to a study from George Washington University, more than 6.6 million tweets were linked to fake news and
conspiracy news publishers during the 2016 election [63]. To combat such threats, we propose machine
learning techniques that can identify social manipulators, recognize the causal influence of misinformation,
and boost the advanced Large Language Models in verifying and clarifying misinformation. By evaluating
the proposed methods on benchmark datasets and analyzing them theoretically, we validate their advantages in tackling the confronted challenges. We further apply these methods to datasets associated with
real-world events such as the presidential election, the COVID-19 pandemic, and COVID-19 vaccines. The
experimental results show that our proposed method can behave well in these real-world scenarios.
1.2 Thesis Outlines
The whole thesis is organized as follows:
In Chapter 2, we will discuss the existing research and discovered phenomenon related to this thesis.
Specifically, we discussed how misinformation impacts our society in different areas, such as politics and
healthcare. Then, we will discuss how social manipulators use misinformation and what people have
already done to alleviate their harmful activities.
In Chapter 3, we will discuss how we prepare the datasets for our thesis. We applied the datasets about
misinformation campaign detection, causal analysis on social media, and hallucination/misinformation
detection.
In Chapter 4, we will present AMDN-HAGE, an effective coordinated account detection model. In this
work, we developed an unsupervised transformer-based model that captures the hidden correlation and
concerted activities of malicious accounts on social media. We applied it to the data about the presidential
election and the COVID-19 pandemic. Experiments indicate the effectiveness of the proposed method.
5

In Chapter 5, we will present VigDet, a knowledge-informed coordinated account detection model. In
this work, we developed a variational inference framework incorporating data-driven detectors with human
experts’ knowledge encoded as interaction intensity graphs. Experiments indicate the proposed method
can significantly boost the performance of data-driven detectors with human knowledge and acquire better
performance.
In Chapter 6, we will present the Counterfactual Neural Temporal Point Process (CNTPP). In this work,
we developed a causal structure model describing how misinformation impacts social media users’ activities.
To infer the parameter, we proposed an adversarial balanced training framework to reduce the bias brought
by the modern recommendation systems of social media platforms. By applying this work to synthetic
data, we indicate the effectiveness and efficiency of the proposed method. We further apply the model to
real-world data and recognize the significant difference between real news and misinformation.
In Chapter 7, we will present Divide-and-Conquer prompting (DaC), which improves the capacity of
large language models to recognize misinformation and deceptive content. In this work, we first derived a
theoretical analysis of the advantage of DaC prompting. Then, we present two cases where DaC can boost
large language and multimodal models in misinformation detection.
6

1.3 Notations
X Input matrix, such as user features, post contents and covariates in causal
analysis
Y Output matrix or vector, such as labels of the input samples (i.e., user assignment and posts authenticity) and the outcomes in causal analysis
x, y Individual input and output, like on row in X and one item in Y
S Event sequences where each each event is attached with a timestamp and a
descriptor (can be user id or event feature)
u, v User id on social media, regarded as marks in marked temporal point process
f Event feature, such as spatial coordinates or text features
t Timestamp
Ht History event set before time t
λv(t|Ht) Intensity function of user v at time t given the history before time t
λ(f, t|Ht) Intensity function with feature f at time t given the history before time t
p(·) Probability density function
p(S, U; θ1,2,...) Probability density of observing an event sequence and corresponding user
ids given parameter θ1,2,...
Hattn History encoding output by an attention module[153]
Q, K, V Query matrix, key matrix and value matrix in Transformer[153]
wi
, µi
, si The weight, mean and standard error of the i-th cluster in a Gaussian mixture
model.
G =< V, E > A graph G with vertex set V and edge set E
wuv The weight of an edge (u, v) in a graph
1(·) Identical function that is 1 when the input is true and 0 otherwise
Φ(·; ·) Energy function with parameters
φ(·) Univariable energy function that only measures the energy of one node
ϕ(·) Binary energy function that measures the mutual energy of a node pair
DKL KL-divergance
E Expect of a random variable
N(·) Counting function that count the event numbers in a data
P(·) The problem set that can be solved by a method (such as prompting strategies)
Table 1.2: Notation Summary
7

1.4 Related Publications
• Chapter 2 is related to the content in: Yizhou Zhang, Karishma Sharma, Lun Du and Yan Liu Toward
Mitigating Misinformation and Social Media Manipulation in Large Language Model Era (Tutorial),
The Web Conference 2024.
• Chapter 3 and 4 are related to the content in: Sharma, Karishma*, Yizhou Zhang*, Emilio Ferrara,
and Yan Liu. Identifying Coordinated Accounts on Social Media through Hidden Influence and Group
behaviors. In Proceedings of the 27th ACM SIGKDD International Conference on Knowledge Discovery
& Data Mining, pp. 324-334. 2021.
• Chapter 3 and 5 are related to the content in: Yizhou Zhang*, Sharma, Karishma*, and Yan Liu.
VigDet: Knowledge Informed Neural Temporal Point Process for Coordination Detection on Social
Media. Advances in Neural Information Processing Systems 34 (2021): 3218-3231.
• Chapter 3 and 6 are related to the content in: Yizhou Zhang*, Defu Cao*, and Yan Liu. Counterfactual
Neural Temporal Point Process for Estimating Causal Influence of Misinformation on Social Media.
Advances in Neural Information Processing Systems 35 (2022): 10643-10655.
• Chapter 3 and 7 are related to the content in: Yizhou Zhang, Lun Du, Defu Cao, Qiang Fu, Yan Liu.
An Examination on the Effectiveness of Divide-and-Conquer Prompting in Large Language Models.
arXiv preprint arXiv:2402.05359 (2024).
8

Chapter 2
Existing Works in Misinformation Mitigation
The pervasive abuse of misinformation to influence public opinion on social media has become increasingly
evident in various domains, such as politics, as seen in presidential elections, and healthcare. For example,
a study from George Washington University [63] showed more than 6.6 million tweets linking to fake
news and conspiracy news publishers during the 2016 election. Also, during the COVID-19 pandemic, the
widespread dissemination of misleading information, which downplayed the seriousness of the epidemic
and exaggerated the side effects of COVID-19 vaccines, has had a detrimental impact on public health
and eroded trust in credible sources [44, 97]. Moreover, misinformation has frequently been harnessed to
manipulate social outcomes and public opinion, severely undermining content credibility on social media
platforms [50, 137]. Researchers have been persistently working on combating misinformation and its
manipulation of social media [3, 51, 72, 133, 134, 152].
Recent studies have primarily concentrated on detecting fake news and understanding misinformation
campaigns, with significant efforts directed toward analyzing user engagement with misleading content
and narratives [136, 139]. While some researchers have explored the correlation between misinformation
and behavioral responses, there remains a gap in understanding the causal effects of misinformation on
users. Most existing work tends to rely on psychology-focused randomized controlled trials, which, though
effective, are difficult to scale in the dynamic landscape of social media [152].
9

The manipulation of information through coordinated accounts has further complicated the scenario.
These accounts, whether human-operated or automated, can amplify specific narratives, undermining the
credibility of social media platforms[97, 138]. Researchers have made strides in identifying individual and
collective behaviors characteristic of these accounts, yet many methodologies still depend on hand-crafted
features, which can limit their effectiveness and adaptability.
Various detection methodologies have been proposed to address these challenges, ranging from manualcrafted feature analysis to advanced representation learning techniques[171, 97]. These methods aim
to capture the nuances of coordinated behavior on social media, striving for a more comprehensive
understanding of how misinformation propagates and influences user actions.
This section discussed the existing works on social media manipulation detection, causal inference
about the influence of misinformation campaigns, and the application of emerging technologies, particularly
large language models (LLMs), in misinformation detection. By examining the current state of research, we
aim to highlight gaps and opportunities for future studies, offering insights into how these technologies can
be harnessed to combat misinformation while appropriately motivating the research works we proposed
in this thesis. Through this comprehensive analysis, we seek to contribute to a deeper understanding of
misinformation’s impact on society and the mechanisms that can be employed to mitigate its effects.
2.1 Influence of Misinformation
Recent research has revealed that misinformation has a negative societal impact in various areas, such
as politics, healthcare, and public attitudes toward science agendas. In [131], the authors conducted a
series of analysis on the online user engagement with misinformation during the 2020 U.S. Presidential
Election. They found that accounts belonging to misinformation campaigns "try to engage in active
discussions (bidirectional replies) with left-leaning accounts"[131]. In [136, 140], the authors analyze how
misinformation on social media influences people’s ideas about the COVID-19 pandemic and vaccines.
10

They find that in the states where misinformation engagement is high, people’s hesitancy to get vaccinated
is more serious. In [152], the authors recruit volunteers to conduct a randomized controlled trial on how
conspiracies impact people’s recognition. Their experimental results indicate that people more exposed to
conspiracies and misinformation are more susceptible to anti-science statements.
The advantage of misinformation over true information in spreading online makes the above problems
more serious. In [155], researchers present that compared to true information, misinformation can reach
more people and spread faster. The authors attribute this phenomenon to the degree of novelty and
emotional reaction brought by misinformation. Other researchers have also reported similar findings. For
example, in [53], researchers from Facebook (now Meta) reveal that the diffusion cascade of rumors goes
longer than normal. In [150], the author attempts to explain this phenomenon with "Echo Chambers", which
means that users tend to reshare information consistent with their opinions, even for misinformation.
In addition to the impact on public opinions, misinformation also increases the risk of a polarized society.
For example, in [6], the authors illustrated how misinformation leads to ideology polarization. Furthermore,
spreading misinformation can trigger discrimination against specific demographics/communities. For
instance, during the COVID-19 pandemic, the widespread fake news and hate speech increases the Asianhate issues [64].
2.2 Social Media Manipulation
The increasing online misinformation has seriously harmed the credibility of social media platforms. To
address this challenge, researchers made a lot of efforts, ranging from fake news detection to misinformation
diffusion pattern analysis[126, 134]. A newly reported social media threat recently attracted researchers’
attention is the abuse of coordinated account[97, 135]. Unlike traditional malicious accounts, coordinated
accounts controlled by human operators or automatic scripts can efficiently raise the visibility and influence
of specific narratives by leveraging some strategies.
11

To address this new threat, researchers proposed to capture two kinds of information:
1. Individual Features. Compared to normal users, coordinated accounts usually have different
individual patterns. Existing works propose two kinds of differences. The first kind is pre-defined
explicit features, such as linguistic cues, meta data and hashtags [3, 69, 174]. The second kind is
based on inherent features, such as the motivation that drives the account activity. Such inherent
features are hard to model with hand-crafted features. Therefore, a previous research proposes to
apply Inverse Reinforcement Learning to learn the reward that drive the account activities[97].
2. Collective Behaviours. The most important reason why coordinated accounts can influence information diffusion much more significantly than normal users is that they can collectively act on social
media. As a result, they may show some anomalous collective behaviours, such as high co-appearance,
anomaly similar hashtags and time synchronization [20, 59, 113]. To detect coordinated accounts
with the above behaviours, researchers usually construct a similarity or interaction graph based
on the pre-defined coordinated behaviours. Then they conduct clustering to detect the anomalous
groups on the graphs. However, the performance of the above methods are limited by the quality
of the hand-crafted feature. Therefore, in this work, we hope to propose a methodology that can
automatically learn the collective behaviours.
To integrate the above two kinds of information, the researchers have proposed some methodologies,
which can be divided into multiple types: classical feature based method, structural feature based
method, and activity representation learning based method
2.2.1 Classical Feature based Social Manipulator Detection
To accurately classify and detect social media manipulators, we rely heavily on a detailed analysis of
classical features, including metadata and linguistic features [171]. Metadata encompasses a range of basic
but powerful indicators, such as the number of statuses, followers, and friends an account has. These
12

statistics provide us with a surface-level view of how active and connected a particular user is, which can
indicate normal or manipulative behavior.
Moving beyond the basics, we can also derive more sophisticated metrics from this metadata. For
instance, we consider the growth rate of followers and friends—real-valued features that help us discern
whether the increase in an account’s network is organic or artificially accelerated, a common trait among
bots and manipulators.
Additionally, linguistic cues are crucial. We can analyze the length and complexity of the names,
descriptions, and the overall likelihood of the screen name being genuine or fabricated [157]. For example,
accounts with many digits in their screen names or overly long user descriptions may indicate automated
setups designed to spread misinformation or manipulate discussions.
2.2.2 Graph based Social Manipulator Detection
2.2.2.1 Graph-Structured Data Generation
One typical coordinated group detection paradigm is constructing a graph measuring the similarity or
interaction between accounts and then conducting clustering on the graph or the embedding acquired by
factorizing the adjacency matrix. There are two typical ways to construct the graph. One way is to measure
the similarity or interaction with pre-defined features supported by prior knowledge or assumed signatures
of coordinated or collective behaviors, such as co-activity, account clickstream, and time synchronization
[20, 126, 158]. The other way is to learn an interaction graph by fitting the data with the temporal point
process models considering mutual influence between accounts as scalar scores as in traditional Hawkes
Process [185]. A critical drawback of both methods is that the interaction between two accounts is simply
represented as an edge with scalar weight, resulting in a poor ability to capture complicated interactions.
In addition, the performances of prior knowledge-based methods are unsatisfactory due to reliance on the
quality of prior knowledge or hypothesis of collective behaviors, which may vary with time [175].
13

To capture more information in addition to the above two methods, researchers propose introducing
more node categories of the constructed graph. For example, in [46], researchers redirect their interests
toward a heterogeneous graph defined on social media data. This graph is defined not only on the user
and their own interactions but also on more sophisticated activity interactions among a heterogeneous
node-set. Specifically, the authors extended the node set from the user account only to a wider collection
that contains users, tweets, and hashtags. Then, the edge sets are extended from containing only user-user
interactions to various relationships, such as followers, references, and so on.
2.2.2.2 Graph-based Detection Models
To make use of the graph generated from the above process, researchers proposed various approaches.
As we mentioned at the beginning, the most intuitive method mentioned is to conduct clustering on the
graph. However, this kind of approach only makes use of graph-structured data and can not aggregate the
information from other sources, such as the manual-crafted feature. To tackle this issue, recent works make
use of the advantage in graph neural networks [79], which support integrating graph-structured data with
various features describing the nodes. In [47], the authors propose to encode the information from user
descriptions, tweets, and metadata as node features and then apply a relational graph convolutional neural
network to aggregate the information and then acquire the predictions. In [172] and [67], the authors notice
that the heterogeneous information in the social media data is not well used by existing works. Therefore,
they developed multiple neural network architectures that can capture such heterogeneous information
in graph data and apply them to boost the performance of the model. In [187], the authors propose to
incorporate graph-based detectors with Large Language Models, which significantly boost the capacity of
the models to capture language information into the graph.
14

2.2.3 Activity Representation Learning based Social Manipulator Detection
In addition to the traditional modalities of metadata, linguistic cues, and graph-structured data, recent
researchers have also tried to learn account representations directly from the observed activity traces of
the user accounts. The intuition behind this idea is that, compared to real users who use social media for
more random and diverse purposes, social manipulators are more targeted at specific events, topics, and
narratives. As a result, their social media usage will have different patterns and characteristics. For example,
normal users usually use social media during their off-work time or breaks. But for social manipulators,
their active time might be quite different since social bots can be active at any time, and human operators
usually keep active in their working time. To incorporate such information, researchers propose various
methods.In [22], the authors propose that social bots’ temporal patterns differ from those of human users.
They applied a series of statistical analysis tools to localize such differences and construct corresponding
features helpful for social bot detection. In [103], the authors propose that the feature within activity traces
can be effectively learned, instead of manually designed, by auto-encoder in an unsupervised manner. In
particular, the authors train an LSTM-based autor-encoder to extract the hidden representations of user
active traces and apply clustering algorithms on the learned features. The experimental results show that
such a paradigm can distinguish normal users and social bots effectively. In [97], Inverse Reinforcement
Learning (IRL) is applied to learn the reward behind an account’s observed behavior, and the learned reward
is forwarded into a classifier as features. Specifically, IRL models social media accounts as agents trying
to maximize their rewards from the environment, i.e., the social media. Then, it aims to learn the reward
function of each agent, given their activities and the environment traces. This method uses the fundamental
difference between normal users and manipulative accounts: normal users’ reward functions are more
diverse and natural, while manipulative accounts have specific targets.
15

2.3 Large Language Model and Social Manipulators
As we transition into the era of large language models (LLMs), our ability to detect social manipulators has
evolved, offering both new opportunities and presenting unique challenges. The rise of state-of-the-art
LLMs like ChatGPT enhances our capabilities in several ways. Opportunities arise from these advances
because these state-of-the-art models can analyze data with high accuracy and efficiency. For instance,
classical features such as metadata and linguistic cues, along with structural features from network data,
can now be integrated into LLM frameworks [187]. This integration allows for better understanding and
learning directly from the context of natural language interactions, significantly boosting the accuracy
and effectiveness of detecting social bots and other manipulative entities. However, this technology also
brings challenges. The same power that enhances detection can also be abused to create more deceptive
bots. These advanced manipulators can adapt and learn how to better mimic human behavior, making
them harder to detect. As LLMs become more capable, so does the potential for their misuse increase,
necessitating advanced defensive measures that are continually updated to keep pace with evolving threats.
2.3.1 Opportunities of LLM based Manipulator Detection
We first discuss the significant opportunities that LLMs present in the field of social manipulator detection.
With recent advancements in LLM, researchers have developed various methods to leverage these models in
social bot detection [48]. One key approach is In-Context Learning. LLMs, such as ChatGPT, are capable of
understanding and processing both contextual information and instructions in a natural language input. We
can feed these models with ’few-shot’ examples (small-scale samples of labeled accounts with their metadata
like user statistics and account behavior) to teach them what a manipulative account looks like. Another
method is Instructional Tuning, which involves fine-tuning the LLM for specific tasks. By updating the
model’s parameters, we can optimize its performance for social manipulation detection. This might include
16

tuning the model to better interpret the nuances in account descriptions and structural relationships within
networks. These techniques allow us to harness the power of LLMs to enhance our detection capabilities.
By converting metadata, text, and network structure into a format that these models can understand and
learn from, we are setting new benchmarks in our ability to identify and mitigate the impact of social
manipulators.
2.3.2 Challenges in Detecting LLM based Manipulator
While large language models present significant opportunities to detect manipulators, they also bring serious
challenges. Unfortunately, those seeking to deceive can also employ the same technologies we use to identify
social manipulators. Firstly, manipulators can utilize LLMs to craft and refine text, making automated
accounts appear more human-like [23]. This process applies zero-shot or few-shot learning methods to
generate text that can increase the difficulty distinguishing from that of a genuine human user. For example,
an LLM can iteratively refine a bot’s description based on feedback from a classifier, gradually improving
its score and susceptibility. Secondly, manipulators can use LLMs to strategically alter the structure of their
social networks to avoid detection [48]. As depicted on the right, this could involve adding or removing
connections to mimic genuine human social patterns more closely. A manipulative account can better blend
in by instructing an LLM to suggest optimal users to follow or unfollow, complicating the detection efforts.
By crafting appropriate prompts, manipulators can engage their social bots autonomously with other
accounts. Such interactive social bots are considerably harder to detect than their traditional counterparts,
which merely follow pre-defined scripts [49, 170, 4]. These challenges require us to continuously advance
our methodologies and develop more sophisticated tools to keep pace with manipulators’ evolving tactics.
They underscore the need for a dynamic approach to combating misinformation, ensuring our detection
systems are as adaptive and inventive as the manipulators themselves.
17

Figure 2.1: This figure describe a basic case of correlation vs causality. Due to the confounding factor
(political stance), we can not tell whether the the anti-vaccine attitudes is caused by the conspiracy or not.
2.4 Causal Inference on Social Media
2.4.1 Correlation vs Causality on Social Media
Recent researches about misinformation mainly focus on detecting fake news[134, 117, 128, 130], misinformation campaign detection [137, 182] and understanding how fake news attract user engagement[28, 29].
Some researcher attempts to study the relation between misinformation and people’s behaviors [72, 152,
141, 84]. However, most of them focus on mining the correlation between misinformation and people’s
behaviors rather than causal effects. The golden standard of measuring ground-truth causal effect is random
controlled trial (RCT) [145]. In the scope of social media, the most well-known RCT is AB-test [80]. Only
a limited amount of works, such as [152], try to understand the causal effect of RCT. However, they are
usually from psychology and mainly rely on carefully designed randomized controlled trials. Extending
such trials on large-scale social media platforms brings not only high costs but also potential ethical risks.
18

Figure 2.2: The common causal models describing misinformation spreading on social media.
2.4.2 Causal Analysis on Social Media
In contrast to correlation analysis, causal Inference aims at understanding the causal relationship between
a treatment (i.e., cause, such as smoking) and an outcome (i.e., result, such as lung cancer) [116]. Modern
Social Media platforms commonly apply personalized recommendation systems, which recommend content
to individual users based on their activity pattern (such as interested topics). This causes Information
Cocoons (Echo Chamber). As a result, the covariates (history activities) will be correlated to treatment
(engagement with misinformation), violating the principle of randomized controlled trials, leading to
imbalanced data distribution and causing bias in model training. This issue stopped us from understanding
the effect of misinformation from a causal perspective. A typical example is that an individual user from a
red state retweets more conspiracies and fake news [139]. However, this phenomenon can not tell us that
this user is more susceptible to misinformation because perhaps the recommendation system believes that
this guy is more interested in the posts from red state politicians, so they recommended more content from
some red politicians.
Causal Inference helps us understand how misinformation and manipulated contents causally affect
online users’ activities by mitigating the confounding effect of recommendation systems. One important
application of it in social media analysis is to learn unbiased representations of users [28]. By re-balancing
19

the data distribution among different treatments (i.e., exposure to specific information and narratives), we
can learn an unbiased representation that mitigates the impact of recommendation systems.
2.5 Misinformation Detection in Large Language Model Era
The rise of Large Language Models is a double-edged sword for combating misinformation. On one side,
LLM has significantly eased the creation of highly deceptive misinformation. For instance, they can simulate
the linguistic style of mainstream media to draft fake news and then generate falsified images or retrieve
authentic but out-of-context content to support specific narratives. On the other side, LLM’s advanced
capacity in tackling text as well as inputs in other modalities (e.g., images, graph, and tabular data [89, 58,
146]) benefits automatic fact verification.
2.5.1 Paradigm of Applying LLM in Misinformation Detection
Existing works mainly apply the following three types of paradigm to apply LLM in misinformation
detection.
Detection with Internal Knowledge The most straightforward approach to utilizing LLMs for misinformation detection involves well-crafted prompts that guide the model in analyzing the content in question.
Researchers typically attach a fixed prompt to the suspected misinformation and submit it to the LLM
for evaluation [21, 68]. This method leverages the LLM’s extensive linguistic capabilities and the internal
knowledge it has amassed during pre-training on large corpora.
While this paradigm offers a simple and direct method for misinformation detection, it has limitations.
Specifically, this approach tends to be effective only under certain conditions:
1. The misinformation can be identified through linguistic cues, such as emotional bias or internal
contradictions, rather than through factual inaccuracies.
20

2. The facts related to the misinformation have already been covered in the model’s pre-training or
fine-tuning corpus, such as well-known urban legends.
Detection with External Knowledge Due to the above limitations, researchers propose incorporating
external knowledge with LLM-based misinformation detection by borrowing the idea in retrieval augmented
generation [88]. The central idea of such retrieval-augmented (or search-augmented) detection paradigm
generally converts the content to be verified as a series of queries [2]. After that, the model calls an API to
acquire relevant documents from a database or Internet as an augmented context for verification.
LLM as a supporter Another innovative approach involves utilizing LLMs as supportive tools rather
than primary detectors. In this model, LLMs act as advisors that generate rationales for the outputs of
smaller language models [66]. This collaborative process allows researchers to fine-tune the smaller models
based on the rationales provided by the more powerful LLMs, enhancing the smaller model’s ability to
detect misinformation effectively.
2.5.2 LLM Prompting Strategies in Misinformation Detection
When deploying LLMs for misinformation detection, selecting an appropriate prompting strategy is crucial
for achieving optimal performance and accuracy. The most basic prompting strategy, known as IOprompting [173], simply asks the LLM to generate responses based on in-context samples. However,
research has indicated that this method may fall short in effectively identifying misinformation [23].
Advancements in prompting strategies have led to more sophisticated techniques. For instance, the
Chain-of-Thought method, also called step-by-step reasoning, has shown promise in improving the performance of LLMs [161]. Experimental results have demonstrated that this approach enhances the ability
of LLMs to detect misinformation. Furthermore, in a study by [179], researchers developed a hierarchical
21

step-by-step method that addresses misinformation at varying complexity levels, further improving LLM
performance.
Through these evolving paradigms and strategies, the potential for LLMs in misinformation detection continues to grow. They offer innovative solutions and pose new challenges that demand careful
consideration and ongoing research.
2.6 New Challenges and Opportunities in Foundation Model Era
Researchers have been persistently working on combating misinformation and its manipulation toward
social media [3, 51, 72, 133, 134, 152]. The proposed techniques have achieved success to a certain
extent when addressing traditional threatens, e.g. social bots incorporated with predefined contents
that human editors fabricate. However, the rapid advances in AI-generated content (AIGC) have opened
Pandora’s box. Incorporated with large language models (LLMs) and in-context learning, the manipulators of
misinformation campaigns are bringing more serious threats, including more impactful social manipulation
and more powerful fake news generation tools.
2.6.1 LLM-boosted Social Manipulation
By crafting appropriate prompts, manipulators can design social bots that engage with other accounts
autonomously, effectively mimicking human-like interactions. Such interactive social bots are considerably
harder to detect than traditional bots, which typically operate on pre-programmed scripts with limited
flexibility [49, 170, 4]. Particularly, these more advanced bots can autonomously generate responses that
are relevant to the dialogue context. This evolution in bot technology presents a serious challenge to social
media platforms and researchers in social manipulation detection.
One of the most straightforward methods manipulators use to enhance the capabilities of their bots is
by leveraging large language models (LLMs) to craft and refine text, making automated accounts appear
22

more human-like [49]. LLMs, such as ChatGPT models [93], are capable of generating highly coherent and
contextually appropriate text in response to various prompts, enabling social bots to engage in conversations,
post content, and reply to other users. A recent work [48] reported this probability, which involves using
zero-shot or few-shot learning methods to generate text that is increasingly difficult to distinguish from
that of a genuine human user. Specifically, the authors build up an LLM that can iteratively refine a bot’s
description based on feedback from a classifier, gradually improving its score and believability. A recent
study [35] highlighted how this technology is being applied in practice.
Additionally, manipulators can use LLMs to strategically edit the structure of their social networks to
avoid detection [48]. This could involve adding or removing connections to mimic genuine human social
patterns more closely. The authors [48] instructing an LLM to suggest optimal users to follow or unfollow,
a manipulative account can better blend in, complicating the detection efforts.
In addition to refining text, manipulators can also use LLMs to strategically adjust the structure of
their social networks to further avoid detection [48]. This could involve manipulating the relationships
between accounts to make the network of automated bots appear more organic by adding or removing
social connections, such as followers or friends. Specifically, the authors instruct an LLM to suggest optimal
users to follow or unfollow for a manipulative account so that the account can more closely resemble a real
user’s network. This is especially critical since many bot detection systems rely on identifying patterns in
social graphs, such as suspicious clustering of accounts or abnormal activity between certain users. By
adjusting these patterns, manipulators can effectively obscure the true nature of their accounts.
Moreover, LLMs can be used to optimize these manipulations by suggesting specific users for a bot to
follow or unfollow, thus enhancing the bot’s social footprint [48]. For example, if an automated account is
flagged for having an unnatural following pattern, an LLM might recommend that it follow accounts that
are part of a similar demographic or interest group, making its behavior appear more aligned with that of a
typical human user. By adopting such strategies, manipulative accounts can blend in more seamlessly with
23

legitimate users, complicating the detection efforts further. This level of sophistication creates a moving
target for researchers and platform moderators who are working to identify and mitigate the influence of
social bots.
The increasing capabilities of LLMs in both text generation and social network manipulation highlight
the ongoing arms race between those developing detection systems and the manipulators themselves.
As manipulators continuously refine their methods, detection systems must evolve in tandem [52]. The
challenges outlined by recent studies [48, 23] underscore the need for a dynamic approach to combating
misinformation and disinformation in online spaces. It is no longer sufficient to rely on static detection
tools that are only capable of identifying known bot behaviors. Instead, there is a pressing need for more
adaptive and inventive detection techniques, ones that can respond in real-time to new, evolving tactics [49,
132].
2.6.2 LLM-boosted Fake News Generation
In the recent past, deploying neural networks to automatically generate misinformation required a solid
understanding of programming and deep learning frameworks [177]. Improving the quality of the generated
content or tailoring it to specific topics and narratives was even more challenging, as fine-tuning neural
content generators demanded advanced expertise. However, with the development of interactive content
generation AI, misinformation creators can now easily exploit language models to generate and refine
misleading content with human-level language quality due to user-friendly interfaces and natural language
commands [40]. These manipulators can harness the extensive capabilities of LLMs to emulate various
linguistic styles, making their misinformation more convincing to specific demographics or the general
public. For example, the manipulators can imitate linguistic styles of mainstream media to improve their
deceptiveness and the willingness of users to spread [49, 10]. Furthermore, they can exploit LLMs’ tendency
to produce fabricated references and evidence, further amplifying the deception [40].
24

Besides, recent research has revealed that large multimodal models can used to efficiently generate
multimodal misinformation containing both vision (image or video) and language information. Typically,
three kinds of AI-generated multimodal misinformation are widely operated:
• AI-edited Multimodal Misinformation: This kind of misinformation is the earliest attempt to
target fake news generation, which edits the original genuine visual contents such as images and
video with AI models. A typical model is DeepFake [163], which replaces the original face in a
genuine image or video with another one’s.
• AI-retrieved Out-of-Context Misinformation: This kind of misinformation targets generating
out-of-context multimodal misinformation, i.e., the content of the vision modality is genuine, but
overall, the whole information is out-of-context. For instance, given a piece of misinformation
text, pre-trained multimodal models can easily retrieve authentic images (e.g., CLIP) to construct
out-of-context media (i.e., images and text are both unaltered but incongruent, thereby supporting
specific narratives) [114].
• AI-generated Multimodal Misinformation. This kind of misinformation targets generating fake
visual information from scratch. The emergent advance of large vision models makes this paradigm
feasible. For example, given a piece of text, state-of-the-art large vision models can generate fake
images (e.g., DALL-E) [101] or vivid videos (e.g., SORA) [186] based on text prompt. This flexible and
powerful tool enables social manipulators to generate misinformation about whatever topics that
they are interested in.
25

Chapter 3
Data Collection
3.1 Coordination Detection Dataset
3.1.1 U.S. 2016 Election
The investigation of the U.S. Congress reveals that the IRA tried to manipulate the U.S. 2016 Election by
operating coordinated accounts (also known as Trolls) to spread or influence the spread of specific narratives.
The investigation provided researchers with a list of coordinated account IDs. Based on this list, in [97], the
authors collect the activity traces of the coordinated accounts with a paid API∗
. In contrast, they also collect
the activities of some normal users whose active time and topics of interest are similar to the coordinated
accounts. Based on the data they collected, we constructed a dataset that contains 2025 accounts (312
coordinated accounts of IRA and 1713 normal users) and 66k election-related tweets in which they get
involved. The account ID and timestamp of the engagements (like posting, commenting, and thumbing up)
naturally form the activity traces in the order of time. The whole account set and 75% traces are applied to
train the AMDN-HAGE model. And 15% are held out for validation.
∗The dataset can only be provided by the authors in this paper
26

3.1.2 COVID-19 Pandemic Twitter
We use Twitter’s official streaming API to collect the tweets containing keywords related to COVID-19
from March 1 to July 22, 2020†
. The collected dataset contains 119k active accounts with at least twenty
appearances and 13.9M tweets. However, in this dataset, we do not have a ground-truth annotation of
coordinated accounts. Instead, we are only aware of 9k suspended accounts, 81 stat-backed accounts, 602
coupled accounts, and 18.5k accounts targeted by specific manipulation narratives.
3.1.3 COVID-19 Vaccine Twitter
We collect tweets related to COVID-19 Vaccines using Twitter’s public API, which provides a 1% random
sample of Tweets. The dataset‡
contains 62k activity sequences of 31k accounts after filtering accounts
collected less than 5 times in the collected tweets and sequences shorter than length 10. Although the data
of tweets about the COVID-19 Vaccine does not have ground-truth labels, we can apply VigDet to detect
suspicious groups and then analyze the collective behavior of the group.
3.2 Causal Analysis Dataset
3.2.1 Synthetic Dataset
We introduce how to generate the synthetic data§
in detail. We present the basic ideas in Figure 3.1. To
generate the data, we design a system to simulate social media activities and events in the real world. The
basic elements in this system are users and news.
• User: Each user i is represented with a hidden vector ui
, which represents the hidden status of a
social media user, such as interests, ideas, and political trends in reality. If a user engages with a piece
†Data available in https://github.com/ksharmar/AMDN-HAGE-KDD21.git
‡Data available in the supplementary materials in https://openreview.net/forum?id=sYNr-OqGC9m
§Data provided in https://github.com/yizhouzhang97/CNTPP.git
27

Jointly decide a
piece of news
to engage and
the timestamp
of engagement
Covariates: Historical
events of the user
Generate a random
number of new
orginial posts and
update the post
Sequence
Engage Engage
Hidden Vector:
Characeterzing the user’s
interest and ideas
⊕
Treatment: New
engagement event
with news content
and timestamp
Update the hidden
vector, simulating the
impact of news on
user’s mind
Updated Hidden Vector: the
user’s interest and ideas
changed by the engagement
A Neural Network simulating
the effect of recommendation
system on information filtering
Post Post Post
Post Post Post Post Post
Update the Engage Engage Engage engagement
sequence
Figure 3.1: The general idea of generating the synthetic dataset.
of news, then this user’s hidden vector will change accordingly. We randomly assign each user with
a randomly initialized vector to simulate the fact that users may hold different ideas and interests
before starting to use social media.
• News: Each piece of news n is characterized by two randomly generated hidden feature vectors: a
topic vector vtopic(n) and an inherent influence vector vin(n). These two vectors decide which users
will be attracted by the news and how they will be influenced by the news.
In this system, a user has two kinds of activities: (1) engaging with a piece of news and (2) posting a
post with original content. These activities are characterized by two temporal point processes respectively.
For the process of engaging with a piece of news, the intensity function of a user’s (i) engagement with a
specific piece of news n is defined as:
λe(n, t|ui
, S(e)
h
, S(g)
h
) = vtopic(n) · ui +
X
(fj ,tj )∈S
(e)
h
∪S
(g)
h
exp(tj − t)NNrec(fj , vtopic(n)) (3.1)
where S
(e)
h
and S
(g)
h
denote the historical sequences of a user’s engagement and posting activities, respectively. In this equation, the first term models the consistency of news topic and user interest, and the
NNrec(·, ·) in the second term is a random parameterized neural network simulating the recommendation
system that recommends contents based on history. exp(tj −t) makes sure that the influence of a historical
28

event decreases as time goes by, simulating the property that recommendation systems usually give greater
attention to recent events.
Similarly, the intensity function of a user’s (i) process generating original posts is defined as:
λg(t|ui
, S(e)
h
, S(g)
h
) = ui
· ui +
X
(fj ,tj )∈S
(e)
h
∪S
(g)
h
exp(tj − t)NNpost(fj , ui) (3.2)
where ui
· ui models the user’s reliance on social media, and the second term reflects the influence of
historical events on future events.
In the real world, the news topic, its influence on the user’s mind, and the user’s mental status are
not observable. Instead we can only observe the tweet content text features, such as the representation
extracted with a pre-trained language model. To simulate this fact, we use two neural networks NNnews
and NNuser with random parameters to project the vectors to observable features:
fnews(n) = NNnews(vtopic(n), vin(n)), fuser(i, t) = NNuser(ui
, t) (3.3)
where fnews(n) is the observable content feature of news n and fuser(i, t) is the observable content feature
of the post generated by user i at time t. fnews(n) does not change over time because once a piece of
news is reported, its content does not change significantly. Note that a normal user’s activity is usually
uncertain. To simulate such uncertainty, in each hidden layer of NNuser, we add a random noise. Since the
two networks are with random parameters, the projection will be highly non-linear and thus bring enough
complexity to the learning of causal impact.
The overall algorithm is shown in Algorithm 1.
29

Algorithm 1 Synthetic Data Generation
1: Set the ending time T. {The simulation will end at time T}
2: Randomly generated user hidden representations and news features and a posting time of each
news.{Each piece of news will only be engaged with after it is posted.}
3: for news n do
4: fnews(n) = NNnews(vtopic(n), vin(n))
5: end for
6: for user i do
7: tend = 0
8: S
(e)
h
, S(g)
h = ∅, ∅
9: repeat
10: Draw a posting event timestamp tg based on intensity function λg(t|ui
, S(e)
h
, S(g)
h
)
11: tcur = tg
12: fcur = fuser(i, tg)
13: flag =′
generating′
14: for news n do
15: if tp(n) ≥ tcur then
16: Continue.
17: end if
18: Draw an event timestamp te based on intensity function λe(n, t|ui
, S(e)
h
, S(g)
h
)
19: if te < tcur then
20: tcur = te
21: fcur = fnews(n)
22: flag =′
engagement′
23: ncur = n
24: end if
25: end for
26: tend = tcur
27: if flag is ′
engagement′
then
28: S
(e)
h = S
(e)
h ∪ (fcur, tcur)
29: ui = ui + NNscale(vtopic(ncur), ui)vin(ncur) {Update the user status. NNscale is a neural
network with random parameters to adjust news impact scale.}
30: else
31: S
(g)
h = S
(g)
h ∪ (fcur, tcur)
32: end if
33: until tend ≥ T
34: end for
30

3.3 Fact Verification Data
3.3.1 Hallucination Detection
In this thesis, we apply the HaluEval-Summary dataset, which is one of the three datasets in the HaluEval benchmark for hallucination detection, which contains the hallucination generated by ChatGPT-3.5.
HaluEval-Summary has the longest context and generated contents among all three tasks in this benchmark
[90]. Thus, detecting hallucination on this dataset requires repeatedly verifying each sentence in the
response, making standard prompting strategies acquire the worst accuracy across all three tasks.
3.3.2 Misinformation Detection
In this thesis, we apply a variant of the SciFact dataset [156]. In this dataset, each sample is a pair of news
and evidence, where the evidence is the abstract of a peer-reviewed paper and the news is a sentence
of a claim. To better simulate the real-world scenario where news on social media usually appears as a
paragraph of the post, following Chen and Shu, we generate a dataset of paragraph-level misinformation
based on the SciFact dataset. Specifically, for a given claim, we apply ChatGPT-4 to extend the claim as an
article based on the evidence.
31

Chapter 4
Data-Driven Detection of Coordinated Social Manipulation
Recent research has revealed the existence of coordinated accounts operated by misinformation campaigns,
such as bots (i.e., automatic accounts) and trolls (i.e., accounts with human operators)[97, 20, 59]. By
applying some special collaborative strategies, like interacting (e.g., thumbing up and commenting) with
each other frequently, these coordinated accounts can rapidly increase the visibility and influence of the
misinformation that they try to spread[135]. An early reported coordinated effort to manipulate public
opinions is from the Internet Research Agency (IRA), which operated coordinated accounts to spread
misinformation on multiple social media platforms to intervene in the 2016 U.S. presidential election[97].
In addition, recent research also points out that the ongoing COVID-19 epidemic is becoming an easy
target of misinformation campaigns as it increases people’s reliance on acquiring information from social
media[135]. To address the above challenges, computational coordination detection methodologies are
urgently required.
Existing researchers have developed methodologies from the perspectives of both individual features and
collective behaviours[20, 59, 97, 135]. However, those methods based on individual features either require
partially annotated data to learn the coordinated cues or make assumptions on the automated behaviors of
the coordinated accounts, which are unable to help recognize accounts with human operators. Meanwhile,
those methods based on collective behaviors usually applied hand-crafted coordination signatures, such as
32

time synchronization and co-activity[59], which have very limited expressive power and heavily rely on
the quality of human prior knowledge. As a result, the algorithm will fail in those coordination cases in
which the hypothesis in the human prior knowledge may not hold true. Figure 4.1 presents an example
of coordinated accounts officially suspended by Twitter (also detected as coordinated accounts by our
proposed method). They shared the same content. However, the time differences in their activity range
from less than 6 hours to more than half a week. Such a time variance can make the time synchronization
hypothesis invalid.
ーー
‼
⁉ ⁉ ⁉
‼
⁉ ⁉ ⁉
Figure 4.1: Coordinated accounts suspended by Twitter officially (also detected by our proposed method).
Example tweets in the figure spread political conspiracies collaboratively. The time differences between the
tweets with same contents vary from less than 6 hours to more than half a week.
To address the above challenges, we propose that the following general properties of coordinated
activities should be modeled:
• Strong correlation: if two accounts collaborate to spread information on social media, then their
activities should be strongly correlated, in other words, influenced by each other. For example, the
automated bots for computational propaganda on social media are usually set to retweet each other
[166].
33

• Highly concerted activities: to spread misinformation more rapidly than normal users, the coordinated
accounts collectively behave anomalously, compared to normal users which have more random and
less organized activities [61].
To model the above properties, we propose AMDN-HAGE (Attentive Mixture Density Network with
Hidden Account Group Estimation), which can learn account embeddings by jointly modeling the account
activities and group behaviors via Neural Temporal Point Process (NTPP) [42, 104, 178, 193, 143, 111]
and Gaussian Mixture Model (GMM). To capture the correlation within account activities, for a piece
of information (e.g., a tweet), we model the distribution of future events (the account ID that interacts
with the tweet and the timestamp when the interaction happens) conditioned on all the past activities
from different accounts. To jointly learn the collective behaviors, we regularize the account embeddings
with a GMM model so that the embeddings are encouraged to form clusters in the hidden space. Since
both modules can be trained by optimizing the likelihood of the observed data, AMDN-HAGE can learn
account representations and their group memberships in an unsupervised manner. Therefore, unlike
previous works, it gets rid of the reliance on annotated data (we can still further boost the model with
ground-truth labels if provided). Meanwhile, our methodology is a data-driven approach based on only
two assumptions. Therefore, compared to those existing methods relying on hand-crafted signatures and
human prior knowledge, our model has stronger expressive power and can acquire knowledge from the
data by itself.
However, a problem arises when combing NTPP and GMM together: Stochastic Gradient Descent
(SGD), which is the most widely applied optimization algorithm family [77, 109] for neural networks, may
result in invalid parameters when being used to train GMM. In practice, people usually use the ExpectationMaximization algorithm to train GMM [142]. To address this challenge and jointly optimize the parameters
in NTPP and GMM, we designed a bilevel optimization algorithm that applies both SGD and EM and has a
theoretic guarantee of convergence.
34

In general, our contributions are:
• We propose AMDN-HAGE, an unsupervised model that can detect coordinated campaigns from
account activities based on NTTP and GMM.
• We design a bi-level optimization algorithm for joint optimizing NTPP and GMM with theoretical
and empirical guarantee on convergence and parameter validation.
• Experimental results on real-world Twitter data with ground-truth coordinated campaign verify the
effectiveness of our model. We further apply the method to identify coordinated campaigns in a
COVID-19 dataset without ground-truth label, and find suspicious coordinated efforts on spreading
contents in different languages (NoMask, NoVaccine, NoALaVacuna, NoAlNuevoOrdenMundia,
QAnon) about no masks, no vaccine, no new world order conspiracies, opposing Bill Gates, that
suggest COVID-19 is a hoax and political scam.
4.1 Preliminary
4.1.1 Task Definition
In this section, we formulate the task of coordination detection on social media platforms. Coordinated
accounts, usually operated by misinformation campaigns, cooperate to increase the visibility of specific
information and thus manipulate public opinions. In coordination detection, we aim at finding such
coordinated accounts by modeling the latent account groups in the activity traces and extracting the
groups with anomalistic concerted activities.
Activity Traces To guarantee that our work can be generalized to different languages and social media
platforms, in this work, we only consider the activity traces of accounts, which are available on most of the
post-based social media platforms such as Twitter, Facebook, and Reddit. Other data such as text contents,
35

the types of action (“thumb up" or “like"), and account metadata are not considered in this work, although
they can be incorporated into our framework. An activity trace is defined as a temporal sequence of events:
S = [(u1, t1),(u2, t2),(u3, t3), · · ·(un, tn)] (4.1)
where each tuple (ui
, ti) is an event that account ui have an activity at time ti
. The activity represents
account actions on the platform such as posting original content, re-sharing, replying, or reacting to other
posts.
Hidden Groups The accounts with similar activities and interests form groups, including both naturally
formed communities and anomalous coordinated groups. Note that such groups do not necessarily have
visible tags. Thus, the membership of accounts is unknown in many cases [11]. So, we call such groups as
hidden groups. In this work, to detect the coordinated groups, we propose to first model all hidden groups
and then extract coordinated groups based on the following two assumptions:
• The size of a coordinated group is usually smaller than a naturally formed community.
• The activities of accounts within a coordinated group are very concerning.
To formulate the task of modeling hidden groups from activity traces, we suppose that there are N groups
in the account set U. Then, we can define a membership function M : U → {1, . . . , N} that projects each
account to its corresponding group. Our objective is to learn this membership function from the activity
trace data.
36

4.1.2 Marked Temporal Point Process
A marked temporal point process (MTPP) is a stochastic process that generates a time series of discrete
events happening in continuous time:
S = [(v1, t1),(v2, t2),(v3, t3), · · ·(vn, tn)] (4.2)
where vi ∈ V is the mark of event i and ti ∈ R
+ is the timestamp [37]. Denoting the set of all events before
time t as Ht = {(vi
, ti)|ti < t}, the conditional distribution of the happening time t of the next event with
mark v ∈ V is formulated as:
pv(t|Ht) = λv(t|Ht) exp
−
Z t
ti−1
λv(s|Ht)ds!
(4.3)
where λv(t|Ht), also known as intensity function, describes the contribution from the history Ht to the
future. Different ways to model the intensity function lead to different temporal point processes. In the
social media analysis area, the most commonly used temporal point process is Hawkes Process (HP) [185],
where the intensity function is defined as:
λv(t|Ht) = αv +
X
(vi,ti)∈Ht
αv,viκ(t − ti) (4.4)
where αv > 0 is the self activating term, αv,vi > 0 is the mutual term modeling the influence of mark vi
on v and κ is a decay kernel to model influence decay over time. However, the above formulation has few
learnable parameters, leading to poor expressive power. As a result, recent works make efforts on modeling
the λv(t|Ht) or pv(t|Ht) with deep neural networks [42, 104, 178, 193, 143, 111].
37

Conditional
density function
p(|history)
Time
t
Event history on the network
Emb.
MH-Attn MH-Attn MH-Attn MH-Attn
Emb. Emb. Emb.
Position and time
embedding
Masked
Self-Attention
h1 h2 h3 h4
Context
Activity
Trace
Modeling
Normal Community
Anomalous group
(Coordinated)
Social
Group
Modeling
Jointly
Learning
P(Cs|U,θa,E)
P(U|θg,E)
Figure 4.2: Architecture of proposed (AMDN-HAGE) to model conditional density function of account
activities and hidden groups on social media.
4.2 AMDN-HAGE: Attentive Mixture Density Networks with Hidden
Account Group Estimation
In order to make use of the strong expressive power of the deep neural networks to model the hidden
correlation between account activities as well as capture the group behaviors, we propose AMDN-HAGE. It
consists of two components:
• Attentive Mixture Density Networks (AMDN): a Neural Temporal Point Process model (NTPP)
that explicitly models the account mutual influence with masked attention.
• Hidden Account Group Estimation (HAGE): a Gaussian Mixture Model (GMM) that is jointly
learned with the AMDN part and can model the group of accounts with similar patterns in the latent
embedding space.
An overview is provided in Figure 4.2. In general, the two components are bridged by sharing the
account embeddings and modeling the process of how the activity traces are observed: the accounts are
first drawn from the hidden groups and then interact mutually. To learn the account representations as well
38

as the membership function and activity trace model, we optimize the parameters to maximize the joint
likelihood of the observed traces and account set. Denoting the account embeddings as E, the parameters
in AMDN as θa and the parameters in HAGE as θg, the joint likelihood function can be written as:
log p(S, U; θg, θa, E) = log p(S|U; θa, E) + log p(U; θg, E)
(4.5)
where p(S|U; θa, E) is the conditional probability density that the activity traces are observed given a
known account set, and p(U; θg, E) is the conditional probability density that we observe the account set
drawn from the hidden social groups given their embeddings and the parameters of the social groups in the
latent space. After acquiring the account representations as well as the parameters of the social groups,
we can calculate the likelihood that an account is drawn from a cluster and detect coordinated groups by
identifying clusters with anomalously small sizes or small variances. Next, we are going into the details of
the two components and the optimization algorithm.
4.2.1 Attentive Mixture Density Networks: Modeling Activity Traces
In this section, we illustrate how to model the activity traces with AMDN architecture. This architecture
contains two main components: the history encoder and the future event decoder. The history encoder
first represents the historical sequence Hti
as a contextual vector Cti
. Then, the future event decoder takes
the contextual vector as input and predicts the happening time and account id of the next event (ui
, ti)
respectively. Under this framework, we can write the likelihood of an observed sequence S as:
log p(S|U; θa, E) = X
|S|
i=2
[log p(ui
|Hti
; θa, E) + log p(ti
|Hti
; θa, E)] (4.6)
Previous researchers have proposed a lot of neural temporal point process models, all of which can
model the above likelihood function. However, for the purpose of coordination detection, we hope our
39

encoder and decoders can satisfy some requirements. Our encoder is expected to have the following
properties:
• Capturing Long-term Influence: As shown in Figure 4.1, the time difference of coordinated activities
may vary from less than 6 hours to more than half a week. Consequently, our encoder must be able
to capture the hidden influence over a long time.
• Explicit Influence Scores: To better understand the behaviors and strategies of the collaborated
accounts, we hope that the detection results are not only accurate but also interpretable. Thus, an
architecture that explicitly models the influence scores among accounts is helpful.
Meanwhile, we also hope the decoder will satisfy the following requirements:
• Universal-Approximation Expressive Power: As the behavior of online accounts, including both
common users and coordinated accounts, could be very complicated, to model their activities, the
p(S|U; θa, E) output by the decoder must be able to approximate a random valid distribution with
enough precision as long as the parameter size is enough, i.e. with universal-approximation expressive
power.
• Closed-form Likelihood: Some decoders designed by previous works lead to a likelihood without a
closed-form solution. As a result, they have to apply some complicated training strategies such as
Monte Carlo sampling. In this work, we hope to avoid this issue and directly optimize a closed-form
and differential likelihood function.
Unfortunately, to the best of our knowledge, existing works can not satisfy all the above requirements
[42, 104, 178, 193, 143, 111]. Therefore, in this work, we propose AMDN (Attentive Mixture Density
Networks), a novel architecture that can satisfy all the above requirements. In the following sections, we
will illustrate the design of the model in detail.
40

4.2.1.1 Encoder: Masked Attentive History Encoder
Masked self-attention with position encoding. To explicitly model the influence scores of past events
on future events, we apply masked self-attention [153] to encode the history:
Hattn = σ(
QKT
√
d
)V
Q = XWq, K = XWk, V = XWv
(4.7)
where Hattn is the history representation vector, X ∈ R
L×d
is the input sequence encoding (L is the
sequence length, d is the feature dimension), and Wq, Wk, Wv are the learnable weights. After acquiring
Hattn, we apply layer normalization, dropout, and feed-forward layer (or an RNN layer) to Hattn to get
output Hout ∈ R
L×d
. The i-th row of the Hout[i − 1 :] will be used as the context vector Cti
of the time ti
.
Input Sequence Encoding In the above architecture, we encode an sequence containing L events as an
input matrix X ∈ R
L×d
, where each row Xi
is a d-dimensional vector representing an event. To provide
the attention mechanism with sufficient information from the input sequence, we have to make sure that
each Xi contains:
• Account ID in event i.
• Timestamp of event i.
• Positional Information (Unlike RNN, the attention mechanism does not preserve the order of the
events.)
As mentioned previously, AMDN has an account embedding layer E to learn account representations,
which project each account ui to a row vector Eui
. Therefore, the account ID information can be easily
41

included. As for the positional information, we use the sine-cosine functions for each dimension j of
m-dimensional position encoding, proposed in [153].
P Epos,2j = sin(pos/100002j/m),
P Epos,2j+1 = cos(pos/100002i/m)
(4.8)
However, the timestamp ti ∈ R+, unlike the position and account id, is a continuous variable. To
encode it, following [168] we encode the temporal information through basis ϕ, a function containing a set
of translation-invariant temporal kernel functions that have different frequencies w:
ϕω(t) = [√
c1, · · · √c2j cos(jπt/ω),
√c2j+1 sin(jπt/ω)· · · ] (4.9)
ϕ(t) = [ϕω1
(t), ϕω2
(t)· · · ϕωk
(t)]T
(4.10)
Then, by concatenating the account embedding, positional encoding, and temporal encoding, we
represent the i-th event (ui
, ti) in a sequence as:
Xi = [Eui
, P Epos=i
, ϕ(ti − ti−1)] (4.11)
4.2.1.2 Decoder: Conditional Probability Density Function
To calculate and optimize the likelihood, we need a decoder to model the probability density p(ui
|Hti
; θa, E)
and p(ti
|Hti
; θa, E) conditioned on the context vector. The first one can be easily modeled by forwarding
the context vector into a multi-layer perceptron (MLP) followed by a softmax layer because ui
is a discrete
variable. However, it is more challenging to design a decoder for p(ti
|Hti
; θa, E) because ti
is a continuous
variable. Therefore, we have to guarantee that the output of the decoder is non-negative and it integrates
(instead of summing) to 1 overall t ∈ R+. Meanwhile, we also hope the decoder has universal-approximation
42

expressive power and closed-form likelihood. To this end, following [143], we model the PDF with a mixture
of log-normal distributions:
p(ti
|wi
, µi
, si) = X
K
k=1
w
k
i
1
s
k
i
√
2π
exp
−
(log ti − µ
k
i
)
2
2(s
k
i
)
2

(4.12)
wi = σ(Vwci + bw), si = exp(Vsci + bs), µi = Vµci + bµ (4.13)
where the mixture weights wi
, means µi and scale si are all parameterized by the context vector ci and
learnable parameters. Obviously, the above probability density function has a closed-form solution. And
previous theoretic works [38] guarantee its universal-approximation expressive power:
Theorem 4.2.1. (Dasgupta, 2008, Theorem 33.2 [38]) Let q(x) be a continuous probability density where
x ∈ R. For any continuous probability density function p(x) with x ∈ R and any ϵ > 0, there exists a number
of components K ∈ N, mixture weight vector w ∈ R
K satisfying PK
i=1 wi = 1, mean vector µ ∈ R
K and scale
vector s ∈ R
K such that for the mixture distribution pˆ(x) = PK
i=1 wiq(
x−µi
si
), we have |p(x) − pˆ(x)| < ϵ
If we apply the log-normal Gaussian distribution with zero mean and identical scale as q(x), then every
Gaussian component in the mixture with parameter µi and si corresponds to a q(
x−µi
si
). Thus, we can
easily get the following corollary:
Corollary 4.2.2. For any p(t|Hti
) and any error bound ϵ > 0, there exists a K ∈ N, mixture weight vector
w ∈ R
K satisfying PK
i=1 wi = 1, mean vector µ ∈ R
K and scale vector s ∈ R
K such that:
|p(t|Hti
) −
X
K
k=1
w
k
i
1
s
k
i
√
2π
exp
−
(log ti − µ
k
i
)
2
2(s
k
i
)
2

| < ϵ (4.14)
With the above probability density function with closed-form solutions, we can easily calculate the
likelihood of each observed activity trace. And we can optimize the encoder-decoder parameters (jointly
43

denoted as θa) and the learnable account embedding layer (denoted as E) by maximizing the likelihood
log p(S|U; θa, E) through SGD.
4.2.2 Hidden Account Group Estimation: Modeling Account Groups
We model the hidden account groups as a mixture of multivariate Gaussian distributions (GMM, Gaussian
Mixture Model) in the account embedding space. The i-th GMM component N (µi
, Σi), where µi
is the
cluster mean and the σi
is the covariance matrix, corresponds to a social group. Denoting the prior
probability that an account is from the i-th group as p(i), the probability density that an account is drawn
from the GMM distribution is defined as P
i
p(i)N (µi
, Σi). Then, the likelihood that all accounts in the
account set U are drawn from the GMM distribution is defined as:
X
|U|
j=1
log p(uj ; θg, E) = X
|U|
j=1
logX
N
i=1
p(uj , i; θg, E)
=
X
|U|
j=1
logX
N
i=1
p(i)N (Euj
; µi
, Σi)
(4.15)
In general, we can unsupervisedly learn the membership by maximizing this likelihood and select the
component maximizing N (Euj
; µi
, Σ). However, unlike general GMM, which learns mixture parameters
on a set of observed samples with fixed features, in our model, both the account embedding and the GMM
parameters are learnable. Moreover, the account embedding is also involved in the likelihood maximization
of the AMDN part. As a result, a joint learning algorithm can simultaneously optimize the two likelihood
functions that are connected through the account embedding layer.
4.2.3 Joint Learning
44

Algorithm 2 Training Algorithm for AMDN-HAGE
Require: Activity traces (S), Account set (U)
Ensure: Generative model (θa, θg and E)
1: θ
(0)
a , E(0) ← argmaxθa,E log p(S|U; θa, E)
2: Set i as 1 {Iteration index}.
3: while not converged do
4: θ
(i)
g ← argmaxθg
log p(U; E(i−1), θg) using EM algorithm
5: θ
(i)
a , E(i) ← argmaxθa,E log p(S, U; θ
(i)
g , θa, E) using SGD or its variants
6: i ← i + 1.
7: end while
For joint learning, we maximize the following joint likelihood:
log p(S, U; θg, θa, E) = log p(S|U; θa, E) + log p(U; θg, E)
(4.16)
The above optimization problem has a trivial infinite asymptotic solution where all embedding vectors
are equal to the mean of a cluster and det(Σ) = 0. To avoid this solution, we constraint det(Σ) to be
greater than a small constant λ
n by adding a λI to Σ (n is the size of the matrix)
The two terms are connected through the embedding layer E. A straightforward solution to jointly
optimize the two terms is to apply gradient descent or its variants because the likelihood is differential.
However, the second term requires some constraints like normalized non-negative mixture weights and
positive definite covariance matrix. As a result, the family of gradient descent algorithms that do not respect
the constraint will lead to invalid parameters, breaking the training process (detailed experiments can be
found in the next experiment section). For this reason, researchers usually apply the EM algorithm, which
naturally maintains the validity of the parameters in each EM iteration, to estimate the parameters.
To address the above challenges, we design a learning algorithm based on bilevel optimization. First,
we rewrite the objective function as an equivalent bilevel optimization formulation:
θ
∗
a
, E∗ = argmaxθa,E[log p(S|U; θa, E) + maxθg
(log p(U; θg, E))] (4.17)
45

θ
∗
g = argmaxθg
log p(U; θg, E∗
) (4.18)
Then, we can solve this optimization problem with iterative optimization. In each iteration, we first fix
the AMDN parameters θa and the embedding layer E. Then, we apply the EM algorithm to optimize the
GMM parameters θg until the second term convergence. After that, we fix θg and optimize the E and θa
with gradient-descent-family algorithm to maximize the joint likelihood. To provide an initialization of the
embedding E better than random initialization, we pre-train the embedding and θa by only maximizing
the first term. The pseudo-code is provided in Algorithm. 3.
An intuitive concern on this iterative optimization algorithm is whether it will converge to a solution or
it will shock around some points. Therefore, we provide a theoretic guarantee that an appropriate selection
of the gradient descent based optimizer can make the algorithm converge at a local minimum or saddle
point.
Theorem 4.2.3. Our proposed optimizing algorithm will converge at a local minimum or a saddle point if in
any iteration i the neural network optimizer satisfies following conditions:
• Given the frozen θ
(i)
g acquired by EM algorithm in iteration i, the neural network optimization algorithm
we applied in converges at a local minimum or or a saddle point (θ
(i)
a , E(i)
)
• L(θ
(i)
a , E(i)
, θ(i)
g ) ≤ L(θ
(i−1)
a , E(i−1), θ(i)
g ), where θ
(i−1)
a and E(i−1) are the starting points in iteration
i
Proof. We first prove that the training algorithm converges. In EM, each step increases the likelihood
function of a mixture model. We have:
log P(E
(i−1)|θ
(i)
g
) ≥ log P(E
(i−1)|θ
(i−1)
g
) (4.19)
46

Thus, we obtain
L(θ
(i−1)
a
, E(i−1), θ(i)
g
) ≤ L(θ
(i−1)
a
, E(i−1), θ(i−1)
g
). (4.20)
Then, from the second condition, we know that:
L(θ
(i)
a
, E(i)
, θ(i)
g
) ≤ L(θ
(i−1)
a
, E(i−1), θ(i)
g
) (4.21)
Therefore, we have:
L(θ
(i)
a
, E(i)
, θ(i)
g
) ≤ L(θ
(i−1)
a
, E(i−1), θ(i−1)
g
) (4.22)
which means the loss function monotonically decreases. Since we constraint the variance of the mixture
model in both point processing model and social group model larger than a constant ϵ, the loss function is
bounded by a constant C on a given activity trace set:
L(θ
(i)
a
, E(i)
, θ(i)
g
) ≥ C (4.23)
Thus the loss function converges when i increases. Then we prove that the loss function converges at a
local minimum or a saddle point. First when the parameters converges, we have:
L(θ
(i)
a
, E(i)
, θ(i)
g
) = L(θ
(i+1)
a
, E(i+1), θ(i+1)
g
). (4.24)
Thus:
L(θ
(i+1)
a
, E(i+1), θ(i)
g
) = L(θ
(i+1)
a
, E(i+1), θ(i+1)
g
) (4.25)
log P(E
(i)
|θ
(i+1)
g
) = log P(E
(i)
|θ
(i)
g
). (4.26)
47

In EM, if log P(E(i)
|θ
(i+1)
g ) = log P(E(i)
|θ
(i)
g ) then θ
(i)
g = θ
(i+1)
g . Since (θ
(i)
a , E(i)
) is a local minimum or
a saddle point, we have:
∂L(θ
(i)
a , E(i)
, θ(i)
g )
∂θ(i)
a
=
∂L(θ
(i)
a , E(i)
, θ(i)
g )
∂E(i)
= 0 (4.27)
Since EM is known to converge to a local minimum, we have:
∂ log P(E(i)
|θ
(i+1)
g )
∂θ(i+1)
g
=
∂ log P(E(i)
|θ
(i)
g )
∂θ(i)
g
= 0 (4.28)
Therefore, (θ
(i)
a , E(i)
, θ(i)
g ) is a local minimum or a saddle point.
Theoretically, when our loss function is L-smooth we can guarantee the two conditions by applying
standard Gradient Descent Algorithm with learning rate lower than 1
L
[109]. But we can also alternatively
apply Adam [77] when finding strict local minimum is not as important as training speed and generalization.
4.3 Detecting Coordinated Accounts in U.S. 2016 Election
To indicate the effectiveness of our proposed method of detecting coordinated accounts on social media, we
evaluate AMDN-HAGE on a labeled dataset of the coordinated accounts belonging to the Internet Research
Agency (IRA) on Twitter.
4.3.1 Implementation Details
We maintain the length of each activity no longer than 128. For longer sequences, we split them into multiple
short sequences. We apply a batch size of 256. The models are trained on 4 NVIDIA-2080Ti with Adam [77].
We decide the hyper-parameters based on the likelihood of the validation set. The hyper-parameters include
embedding dimension (32, 64), number of the mixture components of the probability density function in the
AMDN part (8,16,32), and learning rate (1e-2,1e-3,1e-4). For the cluster number of the HAGE part, we apply
48

2 4 6 8 10 12 14 16 18
Number of Clusters
0.05
0.00
0.05
0.10
0.15
Silhouette Score (IRA)
0.48
0.50
0.52
0.54
Silhouette (COVID-19)
Figure 4.3: The silhouette score with different cluster numbers on IRA and COVID-19 datasets.
2 for the IRA dataset and 3 for the COVID-19 dataset based on the silhouette scores on the pre-trained
embedding (see the first line in Algorithm 3, the pseudo-code of the training algorithm). The curves of the
silhouette scores are presented in Figure 4.3. For the co-variance matrix of HAGE, we set it as a shared
diagonal matrix. We set the max epoch number as 1000 and apply early stopping based on the likelihood of
the validation set.
4.3.2 Evaluation Metrics, Baselines and Model Variants
We compare our model with existing methods that detect coordinated accounts based on account activities.
The baselines that we applied can all extract or learn account features (graphs or vectors) from their
activities in an unsupervised manner. Then, we can forward the features into a supervised classifier (like a
neural network) or an unsupervised detector (like K-Means or another clustering model). For unsupervised
detection, we consider the more concerted cluster to be the coordinated group. We apply the following two
kinds of metrics:
49

1. Threshold-free metrics: Average Precision (AP), Area under ROC curve (AUC), and F1 score at the
threshold that maximizes it (MaxF1).
2. Fixed-threshold metrics: F1 score, Precision, Recall and Macro F1. For this kind of metric, we apply
0.5 as the threshold.
Our baselines include:
1. Co-activity clustering [126]. Co-activity, proposed in [126], measures the correlation of two accounts
based on the frequency with which they interact or share the same information. To represent the
accounts as feature vectors by applying SVD to decompose a binary event-participation matrix (row
for accounts and column for its posts, i.e., tweets, retweets, replies). In this way, correlated accounts
will have similar representations. We then conduct clustering in the feature space for coordination
detection
2. Clickstream clustering [158, 113]. Clickstream was first proposed in [158] to analyze account behaviors
and measure account behavior similarity based on activity patterns. It was then applied in [113] to
identify coordinated groups by clustering the accounts with similar activity patterns.
3. Inverse Reinforcement Learning (IRL and IRL-S) [97]. This kind of approach is based on the assumption
that the activities of coordinated accounts are driven by motivations that are different from those of
normal users. Therefore, it applies Inverse Reinforcement Learning to estimate the rewards that drive
the activity of accounts on social media. The estimated rewards are used as features for detection.
IRL-S learns an AdaBoost model, taking these features as input on a labeled data subset. ∗ We
also include an unsupervised variant of the IRL approach, which conducts clustering with GMM or
K-means to detect coordinated groups.
∗
[97] reported different classifiers in their paper. Among them, AdaBoost performs the best.
50

4. HP and HP-S. Hawkes Process is a classical non-neural point process model. It maintains an accountwise influence strength matrix α, where αij represents the influence from account i on account j.
To learn the account representations with the Hawkes Process, we first learn the influence matrix
and then decompose it with SVD. Then, we can apply the learned embedding in both clustering and
supervised detection. The main motivation for including this baseline is to show that latent influence
and interaction patterns among coordinated accounts are so complex that the neural point process is
necessary for extracting these coordination features.
The variants of our model include:
1. AMDN(GMM), AMDN(KM) and AMDN(NN): In this variant, we remove the hidden account group
estimation module and only learn the account embedding by maximizing the log p(S|U; θa, E). Then
for unsupervised setting, we apply GMM and K-Means to conduct clustering on the learnt embedding
respectively. And for supervised setting, we train a two-layer neural network on the embedding for
detection.
2. AMDN-HAGE: The proposed model. We directly apply the membership learnt by the hidden account
group estimation for detection.
3. AMDN-HAGE(KM) and AMDN-HAGE(NN): In this variant, we jointly learn the embedding and
membership. But after that, for unsupervised setting, we apply K-Means to conduct clustering on
the learnt embedding respectively. And for supervised setting, we train a two-layer neural network
on the embedding for detection.
51

4.3.3 Results of Coordination Detection
4.3.3.1 Comparison with Baselines
The main results are reported in Table 4.1. We report the results of the models averaged on 5 random seeds
and the standard deviation. As we can see, AMDN-HAGE, due to its ability to capture interactions between
the coordinated accounts and jointly learn group membership, outperforms all other baselines. In addition,
we can also observe the following phenomenons:
1. Moreover, compared to IRL, all temporal point process-based models (including HP, our model, and
its variants) perform significantly better. We suggest that this is because they are modeling the
inherent feature of the coordinated accounts, i.e., the strong interaction between them.
2. Meanwhile, our model and its variant perform better than HP, indicating the importance of capturing
complicated interactions for coordination detection and the drawbacks of simply modeling the
interaction between two accounts as a scalar.
4.3.3.2 Ablation Study
In this section, we mainly conduct ablation studies on two modules: the hidden account group estimation
module and our bi-level optimization algorithm.
To verify the importance of the joint learning of both activity traces and group membership, we
compare AMDN-HAGE against AMDN, where the hidden account group estimation module is removed,
and the objective function is only to maximize the likelihood of the activity traces. As we can see, for
every backend detector, AMDN-HAGE performs better on most of the metrics. We suggest that this is
because the HAGE regularizes the model to focus more on the patterns shared by a group of accounts, i.e.,
collective behaviors. Consequently, AMDN-HAGE can learn account embeddings that are more suitable for
coordination detection.
52

To verify the effectiveness of the proposed bi-level algorithm, we present its validation loss curve and
compare it with directly optimizing the joint loss function with Adam [77]. The curve of Adam is shown in
Figure 4.4a. As shown in the figure, at the first several epochs, the validation loss decreases fast. However,
at epoch 10, the optimization leads to invalid parameters and causes NaN validation loss. In contrary, as
shown in Figure 4.4b, our optimization algorithm performs steadily in both pre-training stage and joint
training stage.†
Table 4.1: Results on detection of coordinated accounts (IRA dataset) on Twitter in 2016 U.S. Election
Method (U) AP AUC F1 Prec Rec MaxF1 MacroF1
Co-activity 0.208 ± 0.01 0.592 ± 0.03 0.292 ± 0.02 0.206 ± 0.02 0.510 ± 0.04 0.331 ± 0.03 0.515 ± 0.02
Clickstream 0.169 ± 0.02 0.535 ± 0.04 0.215 ± 0.06 0.205 ± 0.05 0.228 ± 0.08 0.215 ± 0.06 0.532 ± 0.03
IRL 0.200 ± 0.00 0.610 ± 0.02 0.265 ± 0.02 0.219 ± 0.02 0.336 ± 0.03 0.340 ± 0.02 0.543 ± 0.01
HP 0.337 ± 0.04 0.694 ± 0.05 0.376 ± 0.05 0.387 ± 0.06 0.365 ± 0.05 0.545 ± 0.03 0.633 ± 0.03
AMDN(GMM) 0.787 ± 0.05 0.894 ± 0.03 0.631 ± 0.06 0.965 ± 0.03 0.472 ± 0.07 0.738 ± 0.05 0.792 ± 0.03
AMDN(KM) 0.731 ± 0.08 0.901 ± 0.02 0.727 ± 0.06 0.806 ± 0.07 0.663 ± 0.06 0.752 ± 0.05 0.841 ± 0.03
A-H 0.804 ± 0.03 0.898 ± 0.02 0.699 ± 0.05 0.941 ± 0.04 0.558 ± 0.06 0.758 ± 0.04 0.828 ± 0.03
A-H(KM) 0.818 ± 0.04 0.935 ± 0.02 0.731 ± 0.04 0.913 ± 0.03 0.611 ± 0.05 0.776 ± 0.03 0.846 ± 0.02
Method (S) AP AUC F1 Prec Rec MaxF1 MacroF1
IRL (S) 0.672 ± 0.08 0.896 ± 0.03 0.557 ± 0.06 0.781 ± 0.06 0.436 ± 0.06 0.633 ± 0.07 0.749 ± 0.03
HP (S) 0.760 ± 0.04 0.925 ± 0.02 0.753 ± 0.02 0.743 ± 0.04 0.769 ± 0.06 0.782 ± 0.03 0.853 ± 0.01
AMDN(NN) 0.814 ± 0.04 0.918 ± 0.02 0.733 ± 0.04 0.710 ± 0.05 0.761 ± 0.05 0.763 ± 0.04 0.841 ± 0.02
A-H(NN) 0.838 ± 0.04 0.926 ± 0.03 0.769 ± 0.04 0.752 ± 0.05 0.789 ± 0.05 0.799 ± 0.04 0.862 ± 0.02
(a) Joint optimization with Adam finally leads to
NaN validation loss.
0 50 100
Training Epochs
6
7
8
Loss on Val. Set
Pre-Train Joint Train
AMDN loss for
Pre-Training
Total loss for
Joint Training
(b) The validation loss with our algorithm steadily
decrease in both pre-training and joint training stage.
Figure 4.4: Comparison between our proposed optimization algorithm and standard Adam.
53

39%
35%
13%
11%
(a) Averaged influence weights captured by
AMDN-HAGE. The averaged influence between
coordinated accounts is the highest.
(b) Visualization of the account embeddings learnt by
AMDN-HAGE. Red points are coordinated accounts
and green points are normal users
Figure 4.5: Analysis to the learnt influence and account embedding of AMDN-HAGE.
(a) Overall trend of the account-wise influence. (b) Trend of the strongly interacted account pairs.
Figure 4.6: Analysis to the learnt timely influence of AMDN-HAGE.
4.3.4 Analysis of Coordination Detection
In this section, we analyze the influence learned by the AMDN-HAGE model. We quantify the influence
strength with the attention weights in the encoder of AMDN-HAGE. If the attention weight on event (ui
, ti)
is high when calculating the context vector of event (uj , tj ), then we say that the influence on account uj
from account ui
is high in this tweet at time tj .
In Figure 4.5a, we present the average influence score of different kinds of account pairs. The NT
represents non-trolls (normal users), and the T represents trolls (coordinate accounts). As we can see, the
influence between coordinated accounts is the strongest, and the influence from coordinated accounts on
normal users is the weakest. This result justifies the usage of account influence in detecting coordinated
†The sudden increase at the blue line is because the pre-training ends and we add one more term into the loss.
54

accounts. Also, we visualize the account embedding in Figure 4.5b. As we can see, the coordinated accounts
form a much more concerted cluster compared to the normal users.
In Figure 4.6a and 4.6b, we present how the influence score of two accounts varies when the time gaps
between their appearance in the same tweet change. The blue points are coordinated accounts, and the
green points are normal users. In Figure 4.6a, we present all the account pairs in all tweets so that the
overall trend is shown. In Figure 4.6b, we only present the account pairs with the highest influence score
within each 24-hour time window to show the trend of the strongly interacted account pairs. From the
two figures, we can clearly see that the influence among coordinated accounts drops significantly faster
than normal users. This phenomenon suggests that compared to real social interaction of normal users, the
coordination on purpose is more temporal. Meanwhile, there are still a lot of coordinated account pairs
that maintain strong influence after 2000 hours (3 months). This phenomenon indicates that the methods
based on time synchronization may fail in such cases.
4.4 Uncovering Suspicious Coordinated Efforts on Social Media during
the COVID-19 Pandemic
The COVID-19 pandemic, the most serious global pandemic in the 21st century, is now targeted by a lot of
misinformation campaigns[135]. Therefore, in this section, we try to apply our proposed method to detect
the suspicious coordinated efforts of the COVID-19 pandemic.
4.4.1 Experiment Set-up
Different from the IRA dataset, we do not have the ground-truth label of coordinated groups on this dataset.
To examine our model in such a case, we first run our model on the dataset to detect suspicious coordinated
accounts. Then, we analyze their activities and the overlap between them and Twitter’s officially suspended
55

accounts (manually suspended by Twitter for violation of the platform policies). It is noticeable that the
overlap can not be regarded as a precise evaluation metric because:
1. Not all coordinated accounts are suspended by Twitter because the list is constructed by Twitter
manually. There could be coordinated accounts that have not been uncovered.
2. The reasons why an account is suspended may include violations other than coordination, such as
spam. Therefore, not all accounts suspended by Twitter are coordinated accounts.
4.4.2 Coordination Detection Results on COVID-19 Dataset
Our model detects two anomalous concerted groups other than the normal users (we decide the group
number based on silhouette score; see the Appendix). The sizes of the two groups are 3.7k and 5.5k,
respectively. The analysis of their overlap with suspended accounts is shown in Table 4.2
Table 4.2: Overlap between suspended Twitter accounts, and identified coordinated groups/ overall accounts
in collected COVID-19 data.
Twitter Overall Cluster 1 Cluster 2
Suspensions Overlap Anomaly (3.7k) Anomaly (5.5k)
Suspended (9k) 7.544 % 12.19 % 11.94 %
State-backed (81) 0.067 % 0.13 % 0.09 %
Coupled (602) 0.504 % 0.72 % 1.33 %
Targets (18.5k) 15.507 % 14.98 % 17.11 %
Among all suspended accounts, some of them are labeled by Twitter as “state-backed" accounts. In
addition, we consider the “coupled" accounts, i.e., the accounts bidirectionally interact with the state-backed
accounts, and “target" accounts, i.e., the account mentioned by the “state-backed" accounts. From Table 4.2,
we can see that the distribution of the first three kinds of accounts is significantly higher in the two detected
groups compared to the overall overlap (i.e., the random choice). As for the targets, the distributions are
more similar. The phenomenon shows the ability of our model to distinguish normal users from coordinated
56

0 2000
DIRECTO
CoronaTimo
Genocidio
CoronaFarsa
CovidHoax
NoAlNuevoOrdenMundial
NoAlBozal
NoALaVacuna
NoALaMascarillaObligatoria
tableau
powerbi
uipath
excel
nlg
qlik
resultsbi
tibco
microstrategy
bi
ai
PeriodistasCobardes
cpa
CoronaPandemic
_
freetrial
AlertaCOVID19SV
alexa
YoSoyLaResistencia
PoliticosAPrision
SanitariosAsesinos
Jaipur
NoMask
NoVaccine
SanitariosCobardes
NoAlConfinamiento
Cluster Anomaly (C2)
0 10000
news
FANTASTICRADIOUK
BELIEVEYOURPOSSIBILITIES
STAYHOMESAFELIVES
HOMEOFPOSSIBILITIES
News
pandemia
virus
Health
BreakingNews
cdnpoli
Pandemia
COVID19India
COVID19Pandemic
StayHomeSaveLives
CoronaUpdate
NHS
Italy
TamilNadu
AI
economy
Maharashtra
health
CoronaLockdown
coronavirusindia
CoronaInfoCH
vaccine
DonaldTrump
Africa
Salud
Quarantine
YoMeQuedoEnCasa
healthcare
Brazil
EEUU
Cluster Others (C0)
Figure 4.7: Top-35 frequent unique tweet hashtags of the normal users and the suspicious coordinated
group (the larger one).
57

Table 4.3: Representative tweets in disinformation topic clusters in identified COVID-19 coordinated
accounts groups.
VIRUS FRAUD - FAKE NEWS In Spite of Leftist Media Hysteria at the Time, SD Governor Noem Confirms there were
ZERO New Cases or ’Outbreaks’ over Trump’s Rushmore Event, Trump Supporters are Clean, Healthy People...
BREAKING: #BillGates Foundation And The #Covid19 VACCINE NETWORK SCANDAL The British People Are Going
To Get Very Angry Very Soon And Will Want ANSWERS.#GlaxoSmithKline #BillGates #Rothschild #WellcomeTrust
#COVID19 #CivilService #NoMasks
#BillGates Negotiated $100 Billion #ContactTracing Deal With #Democratic Congressman in 8/2019 well before
#Coronavirus #Pandemic starts. #Gates holds pandemic drill 10/2019 Harvard finds #SARSCOV2 started in 8/2019 in
#wuhan. Virus in USA by 1/2020.
accounts because the targeted accounts are expected to contain many more normal accounts with whom
the coordinated accounts try to influence and engage.
In Figure 4.7, we present the most frequent unique tweet hashtags of the normal users and the larger
coordinated group (we present the hashtags of the smaller one in the Appendix). From the comparison, we
can see that the detected C2 group delivers more content about anti-mask and anti-vaccine (“No-Masks",
“No-Vaccine," and “NoALaVacuna"). Also, there are some narratives believing that the pandemic is a hoax
(“CovidHoax").
Besides, we also use topic modeling to find the most representative tweets posted by the suspicious
coordinated group. We construct a similarity graph of the tweets based on their word frequency and
construct clustering on it. We present some tweets close to the cluster center in Table 4.3. As we can
see, they show some anti-vaccine narratives and believe that the pandemic is a scam of Bill Gates and
governments to monetize vaccines.
58

Chapter 5
Knowledge-Informed Detection of Coordinated Social Manipulation
In the previous chapter, we discussed that social manipulators’ groups could heavily influence the informationspreading process on social media [7, 134, 135]. Most of such manipulators’ accounts are operated by
misinformation campaigns. To tackle this challenge, in the previous chapter, we present a novel temporal
point process model that can detect coordinated account groups through unsupervised representation
learning. By conducting collective anomaly detection from these account representations, we can recognize
the hidden groups of manipulators.
However, data from social media have a distinct and important characteristic: the appearance of
accounts in diffusion cascades typically follows a long-tail distribution [87] (illustrated in Fig. 5.1a). This
attribute presents a unique challenge: in contrast to very few dominant accounts, most accounts are sparsely
represented in the data, thus hurting the effectiveness of deep representation learning-based models. Some
previous works leveraging predefined collective behaviors [9, 158, 113] can address this challenge. They
primarily share the same paradigm where similarity graphs are first constructed from the data with certain
prior knowledge or hypotheses, followed by graph-based clustering. However, their expressive capacity is
seriously limited since the sophisticated interactions are simply expressed as edges with scalar weights,
and they heavily rely on predefined coordination signatures. Consequently, they seriously suffer from
unsatisfactory accuracy compared to state-of-the-art deep representation learning-based models [137].
59

2021-01-11 22:25:05 RT @CaliVaxChoice: In Rush to
Create Magic-Bullet COVID Vaccines, Have We Made
Matters Worse? • https://t.co/EV4SLEWXUv Study found
vaccines that don’t prevent viral transmission may
accelerate evolution of more virulent strains could mean
leading vaccine candidates may make COVID crisis
worse. https://t.co/NMLHujNeGe
2021-01-12 16:35:17 RT @NVICLoeDown: In Rush to
Create Magic-Bullet COVID Vaccines, Have We Made
Matters Worse? • https://t.co/CQgkKca4nK Study that
found vaccines don’t prevent viral transmission may
accelerate evolution of more virulent strains could mean
leading vaccine candidates may make COVID crisis worse.
https://t.co/Yy3XiSgzBm
2021-02-09 20:31:17 RT @RobertKennedyJr: A
second @nytimes article quotes doctors who
say the mRNA technology used in #COVID
#vaccines may cause immune
thrombocytopenia, a blood disorder that last
month led to the death of a Florida doctor after
getting the #Pfizer vaccine. #TheDefender
https://t.co/9WrMGYtygm
2021-02-09 20:20:20 RT @RobertKennedyJr: A
second @nytimes article quotes doctors who
say the mRNA technology used in #COVID
#vaccines may cause immune
thrombocytopenia, a blood disorder that last
month led to the death of a Florida doctor after
getting the #Pfizer vaccine. #TheDefender
https://t.co/9WrMGYtygm
(a) Example of collaborated behaviours.
0 500 1000 1500 2000
# Rank
0
2000
4000
6000
# Frequency
(b) Frequency statistic of accounts.
Figure 5.1: The figure (a) is an example of coordinated accounts detected by our method on Twitter. They
retweet similar anti-vaccine contents about COVID-19 Vaccines from same or different sources. The figure
(b) is the frequency statistic of accounts in IRA dataset about the U.S. 2016 Election.
To tackle the challenges above, we propose a knowledge-informed neural temporal point process model
called VigDet (Variational Inference for Group Detection). It incorporates data-driven detectors with
domain knowledge about collective behaviors of coordinated accounts by defining various coordination
signatures, such as accounts that co-occur or synchronize in time and are more likely to be coordinated.
Different from prior attempts that heavily rely on assumed prior knowledge and struggle to learn knowledge
effectively from the data [9, 158], VigDet encodes prior knowledge as a graph through temporal logic and
power functions so that we can apply the graph as an important regularization to data-driven learning.
This approach guides the learning process of the neural point process model and facilitates the effective
inference of coordinated accounts. Furthermore, VigDet maintains a distribution over group assignments
and defines a potential score function that measures the consistency of group assignments in terms of
both embedding space and prior knowledge. Consequently, VigDet can make efficient inferences over the
constructed prior knowledge graph while simultaneously learning the account embeddings using neural
point processes.
A critical challenge in VigDet lies in the aforementioned group assignment distribution, which is
defined as a Gibbs distribution on a Conditional Random Field [85]. This Gibbs distribution applied a
partition function as a normalizer [83], which is NP-hard to be precisely computed or sampled from. Such
characteristics make the Gibbs distribution intractable from which to be trained or sampled. Consequently,
both learning and inference processes encounter significant challenges [15, 82]. To tackle this obstacle, we
60

employ variational inference [108]. Specifically, we approximate the Gibbs distribution with a mean field
distribution [112]. Subsequently, we employ the EM algorithm to jointly learn the approximation and the
learnable parameters, aiming to maximize the evidence lower bound (ELBO) [108] of the observed data
likelihood. During the E-step, we fix the learnable parameters and infer the optimal approximation, while
during the M-step, we fix the approximation and update the parameters to maximize an objective function,
which serves as a lower bound of the ELBO with a theoretical guarantee. Our experiments on a real-world
dataset [97] involving coordination detection validate the effectiveness of our model compared with other
baseline models, including the current state of the art. We further apply our method to a dataset of tweets
about the COVID-19 vaccine without ground-truth coordinated group labels. The analysis of the detection
result suggests the existence of suspicious coordinated efforts to spread misinformation and conspiracies
about COVID-19 vaccines.
5.1 Task Definition: Coordinated Group Detection on Social Media
In this section, we recall the definition of coordinated group detection on social media. In coordinated group
detection, we are given a temporal sequence dataset S = {S1, ..., S|D|} from social media, where each
sequence Si = [(vi1, ti1),(vi2, ti2), · · · ] corresponds to a piece of information, e.g. a tweet, and each event
(vij , tij ) means that an account vij ∈ V (corresponding to a type mark in MTPP) interacts with the tweet
(like comment or retweet) at time tij . Supposing that the V consists of M groups, our objective is to learn a
group assignment Y = {yv|v ∈ V, yv ∈ {1, ..., M}}. This task can be conducted in an unsupervised or
semi-supervised setting. In an unsupervised setting, we do not have the group identity of any account. As
for the semi-supervised setting, the ground-truth group identity YL of a small account fraction VL ⊂ V is
accessible. The current state-of-the-art model on this task is AMDN-HAGE with k-Means. It first learns the
account embeddings E with AMDN-HAGE. Then, it obtains group assignment Y using k-Means clustering
on learned E.
61

5.2 Proposed Method: VigDet
In this section, we introduce our proposed model called VigDet (Variational Inference for Group Detection),
which bridges neural temporal point process and graph based method based on prior knowledge. Unlike
the existing methods, in VigDet we regularize the learning process of the account embeddings with the
prior knowledge based graph so that the performance can be improved. Such a method addresses the heavy
reliance of deep learning model on the quality and quantity of data as well as the poor expressive power of
existing graph based methods exploiting prior knowledge.
5.2.1 Prior Knowledge-based Graph Construction
For the prior knowledge-based graph construction, we apply co-activity [126] to measure the similarity of
accounts. This method assumes that the accounts that always appear together in the same sequences are
more likely to be in the same group. Specifically, we construct a dense graph G =< V, E > whose node set
is the account set and the weight wuv of an edge (u, v) is the co-occurrence:
wuv =
X
S∈S
1((u ∈ S) ∧ (v ∈ S)) (5.1)
However, when integrated with our model, this edge weight is problematic because the coordinated accounts
may also appear in the tweets, attracting normal accounts. Although the co-occurrence of coordinated
account pairs is statistically higher than other account pairs, since coordinated accounts are only a small
fraction of the whole account set, our model will tend more to predict an account as a normal account.
Therefore, we apply one of the following two strategies to acquire filtered weight w
′
uv:
62

Power Function based filtering: the co-occurrence of a coordinated account pair is statistically higher
than a coordinated-normal pair. Thus, we can use a power function with exponent p > 1 (p is a hyperparameter) to enlarge the difference and then conduct normalization:
w
′
uv = (X
S∈S
1((u ∈ S) ∧ (v ∈ S)))p
(5.2)
where u ∈ S and v ∈ S mean that u and v appear in the sequence respectively. Then, the weight with a
relatively low value will be filtered via normalization (details in next subsection).
Temporal Logic [91] based filtering: We can represent some prior knowledge as a logic expression
of temporal relations, denoted as r(·), and then only count those samples satisfying the logic expressions.
Here, we assume that the active time of accounts of the same group are more likely to be similar. Therefore,
we only consider the account pairs whose active time overlap is larger than a threshold (we apply half a
day, i.e., 12 hours):
w
′
uv =
X
S∈S
1((u ∈ S) ∧ (v ∈ S) ∧ r(u, v, S)),
r(u, v, S) = 1(min(tul, tvl) − max(tus, tvs) > c)
(5.3)
where tul, tvl are the last time that u and v appears in the sequence and tus, tvs are the first (starting) time
that u and v appears in the sequence.
5.2.2 Integrate Prior Knowledge and Neural Temporal Point Process
To integrate prior knowledge and neural temporal point process while maximizing the likelihood of the
observed sequences log p(S|E) given account embeddings, VigDet simultaneously learns a distribution
63

E-step: Enhance Prediction with
Belief Propagation on Prior
Knowledge based Graph G
M-step: Update Embedding and Potential
Function with SGD to Fit the Knowledge
Informed Prediction and the Observed Data
Modeling the
Embedding-based
Potential Function via
a Neural Network
Knowledge
Informed
Prediction
Observed
Data
Data-Driven
Model
Graph
Construction
φθ
Prior
Knowledge
Figure 5.2: The overview of VigDet. In this framework, we aim at learning a knowledge informed datadriven model. To this end, based on prior knowledge we construct a graph describing the potential of
account pairs to be coordinated. Then we alternately enhance the prediction of the data-driven model with
the prior knowledge based graph and further update the model to fit the enhanced prediction as well as the
observed data.
over group assignments Y defined by the following potential score function given the account embeddings
E and the prior knowledge based graph G =< V, E >:
Φ(Y ; E, G) = X
u∈V
φθ(yu, Eu) + X
(u,v)∈E
ϕG(yu, yv, u, v) (5.4)
where φθ(yu, Eu) is a learnable function measuring how an account’s group identity yu is consistent
with the learned embedding, e.g., a feedforward neural network. And ϕG(yu, yv, u, v) is pre-defined as:
ϕG(yu, yv, u, v) = wuv √
dudv
1(yu = yv) (5.5)
where du, dv =
P
k wuk,
P
k wvk are the degrees of u, v and 1(yu = yv) is an indicator function that
equals 1 when its input is true and 0 otherwise. By encouraging co-appearing accounts to be assigned
to the same group, ϕG(yu, yv, u, v) regularizes E and φθ with prior knowledge. With the above potential
64

score function, we can define the conditional distribution of group assignment Y given embedding E and
the graph G:
P(Y |E, G) = 1
Z
exp(Φ(Y ; E, G)) (5.6)
where Z =
P
Y
exp(Φ(Y ; E, G)) is the normalizer keeping P(Y |E, G) a distribution, also known as
partition function [83, 81]. It sums up exp(Φ(Y ; E, G))for all possible assignment Y . As a result, calculating
P(Y |E, G) accurately and finding the assignment maximizing Φ(Y ; E, G) are both NP-hard [15, 82].
Consequently, we approximate P(Y |E, G) with a mean field distribution Q(Y ) = Q
u∈V Qu(yu). To
inform the learning of E and φθ with the prior knowledge behind G we propose to jointly learn Q, E, and
φθ by maximizing the following objective function, which is the Evidence Lower Bound (ELBO) of the
observed data likelihood log p(S|E) given embedding E:
O(Q, E, φθ; S, G) = log p(S|E) − DKL(Q||P) (5.7)
In this objective function, the first term is the likelihood of the observed data given account embeddings,
which can be modeled as P
S∈S log pθa
(S|E) with a neural temporal point process model like AMDN. The
second term regularizes the model to learn E and φθ such that P(Y |E, G) can be approximated by its
mean-field approximation as precisely as possible. Intuitively, this can be achieved when the two terms in
the potential score function, i.e. P
u∈V φθ(yu, Eu) and P
(u,v)∈E ϕG(yu, yv, u, v) agree with each other on
every possible Y . The above lower bound can be optimized via variational EM algorithm [108, 120, 121,
147].
5.2.2.1 E-step: Inference Procedure.
In the E-step, we aim at inferring the optimal Q(Y ) that minimizes DKL(Q||P). Note that the formulation
of the Φ(Y ; E, G) is the same as the Conditional Random Fields (CRF) [85] model, although their learnable
65

parameters are different. In the E-step, such difference is not important as all parameters in Φ(Y ; E, G) are
frozen. As existing works about CRF [83, 81] have theoretically proven, the following iterative updating
function of belief propagation converges at a locally optimal solution:
Qu(yu = m) = Qˆ
u(yu = m)
Zu
=
1
Zu
exp{φθ(m, Eu) +X
v∈V
X
1≤m′≤M
ϕG(m, m′
, u, v)Qv(yv = m′
)} (5.8)
where Qu(yu = m)is the probability that account u is assigned into group m and Zu =
P
1≤m≤M Qˆ
u(yu =
m) is the normalizer keeping Qu as a valid distribution.
Here, we provide a detailed justification for the above E-step based on previous works [81, 83]. Let us
recall the definition of the potential function Φ(Y ; E, G) and the Gibbs distribution defined on it P(Y |E, G):
Φ(Y ; E, G) = X
u∈V
φθ(yu, Eu) + X
(u,v)∈E
ϕG(yu, yv, u, v) (5.9)
P(Y |E, G) = 1
Z
exp(Φ(Y ; E, G)) (5.10)
where Z =
P
Y
exp(Φ(Y ; E, G)). With the above definitions, we have the following theorem:
Proposition 5.2.1. (Theorem 11.2 in [81])
DKL(Q||P) = log Z − EY ∼QΦ(Y ; E, G) − H(Q) (5.11)
where H(Q) is the information entropy of the distribution Q.
A more detailed derivation of the above equation can be found in the appendix of [83]. Since Z is fixed
in the E-step, minimizing DKL(Q||P) is equivalent to maximizing EY ∼QΦ(Y ; E, G) + H(Q). For this
objective, we have the following theorem:
66

Proposition 5.2.2. (Theorem 11.9 in [81]) Q is a local maximum if and only if:
Qu(yu = m) = 1
Zu
exp(EY −{yu}∼QΦ(Y − {yu}; E, G|yu = m)) (5.12)
where Zu is the normalizer and EY −{yu}∼QΦ(Y − {yu}; E, G|yu = m) is the conditional expectation of Φ
given that yu = m and the labels of other nodes are drawn from Q.
Meanwhile, note that the expectation of all terms in Φ that do not contain yu is invariant to the
value of yu. Therefore, we can reduce all such terms from both numerator (the exponential function) and
denominator (the normalizer Zu) of Qu. Thus, we have the following corollary:
Corollary 5.2.3. Q is a local maximum if and only if:
Qu(yu = m) = 1
Zu
exp{φθ(m, Eu) +X
v∈V
X
1≤m′≤M
ϕG(m, m′
, u, v)Qv(yv = m′
)} (5.13)
where Zu is the normalizer
A more detailed justification of the above corollary can be found in the explanation of Corollary 11.6 in
Sec 11.5.1.3 of [81]. Since the above local maximum is a fixed point of DKL(Q||P), fixed-point iteration
can be applied to find such a local maximum. More details, such as the stationary of the fixed points, can be
found in Chapter 11.5 of [81]
5.2.2.2 M-step: Learning Procedure.
In M-step, given fixed inference of Q we aim at maximizing OM:
OM = log p(S|E) − DKL(Q||P) = log p(S|E) + EY ∼Q log P(Y |E, G) + const (5.14)
67

The key challenge in M-step is that calculating EY ∼Q log P(Y |E, G) is NP-hard [15, 82]. To address this
challenge, we propose to alternatively optimize following theoretically justified lower bound:
Proposition 5.2.4. Given a fixed inference of Q and a pre-defined ϕG, we have following inequality:
EY ∼Q log P(Y |E, G) ≥ EY ∼Q
X
u∈V
log exp{φθ(yu, Eu)}
P
1≤m′≤M exp{φθ(m′
, Eu)}
+ const
=
X
u∈V
X
1≤m≤M
Qu(yu = m) log exp{φθ(m, Eu)}
P
1≤m′≤M exp{φθ(m′
, Eu)}
+ const
(5.15)
Proof. To simplify the notation, let us apply following notations:
Φθ(Y ; E) = X
u∈V
φθ(yu, Eu), ΦG(Y ; G) = X
(u,v)∈E
ϕG(yu, yv, u, v) (5.16)
Let us denote the set of all possible assignment as Y, then we have:
Ey∼Q log P(y|E, G) = EY ∼Q log exp(Φ(Y ; E, G))
P
Y ′∈Y exp(Φ(Y ′
; E, G))
= Ey∼QΦ(Y ; E, G) − log X
Y ′∈Y
exp(Φ(Y
′
; E, G))
= Ey∼Q(Φθ(Y ; E) + ΦG(Y ; G)) − log X
Y ′∈Y
exp(Φ(Y
′
; E, G))
(5.17)
Because ϕG is pre-defined, ΦG(Y ; G) is a constant. Thus, we have
Ey∼Q log P(y|E, G) = Ey∼QΦθ(Y ; E) − log X
Y ′∈Y
exp(Φ(Y
′
; E, G)) + const (5.18)
68

Now, let us consider the log P
Y ′∈Y exp(Φ(Y
′
; E, G)). Since ϕG is pre-defined, there must be an assignment
Ymax that maximize ΦG(Y ; G). Thus, we have:
log X
Y ′∈Y
exp(Φ(Y
′
; E, G)) ≤ log X
Y ′∈Y
exp(Φθ(Y ; E) + ΦG(Ymax; G))
= log exp(ΦG(Ymax; G)) X
Y ′∈Y
exp(Φθ(Y ; E))
= ΦG(Ymax; G) + log X
Y ′∈Y
exp(Φθ(Y ; E))
(5.19)
Since ϕG is pre-defined, ΦG(Ymax; G))is a constant during the optimization. Note thatP
Y ′∈Y expθ
(Φ(Y
′
; E))
sums up over all possible assignments Y
′ ∈ Y. Thus, it is actually the expansion of following product:
Y
u∈V
X
1≤m′≤M
exp(φθ(m′
, Eu)) = X
Y ′∈Y
Y
u∈V
exp(φθ(y
′
u
, Eu)) = X
Y ′∈Y
exp(Φθ(Y
′
; E)) (5.20)
Therefore, for Q which is a mean-field distribution and φθ which model each account’s assignment
independently, we have:
EY ∼Q log P(y|E, G) ≥ Ey∼QΦθ(Y ; E) − log X
Y ′∈Y
exp(Φθ(Y
′
; E)) + const
= EY ∼QΦθ(Y ; E) − log Y
u∈V
X
1≤m′≤M
exp(φθ(m′
, Eu)) + const
= EY ∼QΦθ(Y ; E) −
X
u∈V
log X
1≤m′≤M
exp(φθ(m′
, Eu)) + const
= EY ∼Q
X
u∈V
log exp{φθ(yu, Eu)}
P
1≤m′≤M exp{φθ(m′
, Eu)}
+ const
=
X
u∈V
X
1≤m≤M
Qu(yu = m) log exp{φθ(m, Eu)}
P
1≤m′≤M exp{φθ(m′
, Eu)}
+ const
(5.21)
69

Intuitively, the above objective function treats the Q as a group assignment enhanced via label propagation on the prior knowledge-based graph and encourages E and φθ to correct themselves by fitting the
enhanced prediction. Compared with pseudolikelihood [13], which is applied to address similar challenges
in recent works [120], the proposed lower bound has a computable closed-form solution. Thus, we do not
really need to sample Y from Q so that the noise is reduced. Also, this lower bound does not contain ϕG
explicitly in the non-constant term. Therefore, we can encourage the model to encode graph information
into the embedding.
5.2.2.3 Joint Training:
The E-step and M-step form a closed loop. To create a starting point, we initialize E with the embedding
layer of a pre-trained neural temporal process model (in this paper, we apply AMDN-HAGE) and initialize
φθ via clustering learned on E (like fitting the φθ to the prediction of k-Means). After that, we repeat the
E-step and M-step steps to optimize the model. The pseudo-code of the training algorithm is presented in
Alg. 3.
Algorithm 3 Training Algorithm of VigDet.
Require: Dataset S and pre-defined G and ϕG
Ensure: Well trained Q, E and φθ
1: Initialize E with the embedding layer of AMDN-HAGE pre-trained on S.
2: Initialize φθ on the initialized E.
3: while not converged do
4: Acquire Q by repeating Eq. 5.8 with E, φθ and ϕG until convergence.{E-step}
5: φθ, E ← argmaxφθ,E log p(S|E) + EY ∼Q
P
u∈V log P
exp{φθ(yu,Eu)}
1≤m′≤M exp{φθ(m′
,Eu)}
. {M-step}
6: φθ, E ← max log p(S|E) + P
u∈V Eyu∼Qu
log softmax(φθ(yu, Eu)). {M-step}
7: end while
5.2.3 Semi-supervised extension
The above framework does not make use of the ground-truth label in the training procedure. In a semisupervised setting, we actually have the group identity YL of a small account fraction VL ⊂ V. Under
70

this setting, we can naturally extend the framework via following modification to Alg. 3: For account
u ∈ VL, we set Qu as a one-hot distribution, where Qu(yu = y
′
u
) = 1 for the groundtruth identity y
′
u
and
Qu(yu = m) = 0 for other m ∈ {1, ..., M}.
5.3 Experiments
5.3.1 Coordination Detection on IRA Dataset
We utilize the Twitter dataset containing coordinated accounts from Russia’s Internet Research Agency (IRA
dataset [97, 137]) attempting to manipulate the U.S. 2016 Election. The dataset contains tweet sequences
(i.e., tweets with account interactions like comments, replies, or retweets) constructed from the tweets
related to the U.S. 2016 Election.
This dataset contains activities involving 2025 Twitter accounts. Among the 2025 accounts, 312 are
identified through U.S. Congress investigations∗
as coordinated accounts and other 1713 accounts are normal
accounts joining in discussion about the Election during the period of activity those coordinated accounts.
This dataset is applied for evaluation of coordination detection models in recent works [97, 137]. In this
paper, we apply two settings: unsupervised setting and semi-supervised setting. For unsupervised settings,
the model does not use any ground-truth account labels in training (but for hyperparameter selection,
we hold out 100 randomly sampled accounts as a validation set and evaluate with reported metrics on
the remaining 1925 accounts as the test set). For the semi-supervised setting, we similarly hold out 100
accounts for hyperparameter selection as a validation set and another 100 accounts with labels revealed in
the training set for semi-supervised training). The evaluation is reported on the remaining test set of 1825
accounts. The hyperparameters of the backbone of VigDet (AMDN) follow the original paper [137]. Other
implementation details are in the Appendix.
∗
https://www.recode.net/2017/11/2/16598312/russia-twittertrump-twitter-deactivated-handle-list
71

5.3.1.1 Evaluation Metrics and Baselines
In this experiment, we mainly evaluate the performance of two versions of VigDet: VigDet (PF) and VigDet
(TL). VigDet (PF) applies Power Function-based filtering, and VigDet (TL) applies Temporal Logic-based
filtering. For the p in VigDet (PF), we apply 3. We compare them against existing approaches that utilize
account activities to identify coordinated accounts.
Unsupervised Baselines: Co-activity clustering [126] and Clickstream clustering [158] are based on
pre-defined similarity graphs. HP (Hawkes Process) [185] is a learnt graph based method. IRL[97] and
AMDN-HAGE[137] are two recent representation learning method.
Semi-Supervised Baselines: Semi-NN is a semi-supervised feedforward neural network without
requiring additional graph structure information. It is trained with a self-training algorithm [189, 118].
Label Propagation Algorithm (LPA) [188] and Graph Neural Network (GNN) (we use the GCN [79], the most
representative GNN) [79, 154, 60] are two baselines incorporated with graph structure. In LPA and GNN,
for the graph structures (edge features), we use the PF and TL based prior knowledge graphs (similarly
used in VigDet), as well as the graph learned by HP model as edge features. For the node features in GNN,
we provide the account embeddings learned with AMDN-HAGE.
Ablation Variants: To verify the importance of the EM-based variational inference framework and
our proposed objective function in M-step, we compare our models with two variants: VigDet-E and
VigDet-PL (PL for Pseudo Likelihood). In VigDet-E, we only conduct the E-step once to acquire group
assignments (inferred distribution over labels) enhanced with prior knowledge but without alternating
updates using the EM loop. It is similar to some existing works conducting post-processing with CRF to
enhance prediction based on the learned representations [24, 71]. In VigDet-PL, we replace our proposed
objective function with a pseudo-likelihood function from existing works.
72

Metrics: We compare two kinds of metrics. One kind is threshold-free: Average Precision (AP), the
area under the ROC curve (AUC), and maxF1 at the threshold that maximizes the F1 score. The other kind
needs a threshold: F1, Precision, Recall, and MacroF1. For this kind, we apply 0.5 as the threshold for the
binary (coordinated/normal account) labels.
5.3.1.2 Implementation details on IRA dataset
We split the sequence set to 75/15/15 fractions for training/validation/test sets. For the setting of AMDN
and AMDN-HAGE [137], we use the default setting from the original paper, including activity sequences of
maximum length 128 (we split longer sequences), batch size of 256 (on 1 NVIDIA-2080Ti GPU), embedding
dimension of 64, number of mixture components for the PDF in the AMDN part of 32, single head and
single layer attention module, component number in the HAGE part of 2. Our implementation is totally
based on PyTorch and Adam optimizer with 1e-3 learning rate and 1e-5 regularization (same as [137]). The
number of loops in the EM algorithm is picked up from {1, 2, 3} based on the performance of the validation
account set. In each E-step, we repeat the belief propagation until convergence (within ten iterations) to
acquire the final inference. In each M-step, we train the model for a maximum of 50 epochs with early
stopping based on the validation objective function. The validation objective function is computed from
the sequence likelihood on the 15% held-out validation sequences and KL-divergence on the whole account
set based on the inferred account embeddings in that iteration.
5.3.1.3 Results
Table 5.1 and 5.2 provide results of model evaluation against the baselines averaged in the IRA dataset
over five random seeds. As we can see, VigDet, as well as its variants, outperforms other methods on both
unsupervised and semi-supervised settings due to their ability to integrate neural temporal point process,
which is the current state-of-the-art method, and prior knowledge, which is robust to data quality and
73

Table 5.1: Results on unsupervised coordination detection (IRA) on Twitter in 2016 U.S. Election
Method (Unsupervised) AP AUC F1 Prec Rec MaxF1 MacroF1
Co-activity 16.9 52.5 24.6 17.8 40.7 27.1 49.5
Clickstream 16.5 53.2 21.0 20.6 21.6 21.0 53.1
IRL 23.9 68.7 35.3 27.5 49.4 38.6 58.8
HP 29.8 56.7 44.2 42.1 46.6 46.0 66.7
AMDN-HAGE 80.5 89.9 69.6 94.3 55.5 75.8 82.7
AMDN-HAGE + k-Means 82.0 93.3 73.0 90.9 61.2 77.0 84.5
VigDet-PL(TL) 83.3 94.0 70.7 89.6 59.0 77.8 83.2
VigDet-E(TL) 85.5 94.6 73.1 95.3 59.4 79.5 84.6
VigDet(TL) 86.1 94.6 73.4 95.1 59.9 79.6 84.8
VigDet-PL(PF) 84.5 95.0 71.9 91.4 59.6 79.3 83.9
VigDet-E(PF) 85.1 94.3 73.6 92.7 61.2 78.8 84.9
VigDet(PF) 87.2 95.0 75.2 91.7 63.9 79.3 85.7
quantity. It is noticeable that although GNN-based methods can also integrate prior knowledge-based graphs
and representation learning from the state-of-the-art models, our model still outperforms it by modeling
and inferring the distribution over group assignments jointly guided by consistency in the embedding and
prior knowledge space.
Ablation Test: Besides baselines, we also compare VigDet with its variants VigDet-E and VigDet-PL. As
we can see, for the Power Filtering strategy, compared with VigDet-E, VigDet achieves significantly better
results on most of the metrics in both settings, indicating that leveraging the EM loop and proposed M-step
optimization can guide the model to learn better representations for E and φθ. As for the Temporal Logic
Filtering strategy, VigDet also brings boosts, although relatively marginal. Such phenomenon suggests
that the performance of our M-step objective function may vary with the prior knowledge we applied.
Meanwhile, the VigDet-PL performs not only worse than VigDet, but also worse than VigDet-E. This
phenomenon shows that the pseudolikelihood is noisy for VigDet and verifies the importance of our
objective function.
74

Table 5.2: Results on semi-supervised coordination detection (IRA) on Twitter in 2016 U.S. Election
Method (Semi-Supervised) AP AUC F1 Prec Rec MaxF1 MacroF1
LPA(HP) 63.3 76.8 68.1 76.2 61.8 71.6 81.5
LPA(TL) 69.7 85.9 62.3 88.5 48.6 66.1 78.6
LPA(PF) 71.1 85.3 60.8 66.5 56.4 68.3 77.2
AMDN-HAGE + Semi-NN 77.1 87.8 70.5 76.6 65.5 72.3 82.8
AMDN-HAGE + GNN (HP) 75.5 84.0 72.0 83.0 65.1 76.6 83.7
AMDN-HAGE + GNN (PF) 80.6 89.5 73.0 86.3 63.7 76.4 84.5
AMDN-HAGE + GNN (TL) 81.3 90.2 73.6 78.2 70.2 77.2 84.6
VigDet-PL(TL) 87.7 95.5 73.9 94.2 61.4 80.0 85.1
VigDet-E(TL) 88.1 95.7 73.4 94.6 60.4 80.8 84.8
VigDet(TL) 88.0 95.7 73.6 94.2 60.9 80.8 84.9
VigDet-PL(PF) 85.1 95.3 69.7 93.4 55.9 79.0 82.8
VigDet-E(PF) 87.1 95.2 74.4 92.8 62.4 79.7 85.3
VigDet(PF) 87.6 95.6 76.1 87.2 68.1 79.8 86.2
5.3.2 Analysis on COVID-19 Vaccines Twitter Data
5.3.2.1 Implementation details on COVID-19 Vaccine Tweets dataset
We apply the Cubic Function-based filtering because it shows better performance on unsupervised detection
on the IRA dataset. We follow all the rest of the settings of VigDet (CF) in IRA experiments except the GPU
number (on 4 NVIDIA-2080Ti). Also, for this dataset, since we have no prior knowledge about how many
groups exist, we first pre-train an AMDN by only maximizing its observed data likelihood on the dataset.
Then, we select the best cluster number that maximizes the silhouette score as the group number. The final
group number we select is 2. The silhouette scores are shown in Fig. 5.3. After that, we train the VigDet on
the dataset with a group number of 2. As for the final threshold we select for detection, we set it as 0.8
because it maximizes the silhouette score on the final learned embedding†
.
†
0.9 can achieve a better silhouette score but gets worse scores on some intermediate metrics like unreliable hyperlink source
ratio
75

2 3 4 5 6 7
Cluster Number
0.015
0.020
0.025
0.030
0.035
0.040
0.045
Silhouette Score
0.043
0.033
0.03
0.025
0.02
0.019
Figure 5.3: The silhouette scores of different group number.
5.3.2.2 Results on COVID-19 Vaccine Tweets dataset
Detection: VigDet detects 8k suspicious accounts from the 31k Twitter accounts. We inspect tweets and
account features of the identified suspicious group of coordinated accounts.
Representative tweets: We use topic mining on tweets of detected coordinated accounts and show
the text contents of the top representative tweets in Table 5.3.
Account features: The two groups (identified coordinated and normal accounts) are clearly distinguished in the comparison of top-30 hashtags in tweets posted by the accounts in each group (presented
in Fig. 5.4). In bold are the non-overlapping hashtags. The coordinated accounts seem to promote
that the pandemic is a hoax (#scamdemic2020, #plandemic2020), as well as anti-mask, anti-vaccine and
anti-lockdown (#notcoronavirusvaccines, #masksdontwork, #livingnotlockdown) narratives, and political
agendas (#trudeaumustgo). The normal accounts’ narratives are more general and show more positive
attitudes toward vaccines, masks, and prevention protocols.
Also, we measure the percentage of unreliable and conspiracy news sources shared in the tweets of
the detected coordinated accounts, which is 55.4%, compared to 23.2% in the normal account group. The
percentage of recent accounts (created in 2020-21) is higher in the coordinated group (20.4%) compared to
76

0 2000
covid19
vaccine
cdnpoli
covidvaccine
auspol
coronavirus
onpoli
covid
vaccines
trudeauvaccinefail
trudeaufailedcanada
pfizer
trudeaumustgo
canada
vaccination
astrazeneca
lockdown
trudeauvaccinefailure
scamdemic2020
covid19vaccine
cdnmedia
crimesagainsthumanity
uk
masksdontwork
notocoronavirusvaccines
trudeau
plandemic2020
lockdownprotest
livingnotlockdown
lockdownchaos
Coordinated group
0 10000
covid19
vaccine
coronavirus
covidvaccine
covid
staysafe
maskup
smallbusiness
washyourhands
redbubble
covid_19
vaccines
pfizer
healthcare
machinelearning
artificialintelligence
vaccination
astrazeneca
cdnpoli
covid19vaccine
largestvaccinedrive
wearamask
moderna
health
trump
christmas
indiafightscorona
newyear
uk
operationbreathe
freshcleanair
Normal group
Figure 5.4: Top-30 hashtags in tweets of suspicious coordinated group and normal group
15.3% otherwise. Disinformation and suspensions are not exclusive to coordinated activities, and suspensions
are based on Twitter’s manual process and get continually updated over time; also, accounts created earlier
can include recently compromised accounts; therefore, these measures cannot be considered as absolute
ground-truth.
77

Table 5.3: Representative tweets from topic clusters in tweets of identified coordinated accounts.
If mRNA vaccines can cause autoimmune problems and more severe reactions to coronavirus’
maybe that’s why Gates is so confident he’s onto a winner when he predicts a more lethal
pandemic coming down the track. The common cold could now kill millions but it will be
called CV21/22?
This EXPERIMENTAL “rushed science" gene therapy INJECTION of an UNKNOWN substance (called a “vaccine" JUST TO AVOID LITIGATION of UNKNOWN SIDE EFFECTS) has
skipped all regular animal testing and is being forced into a LIVE HUMAN TRIAL.. it seems
to be little benefit to us really!
This Pfizer vax doesn‘t stop transmission,prevent infection or kill the virus, merely reduces
symptoms. So why are they pushing it when self-isolation/Lockdowns /masks will still be
required. Rather sinister especially when the completion date for trials, was/is 2023
It is ¯- You don´t own anything, including your body. - Full and absolute ownership of your
biological being. - Disruption of your immune system. - Maximizing gains for #BillGatesBioTerrorist. - #Transhumanism - #Dehumanization’
It is embarrassing to see Sturgeon fawning all over them. The rollout of the vaccine up here
is agonisingly slow and I wouldn’t be surprised if she was trying to show solidarity with the
EU. There are more benefits being part of the UK than the EU.
It also may be time for that “boring” O’Toole (as you label him) to get a little louder and
tougher. To speak up more. To contradict Trudeau on this vaccine rollout and supply mess.
O’Toole has no “fire”. He can’t do “blood sport”. He’s sidelined by far right diversions.
78

Chapter 6
Revealing the Causal Influence of Misinformation on Social Media Users
Recent research reveals that widespread fake news and misleading information have been exploited by
misinformation campaigns to manipulate public opinions in different areas, such as healthcare [135, 141,
84] and politics [97]. To address this crucial challenge, research efforts from different perspectives have
been devoted, such as fake news detection and coordination detection [134, 135, 141].
However, an essential associated research question has not been explored sufficiently: how to know
a piece of misinformation’s causal influence on a user’s beliefs and activities on large-scale social media.
Precisely estimating such impact is crucial for misinformation mitigation in various areas, e.g., delivering the corresponding clarification contents to the users that are most likely to be affected, allocating
resources for more efficient and effective misinformation mitigation, and helping researchers understand
misinformation campaigns better. Nevertheless, most of the existing research in social media analysis
focused on understanding the correlation between misinformation and user activities rather than causal
effect [115, 141, 84, 165]. As a result, they can not distinguish the effect from personal prior beliefs and
engagement with misinformation. Current research on misinformation’s causal influence on people is
mainly from the psychology field [72, 152]. They are usually based on carefully designed psychological
randomized controlled trials on recruited subjects. Thus, it is impossible to extend them onto large-scale
79

social media platforms due to the high cost of recruiting enough subjects and the ethical risk in conducting
such large-scale psychology experiments.
Since personal beliefs are usually unobservable, researchers usually apply the feature of the tweets
generated or retweeted by the users as a proxy [141]. However, the lack of appropriate algorithmic tools to
conduct causal analysis on social media posts prevents researchers from understanding the causal effect of
misinformation. The processes in which social media users generate original posts and engage with existing
posts are typical temporal point processes. But existing methodologies for temporal causal effect estimation
mostly focus on covariates and outcomes continuously distributed on timeline [14, 12, 8, 19, 75], rather
than discrete event points randomly scattered on timeline. Although the essential theory for counterfactual
analysis of point process is already established[125], most works are motivated by healthcare and thus focus
on the hazard models, e.g., survival analysis [1] or the chance to catch cancer [127], which only consider the
single occurrence of the most recent future event. However, on social media, we care more about multiple
events happening in a time window. [56] and [110] are rare works studying the causal effect on multiple
occurrences in the temporal point process. But [110] mainly focuses on simulating the counterfactual
events given an intervened intensity function rather than learning the treatment effect of specific factors
on the process. As for [56], one of its assumptions is that the event marks must be categorized into a finite
number of classes, leaving no space for the rich, continuous features of social media posts, such as user
sentiment and subjectivity scores.
In this work, we propose a framework that models the causal effect of a given piece of information
on user beliefs and activities via counterfactual analysis on temporal point process [137, 42, 104, 178, 193,
111, 25] with continuous features. We first define a causal structure model that characterizes the impact of
misinformation as how engagement with misinformation changes a user’s intensity function in generating
original posts. In this model, the engagement with misinformation is considered as the treatment, and the
user’s future conditional intensity function is considered as the outcome. Then, we design a function that
80

converts the change of two functions to a vector with intuitive physical meaning [105]. To estimate the
effect, we design a neural temporal point process model. It disentangles the distribution of event timestamp
and post feature (e.g. the text embedding). Then it models the distribution of post features and event
timestamp with Gaussian Mixture Model and temporal point process respectively. Such design enables it to
acquire a closed-form solution of the feature expect without losing expressive power, leading to a balance
between precision and efficiency.
A critical challenge in training neural networks to recognize causal effects is the hidden bias in the
dataset. In social media data, the most crucial bias is from information cocoons [192]: users tend more
to engage with the contents that they are interested in, and thus personalized recommendation systems
will deliver users more content that they are interested in to increase user engagement. Such bias leads
to a data distribution different from randomised controlled trials and thus make neural networks give
biased estimation. To decorrelate time-varying treatment from the user’s covariates and history in the point
process, we apply adversarial training to optimize a min-max game. More specifically, the encoder tries to
minimize the likelihood of the observed treatments while a treatment predictor tries to maximize it. Our
theoretic analysis proves that any balanced solution of the min-max game, rather than the global optimal
solution in existing works [14], can help us remove the bias from information cocoons. In addition, the
extensive experiments on synthetic datasets and real-world datasets indicate that our framework is able to
approximate unbiased and identifiable estimation of the causal effect. In conclusion, the main contributions
of the proposed model are as follows:
• We propose a novel research problem on misinformation impact, which aims to find the causal effect
of misinformation on users’ belief and activities on social media.
• We propose a causal structure model to quantify the causal effect of misinformation and further
design a neural temporal process model to conduct unbiased estimation to the effect.
81

• We evaluate our model on synthetic datasets to examine its effectiveness and efficiency and ues it to
recognize identifiable causal effect of misinformation from real-world data.
6.1 Preliminary and Related Research
6.1.1 Temporal Point Process with Event Features
The process that a user retweets or posts tweets can usually be modeled as a temporal point process with
event feature [137, 194, 129, 39]. A temporal point process (TPP) with event feature is a stochastic process
whose realization is a sequence of discrete events in a continuous timeline: S = [(f1, t1),(f2, t2), ...],
where f
∗
is the event feature (a scalar or a vector) and t is the timestamp of the event. A TPP is fully
characterized by an intensity function λ(f, t|HT1) defined in the following integral equation:
E(N(F, T1, T2)|HT1) = Z
F
df
Z T2
T1
λ(f, t|HT1)dt (6.1)
where F is an area in the feature space, HT1 is the historical sequence of all events happening before time
T1, N(F, T1, T2) is the number of events whose feature vectors are in F and timestamps are in the range
[T1, T2]. The meaning of λ(f, t|HT1) is the expected instantaneous speed that the user generates posts
at point f in the feature space on time t. The process after time ti
is fully described by λ(·, ·|HT1) [25].
Recent works propose to apply neural networks to model the λ function [42, 104, 178, 193, 111, 25].
6.1.2 Counterfactual Analysis on Temporal Point Process and Continuous Time Series
The works focusing on studying the causal effect on multiple occurrences in the temporal point process are
rare. In [110], the authors mainly focus on the sampling of counterfactual events rather than learning the
influence of specific factors on the intensity function. Another work[56] proposes a counterfactual analysis
∗We use bold font to emphasize that f is a feature vector rather than a function.
82

framework to understand the causal influence of event pairs in the temporal point process. It defines the
individual treatment effect (ITE) of an event toward future process as:
IT E = µ
1
y
(t, t + T) − µ
0
y
(t, t + T) = 1
T
Z t+T
t
λ
1
y
(t) − λ
0
y
(t)dt (6.2)
where µy is the expect of the event number of type y per unit time in time range [t, t + T]. µ
1
y
indicates the
case that a treatment is applied (exposed to misinformation), and µ
0
y
is the contrary. However, this metric is
only suitable in the case that the events can be categorized into finite discrete types. This is not applicable
to social media posts because most meaningful features of the posts, such as geographical information,
sentiment score, and subjective score, are naturally continuous. Forcibly discretizing them will cause a
loss of meaningful information. Besides, for counterfactual analysis of time series, [14] proposes CRN, a
neural model that can learn unbiased estimation to counterfactual world and causal effect. [12] proposes
to analyze counterfactual estimation using synthetic controls via a novel neural controlled differential
equation model. [8] introduces a new causal prior graph to avoid undesirable explanations that include
confounding or noise and uses a multivariate Gaussian distribution to model the real continuous values.
However, all of them focus on modeling an observable variable existing on a continuous timeline instead of
a temporal point process. Unless getting heavily revised, such previous work can not be simply transferred
to our problem scenario.
6.2 Proposed Causal Structure Model and Treatment Effect
6.2.1 Causal Structure Model
In this study, we focus on understanding how a user’s engagement with the misinformation post causally
influences the characters of the posts generated by the user in a fixed future time window. We formulate
the process in which a user interacts with the posts shared by others and generates social media posts
83

Treatments:
Engagement with
Misinformation or
Information
Covariates:
History Activities
�
Sampling
Post Post Post
Post Engage Post
Engage Engage Post
…
Figure 6.1: The proposed causal structured model describing the impact of a piece of information on user.
as two temporal point processes where each event carries a continuous outcome vector. We denote the
process of engaging the diffusion of a post as Pe and the process generating new posts as Pg. Then the
realization of the two temporal point processes are respectively, two sequences Se and Sg of discrete events
with continuous outcome vectors in a continuous time range:
Se = [(f
(e)
1
, t(e)
1
),(f
(e)
2
, t(e)
2
), ...], Sg = [(f
(g)
1
, t(g)
1
),(f
(g)
2
, t(g)
2
), ...] (6.3)
where f is the feature vector characterizing the event and t is the time stamp. For Se, each event
corresponds to an interaction (e.g., "like“ or comment), and the vector f
(e)
is the feature of the content
(like the text representation, sentiment score, and metadata) from others. Similarly, for Sg, the vector
f
(g)
is the feature of the content generated by the user. To examine the causal effect of an interaction
event on the posts generated by the user in the future, we construct the following causal structure model,
formulated as < X, Y, T r >, where X is the covariate, Y is the outcome, and T r is the treatment. In
this model, given an interaction event (f
(e)
i
, t(e)
i
) whose causal effect is to be examined, we consider this
event as the treatment T r, and all the events, including both engagement events and posting events, that
happen before t
(e)
i
are considered as the covariates. As for the outcome, rather than simply considering the
most next-generation event after t
(e)
i
, we need a representation that can reflect the change of the whole
generating process Sg in a fixed future time window T. Thus, we apply the conditional intensity function
84

of the future process, denoted as λ(f, t|T r ∪ X), as an outcome. As discussed in the related work section,
the λ function completely describes the future process. The overview of the model is presented in Figure.
6.1.
6.2.2 Treatment Effect Evaluation
In traditional counterfactual analysis, the outcome is usually a scalar or a vector with finite dimensions.
Thus, the treatment effect can be trivially computed by comparing the difference between the outcomes
from the real world and the counterfactual world. However, in our framework, it is non-trivial to compute
the difference between two functions. To overcome this challenge, we propose to first apply a functional F
to project the λ function to a vector with finite dimensions:
FT (λ, T r ∪ X) = ϕ(t, t + T, λ, T r ∪ X)
µ(t, t + T, λ, T r ∪ X)
(6.4)
ϕ(t, t + T, λ, T r ∪ X) = ES∼P(S|λ,T r∪X)
[
X
(fi,ti)∈St:t+T
fi
] (6.5)
µ(t, t + T, λ, T r ∪ X) = ES∼P(S|λ,T r∪X)
|St:t+T | = E(N(sup(f), t, t + T)|T r ∪ X) (6.6)
where P(S|λ, T r ∪ X) is the distribution of the event sequence S sampled from the temporal point process
described by λ(·, ·|T r ∪ X), T is the time window that is a hyper-parameter, sup(f) is the support set
of f (the area where the probability density is larger than 0), and St:t+T is a sub-sequence of S. St:t+T
contains every event in S that happens at a time between t and t + T. The intuitive meaning of F is the
expected mean feature vector of all posts generated by a user. Thus, by comparing the outputs of F in
the real world and the counterfactual world, we can see how engagement with a specific post changes
85

the average features, e.g., general sentiment scores or text embedding. With this function, we can simply
compute the individual treatment effect as:
IT E = FT (λ, T r ∪ X) − FT (λ, ∅ ∪ X) (6.7)
where ∅ is an empty set, FT (λ, ∅ ∪ X) is functional from the counterfactual world in which we assume that
the treatment is not applied (e.g., the misinformation post is not recommended or labeled as misinformation).
For a brief, we write FT (λ, ∅ ∪ X) as FT (λ, X) The overall impact of the treatment can be represented
with the average treatment effect:
AT E = E(X,T r)∼U [FT (λ, T r ∪ X) − FT (λ, X)] (6.8)
where U is the set of users who engaged with the treatment post.
6.2.3 Treatment Effect Calculation
In the above sections, we define the causal structure model and the treatment effect. However, the above
formulas are hard to compute. Therefore, in this subsection, we will derive a computable formulation of the
treatment effect. We will start from the following theorem:
Theorem 6.2.1. For a user u, if the intensity function λ(f, t|T r ∪ X) is known, then we have:
µ(t, t + T, λ1, T r ∪ X) = Z
sup(f)
df
Z t+T
t
λ(f, t|T r ∪ X)dt (6.9)
ϕ(t, t + T, λ1, T r ∪ X) = Z
sup(f)
fdf
Z t+T
t
λ(f, t|T r ∪ X)dt (6.10)
Proof. The first equation can be trivially proved by replacing the F in the definition of λ with the support
set. The second one can be proved with the Campbell’s Theorem [18]:
86

Lemma 6.2.2. Campbell’s Theorem (Campbell, 1909): For a point process S, and a measurable function
f : R
d → R
d
′
:
E[
X
x∈S
f(x)] = Z
Rd
f(x)E(N(x, x + dx)) (6.11)
Note that the above theorem is for all point process. For the specific scenario of temporal point process,
we also need to include the time range into consideration:
E[
X
x∈S[t1:t2]
f(x)] = Z
Rd
f(x)E(N(x, x + dx, t1, t2)) (6.12)
From the definition of ϕ, we know that the f(x) = x. Recall the definition of intensity function (Equation.
1 in the main content):
E(N(F, T1, T2)|HT1) = Z
F
df
Z T2
T1
λ(f, t|HT1)dt (6.13)
We can convert this integral to a derivative formula:
E(N(f, f + df, T1, T2)|HT1) = df
Z T2
T1
λ(f, t|HT1)dt (6.14)
With the above formula and Campbell’s theorem, we have:
ϕ(t, t + T, λ1, T r ∪ X) = Z
sup(f)
fdf
Z t+T
t
λ(f, t|T r ∪ X)dt (6.15)
The first equation can be trivially proved by replacing the F in Equation 6.13 with the support set. The
second one can be proved with Campbell’s Theorem [18]. The above formulas contain double integral,
87

which is inefficient to compute. To transform the double integral to a single integral, based on previous
work in spatial-temporal point process [25], we have:
λ(f, t|T r ∪ X) = λ(t|T r ∪ X)p(f|t, T r ∪ X) (6.16)
Thus, we can disentangle λ(f, t) and respectively model λ(ti) and p(f|t). More importantly, we can simply
model µ as:
µ(t, t + T, λ, T r ∪ X) = Z t+T
t
λ(t|T r ∪ X)dt Z
sup(f)
p(f|t, T r ∪ X)df =
Z t+T
t
λ(t|T r ∪ X)dt (6.17)
And with this formula, we have:
ϕ(t, t + T, λ, T r ∪ X) = Z
sup(f)
Z t+T
t
fλ(t|T r ∪ X)p(f|t, T r ∪ X)dfdt
=
Z t+T
t
λ(t|T r ∪ X)dt Z
sup(f)
fp(f|t, T r ∪ X)df
=
Z t+T
t
λ(t|T r ∪ X)E[f|t, T r ∪ X]dt
(6.18)
The above formulas contain only single integrals. Thus, they can be efficiently approximated with summation: R x2
x1
f(x)dx ≈
P(x2−x1)/∆x
i=0 f(x1 + i∆x)∆x
6.3 Neural Estimation of Treatment Effects
The above section constructs a causal framework that can measure the impact of a given social media post
based on the change of λ(t|T r ∪ X) and p(f|t, T r ∪ X). In this section, as shown in Figure 6.2, we will
further discuss how to estimate the impact with a neural temporal point process model.
88

6.3.1 Learning Conditional Intensity Function via Maximum Likelihood Estimation
The log-likelihood of an observed event (f, t) (no matter an engagement event or a generation event) can
be written as:
log p(f, t|T r ∪ X) = log λ(t|T r ∪ X) −
Z t
tn
λ(t|T r ∪ X)dt + log p(f|t, T r ∪ X) (6.19)
where tn is the timestamp of the last event in the set T r ∪ X. The above equation provides us with
a way to learn λ(t|T r ∪ X) and p(f|t, T r ∪ X) by maximizing the likelihood of each event given the
historical information (treatment and covariates). To enable the model to make correct predictions for both
λ(f, t|T r ∪ X) and λ(f, t|X), we construct two kinds of samples to train the functions:
Treatments
Covariates
Sampling
Post Post Post
Post Engage Post
Engage Engage Post
Engage Post Post Engage Post
Share Encoder (H)
Intensity
Function
Feature
Distribution
GRL: adversarial
training
Treatment
Predictor
Figure 6.2: The proposed neural model to estimate the impact of misinformation.
Samples with valid Treatment: If for a generating event (f
(g)
, t(g)
), its most recent previous event
is an engagement event (f
(e)
, t(e)
) (in other words, the user does not have other activity between the
89

engagement event and the generating event), then we can construct a sample (Y, T r ∪ X), where Y =
(f
(g)
, t(g)
), T r = (f
(e)
, t(e)
), and X is a sequence that contains all engagement events and generation
events before T r.
Samples without Treatment: If for a generating event (f
(g)
, t(g)
), its most recent previous event is
still a generating event (in other words, the user generate two original posts without engaging with other
posts), then we can construct a sample (Y, X), where Y = (f
(g)
, t(g)
) and X is a sequence that contains
all engagement events and generation events before Y . In this sample, between the last generation event in
X and Y , there is no interruption from treatment. Thus, it helps the model to learn λ(t|X) and p(f|t, X)
For a sample (Y, T r ∪ X) or (Y, X), we first use a shared encoder H(·) to project (T r ∪ X) (or X)
to a representation vector h = H(T r ∪ X) (or h = H(X) for the case where treatment is NULL). Then,
we model the intensity function and feature distribution as λ(t|h) and p(f|t, h). For λ(t|h), following
FullyNN, we use a multi-layer perceptron MLP(h, t) to model its integral R t
tn
λ(t|h)dt. The MLP’s partial
derivative with respect to t is λ(t|h).
To model p(f|t, h), a straightforward solution borrowed from generative deep learning is to apply a
neural network, i.e., the decoder, to transform a simple distribution, e.g., a Gaussian distribution whose
parameters are decided based on h and t, to a complicated distribution. The decoder can be trained via
different loss functions, like reconstruction error (variational autoencoder) and likelihood (normalizing
flow)†
[159, 25]. However, this method has an important drawback: its conditional expect E[f|t, h] does
not have a closed-form solution. To compute the expected, we can only apply sampling or approximation,
e.g., forwarding the expect of the simple distribution into the decoder‡
. To address the above challenge, we
propose to explicitly model p(f|t, h) with a mixture of Gaussian distributions:
p(f|t, h) = Xm
j=1
wj (t, h)g(
fi − uj (t, h)
σj (t, h)
) (6.20)
†GAN is not suitable because for fixed h and t we only have one sample, which can be easily memorized.
‡Because the decoder D is a non-linear function, E[D(x)] is usually different from D(E[x])
90

w(t, h) = softmax(MLPw(h, t)), σ(t, h) = exp(MLPσ(h, t)), µj (t, h) = MLP(j)
µ (h, t) (6.21)
where wj is the mixture weight, σj is a scalar, uj is a vector with the same dimension as f, and g(·)
is a standard multivariate gaussian distribution N (0, I) whose covariant matrix is an identical matrix.
Although the formula of each component is simple, their mixture has a theoretical guarantee of universal
approximation to all distributions [176]. Because the expect of a mixture distribution is the mixture of the
expects, we have a closed-form solution for E[f|t, h]:
Ef∼p(f|t,h)
[f|t, h] = Xm
j=1
wj (t, h)uj (t, h) (6.22)
6.3.2 Adversarial Balanced Neural Temporal Point Process
As discussed in the related work section, by maximizing the likelihood of the posts generated by the users,
we can train a neural network that predict λ(t|T r ∪ X) and p(f|t, T r ∪ X). However, previous works have
proved that if we do not balance the bias from the correlation between treatment and covariates, the model
will tend to give biased prediction and thus can not give precise estimation of the treatment effect. A crucial
bias in social media data is information cocoon: personalized recommendation systems will deliver user the
contents that they are interested in. For example, it will deliver more anti-vaccine posts to anti-vaccine
users because they are more likely to be interested in those contents. As a result, the anti-vaccine users will
engage more with anti-vaccine posts.
To address the above issue, following previous works in neural counterfactual prediction, we apply
domain adversarial training to learn a representation h that is invariant to such a bias[14]. More specifically,
we hope to learn a encoder H such that for any two users with different history X1 and X2, p(T r|H(X1)) =
p(T r|H(X2)) for the same T r. In other words, in the representation space, the probability that the two
users interact with the same post at the same time should be same, which is the same as psychology
91

experiments that divide the experimental and controlled groups randomly. To achieve this object, we apply
adversarial training to remove the information about future treatment from the representation of covariates.
More specifically, we additionally train a treatment predictor pˆ(T r|H(X)) by modeling λtr(t
(e)
|h) and
ptr(f
(e)
|t, h) with the encoding h of the historical covariates X. However, between the treatment predictor
and the encoder, we insert a gradient reversal layer (GRL)[54, 123, 14] to reverse the sign of the gradient.
Thus, when we optimize the treatment predictor to maximize the likelihood of the observed treatment
T r = (f
(e)
, t(e)
), the GRL will make the encoder to minimize the likelihood. This process leads to the
following min-max game:
minHmaxpˆET r,X∼p(T r,X)
log ˆp(T r|H(X)) (6.23)
The following theorem provide theoretic guarantee that the above adversarial training reduce the bias
introduced by treatment-covariate correlation (e.g. recommendation system and personal interest):
Theorem 6.3.1. Given the following min-max game:
minHmaxpˆET r,X∼p(T r,X)
log ˆp(T r|H(X)) (6.24)
the min gamer’s Nash balanced solution H∗
, ensures for any X1, X2, the following equation holds:
p(T r|H∗
(X1)) = p(T r|H∗
(X2)) (6.25)
where p denote the ground-truth conditional distribution of treatment given encoding.
92

Proof. For the objective of the max gamer, we have:
ET r,X∼p(T r,X)
log ˆp(T r|H(X)) =ET r,H(X)∼p(T r,H(X)) log ˆp(T r|H(X))
=EH(X)∼p(H(X))ET r∼p(T r|H(X)) log ˆp(T r|H(X))
=EH(X)∼p(H(X)) − KL(p(T r|H(X)), pˆ(T r|H(X))
− Q(p(T r|H(X)))
(6.26)
where Q(p(T r|H(X))) is the entropy of p(T r|H(X)). Note that for a fixed H, Q(p(T r|H(X))) is a
constant. Thus, the max objective is the same as minimizing the KL-divergence between p(T r|H(X)) and
pˆ(T r|H(X) for every X:
maxpˆET r,X∼p(T r,X)
log ˆp(T r|H(X)) ⇐⇒ minpˆEH(X)∼p(H(X))KL(p(T r|H(X)), pˆ(T r|H(X)) (6.27)
It is known that the KL-divergence is minimized if and only if pˆ(T r|H(X) perfectly fit p(T r|H(X)) at each
point. Thus, the balance solution of the max gamer is to learn pˆ(T r|H(X)) = p(T r|H(X)). Meanwhile, if
there are two X1 and X2 such that p(T r|H(X1)) ̸= p(T r|H(X2)), then the min gamer can always update
the H by swapping H(X1) and H(X2) while maintaining other points the same. This is doable when H is
a universal function approximator. As a result, the KL-divergence will increase, which violates the definition
of Nash balance. Therefore, at the balance point for any X1 and X2, p(T r|H(X1)) = p(T r|H(X2))
6.4 Experiments
On real-world social media platforms, the ground truth causal effects of user engagement with posts, no
matter whether misinformation posts or information posts are unknown. To address the unknown ground
truth causal effect, previous works of causality analysis evaluate their models on synthetic datasets. In this
paper, following previous works, we evaluate the performance of our model and compare it with baselines
93

Table 6.1: Estimation Error to the ground-truth ITE
Method Accuracy ↑ RAE ↓ RRSE ↓ Decoder Inference Time
FullyNN 73.0% 0.865 0.901 7.13ms
CNTPP-VAE (Approximation) 85.9% 0.279 0.503 4.05ms
CNTPP-VAE (Sampling) 87.8% 0.237 0.454 29.34ms
CNTPP(Ours) 88.0% 0.234 0.448 7.12ms
on synthetic datasets. Then we apply our proposed method to evaluate the impact of misinformation on
real-world social media data about the COVID-19 vaccine collected from Twitter§
.
6.4.1 Experiments on Synthetic Data
To simulate the real-world social media, we generate 15000 users and 120 posts of news. Each user i is
represented with a hidden vector ui
, which corresponds to the status of a social media user. Each piece
of news n has two randomly generated feature vectors: a topic vector vtopic(n) and an inherent influence
vector vin(n). Each user has two kinds of activities: (1) engaging with one of the 120 news posts and (2)
posting a post with original content. The chance that a user engages with a post is decided by vin(n) and
ui
, simulating information cocoons. Engagement event with news post n will change the hidden status u
of the user. The scale and direction of the change are jointly decided by the current user status, the topic
vector, and the inherent influence vector. For each user, the engagement events and the posting events
are modeled through two Hawkes processes, respectively. Both Hawkes processes are influenced by user
status u. Also, the feature of each posting event f is drawn from a distribution P(f|u, t) characterized by
a random parameterized multi-layer perceptron (MLP) taking (u, t) and random noises as input and output
f. Thus, the engagement with the news post will have causal effects on the two processes. Since we have
all the parameters of the model, we can calculate the ground-truth ITE defined in Eq. 6.7 for the synthetic
dataset.
§Code and data will be provided in https://github.com/yizhouzhang97/CNTPP
94

Baselines: To the best of our knowledge, the causality effect on temporal user behavior from misinformation has not been explored by previous works. Thus, we lack well-established baselines for this specific
task. To address this issue, we select some baselines from previous works on temporal point process and
temporal causal inference and extend them to adapt our setting. FullyNN [111] is a non-causal neural
temporal point process that predicts the user’s future behaviors without considering causal effects. We
selected it because it has the same neural architecture as our model. It can also be regarded as our model’s
variant w/o adversarial balancing. Neural-CIP¶
is an extension of CIP [56], which aims at discovering
the causal effect of event pairs in temporal point process. We further compare our model with an ablation
variant: CNTPP-VAE. It replaces our GMM-based decoder with a Variational Auto-Encoder [78]. Since
VAE does not have a closed-form solution of feature expect, we report the results by applying sampling and
approximation separately.
In this work, we will evaluate the proposed model in two aspects:
Table 6.2: Causal Effect Inference
Method MatDis↓ LinCor↑
Neural-CIP 0.90 0.04
FullyNN 0.93 0.236
CNTPP-VAE (Approximation) 0.84 0.303
CNTPP-VAE (Sampling) 0.76 0.287
CNTPP (Ours) 0.77 0.310
ITE Estimation: we will evaluate the
model by comparing the ITE estimated
by the model and the ground-truth ITE.
We report three metrics: Accuracy (the
model needs to correctly predict whether
the engagement increases or decreases
each dimension of the expected average
feature), Relative Absolute Error (RAE),
and Relative Root Square Error (RRSE).
In addition, we also report the inference time of our model to reflect our model’s efficiency.
¶Because the authors did not opensource the code of the original CIP and one important precious work that is crucial for CIP,
we implement a version of CIP that applies a neural network rather than a graphical causal model.
95

As shown in Table 6.1, our model establishes new state-of-the-art on all three quantitative metrics.
This means that our model can fully utilize the causal information from the multivariates point process
for unbiased treatment effect estimation. In particular, the model with causal analysis outperforms the
model with direct neural network prediction (FullyNN), which leads to results that are not causally related.
Simultaneously, our model outperforms all baselines in all three metrics of estimation precision. Although
CRN-VAE incorporated with the sampling method can achieve performance very close to us, it spends
substantially longer inference time.
0.050
0.025
0.000
0.025
0.050
S
u
bj
e
c
tivit
y
S
c
o
r
e
Information
Misinformation
Sentiment Score
0.01 0.00 0.01
(a) Impact on sentiments and subjectivity.
0.02 0.01 0.00 0.01
PCA Vector1
0.2
0.0
0.2
Information
Misinformation
P
C
A
V
e
c
t
o
r
2
(b) Impact on text representation.
Figure 6.3: Analysis on real world social media data.
Causal Effect Inference: CIP defines the treatment effect in a way different from our model. Thus, the
ITE estimation experiment is not fair for it. For a fair comparison and to prove that our model can achieve
unbiased estimation further, we use all the models to predict the ATE of each news post. Then, we evaluate
the correlation between the learned ATEs and the ground-truth average change in the news post to all users’
hidden statuses. The more correlated the learned ATE is, the better it reflects the inherent causal effect of
the engagement on users. To evaluate the correlation, we apply the following two metrics: MatDis and
LinCor. MatDis evaluates the similarity between the ATE-Distance matrix and Hidden-Status-Distance
matrix. LinCor evaluates the linear correlation between the learned ATE and the ground-truth average
hidden status change. From Table 6.2, the ATE of our model best reflects the ground-truth average change
of the news post on the simulated data for both two evaluation metrics. This suggests that our model has
96

Table 6.3: Comparison of (normalized) Average Sum of Distances with different methods on the real-world
dataset. This metric reflect how well the a group of data points is clustered.
Methods ASD↓ ASDin ↓ ASDmis ↓
Event Feature 0.123 0.123 0.122
FullyNN 0.073 0.069 0.072
CNTPP (Ours) 0.045 0.042 0.044
the potential to discover the influence of misinformation on social media users’ hidden status, e.g., interests
and ideas.
6.4.2 Experiments on Real World Data
In this section, we apply our proposed model to the Twitter dataset to estimate the impact of misinformation
on social media scenarios. We apply the data set collected in [182, 141], including a total of 16,9008 tweets
with labels from 24,192 users over a 5-month period from 2020/12/09 to 2021/04/24. Notably, we focus on
understanding how the tweets that users retweeted influence their behavior of posting original tweets.
To collect the Twitter dataset related to COVID-19 vaccines, we use the tracked keywords, including
vaccine, Pfizer, BioNTech, Moderna, Janssen, AstraZeneca, and Sinopharm to filter all tweets via the
streaming Twitter API, which returns a 1% sample of all tweets. We collected the tweets just before
Pfizer-BioNTech and Moderna were approved by the FDA for Emergency Use Authorization (EUA) from
December 9, 2020 - April 24, 2021. After that, we only used the tweets with labels of ’MisInformation’ and
’Information’, including a total of 169,008 tweets from 24,192 users.
For each post, we consider two kinds of feature f: (1) text representation and (2) sentiment and
subjectivity score.
For the impact on text representation, we use the pre-trained BERT model [41] to extract the representation vectors of each tweet’s content with 768 dimensions. After that, we use PCA tools to reduce into two
dimensions and use the new vectors as the covariates in the model. The training setting is the same with
the impact on sentiments and subjectivity experiments.
97

For the impact on sentiments and subjectivity experiments, we first use TextBlob sentiment analyze
tool [96, 124] to generate the continuous sentiment and subjectivity score of the content user posts, where
the former reflects the emotional charge of a statement and the latter reflects the degree of objectivity. After
that, we use the tweets’ timestamp to calculate the time interval and use the sentiment and subjectivity
score, including whether the news is positive (1) or not (0), as the covariates of our causal structure. We
split the training set, validation set, and test set according to the ratio of 80%/10%/10% to train the proposed
model. Specifically, we adopt the Adam optimizer with a learning rate of 1e − 5, and the epoch is set to be
500 with a batch size of 128. In addition, we add the dropout with 0.1 rate.
6.4.2.1 Identifiability between misinformation and information in influencing people’s narratives
For the desired outcome, we analyze the distinguishability between "retweeting fake news" and "retweeting
true news" events. More specifically, for each retweeting event, we use its ITE estimated with our model as
a feature (dimension reduced via PCA [102]) and whether the content is information or misinformation
as a label. As shown in Figure 6.3b, we can verify the identifiability of our proposed method as the
treatment effects of the two types of news are substantially different. This discovery not only supports that
misinformation and information influence people’s behavior in different ways but also provides us with a
new paradigm for detecting fake news.
We also calculate Normalized Averaged Sum of Distances (NASD) for the information cluster,
misinformation cluster, and their joint set. More specifically, we first normalize the scale range of each
embedding dimension (ITE vector or raw input) to be the same for different methods to ensure that the
scale embedding spaces of different methods are comparable. Then, we calculate their Averaged Sum of
Distances (ASD), which is defined as ASD =
Pk
i=0
P
p∈Ci
(p − mi)/n, where Ci
is the set of ITE results
(or event feature) and i = 1 means the information results, i = 0 means the misinformation results, p
stands for an instance in Ci
, mi
is the center of all instance representations in Ci
, n is the total number of
98

instance. The lower these metrics are, the better that information and misinformation are distinguished.
The comparison of our model against FullyNN and event features on this metric is shown in Table 6.3. As
we can see, the ITE learned by our model can identify information and misinformation better than the
baselines.
We also evaluate FullyNN (a baseline) on the real-world dataset. Its results are visualized in Figure 6.4a
and 6.4b. As we can see, it can not acquire recognizable differences between information and misinformation.
We further visualize the raw event features (BERT embedding with dimension reduction of PCA, not
finetuned on fake news detection) in Figure 6.4c. As we can see, it is also hard for a pre-trained language
model to distinguish information and misinformation in an unsupervised manner.
(a) Impact on sentiment
scores, estimated by FullyNN.
(b) Impact on text representation, estimated by FullyNN.
(c) Visualization of raw event
feature.
Figure 6.4: Visualization of the results from baselines on real world.
6.4.2.2 Misinformation is hurting people’s subjective emotion related to COVID vaccine
To understand the influence of misinformation in a more intuitive way, we analyze the impact of retweeting
events on users’ average sentiment score and subjectivity score in the future. The more subjectivity the
content gets, the more personal opinions rather than factual information the text contains. Then, we use
the proposed model to generate the estimated ITE for each retweeting event and plot the results of fake
news and real news. As shown in Figure 6.3a (x-axis for sentiment score and y-axis for subjectivity score),
we find that both information and misinformation do not substantially influence the users’ subjectivity.
However, information tends to make people optimistic about vaccines (true news increases the sentiment
99

score), while fake news tends to make people feel negative about vaccines. This discovery strongly supports
the hypothesis that misinformation is hurting people’s subjective emotions toward COVID-19 vaccines and
suggests that misinformation could be causally responsible for vaccine hesitancy.
100

Chapter 7
Making LLM Discerning to Misinformation with Divide-and-Conquer
Prompt
Large language models (LLM) based on the Transformer architecture have led to major breakthroughs in
natural language processing and other related fields in artificial intelligence[16, 122, 151]. State-of-theart general-purpose language models have demonstrated remarkable advancements in various domains,
including question answering, graph learning, reading comprehension, text generation, and machine
translation [27, 148, 62, 100, 191]. These developments pave the way towards general-purpose problem
solvers [17].
However, as pointed out in [161], significant challenges arise when scale-up models are applied to tasks
involved with deceptive content and misinformation, particularly when they are hidden within a long
context. A series of theoretic works attribute this challenge to Parallelism Tradeoff [107], a fundamental
limitation of Transformers. Specifically, unlike Recurrent Neural Network, whose computational depth
is linear to the input sequence length (i.e., the depth is O(n), where n is the input sequence length),
Transformer does not contain any recurrent structure. Such design, while achieving superior parallelizability
than RNN, makes Transformers suffer from limited expressive power. Merrill and Sabharwal proved that
the expressive power of fixed-depth log-precision Transformer, which is very close to the most commonly
applied Transformer architecture for LLMs, is bounded by constant-depth logspace-uniform threshold
101

#Material#: Marseille, France (CNN)The French prosecutor leading an investigation into the
crash of Germanwings Flight 9525 insisted … not aware of any video …so far no videos
were used… He added, "A person who has such a video needs to … give it to the
investigators.” … Jean-Marc Menichini… that the reports were "completely wrong" and …
Task: Verify the truthfulness of a summary.
#Summary#: A video showing the final moments of Germanwings Flight 9525 has been
recovered by investigators from the wreckage site. Marseille prosecutor Brice Robin urged
anyone who might have more footage to turn it over immediately. Andreas Lubitz, the copilot accused of deliberately crashing the plane, had a history of severe depression and
suicidal tendencies.
Q: You are given the above material and a summary. Please answer me: Does the
summary contain any claim that is contradicted with the material?
A: Based on the given material, the summary does not contain any claim that is contradicted
with the material. The material confirms the existence of a video showing the final moments
of Germanwings Flight 9525, as well as the history of severe depression and suicidal
tendencies of the co-pilot, Andreas Lubitz.
Example of Entangled Problem Solving
Sub-task 1:
Q: Does the claim 1 contradict with the material? #Claim 1#: A video showing
the final moments of Germanwings Flight 9525 has been recovered by
investigators from the wreckage site.
A: Yes, the claim is contradicted by the material. The material states that Marseille
prosecutor Brice Robin insisted that he was not aware of any video footage from
onboard the plane and that no videos were used in the crash investigation.
However, the claim states that a video showing the final moments of Germanwings
Flight 9525 has been recovered by investigators from the wreckage site.
Sub-task 2:
Q: Does the claim 2 contradict with the material? #Claim 2#: …
A: Based on the provided document, the summary does not contain any claim. …
Sub-task 3:
Q: Does the claim 3 contradict with the material? #Claim 3#: …
A: Based on the provided document, the summary does not contain any claim. …
Example of Divide-and-Conquer Problem Solving
Figure 7.1: An illustrative example of hallucination detection with entangled problem solving (i.e., directly
forward all inputs into the LLM) and divide-and-conquer problem solving (i.e., divide the problem inputs to
parallel sub-tasks and tackle them parallelly). The sentence marked with red back font in the material is
the evidence that contradicts the first claim in the summary (marked with red font).
circuits. Thus, they fail to accurately tackle the tasks requiring long solution paths. Consequently, when
deceptive content and misinformation are hidden in a long context, they generally fail to discover them.
To address this challenge, carefully designed prompting strategies have been developed to tackle tasks
that require stronger expressive power [45]. A series of works focus on prompting the LLM to output the
intermediate steps that derive the final answer in an autoregressive manner, such as Chain-of-Thoughts
(CoT) [161, 160, 173, 184, 26]. Theoretically, these prompting strategies convert the role of Transformer from
the complete problem solver to a sub-problem solver in a dynamic programming or tree searching algorithm
[106]. In this way, these prompting strategies expand the expressive power of the LLMs and successfully
improve the reasoning and searching ability of LLMs [45], leading to better performance in misinformation
detection [23]. However, in most of these aforementioned prompting strategies, the processes of sub-problem
decomposition, sub-problem resolution, and sub-resolution assembly are intertwined together during
the autoregressive token decoding. Such entangled design makes the sub-problem generation process
lack control from the human or agent side and susceptible to disruptions of the task resolution process.
As a result, when the task requires a large number of repetitive tasks (e.g., article-level misinformation
verification), LLM is prone to intermediate errors, such as missing some inputs or generating wrong
sub-tasks, leading to problematic answers [5]. This phenomenon is especially serious when the task input
102

contains deceptive information or contents that could trigger hallucination [23, 90]. An illustrative example
of such an entangling issue is presented in Fig. 7.1. To alleviate this issue, some program-guided prompting
strategies such as Least-to-Most [184] (LtM) and Decomposed Prompting [76] propose to disentangle
sub-task generation and resolution. However, they implement the task decomposer through multi-round
conversation with an LLM to sequentially raise and solve the sub-problems in an alternate manner. When
tackling deceptive text and misinformation, such intertwined process often guides the LLM to sequentially
tackle the corpus and follow the context’s flow, making LLM prone to deception.
In fact, human brains also suffer from similar hallucination issues [94], especially when the tasks are
too hard or too complex. For example, when reviewing a long academic paper, some reviewers produce
low-quality reviews [57, 149, 33] containing hallucination-like intermediate errors, such as pointing out
some ‘missing baselines’ that have already been sufficiently discussed by authors. To avoid such ridiculous
mistakes, experienced reviewers usually think slowly [74] to follow a Divide-and-Conquer paradigm
to handle this task. Specifically, they decompose the paper review as examinations of multiple central
opinions. Then they retrieve the supporting corpus to verify them respectively. Similar strategies are also
applied in decoding to specifically improve LLM’s capacity in math [86] and automated evaluation [36].
Inspired by the aforementioned human experience and empirical finding shown in Fig. 7.1, in this paper,
we explore guiding LLM with a Divide-and-Conquer program to unlock the LLM’s ability to handle tasks
with repetitive sub-tasks. This strategy breaks down the whole task resolution process into three distinct
sub-processes: task decomposition, sub-task resolution, and solution merge. The task decomposition process
will prompt the model to separate the whole task into multiple parallelly solvable sub-tasks (i.e., solving
one sub-tasks does not require the result of other sub-tasks) and list them explicitly recursively. After
that, the sub-task resolution process prompts the LLM to output the answers to all sub-tasks. Finally, the
solution-merge process assembles the solutions recursively along the decomposition path. The above subprocesses follow a key principle that every upstream sub-process (e.g., sub-task resolution) only forwards
103

the final answer to its downstream sub-processes (e.g., solution merge) and does not forward its input and
intermediate steps. We denote this principle as disentangled-sub-process principle. An illustrative figure
that explains the difference between our method and previous works is provided in Fig. 7.2
However, unlike Chain-of-Thoughts, whose advance in expressive power is supported by theoretic
analysis [45], the performance boost from the Divide-and-Conquer paradigm lacks rigorous theoretic
support. As a result, we are not aware of the conditions under which the Divide-and-Conquer paradigm
can acquire more accurate answers. To tackle this challenge, in this chapter, we also aim to understand the
utility of DaC prompting. More specifically, we attempt to answer the following two research questions:
1. RQ1: Compared to straightforward instructional prompting, does DaC have theoretically
guaranteed advantages similar to CoT and its variants?
2. RQ2: Compared CoT and its variants, what utility and limitations does DaC have?
To answer these questions, we first provide a theoretic paradigm that can help us analyze how divide-andconquer strategy expand the expressive power of fixed-depth log-precision Transformer on a given task. In
this way, we provide a framework that can provide theoretic guarantee to DaC paradigm in various tasks.
In this way, we present some conditions under which DaC have advantages compared to other prompting
strategies. We then empirically evaluate DaC prompting and representative baselines on tasks that satisfy
the proposed conditions and are challenging to existing prompting strategies even on state-of-the-art
LLMs: Hallucination Detection AND Article-level Fact Verification [30, 90, 156, 65, 167]. These tasks
require tackling long text as well as containing deceptive content (e.g., hallucination detection and fact
verification), making existing methods like Chain-of-Thought prompting prone to intermediate errors.
Our experimental results show that the proposed method outperforms the baselines on both tasks, which
supports our theoretic analysis.
104

Figure 7.2: The comparison between DaC and the existing methods for prompting. The ellipse marks
represent sub-tasks, the right-angled rectangles represent sub-task solutions, and the rounded rectangles
represent intermediate steps that entangle sub-tasks and sub-solutions. The different shades in Tree of
Thoughts (subfigure D) indicate the rates of different search directions. In CoT (Chain-of-Thoughts), CoT-SC,
and ToT, the Large Language Models must simultaneously generate and resolve sub-tasks. Least-to-Most
(also Decomposed Prompting) disentangles sub-task generation and resolution. However, its sub-task
resolution and resolution assembly process are intertwined as it sequentially attaches new sub-tasks to
the previous resolution. Different from them, DaC totally disentangles the sub-task generation, sub-task
resolution, and resolution assembly process.
7.1 Preliminary
7.1.1 Expressive Power of Transformer
As discussed in previous works [107, 45], the expressive power of fixed-length log-precision transformers,
which are widely applied in modern Pre-trained Large Language Models, is actually much more limited than
people’s expects. Merrill and Sabharwal give a theoretic proof that the expressive power of fixed-length
log-precision transformers is upper-bounded with TC0
. Feng et al. further extend their analysis to explain
that a lot of common problems exceed the expressive power of fixed-length log-precision transformers.
Such results explain why the powerful LLM may make some ridiculous mistakes and how CoT improves
the performance.
105

7.1.2 Prompting Strategies of LLM
In this subsection, we introduce the existing prompting and discuss their limitations and drawbacks.
Following the notations in [173], we denote the Large Language Models with parameter θ as pθ and use
lower case letters x, y, z to denote input sequence, result, and intermediate steps, respectively.
Input-Output (IO) Prompting is the standard prompting strategy that attach input x with instructions
and/or few-shot in-context-learning examples to aqcuaire a prompt, denoted as prompt(x) [173]. The LLM
takes prompt(x) as input and predict result, i.e. y ∼ pθ(y|prompt(x)).
Chain-of-Thought (CoT) Prompting [161] aims at simulating a human’s thinking process that
handles complicated tasks (e.g., combinational reasoning and mathematical calculation) in a step-by-step
manner. More specifically, the LLM is guided to output a series of intermediate steps z1, z2, ..., zn (also
known as thoughts) autoregressively, i.e. zi ∼ pθ(zi
|prompt(x), z1, ..., zi−1). Then the LLM output the
prediction of result y based on the thoughts, i.e. y ∼ pθ(zi
|prompt(x), z1, ..., zn).
Exploration-of-Thought (EoT) Prompting and Program-guided Prompting are two variants
of CoT. EoT includes a series of CoT’s variants, such as Self-consistency with CoT (CoT-SC) prompting
[160] and Tree-of-Thoughts (ToT) prompting [173], which aim at addressing the limitation of CoT in
exploration. Their common central idea is to generate multiple chains of thought through sampling or
proposing prompting and then ensemble them to acquire a final prediction. Program-guided Prompting
aims at controlling the LLM’s generation process with symbolic programs or pre-defined procedure [190,
73, 184, 76, 34, 55]. Among them, the Least-to-Most (LtM) Prompting [184] and Decomposed Prompting
[76] are close to this work. They are the earliest attempts that explicitly prompt the LLM to decompose
the task as a series of sub-tasks and sequentially tackle them. LtM prompts an LLM to iteratively raise
sub-tasks and sequentially solve them to acquire the final resolution. Decomposed Prompting can regarded
as an upgraded version of LtM. It introduces special notations into the prompt to represent program states
106

and thus can call itself (i.e., recursion) or other modules (i.e., hierarchical decomposition), endowing it
with stronger expressive power. Such design increased the compositional generalization ability of LLMs in
different areas, such as symbolic manipulation and multi-hop QA [76].
Sub-task Sub-Solution
Input
Output
…
(A). Least to Most
Input
Output
A
C
…
(B). Decomposed
Prompting (DeP)
B
CoT
LtM
DeP
Figure 7.3: Comparison of Least-to-Most (LtM) Prompting and Decomposed Prompting (DeP).
Here, we will provide a more detailed comparison of Least-to-Most (LtM) Prompting [184]
and Decomposed Prompting [76], which is
shown in Fig. 7.3. Decomposed Prompting can
regarded as an upgraded version of LtM. It introduces special notations into the prompt to represent program states so that when sequentially
tackling the sub-tasks, it can call heterogeneous
modules to tackle them. Such design enables
the LLM to call external programs (e.g., retrieval
documents on Wikipedia and program-based calculator) and/or itself (i.e., recursion). Such design endows
it stronger expressive power and increases the compositional generalization ability of LLMs in different
areas, such as symbolic manipulation and multi-hop QA [76]. Also, it endows LLM with the ability to do
open-domain QA by retrieving from the external knowledge base.
The aforementioned CoT and EoT families incorporate LLM with stronger expressive power than IO
prompting. However, a critical issue for them is that they could miss or ignore some important intermediate
steps or contents [95]. This problem is even worse when we are handling tasks involved with long input (e.g.
long documents and large numbers). Typical examples include large numbers of arithmetic calculations
and fact verification in long documents. Compared to them, Least-to-Most prompting and Decomposed
Prompting introduces explicit task decomposition to enumerate sub-tasks. However, their task decomposers
are based on multi-round conversation or question-answering, which navigate the LLM through the
107

deceptive content’s flow sequentially and propagate the hallucination/deception in the contexts [43, 169],
leading to decreased performance.
7.2 Divide-and-Conquer Prompting
To avoid the task decomposition and task resolution from interweaving and interrupting each other, we
propose to guide LLM with a Divide-and-Conquer (DaC) program that consists of three distinct stages: task
decomposition stage, sub-task resolution stage, and solution merge stage. In the task decomposition stage,
the LLM is prompted to explicitly decompose the task as a series of parallel homogeneous sub-tasks with
smaller problem sizes (e.g., divide a long paragraph into sentences). Such design avoids the multi-round
conversation or question-answering in LtM and Decomposed Prompting, making the model less prone
to deception. After that, in the sub-task resolution stage, the LLM is prompted to provide the solutions
for every sub-task. Finally, in the solution merge stage, the LLM is prompted to assemble the solutions of
subtasks and acquire the final answer. In this process, all three stages are isolated to avoid interruption.
They are all guided by a program rather than an LLM to avoid hallucination or deception from the input
context. To tackle tasks of different sizes, we propose two variants: Single-Level DaC Solver and Multi-Level
DaC Solver.
Algorithm 4 Single-Level Divide-and-Conquer Solver T(S, a, t, L, f)
Require: Input Sequence S, Prompt m (for solution merge), Prompt t (for sub-task tackling), Prompt d
(for task decomposition), LLM L
Ensure: Results of the task on input sequence S
1: {S1, S2, ..., Sk} ← L(d, S)
2: Result ← ∅
3: for i = 1, 2, ..., k do
4: Result ← Result +[SEP] + L(t, Si)
5: end for
6: Return L(m, Result)
Single-level Divide-and-Conquer Solver decomposes the task in one call to the LLM, which expands the
original task as a tree of one level. The algorithm is presented in the Alg. 4. The advantage of this variant is
108

its simplicity and efficiency. However, when the original input is too long, a single-level Divide-and-Conquer
Solver may acquire sub-tasks with large problem sizes that will still trigger intermediate errors. In such a
case, following [76], we can recursively expand the task as a multi-level tree. More specifically, we repeat
the aforementioned steps to divide the sub-tasks hierarchically further until they are easy enough for the
LLM to handle. This can be done through a recursion program as presented in Alg. 5. More discussions
on the proposed method’s appliance scope, including its comparison with other prompting strategies and
limitations, can be found in 7.4.3
Algorithm 5 Multi-Level Divide-and-Conquer Solver Recursion T(S, m, t, d, f, n, L)
Require: Input Sequence S, Problem Size Metric Function f(·) (a function that measure the problem
size), hyper-parameter w, Prompt m (for merge), Prompt t (for sub-task tackling), Prompt d (for task
decomposition), Large Language Model L
Ensure: Results of the task on input sequence S
1: S1, S2, ..., Sk ← L(d, S)
2: Result ← ∅
3: for i = 1, 2, ..., k do
4: if f(Si) > w then
5: Result ← Result +[SEP] + T(Si
, m, t, d, f, w, L)
6: else
7: Result ← Result +[SEP] + L(t, Si)
8: end if
9: end for
10: Return L(m, Result)
7.3 Main Theoretic Results
In this section, we provide theoretic analysis to the utility and limitations of the Divide-and-Conquer
prompting. In the first subsection, we provide a comparison of IO prompting (common fixed-length
instructional prompting) and DaC prompting in expressive power perspective. This part answers the first
research question: the expressive power of IO prompting is a subset of DaC prompting. In the second
subsection, we provide a comparison between Chain-of-Thoughts and DaC prompting in expressive power.
109

Our comparison suggests that, although the expressive power of DaC prompting is a subset of Chain-ofThoughts, for tasks satisfying specific conditions, DaC prompting can solve the problem with lower average
context window length when decoding the tokens. Such property has empirically proved to be helpful in
reducing intermediate errors and thus boosting performance.
7.3.1 Divide-and-Conquer vs. IO Prompting
We show that the expressive power of Divide-and-Conquer is stronger than IO Prompting:
Theorem 7.3.1. We denote the set of problems that a fixed-precision transformer with fixed-length IO
prompting can tackle as P(IO). Similarly, we denote the set of problems that a fixed-precision transformer
with DaC prompting can tackle as P(DaC). Then we have the following results:
P(IO) ⊂ TC0 ⊆ NC1 ⊆ P(DaC) (7.1)
Proof Sketch: The conclusion that P(IO) ⊂ TC0
is a corollary of the main results in [31]. In this
paper, we mainly focus on proving NC1 ⊆ P(DaC). Specifically, we exploit the 2-color Binary Subtree
Isomorphism (2-BSI) problem for the proof. In [70], 2-BSI problem is proved to be an NC1
-complete problem.
Its definition is:
Definition 1. 2-color Binary Subtree Isomorphism problem is that, given a pattern 2-color binary tree
tp and a base 2-color binary tree tb, a solver is required to judge whether the pattern tree is isomorphic to a
sub-tree of tb
In [70], the authors pointed out that the encoding of the problem will influence the hardness of the
problem. In this paper, we focus on the pointer list encoding of 2-BSI.
110

Theorem 7.3.2. There exists a log-precision transformer with fixed depth L and hidden dimension d that can
solve the 2-BSI of any size with fixed-length prompt m (for merge), t (for sub-task tackling) and d (for task
decomposition).
Proof Sketch: Here, we first give a brief flow of the proof. To prove this theorem, we first show an
algorithm that can solve the problem with a divide-and-conquer strategy. Then we prove that there exists a
log-precision transformer with fixed depth L and hidden dimension d that can express the modules in the
algorithms with different but fixed-length prompts. In this way, we can prove the theorem.
Now we give the detailed proof of the theorem. Before providing the proof, we first formally define
how to organize the inputs (i.e., two 2-color trees) as a sequence. We assume that we acquire two trees tp
of size n and tb of size m. They are organized as two sequences of nodes in a random order. Each node
has three variables: color, left child index, and right child index. If any child is null, then the index is filled
with 0. Then we can organize them as two sequences Xp ∈ R
n×3
and Xb ∈ R
n
′×3
, where each item in the
sequence is a vector of 3 dimensions. The first dimension is the index of the left child, the second dimension
is the index of the right child, and the third dimension is the color indicator (0 or 1). In addition, we have a
root vector r with three dimensions. The first dimension is the index of the root node of tp (i.e., pointing to
the root node of tp), and the second is the index of the root node of tb (i.e., pointing to the root node of tb).
The third dimension of r is filled with 0 to make it have the same dimension as the items in Xp and Xb.
This expression of trees is also called pointer list encoding according to [70]. Note that in the following
proof, we assume that all indices start from 1. Thus, 0 is regarded as a NULL pointer.
Following the proof sketch, we first provide the following divide-and-conquer algorithm that can solve
the above problem:
The algorithm described above is a typical divide-and-conquer algorithm for solving rooted tree
isomorphism. Its justification can be found in many textbooks introducing algorithms, such as Introduction
111

Algorithm 6 Recursion Divide-and-Conquer Algorithm for 2-BSI BSI(r, Xp, Xb, m, t, d, f, w)
Require: Inputs r, Xp, Xb, problem size metric function f(·), hyper-parameter w, merge function m,
sub-task tackling function t, task decomposition function d
Ensure: A 0-1 indicator vector v: if there exists a subtree with node i as root that is isomorphic with
pattern tree tp defined with inputs r, Xp, Xb, then the v[i] is 1. Otherwise, v[i] is 0.
1: rl
, rr ← d(r, Xp, Xb)
2: for i ∈ {l, r} do
3: if f(ri
, Xp, Xb) > w then
4: vi ← BSI(ri
, Xp, Xb, m, t, d, f, w)
5: else
6: vi ← t(ri
, Xp, Xb)
7: end if
8: end for
9: Return m(r, Xp, Xb, vl
, vr)
Algorithm 7 Implementation of d(r, Xp, Xb) when the depth of the tree indicated by r is not longer than 2
Require: Inputs r ∈ R
3
, Xp ∈ R
n×3
, Xb ∈ R
n
′×3
Ensure: A 0-1 indicator vector v: if there exists a subtree with node i as root that is isomorphic with
pattern tree tp defined with inputs r, Xp, Xb, then the v[i] is 1. Otherwise, v[i] is 0.
1: rl ←< Xp[r[1], 2], r[2], r[3] >
2: rr ←< Xp[r[1], 3], r[2], r[3] >
3: Return rl
, rr
Algorithm 8 Implementation of t(r, Xp, Xb) when the depth of the tree indicated by r is not longer than 2
Require: Inputs r ∈ R
3
, Xp ∈ R
n×3
, Xb ∈ R
n
′×3
Ensure: A 0-1 indicator vector v: if there exists a subtree with node i as root that is isomorphic with
pattern tree tp defined with inputs r, Xp, Xb, then the v[i] is 1. Otherwise, v[i] is 0.
1: Initialize v as all q vector with a length of n
′
2: if r[1] == 0 then
3: Return v
4: end if
5: for i ∈ {1, 2, ..., m} do
6: if Xb[i, 3]! = Xp[r[1], 3] then
7: v[i] ← 0
8: end if
9: end for
10: Return v
to Algorithms [32]. Here, we provide the detailed definition and implementation of problem size metric f(·),
hyper-parameter w, merge function m(), sub-task tackling function t(·), task decomposition function d(·):
112

Algorithm 9 Implementation of m(r, Xp, Xb, vl
, vr)
Require: Inputs r ∈ R
3
, Xp ∈ R
n×3
, Xb ∈ R
n
′×3
, vl ∈ R
n
, vr ∈ R
n
Ensure: A 0-1 indicator vector v: if there exists a subtree with node i as root that is isomorphic with
pattern tree tp defined with inputs r, Xp, Xb, then the v[i] is 1. Otherwise, v[i] is 0.
1: Initialize v as all 0 vector with a length of n
′
2: if r[1] == 0 then
3: Return v
4: end if
5: for i ∈ {1, 2, ..., m} do
6: if Xb[i, 3] == Xp[r[1], 3] then
7: if vl
[Xb[i, 1]]] == 1 and vr[Xb[i, 2]]] == 1 then
8: v[i] ← 1
9: else if vl
[Xb[i, 2]]] == 1 and vr[Xb[i, 1]]] == 1 then
10: v[i] ← 1
11: end if
12: end if
13: end for
14: Return v
• w = 1, and f(r, Xp, Xb) is defined as the depth of the pattern tree tp indicated with root vector
r. Although precisely calculating f(r, Xp, Xb) is of O(n), judging whether f(r, Xp, Xb) > 1 only
require us to check whether the root node has child. If not, then return False.
• d(r, Xp, Xb) = rl
, rr returns two new root vectors rl
, rr. Both rl
, rr have the same second and third
dimension as r. The rl
’s first dimension is updated to be the index of the left child of the root node
that r points to. The rr’s first dimension is updated to be the index of the right child of the root node
that r points to. The updating function can be written as:
• t(r, Xp, Xb) = v returns a 0-1 indicator vector v ∈ R
m with the same length of the base tree size. If
there exists a subtree with node i as root that is isomorphic with pattern tree tp defined with inputs
r, Xp, Xb, then the v[i] is 1. Otherwise, v[i] is 0. When the pattern tree’s depth is not higher than 1
(i.e., 1-node tree), t(r, Xp, Xb) is equivalent to output a 0-1 vector indicating the nodes in the base
tree that have the same color of the root node of pattern tree. The implementation is provided in Alg.
8.
113

• m(r, Xp, Xb, vl
, vl) = v merge the results vl
, vl
to acquire a 0-1 indicator vector v ∈ R
m with the
same length of the base tree size. If there exists a subtree with node i as root that is isomorphic with
pattern tree tp defined with inputs r, Xp, Xb, then the v[i] is 1. Otherwise, v[i] is 0. This function
can be implemented by checking whether the pattern root’s children have a perfect match with each
node’s children. Since each node has at most two children, checking the perfect match can be done
in constant time. The implementation is provided in Alg. 9.
After providing the detailed implementation of the functions d(·), t(·), m(·), we are going to prove that
there exists one unified transformer that can handle all these tasks with different prompts d, t, m. First, we
will provide the following Lemma:
Lemma 7.3.3. Any fixed-size logic circuit that only contains multi-fan-in AND gates, multi-fan-in OR gates,
NOT gates and has no recurrent structure can be precisely simulated by a multi-layer perceptron (MLP) with
ReLU activation function and a width of O(|Input| + |Circuit|) and a depth of O(|Circuit|), where |Input|
denotes the size of input and |Circuit| denotes the number of gates in the circuit.
Proof. Assume that we are given a series of input pins with logic variable of 0 or 1, organized as a 0-1 vector
x ∈ R
h
. We first prove that all gates can be simulated by a two-layer perceptron. Then we can serialize
all gates in the circuits and stack their corresponding 2-layer simulators accordingly to acquire a MLP
simulator. An AND gate that take x as input can be simulated as:
AND(x) = σ(wAx − h + 1) (7.2)
114

where σ is the ReLU activation function, and wA is a weight vector with all dimensions equal to 1. If some
dimensions of x are not the input of the gate, we can set the corresponding dimensions in the weight vector
as 0 and adjust the h as the input pin number. Similarly, an OR gate that take x as input can be simulated as:
OR(x) = 1 − σ(wOx + h + 1) (7.3)
where σ is the ReLU activation function, and wO is a weight vector with all dimensions equal to -1. A NOT
gate is different, since it only takes one input pin. In such a case, we denote the index of the input pin as i,
then we can simulate a NOT gate as:
NOT(x) = σ(wN x + 1) (7.4)
where wN is a weight is a weight vector whose i-th dimension equals to -1 and all other dimensions equal
to 0. Also, since the x is a 0-1 vector, the activation function is equivalent to a identical function to x:
x = σ(x) (7.5)
To construct a MLP that can simulate a fixed-size logic circuit without recurrent structure, we apply
the circuit serialization in [106] which order the gates based on topological order. In this way, we can
represent the circuit as a sequence GATE[1], GATE[2], GATE[3],...,GATE[L], where each GATE[i]’s input
only contains the output of the previous gates and the original input x. Therefore, we can construct a
2L-layer MLP base on the above serialization. Specifically, the 2i-th and 2i + 1-th layers of the MLP will
simulate the GAT E[i] as well as copy all previous inputs with activation function and concatenate them
together. This can be done by concatenate an identical matrix on the GATE’s weight vector (wA, wO or wN ).
In this way, we can construct a MLP that precisely simulate the circuit. Since every time we concatenate
115

the output of a gate with the input of it, the input dimension number of the final layer can be bounded by
O(|x| + L). In the worst case, for a circuit of size L, we needs 2L layers to precisely simulate it. However,
in many cases, a lot of gates in the circuits can be run parallelly. In such cases, the MLP could be much
more shallow.
Now, we can start to prove our main theorem 7.3.2:
Proof. We prove this theorem by constructing a Transformer that can tackle this problem. First we define
how to organize the input given r, Xp, Xb and the prompt. Specifically, we construct a feature sequence
X ∈ R
(3+n+n
′
)×7
. Each item in this sequence is a feature of 7 dimensions, indicating a token. The first two
dimensions indicate whether the token is a prompt (’00’), a root vector (’01’), a pattern tree node (’10’), or a
base tree node (’11’). The third to fifth dimensions carries the information about the token. For a prompt
token, ’100’ indicates merge prompt m, ’010’ indicates sub-task tackling prompt t, and ’001’ indicates task
decomposition prompt d. For other cases, these three dimensions are with the same formula as the three
dimensions in r, Xp, Xb. The rest two dimensions are allocated specifically for the merge function m(·)
to store vl and vr. More specifically, for the feature of token indicating the i-th base tree node, its sixth
dimension is vl
[i] and its seventh dimension is vr[i]. For other tokens, these two dimensions are filled with
0. In X[1] we store the prompt token. In X[2] and X[3] we store the input root vector r duplicately. We
store the same token twice so that we can tackle rl and rr separately. To separate this two token, we use
the last dimension, which was padded as 0 in r, to distinguish them. X[2, 5] is set as 0 and X[3, 5] is set
as 1. From X[4] to X[3 + n], we store Xp. From X[4 + n] to X[3 + n + n
′
], we store Xb. For all node
indices of pattern tree, we add them by 3. For all node indices of base tree, we add them by 3+n, so that the
indices can be applied to directly retrieve the positional embeddings. After preparing the inputs, we start
to construct our Transformer. The transformer first attach the position index for each token (positional
116

Algorithm 10 Logic circuit for MLP of the second Transformer layer
Require: Input feature x
′′ ∈ R
42
Ensure: Output feature y ∈ R
7
1: y ← x
′′[1 : 7] {Initialize y}
2: if x
′′[1 : 2] == 00 or x
′′[1 : 2] == 10{Prompt Token or Pattern Tree Node} then
3: Return y
4: else if x
′′[1 : 2] == 01 {Root Vector Token} then
5: if x
′′[24 : 26] == 001{Prompt is d} then
6: if x
′′[5] == 0 then
7: y[3] ← x
′′[10] {get rl
, similar as line 1 in Alg. 7}
8: else if x
′′[5] == 1 then
9: y[3] ← x
′′[11] {get rr, similar as line 2 in Alg. 7}
10: end if
11: end if
12: else if x
′′[1 : 2] == 11 {Base Tree Node Token} then
13: if x
′′[24 : 26] == 010{Prompt is t} then
14: if x
′′[40] == x
′′[5]{Line 6 in Alg. 8} then
15: y[5] ← 1
16: else
17: y[5] ← 0
18: end if
19: else if x
′′[24 : 26] == 100{Prompt is m} then
20: if x
′′[13] == 1 and x
′′[21] == 1 {Line 7 in Alg. 9} then
21: y[5] ← 1
22: else if x
′′[14] == 1 and x
′′[20] == 1{Line 9 in Alg. 9} then
23: y[5] ← 1
24: else
25: y[5] ← 0
26: end if
27: end if
28: end if
embedding). After that, the inputs are forwarded into a transformer with depth of 2. Each transformer
layer contains a multi-head attention layer followed by a MLP. As proved by [106, 45], the attention layer
of Transformer can retrieve the feature of tokens whose positional embeddings satisfy specific conditions.
For multi-head attention, different heads can retrieve tokens with different conditions. In the following
construction, we will use this conclusion to construct attention heads with different functions.
In the first Transformer layer, the function of each attention head is defined as:
• Head 1 only attends to the token itself to store X[i] for token i.
117

• Head 2 attends to the token with a positional embedding matches the X[i, 3] and copy this token’s
5-dimension feature. For tree node tokens, this head’s job is to retrieve the feature of X[i]’s left child.
For root vector tokens, this head’s job is to retrieve the feature of pattern tree root node. For the first
token (prompt token), this head’s retrieved feature will not be applied in the afterwards layers and
thus does not influence the correctness of the model.
• Similar as Head 2, Head 3 attends to the token with a positional embedding matches the X[i, 4] and
copy this token’s 5-dimension feature. This head’s job is to retrieve the feature of X[i]’s right child.
For root vector tokens, this head’s job is to retrieve the feature of base tree root node.
• Head 4 attends to the first token (prompt token) and copy this token’s 7-dimension feature. This
head’s job is to retrieve the prompt indicator.
• Head 5 attends to the second token (root token) and copy this token’s 7-dimension feature. This
head’s job is to retrieve the root information.
With the above 5 heads, the attention layer will output a 35-dimension feature for each token. We denote
these features as X′ ∈ R
(3+n+n
′
)×35. After that, these features are forwarded into a MLP fitting identical
mapping to acquire the input features for the second Transformer layer.
In the second Transformer layer, the function of each attention head is defined as:
• Head 1 only attends to the token itself to store X′
[i] for token i.
• Head 2 attends to the token with a positional embedding matches the X′
[i, 31] and copy this token’s
1-7 dimension features (X′
[X′
[i, 31], 1 : 7]). This head’s job is to broadcast the feature of the pattern
tree root node to every token.
With the above 2 heads, the attention layer will output a 42-dimension feature for each token. We denote
these features as X′′ ∈ R
(3+n+n
′
)×42. For root vector token, only the features from head 1 and head 4 are
118

useful. For base tree node tokens, all 42 dimensions are useful. Then each token’s feature are parallely
forwarded into a MLP. We will use this MLP to fit the logical circuit described in Alg. 10. The function of
Alg. 10 is to aggregate the functions of m(·), t(·), d(·) together and assign the correct value based on the
prompt indicator. In Alg. 10, all operations are AND, OR, NOT, SELECTOR, and ASSIGN and there is not
loop. Thus, it is a static logical circuit and can be implemented with multi-fan-in AND, OR, NOT gates.
Thus, it can be precisely simulated by a MLP according to our Lemma 7.3.3.
After acquiring the y ∈ R
7
for each token, we can organize them as a feature sequence Y ∈ R
(3+n+n
′
)×7
.
When the prompt is d, we return Y[2, 3 : 5] as rl and Y[3, 3 : 5] as rr. If the prompt is t or m, then we can
output Y[3 + n + 1 : 3 + n + n
′
, 5] as the expected v.
With the above theorem, we can prove that NC1 ⊆ P(DaC), which finishes the proof. With this
theoretic results, we can answer the RQ 1:
Compared to IO prompting, DaC have theoretically guaranteed advantages in expressive power.
7.3.2 DaC vs. CoT
In this section, we compare Divide-and-Conquer with Chain-of-Thoughts in order to understand the utility
and limitation of DaC prompting. The limitation of DaC prompting is that its expressive power is a subset
of CoT prompting:
Proposition 7.3.4. We denote the set of problems that a fixed-precision transformer with DaC prompting
can tackle as P(DaC). Similarly, we denote the set of problems that a fixed-precision transformer with CoT
prompting can tackle as P(CoT) Then we have the following results:
P(DaC) ⊆ P(CoT) (7.6)
119

Proof. The proof of this proposition is very straightforward. For any problem that DaC can solve, we can
concatenate all outputs of LLM by dividing, tackling, and merging them as a sequence. Then, we can prompt
LLM with CoT to output this sequence. Thus, the problem set that DaC can resolve is a subset of CoT.
The limitation revealed by the above theorem shows that compared to CoT, the appliance scope of
Divide-and-Conquer is limited. However, by analyzing the average decoding context window size, we show
that on specific tasks, divide and conquer can reduce the problem complexity:
Definition 2. Decoding Context Window Size: In auto-regressive decoding, each token is decoded from a
window that covers all previous tokens. We denote the length of the window as the Decoding Context Window
Size of the token.
Proposition 7.3.5. Suppose that a task contains k sub-tasks, each of which does not rely on the answers of
other sub-tasks. We define such sub-tasks as parallel sub-tasks. If an LLM tackles these sub-tasks sequentially
with CoT, then the average decoding context window size of the sub-tasks resolution will be C +
Pk
i=1 ri−1
2
,
where ri
is the length of the response to the i-th sub-task and C is the length of input context. If we tackle
them parallelly with DaC, then the average decoding context window size of the sub-tasks resolution will be
C +
Pk
i=1
(ri−1)2
2
Pk
j=1 rj
< C +
Pk
i=1 ri−1
2
.
Proof. Suppose that the LLM is auto-regressively decoding n tokens from an input context window with a
length of C. Then, the decoding window of the i-th token is C + i − 1. Thus, the average window size will
be:
Pn
i=1(C + i − 1)
n
=
C + n − 1
2
(7.7)
Thus, when we sequentially decode all the sub-task resolutions, the total length of the decoded sequence
will be Pk
i=1 ri
. Thus, the average window size will be:
C +
Pk
i=1 ri − 1
2
(7.8)
120

Meanwhile, when we apply Divide-and-Conquer, we parallelly decode each sub-tasks resolution. Thus, for
each sub-task, the total window size will be C
Pk
j=1 rj +
Pk
i=1
(ri−1)ri
2
. Thus the average window size
will be C +
Pk
i=1
(ri−1)ri
2
Pk
j=1 rj
. Meanwhile, with Jensen’s inequality, we have:
X
k
i=1
(ri − 1)ri <
X
k
i=1
(ri − 0.5)2 ≤ (
X
k
i=1
(ri − 0.5))2 ≤ (
X
k
i=1
ri − 0.5k)
2
(7.9)
Thus, when k ≥ 2, we have:
X
k
i=1
(ri − 1)ri < (
X
k
i=1
ri − 1)2
(7.10)
Thus, we have:
C +
X
k
i=1
(ri − 1)2
2
Pk
j=1 rj
< C +
Pk
i=1 ri − 1
2
(7.11)
The above proposition shows that when a task contains a large amount of parallel sub-tasks, DaC
is more helpful for reducing the average decoding context window size than CoT. Existing works have
empirically shown that a long decoding context window will propagate intermediate errors and thus
increase the probability of generating hallucination [169]. Thus, we can draw a conclusion that DaC is
competitive on tasks that contain a large amount of parallel sub-tasks and are bothered by intermediate
errors and hallucinations. With these theoretic results, we can answer the RQ 2:
Compared to CoT and its variants, DaC prompting’s expressive power is weaker. However, on tasks
containing a large amount of parallel sub-tasks, DaC is more helpful.
121

7.3.3 Advantages of DaC
The above analysis answers the two research questions that we proposed. By summarizing these two
answers, we can acquire the two conditions such that when a task simultaneously satisfies both conditions,
DaC brings performance boost:
• Condition 1: the task is harder than P(IO), such as TC0
-complete problems and NC1
-complete
problems.
• Condition 2: the task contains a large number of parallel sub-tasks and is bothered by hallucinations
or intermediate errors.
To better assist the prompt engineering on different tasks, we list the typical tasks that satisfy and
dissatisfy the proposed conditions. In common tasks, the following tasks satisfy the proposed conditions. For
such tasks, searching for a good decomposition prompt for DaC is likely to be helpful for the performance:
1. Fact Verification on Long Text
2. Auto Evaluation on Long Text
3. Article-level Summary
The following tasks typically do not satisfy the proposed conditions. For such tasks, searching for a good
decomposition prompt for DaC is not very likely to be helpful for the performance:
1. Multi-Round Question-Answering: It is a typical sequential task, thus violating condition 2
2. Planning: It is a typical sequential task, thus violating condition 2
7.4 Experiments
In this section, we present our conditions that can be applied to natural language verification tasks.
Specifically, we present the performance of baselines and Divide-and-Conquer on fact verification of long
122

Strategies GPT-3.5-Turbo GPT-4
F1 Acc Prec Recall F1 Acc Prec Recall
Io-Prompting 61.69 61.27 62.11 61.28 64.07 72.66 93.41 48.76
Chain-of-Thoughts 46.85 64.26 91.36 31.50 71.05 76.10 90.08 58.66
CoT-SC 47.70 64.25 88.83 32.60 71.39 76.36 90.41 58.98
Tree-of-Thoughts 70.40 59.91 55.83 95.34 69.41 71.73 75.53 64.28
Least-to-Most 56.43 64.91 74.42 45.44 72.51 77.11 90.74 60.38
Divide-and-Conquer 74.84 75.55 77.41 72.03 76.92 78.99 85.36 70.01
Table 7.1: Performance of different prompting methods on HaluEval dataset.
text. In this task, the LLM is required to determine whether a long corpus is aligned with base knowledge.
This task satisfied the proposed two conditions. For the first condition, we can reduce a 2-BTI problem
to fact verification by describing the two trees with natural language. In this way, we can convert the trees
to two paragraphs, and we need to ask the LLM to judge whether the two paragraphs are aligned or not.
For the second condition, since we are tackling long text, then each sentence can be regarded as parallel
sub-tasks. We select two benchmarks of fact verification: Fact-Verification for Hallucination Detection
and Fact-Verification for Misinformation Detection
7.4.1 Hallucination Detection
Although Large Language Models have achieved impressive performance on various NLP tasks, they are
bothered by hallucination problem [99], especially when the generated content or the input context is too
long for the user to have a thorough review [183]. In this paper, we focus on evaluating the performance of
different strategies in guiding LLM to recognize inconsistency between given context and model response
with hallucination.
Strategies GPT-3.5-Turbo GPT-4
F1 G-M Prec Recall F1 G-M Prec Recall
Io-Prompting 72.12 72.77 83.22 63.64 69.15 71.77 94.44 54.55
Chain-of-Thoughts 56.09 60.64 90.48 40.64 74.03 75.79 94.21 60.96
CoT-SC 56.83 61.44 91.67 41.18 70.09 73.45 100.0 53.95
Tree-of-Thoughts 69.91 73.30 53.74 100.0 77.34 78.00 88.89 68.45
Least-to-Most 54.08 54.15 51.46 56.99 73.56 74.25 85.21 64.71
Divide-and-Conquer 76.88 77.13 83.65 71.12 81.11 81.24 76.67 86.10
Table 7.2: Performance of different prompting methods on SciFact dataset.
123

Task Setup: We use the HaluEval-Summary dataset. We report the Accuracy, F1 score (the hallucination
pairs are positive samples), Precision, and Recall.
Setup of baselines, ablation variants, and DaC: In this task, our baselines include IO prompting,
Chain of Thought, CoT-SC, Tree-of-Thoughts Least-to-Most, and Decomposed Prompting. In this task, the
sub-tasks are verifying fragments of the summary, which are homogeneous and do not require recurssion.
In such a setting, Decomposed Prompting is equivalent to LtM. For this task, we apply single level Divideand-Conquer solver to decompose the summary to multiple sentences, handle them separately and then
merge the conclusions of all sentences.
Hallucination Detection in Long Context: We divide the summary into sentences. After that, we
will verify the sentences in parallel. Finally, we merge the verification to each sentence:
Decomposer Prompt d: Please help me segment the following paragraph as sentences. The separated
sentence should be output as: #Statement 1#: ...#Statement 2#: ...Do not say anything else. Just return the
statements in the given format.
Paragraph
Sub-task Tackling Prompt t: I want you to act as a factual contradiction checker. You are given a set of
statements and a document. Among the statements, there might be one or more statements that contain
contradictions with the document. Please find the problematic statement if it exists by analyzing the
statements one by one. For each statement, please make a choice:
• A: The statement is totally aligned with the document for sure.
• B: The statement contradicts the document.
Results: Experimental results are shown in Tab. 7.1. For both GPT-3.5 and GPT-4, our proposed
prompting strategy outperforms the baselines, presenting the advantage of DaC. More specifically, compared
to IO-prompting, DaC achieved better performance in general, indicating the advantage brought by stronger
124

expressive power. Meanwhile, compared to CoT and CoT-SC results, DaC clearly achieved much better
recall. Tree-of-Thoughts benefited from its searching ability and acquired a significantly better recall score
compared to other baselines. However, its significantly lower precision substantially harms its overall
performance and leads to accuracy that is even worse than standard IO-prompting. On the contrary, DaC
carefully checked all sentences, located the ones containing factual errors, and merged the answers.
7.4.2 Misinformation Detection
The increasing abuse of misinformation toward manipulating public opinions on social media has been
observed in different areas, such as healthcare (e.g., the recent COVID-19 pandemic) [135, 139]. This threat
is increasingly serious due to LLM’s capacity in content generation [92, 162, 180]. This challenge raises the
importance of fact-verification, which aims at judging the authenticity of an article based on a collection of
evidence from verified sources [164, 179]. In this experiment, we present that DaC can outperform other
baselines in fact verification involved with news articles.
Task Setup: In this experiment, we mainly adopt SciFact dataset [156]. For this task, similar to
hallucination detection, we apply a single-level Divide-and-Conquer solver to decompose the news article
into multiple sentences, handle them separately, and then merge the conclusions of all sentences. Also,
the baselines in this experiment are the same as those in hallucination detection. The evaluation metrics
include F1 score, G-Mean score (geometric mean of precision and recall), Precision, and Recall. We do not
apply accuracy as the positive and negative classes are not balanced.
Fact-Verification for Misinformation Detection: Similar to hallucination detection, we divide
the summary into sentences. After that, we will verify the sentences in parallel. Finally, we merge the
verification to each sentence. Thus, our decomposer prompt and sub-task tackling prompt are the same as
hallucination detection. The only difference is the merge prompt.
125

Merge Prompt m: If we connect the above statements to be a news article, based on the above analysis,
please answer me: Is there any contradiction between the document and the article?
Results: Experimental results are shown in Tab. 7.2. Notably, GPT-3.5 incorporated with our proposed
prompting strategy even outperformed the performance of GPT-4 incorporated with IO-prompting, Leastto-Most, CoT, and CoT-SC, which have significantly lower recall scores, indicating their proneness to
deception. Only Tree-of-Thoughts, which benefited from its advantage in exploring various options,
acquired the best results among all baselines but was still defeated by DaC. Moreover, as we can see, for
GPT-4 the performance of CoT-SC is even worse than CoT, which is supposed to be a specific case of CoT-SC
without exploration. These results suggest that, when facing deceptive content generated on purpose, the
improvement of existing works may not be robust.
7.4.3 Discussions and Limitations
In summary, the proposed method has the following advantages:
Comparison with IO-Prompting: Superiority in Expressive Power As we proved in Sec. 7.3,
Compared to IO-prompting, DaC has stronger expressive power and thus can solve harder problems.
Comparison with CoT and EoT: Disentangling the task decomposition and task resolution
Compared to the prompting family of CoT and EoT, DaC explicitly separates the task decomposition stage
and task resolution stage. Therefore, we can acquire explicit decomposed sub-tasks rather than intermediate
thoughts proposed during decoding. Consequently, we can explicitly enumerate all sub-tasks output by the
decomposition module and prevent the model from missing important sub-tasks.
Comparison with LtM and Decomposed Prompting: Parallel Sub-task Handler and Sequential
Sub-task Handler Similar to DaC, some program-guided prompting like LtM and Decomposed Prompting
also explicitly separate the task decomposition stage and task resolution stage. However, they are mainly
126

designed for multi-step reasoning for complex tasks. Thus, they sequentially tackle the sub-tasks and
assembly the resolutions. As a result, they tend to follow the flow of the deceptive content, leading to
proneness to deceptive content.
Although DaC DaC surpasses the baselines on the proposed tasks, it still has some limitations. The
first issue is that the appliance scope of DaC is still limited. More specifically, CoT, EoT, LtM, and DaC are
based on different algorithmic paradigms, learning to different Appliance Scopes. As pointed out by Feng
et al., CoT and LtM can be considered as a neural dynamic programming algorithm. Thus, CoT is more
suitable for tasks that can be bridged to dynamic programming, such as multi-step question answering.
Differently, EoT is based on exploration and search, which is more suitable for planning and search, such
as Game of 24 [173]. DaC is based on Divide-and-Conquer algorithm. Thus, it is more suitable for tasks
that can be decomposed into a series of sub-tasks that are disjoint or only slightly overlapped. Our future
work will focus on further expanding the appliance scope of DaC to more areas like question answering.
127

Chapter 8
Conclusions and Future Work
In the above chapters, we have made efforts to tackle emergent problems in combating misinformation and
social manipulation by developing knowledge-informed data-driven models, causal inference frameworks,
and prompting strategies for LLM on misinformation detection. Experimental results indicate the advantages
of the proposed method on datasets collected from the real world. For future works of these works, we
propose the following possible directions:
Linking Causal Analysis and LLM for better Misinformation Mitigation. Our work in chapter
6 has presented the potential of learning treatment effects from biased social media data. This estimator
can work as a reward model that helps us guide Large Language Models to learn people’s reactions to
various information and draft more persuasive narratives. Therefore, in future works, we aim to develop a
reinforcement learning fine-tuning algorithm that can train LLM to learn how to draft more persuasive and
impactful clarifications to misinformation.
Human-in-loop Learning for Combating Misinformation Campaigns. It is noticeable that most
of the models for combating misinformation campaigns are statistical machine learning models. For
example, VigDet, as a statistical learning model, although integrated with prior knowledge, may have
wrong predictions, such as mislabeling normal accounts as coordinated or missing some truly coordinated
128

accounts. Therefore, we insist that the estimations from existing statistical learning-based models should be
only considered as an efficient and effective assistant tool for human verifiers or researchers to accelerate
the filtering of suspicious accounts for further investigation or analysis. Also, misinformation campaigns
could use the proposed approach to direct their editors to write more impactful fake news. A probable
strategy to address this potential problem is to require social media platforms to recruit human verifiers and
build up human-in-loop learning frameworks to correct and supplement the statistical machine-learning
models.
Multimodal Knowledge-guided Learning of the Social Manipulation Detection. Our current
coordination detection framework can only support one prior knowledge-based graph as input. Consequently, if there are multiple kinds of prior knowledge, we have to manually define integration methods
and parameters like weight. If an automatic integration module can be proposed, we expect that the model
performance can be further improved.
Dataset Development. The rise of LLM has already shown the potential of training large models with
large-scale unsupervised or weakly-supervised data with a small fraction of high-quality data. Therefore,
for combating misinformation, we can follow similar paradigms to enlarge the dataset scale by widely
collecting news claims from various resources (i.e. newspapers, new media and so on) while reducing the
annotation cost by only labeling part of them.
129

Bibliography
[1] Odd O Aalen, Mats J Stensrud, Vanessa Didelez, Rhian Daniel, Kjetil Røysland, and
Susanne Strohmaier. “Time-dependent mediators in survival analysis: Modeling direct and indirect
effects with the additive hazards model”. In: Biometrical Journal 62.3 (2020), pp. 532–549.
[2] Sahar Abdelnabi, Rakibul Hasan, and Mario Fritz. “Open-domain, content-based, multi-modal
fact-checking of out-of-context images via online resources”. In: Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition. 2022, pp. 14940–14949.
[3] Aseel Addawood, Adam Badawy, Kristina Lerman, and Emilio Ferrara. “Linguistic cues to
deception: Identifying political trolls on social media”. In: ICWSM. 2019.
[4] Mohammad Majid Akhtar, Rahat Masood, Muhammad Ikram, and Salil S Kanhere. “False
Information, Bots and Malicious Campaigns: Demystifying Elements of Social Media
Manipulations”. In: arXiv preprint arXiv:2308.12497 (2023).
[5] Ilaria Amaro, Attilio Della Greca, Rita Francese, Genoveffa Tortora, and Cesare Tucci. “AI
unreliable answers: A case study on ChatGPT”. In: International Conference on Human-Computer
Interaction. Springer. 2023, pp. 23–40.
[6] Cheuk Hang Au, Kevin KW Ho, and Dickson KW Chiu. “The role of online misinformation and
fake news in ideological polarization: barriers, catalysts, and implications”. In: Information systems
frontiers (2022), pp. 1–24.
[7] Adam Badawy, Aseel Addawood, Kristina Lerman, and Emilio Ferrara. “Characterizing the 2016
Russian IRA influence campaign”. In: Social Network Analysis and Mining 9.1 (2019), pp. 1–11.
[8] Mohammad Taha Bahadori and David Heckerman. “Debiasing Concept-based Explanations with
Causal Analysis”. In: International Conference on Learning Representations. 2021. url:
https://openreview.net/forum?id=6puUoArESGp.
[9] Nicola Barbieri, Francesco Bonchi, and Giuseppe Manco. “Cascade-Based Community Detection”.
In: Proceedings of the Sixth ACM International Conference on Web Search and Data Mining. WSDM
’13. 2013, pp. 33–42.
[10] Amirsiavosh Bashardoust, Stefan Feuerriegel, and Yash Raj Shrestha. “Comparing the willingness to
share for human-generated vs. AI-generated fake news”. In: arXiv preprint arXiv:2402.07395 (2024).
130

[11] Jeff Baumes, Mark Goldberg, Malik Magdon-Ismail, and William Al Wallace. “Discovering hidden
groups in communication networks”. In: ICISI. Springer. 2004.
[12] Alexis Bellot and Mihaela van der Schaar. “Policy Analysis using Synthetic Controls in
Continuous-Time”. In: ICML. 2021.
[13] Julian Besag. “Statistical Analysis of Non-Lattice Data”. In: The Statistician 24.3 (1975), pp. 179–195.
[14] Ioana Bica, Ahmed M Alaa, James Jordon, and Mihaela van der Schaar. “Estimating counterfactual
treatment outcomes over time through adversarially balanced representations”. In: International
Conference on Learning Representations (2020).
[15] Yuri Boykov, Olga Veksler, and Ramin Zabih. “Fast approximate energy minimization via graph
cuts”. In: IEEE Transactions on pattern analysis and machine intelligence 23.11 (2001), pp. 1222–1239.
[16] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. “Language models are
few-shot learners”. In: Advances in neural information processing systems 33 (2020), pp. 1877–1901.
[17] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar,
Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. “Sparks of artificial general intelligence:
Early experiments with gpt-4”. In: arXiv preprint arXiv:2303.12712 (2023).
[18] N. Campbell. “The study of discontinuous phenomena”. In: Proc Cambr. Phil. Soc vol. 15 (1909), pp.
117–136.
[19] Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong,
Bixiong Xu, Jing Bai, Jie Tong, et al. “Spectral temporal graph neural network for multivariate
time-series forecasting”. In: Advances in neural information processing systems 33 (2020),
pp. 17766–17778.
[20] Qiang Cao, Xiaowei Yang, Jieqi Yu, and Christopher Palow. “Uncovering large groups of active
malicious accounts in online social networks”. In: Proceedings of the 2014 ACM Conference on
Computer and Communications Security. 2014, pp. 477–488.
[21] Kevin Matthe Caramancion. “Harnessing the power of ChatGPT to decimate mis/disinformation:
Using ChatGPT for fake news detection”. In: 2023 IEEE World AI IoT Congress (AIIoT). IEEE. 2023,
pp. 0042–0046.
[22] Nikan Chavoshi, Hossein Hamooni, and Abdullah Mueen. “Temporal patterns in bot activities”. In:
Proceedings of the 26th international conference on world wide web companion. 2017, pp. 1601–1606.
[23] Canyu Chen and Kai Shu. “Can LLM-Generated Misinformation Be Detected?” In: arXiv preprint
arXiv:2309.13788 (2023).
[24] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.
“Semantic image segmentation with deep convolutional nets and fully connected crfs”. In: arXiv
preprint arXiv:1412.7062 (2014).
131

[25] Ricky TQ Chen, Brandon Amos, and Maximilian Nickel. “Neural spatio-temporal point processes”.
In: arXiv preprint arXiv:2011.04583 (2020).
[26] Yuyan Chen, Qiang Fu, Yichen Yuan, Zhihao Wen, Ge Fan, Dayiheng Liu, Dongmei Zhang,
Zhixu Li, and Yanghua Xiao. “Hallucination detection: Robustly discerning reliable answers in large
language models”. In: Proceedings of the 32nd ACM International Conference on Information and
Knowledge Management. 2023, pp. 245–255.
[27] Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang,
Dawei Yin, Wenqi Fan, Hui Liu, et al. “Exploring the potential of large language models (llms) in
learning on graphs”. In: arXiv preprint arXiv:2307.03393 (2023).
[28] Lu Cheng, Ruocheng Guo, Kai Shu, and Huan Liu. “Causal understanding of fake news
dissemination on social media”. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge
Discovery & Data Mining. 2021, pp. 148–157.
[29] Lu Cheng, Ahmadreza Mosallanezhad, Paras Sheth, and Huan Liu. “Causal Learning for Socially
Responsible AI”. In: 30th International Joint Conference on Artificial Intelligence, IJCAI 2021.
International Joint Conferences on Artificial Intelligence. 2021, pp. 4374–4381.
[30] Vincent Cheng and Yu Zhang. “Analyzing ChatGPT’s Mathematical Deficiencies: Insights and
Contributions”. In: Proceedings of the 35th Conference on Computational Linguistics and Speech
Processing (ROCLING 2023). Ed. by Jheng-Long Wu and Ming-Hsiang Su. Taipei City, Taiwan: The
Association for Computational Linguistics and Chinese Language Processing (ACLCLP), Oct. 2023,
pp. 188–193. url: https://aclanthology.org/2023.rocling-1.22.
[31] David Chiang, Peter Cholak, and Anand Pillay. Tighter Bounds on the Expressivity of Transformer
Encoders. 2023. arXiv: 2301.10743 [cs.LG].
[32] Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, and Clifford Stein. Introduction to
algorithms. 2022.
[33] Corinna Cortes and Neil D Lawrence. “Inconsistency in conference peer review: revisiting the 2014
neurips experiment”. In: arXiv preprint arXiv:2109.09774 (2021).
[34] Antonia Creswell and Murray Shanahan. “Faithful reasoning using large language models”. In:
arXiv preprint arXiv:2208.14271 (2022).
[35] Wanyun Cui, Linqiu Zhang, Qianle Wang, and Shuyang Cai. “Who said that? benchmarking social
media ai detection”. In: arXiv preprint arXiv:2310.08240 (2023).
[36] Wendi Cui, Jiaxin Zhang, Zhuohang Li, Damien Lopez, Kamalika Das, Bradley Malin, and
Sricharan Kumar. “A Divide-Conquer-Reasoning Approach to Consistency Evaluation and
Improvement in Blackbox Large Language Models”. In: Socially Responsible Language Modelling
Research. 2023.
[37] Daryl J Daley and David Vere-Jones. An introduction to the theory of point processes: volume II:
general theory and structure. Springer, 2007.
132

[38] Anirban DasGupta. Asymptotic theory of statistics and probability. Vol. 180. Springer, 2008.
[39] Abir De, Utkarsh Upadhyay, and Manuel Gomez-Rodriguez. “Temporal point processes”. In:
Technical report, Technical report, Saarland University (2019).
[40] Luigi De Angelis, Francesco Baglivo, Guglielmo Arzilli, Gaetano Pierpaolo Privitera,
Paolo Ferragina, Alberto Eugenio Tozzi, and Caterina Rizzo. “ChatGPT and the rise of large
language models: the new AI-driven infodemic threat in public health”. In: Frontiers in Public Health
11 (2023), p. 1166120.
[41] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. “BERT: Pre-training of Deep
Bidirectional Transformers for Language Understanding”. In: Proceedings of the 2019 Conference of
the North American Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers). Minneapolis, Minnesota: Association for
Computational Linguistics, June 2019, pp. 4171–4186. doi: 10.18653/v1/N19-1423.
[42] Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodriguez, and Le Song.
“Recurrent marked temporal point processes: Embedding event history to vector”. In: ACM SIGKDD.
2016, pp. 1555–1564.
[43] Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin,
Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, et al. “Faith and fate: Limits of
transformers on compositionality”. In: Advances in Neural Information Processing Systems 36 (2024).
[44] Sarah Evanega, Mark Lynas, Jordan Adams, Karinne Smolenyak, and Cision Global Insights.
“Coronavirus misinformation: quantifying sources and themes in the COVID-19 ‘infodemic’”. In:
JMIR Preprints 19.10 (2020), p. 2020.
[45] Guhao Feng, Yuntian Gu, Bohang Zhang, Haotian Ye, Di He, and Liwei Wang. “Towards Revealing
the Mystery behind Chain of Thought: a Theoretical Perspective”. In: arXiv preprint
arXiv:2305.15408 (2023).
[46] Shangbin Feng, Zhaoxuan Tan, Herun Wan, Ningnan Wang, Zilong Chen, Binchi Zhang,
Qinghua Zheng, Wenqian Zhang, Zhenyu Lei, Shujie Yang, et al. “Twibot-22: Towards graph-based
twitter bot detection”. In: Advances in Neural Information Processing Systems 35 (2022),
pp. 35254–35269.
[47] Shangbin Feng, Herun Wan, Ningnan Wang, and Minnan Luo. “BotRGCN: Twitter bot detection
with relational graph convolutional networks”. In: Proceedings of the 2021 IEEE/ACM international
conference on advances in social networks analysis and mining. 2021, pp. 236–239.
[48] Shangbin Feng, Herun Wan, Ningnan Wang, Zhaoxuan Tan, Minnan Luo, and Yulia Tsvetkov.
“What Does the Bot Say? Opportunities and Risks of Large Language Models in Social Media Bot
Detection”. In: arXiv preprint arXiv:2402.00371 (2024).
[49] Emilio Ferrara. “Social bot detection in the age of ChatGPT: Challenges and opportunities”. In: First
Monday (2023).
133

[50] Emilio Ferrara, Herbert Chang, Emily Chen, Goran Muric, and Jaimin Patel. “Characterizing social
media manipulation in the 2020 US presidential election”. In: First Monday (2020).
[51] Emilio Ferrara, Onur Varol, Clayton Davis, Filippo Menczer, and Alessandro Flammini. “The rise of
social bots”. In: Communications of the ACM 59.7 (2016), pp. 96–104.
[52] Emilio Ferrara, Onur Varol, Filippo Menczer, and Alessandro Flammini. “Detection of promoted
social media campaigns”. In: tenth international AAAI conference on web and social media. 2016.
[53] Adrien Friggeri, Lada Adamic, Dean Eckles, and Justin Cheng. “Rumor cascades”. In: proceedings of
the international AAAI conference on web and social media. Vol. 8. 1. 2014, pp. 101–110.
[54] Yaroslav Ganin and Victor Lempitsky. “Unsupervised domain adaptation by backpropagation”. In:
International conference on machine learning. PMLR. 2015, pp. 1180–1189.
[55] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and
Graham Neubig. “Pal: Program-aided language models”. In: International Conference on Machine
Learning. PMLR. 2023, pp. 10764–10799.
[56] Tian Gao, Dharmashankar Subramanian, Debarun Bhattacharjya, Xiao Shou, Nicholas Mattei, and
Kristin Bennett. “Causal Inference for Event Pairs in Multivariate Point Processes”. In: Advances in
Neural Information Processing Systems. Ed. by A. Beygelzimer, Y. Dauphin, P. Liang, and
J. Wortman Vaughan. 2021. url: https://openreview.net/forum?id=x8qirBbT9xp.
[57] Jose A Garcia, Rosa Rodriguez-Sánchez, and J Fdez-Valdivia. “Quality censoring in peer review”. In:
Scientometrics 126 (2021), pp. 825–830.
[58] Jiayan Guo, Lun Du, and Hengyu Liu. “GPT4Graph: Can Large Language Models Understand
Graph Structured Data? An Empirical Evaluation and Benchmarking”. In: arXiv preprint
arXiv:2305.15066 (2023).
[59] Sonu Gupta, Ponnurangam Kumaraguru, and Tanmoy Chakraborty. “Malreg: Detecting and
analyzing malicious retweeter groups”. In: Proceedings of the India Joint International Conference on
Data Science and Management of Data. 2019, pp. 61–69.
[60] William L Hamilton, Rex Ying, and Jure Leskovec. “Inductive representation learning on large
graphs”. In: arXiv preprint arXiv:1706.02216 (2017).
[61] Douglas M Hawkins. Identification of outliers. Vol. 11. Springer, 1980.
[62] Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr,
Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. “How good are
gpt models at machine translation? a comprehensive evaluation”. In: arXiv preprint
arXiv:2302.09210 (2023).
[63] Matthew Hindman and Vlad Barash. “Disinformation, and influence campaigns on twitter”. In:
Knight Foundation: George Washington University (2018).
134

[64] Traci Hong, Zilu Tang, Manyuan Lu, Yunwen Wang, Jiaxi Wu, and Derry Wijaya. “Effects of#
coronavirus content moderation on misinformation and anti-Asian hate on Instagram”. In: new
media & society (2023), p. 14614448231187529.
[65] Beizhe Hu, Qiang Sheng, Juan Cao, Yuhui Shi, Yang Li, Danding Wang, and Peng Qi. “Bad actor,
good advisor: Exploring the role of large language models in fake news detection”. In: arXiv
preprint arXiv:2309.12247 (2023).
[66] Beizhe Hu, Qiang Sheng, Juan Cao, Yuhui Shi, Yang Li, Danding Wang, and Peng Qi. “Bad actor,
good advisor: Exploring the role of large language models in fake news detection”. In: Proceedings
of the AAAI Conference on Artificial Intelligence. Vol. 38. 20. 2024, pp. 22105–22113.
[67] Haitao Huang, Hu Tian, Xiaolong Zheng, Xingwei Zhang, Daniel Dajun Zeng, and Fei-Yue Wang.
“CGNN: A Compatibility-Aware Graph Neural Network for Social Media Bot Detection”. In: IEEE
Transactions on Computational Social Systems (2024).
[68] Yue Huang and Lichao Sun. “Harnessing the power of chatgpt in fake news: An in-depth
exploration in generation, detection and explanation”. In: arXiv preprint arXiv:2310.05046 (2023).
[69] Jane Im, Eshwar Chandrasekharan, Jackson Sargent, Paige Lighthammer, Taylor Denby,
Ankit Bhargava, Libby Hemphill, David Jurgens, and Eric Gilbert. “Still out there: Modeling and
identifying russian troll accounts on twitter”. In: ACM Web Science. 2020, pp. 1–10.
[70] Birgit Jenner, Johannes Köbler, Pierre McKenzie, and Jacobo Torán. “Completeness results for
graph isomorphism”. In: Journal of Computer and System Sciences 66.3 (2003), pp. 549–566.
[71] Di Jin, Xinxin You, Weihao Li, Dongxiao He, Peng Cui, Françoise Fogelman-Soulié, and
Tanmoy Chakraborty. “Incorporating network embedding into markov random field for better
community detection”. In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 01.
2019, pp. 160–167.
[72] Daniel Jolley and Karen M Douglas. “The effects of anti-vaccine conspiracy theories on vaccination
intentions”. In: PloS one 9.2 (2014), e89177.
[73] Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and
Yejin Choi. “Maieutic prompting: Logically consistent reasoning with recursive explanations”. In:
arXiv preprint arXiv:2205.11822 (2022).
[74] Daniel Kahneman. Thinking, fast and slow. macmillan, 2011.
[75] Nitin Kamra, Yizhou Zhang, Sirisha Rambhatla, Chuizheng Meng, and Yan Liu. “PolSIRD: Modeling
Epidemic spread under intervention policies”. In: Journal of Healthcare Informatics Research 5.3
(2021), pp. 231–248.
[76] Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and
Ashish Sabharwal. “Decomposed prompting: A modular approach for solving complex tasks”. In:
arXiv preprint arXiv:2210.02406 (2022).
[77] D. Kingma and J. Ba. “Adam: A method for stochastic optimization”. In: arXiv:1412.6980 (2014).
135

[78] D. P. Kingma and M. Welling. “Auto-Encoding Variational Bayes”. In: International Conference on
Learning Representations (ICLR). 2014.
[79] Thomas N Kipf and Max Welling. “Semi-supervised classification with graph convolutional
networks”. In: arXiv preprint arXiv:1609.02907 (2016).
[80] Ron Kohavi and Roger Longbotham. “Online controlled experiments and A/B tests”. In:
Encyclopedia of machine learning and data mining (2015), pp. 1–11.
[81] Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT
press, 2009.
[82] Vladimir Kolmogorov and Ramin Zabin. “What energy functions can be minimized via graph cuts?”
In: IEEE transactions on pattern analysis and machine intelligence 26.2 (2004), pp. 147–159.
[83] Philipp Krähenbühl and Vladlen Koltun. “Efficient Inference in Fully Connected CRFs with
Gaussian Edge Potentials”. In: Advances in Neural Information Processing Systems. Ed. by
J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Q. Weinberger. Vol. 24. Curran Associates,
Inc., 2011. url:
https://proceedings.neurips.cc/paper/2011/file/beda24c1e1b46055dff2c39c98fd6fc1-Paper.pdf.
[84] Katherine Kricorian, Rachel Civen, and Ozlem Equils. “COVID-19 vaccine hesitancy:
Misinformation and perceptions of vaccine safety”. In: Human Vaccines & Immunotherapeutics 18.1
(2022), p. 1950504.
[85] John Lafferty, Andrew McCallum, and Fernando CN Pereira. “Conditional random fields:
Probabilistic models for segmenting and labeling sequence data”. In: (2001).
[86] Soochan Lee and Gunhee Kim. “Recursion of thought: A divide-and-conquer approach to
multi-context reasoning with language models”. In: arXiv preprint arXiv:2306.06891 (2023).
[87] Kristina Lerman, Rumi Ghosh, and Tawan Surachawala. “Social contagion: An empirical study of
information spread on Digg and Twitter follower graphs”. In: arXiv preprint arXiv:1202.3162 (2012).
[88] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal,
Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. “Retrieval-augmented
generation for knowledge-intensive nlp tasks”. In: Advances in Neural Information Processing
Systems 33 (2020), pp. 9459–9474.
[89] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. “Blip-2: Bootstrapping language-image
pre-training with frozen image encoders and large language models”. In: arXiv preprint
arXiv:2301.12597 (2023).
[90] Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. “Halueval: A large-scale
hallucination evaluation benchmark for large language models”. In: Proceedings of the 2023
Conference on Empirical Methods in Natural Language Processing. 2023, pp. 6449–6464.
136

[91] Shuang Li, Lu Wang, Ruizhi Zhang, Xiaofu Chang, Xuqin Liu, Yao Xie, Yuan Qi, and Le Song.
“Temporal Logic Point Processes”. In: Proceedings of the 37th International Conference on Machine
Learning. Ed. by Hal Daumé III and Aarti Singh. Vol. 119. Proceedings of Machine Learning
Research. PMLR, 2020, pp. 5990–6000. url: http://proceedings.mlr.press/v119/li20p.html.
[92] Siyu Li, Jin Yang, and Kui Zhao. “Are you in a masquerade? exploring the behavior and impact of
large language model driven social bots in online social networks”. In: arXiv preprint
arXiv:2307.10337 (2023).
[93] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. “GPT
understands, too”. In: AI Open (2023).
[94] Xueqing Liu and Paul Sajda. “Roe: A Computational-Efficient Anti-hallucination Fine-Tuning
Technology for Large Language Model Inspired by Human Learning Process”. In: International
Conference on Brain Informatics. Springer. 2023, pp. 456–463.
[95] Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng,
Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. “Trustworthy LLMs: a Survey and
Guideline for Evaluating Large Language Models’ Alignment”. In: arXiv preprint arXiv:2308.05374
(2023).
[96] Steven Loria et al. “textblob Documentation”. In: Release 0.15 2 (2018), p. 269.
[97] Luca Luceri, Silvia Giordano, and Emilio Ferrara. “Detecting troll behavior via inverse
reinforcement learning: A case study of Russian trolls in the 2016 US election”. In: Proceedings of
the International AAAI Conference on Web and Social Media. Vol. 14. 2020, pp. 417–427.
[98] Jing Ma, Wei Gao, and Kam-Fai Wong. “Rumor Detection on Twitter with Tree-structured
Recursive Neural Networks”. In: Proceedings of the 56th Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers). Vol. 1. 2018, pp. 1980–1989.
[99] Potsawee Manakul, Adian Liusie, and Mark JF Gales. “Selfcheckgpt: Zero-resource black-box
hallucination detection for generative large language models”. In: arXiv preprint arXiv:2303.08896
(2023).
[100] Rui Mao, Guanyi Chen, Xulang Zhang, Frank Guerin, and Erik Cambria. “GPTEval: A survey on
assessments of ChatGPT and GPT-4”. In: arXiv preprint arXiv:2308.12488 (2023).
[101] Gary Marcus, Ernest Davis, and Scott Aaronson. “A very preliminary analysis of DALL-E 2”. In:
arXiv preprint arXiv:2204.13807 (2022).
[102] Aleix M Martinez and Avinash C Kak. “Pca versus lda”. In: IEEE transactions on pattern analysis and
machine intelligence 23.2 (2001), pp. 228–233.
[103] Michele Mazza, Stefano Cresci, Marco Avvenuti, Walter Quattrociocchi, and Maurizio Tesconi.
“Rtbust: Exploiting temporal patterns for botnet detection on twitter”. In: Proceedings of the 10th
ACM conference on web science. 2019, pp. 183–192.
137

[104] Hongyuan Mei and Jason M Eisner. “The neural hawkes process: A neurally self-modulating
multivariate point process”. In: NIPS. 2017, pp. 6754–6764.
[105] Chuizheng Meng, Sungyong Seo, Defu Cao, Sam Griesemer, and Yan Liu. “When Physics Meets
Machine Learning: A Survey of Physics-Informed Machine Learning”. In: arXiv preprint
arXiv:2203.16797 (2022).
[106] William Merrill and Ashish Sabharwal. “The Expresssive Power of Transformers with Chain of
Thought”. In: arXiv preprint arXiv:2310.07923 (2023).
[107] William Merrill and Ashish Sabharwal. “The Parallelism Tradeoff: Limitations of Log-Precision
Transformers”. In: Transactions of the Association for Computational Linguistics 11 (2023),
pp. 531–545. doi: 10.1162/tacl_a_00562.
[108] Radford M. Neal and Geoffrey E. Hinton. “A View of the EM Algorithm That Justifies Incremental,
Sparse, and Other Variants”. In: Proceedings of the NATO Advanced Study Institute on Learning in
Graphical Models. USA, 1998, pp. 355–368.
[109] Yurii Nesterov. “Introductory lectures on convex programming volume i: Basic course”. In: Lecture
notes 3.4 (1998), p. 5.
[110] Kimia Noorbakhsh and Manuel Gomez Rodriguez. “Counterfactual Temporal Point Processes”. In:
arXiv preprint arXiv:2111.07603 (2021).
[111] Takahiro Omi, Kazuyuki Aihara, et al. “Fully neural network based model for general temporal
point processes”. In: NIPS. 2019, pp. 2122–2132.
[112] Manfred Opper and David Saad. Advanced mean field methods: Theory and practice. MIT press, 2001.
[113] Diogo Pacheco, Pik-Mai Hui, Christopher Torres-Lugo, Bao Tran Truong, Alessandro Flammini,
and Filippo Menczer. “Uncovering Coordinated Networks on Social Media: Methods and Case
Studies”. In: Proceedings of the International AAAI Conference on Web and Social Media 15.1 (May
2021), pp. 455–466. url: https://ojs.aaai.org/index.php/ICWSM/article/view/18075.
[114] Reza Arkan Partadiredja, Carlos Entrena Serrano, and Davor Ljubenkov. “AI or human: the
socio-ethical implications of AI-generated media content”. In: 2020 13th CMI Conference on
Cybersecurity and Privacy (CMI)-Digital Transformation-Potentials and Challenges (51275). IEEE.
2020, pp. 1–6.
[115] Judea Pearl. Causality. Cambridge university press, 2009.
[116] Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect. Basic books,
2018.
[117] Verónica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea. “Automatic
Detection of Fake News”. In: Proceedings of the 27th International Conference on Computational
Linguistics. Santa Fe, New Mexico, USA: Association for Computational Linguistics, Aug. 2018,
pp. 3391–3401. url: https://www.aclweb.org/anthology/C18-1287.
138

[118] V Jothi Prakash and Dr LM Nithya. “A survey on semi-supervised learning techniques”. In: arXiv
preprint arXiv:1402.4645 (2014).
[119] Feng Qian, Chengyue Gong, Karishma Sharma, and Yan Liu. “Neural User Response Generator:
Fake News Detection with Collective User Intelligence.” In: IJCAI. Vol. 3834. 2018, p. 3840.
[120] Meng Qu, Yoshua Bengio, and Jian Tang. “GMNN: Graph Markov Neural Networks”. In: arXiv:
Learning (2019).
[121] Meng Qu and Jian Tang. “Probabilistic Logic Neural Networks for Reasoning”. In: Advances in
Neural Information Processing Systems 32 (2019), pp. 7712–7722.
[122] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. “Language
models are unsupervised multitask learners”. In: ().
[123] Edward Raff and Jared Sylvester. “Gradient Reversal against Discrimination: A Fair Neural Network
Learning Approach”. In: 2018 IEEE 5th International Conference on Data Science and Advanced
Analytics (DSAA). 2018, pp. 189–198. doi: 10.1109/DSAA.2018.00029.
[124] Aijaz Ahmad Reshi, Furqan Rustam, Wajdi Aljedaani, Shabana Shafi, Abdulaziz Alhossan,
Ziyad Alrabiah, Ajaz Ahmad, Hessa Alsuwailem, Thamer A Almangour, Musaad A Alshammari,
et al. “COVID-19 Vaccination-Related Sentiments Analysis: A Case Study Using Worldwide Twitter
Dataset”. In: Healthcare. Vol. 10. 3. MDPI. 2022, p. 411.
[125] Kjetil Røysland. “Counterfactual analyses with graphical models based on local independence”. In:
The Annals of Statistics 40.4 (2012), pp. 2162–2194.
[126] Natali Ruchansky, Sungyong Seo, and Yan Liu. “CSI: A Hybrid Deep Model for Fake News
Detection”. In: Proceedings of the 2017 ACM on Conference on Information and Knowledge
Management. ACM. 2017, pp. 797–806.
[127] Pål Christie Ryalen, Mats Julius Stensrud, Sophie Fosså, and Kjetil Røysland. “Causal inference in
continuous time: an example on prostate cancer therapy”. In: Biostatistics 21.1 (2020), pp. 172–185.
[128] Fatima K Abu Salem, Roaa Al Feel, Shady Elbassuoni, Mohamad Jaber, and May Farah. “FA-KES: a
fake news dataset around the Syrian war”. In: Proceedings of the International AAAI Conference on
Web and Social Media. Vol. 13. 01. 2019, pp. 573–582.
[129] Karan Samel, Zelin Zhao, Binghong Chen, Shuang Li, Dharmashankar Subramanian, Irfan Essa, and
Le Song. “Learning Temporal Rules from Noisy Timeseries Data”. In: arXiv preprint
arXiv:2202.05403 (2022).
[130] AR Sanaullah, Anupam Das, Anik Das, Muhammad Ashad Kabir, and Kai Shu. “Applications of
machine learning for COVID-19 misinformation: a systematic review”. In: Social Network Analysis
and Mining 12.1 (2022), pp. 1–34.
[131] Karishma Sharma, Emilio Ferrara, and Yan Liu. “Characterizing Online Engagement with
Disinformation and Conspiracies in the 2020 U.S. Presidential Election”. In: Proceedings of the
International AAAI Conference on Web and Social Media. 2022.
139

[132] Karishma Sharma, Emilio Ferrara, and Yan Liu. “Construction of large-scale misinformation labeled
datasets from social media discourse using label refinement”. In: Proceedings of the ACM Web
Conference 2022. 2022, pp. 3755–3764.
[133] Karishma Sharma, Xinran He, Sungyong Seo, and Yan Liu. “Network Inference from a Mixture of
Diffusion Models for Fake News Mitigation”. In: Proceedings of the International AAAI Conference on
Web and Social Media. 2021.
[134] Karishma Sharma, Feng Qian, He Jiang, Natali Ruchansky, Ming Zhang, and Yan Liu. “Combating
fake news: A survey on identification and mitigation techniques”. In: ACM Transactions on
Intelligent Systems and Technology (TIST) 10.3 (2019), pp. 1–42.
[135] Karishma Sharma, Sungyong Seo, Chuizheng Meng, Sirisha Rambhatla, and Yan Liu. “Coronavirus
on social media: Analyzing misinformation in Twitter conversations”. In: arXiv preprint
arXiv:2003.12309 (2020).
[136] Karishma Sharma, Sungyong Seo, Chuizheng Meng, Sirisha Rambhatla, and Yan Liu. “Covid-19 on
social media: Analyzing misinformation in twitter conversations”. In: arXiv e-prints (2020),
arXiv–2003.
[137] Karishma Sharma, Yizhou Zhang, Emilio Ferrara, and Yan Liu. “Identifying Coordinated Accounts
on Social Media through Hidden Influence and Group Behaviours”. In: Proceedings of the 27th ACM
SIGKDD Conference on Knowledge Discovery & Data Mining. KDD ’21. Virtual Event, Singapore:
Association for Computing Machinery, 2021, pp. 1441–1451. isbn: 9781450383325. doi:
10.1145/3447548.3467391.
[138] Karishma Sharma, Yizhou Zhang, Emilio Ferrara, and Yan Liu. “Identifying coordinated accounts
on social media through hidden influence and group behaviours”. In: Proceedings of the 27th ACM
SIGKDD Conference on Knowledge Discovery & Data Mining. 2021, pp. 1441–1451.
[139] Karishma Sharma, Yizhou Zhang, and Yan Liu. “Covid-19 vaccine misinformation campaigns and
social media narratives”. In: Proceedings of the International AAAI Conference on Web and Social
Media. Vol. 16. 2022, pp. 920–931.
[140] Karishma Sharma, Yizhou Zhang, and Yan Liu. COVID-19 Vaccines: Characterizing Misinformation
Campaigns and Vaccine Hesitancy on Twitter. 2021. arXiv: 2106.08423 [cs.SI].
[141] Karishma Sharma, Yizhou Zhang, and Yan Liu. “COVID-19 Vaccines: Characterizing
Misinformation Campaigns and Vaccine Hesitancy on Twitter”. In: arXiv preprint arXiv:2106.08423
(2021).
[142] Shikhar Sharma. “Unsupervised learning”. In: (1986).
[143] Oleksandr Shchur, Marin Biloš, and Stephan Günnemann. “Intensity-Free Learning of Temporal
Point Processes”. In: ICLR. 2020.
140

[144] Karandeep Singh, Gabriel Lima, Meeyoung Cha, Chiyoung Cha, Juhi Kulshrestha, Yong-Yeol Ahn,
and Onur Varol. “Misinformation, Believability, and Vaccine Acceptance Over 40 Countries:
Takeaways From the Initial Phase of The COVID-19 Infodemic”. In: arXiv preprint arXiv:2104.10864
(2021).
[145] Harald O Stolberg, Geoffrey Norman, and Isabelle Trop. “Randomized controlled trials”. In:
American Journal of Roentgenology 183.6 (2004), pp. 1539–1544.
[146] Yuan Sui, Jiaru Zou, Mengyu Zhou, Xinyi He, Lun Du, Shi Han, and Dongmei Zhang. “TAP4LLM:
Table Provider on Sampling, Augmenting, and Packing Semi-structured Data for Large Language
Model Reasoning”. In: arXiv preprint arXiv:2312.09039 (2023).
[147] Zhiqing Sun and Yiming Yang. “An EM approach to non-autoregressive conditional sequence
generation”. In: International Conference on Machine Learning. PMLR. 2020, pp. 9249–9258.
[148] Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi. “Evaluation of
ChatGPT as a question answering system for answering complex questions”. In: arXiv preprint
arXiv:2303.07992 (2023).
[149] Jonathan P Tennant and Tony Ross-Hellauer. “The limitations to our understanding of peer review”.
In: Research integrity and peer review 5.1 (2020), p. 6.
[150] Petter Törnberg. “Echo chambers and viral misinformation: Modeling fake news as complex
contagion”. In: PLoS one 13.9 (2018), e0203958.
[151] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux,
Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. “Llama: Open
and efficient foundation language models”. In: arXiv preprint arXiv:2302.13971 (2023).
[152] Sander Van der Linden. “The conspiracy-effect: Exposure to conspiracy theories (about global
warming) decreases pro-social behavior and science acceptance”. In: Personality and Individual
Differences 87 (2015), pp. 171–173.
[153] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,
Łukasz Kaiser, and Illia Polosukhin. “Attention is all you need”. In: NeurIPS. 2017, pp. 5998–6008.
[154] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and
Yoshua Bengio. “Graph attention networks”. In: arXiv preprint arXiv:1710.10903 (2017).
[155] Soroush Vosoughi, Deb Roy, and Sinan Aral. “The spread of true and false news online”. In: science
359.6380 (2018), pp. 1146–1151.
[156] David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and
Hannaneh Hajishirzi. “Fact or Fiction: Verifying Scientific Claims”. In: Proceedings of the 2020
Conference on Empirical Methods in Natural Language Processing (EMNLP). Ed. by Bonnie Webber,
Trevor Cohn, Yulan He, and Yang Liu. Online: Association for Computational Linguistics, Nov.
2020, pp. 7534–7550. doi: 10.18653/v1/2020.emnlp-main.609.
141

[157] Claudia Wagner, Silvia Mitter, Christian Körner, Markus Strohmaier, et al. “When social bots
attack: Modeling susceptibility of users in online social networks.” In: # MSM. 2012, pp. 41–48.
[158] Gang Wang, Xinyi Zhang, Shiliang Tang, Haitao Zheng, and Ben Y Zhao. “Unsupervised
clickstream clustering for user behavior analysis”. In: Proceedings of the 2016 CHI Conference on
Human Factors in Computing Systems. 2016, pp. 225–236.
[159] Qi Wang, Guangyin Jin, Xia Zhao, Yanghe Feng, and Jincai Huang. “CSAN: A neural network
benchmark model for crime forecasting in spatio-temporal scale”. In: Knowledge-Based Systems 189
(2020), p. 105120.
[160] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang,
Aakanksha Chowdhery, and Denny Zhou. “Self-consistency improves chain of thought reasoning
in language models”. In: arXiv preprint arXiv:2203.11171 (2022).
[161] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le,
Denny Zhou, et al. “Chain-of-thought prompting elicits reasoning in large language models”. In:
Advances in Neural Information Processing Systems 35 (2022), pp. 24824–24837.
[162] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang,
Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. “Ethical and social risks of harm
from language models”. In: arXiv preprint arXiv:2112.04359 (2021).
[163] Mika Westerlund. “The emergence of deepfake technology: A review”. In: Technology innovation
management review 9.11 (2019).
[164] Chenxi Whitehouse, Tillman Weyde, Pranava Madhyastha, and Nikos Komninos. “Evaluation of
fake news detection with knowledge-enhanced language models”. In: Proceedings of the
International AAAI Conference on Web and Social Media. Vol. 16. 2022, pp. 1425–1429.
[165] Steven Lloyd Wilson and Charles Wiysonge. “Social media and vaccine hesitancy”. In: BMJ Global
Health 5.10 (2020), e004206.
[166] Samuel C Woolley and Philip N Howard. Computational propaganda: political parties, politicians,
and political manipulation on social media. Oxford University Press, 2018.
[167] Yuanhao Wu, Juno Zhu, Siliang Xu, Kashun Shum, Cheng Niu, Randy Zhong, Juntong Song, and
Tong Zhang. “RAGTruth: A Hallucination Corpus for Developing Trustworthy
Retrieval-Augmented Language Models”. In: arXiv preprint arXiv:2401.00396 (2023).
[168] Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. “Self-attention with
functional time representation learning”. In: NeurIPS. 2019.
[169] Chenghao Yang and Allyson Ettinger. “Can You Follow Me? Testing Situational Understanding for
ChatGPT”. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language
Processing. 2023, pp. 6385–6398.
[170] Kai-Cheng Yang and Filippo Menczer. “Anatomy of an AI-powered malicious social botnet”. In:
arXiv preprint arXiv:2307.16336 (2023).
142

[171] Kai-Cheng Yang, Onur Varol, Pik-Mai Hui, and Filippo Menczer. “Scalable and generalizable social
bot detection through data selection”. In: Proceedings of the AAAI conference on artificial intelligence.
Vol. 34. 01. 2020, pp. 1096–1103.
[172] Yingguang Yang, Renyu Yang, Yangyang Li, Kai Cui, Zhiqin Yang, Yue Wang, Jie Xu, and
Haiyong Xie. “Rosgas: Adaptive social bot detection with reinforced self-supervised gnn
architecture search”. In: ACM Transactions on the Web 17.3 (2023), pp. 1–31.
[173] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and
Karthik Narasimhan. “Tree of thoughts: Deliberate problem solving with large language models”.
In: arXiv preprint arXiv:2305.10601 (2023).
[174] Savvas Zannettou, Tristan Caulfield, Emiliano De Cristofaro, Michael Sirivianos,
Gianluca Stringhini, and Jeremy Blackburn. “Disinformation warfare: Understanding
state-sponsored trolls on Twitter and their influence on the web”. In: Companion proceedings of
WebConf. 2019, pp. 218–226.
[175] Savvas Zannettou, Tristan Caulfield, William Setzer, Michael Sirivianos, Gianluca Stringhini, and
Jeremy Blackburn. “Who let the trolls out? towards understanding state-sponsored trolls”. In: ACM
WebSci. 2019, pp. 353–362.
[176] Assaf J. Zeevi and Ronny Meir. “Density Estimation Through Convex Combinations of Densities:
Approximation and Estimation Bounds”. In: Neural Networks 10.1 (1997), pp. 99–109. issn:
0893-6080. doi: https://doi.org/10.1016/S0893-6080(96)00037-8.
[177] Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and
Yejin Choi. “Defending against neural fake news”. In: Advances in neural information processing
systems 32 (2019).
[178] Qiang Zhang, Aldo Lipani, Omer Kirnap, and Emine Yilmaz. “Self-Attentive Hawkes Process”. In:
ICML. 2020.
[179] Xuan Zhang and Wei Gao. “Towards llm-based fact verification on news claims with a hierarchical
step-by-step prompting method”. In: arXiv preprint arXiv:2310.00305 (2023).
[180] Yizhou Zhang, Defu Cao, and Yan Liu. “Counterfactual neural temporal point process for
estimating causal influence of misinformation on social media”. In: Advances in Neural Information
Processing Systems 35 (2022), pp. 10643–10655.
[181] Yizhou Zhang, Lun Du, Defu Cao, Qiang Fu, and Yan Liu. An Examination on the Effectiveness of
Divide-and-Conquer Prompting in Large Language Models. 2024. arXiv: 2402.05359 [cs.AI]. url:
https://arxiv.org/abs/2402.05359.
[182] Yizhou Zhang, Karishma Sharma, and Yan Liu. “VigDet: Knowledge Informed Neural Temporal
Point Process for Coordination Detection on Social Media”. In: Advances in Neural Information
Processing Systems 34 (2021).
143

[183] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao,
Yu Zhang, Yulong Chen, et al. “Siren’s song in the ai ocean: A survey on hallucination in large
language models”. In: arXiv preprint arXiv:2309.01219 (2023).
[184] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang,
Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. “Least-to-most prompting enables
complex reasoning in large language models”. In: arXiv preprint arXiv:2205.10625 (2022).
[185] Ke Zhou, Hongyuan Zha, and Le Song. “Learning social infectivity in sparse low-rank networks
using multi-dimensional hawkes processes”. In: Artificial Intelligence and Statistics. 2013,
pp. 641–649.
[186] Kyrie Zhixuan Zhou, Abhinav Choudhry, Ece Gumusel, and Madelyn Rose Sanfilippo. “" Sora is
Incredible and Scary": Emerging Governance Challenges of Text-to-Video Generative AI Models”.
In: arXiv preprint arXiv:2406.11859 (2024).
[187] Ming Zhou, Dan Zhang, Yuandong Wang, Yangli-ao Geng, Yuxiao Dong, and Jie Tang. “LGB:
Language Model and Graph Neural Network-Driven Social Bot Detection”. In: arXiv preprint
arXiv:2406.08762 (2024).
[188] Xiaojin Zhu and Zoubin Ghahramanih. “Learning from labeled and unlabeled data with label
propagation”. In: (2002).
[189] Xiaojin Zhu and Andrew B Goldberg. “Introduction to semi-supervised learning”. In: Synthesis
lectures on artificial intelligence and machine learning 3.1 (2009), pp. 1–130.
[190] Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Ruyi Gan, Jiaxing Zhang, and Yujiu Yang.
“Solving math word problem via cooperative reasoning induced language models”. In: arXiv
preprint arXiv:2210.16257 (2022).
[191] Mingyu Zong and Bhaskar Krishnamachari. “Solving math word problems concerning systems of
equations with gpt-3”. In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. 13.
2023, pp. 15972–15979.
[192] Frederik Zuiderveen Borgesius, Damian Trilling, Judith Möller, Balázs Bodó, Claes H De Vreese,
and Natali Helberger. “Should we worry about filter bubbles?” In: Internet Policy Review. Journal on
Internet Regulation 5.1 (2016).
[193] Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. “Transformer Hawkes
Process”. In: NeurIPS. 2020.
[194] Simiao Zuo, Tianyi Liu, Tuo Zhao, and Hongyuan Zha. “Differentially Private Estimation of
Hawkes Process”. In: arXiv preprint arXiv:2209.07303 (2022).
144

Appendix A
Experiment Details on COVID-19 Dataset
We also present the hashtag distribution of the smaller detected anomalous group in Figure A.1. A surprising
fact is that, although the suspended account ratio in this group is much higher than normal, the hashtags of
this group are not like the contents of researchers’ typical stereotypes about conspiracy. This finding suggests
that coordination may not only happen in the spreading of misinformation in researchers’ stereotypes.
CoronaVirusOutbreak
IR
TousEnsemble
BoycottCAC40 Raoult
StayAtHomeSaveLives
DesobeissanceCivile
leadership
IndiaFightsCoronavirus GeniouxMG
MeaningfulGrowth
DigitalTransformation GiletsJaunes
CautionYesPanicNo 9News
Japan
Germany
Management
HealthForAll OMS
SwasthaBharat
PaliardFranco
GreveGenerale
facemasks
ENDIRECTE
HerdImmunity CoVId19
SputnikUpdates
TestTraceIsolate
CoronaAlert
CoronaCrisis
Ordnungsamt_Wolfsburg Ortsrat_Detmerode
Polizei_Suedstadt WVG
Nötigung
Körperverletzung
RAEconsultas QAnon
DetmerodeRB
Entérate
Perú
TrumpLiesAmericansDie
love
Texas
Israel
WashYourHands
NuevaNormalidad Auspol
Resist
EEUU
0
1000
2000
3000
4000
Figure A.1: The hashtag distribution in the smaller anomalous group (C1).
145

Abstract (if available)

Abstract Over the recent years, public opinion and online credibility have been suffering from the manipulation of campaigns that control malicious accounts to document and spread misinformation with specific narratives such as fake news and conspiracy theories. Such campaigns, also known as misinformation campaigns, are increasingly threatening various areas related to public opinions and decisions, such as politics and public health. Such threats, prominent in highly scrutinized societal events like the U.S. Presidential Elections and the COVID-19 pandemic, have significantly undermined societal trust and public interests.

My thesis will discuss how to exploit machine learning to discover knowledge and skills that help combat this aforementioned social manipulation. More specifically, my thesis will present my research attempts to apply machine learning algorithms in three directions: Manipulation Source Identification, Susceptible Population Recognition, and Automated Authenticity Verification. To identify the online manipulation from misinformation campaigns, my collaborators and I developed a series of neural temporal point process models that can recognize patterns of coordinated manipulators with data-driven learning and domain knowledge. To identify users susceptible to specific misinformation, we developed a counterfactual neural network that can estimate the causal effect of a piece of misinformation on an individual user or a group of populations. To complete our target of automated authenticity verification, we use the advances of large language models (LLM), which can clarify misinformation and provide a reference for accurate information. To achieve this goal, more work is conducted on developing robust prompting engineering strategies to prevent the LLM from being deceived by misinformation when verifying the genuineness of a given text.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Robust causal inference with machine learning on observational data

PDF

Accelerating robot manipulation using demonstrations

PDF

Learning to optimize the geometry and appearance from images

PDF

Building generalizable language models for code processing

PDF

Sample-efficient and robust neurosymbolic learning from demonstrations

PDF

Trustworthy spatiotemporal prediction models

PDF

Learning social sequential decision making in online games

PDF

Socially-informed content analysis of online human behavior

PDF

Planning and learning for long-horizon collaborative manipulation tasks

PDF

Learning at the local level

PDF

Learning controllable data generation for scalable model training

PDF

Artificial Decision Intelligence: integrating deep learning and combinatorial optimization

PDF

Measuing and mitigating exposure bias in online social networks

PDF

Common ground reasoning for communicative agents

PDF

No-regret learning and last-iterate convergence in games

PDF

Robust and proactive error detection and correction in tables

PDF

Predicting and planning against real-world adversaries: an end-to-end pipeline to combat illegal wildlife poachers on a global scale

PDF

Physics-aware graph networks for spatiotemporal physical systems

PDF

High-throughput methods for simulation and deep reinforcement learning

PDF

Data-driven and logic-based analysis of learning-enabled cyber-physical systems

Asset Metadata

Creator Zhang, Yizhou (author)

Core Title Towards combating coordinated manipulation to online public opinions on social media

School Viterbi School of Engineering

Degree Doctor of Philosophy

Degree Program Computer Science

Degree Conferral Date 2024-12

Publication Date 11/14/2024

Defense Date 10/29/2024

Publisher Los Angeles, California (original), University of Southern California (original), University of Southern California. Libraries (digital)

Tag artificial intelligence,causal analysis,machine learning,misinformation detection,social media analysis

Format theses (aat)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Liu, Yan (committee chair), Drakopoulos, Kimon (committee member), Zhao, Jieyu (committee member)

Creator Email drivechang2023@gmail.com,zhangyiz@usc.edu

Unique identifier UC11399DNDM

Identifier etd-ZhangYizho-13632.pdf (filename)

Legacy Identifier etd-ZhangYizho-13632

Document Type Dissertation

Format theses (aat)

Rights Zhang, Yizhou

Internet Media Type application/pdf

Type texts

Source 20241118-usctheses-batch-1223 (batch), University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the author, as the original true and official version of the work, but does not grant the reader permission to use the work if the desired use is covered by copyright. It is the author, as rights holder, who must provide use permission if such use is covered by copyright.

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA

Repository Email cisadmin@lib.usc.edu