Close
About
FAQ
Home
Collections
Login
USC Login
Register
0
Selected
Invert selection
Deselect all
Deselect all
Click here to refresh results
Click here to refresh results
USC
/
Digital Library
/
University of Southern California Dissertations and Theses
/
Diffusion network inference and analysis for disinformation mitigation
(USC Thesis Other)
Diffusion network inference and analysis for disinformation mitigation
PDF
Download
Share
Open document
Flip pages
Contact Us
Contact Us
Copy asset link
Request this asset
Transcript (if available)
Content
DIFFUSION NETWORK INFERENCE AND ANALYSIS FOR DISINFORMATION MITIGATION by Karishma Sharma A Dissertation Presented to the FACULTY OF THE USC GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree DOCTOR OF PHILOSOPHY (COMPUTER SCIENCE) December 2022 Acknowledgements I would like to thank everyone who has been a constant source of strength and support. I am extremely grateful to my advisors Dr. Yan Liu and Dr. Emilio Ferrara, committee members, and labmates for their feedback and discussions. I am fortunate to have two great advisors in Prof Liu who contributed so much to the vision for a well-rounded thesis, and Prof Ferrara who added so much value to the contributions. My committee members over the years, Prof Fred Morstatter, Kimon Drakopoulos, Barath Raghavan, and David Kempe have provided great feedback and suggestions for the proposal and thesis. I would also like to thank and acknowledge the contributions of my invaluable co-authors Yizhou Zhang, Feng Qian, Sungyong Seo, Xinran He, Natali Ruchansky, Chuizheng Meng, andSirishaRambhatla. MylabmatesinbothYanandEmilio’slabhavefurtheredmyknowl- edge in machine learning and data science and inspired me with their strong personalities to really become a well-rounded individual. Special thanks to Luca Luceri and Goran Muric and everyone I worked with or got feedback from in the lab. I am grateful for the support of my friends at USC and outside who have guided me based on their PhD experiences. Ialsowanttothankeverymemberofthedepartment, ourdirectorofstudentaffairsLizsl De Leon without whom it would be impossible to get through challenging moments, and Dr. PremNatarajanandDr. HaipengLuowhohavereallybeensosupportiveintheearlyyears. I will also be forever indebted to Dr. Pinar Donmez, Dr. Zeki Yalniz, Dr. Lokendra Shastri, Dr. Rajiv Gandhi, Dr. Enming Luo, Dr. Umut Ozertem, and other colleagues and teachers I have had the pleasure to work with. Lastly, I sincerely thank my Mother, my family, and mylovedoneswithoutwhoseencouragementandstrengththiswouldnothavebeenpossible. ii Table of Contents Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Challenges in Disinformation Mitigation . . . . . . . . . . . . . . . . . . . . 2 1.2 Thesis Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.3 Summary of Thesis Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.4 Related Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.1 Disinformation Detection and Mitigation . . . . . . . . . . . . . . . . . . . . 11 2.1.1 Disinformation Detection . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.1.2 Malicious Accounts Detection . . . . . . . . . . . . . . . . . . . . . . 13 2.1.3 Network Interventions . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.1.4 Disinformation and Conspiracy Engagements . . . . . . . . . . . . . . 16 2.2 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.2.1 Diffusion Inference and Influence Maximization . . . . . . . . . . . . 18 2.2.2 Neural Diffusion Modeling . . . . . . . . . . . . . . . . . . . . . . . . 23 3 Data Collection and Disinformation Labeling . . . . . . . . . . . . . . . . 26 3.1 Definition of Disinformation . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.1.1 Types of Disinformation . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.1.2 Types of Disinformation Motives . . . . . . . . . . . . . . . . . . . . 27 3.1.3 Disambiguation of Related Terms . . . . . . . . . . . . . . . . . . . . 28 3.1.4 Disinformation Perception and Spread . . . . . . . . . . . . . . . . . 29 3.2 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.2.1 Social Media Engagements . . . . . . . . . . . . . . . . . . . . . . . . 31 3.3 Disinformation Labeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.3.1 Fact-Checked Labels . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.3.2 News Credibility Based-Labels . . . . . . . . . . . . . . . . . . . . . . 36 3.4 Constructing Disinformation Datasets . . . . . . . . . . . . . . . . . . . . . . 36 3.5 Limitations and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 4 Timely and Scalable Disinformation Labeling in New Domains . . . . . . 45 4.1 Disinformation Labeling Guidelines . . . . . . . . . . . . . . . . . . . . . . . 46 4.2 News-Source Credibility Based Weak Labels . . . . . . . . . . . . . . . . . . 47 4.2.1 Weakly-Labeling Using News-Sources . . . . . . . . . . . . . . . . . . 48 iii 4.2.2 Correlation with Fact-Checked Labels . . . . . . . . . . . . . . . . . . 50 4.3 Model-Guided Label Refinement . . . . . . . . . . . . . . . . . . . . . . . . . 51 4.3.1 Self-Supervision from Disinformation Detection Modeling . . . . . . . 52 4.3.2 Social Context Modeling . . . . . . . . . . . . . . . . . . . . . . . . . 53 4.3.3 Iterative Label Refinement . . . . . . . . . . . . . . . . . . . . . . . . 57 4.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.4.1 Evaluation Tasks and Metrics . . . . . . . . . . . . . . . . . . . . . . 59 4.4.2 Experiment Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 4.5 Discussions and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 5 Disinformation Detection Leveraging Social Media Responses . . . . . . 65 5.1 Early Detection Leveraging Historical Responses . . . . . . . . . . . . . . . . 65 5.2 Early Detection Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 5.2.1 Two-Level Convolutional Neural Network. . . . . . . . . . . . . . . . 67 5.2.2 User Response Generator . . . . . . . . . . . . . . . . . . . . . . . . . 68 5.2.3 Integrated Framework of TCNN-URG . . . . . . . . . . . . . . . . . 70 5.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 5.3.1 Experimental Set-Up . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 5.3.2 Detection Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 5.4 Discussions and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 6 Detection of Coordinated Manipulation to Spread Disinformation . . . . 75 6.1 Task Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 6.2 Coordinated Accounts Detection Model . . . . . . . . . . . . . . . . . . . . . 79 6.2.1 Modeling Latent Influence . . . . . . . . . . . . . . . . . . . . . . . . 80 6.2.2 Modeling Hidden Groups . . . . . . . . . . . . . . . . . . . . . . . . . 83 6.2.3 Jointly Learning and Optimization . . . . . . . . . . . . . . . . . . . 84 6.3 Incorporating Domain Knowledge . . . . . . . . . . . . . . . . . . . . . . . . 86 6.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 6.4.1 Experimental Set-Up . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 6.4.2 Results on Coordination Detection . . . . . . . . . . . . . . . . . . . 96 6.4.3 Analysis on Coordination Detection . . . . . . . . . . . . . . . . . . . 99 6.5 Discussions and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 7 Network Inference to Limit Disinformation Propagation . . . . . . . . . . 108 7.1 Analysis of Diffusion Cascades . . . . . . . . . . . . . . . . . . . . . . . . . . 109 7.2 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 7.2.1 Mixture of Independent Cascade Models . . . . . . . . . . . . . . . . 113 7.2.2 Task Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 7.3 Network Inference for Diffusion Mixture Model . . . . . . . . . . . . . . . . . 115 7.3.1 PAC Learnability of Diffusion Mixture . . . . . . . . . . . . . . . . . 115 7.3.2 Parameter Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . 117 7.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 7.4.1 Experimental Set-Up . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 7.4.2 Results and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 iv 7.5 Discussions and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 8 Characterization of Engagement and Interventions . . . . . . . . . . . . . 129 8.1 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 8.2 Data and Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 8.2.1 Inferring Political Leaning . . . . . . . . . . . . . . . . . . . . . . . . 131 8.3 Results and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 8.3.1 Disinformation Topic Modeling . . . . . . . . . . . . . . . . . . . . . 134 8.3.2 Conspiracy Group Engagement . . . . . . . . . . . . . . . . . . . . . 136 8.3.3 Twitter Intervention Effects . . . . . . . . . . . . . . . . . . . . . . . 141 8.3.4 Propagation Dynamics of Cascades . . . . . . . . . . . . . . . . . . . 145 8.4 Discussions and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 9 Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . 149 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 Appendices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 A Disinformation Labeling for Dataset and Analysis . . . . . . . . . . . . . . . 179 B Supplementary Analysis of U.S. 2020 Election Dataset . . . . . . . . . . . . 191 C Proof and Analysis in Coordinated Campaigns Detection . . . . . . . . . . . 195 D Proof of PAC-Learnability and Runtime Analysis of MIC . . . . . . . . . . . 204 v List of Tables 3.1 Disinformation dataset statistics for news articles and Twitter engagements . 37 3.2 Dataset statistics of Election discourse content engagement cascades . . . . . 40 3.3 Results on detection of unreliable/conspiracy cascades in the election dataset 40 3.4 Data statistics for Twitter [88, 104] and Weibo [104] datasets . . . . . . . . . 42 3.5 Dataset of coordinated manipulation campaign in the 2016 US Election . . . 43 4.1 Covid-19 vaccines Twitter dataset (collected Dec 9, 2020 - Feb 24, 2021) . . 47 4.2 (Mis)information communities in the 3-core of the Retweet Graph . . . . . . 54 4.3 Top Tweeted URLs and Top Retweeted Accounts in (Mis)information com- munities in 3-core Retweet Graph. . . . . . . . . . . . . . . . . . . . . . . . . 55 4.4 Results on classification performance on test set from detection model with label refinement proposed approach for disinformation dataset construction on COVID-19 vaccines. Metrics: AP (average precision), AUC (ROC), F1 and Macro F1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 4.5 Results for noise detection in weak labels with label refinement proposed ap- proach for disinformation dataset construction on COVID-19 vaccines. Eval- uation metrics: Rec (noise recall), Prec (precision), FracUQ (fraction of un- wanted queries), F1 (F1 of detected noise in weak labels) . . . . . . . . . . . 61 4.6 Fine-grained classification from human labeled class prototypes to remaining examples in the dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 5.1 Datasetsusedforearlydetection, contentswithengagementsi.e., userresponses 71 5.2 Results on Weibo dataset with different % of training data used . . . . . . . 72 5.3 Results on our Twitter dataset with different % of training data used . . . . 72 6.1 Summary of neural point process models . . . . . . . . . . . . . . . . . . . . 81 6.2 Results on detection of Russian coordinated manipulation (IRA dataset) . . 96 6.3 Ablation on unsupervised coordination detection (IRA) with VigDet . . . . . 98 6.4 OverlapbetweensuspendedTwitteraccounts,andidentifiedcoordinatedgroups/ overall accounts in collected COVID-19 data. (% provided in table) . . . . . 102 6.5 Representative tweets in disinformation topic clusters in identified COVID-19 coordinated accounts groups . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 7.1 Data statistics for Twitter-1 and Twitter-2. Follower graph stats. in Twitter-1110 7.2 Hypothesis testing results (p-values) to verify that average time between en- gagements is higher in disinformation cascades (temporal) and that ratio of connected components to total engagements is higher in disinformation cas- cades (structural) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 7.3 Results on clustering cascades or separability of disinformation from unsuper- vised parameter estimation of the diffusion mixture model . . . . . . . . . . 122 vi 7.4 Results of IC vs. MIC modeling heterogeneous user behaviors in terms of Avg. NLL on train and held-out validation cascades. Lower NLL indicates better fit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 7.5 CharacteristicsofidentifiedinfluentialusersforbothtypesofcascadesTwitter- 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 8.1 Number of accounts labeled as left or right-leaning (by media URLs, account profile description, and human verification) for validation, with error rate (%) in each type based on the inferred political leaning of those accounts . . . . . 134 8.2 Examples of unreliable/conspiracy tweets with most engagements in the data 136 8.3 QAnon conspiracy keywords along with their occurrence frequency in tweets (original,replyorquotedtweetsi.e.,excludingretweets)containingthekeywords137 8.4 Verification of 100 accounts sampled from inferred right/left-leaning accounts posting QAnon associated keywords. Verified as: (Q) is QAnon conspirator; else not (N) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 8.5 QAnon interactions quantified over all accounts in the dataset. Influenced (ID) are accounts that replied, retweeted or quoted tweets from QAnon ac- counts. Influencer (IR) are accounts that were replied, retweeted or quoted by QAnon accounts. Table contains: # Accounts (%) . . . . . . . . . . . . . 138 8.6 Top 10 hashtags that declined (β < 0) in usage by QAnon post Twitter Action, estimatedbyregressiondiscontinuitydesignintop10Khashtagsused by QAnon accounts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 8.7 Top 20 hashtags that increased (β > 0) in usage by QAnon post Twitter Action, estimatedbyregressiondiscontinuitydesignintop10Khashtagsused by QAnon accounts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 8.8 % Decrease in ratio of direct engagements (reply to, retweet, quote) with QAnonaccountstweetstovolumeofQAnonaccountstweets, beforeandafter Twitter Action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 A.1 Misinformation topic clusters word distribution with highest TF-IDF scores . 187 A.2 CSI label and human validation label with unreliable (1), and reliable (0) . . 188 A.3 Model label and human validation label with unreliable (1), and reliable (0) 189 A.4 Model label and human validation label with unreliable (1), and reliable (0) 190 B.1 Regression stats. and goodness of fit for different degree polynomials. Note: *(p-value <= 0.05). AIC, BIC were lower for degree-1 compared to higher degree curves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 C.1 Results on semi-supervised coordination detection (IRA) in 2016 U.S. Election202 vii List of Figures 1.1 Proposed framework for Disinformation Mitigation . . . . . . . . . . . . . . 4 4.1 TweetvolumetimelinefromtheEmergencyUseAuthorization(EUA)inU.S. till U.S. Adults vaccinations for Covid-19 vaccines Twitter dataset . . . . . . 48 4.2 News-sourcecredibilitylabelscorrelationwithfact-checkedclaimbasedlabels on 150 validation and 256 test tweet cascades . . . . . . . . . . . . . . . . . 49 4.3 Proposedapproach: Model-guidedlabelrefinementwithself-supervisionfrom a generic disinformation detection model and social context modeling to con- struct large-scale disinformation labeled social media datasets in a timely manner. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 4.4 Characterization of communities in the 3-core of the Retweet (RT) graph to find misinformed and informed communities to model the social context . . . 53 4.5 Predicted probabilities from detection model after label refinement: correla- tion with true labels on ground-truth fact-checked human-labeled test and validation set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 5.1 Anexampletoshowthatwhyhistoricaluserresponsesonsocialmediacanbe utilized as rich soft semantic labels to help the early disinformation detection from contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 5.2 Model architecture for early detection . . . . . . . . . . . . . . . . . . . . . . 68 5.3 Top 20 response words generated by URG presented in alphabetical order . . 73 6.1 Coordinated accounts suspended by Twitter in COVID-19 data. The time difference of their coordinated activity varies from less than 6 hours to half a week . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 6.2 Architecture of the proposed AMDN-HAGE to model conditional density function of account activities and hidden groups on social media . . . . . . . 79 6.3 The overview of VigDet. In this framework, we aim at learning a knowledge informed data-driven model. To this end, based on prior knowledge we con- struct a graph describing the potential of account pairs to be coordinated. Thenwealternatelyenhancethepredictionofthedata-drivenmodelwiththe priorknowledgebasedgraphandfurtherupdatethemodeltofittheenhanced prediction as well as the observed data . . . . . . . . . . . . . . . . . . . . . 88 6.4 Comparison of iterative optimization and Adam for the joint training objective 98 6.5 Analysis of learned influence strength between Coordinated (“trolls” T) and Normal (non-trolls NT) in the IRA dataset . . . . . . . . . . . . . . . . . . . 100 6.6 Analysis on how influence weights of account pairs vary with time difference. Green points: Normal (NT) account pairs. Blue: Coordinated (“trolls” T) account pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 6.7 Top-35 (most frequent) unique hashtags in tweets of identified coordinated group and normal accounts . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 viii 6.8 Top-35 hashtags of normal and identified suspicious coordinated accounts. Unique ones in each group are highlighted in bold . . . . . . . . . . . . . . . 103 6.9 Tweetsfromapairofaccounts(A,B)inthedetectedcoordinatedgroup. Left: TweetsfromtheTwitterprofileofaccountsAandBsuggestinganti-lockdown and anti-government narratives. Right: Three example tweets from the col- lected dataset, of the same pair of accounts (A, B) suspected of amplifying misinformation by coordinatedly sharing similar agendas . . . . . . . . . . . 104 6.10 Bot score distribution. Mann-Whitney U-Test for suspicious coordinated Bioweapon (CCP) vs. Normal accounts sample (z-score -2.56, p-val 0.00523 < 0.05) and suspicious coordinated (Great Reset) vs Normal accounts sample (z-score -1.35, p-val 0.0869 < 0.1) . . . . . . . . . . . . . . . . . . . . . . . . 105 6.11 Mutualtriggeringeffect(influence). Betweenactivitiesofaccounts, estimated from data by AMDN-HAGE shown as Avg. estimated triggering effect from Influenceraccounts(whoseactivitiestriggerfutureactivitiesintime). Normal accounts have weaker influence patterns (more random activities) compared to coordinating accounts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 7.1 Example of diffusion cascades on Twitter. The plot shows the cumulative frequency distribution of # tweets for (a) legitimate cascade related to the emergency landing of an airliner in Hudson river in 2009 (b) disinformation cascade related to information suggesting that the combination of Coke and Mentos can lead to death, circulated in 2006 . . . . . . . . . . . . . . . . . . 109 7.2 Statistical tests distributions (Twitter-1) . . . . . . . . . . . . . . . . . . . . 112 7.3 Diffusion Mixture Model as Mixture of Independent Cascade (MIC) . . . . . 114 7.4 Resultsonqualityofinfluentialusersselectedbasedontheestimateddiffusion parameters using greedy maximization of each component IC. (a) Twitter-1 and (b) Twitter-2. Inf(T) and Inf(F) are inferred influential users for legiti- mate and disinformation cascades . . . . . . . . . . . . . . . . . . . . . . . . 123 7.5 General characteristics of identified influential users in Twitter-1 . . . . . . . 125 7.6 Intervention analysis. (a, b) Node interventions (c, d) Edge interventions . . 126 8.1 Topic clusters for unreliable/conspiracy tweets with top representative tweets 134 8.2 QAnonaccountsinteractiongraphinactive1.2Maccounts. Edgesareretweets/ quotes/ replies from source to destination node, normalized by # accounts in source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 8.3 Account interactions with QAnon accounts by tweet type and inferred leaning 139 8.4 QAnon activity and account creation timeline with respect to Twitter Action banning QAnon, and Democratic/Republican National Conventions (DNC) and (RNC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 8.5 RDD data plot for example hashtags (Left) “wwg1wga” (Middle) “walka- wayfromdemocrats”(Right)“nomailinvoting”beforeandafterTwitteraction (fraction of hashtag usage in volume of QAnon tweets ’y’ vs. day ’x’) . . . . 144 ix 8.6 Comparison of information propagation dynamics of reliable vs. unreliable/ conspiracycascadesidentifiedinelection-relatedtweets. (a)CCDFofcascade size. (b) Mean breadth to depth ratio. Reliable cascades run broader at shorter depths of cascade propagation trees. (c) Avg. unique users reached at each cascade size with more repeated engagements from same accounts in unreliable cascades. (d) Mean time to reach unique users is higher and more bursty for unreliable cascades . . . . . . . . . . . . . . . . . . . . . . . . . . 145 A.1 Labeling guidelines provided to annotators for weak label refinement. . . . . 180 A.2 Labeling guidelines provided to annotators for weak label refinement. . . . . 180 A.3 Labeling guidelines provided to annotators for weak label refinement. . . . . 181 A.4 Account statistics boxplot for (mis)informed communities . . . . . . . . . . . 181 A.5 Account statistics for communities (a) account age, (b) number of collected vaccine tweets (original, retweet, reply) by account, and Engagements (ac- count retweeted, replied, mentioned by others) in collected vaccine tweets . . 182 A.7 Frequency correlation of side-effects discussed on Twitter compared with that recorded in VAERS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 B.1 Cascade propagation comparison only on news source labeled cascades set . 194 C.1 The silhouette scores of different group number . . . . . . . . . . . . . . . . 201 C.2 Left. Selection of number of clusters based on silhouette scores in COVID-19 andIRAdatasets. Right. DistributionofaccountcreationyearsinCOVID-19 dataset for each identified accounts group or cluster . . . . . . . . . . . . . . 202 C.3 Hashtag distribution for cluster C1 (COVID-19 dataset) . . . . . . . . . . . 203 D.1 MIC: runtime and convergence analysis . . . . . . . . . . . . . . . . . . . . . 206 x Abstract The proliferation of false and misleading information on social media has greatly reduced trust in online systems. Increasing reliance on social media, combined with sophistication in malicious operations to promote disinformation as a tool to influence public opinion and social outcomes has become a significant threat to society. In this thesis, we address chal- lenges in disinformation mitigation by leveraging the diffusion or propagation dynamics of disinformation on social media, using diffusion network inference and analysis. We consider two aspects in disinformation mitigation 1) Detection of disinformation and malicious efforts 2) Interventions to limit disinformation. In this thesis, for the first aspect we focus on timely detection of disinformation. We address the challenge of disinformation labeling in new and evolving domains in a timely and scalable manner. We propose to weakly-label social media posts using news-source credibility analysis, and leverage model- guidedrefinementofweaklabelsfordisinformationlabeling,bymodelinginstancecredibility jointly with user credibility or stance from the content and social context of a post. Fur- thermore, we propose a disinformation detection model for early detection, i.e., before the content propagation. In order to improve early detection, we learn a generative model of so- cial media responses conditioned on the content, leveraging historical social media responses to disinformation contents to enrich semantic understanding of why a content is labeled as disinformation, and thereby improve early detection when only contents without the social media responses are available. Secondly, we investigate how disinformation spreads and pro- pose an unsupervised, generative model for detection of malicious coordinated campaigns employed for opinion manipulation and amplifying the spread of disinformation. The pro- posed model detects malicious groups by learning to infer unobserved or latent influence between accounts’ activities, and their collective group anomalous behaviors from observed activities. The data-driven estimation of latent influence and group behaviors provides large improvements over state-of-the-art methods based on predefined coordination patterns or xi individual behaviors. We can also incorporate domain knowledge with data-driven learning by encouraging consistency between the group assignments using variational inference. For the second part, we focus on interventions to limit disinformation, and characterize disinformation engagement to further inform detection and mitigation strategies. We ad- dress the problem of learning network interventions to limit disinformation propagation and prevent viral cascades by proposing a mixture model to infer diffusion dynamics of disinfor- mation and legitimate contents from observed, unlabeled diffusion cascades. In addition, we use data-driven analysis to characterize engagements and platform interventions for social and political events, i.e., the pandemic, vaccines, and U.S. Election discourse. Using our disinformation labeling and detection methods, we examine disinformation and uncover sus- picious coordinated groups in the pandemic and vaccine data. Furthermore, we investigate engagement with disinformation and conspiracies in the U.S. 2020 Election, and the effect of Twitter’s ban and restrictions on the QAnon conspiracy group. We examine causal changes in content-posting strategies and sustained engagement with a regression discontinuity de- sign. Our findings suggest that conspiracy groups’ sustained activity and alternate content posting strategies pose challenges to mitigation measures. The outcome of this thesis is to improve the characterization of disinformation engagement and manipulation, and leverage diffusion inference to inform timely detection and improve mitigation interventions. xii Chapter 1 Introduction The proliferation of false and misleading information (also commonly referred to as fake news or disinformation) on social media has increased concerns in society due to its ability to influence public opinions. Therefore, computational methods for timely identification and containment of fake news and associated malicious activities on social media platforms involved in spreading such contents are critical to address disinformation mitigation. Disinformationhaspotentialtoinfluencepublicopinionsandmanipulatesocialoutcomes, and can greatly reduce trust in online systems. It has persistently threatened the integrity of elections and democracies around the world [172], and globally targeted public health [26] and social well-being in real-world events such as the COVID-19 pandemic [42]. At an individual level, exposure to disinformation has been found to influence people’s perception ofthetruthandcausedecreasedacceptanceofscienceand pro-socialbehaviors suchasintent to participate in environmental actions to reduce global warming [161, 120]. At a societal level, it has been observed to exacerbate prejudices and ideological separations [73, 160]. Disinformation has always existed and is not a new problem. However, the popularity of social media facilitating rapid, decentralized, anonymous distribution and communication, and easy creation of social bots and fake online identities and news domains, have made the problem much more difficult to combat [48, 86]. The risks associated with disinformation are more significant due to the scale and reach of social media; the last decade itself has seen morethanaten-foldincreaseinsocialmediausage[128]. TheWorldEconomicForumGlobal Risks Report in 2013 entitled “Digital Wildfires in a Hyperconnected World” warned of the increasing danger of disinformation spread by social media, revisited in the 2018 Report. With increasing pressure on social media platforms to enforce content regulation, the relianceonhumanmoderators hasincreased. Facebookhasgloballyhiredcontentmoderators 1 specialized in different languages to review flagged content 1 . In 2017, Germany passed the Network Enforcement Act (NetzDG) to penalize social media platforms with fines of up to 50 million euros if they fail to remove unlawful content within 24 hours (or 7 days) of it being reported 2 . However, human moderation is not a scalable or viable solution. It is expensive and cannot keep pace with the rapid rate of content creation and propagation. Themajorimpactsofdisinformationhavebeeninglobalsocial,economicandpoliticalissues [113]. The most prominent cases of disinformation in recent times are related to the U.S. 2020PresidentialElectionandtheCOVID-19pandemic. Inthisthesis, westudythegeneral problem of disinformation and its specific characterization in the two mentioned events. 1.1 Challenges in Disinformation Mitigation Wediscusschallengesinmitigatingdisinformationandopinionmanipulationonsocialmedia. • First, the nature of social networks facilitates complex content propagation dynamics and interactions between accounts [138]. Interdependent network dynamics result in non- i.i.d observations of a stochastic process consisting of account actions or engagements in continuous time, with past actions influencing future ones, resulting in complex interactions. • Second, content-based solutions [40, 170, 157] need to be adapted to the diversity of disinformation content topics and deception techniques, and require up-to-date fact verifica- tion. Also,disinformationlabelinghasseveralchallenges,suchas,itrequireshumanintensive fact-checking, and disinformation contents include varied degree of falsehood ranging from false, to partly false, misleading, out-of-context and other types of distortion. • Recent years have observed increasing presence of sophisticated coordinated disinfor- mation campaigns, where group of accounts collude externally or internally on the network (i.e. concealed/hiddensuchasstate-backedoperations[7]orovertsuchasconspiracygroups 1 https://www.theguardian.com/world/2018/jan/05/tough-new-german-law-puts-tech-firms-and-free- speech-in-spotlight 2 https://cdt.org/insight/overview-of-the-netzdg-network-enforcement-law/ 2 [158]) to conduct disinformation and influence operations. Hidden coordinating groups are extremely hard to detect and distinguish from organic accounts interested in similar con- tents, and result in many false positives if assumed signatures or pre-defined features are used for detection. Also, coordinated behaviors are inconsistent over time and groups and therefore detection techniques have to be unsupervised in order to be relevant and useful. • The fourth challenge is the necessity to respond to disinformation quickly before it reaches millions of views through timely detection and network interventions [146]. During theCOVID-19pandemic,theconspiracyvideo“Plandemic”got1.8millionFacebookand7.1 million YouTube views before it was removed [5]. Another severe instance was the infamous “Pizzagate” incident wherein physical violence ensued as a result of online disinformation [111]. NetworkinterventionssuchasidentifyingnodestomonitorinthenetworkareNP-Hard [76] and require approximation algorithms and efficient inference methods for estimation. Analyzing the impact of platform interventions is equally challenging [146]. • Lastly, a challenge specific to these kind of problems is the increasingly sophisticated behaviors of social media manipulation over time, which necessitates continuous data-driven investigations ofmaliciousoperations,improvingdetectionandinterventionmodels(e.g. bot detection [25]), and adapting responses to specific real-world events that are disinformation targets (e.g. COVID-19 vaccines disinformation, U.S. 2020 Election). Strategies that apply to one context might not apply to another, for instance, conspiracy groups became signifi- cant with QAnon gaining substantial following in the U.S. 2020 Election, compared to the widespread Russian coordinated interference efforts observed during the U.S. 2016 Election. 1.2 Thesis Statement This thesis aims to leverage information diffusion or content propagation dynamics on social media to improve our characterization of disinformation and social media manipulation, and to develop methods for timely detection and interventions to mitigate disinformation 3 Social Media Discourse and Engagements Detection Malicious Accounts and Groups Network Interventions Characterization of Engagements Intervention Disinformation Definition and Labeling Disinformation Detection Characterization of Mitigation Actions Model-guided disinformation labeling with social context in new domains (WWW 22) Timely Labeling and Detection Coordinated Influence Campaigns Framework Challenges Thesis Research Neural-URG: Conditional generative model for early detection (IJCAI 18) MIC: Diffusion mixture model and inference to limit disinformation propagation (ICWSM 21) AMDN-HAGE: Hidden influence and group behaviors for coordination detection (KDD 21, NeurIPS 21) Covid-19 Vaccine Disinformation Campaigns and Narratives Analysis (ICWSM 22) U.S. 2020 Election Conspiracy Group Engagement and Twitter’s Intervention Analysis (ICWSM 22) Disinformation Propagation Dynamics Social and Political Events Figure 1.1: Proposed framework for Disinformation Mitigation contents and malicious coordinated efforts spreading the disinformation. Fig. 1.1 provides an overview of the proposed framework for disinformation mitigation in this thesis. The proposed solution includes detection of disinformation and manipulation alongwithinterventionsandanalysisformitigation. Indetection,wefocusontimelylabeling and detection of disinformation contents especially in new and evolving domains related to real-world social and political events. Secondly, malicious accounts and groups can be iden- tified from the detected disinformation. But there also exist malicious operations focused on targeted, persuasive narratives for opinion manipulation and disinformation amplification. We address the sophistication and difficulties in detecting such social media manipulation behaviors from observed activity traces with an unsupervised model for identifying coordi- nated influence on the network. The techniques proposed for detection of disinformation and malicious operations are leveraged for mitigation interventions and analysis, which in turn contributes to increasing our understanding and characterization of malicious opera- tions and its evolving sophistication to further inform improvements in the detection and mitigation techniques. For disinformation mitigation, we focus on network interventions to limit the propagation or spread by inferring the diffusion dynamics of disinformation and legitimate contents. Furthermore, we characterize engagements with disinformation corre- spondingtorecentsocialandpoliticalcontexts,andanalyzetheeffectofplatformmitigation 4 actions on the activities and engagements of malicious groups promoting disinformation and conspiracies, with an emphasis on the COVID-19 pandemic and U.S. Elections. 1.3 Summary of Thesis Work Inthisthesis,wereviewchallengesandapproachesindefiningandlabelingdisinformationor distortionoffactscirculatedonsocialmedia[171,104,15]. Disinformationlabelingnormally requires expensive journalistic verification of claims by fact-checking organizations to deter- mine the degree of truth or falsehood. This human-intensive process has limited the ability toscaleandachievetimelydetectionofdisinformationcontents, especiallyinnewandevolv- ing domains. To overcome this limitation of existing approaches to disinformation labeling [15, 104], we propose a framework for constructing large-scale disinformation datasets from social media posts, leveraging weak labels based on news source credibility with machine learning modeling of the instance credibility and its social context. We develop annotation guidelines that can label the more nuanced distortion and conspiratorial claims observed in recent years on social media [45, 146] and still be general enough to be applied to different domains. We also review social psychology theories behind why disinformation persists and spreads mainly from individual and social factors of human inability to accurately discern false information, and normative influence and naive realism that propels individuals to believe ideologically aligned disinformation through repeated exposures [150, 142]. While early detection remains challenging, existing disinformation detection methods tend to leverage useful signals from social media comments to a post to predict its veracity [138, 105]. The obvious drawback of this is that the disinformation detection must wait to collectenoughsocialmediaresponsestoreliablyestimateveracitytradingoffearlydetection. We assume that only the post or content is available for early detection and propose to in- corporatehistoricalsocialmediaresponsestolearnaconditionalgenerativemodeltoinstead simulate social media responses to new contents. These act as soft semantic labels informing why a post is labeled as disinformation, and improves content-based early detection [134]. 5 The increasing use of coordinated efforts orchestrated by state-backed agencies [49] and conspiracygroups[146]tospreaddisinformationandattempttomanipulateopinionsthrough socialmediahavebecomeaglobalthreattodemocracyandpublichealth[143]. Socialmedia companies have reported more of these malicious operations on their platforms consistently in recent years [49]. These sophisticated coordinated operations are a challenge to tradi- tional detection methods. For instance, earlier research is largely focused on individual and automated behaviours of accounts e.g. social bot detection [43, 25] which overlook jointly operated or coordinating groups of human and/or automated accounts colluding to amplify disinformation spread or manipulate opinions on social networks. A few recent techniques study these coordinated campaigns but propose to detect them with strict and limiting assumptions on the pattern of coordination, or the account group being partly known for supervised techniques [124, 1]. These assumptions severely limit the detection performance in terms of both recall and precision, and are not applicable when the coordinated behaviors are known to be inconsistent over time and groups [182]. We address these challenges by proposing a data-driven unsupervised approach to learn coordinated behaviors directly from observedactivitytracesofaccountsonthenetwork. Wemodelthecharacteristicsinherentto coordinationi.e.,latentinfluencebetweenaccountactivitiesandgroupanomalousbehaviors, both of which are jointly estimated from the observed activities [145]. It makes significant advances over prior approaches in coordination detection. We also propose a framework to incorporate domain-knowledge to jointly leverage the data-driven signals and known prior knowledge to further guide the detection, when such domain knowledge is available [185]. Lastly, mitigation actions or interventions on the social network form an important com- ponenttodisinformationmitigation. Inthisthesis,weinvestigateandcharacterizesocialme- dia engagements and the effect of platform interventions from large-scale datasets collected for emerging real-world events [146, 143, 148]. The contributions include one of the first characterizations of conspiracy engagement and effects of platform moderation on content posting strategies and engagements of conspiracy groups. In comparison to earlier studies 6 [129, 45, 125, 141], we provide insights into the effectiveness of platform moderation efforts on conspiracy groups and changes in their activities to counter the imposed restrictions with observational causal analysis [146]. We observe that the diffusion patterns of disinformation and legitimate contents are heterogeneous with statistically significant differences. Also, we propose how to infer propagation dynamics from observed unlabeled activity traces on the network to enable network interventions to mitigate the diffusion of disinformation [145]. We summarize the following main findings and contributions of the thesis, Disinformation Labeling and Timely Detection • We provide a framework to construct large-scale disinformation datasets from social media posts by leveraging scalable news-source credibility labels as weak labels [147]. The proposed framework models the instance credibility and account credibility from social media context to actively refine labels, reducing expensive annotation resources. It has applications in enabling timely detection of diverse disinformation claims with limited human fact-checking especially for emerging disinformation target domains. • Disinformationdetectionmodelsthatusesocialmediaengagementsasfeedbackassume the availability of engagements during inference i.e., prediction for new contents. This limits their applicability to stricter early detection i.e., before the content has spread to users on the social network. We propose to improve early detection when only the content is available by proposing to use only historical social media engagements as semantic soft labels to learn a conditional variational auto-encoder (Neural User Response Generator [134]) to instead simulate social media responses to new content. Coordinated Efforts to Spread Disinformation • We propose an unsupervised approach to detect coordinated disinformation campaigns attempting to amplify disinformation and influence opinions. We infer the diffusion 7 dynamics on the network from observed activity traces of accounts to capture the hid- den mutual influencebetweentheir activitiesand detectcoordinated groupsfrom their collective anomalous behavior (Attentive Mixture Density Network with Hidden Ac- countGroupEstimation[145]model). Themodelusesdata-drivenlearningratherthan assumed coordination behaviors or supervised coordinated account labels to identify group anomalous behaviors, outperforming existing approaches by a large margin. • Theframeworkisagnostictolinguisticandplatform-specificfeaturesallowingittocap- ture more diverse coordinated groups. It generalizes across platforms and languages or countries from where disinformation campaigns originate and is therefore widely applicable. However, it is flexible enough to incorporate linguistic and metadata fea- tures with trivial inclusion of the features in the input observed activity traces. We also demonstrate the integration of prior domain knowledge of coordinated behaviors in the proposed framework using variational inference to optimize the predictions from the data-driven learning to be consistent with the prior knowledge (VigDet [185]). Mitigation Interventions and Analysis • We use the proposed techniques to detect and characterize disinformation and coordi- natedgroupsinlarge-scaledatasetsthatwecollectedfromtheprominentnews-focused social media platform, Twitter. We study social media engagements and discourse for real-world events i.e., the COVID-19 pandemic, vaccines, and U.S. 2020 Election. • We uncovered suspicious coordinated campaigns promoting a ‘Great Reset’ theory that suggests it is a planned pandemic to reset the economy, and a ‘Bioweapon’ theory suggesting that the pandemic and vaccines are an alleged biological warfare [148]. In a large-scale US 2020 Election dataset, we found disinformation on mail-in voting, social mediacensorship, democraticcandidates, BlackLivesMatter, Covid-19andVenezuela based tweets, along with conspiracy groups such as the far-right QAnon group. We 8 observe that the propagation patterns of disinformation differ significantly from true news in that they tend to be less viral and have more connected components, but can get high engagements. The QAnon conspiracy group is engaged especially with active left-leaning and right-leaning accounts both, through bidirectional Reply tweets, and with active right-leaning accounts through Retweet endorsements [146, 144]. • Weobservedthatthediffusioncharacteristicsoflegitimateanddisinformationcontents differstatisticallywiththetimebetweenengagementsandtheproportionofconnected components being higher in disinformation cascades. We exploit this heterogeneity in engagements to model the diffusion process and propose an expectation-maximization method to infer the dynamics from unlabeled observed cascades [144]. We show that network interventions based on the heterogeneous diffusion modeling can improve esti- mationofinfluentialuserscomparedtootherheuristics,inofflinerealdataevaluations. In addition, we analyzed real world social media platform interventions in the case of the Twitter ban and restriction announced on the QAnon conspiracy group in 2020. Using a regression discontinuity design, we found that Twitter’s moderation led to changes in content posting strategies of the conspiracy accounts, wherein the con- spiracy group activity was sustained by existing accounts using alternate hashtags and increasedactivityrates,makingithardertodesigneffectivemitigationstrategies[146]. 1.4 Related Publications • Sharma, Karishma, Emilio Ferrara, Yan Liu. Construction of Large-Scale Misin- formation Labeled Datasets from Social Media Discourse using Label Refinement. In Proceedings of the ACM Web Conference, WWW 2022. • Sharma, Karishma, Emilio Ferrara, Yan Liu. Characterizing Online Engagement with Disinformation and Conspiracies in the 2020 U.S. Presidential Election. In Pro- ceedings of 16th International AAAI Conference on Web and Social Media, 2022. 9 • Sharma, Karishma, Yizhou Zhang, Yan Liu. COVID-19 Vaccine Misinformation Campaigns and Social Media Narratives. In Proceedings of 16th International AAAI Conference on Web and Social Media, ICWSM 2022. • Zhang,Yizhou,KarishmaSharma,andYanLiu. VigDet: KnowledgeInformedNeu- ral Temporal Point Process for Coordination Detection on Social Media. In Advances in Neural Information Processing Systems 35, NeurIPS 2021. • Sharma, Karishma, Yizhou Zhang, Emilio Ferrara, and Yan Liu. Identifying Coor- dinated Accounts on Social Media through Hidden Influence and Group behaviors. In Proceedings of the 27th ACM SIGKDD International Conference on Knowledge Dis- covery & Data Mining, pp. 324-334. 2021. • Sharma, Karishma, Xinran He, Sungyong Seo, Yan Liu. Network Inference from a Mixture of Diffusion Models for Fake News Mitigation. In Proceedings of 15th Inter- national AAAI Conference on Web and Social Media, ICWSM. 2021. • Qian,Feng,ChengyueGong,KarishmaSharma,andYanLiu. NeuralUserResponse Generator: Fake News Detection with Collective User Intelligence. In Proceedings of 27th International Joint Conference on Artificial Intelligence . 2018. • Sharma, Karishma, Feng Qian, He Jiang, Natali Ruchansky, Ming Zhang, Yan Liu. Combating fake news: A survey on identification and mitigation techniques. ACM Transactions on Intelligent Systems and Technology (TIST). 2019 ; 10(3):1-42. 10 Chapter 2 Preliminaries In this chapter, we review related works in disinformation detection, detection of malicious disinformation promoters including social bots and coordinated disinformation campaigns, along with mitigation mechanisms based on network interventions, and disinformation and conspiracy group engagements on social media. In the preliminaries, we discuss diffusion processes on social networks, specifically diffusion modeling, inference, and influence maxi- mization, used in detection and interventions, followed by neural network advances. 2.1 Disinformation Detection and Mitigation 2.1.1 Disinformation Detection Content-BasedDetection Content-baseddisinformationdetectionleveragesdiscrimina- tive features extracted from the legitimate and disinformation contents, including false and manipulated texts, images, videos, and multi-modal contents [77] to learn a model for disin- formation detection. In text analysis, two prominent approaches involve deception detection basedonwritingstyles[123,166],anddirectclaimverificationusingexternalknowledgefrom the web [132]. The writing style analysis traditionally used linguistic cues for deception de- tection e.g. SCAN [32] which examines cues such as emotive words, objective or subjective words, pronouns, first person, singular, and past tense verbs. Later works have considered features based on LIWC psycho-linguistic cues [48], part-of-speech tags [123] where more verbs, adverbs, pronouns and pre-determiners were signals of deceptive writing, and more nouns, adjectives, prepositions, determiners and coordinating conjunctions were indicators of legitimate contents. The linguistic features from the contents are used to train a discrimi- native classifier for disinformation detection. In later works, instead of pre-defined linguistic 11 features, extraction of features directly from raw content representations, using recurrent or convolutional networks is employed [170]. In work on claim verification using external knowledge from the web, proposed techniques focus on retrieval of articles from the web related to a particular claim, and learn the stance (supporting/refuting the claim) of each article to the claim, and use the aggregate stance to determine the credibility of the claim from the supporting evidence [131]. In multi-modal contents, detection of tampered images is proposed by learning consis- tency between image patches using convolutional neural network based architectures [66], or recurrent convolutional networks for video frames are trained to distinguish fake (artificially generated such as DeepFakes or manipulated videos [58]. Multi-modal content representa- tions with joint representation of text and image or other contents are modeled using shared layers in neural network architectures trained for detection [77]. Most of the approaches require labeled datasets for training [77], while some rely on self-consistency learning only from set of real images [66]. Feedback-Based Detection Disinformation content is crafted for deception and hard to detect for both human and machines as it requires dynamic knowledge of facts, diverse con- textualdatasets,andabilitytodetectnuancedandsophisticatedcontentmanipulations[142]. Addressingthesedrawbacks, existingworkshavefoundthatincorporatinguserengagements with the content on social media (also called social context) are useful for detection. Exist- ing work incorporating user feedback for fake news detection include user stance detection methods [192], and utilizing user comments in time series [103, 104] and propagation tree based methods [105, 106]. The diffusion patterns and user feedback captured in the time- ordered sequence of user comments is largely used in all of these methods, modeled using Recurrent Neural Networks (RNN) for feature extraction from the text and temporal obser- vations. Additionally to incorporate group behaviours, Ruchansky et al. [138] added a user similarity graph based on co-participation of users in engagement with different contents. 12 Few works also leverage crowd-sourced feedback through reporting or flagging mechanisms on social media platforms, where they model the rate of flags and propagation of contents to prioritize verification [78], and jointly model the flagging accuracies of users as an online es- timation problem [159]. Feedback-based methods capture the collective judgement of social mediausersabouttheveracity, malicioususerbehaviors, anddiffusionpatternsofsocialme- dia users. This thesis contributes to this line of work by leveraging social engagements and addressing challenges specific to early detection, network interventions, diffusion analysis, and detection of coordinated disinformation. 2.1.2 Malicious Accounts Detection Detection of Social Bots While detection of disinformation contents are important, de- tection of malicious accounts promoting disinformation is equally necessary to address the problematitssource. Asignificantareaofresearchinthisdirectionisthedetectionofsocial bots and fake accounts [43]. Social bots are automated and semi-automated/software con- trolledaccountsanddetectionofmalicioussocialbotsattemptingtomanipulatesocialmedia is conducted largely from account and network features with classifiers trained on labeled social bot datasets. The human annotations for such labeled datasets are done by examin- ing the account behaviors such as template-based or bot-like tweeting patterns, systematic deletion of tweets, fake followers, spam bots, self-promoters, re-posters. Recent works have also inspected the evolution of social bots [25] and proposed ensemble of supervised classi- fiers trained on different social bot annotated datasets, to improve generalization to novel social bots with more sophisticated behaviors [140]. Analysis of prevalence of social bots, their tweet characteristics, and interactions with human accounts are conducted in different contexts [162, 45, 42]. DetectionofCoordinatedCampaigns Theproblemofdisinformationandsocialmedia abuse has reduced trust in online platforms, and resulted in efforts to understand accounts 13 involved in its spread [43]. Different from social bots and individual malicious accounts, a growing area of research is the detection of coordinated accounts (disinformation or influ- encecampaigns, alsocalled“trollfarms”), orchestratedbygroupofhumanandbotaccounts that are made to work jointly to promote disinformation or persuasive narratives [101, 107]. Coordinated disinformation campaigns have become relevant in recent times, with earliest evidenceofthesemaliciousoperationsintheU.S.2016Election. U.S.Congressinvestigation revealed Russian accounts operated by the Internet Research Agency in Russia, attempt- ing to manipulate the election and political discourse with disinformation, persuasion and polarized narratives [107]. While much of the work in detecting coordinated campaigns as- sumessynchronizedactivities[18]orsimilarityincontentsharingactivities,orotherassumed features [124], other works assume part of the coordinated accounts are known and try to detect the rest using account and network features to train a supervised classifier on the known accounts in the coordinated group [1, 101]. This results in high false positives, strict assumptions, and heuristic solutions. The main techniques can be grouped based on the following. Individual behaviours. Existing works mainly uses two kinds of individual behaviours. The first one is the participation in disinformation spreading [144, 138]. For instance, [138] proposeafakenewsdetectionmodelthatassignsasuspiciousnessscoretoaccountsbasedon theirparticipationinfakenewscascades[138]. Thesecondkindareindividualcharacteristics such as deceptive linguistic features, number of shared links, hashtags and device of posting and cross-platform activity [1, 67, 182]. Apart from above pre-defined features, the activity traces of troll accounts have been found useful for understanding malicious behaviours. In recentwork,thetweet,retweetandreplypatternsofTwitteraccountsareutilizedtoinferthe incentives or rewards behind their activities, formulated as an inverse reinforcement learning problem [101]. Based on the estimated rewards, the authors found that the behaviours of trolls was different as they appeared to perform their activity regardless of the responses. Collective behaviours. Approaches that examine the collective or group behaviours as a 14 wholetodetectanomalousmaliciousaccountsarerelatedtoourapproach. [18]and[59]clus- ter accounts that take similar actions around the same time, based on the assumption that malicious account activities are synchronized in time [18, 59]. Other works cluster or parti- tion a account similarity graph defined over hand-crafted features assumed to be indicative of coordinated behaviours, including the sequence of hashtags or articles shared collectively by a large group of accounts [124, 168]. The significant limitation of such approaches is that the assumptions or hand-crafted features used to define coordination might not hold. 2.1.3 Network Interventions Relatively fewer works address intervention mechanisms to mitigate fake news [62, 37, 53]. These methods provide algorithmic mechanisms to select subset of accounts or nodes in a networkformonitoringtoidentifydisinformation[4,180],ortowardstriggeringinterventions to accelerate or decelerate the diffusion of legitimate or disinformation contents. Forinstance,Nguyenetal.[119]modelthediffusionprocessusingcommonmathematical modelsofnetworkdiffusion, andproposeagreedyalgorithmtoselectsubsetofnodestotrig- ger the diffusion of true news from, such that the objective of reaching or decontaminating at least a β -fraction of the users exposed to disinformation can be achieved. Similarly, other approachesmodelthesimultaneousdiffusionofbothlegitimateanddisinformationcontents, and find nodes to trigger true news spread from, such that exposures to true news over disinformation is maximized, assuming that the originating nodes (seed sets) of the disinfor- mation cascade are known, when optimizing for the true news node set selection [16, 62, 37]. Neither of these works focus on learning or challenges in learning the diffusion dynamics from observed data cascades, and instead assume it is known under the assumed diffusion model. Secondly, these intervention mechanisms rely on already having identified disinfor- mation contents, and the nodes/accounts from which the disinformation cascade originated. We address these drawbacks and focus on the learning aspect of network interventions. 15 2.1.4 Disinformation and Conspiracy Engagements Elections Several researchers investigate different research questions pertaining to disin- formation and conspiracy groups on social media. Studies of disinformation related to US Elections [172, 7] such as analysis of narratives and targets of known accounts from Rus- sia’s coordinated campaign “troll farm” in 2016 US Election have been conducted, with attempts to identify users susceptible to such manipulation campaigns [8], determining that their political ideology, bot-like behaviors and number of followers and tweets were predic- tive of accounts spreading content of Russian coordinated operators, as well as studies of disinformation and social bots in Elections of other countries are conducted [172, 43, 45]. Recent works have studied far-right QAnon conspiracy group on social media platforms that were highly active online and present in offline rallies in the U.S. 2020 Elections, and eventuallybannedfromTwitterandFacebook, aslateasOct2020[28]. deZeeuwetal.2020 traced the normification of QAnon from fringe online communities to mainstream news and socialmedia,usingcross-platformanalysis. OtherresearchershavestudiedQAnonnarratives and contents on different platforms [70, 125] finding high activity and baseless political conspiracies promoted by the group, promoted as “Q-drops” or insider information about Deep State or hidden operations by left-leaning political candidates. Addressing conspiracy groups, Phadke et al. [129] investigated what individual and social factors promote users to joinconspiracygroups. TheirfindingsonRedditsuggestthatdirectinteractions(i.e.,replies) with conspiracists is the most important social precursor for joining (i.e., first contribution to) conspiracy groups. In our work, we focus on prevalence of social media engagements with QAnon accounts and effectiveness of the Twitter ban imposed on QAnon. Pandemic DuringtheCOVID-19pandemic, therehasbeenwidespreadconcernregarding exposures and engagements with disinformation. Disinformation attempts to manipulate public information and opinion on vaccines and the protocols like masks, social distanc- ing, lockdowns have been rampant along with political conspiracies about Bill Gates, the 16 pandemic being a hoax, hydroxychloroquine promotion and the like [143]. COVID-19 dis- information and engagements have been studied in several recent works. Silva et al. [152] investigated which factors are most predictive of engagements with factual and misleading tweets about COVID-19, finding follower-friend ratio, banner image or URL in user profile, presence of image in the tweet, as most relevant. Bot-analysis also suggests presence of activity from social bots in promoting COVID-19 conspiracies [42, 100, 45]. In our work, we found suspicious malicious coordinated efforts to promote political conspiracies, and anti-mask, anti-vaccine propaganda in COVID-19 social media engagements collected from Twitter. Vaccine hesitancy and disinformation on social media and e-commerce platforms has gained much attention in the past few years [24, 74]. Cossard et al. [24] studied the Ital- ian vaccine debate finding echo chambers of anti-vaccine and pro-vaccine groups in 2016 on Twitter,withinteractionbetweenthecommunitiesbeingasymmetrical,asvaccineadvocates ignoretheskeptics. Similarly,Miyazakietal.[114]foundanti-vaccineaccountsreplyingmost to neutral accounts using toxic and emotional content. Studies have found the growing de- bateaboutmeritsofvaccinationonsocialmediatobeaccompaniedbyreducedvaccinations, reduced intent to vaccinate and reappearance of diseases like measles [155, 24, 122]. Memon and Carley [110] characterized health disinformation about COVID-19, among other simi- lar studies. Given the widespread disinformation, government agencies such as the CDC, and fact-checking websites, and communication studies [92] have curated lists of COVID-19 vaccine myths on their websites to inoculate the public against vaccine disinformation. Recenteffortshavestudiedimpactsofvaccinedisinformationusinguserstudysurveys[72, 35, 154, 130]. They found that disinformation and conspiracies are correlated with increased COVID-19 vaccine hesitancy, and the effects are different across different demographics. Moreover, Jamison et al. [69] also found that vaccine opponents share greater proportions of unreliable information, where vaccine opponents shared the greatest proportion (35.4%) of unreliable information including a mix of conspiracy theories, rumors, and scams. Vaccine 17 proponents shared a much lower proportion of unreliable information (11.3%). Pierri et al. [130]reportedhigherhesitancyanddisinformationinRepublicancounties,butlargerchanges in hesitancy with disinformation rate in Democratic counties. Dataset of English Tweets related to COVID-19 vaccines [29] and longitudinal dataset of accounts promoting anti- vaccine hashtags sampled from Twitter discussions [115] have been curated recently by the research community. Lastly, another line of work investigates effects of platform actions on vaccine disinformation where Sharevski et al. [141] examined Twitter’s soft moderation efforts i.e., warning labels and covers on Tweets with unreliable information, and found that warning covers work but not labels in reducing perceived accuracy, through participant study. Kim et al. [79] considered YouTube’s interventions on likely disinformation videos, observing reduced traffic on it. 2.2 Preliminaries 2.2.1 Diffusion Inference and Influence Maximization Information diffusion or propagation is widely studied using probabilistic models in domains related to viral marketing [30], and disease and epidemics [118]. Diffusion models provide a way to solve important computational problems in each domain. For instance, Domingos and Richardson [30] addressed an important question in viral marketing, that is - to trigger a large cascade of product adoptions, who are the most influential users to target in ad campaigns? Such problems can be efficiently solved using submodular optimization under certain diffusion models such as the Independent Cascade model [76]. The choice of model dictates how efficient it is to optimize for important problems such as this. It also affects whether it is possible to derive analytical solutions for learning algorithms in order to infer the parameters of the diffusion model from real observed cascades. We discuss the basics of diffusion modeling and inference with commonly used models in social network analysis. 18 Cascade An engagement or information cascade is generally defined as a time-ordered sequence of user responses/ engagements that a piece of information (content) receives, when it is circulated on a social network. It can be labeled as a legitimate or disinformation cascade, in accordance with the veracity of the propagating content. In most cases, we will useengagementcascadesthat represent theuseridand timestamp oftheengagement(tweet or post) in the diffusion cascade, as follows. At times the content of the tweet might be associated with each engagement (event) and it represents the activity traces of accounts on the network. C = [(u 1 ,t 1 ),(u 2 ,t 2 )··· ] Diffusion Models We discuss the common diffusion models first, namely Independent Cascade(IC)andTemporalPointProcess(TPP).OthermodelssuchastheLinearThreshold model [76] which is also a probabilistic triggering model like the IC model, and SIR for epidemic modeling [118], or its adaptation SEIZ for rumor modeling [71] are also presented inpreviousworks. Thelattertwoarebasedondifferentialequationsoftherateatwhichusers become exposed and infected from susceptible or removed states in the diffusion process. Independent Cascade (IC) Model First, we discuss the formulation of the IC model [76]. G = (V,E) is the directed graph with n =|V| number of nodes (users) and m =|E| edges. A node is activated in an information cascade, if its user has an engagement with the content being propagated. Each edge (u,v)∈E is associated with a parameter p u,v ∈ [0,1]. The diffusion process starts with an initial set of seed nodes assumed to be activated at the first timestep. At each following time step of the diffusion process, a node u activated at the previous time step t, independently makes a single activation attempt on each inactive neighbor v. The activation succeeds with probability p u,v and a node once activated remains activated in the diffusion process. The influence function σ is a function of the seed set S and σ θ (S) is defined as the expected number of nodes activated by the end of the diffusion process starting at seeds S, where θ ={p u,v |(u,v)∈ E} refers to the parameter set. It is a 19 discrete time model, where the diffusion proceeds in discrete timesteps, however, later works have also considered continuous time activation functions on edges [55]. TemporalPointProcessAtemporalpointprocess(TPP)isastochasticprocesswhose realization is a sequence of discrete events in continuous time t ∈ R + [27]. The history of events in the sequence up to time t are generally denoted as H t = {(u i ,t i )|t i < t,u i ∈ U} whereU representsthesetofeventtypes(here,accounts). Theconditionalintensityfunction λ (t|H t ) of a point process is defined as the instantaneous rate of an event in an infinitesimal window at time t given the history i.e. λ (t|H t )dt =E[dN(t)|H t ] where N(t) is the number of events up to time t. The conditional density function of the i th event can be derived from the conditional intensity [33] as p(t|H t ) =λ (t|H t )exp − Z t t i− 1 λ (s|H t )ds ! (2.1) In social network data, the widely used formulation of the conditional intensity is the mul- tivariate Hawkes Process (HP) [187], defined as λ i (t|H t ) = µ i + P t j <t α i,j κ (t− t j ), where λ i (t|H t ) is the conditional intensity of event type i at time t with base intensity µ i > 0 and mutually triggering intensity α i,j > 0 capturing the influence of event type j on i and κ is a decay kernel to model influence decay over time. µ and α are learnable parameters. MultivariatepointprocessorHawkesprocessmodelsarealsowidelyusedinsocialnetworks. Diffusion Network Inference Network inference refers to the problem of inferring the diffusion process, under a mathematical model of propagation, from observed information cascades. The objective of network inference is to estimate the parameters of a diffusion model from observed information cascades, which might entail inferring the edges of the diffusion network, or both the edges and the strength of influence (or weights) on the edges. Forinstance,intheIndependentCascademodel[76],foreverypairofusersuandv,thereisa parameterp u,v which represents the probability with which u activatesv, that is information successfully propagates from u to v. In other words it is the strength of influence between 20 u and v. In the multivariate Hawkes process model, parameters α u,v ≥ 0 model mutually- exciting nature or mutual influence between accounts, with conditional intensity functions capturing the instantaneous rate of future events conditioned on past events or network activities. Earlier works sometimes considered topic or time specific networks [176, 169, 61], such as MultiCascades [61] wherein heterogeneous diffusion models are tied together with joint network priors, and inferred from observed but labeled cascades indicating which topic or time specific networks the cascade corresponds to. Inference is meant to capture latent diffusion strengths from observed data. For the discrete IC model, Saito et al. [139] provide an EM estimation algorithm based on maximum likelihood estimation from observed cascades. The model the unobserved variables as parents that could have activated a given node in a given cascade. Through reparameterization of the edge activation parameters as θ uv =− log(1− p uv ), Netrapalli and Sanghavi [117] show that the log-likelihood can be written as a convex optimization with subproblems corresponding to estimation of parameters of out-edges of each node. Gomez- Rodriguez et al. [55] also find the log likelihood of cascades under the continuous time models (withcontinuous edgeactivationdelay distributions)is a convex functionof the edge parameters for several delay distributions including Exponential, Power Law and Rayleigh distributions, and can be solved with convex optimization procedures. For temporal point process, it is seen that directly optimizing the log-likelihood under the point process model does not have a closed-form solution and numerical optimization leads to computational difficulties in large multi-dimensional networks [164]. Therefore, EM algorithm is generally applied, inferring the posterior distribution of latent variable associated with which previous activity triggers its occurrence, and then updating the parameters in EM [153]. Zhou et al. [187] proposed LowRankSparse to extend the MLE with regularizers to infer low-rank and sparse influence matrix in the Hawkes process model, to capture community structure and reduce the parameters to improve the inference. 21 Influence Maximization Influence maximization is the problem of identifying a subset of users (seed nodes) that can trigger the largest cascades. Formally, under the IC model it is defined as argmax |S|≤ k σ θ (S) i.e., seed set S that provides the largest expected number of activated nodes under IC model with parameters (edge activation probabilities) represented by θ . Kempe et al. [76] prove that this objective is a monotone, submodular set function under the IC model and therefore can be optimized by a greedy algorithm with a (1− 1/e) approximation guarantee, provided in Algorithm 1. Estimation of the influence function (expected cascade size) in the algorithm requires Monte Carlo simulations of diffusion cas- cades starting from the seed set over the learned or assumed diffusion model parameters. In each iteration, the algorithm has to compute the expected cascade size for each node in the network. To reduce computation, future work proposed efficient implementations that exploited submodularity of the influence function under the IC model proposing CLEF [91] and CLEF++ [56]. The main insight was that the marginal gain of a node in the current iteration cannot be better than its marginal gain in the previous iterations, based on sub- modular set functions, since the size of the set in the current iteration is greater than in the previous iteration by one. Therefore, nodes are maintained in decreasing order of the marginalgainswrtthecurrentseedsetinanyiteration. Ifaddingthetopnodetothecurrent seed set results in marginal gain that maintains its position at the top, marginal gains of other nodes need not be re-evaluated, reducing the number of expensive computations. Similar to influence maximization in cascade models such as IC model, in point process models triggering account activities is also an optimization problem which aims to find the optimal intervention intensity (external intensity added to the conditional intensity func- tion, over the base intensity and the mutual excitation intensity from past activities) in the Hawkesprocess[36]. Theconvexoptimizationproposedin[36]attemptstomaximizedesired expectedintensityorotherutilityfunctionsoftheexpectedintensityofactivitiesatdifferent nodes in the multivariate Hawkes model. Farajtabar et al. [37] later proposed a multi-stage intervention framework using reinforcement learning to optimize for optimal intervention 22 Algorithm 1 IC: Influence Maximization Greedy Algorithm Require: Initialize S 0 =∅ 1: for i = 1,2,··· k do 2: Select u∈V \S i− 1 maximizing marginal gain σ θ (S i− 1 ∪{u})− σ θ (S i− 1 ) 3: S i =S i− 1 ∪{u} 4: end for 5: return S k intensity of the point process in stages, to maximize the expected cumulative reward such as minimizing the difference or maximizing the correlation of exposures in simultaneous diffusion of two point processes, each represented by a multivariate Hawkes process. 2.2.2 Neural Diffusion Modeling Recent advances also incorporate neural networks for representation learning in diffusion modeling. For instance, Bourigault et al. [13] propose embedded IC which factorizes the edge activations in the IC model as the dot product of vector representations corresponding to the associated nodes. For learning the authors extend the EM algorithm [139] for the IC model with gradient based optimization of the expected log-likelihood. A few works such as DeepCas [93], extract the diffusion cascade features using bidirectional recurrent neural networks, andusethelearnedrepresentationofthecascadeforpredictingcascadeproperties such as the increment in cascade size in a fixed time interval. DeepHawkes [17] is similar to DeepCas, but includes a weighted pooling based of event representations in the sequence basedontimedifferencebetweentheeventandobservationtime(tocapturetheeffectofthe temporaldecaykernelinHawkesprocesses). However,thesepredictivemodels(insteadofthe generative approaches to diffusion modeling discussed earlier) do not capture stochasticity, which is an inherent and key aspect of diffusion models. The more useful neural diffusion models integrate neural representation learning in the generative framework. Neural Point Process models incorporate representation learning into generative point process models. Recurrent marked temporal point process (RMTPP) [33], models the tem- poral sequence of events/activities in the cascade using recurrent neural networks, and uses 23 theextractedhistoryrepresentationtocalculatetheconditionaldensityofthepointprocess. The conditional intensity is modeled as, λ (t|H t ) = exp(v ˙ h j +w(t− t j )+b) wheretheexponentialfunctionensurestheintensityispositive, and v,w,bareparametersof thenetwork,h j istheencodingoftheeventhistory. Thefirsttermcaptureshistoryinfluence, second term is the time difference for the current influence decay, and b is the base intensity. Due to the exponential function, the conditional density and log-likelihood have a closed form solution, and the maximum likelihood estimation is obtained using gradient descent to learn the parameters of the network. Marks (event types) are included as inputs in event history, and separately predicted from the encoded history for future events. NeuralHawkes [109] extend this formulation to continuous time recurrent neural net- works, and relaxes the exponential function to softplus to represent the multivariate point process (instead of a marked 1d point process), wherein each event type has its own con- ditional intensity representation, with mutual influence between the activities of all event types encoded in the history. However, since the conditional intensity no longer is an ex- ponential functional form, the integral in the log-likelihood needs to be approximated with Monte Carlo (MCMC) sampling during gradient-based optimization for learning the model parameters. Similarly,Self-AttentiveHawkesProcess(SAHP)[184]andTransformerHawkes Process (THP) [194] extend this with masked self-attention [163] to capture long-range de- pendencies in the history more effectively than recurrent networks. However, these methods also require sampling to estimate the integral of the likelihood function. This leads to more noisy approximation of gradients in training the model. On the other hand, HP [187] and RMTPP [33] which define specific functional forms, e.g. exponential for intensity parameterization to have closed form likelihoods but are limited in the flexibility of the selected functional form of the intensity. LogNormMix [149] and FullyNN [121] alle- 24 viate these issues by directly parameterizing the conditional density and cumulative density function instead, getting flexibility and closed-form likelihoods. Shchur et al. [149] use a mixture of 1-dimensional log-normal distributions directly to model the conditional prob- ability density function of the next event time given the history of events in the cascade. The parameters of the log-normal distribution are represented as learnable functions of the history representation encoded with recurrent networks (e.g. a single layer MLP on the his- tory vector with softmax activation to model the mixture weights of the log-normal mixture distribution). Like RMTPP, the mark associated with each event is separately predicted as a function of the encoded history, where the event type and mark are assumed to be conditionally independent given the history. However, the neural network parameterizations in RMTPP, LogNormMix, FullyNN are based on recurrent networks and cannot be used to examine the influence between events and event types, which is possible in the other meth- ods. In this thesis, we extend modeling of the conditional density with interpretable neural network parameterizations so that the influence structure can be learned and examined. 25 Chapter 3 Data Collection and Disinformation Labeling In this Chapter, we discuss the details for data collection related to social media engage- ments and discourse, followed by challenges in labeling disinformation contents. Labeled disinformation datasets are required for training and evaluating detection and intervention methods, and for disinformation analysis. Before, we discuss common approaches to collect labeled datasets, we define disinformation or fake news and disambiguate related concepts. The term fake news might have originated with respect to news shared on social media, however its usage has evolved to encompass any form of distortion of narratives or facts to either intentionally or unintentionally deceive people [23]. In the literature, the intentional distortion of facts is generally termed as disinformation, whereas the term misinformation is used regardless of the intention [86]. Misinformation/disinformation are often used inter- changeably and as an umbrella term naturally encompassing false, fabricated, inaccurate, misleading, and out-of-context information, including false rumors, hoaxes and conspiracies. In this thesis, we use the broader term disinformation to refer to false, misleading and other forms of distorted narratives or facts, unless we need to specifically disambiguate them. 3.1 Definition of Disinformation Although originally used to reference false and often sensational information disseminated undertheguiseofnewsreporting 1 , theterm“fakenews”since2016hasbecomesynonymous with the spread of any form of false and misleading narratives [23]. Fake news was tradi- tionally used to refer to “a news article that is intentionally and verifiably false” [3, 150] or “information presented as a news story that is factually incorrect and designed to deceive the consumer into believing it is true” [54]. However the existing definitions are narrow, 1 as defined in the Collins English Dictionary 26 restricted either by the type of information or the intent of deception, and do not capture the broader scope of the term based on its current usage. We define fake news as follows, Definition 1. A news article or message published and propagated through media, carrying false or misleading information regardless of the means and motives behind it. 3.1.1 Types of Disinformation This definition allows us to capture the numerous different types of fake news identified in earlier studies [171, 142], which we can summarized along different axes of distortion as follows. First, disinformation can be differentiated by the means employed to falsify or distort facts or information as identified in previous journalistic studies [171]. • Fabricated content (completely false or made-up) • Misleading content (misleading use of information to frame an issue) • Imposter content (falsely impersonating genuine sources) • Manipulated content (manipulated images and multi-modal contents for deception) • False connection (headlines, visuals or captions that do not support the content) • False context (genuine content shared with false contextual information) In addition, 5 techniques of science denial, namely, false experts, logical fallacies, impossi- ble expectations, cherry-picking, and conspiracy theories (FLICC) [92] are suggested more specificallyforhealthandsciencedisinformationandoverlapwiththecharacterization[171]. 3.1.2 Types of Disinformation Motives The definition we proposed, also allows us to include different types differentiated by the motive or intent of creation, as listed below from prior work [181]. • Malicious intent (to hurt or disrepute) • Profit (for financial gain by increasing views) • Influence (to manipulate public opinion or social outcomes) 27 • Sow discord (to create disorder and confusion) • Passion (to promote ideological biases) • Amusement (individual entertainment) We can also subdivide false information by intent as misinformation and disinformation. Misinformation refers to unintentionally spread false information which can be a result of misrepresentationormisunderstandingstemmingfromcognitivebiasesorlackofunderstand- ingorattention,anddisinformationreferstofalseinformationcreatedandspreadspecifically with the intention to deceive [86]. But in many instances both terms are used interchange- ablyasumbrellatermsencompassinganykindofdistortion,sincetheintentcannotbeeasily evidenced from observations, and are jointly referred to as mis/disinformation. 3.1.3 Disambiguation of Related Terms Another type of information that might be closely connected to fake news is satire - satire presents information that might be factually incorrect, but the intent is not to deceive but rather to call out, ridicule, or expose behavior that is shameful, corrupt, or otherwise “bad” [54]. The intent behind satire seems legitimate enough to exclude it from the definition, however, Wardle [171] does include satire as a type of fake news when there is no intention tocauseharmbutithaspotentialtomisleadorfoolpeople. Also,Golbecketal.[54]mentions that there is a spectrum from fake to satirical news which they found to be exploited by many fake news sites, which used disclaimers at the bottom of their webpages to suggest they were “satirical” even when there was nothing satirical about their articles, to protect them from accusations about being fake. Thereby, our definition does include content that is falsely posed as satire, as well as satirical content that can potentially mislead. Additionally, we disambiguate the terms hoax and rumor which are closely related to fake news. A hoax is considered to be a false story used to masquerade the truth, and by the traditional definition, fake news can be seen as a form of hoax usually spread through news outlets [193]. The term rumor refers to unsubstantiated claims that are disseminated 28 with the lack of evidence to support them. This makes them very similar to fake news, with the main difference being that they are not necessarily false, and may turn out to be true [193]. Rumors originate from unverified sources but may later be verified as true or false, or remain unresolved. Our definition encompasses hoax, false and unverified rumors. Lastly, we define conspiracy theories, which are a part of disinformation or fake news, andgenerallyconsistofbaselessclaimsthatareunprovenandoftencannotbeverifiedeasily. In other words, they are defined as most likely false narratives, oftentimes postulated upon unverifiable information, and typically used as an explanation for an event or situation that invokes a conspiracy, often with a political motive [45]. A notable example of conspiracy theories is the “pizzagate” conspiracy touted on social media in 2016 which falsely claimed the emails contained coded messages that connected several high-ranking Democratic Party officials and U.S. restaurants with an alleged human trafficking and child sex-ring [46]. 3.1.4 Disinformation Perception and Spread Wereviewliteraturefromsociologyandpsychologicallythatexplainthereasonsforexistence and spread of disinformation on social media, at both an individual and social level. Individual Level At an individual level, the inability of humans to accurately discern false information leads to continued sharing and belief in disinformation on social media. The inability to discern has been attributed to cognitive abilities and ideological biases. YouGov [178] found in 1684 adults who were shown six individual news stories, three of whichweretrueandthreeofwhichwerefalse, thatonly4percentwereabletoidentifythem all correctly. Pennycook and Rand [127] identified a positive correlation between analytical thinking and the ability to discern false from true information. Similarly, people with higher education, longer time spent on social media, and older people were found to have more accurate perceptions [3]. Moreover, individuals with lower cognitive ability adjusted their assessments after being told that the information given was incorrect, but not to the same 29 extent as those with higher cognitive ability [137]. Besides cognitive abilities, ideological priors play an important role in information con- sumption. Naive realism (individuals tend to more easily believe information that is aligned with their views), confirmation bias (individuals seek out and prefer to receive information that confirms their existing views), and normative influence theory (individuals choose to share and consume socially safe options as a preference for social acceptance and affirma- tion) are generally regarded as important factors in the perception and sharing of fake news [150]. The survey by Allcott and Gentzkow [3] also found with statistical significance that DemocratsandRepublicansare17.2and14.7percentrespectively,morelikelytobelieveide- ologicallyalignedarticlesthannonalignedones,althoughthedifferencesinmagnitudeacross the two groups (Democrats and Republicans) are not statistically significant. These individ- ual vulnerabilities have been exploited to successfully disseminate disinformation, with the current social media ecosystem deemed as a “post-truth” era where objective facts are less influential in shaping public opinion, than appeals to emotions and personal beliefs [63]. Social Level The nature of social media and collaborative information sharing on on- line platforms provides an additional dimension to disinformation, popularly called the echo chamber effect [150]. The principles of naive realism, confirmation bias and normative in- fluence theory discussed above in Section 3.1.4 essentially imply the need for individuals to seek, consume and share information that is aligned with their own views and ideologies. As a consequence, individuals tend to form connections with ideologically similar individ- uals (social homophily), and algorithms tend to personalize recommendations (algorithmic personalization) by recommending content that suits an individual’s preferences, as well as by recommending connections to similar individuals. Social homophily and algorithmic personalization lead to the formation of echo cham- bers and filter bubbles, wherein individuals get less exposure to conflicting viewpoints and become isolated in their own information bubble [50]. The existence of echo chambers can 30 improvethechancesofsurvivalandspreadoffakenewsexplainedbythephenomenaofsocial credibility and frequency heuristic, where social credibility suggests that people’s perception of credibility increases if others also perceive it as credible, and frequency heuristic refers to increase in perception of credibility with multiple exposures to the same information [150]. 3.2 Data Collection In this thesis, we analyze disinformation on the prominent social media platform, Twitter, whichisoneofthelargestonlinesocialplatformswithmillionsofactiveusers. Itisrecognized to have one with the most news-focused user base, with significant following in U.S. politics [65]. Due to the relevance of information and news sharing on Twitter, it has largely been a target of disinformation and malicious operations in recent years [49, 143]. Also, Twitter actively engages with academic researchers to provide API endpoints to collect sample of the public data for research, compared to other platforms which restrict data access. 3.2.1 Social Media Engagements The standard Twitter API provides a roughly 1% sample of all tweets that match a given query consisting of keywords and parameters that should be matched to the tweet content and metadata. Most data collection for social media posts define a set of keywords to search forortrack. TherearetwomaintypesofAPI-streaming(live)andsearchAPI,forcollecting social media discourse on a topic or engagements with specific contents. Thestreaming APIallowsustoobtaintweetspertainingtoaspecifiedtopicofdiscussion throughtheassociatedpre-definedkeywordsinreal-time. The search APIontheotherhand, allows querying for past tweets, however it imposes a limit of up to 7 previous days on the collection. The latter is generally used to fetch engagements related to specific articles or contents from which keywords are selected to obtain social media engagements associated with the content. There are also other Twitter endpoints which allow requesting tweet content and metadata given tweet IDs (also called the tweet payload), and similarly user 31 metadata or timeline of tweets given user IDs, with rate limits on number of requests. Engagement Types Tweets include the following types of posts or engagements, that inform how information spreads and account/tweet interactions occur on the platform. • Original tweets (accounts can create content and post on Twitter) • Reply tweets (reply to other posts) • Retweeted tweets (reshares without comment) • Quote tweets (embed a tweet i.e., reshare with comment) • Mentions (accounts mention or tag another account’s handle in their post content). Thefulltweetpayloadcontainslinkinformationbetweentweetsi.e., ifthetweetisaretweet, reply or quoted tweet, the link to the parent tweet is available. A user account may also view tweets (on his/her timeline), or like (favourite) a tweet, and follow other user accounts, or include them in topical lists. However, the followers of a user are not available in the user metadata,andalthoughitcanbeseparatelyqueriedinbatchesof5000underratelimits,itis expensive and difficult to obtain the follower graph for millions of user accounts by querying the API, especially since many users have large number of followers. Also, information about likes (favorite) or views is not available in tweet payload, and therefore the analysis is restricted to the other five types of engagements described above. The propagation of content happens through retweets, replies and quoted tweets (engagements) and whenever user accounts engage with tweets, the tweet and engagement is visible on the timeline of the account’s neighbors (followers), who can then subsequently interact with the tweet, thereby creating a diffusion cascade or propagation tree, as described in the preliminaries. Geolocation Inaddition, atweet’sgeolocationinformationcanbeextracteddirectlyfrom tweet metadata if it has geolocation enabled, otherwise it can be extracted from the user reported location description from the user profile metadata [31]. The geolocation is not always be available. If we obtain the geolocation from the user profile, it is possible that it might not be a valid or accurate geolocation since it relies on accounts self-reporting. 32 Extraction of Diffusion Cascades As described in the preliminaries, information diffu- sioncascadesrepresentthediffusionorspreadofinformationonthenetworkasasequenceof engagements (retweets, quotes and replies) with the original (source) tweet or its subsequent engagements. Cascade C j corresponds to an original tweet with user (u), tweet (tw), and temporal (t) features is retrieved by finding connected components on the diffusion graph (retweet, reply, quote edges) between tweets. C j = [(u 1 ,tw 1 ,t 1 ),(u 2 ,tw 2 ,t 2 ),··· ] The diffusion cascades can be extracted by constructing a graph with nodes representing tweetids and edges linking tweets based on the engagement i.e., if it is a retweet, reply or quoted tweet. Extracting weakly-connected components of this retweet graph results in the engagement cascades, each corresponding to one connected component, rooted at the source tweet or post of the propagating content. The cascade can be represented as above either as a time-ordered sequence, or propagation tree based on the model and application. Unreliable, Conspiracy and Reliable URLs In addition, we collect lists of unreliable and conspiracy news sources from three fact-checking resources for labeling low-credibility news sources shared in the social media engagements, namely: Zimdars (2016), Media Bias/Fact 2 , NewsGuard 3 . NewsGuard has actively maintained a repository of unreliable news publishing sources, many of which published recent disinformation about the COVID- 19 pandemic. The listed sources from NewsGuard, accessed on September 22, 2020, along with low and very low factual reporting news sources listed as “questionable” in Media Bias/Fact Check, and sources tagged with unreliable or related labels from Zimdar’s list are utilized for analysis. Additionally, we compiled mainstream reliable news sources listed on Wikipedia 4 , obtaining a list of 124 reliable and 1380 unreliable news sources or domains. 2 https://mediabiasfactcheck.com/ 3 https://www.newsguardtech.com/covid-19-resources/ 4 https://en.wikipedia.org/wiki/Wikipedia:Reliable_sources/Perennial_sources 33 3.3 Disinformation Labeling Disinformation datasets, with labeled contents are useful for training and evaluation of de- tection and intervention methods, as well as for analysis of disinformation in social media discourse. Since fact-checking normally requires journalistic verification of content i.e., arti- clesorsocialmediaposts,itisusuallyexpensivetocreatelarge-scaledisinformationdatasets. The process of disinformation labeling is challenging for several reasons, summarized below. • Fact-checking is based on journalistic process of research into credibility of claims, to identify if they are false or misleading, supported by credible news reporting and other evidence, or to determine whether they are unproven or unverifiable claims. • Another challenge is that fact-checking being a human and painstaking process is time consuming and expensive, therefore cannot scale in real-time with social media posts. • Disinformation lies on a spectrum with partly false, mixed veracity, unverified or un- verifiable claims, and other types discussed earlier which includes missing, misleading context, false connection, of varying intents and motives. This complicates the process of verification due to the nuanced types of distortion, which can often be contextual. • Different fact-checking resources follow different verification process in terms of selec- tion of claims to verify, process of collection of claims, and use different annotation or labeling criteria and scores. For instance, some provide either a list of violated journalistic principles with credibility ratings, or the degree of false, or mixed facts. • Dynamic, evolving facts and rumors pose other challenges to the verification process, sincefactscanchangeandnewinformationmightbecomeavailablewhichcouldchange the labeling of postulated or previously unverified claims later in time. Therefore, it is difficult to acquire large-scale disinformation labels for social media data and online news articles circulated on social media. Existing works apply two two primary 34 approaches to construct disinformation labeled datasets, either from fact-checked claims reviewed by independent journalistic verification by fact-checking organizations, or using credibility of the sources publishing the content, described as follows. 3.3.1 Fact-Checked Labels One approach to collecting labeled disinformation contents in the form of news articles, claims, or social media posts, is from contents verified by fact-checking websites. Fact- checkingwebsitessuchasSnopes.com, PolitiFact.com, FactCheck.org, NewsGuard, APNews Fact Checks are widely reputed for independent journalistic verification of online content. The fact-checked contents are labeled based on the criteria followed by fact-checker and is not standardized. Researchers generally aggregate the fact-checker annotations of contents, and usually group them into binary false or true (disinformation/legitimate) contents [104]. This approach is frequently used to construct datasets with hundreds or few thousand labeleddisinformationcontents[167,104,151,88]. Thesocialmediaengagementsassociated with the contents are collected using keywords extracted from the contents. The matched social media posts containing these keywords are inspected [88] to determine if they are relevant to the content, or keywords are refined till reasonably relevant matches of social media engagements are found [104]. Such labeled datasets with social media engagements are used in analysis of disinformation vs. legitimate content cascades [167], or in training and evaluation of detection methods [104, 151]. Some works also include mainstream news contents or articles into the dataset [88, 104] as legitimate or true news contents. This approachcanintroducebiasesindatacollectionbasedonselectioncriteriaforclaimsbyfact- checking organizations. For instance, PolitiFact.com documents their process as selecting claims based on relevance and popularity and therefore the disinformation uncovered can be biased towards top stories that were more sensational or viral on social media. 35 3.3.2 News Credibility Based-Labels The other commonly used approach for disinformation labeling is based on credibility of news sources publishing the contents [15]. Some fact-checking websites like Media Bias/Fact Check, NewsGuard, PolitiFact also provide list of unreliable and reliable sources based on individual articles fact-checked from these sources. Sources that repeatedly publish disin- formation and conspiracies are noted as unreliable. NewsGuard maintains a list of sources with specific violations of journalistic standards and associated credibility scores. Bozarthetal.[15]founddifferentlistsofsourcesareoftenusedindisinformationlabeling depending on the fact-checker it is compiled from. The differences however were found to affect prevalence, but not the temporal trends or differences in narratives of disinformation vs. legitimate contents labeled by these methods. This approach allows for more diverse and large-scale datasets and for labeling disinformation in social media discourse, but can contain label noise at the level of the engagement or articles in the social media posts. 3.4 Constructing Disinformation Datasets Wedescribethecollectionofsocialmediaengagementstoconstructdisinformationdatasets, aswellas,existingdisinformationandsocialmediamanipulationdatasetsusedinthisthesis. Dataset of News Articles and Social Media Engagements Most fake news detection datasets contain either news article contents, or social media engagements related to fact-checked claims, since most content-based methods rely solely on articles or contents [170], and others focus on modeling engagements or user response only [138, 104]. These engagements or user responses on social media are normally provided as temporal sequences or tree-structured cascades of social media posts corresponding to specific fact-checked disinformation and legitimate claims. Therefore, we collected a dataset with both article contents and user responses to jointly study social media engagements to contents, and leverage them together for disinformation detection. For data collection 36 Table 3.1: Disinformation dataset statistics for news articles and Twitter engagements Statistic Low-credibility News True News # URLs (news articles) 13,552 11,523 # Engagements (user responses) 995,325 1,845,854 # Users (social media accounts) 221,856 587,130 Min. length of article 200 200 Max. length of article 9324 9928 Avg. length of article 821 1101 Min. tweets/URL 4 4 Max. tweets/URL 118,377 72,726 Avg. tweets/URL 73.4 160 # Users engaged in > 1 URL 81,292 182,727 Avg. articles engaged for above users 6 4.3 Max. articles engaged by any user 1574 601 and disinformation labeling, we compile a list of credible mainstream news sources (such as http://www.bbc.co.uk/) from Wikipedia, and a list of low-credibility news sources that repeatedly publish false and misleading disinformation from Zimdars [191]. Article Contents To collect article URLs, for each website, we search for all ‘sub-URLs’ (such as http://www.bbc.co.uk/xxx/xxx/xxx) on Twitter using Twitter search API (which returns tweets only within 7-9 days). These article URLs have at least one user response on Twitter and are labeled by the credibility of the news source. For the collection of article bodies from article URLs, we crawl the article contents from the URLs. 5 User Responses To retrieve user responses to the article on Twitter, we collect tweets containing the article URLs obtained with the Twitter search API. However, since tweets olderthan7dayscannotberetrieved, weadditionallycrawlforpasttweetsdirectlyfromthe Twitter search web interface. Although due to crawling limits, we cannot obtain as many tweets as from the API, we are able to retrieve enough engagements with the article as user 5 we use the python package “newspaper” to crawl article contents from URLs. 37 responses to the contents. 6 Table 3.1 summarizes the dataset statistics of collected cascades. Disinformation Dataset for US Election Analysis For the U.S. 2020 Election disinformation and conspiracy group analysis, we utilize a large- scale dataset collected by earlier work [20]. It contains 242 million election-related tweets collected between June - September 2020, preceding the election by tracking mentions of political candidates. It is collected using the Twitter streaming API which provides a∼ 1% sample of all tweets filtered based on tracked parameters. Around July 22, 2020, Twitter imposed its ban on the far-right QAnon conspiracy group, and therefore, we leverage this largedatasettostudydisinformationnarratives, engagements, andQAnonconspiracygroup activities and effects of Twitter restrictions. This large dataset provides tweets related to the election, but for disinformation and conspiracygroupanalysis,weneedtolabelthetweets. Sincethedatasetisspecificallyabout theElectionandcontainsrecentclaimsbasedonreal-worldeventssuchasfalseclaimsabout “mail-invoterfraud”, wecannotdirectlyapplyadetectionmodeltoidentifydisinformation. Instead,weleveragenewssource-basedcredibilitylabelstofirstobtainalabeledsetoftweets, to train a disinformation detection model, and then apply the model on unlabeled tweets. Model Specifications for Labeling Tweets Original tweets sharing URLs published from the unreliable (or conspiracy) and reliable news sources are thereby labeled as unreli- able/conspiracy and reliable respectively. In addition, we also label tweets that are retweets, replies, or quotes sharing such URLs, if the parent of the tweet is not found in the collected tweets using the Twitter API which returns a 1% sample. We apply CSI, a supervised, deep learning, feedback-based detection model [138]. The modelcapturesthreemainsignalsorfeatures: source,text,andtemporalfeatures,whichare useful for disinformation detection [138, 142]. To classify a tweet, the model considers time- 6 We used python package “got” to collect past tweets that contain a certain article URL, designed to crawl directly from the Twitter web search interface. 38 ordered sequence of engagements that the tweet receives, along with temporal information such as the frequency and time intervals between engagements. In addition, it learns the source characteristic, based on account behaviours i.e., which accounts co-engage with a given tweet. These input signals are useful for differentiating disinformation tweets. The model requires training data containing labeled cascades (i.e., tweets with its engagements). Engagement Cascades and Cascade Statistics To train the detection model, we first extract diffusion cascades from the dataset as described in Section 3.2.1. The cascade is labeled by the URLs linked in its first tweet, if the tweet shares a URL from one of the con- sidered factual or unreliable news sources, otherwise it is left unlabeled. Since the dataset contains over 10M accounts and 242M tweets, for computational efficiency, we subsam- ple accounts and cascades, whilst ensuring minimal information loss, such that at least 75%(∼ 180M) of the tweets are accounted for. We find that this can be achieved by subsam- pling cascades with a minimum number of engagements, and accounts with highest active and passive engagements with other accounts in the dataset 7 . For accounting for >75% of thetweets,wefindcascadesizeof5,and7471highestengagingaccountswassufficient. Note that the subsampled cascade i.e., corresponds to engagements only with the subsampled ac- counts, but considering all accounts in the cascade from the full 10M account set, results in complete or “full cascades” which accounted for at least 75% of all tweets. Full cascades are used in the analysis of identified disinformation cascades, but for training and inference with CSI, the subsampled accounts in these cascades are utilized. In total we have ∼ 200K cascades, where 192K are unlabeled, and the rest are labeled by the news source credibility (3120 unreliable and 4320 reliable cascades). Table 3.2 provides statistics of data cascades. ModelEvaluationforLabelingTweets Table3.3reports5-foldcrossvalidationresults obtainedusingCSIonthelabeledcascades. ThemetricsreportedaretheROCAUC,average 7 Active refers to tweets (original, retweet, reply, quote) from the account, whereas passive means the account has been involved in another account’s tweets (retweeted, replied, quoted, or mentioned) 39 Table 3.2: Dataset statistics of Election discourse content engagement cascades Statistic Count Tweets 242,087,331 Users accounts 10,392,492 Unreliable/conspiracy URLs cascades 3,162 Reliable URLs cascades 4,320 Unlabeled cascades 192,103 Avg. cascade size (# engagements) 57.11 Avg. cascade time (in hrs) 80.42 Avg. time between engagements (hrs) 5.93 Table 3.3: Results on detection of unreliable/conspiracy cascades in the election dataset Method AUC AP F1 Prec Rec Macro-F1 SVM 0.6236± 0.01 0.5025± 0.01 0.5710± 0.01 0.5594± 0.01 0.5835± 0.02 0.6226± 0.01 GRU 0.5244± 0.05 0.4499± 0.04 0.4606± 0.11 0.4367± 0.05 0.5110± 0.17 0.5073± 0.05 CI 0.6326± 0.02 0.5364± 0.01 0.5732± 0.02 0.5307± 0.02 0.6243± 0.03 0.6046± 0.02 CI-t 0.6554± 0.02 0.5661± 0.01 0.5820± 0.03 0.5354± 0.01 0.6426± 0.08 0.6099± 0.01 CSI 0.8054 ± 0.02 0.6826 ± 0.02 0.7597 ± 0.02 0.6611 ± 0.02 0.8944 ± 0.04 0.7608 ± 0.02 precision AP, and we report F1, Precision, Recall and macro-F1 at the detection threshold that achieves maximum geometric mean of validation set sensitivity and specificity in AUC. Forcomparison,weevaluateitagainstseveralbaselines: (i)SVM-RBFontextfeaturesof the cascade source tweet, extracted with doc2vec. (ii) GRU [104] which utilizes a recurrent neural network (RNN) to classify the cascade based on text features of the time-ordered sequence of engagements in the cascade. (iii) CI, which can be considered as a variant of the CSI model, utilizes only tweet text and account metadata features of the engagements as input to a RNN for classification. (iv) CI-t, another variant of the CSI model, utilizes tweet features of engagements, and in addition temporal information of time intervals between engagements as input to a RNN. (v) CSI model, which includes all three features (tweet features of engagements, temporal information, and account behaviours). The CSI model has AUC 0.8 and high recall and is therefore used for inference on unlabeled cascades. The ensemble of CSI models trained over five folds are used for inference. We take the top and bottom 80 percentile, of cascades predicted with the largest margins above and below 40 the detection threshold, resulting in 72,228 unreliable and 81,453 reliable classified cascades respectively. The engagements of the reliable cascades constitute about two-third of the tweets and engagements of unreliable cascades correspond to the remaining one-third. Validation We take a random sample of 50 cascades from the cascades labeled by the model for human validation. The list of validated tweets is provided in the Appendix A.2.1. In 2/50 (4%) the model label differed from the validated label assigned by inspecting the tweet content and its account for suspensions. The model errors included a tweet critical of the postal system and though non-conspiratorial, could have been mistaken for claims related to mail-in voter fraud by the model, and another which strongly supported criticism of China by president Trump, although not from a suspended account and unlikely part of QAnon/conspiracy groups, was predicted unreliable. To reiterate, these validated cas- cades are predictions with the largest margins from the CSI model’s detection threshold, as discussed earlier, using the ensemble of 5-fold classifiers, and the expected lower error rate confirmed by validation suggests it can be reliably utilized for further analysis. Covid-19 Social Media Engagements Dataset Covid-19 Foranalysisofdisinformationduringthepandemic, wecollectedasimilarlarge- scale dataset of Twitter posts using the streaming API service tracking parameters related to COVID-19 (‘Covid19’, ‘coronavirus’, ‘coronavirus’, ‘2019nCoV’, ‘CoronavirusOutbreak’, ‘coronapocalypse’) from March - June, 2020. The COVID-19 crisis was declared a global pandemiconMarch11,2020,fromwhichperiodthedatawascollected. Thedatasetcontains 85.04M tweets from 182 countries. The subset of English tweets equals 54.32M. The tweets are weakly-labeled by the news source credibility as in the previous Election dataset. Covid-19 Vaccines With the advent of COVID-19 vaccines in late 2021, anti-vaccine beliefs and disinformation became an important concern. We extended our collection by tracking vaccine-related tweets with the streaming Twitter API (‘vaccine’, ‘Pfizer’, ‘BioN- 41 Table 3.4: Data statistics for Twitter [88, 104] and Weibo [104] datasets Dataset Twitter-1 [88] Twitter-2 [104] Weibo [104] # Users 117,824 233,719 2,746,818 # Engagements 192,350 529,391 3,805,656 # Disinformation cascades 60 498 2,313 # Legitimate cascades 51 494 2,351 Avg. duration (hrs) 8,177 1,983 2,460 Avg. # engagements 1,733 597 816 Tech’, ‘Moderna’, ‘Janssen’, ‘AstraZeneca’, ‘Sinopharm’) from Dec 9, 2020 to Apr 24, 2021. This marks the period from the first emergency authorization (EUA) of the vaccines with Pfizer-BioNtech and Moderna in December, to the COVID-19 vaccines being made available to all US adults in April, 2021. It contains 29.7M COVID-19 from 7.4M accounts. Existing Disinformation and Manipulation Datasets We also use other existing disinformation datasets for evaluation in our works on early detection, network interventions, and coordinated disinformation campaign detection. Disinformation Datasets The prior work disinformation datasets used in this thesis in- clude two Twitter [88, 104] and one Weibo [104] dataset which contain social media engagements with false and true claims verified by fact-checking organizations. Kwon et al. [88]includesTwitterengagementsofcontentsfact-checkedbySnopesandurbanlegends.com. [104]usesSnopesfact-checks. TheWeibo[104]datasetisbuiltfromreportedmisinformation on the Sina community management center 8 and its engagements on social media platform, Weibo. The datasets span posts dated between 2008-2016. The datasets are used addition- ally in our experiments on proposed methods for early detection, analysis of disinformation propagation and network intervention experiments. The datasets are collected similar to the process described earlier, where fact-checked claims from Snopes etc. are inspected, and keywords from the claim content is used to query the Twitter search API. The engagements returned are represented as a time-ordered sequence or cascade for each fact-checked true 8 http://service.account.weibo.com 42 Table 3.5: Dataset of coordinated manipulation campaign in the 2016 US Election IRA Dataset (Internet Research Agency) Count Coordinated group (T: Russian ‘troll farm’) 312 accounts 495,293 tweets Normal accounts (NT: Non-troll) 1713 accounts 4,118,819 tweets andfalseclaims. DatasetsstatisticsarelistedinTable3.4. Twitteraccountsthataredeleted or suspended cannot be retrieved from the API. Therefore, the dataset statistics represent the remaining accounts that could be fetched from the TweetIDs released in [104]. CoordinatedDisinformationCampaignsDataset Inexperimentsrelatedtodetection of coordinated disinformation promotion from malicious account groups, usually there is no ground-truth or labeled datasets that can be easily constructed. Therefore, we have to rely on analysis of feature distributions of detected groups for evaluation. However, in one specific case, of the U.S. 2016 Election, disinformation promoted from a coordinated group of accounts operated by Russia’s Internet Research Agency (IRA) were uncovered by U.S. Congressofficial investigations withthe Twitterplatform. These accountshad engagedwith more than 1.4 million accounts, who were later notified of the malicious activities post the Election. The coordinated influence campaign was also referred to as the Russian “troll farm” and contained 2,735 human and bot accounts operating under the Russian Agency to deepen the political divide and manipulate the Election outcome, with promotion of disinformation and persuasive and polarized narratives. We obtain part of this labeled dataset from [101] to evaluate our proposed methods. The dataset contains accounts from this Russian campaign and 2016 Election data obtained from Twitter, with 312 coordinated and 1713 normal accounts, corresponding to accounts that had at least 20 active and passive engagements. Thedatasetalsocontains5MtweetsfromtheseaccountsasshowninTable3.5. 43 3.5 Limitations and Discussion As described, the process of disinformation labeling is hard for several reasons. The claims in the content need to be fact-checked. Also, claims can be partly false, out-of-context, unverifiable, making the task more pain-staking and time-consuming. Prior approaches to constructing disinformation datasets utilize either previously fact- checkedclaimsfromfact-checkingorganizations,whichresultindatasetswithhundredtofew thousand claims. In other cases, prior works utilize news source credibility which provides larger-scaledatasets,withcheaperlabelingefforts,buteachapproachhasitsownlimitations. • Fact-checkedclaimsbasedlabelinglimitations. Thisapproachutilizespreviously fact-checked claims and therefore it is biased by the claim selection process of the fact- checkingorganizations,andtheratingcriteriausedbytheorganization. Insomecases, theselectionofclaimstofact-checkisbasedonrelevanceorpopularity 9 ,whichcanbias the analyses, such as analysis of how disinformation spreads or is perceived compared to legitimate contents. Moreover, it is difficult to construct large-scale datasets for analysis or training new models in a timely manner, especially for new domains. • News source credibility-based labeling limitations. This approach can be more effectively applied in new disinformation domains, differentiated by low-quality or mainstream reliable sources. The diversity of information labeled through this ap- proach is higher and less sensitive to biases in selection processes of fact-checkers. The obvious limitation though, is that it contains noise in the labels at the article or social media post level, based on the specific article from the source and post sharing it. In the following chapter, we discuss these limitations further and propose a framework for construction of large-scale disinformation labeled datasets from social media posts for new domains or topics of discourse in a timely manner, minimizing annotation resources. 9 PolitiFact mentions this as their selection criteria under their process description. 44 Chapter 4 Timely and Scalable Disinformation Labeling in New Domains InChapter3,wediscussedseveralchallengesindisinformationlabelingwhichrequireshuman intensivefact-checkinganddifferentiatingdifferenttypesofdistortednarrativessuchasfalse, misleading, missing context, and others, making it untimely and difficult to scale. Due to these difficulties in disinformation labeling, prior approaches resort to utilizing previously fact-checked claims from fact-checking organizations [88, 104], or using cheaper news source credibility-based labeling instead [15, 188]. The limitations though are that the former approach cannot be easily scaled to large diverse disinformation labels especially for newdomainsinatimelymanner. Second, itisalsobiasedbytheselectionprocessandmany fact-checking organizations prioritize popular claims. The other approach of labeling based on news-source credibility is cheaper and scalable but subject to inherent label noise. To reduce the human labeling effort, we consider a strategy of utilizing the news domain- level labels as initial weak labels, i.e., we utilize the credibility of news sources to label social media posts sharing URLs from the news domains. From the weakly-labeled dataset, we propose an active label refinement framework to to obtain more accurate tweet-level labels. We achieve this by employing an active label refinement strategy which combines the au- tomated disinformation detection model and social context, with the human expert, wherein the model can be used to guide the re-labeling in real-world social media data. The human expertcanverifyalimitednumberoflabels, asguidedbythemodelpredictionstoconstruct timely, large-scaledisinformationdatasets. InthisChapter, wedescribetheproposedframe- work applied to construction of a large-scale disinformation dataset on COVID-19 vaccines. 45 4.1 Disinformation Labeling Guidelines Labelingdisinformationisalreadychallenging, moresobecausedisinformationisnoteasyto specify[142]. Itcanlieonaspectrumoftruth,includingfalse,conspiracy[45],andmisleading or distorted information such as missing or misleading contexts or mixture [171, 92]. We find that fact-checking organization Snopes uses a well-defined label schema that is general enoughtofitanydomain , andyetmanagestocoveralltypes andnuancesofdistortion we found upon examining tweets in the vaccines dataset, and generally in the literature [142]. Snopes includes several labels to cover the varying degree of truth and other deceptive tactics like miscaptioned, misattributed, scam. We work with the 6 most relevant Snopes categories, and add the ‘Debunk’ category based on what we observed in tweets. Thelabelschemeisproposedbelow,derivedfromSnopesandtweetinspection. Werefine the label definitions to make the distinction between them and its coverage explicitly clear basedontheinspectedtweetdata, forlabelingsocialmediapostsbasedontheirfactualness. We found that providing clear guidelines to annotators with listed examples of cases in each label category was more effective to reduce subjectivity in labeling. Guidelines Label the tweet based on what the tweet is trying to say or claim, and how factual its claim is. The label is assigned by choosing one of the following types. 1 • True: Primary elements of the claim are demonstrably true. • Debunk: Tweet calls out or debunks inaccurate information. • Mostly true: Primary elements of a claim are demonstrably true, but some of the ancillary details surrounding the claim may be inaccurate. • Mixture: Claim has significant elements of both truth and falsehood (including e.g. significant missing context or misleading, which might cause one to be misled). 1 This label scheme and instruction set was finalized after two iterations. At first, we started with 11 labels with vaccine specific instructions e.g. potential to mislead (can be misinterpreted as unsafe), missing context, etc) but found difficulties in making clear distinctions and following the specified instructions. This label scheme was clear to follow and we found that sufficient examples of tricky and typical cases in each category was very helpful to the annotators, who were asked to review the instructions and examples before annotating, and had access to reference it when in doubt. 46 Table 4.1: Covid-19 vaccines Twitter dataset (collected Dec 9, 2020 - Feb 24, 2021) Statistic Count # Tweets 9M # User accounts 490,638 Unreliable/conspiracy URL cascades 4,267 Reliable URL cascades 10,377 • Mostly false: Primary elements are false, but ancillary details may be accurate. • False: Primary elements of a claim are false or conspiratorial. • Unproven: Insufficient evidence that it is true, but for which declaring it false would require a difficult (if not impossible) task of proving a negative. We evaluated the label scheme and guidelines on a random 200 sample subset from the collected tweet cascades. We compute the inter-annotator credibility for the tweets between two annotators, one graduate non-native English speaker familiar with disinformation re- search vs. one undergraduate native English speaker not familiar with the research. The agreement is moderate if we consider across the 7 label categories (0.61 Cohen’s kappa), and substantial (0.77 Cohen’s kappa) if binarized as (true, debunk, mostly true) vs. (mixture, mostly false, false, unproven) as high-level abstractions. Both annotators followed the same guideline and instructions with typical and difficult examples (noted in Appendix A.1). 4.2 News-Source Credibility Based Weak Labels A social media post and its engagement cascade can be weakly-labeled as disinformation cascades if the post shares a URL from a low-credibility source that repeatedly publishes false and misleading content. There are however two sources of label noise. • First, not all contents published by the source might contain disinformation, although theytendtobeunreliableorrepeatedlyviolatejournalisticreportingprinciples,enough for the source to be included as a low-quality source by experts. 47 12-13 12-23 01-02 01-12 01-22 02-01 02-11 02-21 03-03 03-13 03-23 04-02 04-12 04-22 Timeline 0 50000 100000 150000 200000 250000 Pfizer-BioNTech EUA in U.S. Moderna EUA in U.S. Biden: All U.S. Adults eligible (Apr 19) Biden says Vaccines to U.S. Adults by May end Original Reply Retweet Quote Pfizer tweets Moderna tweets Figure 4.1: Tweet volume timeline from the Emergency Use Authorization (EUA) in U.S. till U.S. Adults vaccinations for Covid-19 vaccines Twitter dataset • Secondly, the post that shares the content from the source might either support or restate the content accurately, or in other cases oppose or distort the true content from the source, which would result in an incorrect label for the cascade. We described in Chapter 3 (Sec. 3.2.1) that we collected social media engagements on COVID-19 vaccines using the streaming Twitter API that returns a∼ 1% sample of tweets matching the tracked vaccine-related keywords. Then, with extraction of diffusion cascades, we get the following sequences of engagements represented by the user accounts (u), tweets (tw), and time-stamps (t), C j = [(u 1 ,tw 1 ,t 1 ),(u 2 ,tw 2 ,t 2 ),··· (u n ,tw n ,t n )] (4.1) The source tweet is the first tweet in the cascade, and subsequent tweets are its propagation tree or time-ordered sequence of tweet engagements. 4.2.1 Weakly-Labeling Using News-Sources We weakly-label each cascade based on news-source credibility. If the source post references one of the news sources i.e., shares a URL or article published by the news source, it is assigned its corresponding weak label. The list of news sources with their credibility is 48 Debunk True Mostly_true Mixture Mostly_false False Unproven Reliable Unreliable Conspiracy 27 202 13 16 2 13 0 0 23 6 16 3 17 6 0 7 0 10 2 41 2 0 25 50 75 100 125 150 175 200 Figure 4.2: News-source credibility labels correlation with fact-checked claim based labels on 150 validation and 256 test tweet cascades collected from fact-checking resources, namely, Media Bias/Fact Check, Zimdars [191], and NewsGuard as detailed in Chapter 3 (Section 3.2.1). The weak label assigned from low- quality sources is the ‘unreliable/conspiracy’ label vs. ‘reliable’ label for the mainstream news sources. Bozarth et al. 2020 found that different works in the literature have utilized different fact-checking resources to compile lists of low-quality news sources. Bozarth et al. 2020 found that the differences in news credibility lists tend to affect the prevalence of identifiedquestionablecontent,butnotthetemporaltrendsornarrativesofthesuchcontent. Since,intheproposedframework,weareinterestedincuratingnewssourcestoprovideweak labelsthatcancoverasdiverseasetofdisinformationclaimsaspossibleinsocialmediaposts, more than the prevalence, it is sufficient to use any reasonable credibility analysis. The extracted tweet cascades from the social media vaccine engagements collected be- tween Dec, 9 2020 and Feb 24, 2021 (Fig. 4.1), retaining user accounts that have at least 5 observed tweets in the dataset, and cascades of minimum length 5, are used for construct- ing the weakly-labeled disinformation dataset. After weakly-labeling, there were 10,377 reliable cascades, and 4,267 unreliable/conspiracy cascades (Table 4.1). These 14.6k cascades with weak labels will be used to construct the disinformation dataset. 49 4.2.2 Correlation with Fact-Checked Labels First, we analyze the correlation between the labels from the two approaches - news-source credibility labeling and fact-checked claim based labeling. We collected fact-checked claims from Snopes 2 and NewsGuard 3 on COVID-19 vaccines. For each fact-checked claim, we find tweet cascades that discuss the claim by searching for text matches to words related to the claim. E.g. “Myth: The COVID-19 vaccine will use microchip surveillance technology created by Bill Gates-funded research.” We search for source tweets with words “chip”, “microchip”, and “surveillance”. If nothing is found, we refine the search with“gates”. NewsGuard provides only Myths (false claims), while Snopes provides varying factuality labels (true, mostly true, mixture, false etc.). From the Snopes collection tagged as COVID- 19 vaccines, we obtained the claims tagged as fact-checked claims and high factual news articles from AP News. AP news is rated as a ‘very high’ rating for factual reporting and ‘least biased’ for media bias by the Media Bias/Fact Check credibility analysts. NPR News also has a similar rating. Therefore, we additionally web-scrape the NPR’s COVID-19 news articles repository and AP News website to incorporate more reliable claims with the ones extracted from the articles and claims fact-checked and listed on Snopes.com. For news articles, we scraped the article heading, claim/short description, and date from NPR and AP News. Along with the Snopes claims, a total of 400 fact-checked claims are retrieved. Ground-Truth Labels Using the extracted claims, we find the matching tweet cascades based on the text matches of claim content to tweets as described earlier. For each matched tweet cascade, we inspected its source tweet. If the tweet discusses the matched claim, we examine the stance of the tweet i.e, whether it supports or distorts the claim. Based on the Snopes fact-check label and tweet stance the fact-checked claim based label is assigned as ‘false’, ‘mostly false’, ‘unproven’, ‘mixture’, or ‘true’, ‘mostly true’, ‘debunk’ i.e., 2 https://www.snopes.com/tag/covid-19-vaccine/ 3 https://www.newsguardtech.com/special-reports/special-report-top-covid-19-vaccine-myt hs/ 50 News Source Credibility Tweets Weak labels Detection Model Social Context {Unreliable/Conspiracy/Reliable} Tweets (High label entropy) {False, Mostly false, Mixture, Unproven} {True, Mostly true, Debunk} Tweets (Low label entropy) Relabel / Self-Supervision News Articles Fact-checks Extracted representation Class Prototypes (k-NN) {False, Mostly false, Mixture, Unproven} {True, Mostly true, Debunk} Figure 4.3: Proposed approach: Model-guided label refinement with self-supervision from a generic disinformation detection model and social context modeling to construct large-scale disinformation labeled social media datasets in a timely manner calling out disinformation. We found 256 tweet cascades to label based on the Snopes fact-checks and factual news articles. Thisformsourevaluationtestsetoftweetcascadeswithground-truthfact-checked claimbasedlabels. Forthevalidationset,weadditionallysampled150tweetcascadeswith stratified sampling from the 14.4k cascades, and labeled them based on similar annotation. Correlation In Fig 4.2, we compare the news-source credibility based labels (unreli- able/conspiracy/reliable) with the inspected fact-checked claim based labels for the human labeled tweets in the evaluation test and validation sets mentioned above. Overall, the news credibility labels appear to be well correlated with actual human labels. Individual inaccu- racies can still exist, but with the large-scale weakly-labeled data smoothing out individual errors, we could learn to refine the weak labels to construct the disinformation dataset. 4.3 Model-Guided Label Refinement In the previous section, we observed that the news credibility labels are correlated overall with actual fact-checked claim based labels, and with the large scale of the weak labeled dataset smoothing out individual errors, we could learn to remove inaccuracies. In Fig. 4.3, we provide the proposed framework for disinformation dataset construction 51 and labeling in new domains at scale. The weak labels unreliable, conspiracy, reliable on tweet cascades are from news-source credibility. It is utilized to construct the initial dataset with y∈{0,1} as weak labels with 1 as unreliable/conspiracy and 0 as reliable. Our goal is to remove or correct inaccurate weakly labeled instances in the dataset, and output high-level distinctions of disinformation label as 1 and reliable information label as 0, with the model-guided predictions of confidence in the labels. 4.3.1 Self-Supervision from Disinformation Detection Modeling In the proposed framework, we make use of any generic disinformation detection model to guide the weak label refinement. Classifiers are often utilized to estimate uncertainty in the instance labels from the loss or model predictions in label noise methods [38, 6]. In this work, we use entropy of the disinformation detection model predictions to measure closeness from the decision boundary [38]. High entropy indicates greater model uncertainty about the label. The entropy S i in the model predictions for an instance i is defined as follows, where p(x i ) is the vector of predicted probabilities from the detection model, and k is the classes, here for y∈{0,1}. S i =− L X k=1 p k (x i )log(p k (x i )) (4.2) We train the detection model on all weak labeled data, and then filter out high entropy instances. We also filter out tweets with low entropy model predictions if the initial weak label and predicted model label for the tweet are inconsistent with each other. This is for instances where the model is confident in its prediction, either has an incorrect weak label or predicted label. With the filtered dataset, we retrain the detection model, and repeat until the model has marginal improvements on a held-out small human-labeled or fact-checked labeled validation set is marginal. This iterative self-training improves the detection model and its signals of the inaccuracies in weak labels. In each iteration, the retrained detection model is applied to all instances in the initial dataset to calculate the entropy scores, and filter for the next iteration as it can now make more informed filtering decisions. 52 Figure 4.4: Characterization of communities in the 3-core of the Retweet (RT) graph to find misinformed and informed communities to model the social context 4.3.2 Social Context Modeling We propose that the social context can also be useful in guiding the construction of disinfor- mation labeled datasets. We incorporate social context of the post using the community of its associated user to model a user account’s credibility and stance in the discourse. Social media discourse tends to be segregated into echo-chambers of user accounts sharing similar opinions [51]. User accounts follow each other based on their interests, and become more exposed to contents that align with their interest and ideologies [142]. The retweet graph between user accounts that retweeted each other’s tweets can be used to identify user com- munities. Retweets are seen as a form of endorsement of content and edges with at least two retweets are retained to capture links of similar interests in the user accounts [51]. We identify user account communities from the retweet graph using Louvain method [11]. Communities Structure Similar to prior work [51], we use RT edges with minimum count of retweets >= 2, including mutual retweets. For the RT graph, we use the 3-core decomposition to exclude users with only weak connections to the primary discussions [114]. A retweet graph with 91k accounts, 121k edges, and avg. degree 2.66 before the k-core decomposition. is obtained and after 3-core decomposition we have 8,974 accounts with 31k edges and an avg. degree of 6.92. Applying Louvain [11], we get 39 communities. 53 Table 4.2: (Mis)information communities in the 3-core of the Retweet Graph Inferred Leaning Low-Quality News URLs No. Language Locations Left Right Und ConspiracyUnreliable Reliable Others 0(20%) EN (97.4%) US (90.3%) 98.7 0.3 1 0.26 2.87 96.87 92.22 1(16%) EN (94.0%) US (48.9%), UK (27.5%) 2.8 6.8 90.4 39.61 33.82 26.57 87.56 2(16%) EN (95.3%) US (57.5%), India (7.6%), UK (6.1%) 94.4 1.2 4.4 0.94 5.47 93.59 91.54 3(11%) EN (96.9%) US (86.6%) 5.8 83.9 10.3 32.31 35.14 32.55 89.62 4(6%) EN (84.5%) India (90.3%) 8.2 0.2 91.6 10.27 37.76 51.97 97.92 5(5%) EN (96.6%) UK (76.9%), US (13.8%) 64.6*0.2 35.2 1.28 10.25 88.47 93.12 6(5%) ES (81.2%), EN (13.3%) Argentina (34.3%), Mexico (11.9%) 56.9*0 43.1 3.48 7.32 89.2 97.68 7(3%) EN (95.6%) Canada (91.0%) 87.8 0 12.2 1.78 4.07 94.15 97.05 8(3%) EN (97.3%) US (94.8%) 0 97.7 2.3 31.86 45.14 23 85.09 9(3%) FR (81.9%), EN (10.6%) France (92.0%) 4 0 96 14.18 62.88 22.94 90.18 10(3%) EN (90.8%) India (84.2%) 92.3 0 7.7 0 2.66 97.34 95.92 11(2%) EN (79.2%), TL (18.8%) Philippines (85.7%) 100 0 0 7.5 1.25 91.25 98.25 12(2%) EN(53.5%), IT (38.7%) Italy (32.7%), US (20.4%) 34.5 1.9 63.6* 22.22 55.19 22.59 91.28 13(2%) EN (88.4%) US (30.8%), Africa (13.8%) 86.9 0.8 12.3 1.79 1.57 96.64 90.33 14(1%) ES (48.6%), EN (26.0%) Netherlands (28.6%), Uruguay (14.3%), Latvia (14.3%) 60.3*1.8 37.9 49.52 29.61 20.87 90.84 Characterization of Misinformed and Informed Communities We characterize the top-20 diffusion communities that account for 96% of the accounts in the 3-core RT graph. Fig. 4.4 presents the communities with its characterization in Table. 4.2 and Table. 4.3 in- cludes the community number with size (% accounts) in the graph, language in tweets from the community, geolocation extracted from geo-enabled tweets/reported valid locations in account profiles [31] to characterize the general demographic of the community. In addition, we infer the political leaning of accounts using left/right media URLs (as classified by all- sides.com) endorsed directly or through retweet structure, similar to [7]. We jointly inspect these with the top retweeted accounts and tweets, and distribution of URL news sources, top URL/news domains, and contents in top retweeted and random subset of tweets. • A large (16.24%) community (C1) of Anti-vaccine misinformation and conspiracies. From accounts that have valid locations reported in the profile or tweets, this commu- nity spans US (48.9%) and UK (27.5%). • Other smaller communities with dominant misinformation or conspiracy tweets cor- respond to the U.S. Far-right conspiracy group that post anti-vaccine content (C8). 54 Table 4.3: Top Tweeted URLs and Top Retweeted Accounts in (Mis)information communi- ties in 3-core Retweet Graph. No. Top News Domains Top Retweeted Accounts 0(20%) nytimes, washingtonpost, cnn, latimes, politico JoeBiden, KamalaHarris, NYGov- Cuomo 1(16%) childrenshealthdefense, dailymail, zerohedge, rt, lifesitenews ChildrensHD, zerohedge 2(16%) reuters, theguardian, nytimes, independent, latimes Reuters, NBCNews, AP, Coro- naUpdateBot 3(11%) truepundit, foxnews, theepochtimes, zerohedge, dailymail Mike Pence, GOPChairwoman, OANN, nypost 4(6%) swarajyamag, indianexpress, dailymail, nationalfile, wsj timesofindia, WIONews, mygovin- dia 5(5%) theguardian, nytimes, telegraph, bbc, express DHSCgovuk, NHSEngland, NHSuk 6(5%) nytimes, reuters, bbc, theguardian, thetimes ReutersLatam, CoronavirusNewv, AlertaNews24 7(3%) nytimes, theguardian, reuters, washingtonpost, bloomberg JustinTrudeau, CBCAlerts, CPHO Canada 8(3%) thegatewaypundit, breitbart, foxnews, lifesitenews, zerohedge 9(3%) francesoir, fr, reseauinternational, childrenshealthdefense, daily- mail sputnik fr, VirusWar, franceinfo, afpfr 10(3%) nytimes, indianexpress, reuters, theguardian, bloomberg CNBCTV18News 11(2%) reuters, buzzfeednews, theguardian, nytimes, prevention ANCAlerts, CNNPhilippines 12(2%) imolaoggi, zerohedge, rt, dailymail, nytimes RT com, SputnikInt 13(2%) npr, nytimes, latimes, theguardian, wsj NPRHealth, WHO, EU Health, CovidSupportSA 14(1%) humansarefree, dailymail, rt, childrenshealthdefense, zerohedge Another Spanish-English tweets community (C14) contains strongly anti-vaccine con- spiracies, very similar to the larger Anti-vaccine misinformation community. A French tweets community (C9) with relatively less conspiracy content but anti-vaccine, and an Italian tweets community (C12) of mixed stance to vaccine hesitancy. • Benign communities included Mainstream News (15.58%) (C2), Health news (1.48%) (C13), U.S. Left Leaning (C0) (with Joe Biden, Kamala Harris as top retweeted ac- counts). The former contain accounts with more global geolocations, while the latter was dominantly with US geolocations. There are several regional news and politics communities centered close to the Mainstream and Health News communities (e.g. UK based with top retweeted accounts corresponding to the National Health Service NHS and Department of Health and Social Care DHSC (C5), Latin America (C6), Philippines News (C11), India and Canada News and Politics (C4, C10 and C7)). These are identified based on tweets language, contents and inferred geolocations. 55 • TheU.S.Right-leaningcommunity(C3)(MikePence,OANN,topRepublicansasmost retweeted accounts), however different from other communities, has roughly equal pro- portions of unreliable, conspiracy and reliable URLs in their tweets. In terms of tweet contents and proximity to other communities, the right-leaning community is closer to the Anti-vaccine misinformation and conspiracy community (C1), as well as the far right group (C8), with relatively sparse edges to the global mainstream informa- tional community (C2,C13). The top news domains in their tweets include relatively credible sources with possible right-leaning bias such as Foxnews, and also conspiracy sources like zero-hedge and Truepundit. Inspecting top retweeted and random sample of tweets suggests a mixture of vaccine news updates, politically biased views and also conspiracies (e.g. numerous anti-China messages, Bill Gates conspiracies), and largely anti-vaccine/protocol stance (including misinformation about the vaccines). In Tables 4.2, note that unreliable, conspiracy, reliable news source URLs proportions are included. For tweets containing no URLs or unidentified URL domains, the fraction of such tweets is listed under Others column in the Table. The inferred political leanings that are for majority of the accounts in a community are highlighted in bold. The asterisk is used for communities with more than a single prominent inferred leaning type. Languages and geolocations smaller than 10% in the tweets are not reported. Top News Domains and Top Retweeted Accounts are presented for each community, and left unmentioned if the top-20 ones contained only individuals that were not agencies or well known figures (e.g. C8). To leverage the social context, we identify communities that dominantly post or share disinformation sources vs. reliable sources. Several works have found that informed and misinformed user accounts exhibit echo-chambers in their network structure [110, 148]. For the identified communities, if tweets of user accounts in the community dominantly contain references to disinformation news sources, the communities are likely less to be credible, or are more misinformed. Disinformation communities would involve either malicious groups promoting disinformation, or groups with beliefs that support or are vulnerable to believing 56 and sharing disinformation on the topic of the discourse [110]. We can thereby leverage this to encode auser account’s credibility and stance with respect to disinformation on the topicofthediscourse. Here,wedenotesocialcontextofatweetasthecommunityoftheuser accountthatpostedthetweet. Giventhecommunitystructure,thetweetcascadeisdetected aspossiblymislabeledif(1)theuseraccountbelongstoanidentifieddominantdisinformation (unreliable/conspiracy) news-source sharing community, but the tweet is weakly-labeled as reliable (2) or, if the account is in a dominantly reliable information sharing community, but the weak label is unreliable/conspiracy. For mixed communities with unclear dominant patterns, we have no definitive social context, and use only the detection entropy. 4.3.3 Iterative Label Refinement We jointly use the social context signal and the detection model entropy to guide the identi- ficationofpost-levelmistakesintheweakly-labeleddata. Theproposedframework(Fig4.3) is iterative and flexible. We can replace the disinformation detection model with any mod- eling choice, and use either self-supervision and/or human/model based relabeling. The detection model is first trained by itself with self-supervision from the model predic- tions. Then the improved detection model signals are jointly combined with social context for further label refinement. The process is iteratively repeated with evaluation of detection model on small held-out human labeled or fact-checked validation set as a proxy for label quality in the large-scale dataset. The procedure for label refinement from detection model and social context is described in Algorithm 2. The subroutine assumes as input the instance (tweet cascade denoted as x i , with its weak label y i ), the detection model M trained in the previous iteration, and the social context S. Given the model state, we generate three possible actions: (1) RETAIN weak label (2) FLIP weak label (3) QUERY label. Action retain keeps the instance with its weak label in the dataset, flip is model-guided relabeling (without human resource), and query is for active human relabeling of the model suggested instance. If the human resource 57 is not available, then QUERY can be replaced by REMOVAL (discarding the instance due to low confidence in its label or due to contradictory confident signals from detection model M and social context S). The states from the detection model M and social context S are defined as follows, for instance x i , • M-lc: If high-entropy S i in detection model prediction (M-lc stands for low confidence, that is, high entropy) • M-consistentandM-inconsistent: IflowentropyS i ,andM predictedlabelequalsweak label then it is consistent (opposite predicted label and weak label, then inconsistent) • S-unk: no social context signal, either its user’s community is not dominantly reli- able or unreliable/conspiracy, but a mixture; or the user is not clustered in any main community. • S-consistent and S-inconsistent: If the social context of a user account (its community label) is (in)consistent with the weak label of its tweet in x i (as described earlier). The objective of the procedure is to minimize human relabel queries, and incorporate high confidence signals from both detection model M and social context S to ultimately remove or correct as many inaccurate weak labels, keeping as many correctly weak labeled instances. If the signals reinforce each other, the procedure can more confidently take an action without human label querying (or removal/discarding of the instance). Given the state, the appropriate action is selected by the procedure Alg. 2. Fine-Grained Semi-Supervised Classification The dataset is refined based on retain- ing weak labels, model based relabeling, and human relabeling or removal of the instance. The retained and refined instances form the output constructed dataset with the associated model confidence in its label. The fine-grained labels are obtained by the human labeling but only on selected instances false, unproven, mixture, mostly false, mostly true, true, anddebunk. Fortheremaininginstances, wecanusea semi-supervised classification setup[81,173]toobtainfine-graineddistinctions. Thefewobtainedhumanlabeledinstances become class prototypes to separate the rest into the seven classes. The distinctions can be 58 Algorithm 2 Label Refinement Procedure Require: Dataset instance x i , weak label y i , and detection model M, and social context S Ensure: Action: RETAIN, FLIP, QUERY label 1: if M-consistent and S-consistent then 2: RETAIN with weak label // reinforced signals from M and S 3: else if M-inconsistent and S-inconsistent then 4: FLIP weak label // reinforced signals from M and S 5: else if (M-consistent and S-inconsistent) or (M-inconsistent and S-consistent) then 6: QUERY label // contrasting signals from M and S 7: else if M-consistent and S-unk then 8: RETAIN with weak label // only signals from M 9: else if M-inconsistent and S-unk then 10: FLIP weak label // only signals from M 11: else 12: QUERY label // detection model high entropy filtering 13: end if very nuanced with varying degrees of truth, and difficult for a model to distinguish very accurately, so we provide these as auxiliary outputs. 4.4 Experiments We study the proposed approach for constructing a large-scale public disinformation dataset on COVID-19 vaccines. We use iterative self-training of the disinformation detection model CSI [138] trained first on the initial weakly-labeled cascades, i.e., 10,377 weakly labeled as reliable and 4,267 weakly labeled as unreliable/conspiracy cascades. 4.4.1 Evaluation Tasks and Metrics The evaluation test set of tweet cascades contains 256 tweets with ground-truth fact- checked claim based labels obtained by searching for tweets related to Snopes fact-checks and AP news/NPR news on COVID-19 vaccines and labeling from the 7 fine-grained labels according to the annotation scheme and fact-checked claims. For experiments, a human- labeled validation set of 150 tweets, based on the annotation scheme and guidelines, is also constructed and held-out from the 14.6k tweet cascades (as described in Sec 4.2.2). 59 Evaluation Tasks We cannot directly measure the quality of the constructed disinforma- tion dataset, since we cannot obtain ground-truth fact-checker (e.g. Snopes) labels on all 14.6k tweet cascades. We instead evaluate on the fact-checked claim based test subset of 256 tweet cascades using the following evaluation metrics (i) Disinformation detection performance on test set. Label quality in the dataset should be positively correlated with disinformation detection accuracy on ground-truth labeled data. (ii) Label correction accuracy on validation and test sets and (iii) # of wasted queries generated in the label refinement procedure, to measure human resources that are inefficiently utilized . Evaluation Metrics (i) The baselines and proposed experiments are evaluated fordisin- formation detection (classification) performance on the ground-truth test set of 256 tweet cascades. The detection model performance is averaged over 5 random seeds. The evaluation includes the classification metrics namely, Area under the precision-recall curve (AP) and Area under the ROC curve (AUC) and F1 and Macro-F1 for the disinformation classification. It measures how well the refined/constructed dataset from baselines or the proposed approach work in separating the disinformation from reliable tweet cascades. (ii) The baselines and proposed experiments are evaluated for label correction accu- racy on the ground-truth test set and validation set. We have the initial weak labels and correct disinformation labels for the test and validation set. Therefore, we can measure the recall (Rec), precision (Prec), and F1 of the noise in the weak labels (i.e., weak label and ground-truth fact-checked label are not aligned). The instances detected as noise by the methodsareonesselectedforFLIPorQUERY(REMOVE)actions(asitispredictedbythe method as having a possibly mislabeled weak label). Recall is the fraction of actual noise in weaklabelsthatarecorrectlydetectedbythemethods, andPrecisionmeasuresthecorrectly recalled noise in weak labels out of all instances detected as noise by the methods. F1 is the harmonic mean of precision and recall. (iii)Weadditionallyproposeametrictoalsomeasurehowefficientlytheresourcesare 60 Table 4.4: Results on classification performance on test set from detection model with label refinement proposed approach for disinformation dataset construction on COVID-19 vac- cines. Metrics: AP (average precision), AUC (ROC), F1 and Macro F1 Experiment AP AUC F1 Macro F1 Weak labels 0.722± 0.03 0.876± 0.01 0.774± 0.02 0.812± 0.01 Self-training (iteration 1) 0.768± 0.01 0.888± 0.0 0.812± 0.01 0.842± 0.01 Self-training (iteration 2) 0.775± 0.02 0.891± 0.0 0.811± 0.01 0.842± 0.01 Social-context only 0.764± 0.02 0.891± 0.01 0.810± 0.01 0.837± 0.01 Social+Detection model 0.785± 0.02 0.895± 0.0 0.813± 0.01 0.842± 0.01 Social+Detection (+label correction) 0.800 ± 0.01 0.895 ± 0.0 0.818 ± 0.01 0.845 ± 0.01 Table4.5: Resultsfornoisedetectioninweaklabelswithlabelrefinementproposedapproach for disinformation dataset construction on COVID-19 vaccines. Evaluation metrics: Rec (noise recall), Prec (precision), FracUQ (fraction of unwanted queries), F1 (F1 of detected noise in weak labels) Test set Validation set Experiment Rec Prec FracUQ F1 Rec Prec FracUQ F1 Naive 1.0 0.1719 1.0 0.2934 1.0 0.1533 1.0 0.2658 Self-training (iteration 2) 0.5682 0.3846 0.1887 0.4587 0.6522 0.4054 0.1732 0.5000 Social-context only 0.4545 0.4444 0.1179 0.4494 0.4348 0.3704 0.1339 0.4000 Social+Detection model 0.8409 0.3978 0.2642 0.5401 0.7826 0.3673 0.2441 0.5000 Social+Detection (+label flipping) 0.8409 0.3978 0.2406 0.5401 0.7826 0.3673 0.2126 0.5000 utilized by the baselines and proposed methods. We define Frac UQ (fraction of correctly weak-labeled instances that are assigned QUERY (REMOVE) action for human relabeling (removal), i.e. unwanted or wasted queries) as follows, Frac UQ = | (QUERY action assigned) & (correct weak label)| |correct weak label| (4.3) The # of instances with correct weak labels assigned QUERY (REMOVE) action is the numerator, measuring human resource wastage. Lower value of Frac UQ is better. 4.4.2 Experiment Results Disinformation Detection In Table 4.4 we provide results of the proposed framework to construct disinformation datasets from weak labels. We trained the CSI [138] disinforma- tion detection model on weak labels from news source credibility to classify disinformation (unreliable/conspiracy) tweets from reliable information tweets, as a baseline. The held-out 61 Table 4.6: Fine-grained classification from human labeled class prototypes to remaining examples in the dataset Expt (5-fold) Macro F1 Weighted F1 F1 (debunk) (true) (mostly true) (mixture) (mostly false) (false) (unproven) Random 0.113 +/- 0.037 0.201 +/- 0.067 0.071 +/- 0.089 0.254 +/- 0.111 0.042 +/- 0.084 0.16 +/- 0.101 0.125 +/- 0.125 0.097 +/- 0.058 0.044 +/- 0.089 Majority 0.105 +/- 0.007 0.428 +/- 0.078 0.0 +/- 0.0 0.733 +/- 0.049 0.0 +/- 0.0 0.0 +/- 0.0 0.0 +/- 0.0 0.0 +/- 0.0 0.0 +/- 0.0 Unweighted MLP 0.219 +/- 0.022 0.552 +/- 0.073 0.04 +/- 0.08 0.757 +/- 0.045 0.0 +/- 0.0 0.257 +/- 0.074 0.0 +/- 0.0 0.483 +/- 0.064 0.0 +/- 0.0 Weighted MLP 0.266 +/- 0.055 0.57 +/- 0.077 0.235 +/- 0.145 0.735 +/- 0.05 0.033 +/- 0.067 0.338 +/- 0.165 0.0 +/- 0.0 0.52 +/- 0.1 0.0 +/- 0.0 validation set is used by the detection model for early stopping in model optimization, and for calculating the threshold for detection based on AUC curve, trading off sensitivity and specificity on the validation set. The reported results are on the held-out ground-truth test set of fact-checked based labels. The same setting is used in all experiments. The removal (entropy filtering) guided by the detection model (self-training iteration 1 and 2) improves the classification on the ground-truth test set, indicative of higher label quality in the retained tweets. After 2 iterations, the improvement was insignificant. Further, incorporating the social context, we first evaluate Social-context only, wherein the tweets with labels opposite to their community label (dominantly reliable or dominantly disinformation)aretobequeriedorremoved. Wefindthatcombiningthesocialcontextmod- elinganddetectionmodelguidanceismoreinformativeaboutpossiblemislabeling(tweetsto be removed) in the weak labels (Social+Detection model). Finally, Social+Detection model (+label correction) is used to correct the labels that the two signals suggest should be op- positely labeled, and remove ones that the model is unsure about either from the detection model or social context (i.e., using the label refinement procedure in Alg 1). We find results in model-guided label refinement for construction of disinformation datasets is effective and significantly improves both recall in disinformation detection, (since the disinformation ex- amples are fewer in the imbalanced data), and the precision of detected disinformation, and other metrics separating the two classes of disinformation and reliable information. Label Correction In Table 4.5, we provide the results of performance on label correction using the signals from the detection model and/or social context. Naive baseline is trivially settoassumeallweaklabelsaremislabeled,andQUERYallofthem. Therefore,allcorrectly 62 debunk mostly_true true 0.10 0.15 0.20 0.25 0.30 Probability of Class 1 (False) Boxplot grouped by human_gt_label false mixture mostly_false unproven 0.4 0.5 0.6 0.7 Probability of Class 1 (False) Boxplot grouped by human_gt_label Figure 4.5: Predicted probabilities from detection model after label refinement: correlation with true labels on ground-truth fact-checked human-labeled test and validation set weak-labeled instances are Queried with worst resource utilization of 1, and low precision, F1 scores. For the removal (entropy filtering) guided by the detection model (self-training iteration 2) has roughly 56% recall in inaccurate weak labels, with reasonable precision and low Frac UQ. It is similarly the case for Social-context only method. With the proposed approach (i.e., using the label refinement procedure in Alg 1) com- bining the social context modeling and detection model guidance is more informative about possible mislabeling in the weak labels (Social+Detection model) and we see a massive in- crease in recall on combining the two signals, resulting in 0.84 recall. This might suggest that the signals provided by detection model entropy filtering and social context are com- plementary to each other, and jointly inform label refinement in the proposed algorithm most effectively. With label FLIP action included (Social+Detection (+label flipping), the difference is only in Frac UQ, where now some of the detected noise will be directly selected forFLIPactioninsteadofforremoval(orquery), minimizingthewastedqueries, iftheFLIP was assigned to actual noisy instances. Analysis In the constructed disinformation dataset, in Fig. 4.5 we examine scatter plot of instances on the predicted probability of disinformation from the detection model, which as we see is correlated with the fine-grained human labels available on the validation and test set, capturing the varying degree of truth. In Table 4.6, we show the fine-grained classification from human labeled class prototypes on remaining examples in the dataset, 63 using 5-fold cross validation on stratified splits of the validation plus test set for evaluation. For classification, we additionally labeled 400 instances to include as human-labeled class prototypes. We used extracted representations of tweet cascades from the CSI detection model used here, to train an MLP. With class-weighting, the fine-grained classifier has 0.57 weighted F1 distinguishing over the 7 nuanced label categories which is a difficult task. 4.5 Discussions and Conclusion The proposed label refinement approach is effective at constructing large-scale datasets from weak labels with high recall of inaccurate weak labels when incorporating social context jointly with entropy filtering, and both provide complementary information. There are some limitations of the approach and potential for future research. • First, the weak labels from news-source credibility introduces bias in the dataset to- wards news-worthy contents. An approach to mitigate this bias might be to consider other directions to augment the weak labels. • Second, the proposed approach models instance credibility and user credibility. In addition, one could also model the news-source credibility for each source separately, since each might introduce different noise rates. For instance, unreliable sources are more mixed than conspiracy sources, as we observed in Fig 4.2. • Lastly,thefine-grainedclassificationandselectionofexamplesforannotatorguidelines is still a difficult task. Future research directions could explore richer semi-supervised fine-grained classification techniques for imbalanced classes. To conclude, the proposed label refinement with social context modeling is a useful, new approach for constructing disinformation datasets, in a timely and scalable way for new or evolvingdomains. Inaddition, thereareusefulopenquestionsthatwarrantfurtherresearch. 64 Chapter 5 Disinformation Detection Leveraging Social Media Responses Detecting disinformation on social media is normally formulated as a discriminative task of learning features that can differentiate disinformation from legitimate contents using la- beled datasets. In earlier chapters, we reviewed different content-based and feedback-based detection models, as well as approaches for constructing labeled disinformation datasets for training and evaluation of disinformation detection models. Existing content-based disinformation detection models rely on writing style and lin- guistic cues for deception detection by training discriminative models on labeled datasets [40, 170]. However, social media context i.e., user responses or engagements to the content are found to provide more informative signals than the content alone, as they provide social media users’ feedback, and are therefore exploited in the feedback-based models [138, 106]. While both content and social media responses are useful for detection, waiting to collect social media engagements results in increased exposures to potential disinformation. Early detection is important to limit the propagation and exposures to disinformation. In this Chapter, we assume that for early detection, only contents to be verified are available whilst socialmediaresponsesorengagementsareunavailableatthetimeofverification. Addressing specifically this challenge of early detection, we propose a model TCCN-URG that can instead leverage historical user responses to improve content-based detection models. 5.1 Early Detection Leveraging Historical Responses Existing content-based methods take as input contents or features extracted from contents to train supervised detection models on labeled disinformation datasets [123, 41, 170]. In contrast, feedback-based models take sequence of social media posts (user responses or en- 65 gagements) which are ordered temporally and consist of comments posted by users on social mediarelatedtothepropagatingcontent. Thesecommentscontainusefulinformationabout the veracity based on collective judgements of social media users, as well as patterns such as negative or questioning responses that can act as signal of content veracity [186]. Content- based and feedback-based signals are normally treated separately in detection models. The prior works in disinformation detection also primarily focus on learning discrimina- tive features, but do not specifically address challenges in early detection. Content-based methods can directly be used for early detection, but lack the useful social context features. Feedback-based models instead rely on engagements, and could be used for early detection where only part of the engagements (in the first 12 hours, 24 hours, etc.) are considered in test set cascades. However, since the models are trained on cascades with several en- gagements, the performance degrades significantly at test time for early detection, when the engagement signals become sparse [104, 138]. Also, it cannot be applied to a strictly early detection setting when no engagements are available yet, as we consider here. Leveraging Historical Social Media Engagements Although in early detection as we consider here, social media responses are not available during verification of new contents, but we find that historical user responses to past contents are available, and can be lever- agedtoenhancecontent-baseddetection. Specifically, wefindthatthesehistoricalresponses can be treated as soft semantic labels, that enrich the binary label of contents (disinforma- tion/legitimate), providing insights into why the content must be labeled as disinformation. Toillustratethis,wesampleadisinformationcontentfromthedatasetwithfourcorrespond- ing user responses as shown in Figure 5.1, related to the banning of a drug by the FDA. The responses contain information that can explain why this content is false, by the users. 66 Disinformation content with social media responses ... FDA quietly bans powerful life-saving intravenous Vitamin C ...It would be naive to think that the FDA endeavors to protect the public’s health as its primary focus ... User 1 This is an absolute disgrace. It is a well known cure. User 2 Whyisthe#FDAquietlybanninglifesaving#natural #medicine ? User 3 It’s funny, I just had it yesterday in the hospital User 4 Not really reliable, since a drug need to be tested re- peatedly before being approved by FDA. They won’t ban something so easily. It costs too much money... Figure 5.1: An example to show that why historical user responses on social media can be utilized as rich soft semantic labels to help the early disinformation detection from contents 5.2 Early Detection Model Based on that, we propose a conditional variational auto-encoder based detection model (TCNN-URG)forearlydetection. Two-LevelConvolutionalNeuralNetwork (TCNN) captures semantic information from content text by representing it at the sentence and word level, and User Response Generator (URG) learns a generative model of user response to content text from historical user responses which it can use to generate responses to new content. Theproposedmodelcanprovidedeepersemanticanalysisandunderstandingofthe content and its veracity through the relationship between the content and the correspond- ing user responses it invokes. The TCNN-URG are combined to perform early fake news detection in that TCNN extracts content representation and URG generates user responses conditionedonthecontentrepresentationandthecontentrepresentationandgenerateduser response are used for final classification. The proposed model architecture is in Fig. 5.2. 5.2.1 Two-Level Convolutional Neural Network TCNN represents content at the sentence and word level. Each word is represented by a distributed vector representation or pre-trained word embeddings [112], and each sentence is obtained by average pooling of the associated word embeddings in the sentence. The 67 Average Pooling Label TCNN URG w1 w3 w2 s1 s2 Article …. …. Latent Distribution Generative Network Convolutional Layer Fully Connected Layer Fully Connected Layer w2 w1 w3 Generated Response (a) Architecture of TCNN-URG User Response Latent Distribution Generated Response Σ ϵ z Article V ector Inference Network Generative Network (b) Generative process of URG condi- tional generator Figure 5.2: Model architecture for early detection content representation is derived from the sentence representations first by concatenation of each sentence representation. A convolution operation then applies a filter w ∈ R hk to a window of h sentences moving through the content to extract semantic information features from the content. A feature c i is generated from a filter t and a window of s i:i+h− 1 by: c i =g(t· s i:i+h− 1 +b) (5.1) where b∈R is the bias and g is an activation function. After that, a max pooling operation is applied to the feature map so that the maximum value within each window is taken as the output of the corresponding filter. Multiple convolutional filters of varying length or parameters are used to capture extract content representations. Finally, the features are used as the input to a fully connected layer and a softmax activation for predicting the veracity, as predicted probability of disinformation f i for contentd i , and can be trained with cross-entropy loss− P d i ∈D log p(f i |d i ,θ ) where θ represents all TCNN parameters and D is the training set. 5.2.2 User Response Generator A generative conditional variational autoencoder [156] is applied as the User Response Gen- erator (URG). More specifically, the variational autoencoder is trained to generate user responses conditioned on the content representation from TCNN, modeling stochastic user 68 responses to a given content. It can learn a distribution over user responses and use to simulate responses conditioned on content, from the learned distribution. URG models the relations between the content y, the user response x and a latent generativevariablez. Theinferencenetworkandthegenerativenetwork,namelytheencoder and decoder is defined as: q ϕ (z|x,y) and p θ (x|z,y), where ϕ,θ are the parameters of the respective networks, since samples of both x and z and drawn under the influence of the content (article) y. Under the influence of content y, encoder encodes user response x into latent variablesz, and then, decoder decodes latent variable z to reconstruct user response x. We construct the input vector y as the content vector generated by TCNN. As for user responsex, we construct a binary vector of vocabulary size to indicate which words appear in the user response. The conditional generative network learns to reconstruct the user response, resulting in a vector of vocabulary size with each component being the probability of the word’s occurrence in the user’s response. To learn the latent parameters of the generative model, we use the re-parameterization trick [80] with z = g ϕ (ϵ ,x,y) = µ +σ ⊙ ϵ where ϵ ∼ N (0,I), so that we can learn the parameters using back propagation by minimizing the following objective over each user response and content pair (there can be multiple user responses for a given content and we treat them as separate training samples indexed by i and j) as follows, E z∼ q ϕ (.| x (i) ,y (i) ) h − logp θ (x (i) |z,y (i) ) i +D KL (q ϕ (z|x (i) ,y (i) )|| p θ (z)) (5.2) The first term is the reconstruction error designed as the negative log-likelihood of the data reconstructed from the latent variable z conditioned on content y. The second term, the regularization, is used to minimize the KL-divergence between the encoder distribution q ϕ (z|x,y) and the prior distribution p θ (z). URG is depicted in Fig. 5.2b. 69 5.2.3 Integrated Framework of TCNN-URG The content vectory extracted from TCNN is used to condition the responses generated by URG.TheuserresponsesgeneratedbyURGarethenputthroughanonlinearneuralnetwork layerand concatenated with the content vector extracted by TCNN.The final featurevector isfedtoafeedforwardsoftmaxclassifierforclassificationinthemodelarchitectureasshown in Figure 5.2 to predict the content veracity. We formulate the problem of predicting output f for the contenty as, p(f|y) = Z x p(f|y,x)p(x|y)dx (5.3) where the integration is intractable, so it is approximated with the expectation for which we can derive a Monte Carlo estimate as p(f|y)≈ E p θ (x|y,z) p(f|y,x) where the expected value is calculated as average over 100 samples generated from URG condition on y. The training process is divided into three steps: (i) TCNN is first trained separately to minimize cross-entropy loss with training set contents with mini-batch gradient descent. (ii) URG is trained next. The training target is to minimize the loss function described in Eq. 5.2 using historical user responses and content representations from trained TCNN in training set contents. (iii) The TCNN and URG are combined together by first obtaining the content vector from TCNN and generating user responses based on the content vector, followed by retraining the unified model to minimize the former cross-entropy loss on predicted labels. 5.3 Experiments 5.3.1 Experimental Set-Up Datasets The early detection method is evaluated on two real-datasets. The first dataset Weibo [104], corresponds to disinformation and legitimate contents identified on a Chinese micro-blogging social media platform. The contents are social media posts usually less than 100 words. Due to unavailability of existing datasets with both long texts or contents and 70 Table 5.1: Datasets used for early detection, contents with engagements i.e., user responses Dataset Weibo [104] Twitter(Ours) Total # Engagements 3,805,656 2,841,179 # Disinformation contents 2,313 13,552 # Legitimate contents 2,351 11,523 Avg # engagements / content 816 113 associated social media engagements, we additionally collected a second Twitter dataset 1 (described in Chapter 3), with 25k news articles labeled as reliable or unreliable based on trustworthy websites and disinformation publishing websites such as NaturalNews compiled in Zimdars [191], with associated engagements on Twitter. This dataset thereby contains longtextsofaveragelengthof950wordscorrespondingtothecollectednewsarticlecontents, alongwithassociateduserresponsesorengagementscollectedusingtheTwitterAPI.Dataset details are in Table 5.1. Preprocessing and Training Details We separate punctuation and replace specific strings with tokens denoting their types. These notations are standardized as: mention notations such as @xx are converted into [[@]], hashtag #xx to [[#]], time notations such as “17:21” and “07:12:2013” are converted into [[time]], data notations such as “17.21” and “3/5”and“60%”areconvertedto[[data]],andmoneynotationssuchas“$11”isconvertedto [[money]]. website urls to [[url]]. The text is tokenized and word embeddings are pre-trained [112] on collected contents. In the experiments, we set the word embedding dimension to be 128 and filter size to 2,4,5. For each filter size, 64 filters are initialized randomly and trained. When generating user responses from URG, we use the average of 100 samples. For training, we use a mini-batch size of 64 and contents of similar length are organized in the same batch. We build and train the model using TensorFlow and use ten-fold cross validation for evaluation of the model. 1 The Twitter dataset collected for early detection is available here: https://drive.google.com/drive /folders/1ArxS2LxcPvK3r8gXFWJ0tUpE_XpoRNJm?usp=sharing 71 Table 5.2: Results on Weibo dataset with different % of training data used 10% 20% 30% 40% 50% 60% 70% 80% 90% LIWC 56.61 58.24 61.25 61.48 63.66 64.27 64.97 65.4 66.06 POS-gram 63.67 63.46 65.82 66.45 67.36 68.88 72.19 72.91 74.77 1-gram 80.19 80.89 81.32 81.35 82.47 83.01 83.59 84.03 84.76 CNN 74.03 81.34 81.89 82.82 84.01 84.56 84.86 85.11 86.23 TCNN 76.06 82.51 84.32 84.72 85.97 86.86 86.92 87.46 88.08 TCNN-URG 79.00 84.52 85.51 86.26 88.05 88.41 88.43 88.56 89.84 Table 5.3: Results on our Twitter dataset with different % of training data used 10% 20% 30% 40% 50% 60% 70% 80% 90% LIWC 51.8 53.47 55.2 56.25 57.74 58.96 60.34 60.9 62.13 POS-gram 57.55 64.44 66.36 68.21 69.06 69.32 69.86 69.87 70.34 1-gram 76.47 77.57 78.23 79.09 79.4 79.69 80.38 80.37 80.69 CNN 70.15 72.2 76.65 78.49 80.26 80.45 81.15 82.74 83.24 TCNN 77.46 77.59 78.21 80.18 81.61 83.73 84.29 85.96 86.02 TCNN-URG 77.47 77.71 79.38 81.92 83.98 86.13 86.68 88.28 88.83 Baselines Therangeofbaselinesincontent-baseddetectionusedifferenttechniquesforde- tecting disinformation from text content ranging from feature engineering to using linguistic techniques as well as neural network models, as stated below: • LIWC. [123] uses hand-crafted Linguistic Inquiry and Word Count (LIWC) [126] features for text analysis, which is a widely used lexicon of psycho-linguistic cues used in deception detection and other social science studies. • 1-gram. Features used in former deception detection works [123, 40]. In order to keep comparisondimensionfair, weusetf-idftochoosevocabwordsforthe1-gramfeatures. • POS-gram. [40, 123] Linguistic cues (Part-of-speech tags) for deception detection. • CNN. Convolutional neural networks have achieved state-of-the-art in text classifica- tiontasksandoutperformedrecurrentneuralarchitectureslikethebidirectionalLSTM (long short-term memory) in former deception detection work [170]. The text is rep- resented at the word-level and fed to the CNN that extracts semantic representations. 72 A disinformation article sampled from the test set: ... FDA quietly bans powerful life-saving intravenous Vi- tamin C ...It would be naive to think that the FDA en- deavors to protect the public’s health as its primary focus ... Top 20 words generated by URG in response: [[ ! ]] [[ ? ]] [[ @ ]] [[ link ]] c care fda food false health intravenous life only problem protect rich tax wait watch why Figure 5.3: Top 20 response words generated by URG presented in alphabetical order 5.3.2 Detection Results Experimental results on Weibo dataset are shown in Table 5.2, and results on self-collected Twitter dataset are shown in Table 5.3. We present the results in terms of the accuracy of detection, on varying percentage (10-90%) of data samples used as training data to evaluate the variation and stability in performance for the evaluated methods. Overall, TCNN out- performs the other methods compared against including CNN, and moreover, URG further improves the accuracy of TCNN even when the training data is limited. TCNN uses convolutions over sentence-level representations, compared to CNN which learns filters over word-level representations where the sliding window is only over neigh- boring words, and therefore TCNN performs better than CNN. Both are better than earlier content-based chosen hand-crafted or linguistic cues as features, since they leverage neural nets for automatic feature extraction. URG further improves the accuracy of TCNN as can be seen in both result tables. URG learns the nature of user responses conditioned on the content text and is able to generate responses to new contents for early detection. By capturing the intricate relationship between contents and user responses, the URG empow- ers the system with user wisdom that is not directly available from the content text alone. To further understand how URG works, we sample an example from the test set for which we already provided the true user responses in Figure 5.1. We specifically eliminated this article from the training data. Even though the true responses are not seen by the URG, 73 sincethecontentisinthetestsetonly,theURGisstillabletogeneratereasonableresponses to the content using the inferred latent parameters encoding the relationship between user responses and contents. We provide the top 20 response words that are generated by URG for this example disinformation content as listed in alphabetical order in Figure 5.3. We can seethatURGgeneratessomenegativeresponsesandquestioningresponsessuchas[[?]], [[!]], false, problem and so on, which are useful signals of its veracity. 5.4 Discussions and Conclusion In this Chapter, we proposed TCNN-URG for early detection from disinformation contents towards preventing disinformation exposures and viral cascades of disinformation contents. TCNN-URG is able to capture semantic reasoning from user responses to past contents, which it can utilize to guide the prediction for new contents, since it indirectly acts as soft semantic labels enriching the simple binary labels of veracity during learning. The simulated responses to disinformation contain more negative and questioning re- sponses such as [[?]], [[!]], fake, problem. Besides improving accuracy over content-based solutions, TCNN-URG alsomake thepredictions more robust, sincethey relyon signalsthat implicitly encode and learn to explain disinformation contents, beyond just the predicted labels and do not suffer the performance degradation in feedback-based methods for early detection. Future work can further explore analysis of the robustness and generalizability of features learned under the proposed discriminative model guided by model interpretability. 74 Chapter 6 Detection of Coordinated Manipulation to Spread Disinformation Inthepreviouschapterweaddressedthedetectionofdisinformationcontentspropagatedon social media. While disinformation contents can spread organically through re-shares from susceptible users who believe the information, there can also be malicious account activities that promote and sustain these disinformation operations. In recent times, an increase in prevalence of coordinated disinformation or influence campaigns has been observed on social media platforms [49]. For instance, in 2016 a group of Russian accounts operated jointly by the “Internet Research Agency” acted together to amplify disinformation and divisivenarratives,inanefforttoinfluenceandmanipulatetheU.S.Election. Thepersistent abuse of social media by foreign state-backed operators and other disinformation campaigns forinfluencingpublicopinionandsocialoutcomeshasbeenincreasinglyconcerning[49,172]. To address the crucial task of identifying coordinated disinformation campaigns, earlier approaches have tried to uncover coordinated accounts based on their individual behaviors, from account features and participation in disinformation promotion [1], or collective group behaviors, such as activities synchronized in time [18, 124]. However, they all face inherent limitations. The methods relying on disinformation cues [1] or automated behaviors [44] to detect coordinated actors fall short in detecting human operated accounts that instead use persuasion, defamation, and polarization to manipulate opinions (characteristics noted in coordinated influence operations [107]). Existing methods exploiting collective behaviors heavily rely on assumed features of coordinated behavior e.g. similar sequence of hashtags as coordination signature or activities synchronized in time [124, 179, 168]. However such features can be inconsistent [183], limiting generalization to unseen accounts. Moreover, reliance on hand-crafted coordination signatures can only capture a limited range of behav- iors,andareineffectiveatreducingfalsepositives,makingstrictassumptionsoncoordinated 75 ー ー ‼ ⁉ ⁉ ⁉ ‼ ⁉ ⁉ ⁉ Figure 6.1: Coordinated accounts suspended by Twitter in COVID-19 data. The time difference of their coordinated activity varies from less than 6 hours to half a week groups which need not hold true. Fig. 6.1 shows an example of accounts officially suspended by Twitter and their activities observed in COVID-19 data, also identified as coordinated with our method. The time differences of coordinated activities observed from the accounts is diverse, with less than 6 hours in one case to more than half a week in another. Other supervisedalternatestocoordinationdetection[1,101]requirepartofthemaliciousaccounts in the coordinated group to be revealed to train supervised detection models to identify the rest of the accounts in the group. Moreover, Zannettou et al. 2019 found that coordinated behaviorsvaryacrossgroupsandtime, thereforesupervisedapproacheswouldnotgeneralize directlytofuturedetection. Inthiswork,toaddresstheseshortcomingsweproposetomodel the following inherent characteristics of coordination. • Strong hidden influence. If accounts coordinate to amplify social media posts or target specific individuals, there should be a strong hidden (latent) influence between theiractivities. Significantrecentcomputationalpropagandaisproducedandoperated by political operatives and governments [172]. • Highly concerted activities. The behaviors of coordinated accounts should be collectivelyanomalousfromnormalaccountswithlessorganizedactivitypatterns(i.e., observations that deviate from normal when generated by a different mechanism [60]). 76 To capture the proposed inherent characteristics of coordination, we propose AMDN- HAGE (Attentive Mixture Density Network with Hidden Account Group Estimation), an unsupervised,generativemodelforidentificationofcoordinatedaccounts,whichjointlymod- els account activities and hidden account groups based on Neural Temporal Point Processes (NTPP) and Gaussian Mixture Model (GMM). To learn the latent interactions or influence betweenaccountactivities, wemodelthedistributionoffutureactivitiesconditionedonpast activities of all accounts with temporal differences, and jointly capture collective anomalous behavior by simultaneously learning the group membership of accounts. To jointly optimize theactivitytraceandgroupbehaviormodeling,weiterativelyoptimizethemodelparameters with gradient descent and expectation-maximization from observed activity traces. 6.1 Task Definition In this section, we introduce the task of detecting coordinated accounts in social networks from collective or group behaviors of the accounts. Coordinated campaigns are orchestrated efforts where accounts collude to inorganically spread and amplify the spread of specific narratives for opinion manipulation, and the task we address is to identify such coordinated accounts. Inthiswork,weproposeAMDN-HAGE(AttentiveMixtureDensityNetworkwithHidden Account Group Estimation), an unsupervised generative model for coordination detection. It jointly models account activity traces and latent account groups, to learn collective group behaviors from observed account activities to detect coordinated accounts. ActivityTraces Theonlyinputweconsiderarethecascadeofaccountactivitiesontheso- cialnetwork. Anactivitytraceorengagementcascadeasdefinedinthepreliminarieschapter, represents a sequence of events ordered in time, C s = [(u 1 ,t 1 ),(u 2 ,t 2 ),(u 3 ,t 3 ),··· (u n ,t n )]. Eachtuple(u i ,t i )correspondstoanactivitybyaccountu i attimet i . Theactivitiesrepresent account engagements such as posting original content, or re-sharing or replying. 77 In order to provide platform/language independent detection, we do not include the type of action, or features such as content of the post, and account metadata, although additional available features can be easily incorporated in the method. The basic input is the most easily available for any social network. Furthermore, the only assumption we make is that compared to normal accounts, the number of coordinated accounts is quite small and coordinated users have highly concerted activity patterns (i.e., collectively anomalous). Hidden Account Group In real social networks, accounts with similar activities form social groups, that can constitute normal communities as well as coordinated groups. Sup- posing that there are N groups in the account set U, we can define a membership function M : U →{1,··· ,N}, which projects each account u i to its group index. This membership in many cases is hidden or unknown [9]. Acquiring M can help us identify collective anoma- lous group behaviors to detect coordinated groups. In this work, we aim to learn the hidden group memberships of accounts from only the observed activity traces. Temporal Point Process A temporal point process (TPP) is a stochastic process whose realization is a sequence of discrete events in continuous time t ∈ R + [27]. The history of events in the sequence up to time t are generally denoted as H t = {(u i ,t i )|t i < t,u i ∈ U} whereU representsthesetofeventtypes(here,accounts). Theconditionalintensityfunction λ (t|H t ) of a point process is defined as the instantaneous rate of an event in an infinitesimal window at time t given the history i.e. λ (t|H t )dt =E[dN(t)|H t ] where N(t) is the number of events up to time t. The conditional density function of the i th event can be derived from the conditional intensity [33] as p(t|H t ) =λ (t|H t )exp − Z t t i− 1 λ (s|H t )ds ! (6.1) In social network data, the widely used formulation of the conditional intensity is the mul- tivariate Hawkes Process (HP) [187], defined as λ i (t|H t ) = µ i + P t j <t α i,j κ (t− t j ), where 78 Conditional density function p( |history) Time t Event history on the network Emb. MH-Attn MH-Attn MH-Attn MH-Attn Emb. Emb. Emb. Position and time embedding Masked Self-Attention h1 h2 h3 h4 Context Activity Trace Modeling Normal Community Anomalous group (Coordinated) Social Group Modeling Jointly Learning P(Cs|U,θa,E) P(U|θg,E) Figure 6.2: Architecture of the proposed AMDN-HAGE to model conditional density func- tion of account activities and hidden groups on social media λ i (t|H t ) is the conditional intensity of event type i at time t with base intensity µ i > 0 and mutually triggering intensity α i,j > 0 capturing the influence of event type j on i and κ is a decay kernel to model influence decay over time. µ and α are learnable parameters. Since HP’sfixedformulationandfewlearnableparameterslimititsexpressivepower, recentworks propose to model the intensity function with neural networks [33, 109, 184, 194, 149, 121]. 6.2 Coordinated Accounts Detection Model Inordertocapturethelatentinfluencebetweenaccount’sactivities, andcollectivebehaviors of coordinated groups, as well as diversity in coordinated activities from such accounts, we introduce the proposed model AMDN-HAGE. AMDN-HAGE consists of two components: an Attentive Mixture Density Network (AMDN) that models observed activity traces as a temporal point process and a Hidden Account Group Estimation (HAGE) component that models account groups as mixture distributions. An overview is shown in Figure 6.2. The two components share the account embedding layer and reflect the complete generative processthattheaccountsarefirstdrawnfrommultiplehiddengroupsandtheninteractwith 79 each other so that activity traces are observed. Using the observed activity traces, we can learn the generative model by maximizing the likelihood function of the joint model, and acquire not only account embeddings but also a activity trace model and group membership function. Denoting the account embeddings as E, the parameters in AMDN as θ a and the parameters in HAGE as θ g , the joint likelihood function can be written as: logp(C s ,U;θ g ,θ a ,E) = logp(C s |U;θ g ,θ a ,E)+logp(U;θ g ,θ,E ) = logp(C s |U;θ a ,E)+logp(U;θ g ,E) (6.2) p(C s |U;θ a ,E) is the probability density that the activity traces are observed given a known account set, and p(U;θ g ,E) is the probability density that we observe the account set drawn from the latent hidden social groups. With the account embeddings and the learned membership, we obtain collectively anomalous groups as latent groups with anoma- lous distributions having small variance or size compared to the rest of the accounts, to detect coordination. 6.2.1 Modeling Latent Influence In this section, we introduce the AMDN component (Attentive Mixture Density Network) to model account activities for coordination detection. AMDN consists of two parts: a history encoder and an event decoder. Suppose we are modeling the activity (u i ,t i ), the history encoder represents all the activities that happened before t i as a vector H t i . Then the event decoder, which is the conditional density function, predicts (u i ,t i ) based on the history representation H t i and the known account set. This encoder-decoder architecture models activities with likelihood p(C s |U;θ a ,E) factorized as: logp(C s |U;θ a ,E) = L X i=1 logp θ a,E (t i |H t i )+logp θ a,E (u i |H t i ) (6.3) We provide architecture details in the following paragraphs. 80 Table 6.1: Summary of neural point process models Model Flexible intensity function Closed-form likelihood Long-range dependen- cies Interpretable Influence HP [187] no y y y RMTPP [33] limited y limited no FullyNN [121] y y limited no LogNormMix [149] y y limited no SAHP [184] limited no y y THP [194] y no y y AMDN y y y y AMDNArchitecture InTable6.1,wesummarizeexistingpointprocessmodels(detailed in Chapter 2.). These models suffer from different drawbacks with a trade-off on flexible intensity function (better expressive power), closed-form likelihood (reducing gradient noise intraining)andinterpretableinfluence(explicitinfluencescoreoneventpairs). Formodeling coordinatedaccounts,abovepropertiesarealluseful. Thus,wedevelopAMDN,whichhasall theaboveproperties. Weusemaskedself-attention[163](withadditionaltemporalencoding forhandlingirregularinter-eventtimes)toencodetheeventhistoryforinterpretableinfluence between past and future events (alternative to the recurrent neural network used in [149]), but still use a log-normal mixture distribution as event decoder to model the conditional density of the next event given the history (similar to [149]), achieving all properties. Encoding Event Sequences with Masked Self-Attention Let τ ∈ R + represent inter-event time, p(τ |H τ ) the conditional density. History H τ is encoded with a neural network to automatically extract useful features, similar to NTPPs. 81 Masked Self-Attention with Position Encoding For interpretable influence of past event on future events, we encode the event sequence with masked self-attention [163]. A =σ (QK T / √ d) and H attn =AV Q =XW q , K =XW k , V =XW v (6.4) with masked attention weights A of pairwise influence from previous events, input sequence representation X ∈ R L× d (L sequence length, d feature dimension), and learnable weights W q ,W k ,W v . At the end, we apply layer normalization, dropout and feed-forward layer to H attn to get output H out ∈R L× d . To maintain ordering of the history events, we represent the position information of the i-th event as an m-dim position encoding PE pos=i with trigonometric integral function following [163]. Account and Time Encoding We also represent account type and time (u i ,t i ) using learnable account embedding matrix E ∈ R |U|xm and translation-invariant temporal kernel functions[174], usingfeaturemapsϕ withmultipleperiodicfunctionsofdifferentfrequencies ω to embed inter-event times. The input X i (to the attention head) of the i th event (u i ,t i ), is a concatenation of event, position and temporal embedding as follows. ϕ ω (t) = [ √ c 1 ,··· √ c 2j cos(jπt/ω ), √ c 2j+1 sin(jπt/ω )··· ] (6.5) ϕ (t) = [ϕ ω 1 (t),ϕ ω 2 (t)··· ϕ ω k (t)] T (6.6) X i = [E u i ,PE pos=i ,ϕ (t i − t i− 1 )] (6.7) Event History Context Vector The attention mechanism gives us representations of each event using attention over prior events. We can use the representation of the last event or a recurrent layer over the attention outputs H out ∈ R Lxd to summarize events histories into context vectors C ∈R Lxd where L is event sequence length. Each c i ∈ C is a context 82 vector encoding history of events up to t i i.e. history H t i of the temporal point process. Decoding Event Sequences with Conditional Density With the encoded event history (context vector), the event decoder (learnable conditional density function p(τ |H τ )) is used to generate the distribution of the next event time con- ditioned on the history. While we can choose any functional form for p(τ |H τ )), the only condition is that it should be a valid PDF (non-negative, and integrate to 1 over τ ∈R + ). To maintain a valid PDF, exponential or other distributions with learnable parameters are generally used in point process models [187, 33, 149]. We define the PDF as mixture of log-normal distributions since the domain of τ ∈R + is non-negative (as in [149]), and mix- ture distributions can approximate any density on R arbitrarily well [149] not restricted to exponential or other monotonic functions. The conditional PDF is defined as, p(τ i |w i ,µ i ,s i ) = K X k=1 w k i 1 τs k i √ 2π exp − (logτ i − µ k i ) 2 2(s k i ) 2 ! (6.8) w i =σ (V w c i +b w ), s i = exp(V s c i +b s ), µ i =V µ c i +b µ (6.9) where the mixture weights w i , means µ i and stddevs s i are parameterized by extracted con- text history c i and learnable V,b. The encoder-decoder parameters (denoted jointly as θ a ) and the learnable account embeddings (E) can be learned using maximum likelihood esti- mation (for log-likelihood defined in Eq. 6.3) from observed activity sequences and account set, and trained with gradient back-propagation as θ ∗ a ,E ∗ = argmax θ a,E logp(C s |U;θ a ,E). 6.2.2 Modeling Hidden Groups For modeling the hidden social groups from the observed activity traces, we model hidden groups of accounts as a mixture of N multivariate Gaussians (GMM) in the account embed- ding space. Formally, the i-th group is modeled as a Gaussian distributionN(µ i ,Σ i ) where µ i is the mean and Σ i is the covariance matrix. Since the group membership is unknown, we 83 assume that account embeddings are drawn from the mixture, as embedding E u j of account u j is distributed as P i p(i)N(E u j ;µ i ,Σ i ), where p(i) is the prior probability of group i. Hidden Group Estimation A notable difference from general Gaussian mixture models, is that we define the GMM over the learnable account embeddings ( latent space), as com- pared to over observed variables. Therefore, the optimization and learning of AMDN-HAGE requires iterative optimization for jointly learning the model parameters and account em- beddings (as discussed in the next section). The model proposed here aims to capture latent or hidden groups from activity traces, rather than from observed account features. This is because coordination signals like activities strongly influenced by each other or a central agency remain in activity traces and provide information about their collective behaviors. Formodelingdistinctgroupsofcoordinatedandnormalaccounts,weusedtiedcovariance Σforallgroups. Denotingtheparameters(means,thesharedcovariance,andprior)ofGMM as θ g , the log-likelihood of social group modeling over accounts U is, |U| X j=1 logp(u j ;θ g ,E) = |U| X j=1 log N X i=1 p(u j ,i;θ g ,E) = |U| X j=1 log N X i=1 p(i)N(E u j ;µ i ,Σ) (6.10) 6.2.3 Jointly Learning and Optimization For joint learning, we maximize the following log-likelihood, logp(C s ,U;θ g ,θ a ,E) = logp(C s |U;θ a ,E)+logp(U;θ g ,E) = L X i=1 logp θ a,E (t i |H t i )+logp θ a,E (u i |H t i ) + |U| X j=1 logp(u j ;θ g ,E) (6.11) Above loss function has a trivial infinite asymptotic solution where all embedding vectors equal to the mean of a cluster and det(Σ) = 0. To avoid this solution, we constraint det(Σ) to be greater than a small constant λ , lower bounding the loss function. 84 Algorithm 3 Training Algorithm for AMDN-HAGE Require: Activity traces (C s ), Account set (U) Ensure: Generative model (θ a , θ g and E) 1: θ (0) a ,E (0) ← argmax θ a,E logp(C s |U;θ a ,E) 2: Set i as 1{Iteration index}. 3: while not converged do 4: θ (i) g ← argmax θ g logp(U;E (i− 1) ,θ g ) using EM algorithm 5: θ (i) a ,E (i) ← argmax θ a,E logp(C s ,U;θ (i) g ,θ a ,E) using SGD or its variants 6: i← i+1. 7: end while Thelog-likelihood(Eq6.11)isafunctionofparametersθ a ,θ g ,E. Inthis,theoptimization of the second term (Gaussian mixture of the shared latent embeddings), is a constrained optimization problem. Therefore, directly optimizing the joint likelihood with Stochastic Gradient Descent (SGD) or its variants like ADAM does not respect the constraints on mixture weights (normalized non-negative) and covariance (positive definite), leading to invalid log-likelihood in training (ablation study of loss with ADAM is in the expt section). To address above disadvantages, we provide an equivalent bilevel optimization formula- tion to solve the joint learning problem: θ ∗ a ,E ∗ = argmax θ a,E [logp(C s |U;θ a ,E)+max θ g (logp(U;θ g ,E))] (6.12) θ ∗ g = argmax θ g logp(U;θ g ,E ∗ ) (6.13) The above optimization can be solved with iterative optimization. In each iteration, we first freezeE andθ ,thenestimateθ g withEMalgorithm. Afterthat,wefreezeθ g anduseSGD(or its variant) to optimize E and θ a . Since the log-likelihood of latent embeddings is optimized in the second term (Eq. 6.11), we require initialization of embeddings by pre-training E and θ a on observed sequences by maximizing the first term in the objective function, before jointly optimizing the two terms. Detailed algorithm is in Algorithm 3. A generic concern to such alternating optimization algorithm is its convergence. There- fore, we give following theoretic guarantee that the proposed algorithm leads the model to 85 converge at least on a local minimum or a saddle point with appropriate selection of the gra- dient based optimizer (denoting L(θ a ,E,θ g ) as the negative log-likelihood (loss function)). Theorem 2. Our proposed optimizing algorithm will converge at a local minimum or a saddle point if in any iteration i the neural network optimizer satisfies following conditions, • Given the frozen θ (i) g acquired by EM algorithm in iteration i, the neural network op- timization algorithm we applied in converges at a local minimum or or a saddle point (θ (i) a ,E (i) ). • L(θ (i) a ,E (i) ,θ (i) g )≤ L(θ (i− 1) a ,E (i− 1) ,θ (i) g ), where θ (i− 1) a and E (i− 1) are the starting points in iteration i. The proof can be found in the Appx C. Theoretically, the two conditions can be guaran- teed when our loss function is L-smooth and we apply standard Gradient Descent Algorithm with learning rate lower than 1 L [116]. But in practice, since finding strict local minimum is notasimportantastrainingspeedandgeneralization, wecanapplyAdamorothervariants. 6.3 Incorporating Domain Knowledge In cases where additional domain knowledge about coordination patterns is available, we can incorporate that knowledge in the proposed model. To do so, we propose Variational Inference for Group Detection (VigDet). We can represent the domain knowledge as a graph between accounts, to capture patterns such as accounts that co-appear in cascades or havemoresynchronizedactivitiescouldbemorelikelytobepartofacoordinatedcampaign. VigDetmaintainsadistributionovergroupassignmentsanddefinesapotentialscorefunction that measures the consistency of group assignments in terms of both AMDN-HAGE account embedding space and the prior knowledge graph. The model is therefore jointly guided by the data-driven learning of AMDN-HAGE and the encoded domain knowledge. 86 Prior Knowledge-Based Graph Construction For the prior knowledge based graph construction, we apply co-activity [138] to measure the similarity of accounts. This method assumes that the accounts that always appear together in same sequences are more likely to be in the same group. Specifically, we construct a dense graph G =<V,E > whose node set is the account set and the weight w uv of an edge (u,v) is the co-occurrence. w uv = X S∈S 1((u∈S)∧(v∈S)) (6.14) However, when integrated with our model, this edge weight is problematic because the coordinated accounts may also appear in the tweets attracting normal accounts. Although the co-occurrence of coordinated account pairs is statistically higher than other account pairs, since coordinated accounts are only a small fraction of the whole account set, our model will tend more to predict an account as normal account. Therefore, we apply one of following two strategies to acquire filtered weight w ′ uv . Power Function based filtering: the co-occurrence of a coordinated account pair is sta- tistically higher than a coordinated-normal pairs. Thus, we can use a power function with exponent p> 1 (p is a hyper-parameter) to enlarge the difference and then normalize w ′ uv = ( X S∈S 1((u∈S)∧(v∈S))) p (6.15) where u∈ S and v ∈ S mean that u and v appear in the sequence respectively. Then the weightwithrelativelylowvaluewillbefilteredvianormalization(detailsinnextsubsection). Temporal Logic [95] based filtering: We can represent some prior knowledge as a logic expressionoftemporalrelations,denotedasr(· ),andthenonlycountthosesamplessatisfying the logic expressions. Here, we assume that the active time of accounts of the same group are more likely to be similar. Therefore, we only consider the account pairs whose active 87 E-step: Enhance Prediction with Belief Propagation on Prior Knowledge based Graph G M-step: Update Embedding and Potential Function with SGD to Fit the Knowledge Informed Prediction and the Observed Data Modeling the Embedding-based Potential Function via a Neural Network Knowledge Informed Prediction Observed Data Data-Driven Model Graph Construction φ θ Prior Knowledge Figure 6.3: The overview of VigDet. In this framework, we aim at learning a knowledge informed data-driven model. To this end, based on prior knowledge we construct a graph describing the potential of account pairs to be coordinated. Then we alternately enhance the prediction of the data-driven model with the prior knowledge based graph and further update the model to fit the enhanced prediction as well as the observed data time overlap is larger than a threshold (we apply half a day, i.e. 12 hours), w ′ uv = X S∈S 1((u∈S)∧(v∈S)∧r(u,v,S)), r(u,v,S) = 1(min(t ul ,t vl )− max(t us ,t vs )>c) (6.16) where t ul ,t vl are the last time that u and v appears in the sequence and t us ,t vs are the first (starting) time that u and v appears in the sequence. Integrating Prior Knowledge To integrate prior knowledge and neural temporal point process, while maximizing the likelihood of the observed sequences logp(S|E) given account embeddings, VigDet simultaneously learns a distribution over group assignments Y defined by the following potential score function given the account embeddings E and the prior 88 knowledge based graphG =<V,E >, Φ( Y;E,G) = X u∈V φ θ (y u ,E u )+ X (u,v)∈E ϕ G (y u ,y v ,u,v) (6.17) where φ θ (y u ,E u ) is a learnable function measuring how an account’s group identity y u is consistent to the learnt embedding, e.g. a feed-forward neural network. And ϕ G (y u ,y v ,u,v) is pre-defined as, ϕ G (y u ,y v ,u,v) = w uv √ d u d v 1(y u =y v ) (6.18) whered u ,d v = P k w uk , P k w vk arethedegreesofu,v and 1(y u =y v )isanindicatorfunction thatequals1whenitsinputistrueand0otherwise. Byencouragingco-appearingaccountsto be assigned in to the same group, ϕ G (y u ,y v ,u,v) regularizes E and φ θ with prior knowledge. With the above potential score function, we can define the conditional distribution of group assignment Y given embedding E and the graphG, P(Y|E,G) = 1 Z exp(Φ( Y;E,G)) (6.19) where Z = P Y exp(Φ( Y;E,G)) is the normalizer keeping P(Y|E,G) a distribution, also known as partition function [84, 82]. It sums up exp(Φ( Y;E,G)) for all possible assignment Y. As a result, calculating P(Y|E,G) accurately and finding the assignment maximizing Φ( Y;E,G) are both NP-hard [14, 83]. Consequently, we approximate P(Y|E,G) with a mean field distribution Q(Y) = Q u∈V Q u (y u ). To inform the learning of E and φ θ with the prior knowledge behind G we propose to jointly learn Q, E and φ θ by maximizing following objective function, which is the Evidence Lower Bound (ELBO) of the observed data likelihood logp(S|E) given embedding E. O(Q,E,φ θ ;S,G) = logp(S|E)− D KL (Q||P) (6.20) 89 Variational Inference In this objective function, the first term is the likelihood of the observeddatagivenaccountembeddings,whichcanbemodeledas P S∈S logp θ a (S|E)witha neural temporal point process model like AMDN. The second term regularizes the model to learn E and φ θ such that P(Y|E,G) can be approximated by its mean field approximation. Intuitively, this can be achieved when the two terms in the potential score function, i.e. P u∈V φ θ (y u ,E u ) and P (u,v)∈E ϕ G (y u ,y v ,u,v) agree with each other on every possible Y. The above lower bound can be optimized via variational EM [135]. In E-step, we aim at inferring the optimal Q(Y) that minimizes D KL (Q||P). Note that the formulation of Φ( Y;E,G) is same as Conditional Random Fields (CRF) [89] model although their learnable parameters are different. In E-step such difference is not important as all parameters in Φ( Y;E,G) are frozen. As existing works about CRF [84, 82] have theoretically proven, following iterative updating function of belief propagation converges. Q u (y u =m) = ˆ Q u (y u =m) Z u = 1 Z u exp{φ θ (m,E u )+ X v∈V X 1≤ m ′ ≤ M ϕ G (m,m ′ ,u,v)Q v (y v =m ′ )} (6.21) where Q u (y u = m) is the probability that account u is assigned into group m and Z u = P 1≤ m≤ M ˆ Q u (y u =m) is the normalizer keeping Q u as a valid distribution. In M-step, given Q we maximize O M but calculating E Y∼ Q logP(Y|E,G) is NP-hard [14, 83]. So we propose to instead optimize the following lower bound (proof in Appx. C.2). O M = logp(S|E)− D KL (Q||P) = logp(S|E)+E Y∼ Q logP(Y|E,G)+const (6.22) 90 Theorem3. Given a fixed inference of Q and a pre-defined ϕ G , we have following inequality, E Y∼ Q logP(Y|E,G)≥ E Y∼ Q X u∈V log exp{φ θ (y u ,E u )} P 1≤ m ′ ≤ M exp{φ θ (m ′ ,E u )} +const = X u∈V X 1≤ m≤ M Q u (y u =m)log exp{φ θ (m,E u )} P 1≤ m ′ ≤ M exp{φ θ (m ′ ,E u )} +const (6.23) Intuitively, the above objective function treats the Q as a group assignment enhanced via label propagation on the prior knowledge based graph and encourages E and φ θ to correct themselves by fitting the enhanced prediction. Compared with pseudo-likelihood [10] which is applied to address similar challenges in recent works [136], the proposed lower bound has a closed-form solution. Thus, we do not really need to sample Y from Q so that the noise is reduced. Also, this lower bound does not contain ϕ G explicitly in the non-constant term. Therefore, we can encourage the model to encode graph information into the embedding. To create a starting point, we initialize E with the embedding layer of a pre-trained neural temporal process model (in this paper we apply AMDN-HAGE) and initialize φ θ via clustering learnt on E (like fitting the φ θ to the prediction of k-Means). The pseudocode of the VigDet training framework over the AMDN-HAGE model is in the Appx. C.3. 6.4 Experiments We verified the effectiveness of the proposed method AMDN-HAGE and training algorithm on real datasets collected from Twitter related to coordinated accounts by Russia’s Internet Research Agency in the U.S. 2016 Election, and for analysis in COVID-19 data. 91 6.4.1 Experimental Set-Up Datasets IRA Dataset As described in Chapter 3, we use the IRA dataset. In 2018, 2752 Twitter accounts were identified by the U.S. Congress 1 as coordinated accounts operated by the Russianagency(referredtoas“trollfarm”)tomanipulatetheU.S.Electionin2016, posting over 1.2M tweets. The social media posts or activities of subset of these accounts were availablethroughpaidTwitterAPIaccessbytheacademiccommunitytostudycoordinated account detection [7, 101]. We obtained the collected dataset [101] which contains 312 of the Russian coordinated accounts (referred to as coordinated “trolls”) with 1713 normal accounts that actively participated in discussion about the U.S. Election during which the coordinated trolls were active (collected based on election related keywords using Twitter API). Accounts in the collected dataset were accounts with at least 20 active and passive tweets. Active tweet is where an account posts (tweet, retweet, reply) and passive is where the account is reacted to (mentioned, retweeted, or replied) [101]. This dataset is used in evaluation of coordination detection the previous state-of-art method proposed in [101]. Activity Traces: Activity traces C s are constructed from the tweets, as any account’s posts and subsequent engagements from others (retweets, replies to the post) forming a time-ordered sequence of activities. The account set (2025 accounts with∼ 5M tweets) and activitytracesareutilizedtotrainAMDN-HAGE,with15%held-outasvalidationcascades. COVID-19PandemicDataset Duetoconcernsarounddisinformationandsocialmedia abusearoundCOVID-19,weutilizethedatacollectedbyus(describedinchapter3)ofsocial mediapostswithkeywordsrelatedtoCOVID-19,usingTwitter’sstreamingAPIfromMarch- July 22, 2021, containing 119,298 active accounts, i.e., ones that have at least 20 active and passive collected tweets, similar to the IRA dataset with their 13.9M tweets. 1 https://www.recode.net/2017/11/2/16598312/russia-twittertrump-twitter-deactivated-handle-list 92 COVID-19 Vaccine Dataset As described in Chapter 3, we collected vaccine related Twitter data between December 2020 and April 2021. The extracted sequences contain 316k activity cascades of 205k accounts, after filtering accounts less than 20 times in the collectedtweets,andcascadesequencesshorterthanlength5. Applyingthemethodresulted in 3 clusters. The method found 2 distinct small group of accounts that are suspicious of coordination, and the large group is the rest of the accounts referenced here as ‘Normal’ i.e., non-coordinated as estimated by the method, where the silhouette score is highest. Unknown Coordination: The COVID-19 pandemic and vaccine data does not have any labeled coordinated groups, unlike the IRA dataset. But it is important to examine if we can uncover any unknown coordinated campaigns for these timely and socially relevant problems. We run AMDN-HAGE on the full account set with their activity traces as in the IRA dataset, and examine tweets from identified coordinated groups, and the overlap of accounts with suspended Twitter accounts (manually suspended by Twitter for violations of platform policies). Note that there can be violating accounts that have not yet been found andsuspendedbyTwitter. Also, accountsthataresuspended, canbeduetovariousreasons (e.g. spam, automation, multiple accounts) and are not restricted to coordinating accounts. Thus we cannot use Twitter suspensions as ground-truth on coordination detection. Baselines and Model Variants Wecompareagainstexistingapproachesthatutilizeaccountactivitiestoidentifycoordinated accounts. Thebaselinesextractfeaturesofcoordinationfromaccountactivitiesandusethem for supervised or unsupervised detection, based on individual or collective behaviors. • Unsupervised Baselines: Co-activity clustering [138] and Clickstream clustering [168] are based on pre-defined activity features. Co-activity models joint activity in content sharing, andClickstreamclusteringmodelspatternsinpost, retweetandreplyactions. The SOTA approach is IRL [101] based on inverse reinforcement learning to extract 93 features from activity traces, for clustering coordinated accounts. • Supervised Baselines: IRL(S) [101] is sota supervised variant of inverse reinforcement approachwhichtrainsasupervisedclassifieronextractedfeaturesfromactivitytraces. WeaddanotherbaselineusingHP(HawkesProcess)[187]tolearnaccountfeaturesunsuper- vised from activity traces. HP(S) is its supervised variant. HP models the influence between accounts with an additive function. This baseline serves as a ablation of the proposed model toshowthatlatentinfluenceandinteractionpatternsforcoordinatedgroupismorecomplex and neural point process can better extract these coordination features. For ablation of different components of proposed model AMDN-HAGE, we also compare with(i)AMDN(withouthidengroupestimation),whichonlylearnstheactivitytracemodel. We can use it to extract account embeddings and cluster with GMM or KMeans to identify the coordinated group as anomalous group. (ii) AMDN-HAGE directly uses the jointly learnt GMM to output the group membership. AMDN-HAGE + Kmeans instead uses account embeddings from AMDN-HAGE with KMeans clustering to find the anomalous coordinated group. (iii) To compare with supervised setting (IRL (S) [101]), we similarly trainaclassifieronextractedfeaturesi.e.,learnedaccountembeddingstodetectcoordinated accounts (assuming subset of labeled coordinated and normal accounts are available for training). The variants are AMDN + NN and AMDN-HAGE + NN which use a two-layer MLP classifier on extracted embeddings from AMDN and AMDN-HAGE respectively. Implementation Details First, we provide the details and implementation specifics of the baselines, as follows. Later, we mention the details of the proposed AMDN-HAGE model and code. • Co-activity clustering [138]. Co-activity features to identify coordinated groups as ac- countsthatrepeatedlysharethesamecontents,wasproposedin[138]Accountfeatures are extracted using SVD on a binary event-participation matrix (of accounts and its 94 posts i.e., tweets, retweets, replies). Clustering on the features is used for detection. • Clickstream clustering [168, 124]. It is proposed by [168] to analyze account behaviors, and is based on hierarchical clustering of accounts with similar activity patterns to identify coordinated groups represented on post, reply and re-share patterns. • IRL (S) [101]. SOTA approach is based on inverse reinforcement learning to discover motives of coordinated accounts from rewards estimated from activity traces. Esti- mated rewards are used as features for detection. IRL(S) trains AdaBoost on these features on a labeled subset of coordinated and normal accounts. [101] reported differ- ent classifiers in their paper, including two-layer MLP and AdaBoost, with the latter outperforming others on their features. For IRL, we use code provided by [101]. • IRL [101]. We also compare an unsupervised variant of the SOTA IRL approach based on clustering with GMM or K-means clustering to detect coordinated groups. • HP and HP (S). In addition, we compare our method to modeling activities using the Hawkes Process (HP) 2 with influence α ij factorized by learnable embeddings of accounts i and j, with both clustering and supervised detection. AMDN-HAGE Details We use activity sequences of maximum length 128, splitting longer sequences, batch size of 256 on 4 NVIDIA-2080Ti, embedding dimension in{32, 64}, number of mixture components for the PDF in the AMDN part between {8,16,32}, single head and single layer attention module. As for the component number in the HAGE part, it is set as 2 for IRA dataset and 3 for COVID dataset based on silhouette scores on learned embeddings from the pre-train step of training algorithm (Fig in Appx. C). For the shared covariance matrix of HAGE part, we constrain it to be a diagonal matrix. The second term of the loss is scaled for embedding dimension and size of user set. We implement the model and training algorithm entirely in PyTorch and use Adam with 1e-3 learning rate and 1e-5 2 Hawkes process code [175] to extract account embeddings from activity traces 95 Table 6.2: Results on detection of Russian coordinated manipulation (IRA dataset) Unsupervised AP AUC F1@TH=0.5 Prec@TH=0.5 Rec@TH=0.5 MaxF1 Co-activity 0.208± 0.01 0.592± 0.03 0.292± 0.02 0.206± 0.02 0.510± 0.04 0.331± 0.03 Clickstream 0.169± 0.02 0.535± 0.04 0.215± 0.06 0.205± 0.05 0.228± 0.08 0.215± 0.06 IRL 0.200± 0.00 0.610± 0.02 0.265± 0.02 0.219± 0.02 0.336± 0.03 0.340± 0.02 HP 0.337± 0.04 0.694± 0.05 0.376± 0.05 0.387± 0.06 0.365± 0.05 0.545± 0.03 AMDN + GMM 0.787± 0.05 0.894± 0.03 0.631± 0.06 0.965± 0.03 0.965± 0.03 0.965± 0.03 0.472± 0.07 0.738± 0.05 AMDN + Kmeans 0.731± 0.08 0.901± 0.02 0.727± 0.06 0.806± 0.07 0.663± 0.06 0.663± 0.06 0.663± 0.06 0.752± 0.05 AMDN-HAGE 0.804± 0.03 0.898± 0.02 0.699± 0.05 0.941± 0.04 0.558± 0.06 0.758± 0.04 AMDN-HAGE + Kmeans 0.818± 0.04 0.818± 0.04 0.818± 0.04 0.935± 0.02 0.935± 0.02 0.935± 0.02 0.731± 0.04 0.731± 0.04 0.731± 0.04 0.913± 0.03 0.611± 0.05 0.776± 0.03 0.776± 0.03 0.776± 0.03 Supervised AP AUC F1@TH=0.5 Prec@TH=0.5 Rec@TH=0.5 MaxF1 IRL (S) 0.672± 0.08 0.896± 0.03 0.557± 0.06 0.781± 0.06 0.781± 0.06 0.781± 0.06 0.436± 0.06 0.633± 0.07 HP (S) 0.760± 0.04 0.925± 0.02 0.753± 0.02 0.743± 0.04 0.769± 0.06 0.782± 0.03 AMDN + NN 0.814± 0.04 0.918± 0.02 0.733± 0.04 0.710± 0.05 0.761± 0.05 0.763± 0.04 AMDN-HAGE + NN 0.838± 0.04 0.838± 0.04 0.838± 0.04 0.926± 0.03 0.926± 0.03 0.926± 0.03 0.769± 0.04 0.769± 0.04 0.769± 0.04 0.752± 0.05 0.789± 0.05 0.789± 0.05 0.789± 0.05 0.799± 0.04 0.799± 0.04 0.799± 0.04 regularization to optimize. We train for max 1000 epochs with early stopping based on validation likelihood of observed activity sequences (75/15/10 splits for tr/va/test sequences are used). The Pytorch code: https://github.com/USC-Melady/AMDN-HAGE-KDD21/. 6.4.2 Results on Coordination Detection Detection Results on IRA Dataset We evaluate on two settings - unsupervised and supervised (as in earlier work [101]). In both, theproposedmodelistrainedasunsupervisedfromactivitytracestoobtainthegroup membership and account embeddings. In the unsupervised setting, the group membership is directly used to report anomalous coordinated group. In supervised setting, the learned embeddings are used as features to train a classifier (to predict coordinated from normal accounts). The classifier is trained on subset of labeled coordinated and normal accounts in IRA dataset, with rest (stratified 20% in 5-folds) held-out for evaluation. Table 6.2 provides results of model evaluation against the baselines averaged in 5-fold stratifiedcross-validationonthelabelednormalandcoordinatedaccountsintheIRAdataset over five random seeds. We compare the Average Precision (AP), area under the ROC curve (AUC), and F1, Precision, Recall, and MacroF1 at 0.5 threshold, and maxF1. AMDN- HAGE outperforms other methods on both unsupervised and supervised settings, due to 96 its ability to capture coordination characteristics with diverse account behaviors by learning latent influence and hidden group behaviors, without pre-specifying features relied on by other baselines. Moreover, the coordination features learned with the proposed method are robust to unsupervised or supervised setting, unlike IRL and IRL(S) [101] (where even though IRL(S) can learn useful features, it performs poorly in unsupervised setting). In comparison, because AMDN-HAGE models the more intrinsic behaviors of coordination, it can extract patterns that can effectively identify anomalous coordinated groups in an unsupervised manner. The margin is larger on unsupervised setting, where group behaviors are more important, since there is no known set of coordinated accounts to train classifiers from extracted features. Ablation of Proposed Model and Training We also compare AMDN-HAGE with its variants to verify the importance of the joint learn- ing and optimization algorithm. To verify the importance of joint learning, in Table 6.2, AMDN-HAGE is compared with AMDN, which only learns the activity trace model, with- out hidden group estimation. Proposed model AMDN-HAGE captures consistently better coordination behaviors, indicating that modeling group behavior jointly is useful over only modeling the latent influence between account pairs through observed activity traces. To demonstrate the effectiveness of the optimization training algorithm, we present the validation loss (negative log-likelihood on held-out 15% of activity traces) in the training process, comparing direct optimization of the joint log-likelihood using Adam (variant of SGD) and our iterative algorithm in Fig. 6.4. As we can see, for the proposed optimization, the loss on the validation set in both the pre-training and joint training stage decline and finally converge. However, in direct optimization with Adam the validation loss decreases to a point but breaks as it reaches an invalid parameter point. Without constraints, Adam reachesacovariancematrixthatisnotpositivedefinite, andaninvalidlog-likelihood(NaN). 97 (a) Loss on validation set with Adam leads to NaN loss function 0 50 100 Training Epochs 6 7 8 Loss on Val. Set Pre-Train Joint Train AMDN loss for Pre-Training Total loss for Joint Training (b) Loss on validation set in training proposed algorithm Figure 6.4: Comparison of iterative optimization and Adam for the joint training objective Table 6.3: Ablation on unsupervised coordination detection (IRA) with VigDet Method AP AUC F1 Prec Rec MaxF1 MacroF1 Co-activity .169± .01 .525± .03 .246± .02 .178± .02 .407± .07 .271± .01 .495± .02 Clickstream .165± .01 .532± .01 .21± .02 .206± .02 .216± .03 .21± .02 .531± .01 IRL .239± .01 .687± .02 .353± .03 .275± .03 .494± .05 .386± .01 .588± .02 HP .298± .03 .567± .03 .442± .03 .421± .02 .466± .04 .46± .03 .667± .01 Amdn-Hage .805± .03 .899± .02 .696± .05 .943± .03 .555± .06 .758± .03 .827± .03 Amdn-Hage(Kmeans) .819± .05 .933± .03 .73± .04 .909± .03 .612± .05 .77± .03 .845± .02 VigDet-PL(TL) .833± .05 .94± .03 .707± .06 .896± .05 .59± .08 .778± .04 .832± .03 VigDet-E(TL) .855± .03 .946± .03 .731± .03 .953± .03 .953± .03 .953± .03 .594± .04 .796± .03 .796± .03 .796± .03 .846± .02 VigDet(TL) .861± .03 .946± .03 .734± .03 .951± .03 .599± .04 .796± .03 .796± .03 .796± .03 .848± .02 VigDet-PL(CF) .845± .04 .95± .02 .719± .05 .914± .04 .596± .07 .793± .03 .839± .03 VigDet-E(CF) .851± .04 .943± .03 .736± .03 .928± .03 .612± .04 .789± .03 .849± .02 VigDet(CF) .872± .03 .872± .03 .872± .03 .95± .03 .95± .03 .95± .03 .752± .03 .752± .03 .752± .03 .917± .04 .639± .04 .639± .04 .639± .04 .793± .03 .857± .02 .857± .02 .857± .02 Ablation of VigDet to Integrate Domain Knowledge InTable.6.3,weshowablationperformanceofVigDetandthebaselines. Earlier,weproposed two encodings of the prior domain knowledge graph. We use both variants in the ablation studyi.e., VigDet(CF)andVigDet(TL),applyingpowerfunctionbasedfilteringwithcubic powerandtemporallogicbasedfiltering. TheproposedmodelAMDN-HAGEandframework VigDet do not use any ground-truth account labels in training. But for VigDet, we hold out 100 randomly sampled accounts as validation set for validation loss, and report metrics on the remaining accounts. Other implementation details are in the Appx. C.3.2. For ablation study of the EM-based variational inference framework and our proposed objective function in M-step, we compare our models with two variants: VigDet-E where we only conduct E-step once to acquire group assignments (inferred distribution over labels) enhanced with prior knowledge, but without alternating updates using the EM loop; and 98 VigDet-PL (PL for Pseudo-Likelihood) where we replace our proposed objective function with pseudo-likelihood function from existing works. VigDet-PL performs not only worse than VigDet, but also worse than VigDet-E. This phenomenon shows that the pseudo- likelihood is noisy for VigDet and verifies the importance of our objective function. 6.4.3 Analysis on Coordination Detection Influence Structure In this section, we examine the latent influence structure and account interactions learned by AMDN-HAGE on the IRA data. The latent influence is captured by the interpretable at- tention weights of the model between account activities in observed traces. Higher attention paid by an event (account activity) on a history event (earlier activity from any account) indicates that the history event has a stronger triggering influence on the future event. In Fig6.5a, we compute the aggregate influence between account pairs learned with AMDN-HAGE over 5 random seeds (as average attention weight from account interactions over all activity traces). The strongest influence ties are between coordinated (“trolls”) (T) accounts, and the least influence is between normal (“non-trolls (NT)”) and coordinated (T) accounts and their activities. In Figure 6.5b, the learned account embeddings of coor- dinated and normal accounts with AMDN-HAGE in the IRA data is visualized. As we see, coordinated accounts form an anomalous cluster distinct from normal accounts. In Fig. 6.6a and 6.6b, we examine how influence weights of account pairs vary with time difference. In the two figures, each point represent an account pair appearing in the same activity sequence. Blue points refer to coordinated (T) pairs and green points refer to normal account (NT) pairs. The influence weights (y-axis) correspond to the time difference betweenactivitiesoftwoaccountsappearinginthesequence(x-axis)isshownforallpointsin Fig. 6.6a to reflect the overall trend. In Fig. 6.6b, we only plot points with highest influence weight in each time difference range of 24 hrs to reflect the trend on strongly interacting pairs in each window. From both figures, we can see that influence weights are higher for 99 39% 35% 13% 11% (a) Avg. influence weights captured by AMDN-HAGE, which is highest for co- ordinated account pairs (b) AMDN-HAGE account embeddings inferred for coordinated “trolls” (red) and normal accounts (green) Figure 6.5: Analysis of learned influence strength between Coordinated (“trolls” T) and Normal (non-trolls NT) in the IRA dataset (a) Overall trend on all account pairs (b) Strongest influence account pairs Figure 6.6: Analysis on how influence weights of account pairs vary with time difference. Green points: Normal (NT) account pairs. Blue: Coordinated (“trolls” T) account pairs shorter time differences between account activities. However, the influence decreases faster for coordinated pairs than normal account pairs with the time between their activities. COVID-19 Dataset As mentioned earlier, the data collected on COVID-19 does not contain a ground-truth set of labeled coordinated accounts. But, we can use the proposed method AMDN-HAGE to uncover any suspicious coordinated accounts with analysis of features. AMDN-HAGE is trained on observed account activities in the COVID-19 data of 119k accounts. The method identifiestwo anomalous clusters ofaccounts (basedon clusteringsilhouette scores, provided in appendix). We inspect the feature distribution in each account group. InFig6.7,wefindmostfrequenthashtagsintweetspostedbyaccountsinthegroups,and plotthetophashtagsuniquetoeachgroup(hashtagsofthesmalleranomalousclusterispro- 100 0 2000 DIRECTO CoronaTimo Genocidio CoronaFarsa CovidHoax NoAlNuevoOrdenMundial NoAlBozal NoALaVacuna NoALaMascarillaObligatoria tableau powerbi uipath excel nlg qlik resultsbi tibco microstrategy bi ai PeriodistasCobardes cpa CoronaPandemic نتصدر_المشهد freetrial AlertaCOVID19SV alexa YoSoyLaResistencia PoliticosAPrision SanitariosAsesinos Jaipur NoMask NoVaccine SanitariosCobardes NoAlConfinamiento Cluster Anomaly (C2) 0 10000 news FANTASTICRADIOUK BELIEVEYOURPOSSIBILITIES STAYHOMESAFELIVES HOMEOFPOSSIBILITIES News pandemia virus Health BreakingNews cdnpoli Pandemia COVID19India COVID19Pandemic StayHomeSaveLives CoronaUpdate NHS Italy TamilNadu AI economy Maharashtra health CoronaLockdown coronavirusindia CoronaInfoCH vaccine DonaldTrump Africa Salud Quarantine YoMeQuedoEnCasa healthcare Brazil EEUU Cluster Others (C0) Figure6.7: Top-35(mostfrequent)uniquehashtagsintweetsofidentifiedcoordinatedgroup and normal accounts videdintheappendix). Wefindthattheprominenttophundredhashtagsinthecoordinated group promote anti-mask and anti-vaccine (“NoMasks”, “NoVaccine”, “NoALaVacuna”), and anti-science theories (“Plandemic”, “Covid-Hoax”), and contain hashtags associated with “QAnon” (“WWG1WGA”), a notorious far-right conspiracy group. In Table 6.5 we use topic modeling to find the most representative disinformation tweets posted by the anomalous accounts. We find 4 topic clusters within tweets that are linked to low-credibility (disinformation) news sources. The table shows tweets closest to the topic clustercenters. ThenarrativeinthehashtagsandtopicssuggestspresenceoftweetsinSpan- ishandEnglish(NoALaMascarillaObligatori,NoALaVacuna,NoAlNuevoOrdenMundia,No- Vaccine) about no new world order, no masks, no vaccine, and QAnon, all opposing Bill Gates, and suggesting that COVID-19 is a hoax and deep state scam to monetize vaccines. In Table 6.4, we compare the distribution of suspended Twitter accounts. Twitter addi- tionally labels some suspended accounts as state-backed i.e, accounts Twitter finds as linked 101 Table 6.4: Overlap between suspended Twitter accounts, and identified coordinated groups/ overall accounts in collected COVID-19 data. (% provided in table) Twitter Overall Cluster 1 Cluster 2 Suspensions Overlap Anomaly (3.7k) Anomaly (5.5k) Suspended (9k) 7.544 12.19 11.94 State-backed (81) 0.067 0.13 0.09 Coupled (602) 0.504 0.72 1.33 Targets (18.5k) 15.507 14.98 17.11 Table 6.5: Representative tweets in disinformation topic clusters in identified COVID-19 coordinated accounts groups Did coronavirus leak from a research lab in Wuhan? Startling new theory is ’nolongerbeingdiscounted’amidclaimsstaff’gotinfectedafterbeingsprayed with blood’ #WWG1WGA #QAnon #MAGA #Trump2020 #COVID19 #scamdemic Since China owns WHO, China must be in on the scam The WHO Lied and Created a Global Panic: Second Extensive Study Finds Coro- navirus Mortality Rate Is 0.4% Not 3.4% - Similar to Seasonal Flu VIRUSFRAUD-FAKENEWSInSpiteofLeftistMediaHysteriaattheTime, SDGovernorNoemConfirmstherewereZERONewCasesor’Outbreaks’over Trump’s Rushmore Event, Trump Supporters are Clean, Healthy People... BREAKING: #BillGates Foundation And The #Covid19 VACCINE NET- WORK SCANDAL The British People Are Going To Get Very Angry Very SoonAndWillWantANSWERS.#GlaxoSmithKline#BillGates#Rothschild #WellcomeTrust #COVID19 #CivilService #NoMasks #BillGatesNegotiated$100Billion#ContactTracingDealWith#Democratic Congressman in 8/2019 well before #Coronavirus #Pandemic starts. #Gates holds pandemic drill 10/2019 Harvard finds #SARSCOV2 started in 8/2019 in #wuhan. Virus in USA by 1/2020. to state-sponsored operations (such as from Russia, etc. that tried to interfere with politics in other countries) [49]. In addition, we consider the “coupled” accounts in the collected data that bidirectionally engage with Twitter’s state-backed accounts, and “targets” as ac- counts mentioned (or targeted) by state-backed accounts in their tweets. Amongst all 119k accounts, the distribution of Twitter accounts (Suspended, State-backed, Coupled) was 1.5- 2 times higher than by random chance in the identified anomalous clusters, even when the numberofsuchaccountstobefoundfromthelargesetofcollectedaccountsissmall. ForTar- gets, the distribution is more uniform, since targeted accounts unlike Coupled only capture a unidirectional engagement from state-backed accounts, as expected because state-backed 102 0 50000 100000 covid19 vaccine astrazeneca coronavirus covidvaccine covid pfizer cdnpoli vaccines covid19vaccine covid_19 auspol vaccination moderna onpoli coronavaccine breaking staysafe corona novaccinepassports eu maskup lockdown vaccin healthcare biontech largestvaccinedrive covid-19 vaccinepassports pandemic washyourhands vaccineswork china smallbusiness india Normal Accounts 0 2000 4000 6000 8000 covid19 vaccine ccpvirus drlimengyan1 astrazeneca covidvaccine unrestrictedbioweapon dryan coronavirus cdnpoli covid takedowntheccp 郭文贵 爆料革命 yanlimeng 闫丽梦 pfizer racialism auspol vaccines onpoli vaccination wuhanvirus 路德社lude covid19vaccine moderna hydroxychloroquine coronavaccine lockdown breaking healthcare vaccinepassports pharmaceutical socialmedia novaccinepassports Supicious Coordinated (CCP) 0 1000 2000 covid19 vaccine astrazeneca covidvaccine coronavirus covid pfizer vaccines auspol cdnpoli moderna onpoli breaking plandemic eu vaccination china corona covid19vaccine greatreset agenda21 covid19ireland who canada id2020 trudeau operationlockstep covidiots trump 7news trudeaufailedcanada fordfailedontario coronavirusde covid_19 covid__19 Supicious Coordinated (Great Reset) Figure 6.8: Top-35 hashtags of normal and identified suspicious coordinated accounts. Unique ones in each group are highlighted in bold accounts attempt to manipulate and mention or engage with other normal accounts. COVID-19 Vaccines Dataset We examine the two identified account groups ( ∼ 8k and 3k accounts) and the remaining ‘Normal’ accounts in terms of tweets features and account behaviors. The tweets from the identified coordinated group contained 5% more misinformation (unreliable/conspiracy URLs in tweets) than over all tweets. Although there were false positives due to the large scale of accounts which makes clustering and learning harder, the groups identified were notably suspicious in terms of the content promoted in their tweets, even though the model has never seen the tweet contents. In Fig. 6.8, we compare the lowercase top-35 hashtags in the tweets of each group (in bold are the non-overlapping hashtags). The coordinated conspiracy group narratives focus on the pandemic being a hoax (‘plandemic’), with one group focusing on the ‘Great Reset’ conspiracy, including ‘Agenda 21’, ‘Operation LockStep‘ which are spin-offs of real-world projects, to falsely suggest malicious intents of world leaders in planning the pandemic 103 Figure 6.9: Tweets from a pair of accounts (A, B) in the detected coordinated group. Left: Tweets from the Twitter profile of accounts A and B suggesting anti-lockdown and anti- government narratives. Right: Three example tweets from the collected dataset, of the same pair of accounts (A, B) suspected of amplifying misinformation by coordinatedly sharing similar agendas towards global economic control. 3 The conspiracy started trending globally after a video of Canadian Prime Minister Justin Trudeau at a UN meeting talking about economic recovery or reset went viral. Tweets from the second coordinated group promote the Bioweapon theory that the virus is a Chinese (CCP) originated Bioweapon. Both coordinated group tweets contain anti-vaccine misinformation. The top hashtags in normal accounts support health interventions (#maskup, #healthcare, #staysafe). Example vaccine misinformation tweet from the coordinated group, “@CNN Remember the Covid vaccine is substantially more dangerous than the virus. Issues range from severe allergic reactions to blindness, stroke and even sudden death! You have been warned! #Plandemic #Agenda21 #ID2020 #Op- erationlockstep #covidvaccine #Coronavirus #Covid19 #Greatreset.” We inspected activities of a sample of the detected coordinated accounts. We randomly sampledaccountpairsthathadretweetedatleastonecommontweetintheobservedcollected 3 https://www.bbc.com/news/55017002 104 0 1 2 3 4 5 Bot score distribution 0.0 0.1 0.2 0.3 0.4 0.5 Frequency Bioweapon (CCP) GreatReset Normal Figure 6.10: Bot score distribution. Mann-Whitney U-Test for suspicious coordinated Bioweapon (CCP) vs. Normal accounts sample (z-score -2.56, p-val 0.00523 < 0.05) and suspicious coordinated (Great Reset) vs Normal accounts sample (z-score -1.35, p-val 0.0869 < 0.1) Normal GreatReset Bio(CCP) INFLUENCER Normal GreatReset Bio(CCP) INFLUENCED 0.0749 (+/-0.224) 0.0212 (+/-0.135) 0.023 (+/-0.137) 0.0359 (+/-0.167) 0.202 (+/-0.326) 0.0 (+/-0.0) 0.0229 (+/-0.13) 0.0026 (+/-0.011) 0.2098 (+/-0.33) 0.00 0.05 0.10 0.15 0.20 Figure 6.11: Mutual triggering effect (influence). Between activities of accounts, estimated from data by AMDN-HAGE shown as Avg. estimated triggering effect from Influencer accounts (whose activities trigger future activities in time). Normal accounts have weaker influence patterns (more random activities) compared to coordinating accounts dataset. For a pair of accounts, we checked their Twitter profile and their tweets in the collected dataset. Fig. 6.9 shows an example account pair (A, B) from the coordinated group, still active on Twitter as of June, 2021. The account names are anonymized here. The tweets of both accounts promoted the same agendas in coordination over similar time periods. In one instance, both retweet different sources that independently posted the same content, seeminglypartofacoordinatednetwork, asshownintheexample@NVICLoeDown and @CaliVaxChoice posted exactly the same content and they re-shared each respectively. Fig6.10examinestheboti.e., automatedaccountscoredistributionofsuspiciouscoordi- nated group accounts. Conspiracies promoted from coordinating account groups (colluding in a hidden, unknown manner) tend to employ both bot (automated) and human actors 105 to push agendas [101, 7]. We evaluate the bot scores using Botometer v4 API [140] on 500 randomlysampledaccountseachfromnormalandcoordinatedgroupstocomparethedistri- bution. We assume the null hypothesis that there is no difference in bot score distributions and use Mann-Whitney U test to compare the distributions. We find statistically significant differences with the normal sample for each coordinated group. As seen in Fig 6.10, for the suspicious Bioweapon (CCP) group the (z-score -2.56, p-val 0.00523) are significant at 0.05. SimilarlyforthesuspiciousGreatResetconspiracy, at0.1significancelevel, suggesting higher distribution of automated behaviours in the detected coordinated account groups. Fig 6.11 examines what the model learns from observed account activities in detecting coordinated groups. We obtain the estimates of the mutual triggering effect (or influ- ence) between accounts from the learned model. The model estimates the density of future activities on the network given past activities, encoding which account pairs trigger each other’s activities. Fig 6.11 shows the average influence weight from accounts in one group (Influencer) on accounts in other groups (Influenced). As we observe, the model picks up stronger influence within accounts of the suspicious coordinating groups. Weaker influence patternswithnormalaccountsindicatemorerandomactivitiesofnormalaccounts(thatmay not be centrally controlled, externally colluding, or jointly collaborating to promote agen- das). Also, the model does not find hidden influence across accounts in the two coordinated groups, suggesting presence of separate efforts, as is also evident from the separate agendas of the two groups (Bioweapon (CCP) vs. Great Reset conspiracy, as seen in Fig 6.8). 6.5 Discussions and Conclusion We proposed AMDN-HAGE to detect coordinated accounts based on their collective behav- iors inferreddirectly fromtheir activitieson social media. Other accountswho are operating individually (not in coordination) would have less organized or more independent and ran- domized activity patterns. Leveraging this, AMDN-HAGE estimates this mutual account influenceandgroupbehaviortouncovercoordinatedgroupsdirectlyfromobservedactivities. 106 It does not observe or utilize any content or tweet features (other than the time-stamped sequence of activities from accounts) and therefore captures coordination behaviors rather than topical interests of accounts. Methods like Louvain community detection [11] have distinct differences that make them less suitable for detecting hidden coordination of collab- oratingaccounts. Communitydetectionwouldcapturedirectinteractionse.g. retweetgraph communities. However organized campaign accounts might operate through a hidden indi- rect influence rather than direct retweets of each other. Either the account group has joint agendasdictatedbyanAgencywithhigher-orderhiddeninteractions,likeintheIRAdataset and target normal accounts in a similar manner [107], or could be self-organized and collud- ing through mechanisms established on black-market or other external platforms. Topical interest based associations from a retweet graph or other pre-defined graphs in community detection result in many false positives compared to the proposed approach [145, 124]. With analysis on Russian Interference and COVID-19 and vaccine datasets, we investi- gated the behaviors of identified coordinated accounts, finding that influence between coor- dinated accounts is higher, decreases faster than non-coordinated pairs over time. Moreover, the proposed method is independent of linguistic, metadata or platform specific features, and hence generalize across platforms and languages or countries from where disinformation campaigns originate. We find that in the COVID-19 dataset, the anomalous group contains Spanish-English tweeting accounts for no-masks and no-vaccine campaign, due to this char- acteristicoftheproposedmodel. Nevertheless,itistrivialtoincorporatechosenfeaturesinto the neural representation encoder modeling the observed activity traces. We also presented aframeworkto incorporateadditionaldomainknowledgein theformof agraphdefined over prior knowledge of coordination patterns, which is flexible and can be used to jointly with data-driven learning to guide the predictions of coordinated accounts. 107 Chapter 7 Network Inference to Limit Disinformation Propagation The propagation of disinformation on social media can result in viral cascades. In order to limit disinformation exposures, it is useful to consider early detection of disinformation contents,aswellasnetworkinterventionstolimitdisinformationspreadbyidentifying,mon- itoring and blocking high transmission paths of disinformation propagation in the network. Towards this end, earlier methods have proposed different network intervention mechanisms forselectionofsubsetofnodesformonitoring[4,180], orfortriggeringtruenewscascadesto counter disinformation exposures [119, 16, 62, 37]. However, these intervention mechanisms assume the diffusion dynamics are known under an assumed model of diffusion. In this Chapter, we focus on the aspect of learning the diffusion dynamics of disinforma- tion and legitimate contents from observed and unlabeled cascades. We first analyze how disinformation spreads and then propose unsupervised inference methods for learning the propagation dynamics for network intervention mechanisms to limit diffusion of disinforma- tion. The primary challenge in learning comes from the fact that collecting disinformation labels for content cascades is expensive and not scalable. Thus, we examine learnability of diffusion dynamics directly from unlabeled content cascades. The veracity of the content de- termines the label of the cascade. In Fig. 7.1, an example of a disinformation and legitimate news cascade on Twitter from the dataset [88] is illustrated in terms of the distribution of engagementswiththecontentovertimeonthenetwork. Thepatternsofpropagationcanbe significantly different for disinformation and legitimate contents based on the user responses each elicits on social media. Therefore, we first study how user behaviors differ with respect to each type of content, and then propose a model of diffusion to capture these behav- iors, followed by unsupervised estimation of the diffusion model parameters from unlabeled cascades. The inferred diffusion model can support disinformation as follows. 108 (a) Legitimate content cascade (b) Disinformation cascade Figure 7.1: Example of diffusion cascades on Twitter. The plot shows the cumulative fre- quency distribution of # tweets for (a) legitimate cascade related to the emergency landing of an airliner in Hudson river in 2009 (b) disinformation cascade related to information sug- gesting that the combination of Coke and Mentos can lead to death, circulated in 2006 • Identifying user influence. The diffusion model can capture the strength of influence between users, that determines how easily information propagates. This allows us to understand the role of different users, how much influence they have in the spread of disinformation and legitimate content, and the network structure governing diffusion. • Predicting cascade dynamics. Learning the diffusion process as a generative model, allows to predict the trajectory of a cascade, its expected size and virality, and other macroscopic properties, useful for prioritizing contents for fact-checking. • Supporting intervention mechanisms. Interventions for mitigation, such as limiting disinformation, or accelerating true news require learning the diffusion parameters to solve for optimal intervention strategies such as ones proposed in [16, 62, 37, 53]. 7.1 Analysis of Diffusion Cascades A few earlier studies identified which features from a set of hand-crafted features were most discriminative in training classifiers for detecting disinformation from legitimate contents [87, 19, 99]. Their findings suggest that fraction of information flow from low to high-degree nodes, multiple periodic spikes, and greater depth to breadth ratio in the diffusion trees of disinformation cascades, are highly predictive features of disinformation propagation. In 109 Table 7.1: Data statistics for Twitter-1 and Twitter-2. Follower graph stats. in Twitter-1 Dataset Twitter-1 [88] Twitter-2 [104] # Users 117,824 233,719 # Engagements 192,350 529,391 # Disinf. Cascades 60 498 # Legit. Cascades 51 494 Avg T length (hr) 8,177 1,983 Avg T interval (hr) 80 65 Avg # engagements 1,733 597 Follower Graph Count # Active users 3K # Edges 27K Avg out-deg 6.54 Max out-deg 126 Avg in-deg 6.55 Max in-deg 137 # Strongly CC 810 # Weakly CC 35 this analysis, we consider both temporal and structural differences in diffusion cascades of disinformation and legitimate contents to investigate whether how user behaviors differ towards each type of content. We answer the following questions for analysis of diffusion cascades, considering two real-world Twitter datasets - 1) Are the diffusion patterns of disinformation cascades significantly different from legitimate content cascades? 2) Are user behaviors with disinformation and legitimate contents non-homogeneous? Real-WorldDatasets WeutilizetwopubliclyavailableTwitterdatasetsdescribedearlier inChapter3, Twitter-1 1 [88]andTwitter-2 2 [104]. Twitter-1wascollectedduring2006-2009 and Twitter-2 from March-Dec 2015. In both datasets, contents are identified as false or le- gitimate from fact-checking website (Snopes.com, urbanlegends.about.com and mainstream news for the former, and Snopes.com for the later). The corresponding engagements on Twitter are obtained by keyword search related to the content. The dataset statistics are summarized in Table 7.1. For inference, we retain users that appear in at least five engage- ments, resulting in 3K and 7K users in the two datasets. The former contains 111 cascades and the later 992 cascades, with Twitter-1 cascades of average time length of 8177 hrs and Twitter-2 of 1983 hrs. The datasets contain cascades in the form of time-stamped sequences ofuserengagements,forexample,cascadeC i = [(u 1 ,t 1 ),(u 2 ,t 2 ),...]whereu j ,t j corresponds to the engagement of user u j at time stamp t j with content corresponding to cascade C i . 1 https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi%3A10.7910%2FDVN%2FBFGAVZ 2 https://www.dropbox.com/s/46r50ctrfa0ur1o/rumdect.zip?dl=0 110 Table 7.2: Hypothesis testing results (p-values) to verify that average time between engage- mentsishigherindisinformationcascades(temporal)andthatratioofconnectedcomponents to total engagements is higher in disinformation cascades (structural) Temporal Structural t-statistic p-value z-score p-value Twitter-1 4.9975 ≤ .00001 1.87577 .03005 Twitter-2 12.760 ≤ .00001 - - For temporal diffusion analysis, we report statistical tests on each dataset based on the observed cascades. For structural diffusion analysis, we consider only Twitter-1, since it additionally provides follower links from which we can construct diffusion structure, similar to [87]. The follower graph represents whether user A follows user B. Diffusion of a content from B to A can occur if A follows B, and B posts before A in that cascade. Therefore, we can construct the diffusion graph of each cascade, from the cascade engagement sequence and the follower graph. In case A has multiple parents, the edge from the latest parent is retained. The follower graph stats for Twitter-1 are in Table 7.1. Studying User Behaviors in Disinformation and Legitimate Cascades First, we perform a two-sample t-test to verify whether the average time delay between engagements or posts is higher in disinformation cascades. The first group of samples S 1 consists of the disinformation cascades in the datasets. The second group S 2 comprises the legitimate contentcascades. Thelog-transformofthedataisnormallydistributed. Thenullhypothesis isthatthereisnosignificantdifferencebetweentheaveragetimedelaybetweenengagements in cascades from the two groups H 0 :µ f =µ t . The alternate hypothesis is the average time delay between engagements is higher for disinformation cascades H 1 : µ f > µ t . The p-value is shown in Table 7.2. The null hypothesis is rejected at significance level α = 0.01 which suggeststhatthereisstatisticallysignificantdifferencebetweenthetemporalcharacteristics. Second, we perform statistical significance test to examine differences in structural char- acteristics of the cascades. We compute the number of connected components (cc) in the diffusion graph of each cascade, constructed as mentioned in the previous subsection. Then 111 −5 0 5 10 Log (Avg Time Delay in Eng.) 0.00 0.05 0.10 0.15 0.20 0.25 PDF Fake True (a) Distribution of avg. time delay afterlog-transformtoreduceskew- ness for t-test in diffusion cascades 0.4 0.6 0.8 1.0 Ratio of Connected Components 0.0 0.2 0.4 0.6 0.8 1.0 Empirical CDF Fake True (b) Empirical CDF of the propor- tion of connected components in diffusion cascades Figure 7.2: Statistical tests distributions (Twitter-1) we define the proportion of connected components in a cascade r = number (cc) number of engagements . The null hypothesis is that there is no significant difference in the proportion of connected com- ponents in the two groups of cascades H 0 : r f = r t . The alternate hypothesis is that it is higher for disinformation cascades H 1 : r f > r t . The data is not normally distributed, so we compute the non-parametric Mann–Whitney U test and report the z-score and p-value in Table 7.2. The null hypothesis is rejected at α = 0.05 which suggests that the proportion of connected components in disinformation cascades is higher. Both statistical tests confirm that diffusion patterns differ based on the type of cascade and the user behaviors towards disinformation and legitimate contents are non-homogeneous. The distribution of average timebetweenengagementsandproportionofconnectedcomponentsisprovidedinFigure7.2 for Twitter-1. The distribution of avg. time between engagements for Twitter-2 cascades is similar to Twitter-1 and thereby omitted. 7.2 Problem Formulation Inthissection,weintroducethediffusionmodelforheterogeneousengagementsanddiffusion phenomenon with disinformation and legitimate contents. Then we discuss the problem formulation for unsupervised diffusion network inference under the diffusion model. 112 7.2.1 Mixture of Independent Cascade Models Given the heterogeneous user engagements with disinformation and legitimate contents, and statistical differences in temporal and structural patterns, we propose to model the heterogeneous behaviors as a mixture of diffusion models. G = (V,E) is the directed graph with n = |V| number of nodes (users/accounts) and m = |E| edges, which are generally unobserved and can represent the latent diffusion structure i.e., influence between accounts. Different mathematical models of diffusion are commonly used in social network analysis [76,55,187]wherethechoiceofmodelaffectstheefficiencyofoptimizationforcomputational problems such as influence maximization. It also affects ease and efficiency of learning algorithms to infer the diffusion structure from observed cascades. Therefore, we extend the Independent Cascade (IC) model [76] commonly used for social networks, to model heterogeneous influence for legitimate and disinformation contents. In the IC model, each edge (u,v)∈ E is associated with a parameter p u,v ∈ [0,1]. A node is considered activated in an content cascade, when its user account has posts or engages with the propagating content. The IC model diffusion starts with seed set (subset of initially activated nodes) and proceeds in discrete timesteps, where each activated node u gets a single independent activation attempt based on p u,v to successfully activate each inactive neighbor v. The influence function σ is a function of the seed set S and σ θ (S) is defined as the expected number of nodes activated at the end of the process. Mixture of Independent Cascade (MIC) Given a social network G = (V,E), we extend the Independent Cascade (IC) model to represent the diffusion of both legitimate and disinformation contents using separate sets of parameters θ T = {p T uv |(u,v) ∈ E} and θ F ={p F uv |(u,v)∈ E}, i.e., both types of contents share the same network skeleton G but with separate parameters for activation probabilities on the edges. The diffusion mixture model is illustrated in Fig. 7.3. We assume π T is the probability with which a legitimate content cascade emerges, and 113 Figure 7.3: Diffusion Mixture Model as Mixture of Independent Cascade (MIC) π F = 1− π T istheprobabilitywithwhichadisinformationcascadeemerges. Letπ = [π T ,π F ] be the mixing weights of the diffusion mixture model, then each cascade c i ∈ C is assumed to be generated independently under MIC as follows: 1. Generated seed set S⊆ V is sampled from some unknown distributionP over V. 2. Generatedcascadecorrespondstocontentbasedontheoutcomeoftherandomvariable h i ∼ Bernoulli (π T ). Cascade labels and mixing weights π are unobserved. 3. Generatedcascadeisdrawnfromthediffusionmixturemodelwith c i ∼ IC(θ T )ifh i = 1 and c i ∼ IC(θ F ) otherwise; diffusion parameters θ T ,θ F are unobserved. 7.2.2 Task Definition Under this mixture MIC model, we study the problem of unsupervised inference of the proposed diffusion mixture model parameters from observed, unlabeled cascades. Network inference refers to the problem of inferring the diffusion process under the mathematical model of propagation, which might entail inferring the edges of the diffusion network, or both the edges and the strength of influence (or weights) on the edges, which in this work is the latter since edges are generally unobserved or capture latent diffusion strengths. First, we formally define the inference problem for the proposed diffusion mixture model MIC.Weassumethattheobservedsetofdiffusioncascades C containsamixtureofunlabeled disinformation and legitimate cascades. We study whether the diffusion processes can be learned directly from C without requiring cascade labels. This makes the inference problem more challenging, but practical when large-scale collection of labeled cascades requiring 114 expert human verification is not scalable. The objective of the network inference problem thereby is to infer θ T ,θ F and π from unlabeled cascades C. 7.3 Network Inference for Diffusion Mixture Model In this section, based on the problem formulation, we examine learnability of the proposed diffusion mixture model, followed by EM algorithm for parameter estimation from real data. 7.3.1 PAC Learnability of Diffusion Mixture ForPAClearnability,weconsiderareductionoftheproblemoflearningthediffusionmixture under the MIC model to the problem of learning a mixture of product distributions over a discrete domain to examine the learnability of the proposed diffusion mixture model (MIC). Each edge e = (u,v) in the diffusion model is associated with the parameter p T e and p F e as stated earlier. Each legitimate content cascade can be alternately represented in terms of a ‘live-edge’ graph, such that each edge e ∈ E is independently declared as live with probabilityp T e andincludedinthegraphorblockedwithprobability1− p T e andnotincluded in the graph. The cascade is then defined by the reachability from seed set S over this graph i.e. a node is activated in the cascade iff there is a directed live edges path from S to the node. Similarly, for disinformation cascades. Therefore, each edge can be represented by a randomvariablex T e andx F e indicatingitslive-edgestatusunderthediffusionmixturemodel, i.e. representingwhethertheedge e isliveorblockedina givendiffusion cascade. Naturally, x T e ∼ Bernoulli (p T e ) and x F e ∼ Bernoulli (p F e ) Let X be a vector of random variables indicating live-edge status of each edge under the mixture diffusion model. According to the generative process of the diffusion mixture model, X is a mixture of k = 2 components X T and X F with mixing weights π . Therefore, X T is a discrete distribution over {0,1} m where m is the number of edges in G and all 115 the x T e are independent. X which is the mixture distribution of X T and X F is simply a mixture of discrete product distributions with mixing weights π . The problem is thus reduced to learning a mixture of discrete product distributions given the live-edge graphs of the observed cascades. Mixture distributions are more generally used in recommendations systems, medicine and other applications [39] and different algorithms can be used to learn the parameters of the mixture distributions, which in our case are p T e ,p F e for all edges in G and mixing weights π by definition. Theorem4. Givenamixtureofunlabeledcascadeswithcompletelyobservedlive-edgegraphs, with diffusion parameters θ M ={p M e |e∈E} with M ∈{T,F} and any ϵ,δ > 0, with mixing weight π M ≥ ϵ mn we can recover in time poly (m 2 n/ϵ )· log(1/δ ), a list of poly (m 2 n/ϵ ) many candidates, at least one of which satisfies the following bound on the influence function σ θ M (S) and its estimate ˆ σ θ M (S) learned from the observed cascades for seed set S drawn from any distribution P over nodes in G, P S∼ P (|ˆ σ θ M (S)− σ θ M (S)|>ϵ )≤ δ with sample complexityO ( n 4 m 8 ϵ 4 ) 3 ln m δ (Proof in Appendix D.1). In practice, the entire live-edge graph cannot be observed. We can only observe the live-edge status of edges outgoing from users that are activated at some time step during the diffusion process, since only the outcomes of activation attempts of activated users on their neighbors can be observed [76]. Empirically, it can still be applied since the calculation of sample estimates from observed cascades (defined in Equation D.1 in the Appendix) can be updated given that we know if the edge is live, blocked or unobserved. We also note that although the derived sample complexity is efficient i.e. polynomial in the size of the graph, for large graphs the degree of the polynomial requires a sizeable number of samples to provide the PAC guarantees on reliable estimation. In the following chapter therefore, we derive a learning algorithm based on EM for parameter estimation that can be used in 116 practice given observed unlabeled cascades. 7.3.2 Parameter Estimation Theestimationofparametersofthediffusionmixturemodel(MIC) θ T ,θ F andmixingweights π from unlabeled cascades that record only the order or timestamps of user activations in the cascades, is obtained by deriving a maximum likelihood based estimation EM algorithm. Notation We use a general notation M ∈{T,F} to denote a component in the mixture model, here the k = 2 component mixture model MIC. θ T ,θ F are the set of edge influence parametersforeachcomponentICmodelinthediffusionmixturemodelwithmixingweights π = [π T ,π F ]. That is for graph G = (V,E), θ T ={p T uv |u,v ∈ E} and θ F ={p F uv |u,v ∈ E}. Weusethenotationstospecifyanobservedsamplecascadebelongingtothesetofcascades C. We define C s (t) as the set of nodes activated at time step t in cascade s and t s (v) as the time of activation of node v in cascade s. Also, we define D s (t) as all activated nodes up to andincludingtimet. Letp M u,v and ˆ p M u,v astheactualandestimatededgeactivationparameter in component M. In addition, we represent γ M s as the posterior probability that cascade s is generated under diffusion component M i.e. γ M s = P(Z s = M;θ ) where Z s indicates the component to which the cascade s belongs and θ is the complete set of parameters θ T ,θ F , and π . Applying Bayes’ rule, γ M s =P(Z s =M;θ ) = π M P(s; θ M ) P k i=1 π i P(s; θ i ) (7.1) Let Pa(v) represent parents of v in G that is, u ∈ Pa(v) if and only if (u,v) ∈ E. Similarly, Ch(v) is the children of v. Let p M s (v) be the probability with which v is activated in cascade s under diffusion component M. By the definition of the IC model, v is activated at time step t s (v) in cascade s iff at least one activation attempt of an active parent of v in 117 s is successful. Therefore, p M s (v) = 1− Y u∈Pa(v)∩Cs(ts(v)− 1) (1− p M u,v ) (7.2) In addition, let A u,v ⊆ C be the subset of cascades in which both u and v are activated and t s (v) = t s (u)+1 and B u,v be the subset of cascades in which u is activated at some time t and v is not activated up to and including time t+1. Derivation and Algorithm We derive an expectation-maximization (EM) based max- imum likelihood estimation (MLE) procedure. The joint log probability of cascades and unobserved cascade labels under the mixture model is, logP(C,Z;θ ) = X s∈C X M∈k 1 {Zs=M} log(π M P(s;θ M )) Our goal is to maximize the expected joint log probability, Q =E logP(C,Z;θ ) = X s∈C X M∈k γ M s logπ M + X s∈C X M∈k γ M s logP(s;θ M ) The maximization of Q with respect to π subject to constraints P M π M = 1,π M ≥ 0, we get, π M = 1 |C| P s γ M s from the first term of Q containing π M . Now to update the estimates for edge probabilities p M u,v , we need to maximize the second term of Q by differentiating Q with respect to p M u,v . Let ˆ p M u,v be the current estimates of edge influence parameters of edge (u,v) for component M. As stated in the notations, C s (t) is the set of activated nodes at times step t in cascade s and t s (v) is the time of activation of node v in cascade s. p M s (v) is the probability with which v is activated in cascade s under diffusion model M. The second term of Q involves the product terms of Equation 7.2 which cannot be solved analytically. However, based on the definition of the IC model, it is possible to approximate Q based 118 on the current estimates of the parameters [57, 139]. We utilize the linear approximation chosenin[139]. PrimarilyQcanbedecomposedintermsofnodesactivatedinacascadeand nodes not activated in a cascade. For the second case of inactive nodes, we will not need any approximation as the likelihood involves log(1− p M s (v)) which eliminates the product form of Equation 7.2. For the first case of active nodes, the form is complex because we do not know which active parent was responsible in activating a given node v. This is because, by the definition of IC, activation attempts of all parents of v activated at a given time step are arbitrarily sequenced. Therefore, following [139], we can instead approximate p M s (v) for this case in terms of ˆ p M u,v ˆ p M s (v) for every active parent u since - the probability that v was activated by u should be proportional to the current estimate ˆ p M u,v of the strength of influence of u on v. Therefore, the second term of Q is as follows, where X = C s (t + 1)∩ Ch(u) and X ′ =Ch(u)\D s (t+1), X s∈C X M∈k γ M s T− 1 X t=0 X u∈Cs(t) h X v∈X h ˆ p M u,v ˆ p M s (v) logp M u,v + 1− ˆ p M u,v ˆ p M s (v) ! log(1− p M u,v ) i + X v ′ ∈X ′ log(1− p M u,v ′) i Differentiating the above with respect to p M u,v and setting it to zero, and considering Pa(v) represents parents of v in base graph G, A u,v is the subset of samples in which both u and v are activated and t s (v) =t s (u)+1 and B u,v is the subset of samples in which u is activated at some time t and v is not activated up to and including time t+1 we get, p M u,v = 1 P s∈Au,v γ M s + P s∈Bu,v γ M s X s∈Au,v γ M s ˆ p M u,v p M s (v) This completes the derivation of the EM algorithm shown in Alg. 4. Relaxation Sinceobservedcascadesonlycontaintheorderofactivationsortimestampsat whichusersareactivated, ratherthandiscretetimesteps, andtheedgesin Gareunobserved; 119 Algorithm 4 MIC: Diffusion Mixture Parameter Estimation Require: observed, unlabeled cascades C Ensure: estimate ˆ θ M ,ˆ π M ,γ M s ; ∀s∈C and M ∈{T,F} 1: ˆ π T , ˆ θ T , ˆ θ F ← init∈ [0,1]; ˆ π F ← 1− ˆ π T . 2: while not converged do 3: // E-Step 4: γ M s ← ˆ π M P(s; ˆ θ M ) P k i=1 ˆ π i P(s; ˆ θ i ) 5: p M s (v)← 1− Q u∈Pa(v) ∩ Cs(ts(v)− 1) (1− ˆ p M u,v ) 6: // M-step 7: ˆ π M ← 1 |C| P s∈C γ M s ; 8: ˆ p M u,v ← 1 P s∈Au,v γ M s + P s∈Bu,v γ M s P s∈Au,v γ M s ˆ p M u,v p M s (v) 9: end while we relax Equation 7.2 to deal with continuous time and let p M s (v), the probability that v is active in s under component M equal 1− Q u∈Cs(ts(v)− W≤ τ<t s(v))) (1− p M u,v ) where W is a lookback window and hyperparameter of the algorithm. Thus, any u activated in W before t v (s) is considered a potential parent and influencer (that can activate) v. W can be set in unit of time or in terms of number of past events. 7.4 Experiments We evaluate the EM estimation on datasets mentioned in Sec. 7.1. Diffusion parameters are latent, but since the ground-truth labels of cascades are present in the dataset, we evaluate inferred parameters on cascade separability and model fit based on likelihood of observed cascades, and inferred influential users and high transmission paths for interventions. 7.4.1 Experimental Set-Up Baselines Unsupervised methods for separating disinformation from legitimate cascades. • TruthFinder (TF) [177] is a credibility propagation algorithm that exploits conflicting sentiments between user comments to the same content. • StanceEval(SE)[134,186]exploitstheaveragesentimentofuserresponsesincascades, 120 since disinformation tends to elicit negative and questioning engagements. • K-Means (KM) clustering [103, 87, 19] based on temporal and propagation features identifiedinpriorworksi.e.,numberofengagements,timedurationofcascade,average time gap between engagements, and fraction of most active users in the cascade. • SEIZ (SZ) [71] is a rumor model proposed for unsupervised rumor detection. It par- titions users as either “susceptible”, “infected”, “exposed” or “skeptic” with regards to the content and models state transitions between them. The model is fit to each cascade separately by solving differential equations. They define a ratio based on the learned parameters of the rumor model for each cascade to classify it as rumor. • Homogeneous IC (HIC) where we assume a single parameter value (f) for the disin- formation component edges p F uv =f,∀(u,v) and similarly (t) for the other component. MIC Appx. D.2 contains implementation details and runtime and convergence analysis of the EM algorithm for parameter estimation of the proposed MIC diffusion mixture model. 7.4.2 Results and Analysis Clustering Cascades and Fitting Results From the inferred parameters, we can determine if an observed cascade is more likely to be considered for disinformation or legitimate content based on the posterior probability of the cascade under each component of the mixture MIC. The predicted component for each cascadeisthusobtainedasargmax M γ M s . Thiswillresultintwoseparateclustersofunlabeled cascades. Each cluster of cascades is then assigned its label as disinformation/legitimate based on a held out one-fifth set of cascades with known labels from the dataset. Table 7.3 reports the clustering result evaluated on the ground-truth labels of the cascades in the dataset for baselines and MIC. 121 Table7.3: Resultsonclusteringcascadesorseparabilityofdisinformationfromunsupervised parameter estimation of the diffusion mixture model Twitter-1 Twitter-2 F1-Score Accuracy F1-Score Accuracy TF 0.576 0.522 0.573 0.536 SE 0.535 0.531 0.388 0.469 KM 0.253 0.522 0.312 0.490 SZ 0.54± 0.03 0.52± 0.03 0.56± 0.03 0.57± 0.01 HIC 0.48± 0.16 0.55± 0.01 0.49± 0.12 0.53± 0.02 MIC 0.67± 0.02 0.61± 0.02 0.63± 0.01 0.59± 0.01 Table 7.4: Results of IC vs. MIC modeling heterogeneous user behaviors in terms of Avg. NLL on train and held-out validation cascades. Lower NLL indicates better fit Twitter-1 Twitter-2 NLL (Tr) NLL (Va) NLL (Tr) NLL (Va) IC 58.45 3757.84 1052.97 1414.25 MIC 6.29 9.035 8.35 8.45 Result In terms of the distribution π estimated in MIC, we report the Mean Absolute Error (MAE) between the estimated value and the true data distribution. The data is near balanced, and the estimated π = [0.44,0.56] in Twitter-1 is close to the true distribution, with Mean Absolute Error (MAE) of 0.04. Twitter-2 estimated π = [0.47,0.53] with MAE of 0.058. Therefore, MIC outputs balanced clusters of cascade types. KMeans (KM) on the other hand produces unequal sized, biased clusters, resulting in close to random accuracy predicting most cascades to one type, with low F1. We find that KM was biased towards producing a single cluster without being able to effectively separate them, HIC does not model heterogeneous influence across user pairs which limits expressivity of the model, and SZ cannot capture common patterns across cascades, as it fits separate parameters per cascade. In comparison with MIC exploit differences in inferred dynamics, TF and SE utilize aggregate sentiments of user responses which are relatively noisy signals of veracity. The sentiment analysis methods like StanceEval (SE) make mistakes in cases where true content evokes negative sentiments such as “Is horrified to read about the missing Air France plane” and also due to sentiment lexicons that map certain words like “missing” to 122 Inf (T) Inf (F) 0 20 40 60 80 100 % Appearance (Fake Cascades) (a) Twitter-1 Inf (T) Inf (F) 0 20 40 60 80 100 % Appearance (Fake Cascades) (b) Twitter-2 Figure 7.4: Results on quality of influential users selected based on the estimated diffusion parametersusinggreedymaximizationofeachcomponentIC.(a)Twitter-1and(b)Twitter- 2. Inf(T) and Inf(F) are inferred influential users for legitimate and disinformation cascades negative, such as in “Air France jet missing with 228 people over Atlantic after running into thunderstorms”. This results in negatively correlated predictions below 50% depending on the sentiment patterns and content in the data. TruthFinder (TF) also utilizes user sentimentsbutismorerobustasitaccountsforconflictingrelationshipsbetweenusers. SEIZ (SZ) is the better baseline based on rumor modeling. But it does depend on an estimated threshold for the ratio per cascade used to determine if the cascade is a disinformation cascade. SEIZ uses median ratio over the set of cascades as the threshold, and any observed ratio above this threshold is considered as disinformation. It can result in lower quality estimates of the threshold with fewer cascades. We additionally compare the IC model with MIC. IC model does not have the proposed parameterization for different cascade types, and hence cannot be compared in clustering. Therefore, we report the avg. Negative Log-likelihood (NLL) per cascade instead, after parameter estimation using IC and MIC in the datasets. Lower NLL indicates better fit to the observed cascades. Average NLL per cascade on a 20% held-out set of cascades (validation set) and on the remaining training set is reported in Table 7.4. Lower Avg. NLL in both Va and Tr cascades in both datasets reported in the Table finds that MIC which is representing heterogeneous user behaviors allows for better diffusion modeling than IC. 123 Table 7.5: Characteristics of identified influential users for both types of cascades Twitter-1 Followers Following # Posts Screen Name Description Tags Legitimate content cascades 54418 1157 24182 HuffPost real life is news, and news is per- sonal. Read more. 17874 0 675 TMZ breaking biggest stories in enter- tainment news 2684 2941 2144 PSPGuru Sending you constant news about the latest PSP news. 1118 142 3191 FOX10News TV news station, serving the Al- abama, Florida, Gulf 22252 23853 4621 OnlyMobileNews We follow the latest in mobile technology news Disinformation content cascades 672 280 8200 unk. F: Swine flu from eating pork, and related to zombies 514 470 3408 Terrypooch Fighting for liberty, justice for all 1 0 32 08kx250f F: xbox720 will launch before 2012 3273 1926 2294 unk. F: Obama not a natural born cit- izen 8829 8362 6375 BuzzFeed F: BigFoot myth, giant man- eating catfish, Montauk monster Influential Users Top-100 influential users for each component IC model with inferred parameters ˆ θ T , ˆ θ F are selectedusinggreedymaximizationalgorithm[56]. Influentialusersareaccountsthatasseed sets would trigger the largest cascades in the propagation of disinformation and legitimate contents respectively, selected using the inferred diffusion model components in MIC. In Fig 7.4, we report the box-plot for selected influential users as per % relative appear- ance in disinformation vs. legitimate cascades. Inferred users identified for disinformation cascades (Inf(F)) have higher positive correlation with relative appearance in disinformation cascades, as seen from the figure, for both datasets. This confirms the quality of the esti- mated parameters in MIC. Also, relative appearance of a uniform random sample of users in disinformation vs. legitimate cascades suggests a higher degree of separation in Twitter-1 with more accounts engaging solely with one type of cascade. 124 (a) Cascade distance (b) Followers count (c) Cascade size Figure 7.5: General characteristics of identified influential users in Twitter-1 Characteristics Table 7.5 lists the features of identified influential users with reported number of followers, and posts from 2009 Twitter-1 snapshot. Inferred influential users identified for legitimate contents, as seen, correspond largely to accounts of known credible news and opinion websites and blogs. In terms of topic distribution, the dominant types of influential users include accounts disseminating news related to politics, entertainment, infotainment, technology updates, and tend to have large number of direct followers. Influential users identified for disinformation cascades include accounts with relatively fewer counts of direct followers. For some of these the screen name and description is unavailable from Twitter API (reported as ‘unk.’ in the table). Several of these do not have a listed description along with their screen name, unlike in the previous case of influential legitimatecascadeusers. Therefore,welistthecontentofdisinformationcascadeswithwhich they engaged. The influential accounts engage in diverse range of disinformation similarly dominatedbypolitics,technology,entertainment,andnews,ortrendingtopicssuchasSwine Flu and current events. They also appear among the larger and more viral disinformation cascades in the dataset. Buzzfeed interestingly has been historically linked to unreliable journalism, especially before 2014 and appears in connection with viral myths related to man-eating catfish and BigFoot. Fig. 7.5 plots the distribution of account characteristics of identified influential users in terms of cascade distance i.e., the number of engagements from the first post in the cascade sequence, count of direct followers, and cascade size i.e., the number of engagements 125 5 15 25 35 # Users 0 20 40 60 80 100 % Reduction in Fake Cascade Size MIC TopU SZ KM SE TF TopFol (a) Twitter-1 5 15 25 35 # Users 0 20 40 60 80 100 % Reduction in Fake Cascade Size MIC TopU SZ KM SE TF TopFol (b) Twitter-2 50 100 250 500 K (# edges removed) 0 20 40 60 80 100 % Reduction in Fake Cascade Size MIC Random (c) Twitter-1 50 100 250 500 K (# edges removed) 0 20 40 60 80 100 % Reduction in Fake Cascade Size MIC Random (d) Twitter-2 Figure 7.6: Intervention analysis. (a, b) Node interventions (c, d) Edge interventions in the cascades in which the influential/all users participated. The account characteristics highlight that disinformation influential users are early spreaders and have lower count of direct followers compared to legitimate content influential users. Both types of influential usersappearinmoreviralcascades,thatislargercascadescontainingmoreuserengagements. Network Interventions In Fig 7.6 we investigate different intervention mechanisms to monitor or intercept the prop- agationpathsof disinformationcascades, leveragingtheinferred diffusiondynamics, soas to its limitits diffusion onthe network. High transmissionpaths as influential nodes (identified by influence maximization on the component IC) and edges (with highest edge activation probability inferred under MIC) for the disinformation component for interventions. Node Interventions We determine which subset of nodes can be monitored in order to block disinformation content diffusion. The selected top-100 disinformation influential users examined earlier are chosen candidates for node intervention under MIC, ranked by its influence. First, we consider that K users are selected for intervention/monitoring. If a disinformationcascadereachesanyofthemonitoredusers,itcanbeinterceptedandremoved from the network, thereby limiting its future spread. Here, we evaluate if the selected nodes are indeed high transmission paths in the disinformation diffusion cascades. We measure the % reduction in disinformation cascade engagements, if the disinformation cascade is cut-off 126 at the point when one of the selected K users is encountered. We evaluate MIC against the previously considered baselines and include two additional baselines TopU andTopFol thatinterceptusersrankedbytheirtotalengagementcountinthesetofobservedcascades/ total followers count from the user profile information. For the other baselines, the selection ofK usersisasfollows: rankusersbytheirtotalengagementcountinthecascadespredicted as disinformation cascades by the baseline method. The results are in Fig 7.6a and 7.6b. MIC identifies nodes that are more influential to the spread, over other heuristics, such as number of times a user engages with disinformation or predicted disinformation cascades. Edge Interventions In edge intervention, we select K edges in the network in order to intercept the propagation of disinformation cascades. The edges are ranked by the weight (strength of influence) p F e under the inferred disinformation component of MIC. These are theidentifiedhightransmissionpathsfordisinformationcascadesandthusremoved/blocked. We again compare the percentage reduction in disinformation cascade size due to edge removal with MIC, against a Random strategy that selects edges uniformly at random from the network, as shown in Fig 7.6c and 7.6d. Here the reduction is calculated over the size of disinformation cascades simulated over 1000 rounds under the disinformation component with and without the K edges removed/intercepted. The simulations are triggered from seeds sampled from users at the head of the sequence of observed disinformation cascades. 7.5 Discussions and Conclusion In this Chapter, we proposed a mixture of independent cascade models (MIC) to express and infer the diffusion dynamics of disinformation and legitimate contents. With statistical analysisonrealdatasets,weconfirmednotabledifferencesinuserbehaviorstowardsfakeand truecontentsintemporalandstructuralaspectsofdiffusion,thatcanbeexpressedwithMIC. Based on that, we derived an unsupervised inference method for parameter estimation from observed unlabeled cascades, and conducted experiments on Twitter datasets with ground- 127 truth labeled cascades. The experiments revealed interesting analysis of the characteristics ofusersidentifiedasinfluentialincontentpropagationundertheinferreddiffusiondynamics, and their effectiveness in identifying high transmission diffusion paths over heuristics. To conclude, we discuss few limitations and extensions. We assumed two sets of parame- tersθ T ,θ F todifferentiatedisinformationandlegitimatecontentcascades, basedonverifying that (i) differences in diffusion patterns of the two types are statistically significant in the datasets, and (ii) the datasets are built from collections of events reported during a specific period with samples across types collected from the same data source and no known col- lection biases across types. In order to account for multiple types (such as satire, or topics andtypescorrelatedwithcredibility), themixturemodel easily generalizestomultipletypes of cascades, when k > 2 components are initialized in Algorithm 1, wherein the provided derivation is already for the general case k. Since our datasets model label only two types of cascades, without more fine-grained distinctions, we restricted our experiments to k = 2. Recentadvancesinneuralnetworkstorepresentdiffusionmodels,suchastheEmbedded- IC model [13] can improve parameter sharing, generalizability and scalability to larger net- works, and the inference uses gradient-based optimization of the log-likelihood of observed cascades. The runtime analysis (D.2) shows that it scales in the order of O(k|C|V 2 ) which is reduced toO(k|C|VW) by setting a constant window W smaller than V, where W is the window size described in Relaxation section under Parameter Estimation, V is the number of users, k is the number of components, and C is the set of cascades. This is a limitation of applying the algorithm to large-scale graphs. In future work, we can integrate dimensional- ity reduction techniques to reduce the number of unique user representations through user graph coarsening [96] or using neural representation in diffusion modeling [17, 13]. 128 Chapter 8 Characterization of Engagement and Interventions Disinformation threatens to influence individual opinions and social dynamics. In previous chapters, we proposed different techniques for disinformation mitigation based on detection, analysis of how it spreads, coordinated efforts and network interventions. Apart from miti- gation, in social media analysis it is important to study and characterize engagement with disinformation to understand the scope of the problem, and the risks and impact of disinfor- mation on public perception. Disinformation promoters in the past have persistently target social media discourse related to politics, Elections, social or global phenomenon such as the COVID-19 pandemic, natural disasters, and other real-world events. In this Chapter, we characterize the engagement with disinformation and conspiracy groups from large-scale datasets collected from Twitter for discourse pertaining to recent real-world events. In this thesis we investigate social media engagement in the U.S. 2020 Election and COVID-19 pandemic and vaccines. Due to the critical nature of these socio- politicalevents,Twitterandotherplatformstookmeasurestocounteractdisinformationand conspiracygroups. OnesuchmeasureannouncedbyTwitterinJuly,2020wastobanthefar- right QAnon conspiracy group and it restricted activities of several accounts in connection withthat. Inourwork,weprovideoneofthefirstcharacterizationsofengagementandeffects of interventions on conspiracy group activities with observational causal analysis from the dataset collected on the U.S. 2020 Election. The analysis is critical to ensuring integrity of elections and democratic systems from attempts to manipulate public opinion. 129 8.1 Research Questions In the context of the 2020 U.S. 2020 presidential election, held on November 3, 2020, since Twitter is recognized as one of the social media platforms with the most news-focused users, with significant following in U.S. politics [65], we investigate and characterize disinformation on Twitter, with an analysis of more than 242 million election-related tweets, collected between June 20, 2020 and September 6, 2020 [20]. We focus on the following research questions to characterize online engagement with disinformation and conspiracies. • R1. What are the prevalent disinformation and conspiracy narratives on Twitter pre- ceding the U.S. 2020 Election? We apply and train a detection model to construct disinformation labels for factual and unreliable (or conspiratorial) claims to examine how the political discourse was manipulated based on topic modeling. • R2. How significant is the impact and reach of disinformation and conspiracy groups in terms of account engagements (characterized by activity level, political leaning, tweet types, and propagation dynamics)? We characterize account engagements with the QAnon conspiracy group, and disinformation tweets, and compare propagation dy- namics of unreliable/conspiracy and reliable cascades. • R3. Did Twitter’s restrictions on conspiracy groups influence its activities and were they effective in limiting the conspiracy? We investigate factors driving the sustained activityandengagementofQAnonaccountsafterTwitter’srestrictions,andinvestigate its observational causal impact using a regression discontinuity design [90]. 8.2 Data and Methodology Data Collection and Disinformation Detection For the analysis, we start from the dataset collected in [20] which tracks election related tweets from May, 2019 onwards, and contains over approximately one billion tweets. 1 The dataset was collected by tracking men- 1 Dataset: https://github.com/echen102/us-pres-elections-2020 130 tions of official and personal accounts of Republican and Democratic candidates in the pres- idential election using Twitter’s streaming API service. The details of the tracked mentions and distribution of frequent hashtags and bigrams in the data are available in [20]. For this analysis,wefocusonthetweetsthatappearedbetweenJune20,2020andSeptember6,2020, in order to study the disinformation surfaced on Twitter in the months preceding the elec- tion. This subset of the data contains 242,087,331 election-related tweets from 10,392,492 unique users. The four types of tweets as described earlier are (i) Original tweets (accounts cancreatecontentandpostonTwitter)(ii)replytweets(iii)retweetedtweets,whichreshare without comment (iv) quote tweets (embed a tweet i.e., reshare with comment). As described in Chapter 3 (Section 3.4), we label disinformation tweets in the dataset basedonunreliable(orconspiracy)sources,totrainanensembleofCSIdetectionmodel[138] which can then be applied to infer labels for unlabeled tweets. The labeled training cascades consistedof3,162unreliableand4,320reliable, accountingforatleast75%ofalltweetsafter subsampling most active users and cascades by minimum length of 5 as described earlier. The unlabeled cascade set included 192,103 such cascades. Using the CSI ensemble, which has a AUC 0.81 and F1 0.76 on 5-fold cross-validation, the predictions with largest margins from the detection threshold, result in 72,228 unreliable and 81,453 reliable cascades were obtained for analysis. The engagements tweets forming reliable cascades constitute about 2/3 of the total tweets in both types of cascades. Human verification of the labeled cascades from the ensemble model is discussed earlier, with supplementary details in the A.2.1. 8.2.1 Inferring Political Leaning For analysis, we also need to infer political leaning of accounts in the collected dataset. We describe the methodology for political leaning inference in this section. We use the list of 29 prominent news outlets classified as left, lean left, center, lean right, right as per ratings provided by allsides.com 2 [45]. We consider left and lean left classified outlets as the 2 https://www.allsides.com/media-bias 131 left-leaning, and right and lean right as right-leaning. ModelSpecifications ForaccountsthattweetedorretweetedURLspublishedfromthese news outlets, we characterize their political leaning by measuring the average bias of the media outlets they endorsed. This gives us a set of labeled accounts with known political leaning labels based on the media URLs. To infer labels of other accounts, we can propagate labelsfromthisknownset(alsocalledtheseedseti.e.,accountswithknownpoliticalleaning labels based on the media URLs) to other accounts based on interactions between accounts. Weutilizetheretweetnetworktoinferpoliticalleaningofotheraccountsstartingfromthe seedsetaccounts. Retweetingisaformofendorsement,differentfromothertweettypes,and accountswhichretweeteachothertendtosharethesamepoliticalbiases[7]. WeuseLouvain method [11] to identify communities in the retweet graph, where edge weights represent the number of retweets between accounts. It optimizes modularity which provides a measure of edge density within communities compared to across communities. We assign political leaning to each identified community, using the average leaning of media URLs endorsed by accounts in the seed set that belong to the community. The seed set accounts with high entropy in distribution of left and right-leaning URLs (close to uniform with margin of 0.2) and ones that shared less than 10 URLs from the media outlets are filtered out. Inference of Political Leanings Using media outlets, we obtain a seed set of 114K ac- counts from 10.4M accounts in the dataset. To limit the size of the retweet network, we consider the top active 1.2M accounts that appeared (in original tweet, retweet, quote, or reply tweet) at least 20 times in the collected dataset. Using the seed set and retweet graph of 1.2M accounts, gives a resulting inferred network with large left-leaning communities of 540,719 and 68,197 accounts and two smaller left-leaning ones, and large right-leaning com- munities of 480,982 and 10,723 accounts, and multiple smaller ones. Of the 1.2M accounts, we were able to infer the political leaning of 92% of the accounts. The rest of the accounts remainundeterminedduetohighentropyinleftandright-leaningURLsshared,orwithcom- 132 munities that had fewer than two seed accounts. We thereby identified 610,430 left-leaning and 500,804 right-leaning in the 1.2M accounts. For validation, we measure the accuracy of inferred leanings based on three types of evaluations (i) Media URL labels i.e., based on the averaged political leaning of left/right leaning media outlets endorsed in tweets from the account (However since media URL labels are also used as seed set labels during inference, therefore for evaluation we report averaged 5-fold results wherein 20% of the seed labels are held-out and kept unseen during inference). (ii) Profile descriptions, i.e., based on whether the account profile description dominantly included left or right leaning hashtags (the hashtags were classified as left/right through human validation of most frequently used 3,000 hashtags, provided in Appx B). (iii) Manual verification based on inspection of tweets of randomly sampled subset of accounts (i.e., based on explicitly stated party affiliation in tweet or account profile, or expressed support of left/right presidential candidate/party, and assigned ‘center’ instead of left/right if non- partisan). Wesampled100inferredaccountsuniformlyatrandom,and124inferredaccounts bystratifiedsamplingbasedondegreedistributionofaccountsintheretweetgraph,toensure coverage of dense and sparsely connected accounts in the retweet graph. In Table 8.1, we report the error rate on each evaluation measure separately for the left- leaning and right-leaning accounts. Here, we added an alternative baseline based on Label propagation [7] for comparison. The total error rate was 4.46% on manually verified labels for both methods (label propagation (LP) and Louvain (Lo) used here, on RT graph from same media URL labeled seed set) and results were robust on both. Error analysis on the manually labeled set, suggests that errors included few accounts that are actually neither left/rightleaning(butcenter),e.g.,U.S.DepartmentofState,reporteraccounts,orunrelated or disinterested in U.S. politics, and were erroneously classified as left/right leaning. Other mistakes included anti-Trump conservatives that were inferred as left-leaning. 133 Table 8.1: Number of accounts labeled as left or right-leaning (by media URLs, account profile description, and human verification) for validation, with error rate (%) in each type based on the inferred political leaning of those accounts MediaURLs Profile Desc. Human Verif. LP-Left 68k (0.71) 29.5k (0.32) 116 (1.72) LP-Right 46k (0.27) 14.0k (0.34) 103 (2.91) Lo-Left 68k (0.67) 29.5k (0.25) 116 (1.72) Lo-Right 46k (0.37) 14.0k (0.58) 103 (2.91) mail, ballots, voting, fraud, voter, vote, ballot, election, absentee, democrats, person, dems, votes news, watch, twitter, media, video, fake, conference, fox, cnn, people, live, breaking, press, see, tweets obama, barack, clinton, hillary, obamagate, administration, campaign, spying, hussein, years, corrupt democrats, party, democrat, democratic, vote, republican, republicans, election, socialist, communist, country, radical, political, liberal, left god, bless, loveoneanother, bebest, good, patriots, morning, lord, pray, please, thank, participate, retweet, thanks, Q Watch Missile Attack on 4th Sec @realDonaldTrump Was Right covid, virus, vaccine, coronavirus, als, people, hydroxychloroquine, deaths, fda, treatment, hcq, china, patients, fauci q, maga, trust, tammy, biden, plan, bbb, rd, love, must, great, bqqm venezuela, por, de, para, urgente, militar, los, en, necesita, libertad, intervencion, socialistas jobs, tax, economy, us, china, american, people, federal, law, mining, taxes, order, need, would, state police, blm, law, antifa, violence, federal, riots, people, cities, democrats, terrorists, violent, crime, terrorist, stop Figure 8.1: Topic clusters for unreliable/conspiracy tweets with top representative tweets 8.3 Results and Analysis 8.3.1 Disinformation Topic Modeling Topic Modeling We use topic modeling to identify prominent topics that were targets of disinformation prior to the election. With identified unreliable or conspiracy cascades, we can model the topics in tweet text associated with the source (first) tweet in the cascade. The text is pre-processed by tokenization, punctuation removal, stop-word removal, and removal of URLs, hashtags, mentions, and special characters, and represented using pre- 134 trained fastText word embeddings [12] 3 . We take the average of word embeddings in the tweet text to represent each tweet. Using pre-trained embeddings trained on large English corpora, we can potentially encode more semantic information and it is useful for short texts where word co-occurrence statistics are limited for utilizing traditional probabilistic topic models [94]. The tweet text representations are clustered using k-means to identify topics. Weselectnumberofclusters(K)usingsilhouetteandDavies-Bouldinmeasuresofcluster separability. Kthatjointlyisinthebestscoresofbothisselectedbetween3-35. Thisgaveus K=30, andinspectingworddistributionandrepresentativetweets(closesttoclustercenter), we discard two clusters unrelated to US politics (ALS treatment and Nigeria violence), and eight small or less distinguished clusters, and merge over-partitioned clusters each related to the Black Lives protests, and to mail-in voter fraud. Topic Clusters The resultant clusters are in Fig 8.1. For each cluster, top words ordered by highest tf-idf scores, along with an example tweet from 100 most representative tweets of the cluster are shown. The major themes relate to false claims about mail-in voter fraud, COVID-19 and pushing hydroxychloroquine as a cure, and protests concerning law enforce- ment and Black Lives Matter. Other topics target specific candidates and entities, such as social media platforms for censorship of unverified and conspiratorial content, conspiracies andallegationsagainstformerpresidentObama,ortargetingthedemocraticpartyasawhole on different social issues, and misleading claims about jobs and economy. The remaining clusters include the QAnon conspiracies, a far-right conspiracy group now banned by sev- eral platforms [28]. Another cluster related to Venezuela appears in support of right-leaning theories potentially about voter fraud, and anti-democratic posts, however, our modeling is limited to English, the most prominent language in the tweets (∼ 94% disinformation tweets were in English, followed by∼ 3% in Spanish, remaining languages less than 0.1% each). InTable8.2,welistexamplesofidentifiedunreliable/conspiracytweetswithintweetswith themostengagementsinthecollecteddataset, discardingfalsepositives, someofwhichhave 3 Pre-trained: https://fasttext.cc/docs/en/english-vectors.html 135 Table 8.2: Examples of unreliable/conspiracy tweets with most engagements in the data Unreliable/conspiracy tweet # Eng. BREAKING: Democratic Presidential nominee @JoeBiden is formally being listed as a criminal suspect by high level Ukraine government officials, in a major case involving his son - Hunter. https://t.co/Xe2bSLEAh8 56K We are being censored! @realDonaldTrump @Facebook is about to unpub- lished our FB page with 6 million followers. The NY Times recent arti- cle claiming we are right wing Provacateurs They are interfering with this election! Conservatives are being censored on FB. PLEASE RETWEET!! https://t.co/xVy8xZ7kyC 48.6K I am tired of the censorship! Anderson Cooper called me a snake oil salesman because I’m trying to get the FDA to test a supplement that I’ve seen work! And Twitter keeps taking my followers! Please RT and follow me as I support President @realDonaldTrump! 48K Can you believe what’s happening!? They give Joe Hiden’ the questions, and he reads them an answer! https://t.co/ivMw6uQ2gp 47K HEARTBREAKING A 60 year old Black Trump Supporter was murdered in coldbloodallbecausehesupportPresident@realDonaldTrumpThisisaHate CrimeHe deserves JusticeLet’smakehis nametrendUse #JusticeForBernell- Trammell https://t.co/XZbdOiHgRR 46.9K NATURE ARTICLE HOAX BUSTED!! Proof that chloroquine let’s covid attack cancer cells but not normal cells. PLEASE RETWEET. @real- DonaldTrump @IngrahamAngle @SteveFDA @drsimonegold @jennybethm https://t.co/XN0YC1liSQ 40.6K Ifwecanstandinlineatagrocerystoreorhardwarestore,wecanstandinline at the polls to vote. President @realDonaldTrump is RIGHT that universal, unmonitored vote-by-mail would be a DISASTER, and we’re already seeing evidence of that across the country. @TeamTrump https://t.co/ai1uNjQi7k 32.5K beendebunkedasfalse,misleadingorlackingevidencebyfact-checkingjournalisticsources. 45 8.3.2 Conspiracy Group Engagement QAnon, a far-right conspiracy group emerged in 2017 on 4chan, and has risen to prominence for its baseless conspiracies that have received significant following and attention [28]. The group advances the conspiracy theory that president Trump is battling a satanic child sex- trafficking ring, and an anonymous ‘Q’ claims to be a US government official with top- clearance, providing insider information about deep state operations [45]. In this subsection, 4 https://www.politifact.com/factchecks/2020/jul/28/viral-image/opening-case-file-doe s-not-mean-joe-biden-criminal/ 5 https://apnews.com/article/shootings-wisconsin-race-and-ethnicity-politics-lifestyle -23668668b3b59fa609a18d023c0bb485 136 Table 8.3: QAnon conspiracy keywords along with their occurrence frequency in tweets (original, reply or quoted tweets i.e., excluding retweets) containing the keywords Keyword Freq. Keyword Freq. wwg1wga 159436 wgaworldwide 18231 #qanon 68039 #qarmy 13577 #obamagate 78574 #pizzagate 13053 #savethechildren 33221 #taketheoath 10994 thegreatawakening 23305 greatawakening 31615 deepstate 25268 deepstatecoup 995 deepstatecabal 1669 deepstateexposed 2188 #pedogate 5211 pedowood 4454 #plandemic 8456 #scamdemic 4674 #sheepnomore 1492 adrenochrome 6397 thestorm 3989 followthewhiterabbit 95 thesepeoplearesick 2843 wearethenewsnow 5540 trusttheplan 2579 pizzagateisreal 698 thestormisuponus 887 newworldorder 901 darktolight 6825 clintonbodycount 1898 Table 8.4: Verification of 100 accounts sampled from inferred right/left-leaning accounts posting QAnon associated keywords. Verified as: (Q) is QAnon conspirator; else not (N) Inferred Leaning Human verification # Accounts Right-leaning (74k) Q 100 Left-leaning (7.6k) N (Reference QAnon) 79 N (Re-purposed hashtag) 2 Q (Incorrect leaning) 19 Undetermined (10k) Q 89 N 11 weanalyzeactivitiesofQAnonaccountsandcharacterizeitsinteractionswithotheraccounts. Identification and Verification We identify accounts that actively form part of the conspiracy group by posting original content related to QAnon conspiracies, referred to as QAnon accounts thereafter. We extract tweets (original, quote, or reply tweets) excluding retweets containing any keywords or hashtags frequently associated with the QAnon group. Table 8.3 lists the QAnon associated keywords with their frequencies. This gives 92,065 accounts with posts containing QAnon associated keywords. 7,661 of these were inferred left-leaning accounts, for 10,085 the political leaning was undetermined (not inferred), and the rest were inferred as right-leaning. For accounts posting QAnon keywords, grouped by theinferredpoliticalleaning,wesampled100accountsuniformlyatrandomfromeachgroup 137 QAnon 5.2% 217 27 84 2 171 2 1 1 56 48 2 32 4 2 58 60 34 3 42 1 Neither 34.42% IR&ID 24.9% IR-only 4.5% ID-only 30.98% 4 92 Figure 8.2: QAnon accounts interaction graph in active 1.2M accounts. Edges are retweets/ quotes/ replies from source to destination node, normalized by # accounts in source Table 8.5: QAnon interactions quantified over all accounts in the dataset. Influenced (ID) are accounts that replied, retweeted or quoted tweets from QAnon accounts. Influencer (IR) are accounts that were replied, retweeted or quoted by QAnon accounts. Table contains: # Accounts (%) 10.3 M (All users) 1.2M (Activity > 20) Group QAnon 74.3k (0.72) QAnon 62.8k (5.2) Sample IR&ID 376k (3.62) 300.7k (24.9) (22.60) IR-only 150.6k (1.45) 54.3k (4.5) (3.00) ID-only 912.5k (8.78) 374k (30.98) (47.01) Neither 8.9M (85.43) 415.7k (34.42) (22.19) (left, right, undetermined), and inspected their tweets to identify whether the accounts are promotersofQAnonconspiracies. Thevalidationanderroranalysisofthesampledaccounts isprovidedinTable8.4. Amongthe100inferredleftaccounts, 19%weremistakenlyinferred as left-leaning (and were promoters of the conspiracy), 79% were not part of the conspiracy group and only referencing the QAnon movement (oppose, ridicule, discuss or tag in their tweetscontainingQAnonkeywords),and2%wereusingthe“pedogate”hashtaginadifferent context as compared to how the hashtag was used by QAnon conspirators. Among the inferred right-leaning accounts, all were verified to be QAnon accounts. Basedonthisobservation,weretainedtheinferredright-leaningaccountspostingQAnon associated keywords as the identified QAnon accounts (since they indicate with high preci- sionsupportofthefar-rightQAnonconspiracytheories). Thistwo-stagefilteringforQAnon 138 Tweet Type Left- Leaning Right- Leaning Un- determined Reply To (RP-TO) 42.54 50.16 7.30 Replied By (RP- BY) 49.97 45.58 4.45 Retweet (RT) 8.95 90.06 0.99 Quoted (QTD) 25.03 73.44 1.53 (a) % of accounts that interacted with QAnon accounts with mentioned tweet types inferred as left or right leaning with active 1.2M accounts RP QTD RT Not (RT/QTD) Left-Leaning 0.00 0.25 0.50 0.75 1.00 RP QTD RT Not (RT/QTD) Right-Leaning 0.00 0.25 0.50 0.75 1.00 (b) Fraction of inferred left/right- leaning accounts in active 1.2M that interacted with QAnon accounts with mentioned tweet types Figure 8.3: Account interactions with QAnon accounts by tweet type and inferred leaning accounts identification, results in 100% precision and 88% recall based on the verified ex- amples weighted by actual number of accounts in each type (words not included as QAnon associated keywords are not considered in the recall measure). QAnon Interaction Graph To quantify interactions between accounts, we consider that twoaccountshaveinteractedifthereexistsaretweet(i.e.,retweetwithoutcomment),quoted tweet(i.e.,retweetwithcomment),orreplytweetedgebetweentheaccounts,inthecollected dataset. Further, to account for directionality of the interactions, we partition the accounts that have interacted with QAnon accounts into “ID” (influenced) and “IR” (influencer). Influenced accounts are those that have been influenced by (retweeted, quoted, or replied to) QAnon accounts tweets, whereas influencer are accounts that QAnon accounts have retweeted, quoted or replied to. Table 8.5 reports what percentage of all accounts were influenced but did not influence QAnon (ID-only), influencer but not influenced (IR-only), and both influencer and influenced in the data (IR&ID). Wefindthat85.43%of10.4MaccountshadnoobservedinteractionwithQAnonaccounts in the collected tweets. However, considering accounts that appeared in at least 20 tweets (original, reply, retweet, quote) in the dataset, we find only 34.42% of the active 1.2M accounts had no interaction with QAnon accounts in the collected tweets. Also, we consider a random sample of accounts from the active 1.2M accounts of the same sample size as % 139 of QAnon accounts in that set (drawn five times, and statistics averaged) for comparison. IR&ID and IR-only are similar. The main difference is in the ID-only and Neither cases, wherein % of ID-only (QAnon influenced) accounts was fewer than the expected influence accounts of the random samples (30.98 vs. 47.01), and % of accounts not interacting with QAnon is higher than that with the random samples (34.42 vs. 22.19). The interaction graph of the 1.2M active accounts (activity > 20 in the dataset) is visualizedinFig.8.2. Thus,5.2%oftheseaccountsareQAnonaccounts,asdescribedearlier. And 24.9% (IR&ID) are closely coupled accounts that both influenced QAnon content and were influenced by it (retweeted, quoted, replied). A larger 30.98% were only influenced (ID-only), while a small fraction of 4.5% were retweeted, quoted, or replied by QAnon, but did not engage back (IR-only). The edge weights correspond to volume of retweets, replies, or quoted tweets of the destination node by the source node, per account in the source node. QAnon Engagement Characteristics Next, we characterize interactions with QAnon accounts by inferred political leaning and tweet types, in the active 1.2M accounts in Fig. 8.3a. As one would expect, the retweeted and quoted tweet interactions are mostly between right-leaning and QAnon accounts (90.06% and 73.44%), as these are more close to endorsements. Replies on the other hand are roughly equally distributed amongst left and right-leaning accounts (49.97% of replies by QAnon are towards left-leaning and 42.54% of the replies to QAnon tweets are from left-leaning accounts). This suggests that QAnon get engaged in active discussions with left-leaning accounts. This might also be indicative of engagement strategies by QAnon to influence liberals by “red pilling”, i.e., have their perspective transformed, illustrated in a QAnon account tweet: “We interact with liberals the most under @realDonaldTrump’s most recent posts. Q gave us a challenge so I have been sharing the truths about COVID & nursing home deaths in hopes of redpilling anyone who sees. Next Trump tweet wanna help out? https://breitbart.com/politics/2020/06/22/exclusiv e-seema-verma-cuomo-other-democrat-governors-coronavirus-nursing -home-policies-contradicted-federal-guidance/.” 140 06-22 06-27 07-02 07-07 07-12 07-17 07-22 07-27 08-01 08-06 08-11 08-16 08-21 08-26 08-31 09-05 Timeline 10K 20K 30K Tweet Vol (QAnon keywords) DNC RNC Twitter Action 200K 300K 400K 500K Tweet Vol (QAnon accounts) (a) QAnon tweets activity timeline. (Red) To- tal daily volume of tweets from the QAnon ac- counts in the data collection period. (Blue) DailyvolumeofQAnonaccounttweetscontain- ing the listed QAnon keywords 06-22 07-02 07-12 07-22 08-01 08-11 08-21 08-31 Timeline 0.0 0.001 0.002 0.003 Frac of QAnon accounts (newly created on each day) DNC RNC Twitter Action 0.0 0.01 0.02 Frac of QAnon Tweet Volume from new accounts (created after Twitter Action) (b) QAnon accounts creation timeline. (Red) Fraction of newly created QAnon accounts per day in the data collection period. (Bottom) Fraction of total tweet volume from QAnon ac- counts that is attributed to new accounts cre- ated after Twitter Action Figure 8.4: QAnon activity and account creation timeline with respect to Twitter Action banning QAnon, and Democratic/Republican National Conventions (DNC) and (RNC) Fig. 8.3b shows distribution of inferred left/right-leaning accounts that interacted with QAnon, by tweet types. 82.17% of left-leaning in 1.2M have not retweeted/quoted QAnon accounts, but only 28.35% of right-leaning have not endorsed content from QAnon accounts. 8.3.3 Twitter Intervention Effects Twitter announced sweeping actions against the QAnon conspiracy group on July 21, 2020, designating them as harmful coordinated activity, taking down more than 7,000 QAnon accounts violating platform manipulation policies with spam, ban evasion, and operation of multiple accounts. Twitter also banned QAnon-related terms and urls from appearing in trending topics and search, limiting 150,000 other such QAnon accounts on the platform. 6 QAnon Activities Timeline Timeline of tweet volume (original, retweet, quote, reply) from QAnon accounts is visualized in Fig. 8.4a. We separately plot the total daily volume of tweetsandthedailyvolumeoftweetscontainingtheQAnonkeywords(frequentlyassociated with QAnon listed earlier in Table 8.3). As observed, although the volume of tweets with QAnonassociatedkeywordssharplydeclinesaftertheTwitterAction,buttheoverallvolume 6 https://www.nbcnews.com/tech/tech-news/twitter-bans-7-000-qanon-accounts-limits-150-0 00-others-n1234541 141 Table 8.6: Top 10 hashtags that declined (β < 0) in usage by QAnon post Twitter Action, estimated by regression discontinuity design in top 10K hashtags used by QAnon accounts Hashtag |β | slope intercept p-value wwg1wga 1.512 -0.047 4.951 0.002 qanon 0.901 -0.03 2.974 0.007 kag 0.411 -0.002 1.419 0.000 q 0.226 -0.004 0.584 0.000 qarmy 0.207 -0.008 0.7 0.048 wwg1gwa 0.139 -0.005 0.47 0.071 qanons 0.108 -0.004 0.42 0.031 patriotstriketeam 0.087 -0.001 0.164 0.010 deepstate 0.082 -0.001 0.233 0.000 inittogether 0.067 -0.002 0.19 0.083 of activities from these accounts is sustained even after imposed bans and restrictions. This clearlysuggestsagapinenforcementactionsanddesiredoutcomes. Soweinvestigateevasion strategies that rendered Twitter Action ineffective. In Fig. 8.4b we inspect whether new QAnon accounts were injected to continue activities after the Twitter Action. Whilst a declining and small fraction of new accounts were intro- duced, the fraction of the total volume of QAnon account tweets attributed to new accounts created after Twitter Action is less than 3%. Clearly, much of the volume is sustained by earlier accounts even after the ban. Regression Discontinuity Design Twitter restrictions attempted to make QAnon con- tent less visible, by banning QAnon-related content from appearing in trends and search. Therefore, we examine changes in QAnon content through analysis of hashtag usage pat- terns, that could have been employed as evasion strategies to sidestep Twitter restrictions. To that end, w e leverage regression discontinuity design (RDD) to estimate causal effects of Twitterintervention,onhashtagsadoptedbyQAnonaccounts. RDDisadesignwhichelicits causal effects of interventions by assigning a threshold above/below which an intervention is assigned [90]. By comparing observations lying closely on either side of the threshold, it is possible to estimate the average treatment effect from observed data, when randomized trials are infeasible. For RDD, we consider each hashtag in the most frequent 10K hash- tags adopted in QAnon account tweets, and fit a linear regression model on the hashtag’s 142 Table 8.7: Top 20 hashtags that increased (β > 0) in usage by QAnon post Twitter Action, estimated by regression discontinuity design in top 10K hashtags used by QAnon accounts Hashtag |β | slope intercept p-value walkawayfromdemocrats 0.138 -0.001 0.145 0.000 saveourchildren 0.099 0.002 0.011 0.158 huge 0.074 -0.001 0.016 0.130 heelsupharris 0.068 0.0 0.006 0.098 vote 0.067 0.0 0.071 0.005 warroompandemic 0.065 -0.001 0.041 0.007 th3d3n 0.064 -0.001 0.016 0.002 womenfortrump 0.061 0.0 0.086 0.133 thesilentmajority 0.059 0.0 0.017 0.122 sallyyates 0.058 -0.001 0.015 0.058 hcqworksfauciknewin2005 0.055 -0.001 0.014 0.000 nomailinvoting 0.053 0.0 0.014 0.000 mo03 0.051 -0.001 0.015 0.032 trump2020victory 0.05 -0.001 0.03 0.001 hermancain 0.05 -0.001 0.016 0.003 bigpharma 0.05 -0.001 0.018 0.000 jimcrowjoe 0.05 -0.001 0.032 0.072 bidenisapedo 0.049 -0.001 0.014 0.011 vppick 0.049 -0.001 0.008 0.172 hcqzinc4prevention 0.048 -0.001 0.015 0.000 daily usage volume during the data collection period, with an additional treatment variable assigned to the regression model to capture intervention effect. The regression model is as follows, y =mx+b+β I{x>x 0 }. Here, the hashtag usage volume per day (i.e., how many timesthehashtagwasadoptedintweetsfromQAnonaccounts),normalizedbytweetvolume from QAnon accounts on that day, is modeled by dependent variable y, over each day x. The slope m and intercept b captures the trend in hashtag usage, and the coefficient β of the treatment variableI{x>x 0 } captures the discontinuity at threshold x 0 , selected as the end of the week that Twitter announced and enforced its restrictions on QAnon. In Table 8.6 and 8.7, we list identified hashtags with highest estimated treatment effects |β | (i.e., most change with respect to Twitter’s intervention on QAnon with p-value of β at most 0.2) and highlighted in bold are p-value ≤ 0.05 i.e., 95% confidence interval. 7 In the assumed function, since a common slope with different intercepts ( b before intervention and b+β after intervention) is used to capture the estimated treatment effect, both coefficients β and m determine whether the hashtag’s usage declined or increased post intervention. For 7 The goodness of fit tests and other regression statistics, with few illustrated examples of identified hashtags are in the Appendix. 143 Figure 8.5: RDD data plot for example hashtags (Left) “wwg1wga” (Middle) “walka- wayfromdemocrats” (Right) “nomailinvoting” before and after Twitter action (fraction of hashtag usage in volume of QAnon tweets ’y’ vs. day ’x’) declining hashtags, β < 0,m ≤ 0 are indicators of decreased intercept post intervention, whereas β > 0,m≥ 0 indicate increased intercept and increased hashtag usage post inter- vention. An error margin of± 0.001 from 0 was considered for the slope based on inspection of scatter plots of data for fitted regressions that were nearly parallel to the x-axis, resulting in difference of averages on either side of the intervention as estimated treatment effects. As per Table 8.6 and 8.7, we can see a change of discourse strategy in the hashtags used. The top declining ones include “WWG1WGA” (where we go one, we go all), “qanon”, “deepstate” and others which are directly related to the QAnon movement and the top increasing ones include “walkawayfromdemocrats”, “nomailinvoting” and hijacked hashtag “saveourchildren”amongothers,whicharenotdirectlyrelatedtoQanon,buttotheconspir- acytheoriestheyarepushing. Forexample,thehashtag“saveourchlidren”hasbeendetected as part of a strategy used by Qanon users to stay under cover and promote the conspiracy of child sex trafficking by Democratic leaders 8 . It was later also banished (or limited) by platforms such as Facebook 9 . Regression statistics in terms of goodness-of-fit for different RDD regression functions are in the Appx. B and illustrations are in Fig. 8.5. Engagement Characteristics We examined the change in volume of engagements (di- rect engagements in the form of replies to, retweets of, quotes of) with QAnon accounts 8 https://www.vox.com/21436671/save-our-children-hashtag-qanon-pizzagate 9 https://techcrunch.com/2020/10/30/facebook-is-limiting-distribution-of-save-our-child ren-hashtag-over-qanon-ties/ 144 Table 8.8: % Decrease in ratio of direct engagements (reply to, retweet, quote) with QAnon accounts tweets to volume of QAnon accounts tweets, before and after Twitter Action Tweet Type Vol./ day (Before) Vol./ day (After) %Increase (Vol./ day) %Decrease (Eng vol./QAnontweet) QAnon 295,267 361,176 22.32 - RP TO 49,161 46,114 -6.2 23.32 RT 177,013 189,387 6.99 12.53 QTD 8908 8985 0.86 17.55 2.5 5.0 7.5 10.0 12.5 Cascade Size (Log-Scale) 0.0 0.2 0.4 0.6 0.8 1.0 CCDF Unreliable/consp. Reliable (a) 0 100 200 300 Depth 0.0 0.2 0.4 0.6 0.8 1.0 Mean Breadth (normalized by cascade size) Unreliable/consp. Reliable 0 100 200 0.00 0.02 0.04 0.06 (b) (c) (d) Figure 8.6: Comparison of information propagation dynamics of reliable vs. unreliable/ conspiracy cascades identified in election-related tweets. (a) CCDF of cascade size. (b) Mean breadth to depth ratio. Reliable cascades run broader at shorter depths of cascade propagation trees. (c) Avg. unique users reached at each cascade size with more repeated engagements from same accounts in unreliable cascades. (d) Mean time to reach unique users is higher and more bursty for unreliable cascades tweets, following Twitter intervention on QAnon (Tab 8.8). While the volume of retweet engagements per day per QAnon account tweet decreased by 12.53%, from earlier part of the timeline to later half after Twitter action, the overall volume of retweet engagements per day increased by ∼ 7%. This can be explained by 22.32% increase in QAnon tweet volume per day, compensating for decrease in engagements per tweet. Similar for quoted tweets, but for reply tweets the engagements volume declined overall by∼ 6% in the latter half. 8.3.4 Propagation Dynamics of Cascades In Fig. 8.6, we provide additional analysis of engagements with unreliable/conspiracy tweets bycomparingcascadepropagationdynamics. Theengagementswiththeunreliable/conspiracy tweets appeared less viral (mean time to reach unique accounts is higher (Fig. 8.6d), and meanbreadthofaccountsreachedatlesserdepthofthepropagationtreeissmaller(Fig.8.6b)). 145 The propagation tree corresponding to each cascade was constructed from available retweet, replyorquotelinksbetweentweets. Yet,therecanbeseveralunreliable/conspiracycascades thatdogetmanyengagements(cascadesizeCCDF(Fig8.6a)). Thefindingsonpropagation structure are similar to previous findings on unverified or rumor cascades [47]. In addition, unreliable/conspiracy cascades appear to have more repeated engagements (reaching fewer unique users for the same cascade size (Fig. 8.6c)), which is also observed in [138]. Vosoughi et al. (2018) studied propagation dynamics of false and true news (verified on fact-checking websites e.g. snopes.com) on Twitter from 2006-2017. They found that fact- checked false news was more viral (diffused farther, faster, deeper and more broadly) than fact-checked true news. We note that the study provides useful findings, however, is spe- cific to information that has been fact-checked (e.g. PolitiFact often fact-checks statements submitted by readers, and mentions that since they cannot check all claims, they select the most newsworthy and significant ones 10 ) Also, the fact-checked true rumors would likely not include mainstream news articles. In comparison, our findings about unreliable/conspiracy cascades being less viral than reliable cascades illustrate a more general comparison, beyond onlypopularfalseornon-mainstreamtruenews. InAppx.B,weadditionallyshowtheprop- agation dynamics for the subset of labeled cascades labeled using the news source domain (unreliable, conspiracy and reliable), without considering the inferred labels from the CSI model, in order to ensure limited model bias in the analysis and find it to be consistent. 8.4 Discussions and Conclusion We studied the disinformation landscape through identification of unreliable/ conspiracy tweets, and analysis of QAnon conspiracy group and its activities. We focused on character- ization oftargeted topics ofdisinformation, and engagements with QAnonbased on political leaning and tweet types, to understand how attempts to manipulate the discourse prior to the election were conducted. Unfortunately, our findings also suggest that Twitter actions 10 PolitiFact.com/How we choose claims to fact-check. 146 to limit the QAnon conspiracy might not have been sufficient and effective, and even known conspiracy groups can adapt and evade imposed restrictions. The findings pertaining to the research questions can be summarized as follows. • Applying CSI [138] to detect unreliable/conspiracy tweets in the election dataset, we obtainvalidationAUC0.8/F10.76. Theprominenttopicsofunreliable/conspiracy tweets preceding the election included mail-in voter fraud, COVID-19, Black Lives Matter, media censorship, and disinformation about political candidates. • While unreliable/conspiracy tweets were widespread and diverse, accounts en- gagements with such tweets were less viral than with reliable tweets (i.e., mean time to reach unique accounts is higher, mean cascade breath is smaller). Yet, there can be several such tweets which manage to receive substantial attention and engagements. • QAnon far-right conspiracies are quite prevalent on Twitter, and we find that 85.43% of all accounts did not have an observed interaction with QAnon accounts (i.e., accounts with QAnon conspiracy related keywords in their tweets), but the frac- tion of such accounts is much smaller (34.42%) in the 1.2M active accounts (i.e., with at least 20 tweets in the dataset). Their influence however extends to fewer accounts, compared to the expected influence of a random sample of the active accounts. Inter- actions of QAnon accounts suggests that they try to engage in active discussions (bidi- rectional replies) with left-leaning accounts to “red pill” (i.e., transform perspectives with disturbing revelations), while right-leaning interactions include replies/retweets. • QAnon accounts’ activities before and after the Twitter restrictions in July, 2020 indicate that older accounts created before the intervention continue to sustain the volume of tweets, but adapt to alternate hashtags related to their conspiracies and increased tweeting volume to continue pushing their conspiracy, and counteract the Twitter restrictions, as inferred by causal effect estimation using a regression disconti- nuity design to identify statistically significant changes in QAnon hashtag usage. 147 In conclusion, we discuss the limitations of the current study. Here, we regard right- leaning accounts that tweet pre-defined QAnon related keywords or hashtags as QAnon accounts. In future work, it will be desirable to make finer distinctions to characterize conspiratorial vs. non-conspiratorial far-right narratives in QAnon tweets, as well as to characterize tweets that debunk QAnon conspiracies, and investigate account interactions based on these distinctions. Another interesting observation is that QAnon accounts influ- enced fewer accounts (ID-only) as compared to the expected influence of a random sample. This could suggest that QAnon accounts are either not as influential or specifically target a smaller audience. More investigation is needed to understand this behavior. We also discuss the limitations of causal estimation from observational data. In RDD, sincewedonotcontrolforconfoundingvariablesthatmightberelatedtoreal-worldevents,it ispossiblethataspikecausedinahashtag’susageneartheinterventionmightbeattributed to the intervention, instead of a possible unrelated confounding event. Also, given the certaintyoftheTwitteractiondate,weuseRDDwithahardthreshold,whichisareasonable assumption based on inspection of the results; however, extensions to fuzzy RDD could be alternatively considered [90]. We also discuss inherent limitations in data collection. The standard API allows access to a∼ 1% sample of the Twitter stream filtered on the keywords. 148 Chapter 9 Conclusions and Future Work This thesis addressed several challenges in disinformation mitigation [142]. We discussed the problem of disinformation labeling [147, 148], proposed techniques for early detection of disinformation content (TCNN-URG) [134], and proposed unsupervised techniques for detection of coordinated disinformation from malicious account groups (AMDN-HAGE and VigDet)leveragingdiffusiondynamicsinferredfromobservedactivitiesofaccountscapturing latent influence between accounts and their collective group anomalous behaviors [145, 185]. Next, we addressed how disinformation propagates with analysis of diffusion cascades, and inference of propagation dynamics and influential users from observed, unlabeled cascades, under a mixture of diffusion models (MIC) [144]. The proposed techniques were aimed at developing timely response to disinformation by limiting viral cascades, and uncovering of suspicious malicious behaviors on the network. Both MIC and AMDN-HAGE use unsuper- visedestimationofthediffusionbehaviorsandlearnlatentinfluencebetweenaccountsbased on the observed accounts activities or diffusion cascades or its propagation patterns. Unsupervised inference methods of the diffusion process governing disinformation and coordinated operations are extremely useful since much of the data is unlabeled, since col- lectinglarge-scale,comprehensiveandaccuratedisinformationlabelsforthecontentsrequire expensivejournalisticverification,comparedtocheaplyavailableunlabeledcontentcascades. Moreover, prior knowledge of accounts involved in coordinated campaigns even partial, or knowing what disinformation campaigns exist, or its pre-defined characteristics is not feasi- ble. This work eliminated these limitations and shows significant improvement over earlier methods. AMDN-HAGE [145] also identified suspicious and likely coordinated activities in promoting anti-mask messages and conspiracies about the pandemic being political hoax with respect to the COVID-19 pandemic and vaccines [148, 145, 143]. 149 Besidesmitigationtechniques,characterizationofsocialmediadiscourseandengagement with disinformation and conspiracy groups enables improved understanding of the impact and risk associated with disinformation. We studied social media engagement preceding the U.S. 2020 Presidential Election [146], the COVID-19 pandemic and vaccines [148, 143]. While we found that disinformation and conspiracy engagements on Twitter are less viral than reliable information, some do receive high engagements. We observed that only a small fraction (34%) of active accounts had no QAnon conspiracy group interactions (and 85% of all accounts had no observed interactions with QAnon). The majority of retweet interactions was from right-leaning active accounts whereas left-leaning active accounts were bidirectionally engaged in reply based discussions. We also studied the impact of Twitter restrictionsimposedontheQAnonconspiracygroupinmidJuly,2020andprovideoneofthe first characterizations of the engagement and effect of interventions on conspiracy groups by social media platforms. Our findings suggest that QAnon content posting strategies change causally,estimatedusingaregressiondiscontinuitydesign,duetoTwitter’srestrictions,inan effort to increase its reach by counteracting restrictions. They employed alternate hashtags related to their conspiracies, and increased activity rates, managing to sustain their tweet volume and retweet engagements, even though the per tweet engagement reduced after the restrictions. This points to the need for more robust platform interventions [146]. Limitations and Future Work We conclude with the limitations and directions of future research in mitigation of disin- formation and social media opinion manipulation. This thesis has demonstrated significant progresstowards alleviatingsome ofthechallengesindisinformationormaliciouscampaigns detection, and mitigation techniques. However, the problem is extremely challenging and has inherent difficulties such as the non-i.i.d network dynamics, and the difficulty of clearly separating misleading contents and nuanced distortion of facts. 150 Fine-Grained Disinformation Labeling In Chapter 4 we proposed a framework for model-guided refinement of weak labels that incorporates social context a post to estimate instance and user credibility and stance. We showed that the proposed approach can min- imize human annotation dependency in new domains allowing for timely and scalable dis- information labeling. It is able to remove weak label noise with high recall when both the social context and self-supervision from a disinformation detection model are jointly lever- aged. This approach is promising with the label annotation guidelines we tested to capture different degrees of distortion. However, we also observed that the fine-grained classifica- tion with machine learning models for different distortion types (false, unproven, mixture i.e., missing or misleading context, true, debunk) is a harder problem. The imbalance in the classes and soft boundaries in label types severely limits the ability to separate at a fine-grained level. These distinctions can be useful for targeted interventions and to detect morecomplexdistortioncases. Wethinkthatsemi-supervisedlearningtechniques[173]with data-augmentation strategies to increase labels in the smaller imbalanced classes can help to address these challenges. Another strategy would be to utilize active learning [85, 2] to select the most informative human labeled prototypes for the imbalanced classes, using signals from model-guided refinement incorporating social context. Multi-Platform Coordinated Manipulation To keep platforms safe from abuse, char- acterizing and advancing the proposed detection and mitigation techniques to identify new and increasing forms of sophistication in malicious operations and changing landscapes of platforms is essential. In Chapter 6, we proposed an unsupervised, generative model to es- timate the latent or unobserved influence between activities of accounts and their collective group behaviors. The learned group anomaly from observed cascades was used to detected malicious coordinated campaigns in a data-driven approach. We further plan to investi- gate coordinated behaviors in multi-platform environments. Recent works have started to investigate and collect colluding account groups datasets from Reddit, Twitter to analyze 151 cross-platform coordination [102, 68], as well as from black-market sites that allow collu- sion to artificially boost followers and content [34]. Our proposed model in this thesis for data-driven detection would serve well if extended to modeling cross-platform influence and social network alignment for multi-platform coordination detection. In addition, there are research opportunities to characterize new platforms like ‘Birdwatch’ introduced by Twitter in 2021 to crowd-source identification and contextualization of misleading posts. This itself can become a target of coordinated manipulation. Also, future studies could inform how to reach consensus about the truth from such crowd-enabled platforms for disinformation mitigation. Diffusion inference in this thesis is critical to solving the above challenges. It does have some limitations such as scalability. High-dimensional networks become computationally challenging both in terms of time and memory requirements for diffusion modeling and learning from observed cascades. Future research could consider representation learning advances in graph neural networks with adaptive sampling [64] or graph coarsening [97] to make the neural diffusion/ point process model more scalable for large-scale networks. Mitigation of User Susceptibility and Bias A broad area that is under-investigated is the causal impact of disinformation on opinions in social media data. Earlier studies [73, 161] have observed that study participants who were shown false and conspiratorial contentshowedreducedpro-socialintentsandwereinfluenced. Thesefindingshighlightthat peoplecanbeinfluencedbydisinformation. RecentworkbyPhadkeetal.2021findsthrough regressionwithcontrolvariablesthatrepliesfromconspiracygroupsarestrongprecursorsto users joining the conspiracy group on Reddit. Cheng et al. [22] examined features that are causally related to engagement with disinformation, but focused on individual rather than community dynamics. In Covid-19 vaccine data, we observed communities of accounts that aremoremisinformedandareseparatedfrominformedcommunitiesthatconsumenewsand healthcare information and support vaccination. We also see that there is a correlation in 152 the political-leaning, anti-vaccine stance, and disinformation (Chapter 4, Appendix A.1.2). An interesting research direction is to use longitudinal social media data to determine the causal relationship between disinformation engagement and opinions controlling for prior beliefs and outside demographic influences. Linguisticanalysiscouldalsoinformhowtoreduceharmfulnetworkinteractions. Study- ing causal linguistic influence from social media posts can help estimate which linguistic properties promote toxic engagement or are drivers of persuasion to join disinformation or conspiracy groups. Some research in natural language processing developed causal text em- beddings and causal linguistic analysis that could be applied for estimation for social media [165], and others that use text to eliminate confounding bias in causal estimation [75]. Another critical factor in disinformation influence on opinion comes from misleading and missing context. Information biases on social media can exacerbate the problem and lead to distorted understanding of the truth, such as more frequent discussion of rarer vaccine side-effects (Fig. A.7a). It would be interesting to study if providing additional context to social media users by presenting aggregate analysis of the discourse can mitigate informa- tion bias and susceptibility to disinformation. One approach could be to present discourse overview through topic models, emerging hashtags, community structure, diffusion cascade visualizations, geo-political analysis [143]. A promising study in reducing political polar- ization proposed Social Mirror, a social network visualization tool that enables a sample of Twitter users to explore the politically-active parts of their social network [52]. 153 References [1] Aseel Addawood, Adam Badawy, Kristina Lerman, and Emilio Ferrara. Linguistic cues to deception: Identifying political trolls on social media. In ICWSM, 2019. [2] UmangAggarwal,AdrianPopescu,andC´ elineHudelot. Activelearningforimbalanced datasets. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1428–1437, 2020. [3] HuntAllcottandMatthewGentzkow. Socialmediaandfakenewsinthe2016election. Journal of Economic Perspectives, 31(2):211–36, 2017. [4] Marco Amoruso, Daniele Anello, Vincenzo Auletta, and Diodato Ferraioli. Contrast- ing the spread of misinformation in online social networks. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, pages 1323–1331. Inter- national Foundation for Autonomous Agents and Multiagent Systems, 2017. [5] Travis M. Andrews. Facebook and other companies are removing viral ‘Plandemic’ conspiracy video, 2020 (accessed June, 2021). URL https://www.washingtonpost.c om/technology/2020/05/07/plandemic-youtube-facebook-vimeo-remove/. [6] Eric Arazo, Diego Ortego, Paul Albert, Noel O’Connor, and Kevin Mcguinness. Un- supervised label noise modeling and loss correction. In International Conference on Machine Learning, pages 312–321, 2019. [7] AdamBadawy,AseelAddawood,KristinaLerman,andEmilioFerrara. Characterizing the 2016 russian ira influence campaign. SNAM, 9(1):31, 2019. [8] Adam Badawy, Kristina Lerman, and Emilio Ferrara. Who falls for online political manipulation? In Companion Proceedings of The 2019 World Wide Web Conference, pages 162–168, 2019. 154 [9] Jeff Baumes, Mark Goldberg, Malik Magdon-Ismail, and William Al Wallace. Discov- ering hidden groups in communication networks. In ICISI. Springer, 2004. [10] Julian Besag. Statistical analysis of non-lattice data. The Statistician, 24(3):179–195, 1975. [11] Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment, 2008(10):P10008, 2008. [12] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. TACL, 5, 2017. [13] Simon Bourigault, Sylvain Lamprier, and Patrick Gallinari. Representation learning forinformationdiffusionthroughsocialnetworks: anembeddedcascademodel. In Pro- ceedings of the Ninth ACM international conference on Web Search and Data Mining, pages 573–582, 2016. [14] Yuri Boykov, Olga Veksler, and Ramin Zabih. Fast approximate energy minimization via graph cuts. IEEE Transactions on pattern analysis and machine intelligence, 23 (11):1222–1239, 2001. [15] Lia Bozarth, Aparajita Saraf, and Ceren Budak. Higher ground? how groundtruth labeling impacts our understanding of fake news about the 2016 u.s. presidential nom- inees. Proceedings of the International AAAI Conference on Web and Social Media, 14(1):48–59, May 2020. URL https://ojs.aaai.org/index.php/ICWSM/article/v iew/7278. [16] Ceren Budak, Divyakant Agrawal, and Amr El Abbadi. Limiting the spread of mis- information in social networks. In Proceedings of the 20th international conference on World wide web, pages 665–674. ACM, 2011. 155 [17] Qi Cao, Huawei Shen, Keting Cen, Wentao Ouyang, and Xueqi Cheng. Deephawkes: Bridging the gap between prediction and understanding of information cascades. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Manage- ment, pages 1149–1158, 2017. [18] Qiang Cao, Xiaowei Yang, Jieqi Yu, and Christopher Palow. Uncovering large groups ofactivemaliciousaccountsinonlinesocialnetworks. In Proceedings of the 2014 ACM Conference on Computer and Communications Security, pages 477–488, 2014. [19] Carlos Castillo, Marcelo Mendoza, and Barbara Poblete. Information credibility on twitter. In Proceedings of the 20th international conference on World wide web, pages 675–684. ACM, 2011. [20] Emily Chen, Ashok Deb, and Emilio Ferrara. #election2020: The first public twitter dataset on the 2020 us presidential election, 2020. [21] Wei Chen, Tian Lin, Zihan Tan, Mingfei Zhao, and Xuren Zhou. Robust influence maximization. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 795–804. ACM, 2016. [22] LuCheng, RuochengGuo, KaiShu, andHuanLiu. Causalunderstandingoffakenews dissemination on social media. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 148–157, 2021. [23] Nicole A Cooke. Posttruth, truthiness, and alternative facts: Information behavior and critical information consumption for a new age. The Library Quarterly, 87(3): 211–221, 2017. [24] Alessandro Cossard, Gianmarco De Francisci Morales, Kyriaki Kalimeri, Yelena Mejova, Daniela Paolotti, and Michele Starnini. Falling into the echo chamber: the italian vaccination debate on twitter. In Proceedings of the International AAAI Con- ference on Web and Social Media, volume 14, pages 130–140, 2020. 156 [25] Stefano Cresci. A decade of social bot detection. Communications of the ACM, 63 (10):72–83, 2020. [26] Enyan Dai, Yiwei Sun, and Suhang Wang. Ginger cannot cure cancer: Battling fake health news with a comprehensive data repository. In Proceedings of the International AAAI Conference on Web and Social Media, volume 14, pages 853–862, 2020. [27] Daryl J Daley and David Vere-Jones. An introduction to the theory of point processes: volume II: general theory and structure. Springer, 2007. [28] Daniel de Zeeuw, Sal Hagen, Stijn Peeters, and Emilija Jokubauskaite. Tracing normiefication. First Monday, 2020. [29] Matthew DeVerna, Francesco Pierri, Bao Truong, John Bollenbacher, David Axelrod, Niklas Loynes, Cristopher Torres-Lugo, Kai-Cheng Yang, Fil Menczer, and John Bry- den. Covaxxy: A global collection of english twitter posts about covid-19 vaccines. arXiv preprint arXiv:2101.07694, 2021. [30] Pedro Domingos and Matt Richardson. Mining the network value of customers. In Proceedings of the seventh ACM SIGKDD international conference on Knowledge dis- covery and data mining, pages 57–66. ACM, 2001. [31] Mark Dredze, Michael J Paul, Shane Bergsma, and Hieu Tran. Carmen: A twitter geolocation system with applications to public health. In Workshops at the Twenty- Seventh AAAI Conference on Artificial Intelligence , 2013. [32] Lawrence N Driscoll. A validity assessment of written statements from suspects in criminal investigations using the scan technique. Police Stud.: Int’l Rev. Police Dev., 17:77, 1994. [33] Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodriguez, 157 and Le Song. Recurrent marked temporal point processes: Embedding event history to vector. In ACM SIGKDD, pages 1555–1564, 2016. [34] Hridoy Sankar Dutta, Udit Arora, and Tanmoy Chakraborty. Abome: A multi- platform data repository of artificially boosted online media entities. In Proceedings of the International AAAI Conference on Web and Social Media, volume 15, pages 1000–1008, 2021. [35] Adam M Enders, Joseph E Uscinski, Casey Klofstad, and Justin Stoler. The different formsofcovid-19misinformationandtheirconsequences. TheHarvardKennedySchool Misinformation Review, 2020. [36] Mehrdad Farajtabar, Nan Du, Manuel Gomez Rodriguez, Isabel Valera, Hongyuan Zha, and Le Song. Shaping social activity by incentivizing users. Advances in Neural Information Processing Systems, 27:2474–2482, 2014. [37] Mehrdad Farajtabar, Jiachen Yang, Xiaojing Ye, Huan Xu, Rakshit Trivedi, Elias Khalil, Shuang Li, Le Song, and Hongyuan Zha. Fake news mitigation via point process based intervention. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 1097–1106. PMLR, 2017. [38] Sergiy Fefilatyev, Matthew Shreve, Kurt Kramer, Lawrence Hall, Dmitry Goldgof, Rangachar Kasturi, Kendra Daly, Andrew Remsen, and Horst Bunke. Label-noise reduction with support vector machines. In Proceedings of the 21st International Con- ference on Pattern Recognition (ICPR2012), pages 3504–3508. IEEE, 2012. [39] Jon Feldman, Ryan O’Donnell, and Rocco A Servedio. Learning mixtures of product distributions over discrete domains. SIAM Journal on Computing, 37(5):1536–1564, 2008. [40] SongFeng, RitwikBanerjee, andYejinChoi. Syntacticstylometryfordeceptiondetec- tion. In Proceedings of the 50th Annual Meeting of the Association for Computational 158 Linguistics: Short Papers-Volume 2, pages 171–175. Association for Computational Linguistics, 2012. [41] Vanessa Wei Feng and Graeme Hirst. Detecting deceptive opinions with profile com- patibility. In IJCNLP, pages 338–346, 2013. [42] Emilio Ferrara. What types of covid-19 conspiracies are populated by twitter bots? First Monday, 2020. [43] Emilio Ferrara, Onur Varol, Clayton Davis, Filippo Menczer, and Alessandro Flam- mini. The rise of social bots. Communications of the ACM, 59(7):96–104, 2016. [44] Emilio Ferrara, Onur Varol, Filippo Menczer, and Alessandro Flammini. Detection of promotedsocialmediacampaigns. In tenth international AAAI conference on web and social media, 2016. [45] Emilio Ferrara, Herbert Chang, Emily Chen, Goran Muric, and Jaimin Patel. Charac- terizing social media manipulation in the 2020 us presidential election. First Monday, 2020. [46] Marc Fisher, John Woodrow Cox, and Peter Hermann. Pizzagate: From rumor, to hashtag, to gunfire in dc. Washington Post, 2016. [47] Adrien Friggeri, Lada A Adamic, Dean Eckles, and Justin Cheng. Rumor cascades. In ICWSM, 2014. [48] ChristieMFuller,DavidPBiros,andRickLWilson. Decisionsupportfordetermining veracity via linguistic-based cues. Decision Support Systems, 46(3):695–703, 2009. [49] Vijaya Gadde and Yoel Roth. Enabling further research of information operations on twitter. Twitter Blog, 17, 2018. 159 [50] Kiran Garimella, Aristides Gionis, Nikos Parotsidis, and Nikolaj Tatti. Balancing informationexposureinsocialnetworks. InAdvances in Neural Information Processing Systems, pages 4663–4671, 2017. [51] Kiran Garimella, Gianmarco De Francisci Morales, Aristides Gionis, and Michael Mathioudakis. Quantifying controversy on social media. ACM Transactions on Social Computing, 1(1):1–27, 2018. [52] Nabeel Gillani, Ann Yuan, Martin Saveski, Soroush Vosoughi, and Deb Roy. Me, my echo chamber, and i: Introspection on social media polarization. In Proceedings of the 2018 World Wide Web Conference on World Wide Web, pages 823–831. International World Wide Web Conferences Steering Committee, 2018. [53] Mahak Goindani and Jennifer Neville. Social reinforcement learning to combat fake news spread. In Ryan P. Adams and Vibhav Gogate, editors, Proceedings of The 35th Uncertainty in Artificial Intelligence Conference , volume 115 of Proceedings of Machine Learning Research, pages 1006–1016. PMLR, 22–25 Jul 2020. URL http: //proceedings.mlr.press/v115/goindani20a.html. [54] Jennifer Golbeck, Matthew Mauriello, Brooke Auxier, Keval H Bhanushali, Christo- pher Bonk, Mohamed Amine Bouzaghrane, Cody Buntain, Riya Chanduka, Paul Cheakalos, Jennine B Everett, et al. Fake news vs satire: A dataset and analysis. In Proceedings of the 10th ACM Conference on Web Science, pages 17–21. ACM, 2018. [55] Manuel Gomez-Rodriguez, David Balduzzi, and Bernhard Sch¨ olkopf. Uncovering the temporaldynamicsofdiffusionnetworks. In Proceedings of the 28th International Con- ference on International Conference on Machine Learning, ICML’11, page 561–568, Madison, WI, USA, 2011. Omnipress. ISBN 9781450306195. [56] Amit Goyal, Wei Lu, and Laks VS Lakshmanan. Celf++: optimizing the greedy 160 algorithm for influence maximization in social networks. In Proceedings of the 20th international conference companion on World wide web, pages 47–48. ACM, 2011. [57] Daniel Gruhl, Ramanathan Guha, David Liben-Nowell, and Andrew Tomkins. Infor- mationdiffusionthroughblogspace. In Proceedings of the 13th international conference on World Wide Web, pages 491–501. ACM, 2004. [58] David G¨ uera and Edward J Delp. Deepfake video detection using recurrent neural networks. In 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pages 1–6. IEEE, 2018. [59] Sonu Gupta, Ponnurangam Kumaraguru, and Tanmoy Chakraborty. Malreg: De- tecting and analyzing malicious retweeter groups. In Proceedings of the India Joint International Conference on Data Science and Management of Data, pages 61–69, 2019. [60] Douglas M Hawkins. Identification of outliers , volume 11. Springer, 1980. [61] Xinran He and Yan Liu. Not enough data?: Joint inferring multiple diffusion net- works via network generation priors. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, pages 465–474. ACM, 2017. [62] XinranHe,GuojieSong,WeiChen,andQingyeJiang.Influenceblockingmaximization in social networks under the competitive linear threshold model. In Proceedings of the 2012 SIAM International Conference on Data Mining, pages 463–474. SIAM, 2012. [63] Kathleen Higgins. Post-truth: a guide for the perplexed. Nature News, 540(7631):9, 2016. [64] Wenbing Huang, Tong Zhang, Yu Rong, and Junzhou Huang. Adaptive sampling towardsfastgraphrepresentationlearning. Advances in Neural Information Processing Systems, 31:4558–4567, 2018. 161 [65] Adam Hughes and Stefan Wojcik. 10 facts about Americans and Twitter, 2019 (ac- cessed March 20, 2020). URL https://www.pewresearch.org/fact-tank/2019/08/ 02/10-facts-about-americans-and-twitter/. [66] Minyoung Huh, Andrew Liu, Andrew Owens, and Alexei A Efros. Fighting fake news: Image splice detection via learned self-consistency. In Proceedings of the European Conference on Computer Vision (ECCV), pages 101–117, 2018. [67] Jane Im, Eshwar Chandrasekharan, Jackson Sargent, Paige Lighthammer, Taylor Denby, Ankit Bhargava, Libby Hemphill, David Jurgens, and Eric Gilbert. Still out there: Modeling and identifying russian troll accounts on twitter. In ACM Web Sci- ence, pages 1–10, 2020. [68] Maurice Jakesch, Kiran Garimella, Dean Eckles, and Mor Naaman. Trend alert: A cross-platform organization manipulated twitter trends in the indian general election. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2):1–19, 2021. [69] Amelia M Jamison, David A Broniatowski, Mark Dredze, Anu Sangraula, Michael C Smith, and Sandra C Quinn. Not just conspiracy theories: Vaccine opponents and proponents add to the covid-19 ‘infodemic’on twitter. Harvard Kennedy School Mis- information Review, 1(3), 2020. [70] AndrzejJarynowski,AlexanderSemenov,andVitalyBelik. Protestperspectiveagainst covid-19riskmitigationstrategiesonthegermaninternet. In International Conference on Computational Data and Social Networks, pages 524–535. Springer, 2020. [71] Fang Jin, Edward Dougherty, Parang Saraf, Yang Cao, and Naren Ramakrishnan. Epidemiological modeling of news and rumors on twitter. In Proceedings of the 7th Workshop on Social Network Mining and Analysis, page 8. ACM, 2013. [72] Daniel Jolley and Karen M Douglas. The effects of anti-vaccine conspiracy theories on vaccination intentions. PloS one, 9(2):e89177, 2014. 162 [73] DanielJolley,RoseMeleady,andKarenMDouglas. Exposuretointergroupconspiracy theoriespromotesprejudicewhichspreadsacrossgroups. BritishJournalofPsychology, 111(1):17–35, 2020. [74] Prerna Juneja and Tanushree Mitra. Auditing e-commerce platforms for algorithmi- cally curated vaccine misinformation. In Proceedings of the 2021 chi conference on human factors in computing systems, pages 1–27, 2021. [75] Katherine Keith, David Jensen, and Brendan O’Connor. Text and causal inference: A review of using text to remove confounding from causal estimates. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5332–5344, 2020. [76] David Kempe, Jon Kleinberg, and ´Eva Tardos. Maximizing the spread of influence through a social network. In Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 137–146. ACM, 2003. [77] Dhruv Khattar, Jaipal Singh Goud, Manish Gupta, and Vasudeva Varma. Mvae: Multimodal variational autoencoder for fake news detection. In The World Wide Web Conference, pages 2915–2921. ACM, 2019. [78] JooyeonKim,BehzadTabibian,AliceOh,BernhardSch¨ olkopf,andManuelGomezRo- driguez. Leveraging the crowd to detect and reduce the spread of fake news and mis- information. In Proceedings of the 11th ACM International Conference on Web Search and Data Mining (WSDM 2018), 2018. [79] Sangyeon Kim, Omer F Yalcin, Samuel E Bestvater, Kevin Munger, Burt L Monroe, and Bruce A Desmarais. The effects of an informational intervention on attention to anti-vaccination content on youtube. In Proceedings of the International AAAI Conference on Web and Social Media, volume 14, pages 949–953, 2020. 163 [80] D. P. Kingma and M. Welling. Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014. [81] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolu- tional networks. arXiv preprint arXiv:1609.02907, 2016. [82] Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and tech- niques. MIT press, 2009. [83] Vladimir Kolmogorov and Ramin Zabin. What energy functions can be minimized via graph cuts? IEEE transactions on pattern analysis and machine intelligence, 26(2): 147–159, 2004. [84] Philipp Kr¨ ahenb¨ uhl and Vladlen Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011. URL https://proceedings.neurips.cc /paper/2011/file/beda24c1e1b46055dff2c39c98fd6fc1-Paper.pdf. [85] Jan Kremer, Fei Sha, and Christian Igel. Robust active label correction. In Interna- tional conference on artificial intelligence and statistics , pages 308–316. PMLR, 2018. [86] Srijan Kumar and Neil Shah. False information on web and social media: A survey. arXiv preprint arXiv:1804.08559, 2018. [87] SejeongKwon, MeeyoungCha, KyominJung, WeiChen, andYajunWang. Prominent features of rumor propagation in online social media. In 2013 IEEE 13th International Conference on Data Mining, pages 1103–1108. IEEE, 2013. [88] Sejeong Kwon, Meeyoung Cha, and Kyomin Jung. Rumor detection over varying time windows. In Harvard Dataverse. Harvard Dataverse, 2016. doi: 10.7910/DVN/BFGA VZ. URL https://doi.org/10.7910/DVN/BFGAVZ. 164 [89] JohnD.Lafferty,AndrewMcCallum,andFernandoC.N.Pereira. Conditionalrandom fields: Probabilistic models for segmenting and labeling sequence data. In Proceed- ings of the Eighteenth International Conference on Machine Learning, ICML ’01, page 282–289, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558607781. [90] David S Lee and Thomas Lemieux. Regression discontinuity designs in economics. Journal of economic literature, 48(2):281–355, 2010. [91] Jure Leskovec, Andreas Krause, Carlos Guestrin, Christos Faloutsos, Jeanne Van- Briesen, and Natalie Glance. Cost-effective outbreak detection in networks. In Pro- ceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 420–429, 2007. [92] Stephan Lewandowsky, John Cook, Philipp Schmid, Dawn Liu Holford, Adam Finn, Julie Leask, Angus Thomson, Doug Lombardi, Ahmed K Al-Rawi, Michelle A Amazeen, et al. The covid-19 vaccine communication handbook. a practical guide for improving vaccine communication and fighting misinformation, 2021. [93] Cheng Li, Jiaqi Ma, Xiaoxiao Guo, and Qiaozhu Mei. Deepcas: An end-to-end pre- dictor of information cascades. In Proceedings of the 26th international conference on World Wide Web, pages 577–586, 2017. [94] Chenliang Li, Haoran Wang, Zhiqian Zhang, Aixin Sun, and Zongyang Ma. Topic modeling for short texts with auxiliary word embeddings. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 165–174, 2016. [95] Shuang Li, Lu Wang, Ruizhi Zhang, Xiaofu Chang, Xuqin Liu, Yao Xie, Yuan Qi, and Le Song. Temporal logic point processes. In Hal Daum´ e III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 165 of Proceedings of Machine Learning Research, pages 5990–6000. PMLR, 2020. URL http://proceedings.mlr.press/v119/li20p.html. [96] Jiongqian Liang, Saket Gurukar, and Srinivasan Parthasarathy. Mile: A multi-level framework for scalable graph embedding. arXiv preprint arXiv:1802.09612, 2018. [97] Jiongqian Liang, Saket Gurukar, and Srinivasan Parthasarathy. Mile: A multi-level framework for scalable graph embedding. Proceedings of the International AAAI Conference on Web and Social Media, 15(1):361–372, May 2021. URL https: //ojs.aaai.org/index.php/ICWSM/article/view/18067. [98] Nut Limsopatham and Nigel Collier. Normalising medical concepts in social media texts by learning semantic representation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1014–1023, 2016. [99] Yahui Liu, Xiaolong Jin, Huawei Shen, and Xueqi Cheng. Do rumors diffuse differ- ently from non-rumors? a systematically empirical analysis in sina weibo for rumor identification. In Pacific-Asia Conference on Knowledge Discovery and Data Mining , pages 407–420. Springer, 2017. [100] Luca Luceri, Felipe Cardoso, and Silvia Giordano. Down the bot hole: action- able insights from a 1-year analysis of bots activity on twitter. arXiv preprint arXiv:2010.15820, 2020. [101] Luca Luceri, Silvia Giordano, and Emilio Ferrara. Detecting troll behavior via inverse reinforcement learning: A case study of russian trolls in the 2016 us election. In ICWSM, volume 14, pages 417–427, 2020. [102] Josephine Lukito. Coordinating a multi-platform disinformation campaign: Internet research agency activity on three us social media platforms, 2015 to 2017. Political Communication, 37(2):238–255, 2020. 166 [103] Jing Ma, Wei Gao, Zhongyu Wei, Yueming Lu, and Kam-Fai Wong. Detect rumors using time series of social context information on microblogging websites. In Pro- ceedings of the 24th ACM International on Conference on Information and Knowledge Management, pages 1751–1754. ACM, 2015. [104] Jing Ma, Wei Gao, Prasenjit Mitra, Sejeong Kwon, Bernard J Jansen, Kam-Fai Wong, andMeeyoungCha. Detectingrumorsfrommicroblogswithrecurrentneuralnetworks. In IJCAI, pages 3818–3824, 2016. [105] Jing Ma, Wei Gao, and Kam-Fai Wong. Detect rumors in microblog posts using propagation structure via kernel learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 708–717, 2017. [106] Jing Ma, Wei Gao, and Kam-Fai Wong. Detect rumor and stance jointly by neural multi-task learning. In Companion of the The Web Conference 2018 on The Web Conference 2018, pages 585–593. International World Wide Web Conferences Steering Committee, 2018. [107] Diego A Martin, Jacob N Shapiro, and Michelle Nedashkovskaya. Recent trends in online foreign influence efforts. Journal of Information Warfare, 2019. [108] Edouard Mathieu, Hannah Ritchie, Esteban Ortiz-Ospina, Max Roser, Joe Hasell, Cameron Appel, Charlie Giattino, and Lucas Rod´ es-Guirao. A global database of covid-19 vaccinations. Nature Human Behaviour, pages 1–7, 2021. [109] Hongyuan Mei and Jason M Eisner. The neural hawkes process: A neurally self- modulating multivariate point process. In NIPS, pages 6754–6764, 2017. [110] Shahan Ali Memon and Kathleen M Carley. Characterizing covid-19 misinformation communities using a novel twitter dataset. arXiv preprint arXiv:2008.00791, 2020. 167 [111] Panagiotis Metaxas and Samantha Finn. The infamous” pizzagate” conspiracy theory: Insights from a TwitterTrails investigation, 2017. URL http://cs.wellesley.edu / ~ pmetaxas/Research/The_infamous_Pizzagate_conspiracy_theory_Insight_fr om_a_TwitterTrails_investigation.pdf. [112] TomasMikolov,IlyaSutskever,KaiChen,GregCorrado,andJeffreyDean.Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems, 26:3111–3119, 2013. [113] MehrnooshMirtaheri,SamiAbu-El-Haija,FredMorstatter,GregVerSteeg,andAram Galstyan. Identifying and analyzing cryptocurrency manipulations in social media. IEEE Transactions on Computational Social Systems, 8(3):607–617, 2021. [114] Kunihiro Miyazaki, Takayuki Uchiba, Kenji Tanaka, and Kazutoshi Sasahara. The strategy behind anti-vaxxers’ reply behavior on social media. arXiv preprint arXiv:2105.10319, 2021. [115] Goran Muric, Yusong Wu, and Emilio Ferrara. Covid-19 vaccine hesitancy on social media: Building a public twitter dataset of anti-vaccine content, vaccine misinforma- tion and conspiracies. arXiv preprint arXiv:2105.05134, 2021. [116] Yurii Nesterov. Introductory lectures on convex programming volume i: Basic course. Lecture notes, 3(4):5, 1998. [117] Praneeth Netrapalli and Sujay Sanghavi. Learning the graph of epidemic cascades. ACM SIGMETRICS Performance Evaluation Review, 40(1):211–222, 2012. [118] Mark EJ Newman. Spread of epidemic disease on networks. Physical review E, 66(1): 016128, 2002. [119] Nam P Nguyen, Guanhua Yan, My T Thai, and Stephan Eidenbenz. Containment 168 of misinformation spread in online social networks. In Proceedings of the 4th Annual ACM Web Science Conference, pages 213–222. ACM, 2012. [120] Brendan Nyhan and Jason Reifler. When corrections fail: The persistence of political misperceptions. Political Behavior, 32(2):303–330, 2010. [121] Takahiro Omi, Kazuyuki Aihara, et al. Fully neural network based model for general temporal point processes. In NIPS, pages 2122–2132, 2019. [122] Rebecca R Ortiz, Andrea Smith, and Tamera Coyne-Beasley. A systematic literature review to examine the potential for social media to impact hpv vaccine uptake and awareness, knowledge, and attitudes about hpv and hpv vaccination. Human vaccines & immunotherapeutics, 15(7-8):1465–1475, 2019. [123] Myle Ott, Claire Cardie, and Jeffrey T Hancock. Negative deceptive opinion spam. In HLT-NAACL, pages 497–501, 2013. [124] Diogo Pacheco, Pik-Mai Hui, Christopher Torres-Lugo, Bao Tran Truong, Alessandro Flammini, and Filippo Menczer. Uncovering coordinated networks on social media. In ICWSM, 2021. [125] Antonis Papasavva, Jeremy Blackburn, Gianluca Stringhini, Savvas Zannettou, and Emiliano De Cristofaro. ” is it a qoincidence?”: A first step towards understanding and characterizing the qanon movement on voat. co. arXiv preprint arXiv:2009.04885, 2020. [126] James W Pennebaker. Linguistic inquiry and word count: Liwc 2001. Lawrence Erl- baum Associates, page 71, 2001. [127] Gordon Pennycook and David G Rand. Who falls for fake news? the roles of bullshit receptivity, overclaiming, familiarity, and analytic thinking. SSRN:3023545, 2018. doi: http://dx.doi.org/10.2139/ssrn.3023545. 169 [128] Andrew Perrin. Social media usage, 2015. URL https://www.pewinternet.org/20 15/10/08/social-networking-usage-2005-2015. [129] Shruti Phadke, Mattia Samory, and Tanushree Mitra. What makes people join con- spiracy communities? role of social factors in conspiracy engagement. ACM HCI, 4 (CSCW3):1–30, 2021. [130] FrancescoPierri,BreaPerry,MatthewRDeVerna,Kai-ChengYang,AlessandroFlam- mini, Filippo Menczer, and John Bryden. The impact of online misinformation on us covid-19 vaccinations. arXiv preprint arXiv:2104.10635, 2021. [131] Kashyap Popat, Subhabrata Mukherjee, Jannik Str¨ otgen, and Gerhard Weikum. Where the Truth Lies: Explaining the Credibility of Emerging Claims on the Web and Social Media. In Proceedings of the 26th International Conference on World Wide Web Companion - WWW ’17 Companion, pages 1003–1012, Perth, Australia, 2017. ACM Press. ISBN 978-1-4503-4914-7. doi: 10.1145/3041021.3055133. URL http://dl.acm.org/citation.cfm?doid=3041021.3055133. [132] Kashyap Popat, Subhabrata Mukherjee, Andrew Yates, and Gerhard Weikum. De- clare: Debunking fake news and false claims using evidence-aware deep learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Pro- cessing, pages 22–32, 2018. [133] V Jothi Prakash and Dr LM Nithya. A survey on semi-supervised learning techniques. arXiv preprint arXiv:1402.4645, 2014. [134] Feng Qian, Chengyue Gong, Karishma Sharma, and Yan Liu. Neural user response generator: Fake news detection with collective user intelligence. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18 , pages 3834–3840. International Joint Conferences on Artificial Intelligence Organiza- 170 tion, 7 2018. doi: 10.24963/ijcai.2018/533. URL https://doi.org/10.24963/ijcai .2018/533. [135] Meng Qu and Jian Tang. Probabilistic logic neural networks for reasoning. Advances in Neural Information Processing Systems, 32:7712–7722, 2019. [136] Meng Qu, Yoshua Bengio, and Jian Tang. Gmnn: Graph markov neural networks. arXiv: Learning, 2019. [137] ArneRoetsetal. ‘fakenews’: Incorrect,buthardtocorrect.theroleofcognitiveability ontheimpactoffalseinformationonsocialimpressions. Intelligence,65:107–110,2017. [138] Natali Ruchansky, Sungyong Seo, and Yan Liu. Csi: A hybrid deep model for fake news detection. In CIKM, pages 797–806. ACM, 2017. [139] Kazumi Saito, Ryohei Nakano, and Masahiro Kimura. Prediction of information dif- fusion probabilities for independent cascade model. In International Conference on Knowledge-Based and Intelligent Information and Engineering Systems, pages 67–75. Springer, 2008. [140] Mohsen Sayyadiharikandeh, Onur Varol, Kai-Cheng Yang, Alessandro Flammini, and Filippo Menczer. Detection of novel social bots by ensembles of specialized classifiers. InProceedingsofthe29thACMInternationalConferenceonInformation&Knowledge Management, pages 2725–2732, 2020. [141] Filipo Sharevski, Raniem Alsaadi, Peter Jachim, and Emma Pieroni. Misinformation warning labels: Twitter’s soft moderation effects on covid-19 vaccine belief echoes. arXiv preprint arXiv:2104.00779, 2021. [142] Karishma Sharma, Feng Qian, He Jiang, Natali Ruchansky, Ming Zhang, and Yan Liu. Combating fake news: A survey on identification and mitigation techniques. ACM TIST, 2019. 171 [143] Karishma Sharma, Sungyong Seo, Chuizheng Meng, Sirisha Rambhatla, and Yan Liu. Coronavirusonsocialmedia: Analyzingmisinformationintwitterconversations. arXiv preprint arXiv:2003.12309, 2020. [144] Karishma Sharma, Xinran He, Sungyong Seo, and Yan Liu. Network inference from a mixtureofdiffusionmodelsforfakenewsmitigation.In ProceedingsoftheInternational AAAI Conference on Web and Social Media, volume 15, 2021. [145] Karishma Sharma, Yizhou Zhang, , Emilio Ferrara, and Yan Liu. Identifying coor- dinated accounts on social media through hidden influence and group behaviours. In Proceedingsofthe27thACMSIGKDDInternationalConferenceonKnowledgeDiscov- ery & Data Mining, KDD ’21, New York, NY, USA, 2021. Association for Computing Machinery. doi: 10.1145/3447548.3467391. URL https://doi.org/10.1145/344754 8.3467391. [146] Karishma Sharma, Emilio Ferrara, and Yan Liu. Characterizing online engagement withdisinformationandconspiraciesinthe2020uspresidentialelection. InProceedings of the International AAAI Conference on Web and Social Media, volume 16, pages 908–919, 2022. [147] Karishma Sharma, Emilio Ferrara, and Yan Liu. Construction of large-scale mis- information labeled datasets from social media discourse using label refinement. In Proceedings of the ACM Web Conference 2022, pages 3755–3764, 2022. [148] Karishma Sharma, Yizhou Zhang, and Yan Liu. Covid-19 vaccine misinformation campaigns and social media narratives. In Proceedings of the International AAAI Conference on Web and Social Media, volume 16, pages 920–931, 2022. [149] Oleksandr Shchur, Marin Biloˇ s, and Stephan G¨ unnemann. Intensity-free learning of temporal point processes. In ICLR, 2020. 172 [150] Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. Fake news detection on social media: A data mining perspective. ACM SIGKDD Explorations Newsletter, 19(1):22–36, 2017. [151] Kai Shu, Deepak Mahudeswaran, Suhang Wang, Dongwon Lee, and Huan Liu. Fak- enewsnet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media. Big Data, 8(3):171–188, 2020. [152] Mirela Silva, Fabr´ ıcio Ceschin, Prakash Shrestha, Christopher Brant, Juliana Fernan- des, Catia S Silva, Andr´ e Gr´ egio, Daniela Oliveira, and Luiz Giovanini. Predicting misinformationandengagementincovid-19twitterdiscourseinthefirstmonthsofthe outbreak. arXiv preprint arXiv:2012.02164, 2020. [153] Aleksandr Simma and Michael I Jordan. Modeling events with cascades of poisson processes. In Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence, pages 546–555, 2010. [154] Lisa Singh, Shweta Bansal, Leticia Bode, Ceren Budak, Guangqing Chi, Kornraphop Kawintiranon, Colton Padden, Rebecca Vanarsdall, Emily Vraga, and Yanchen Wang. A first look at covid-19 information and misinformation sharing on twitter. arXiv preprint arXiv:2003.13907, 2020. [155] PhilipJSmith,SharonGHumiston,EdgarKMarcuse,ZhenZhao,ChristinaGDorell, Cynthia Howes, and Beth Hibbs. Parental delay or refusal of vaccine doses, childhood vaccination coverage at 24 months of age, and the health belief model. Public health reports, 126(2 suppl):135–146, 2011. [156] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output represen- tation using deep conditional generative models. In Advances in Neural Information Processing Systems, pages 3483–3491, 2015. 173 [157] James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Fever: a large-scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809–819, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1074. URL https://www.aclweb.org/anthology/N18-1074. [158] Christopher Torres-Lugo, Kai-Cheng Yang, and Filippo Menczer. The manufac- ture of political echo chambers by follow train abuse on twitter. arXiv preprint arXiv:2010.13691, 2020. [159] SebastianTschiatschek, AdishSingla, ManuelGomezRodriguez, ArpitMerchant, and Andreas Krause. Fake news detection in social networks via crowd signals. In Com- panion of the The Web Conference 2018 on The Web Conference 2018, pages517–524. International World Wide Web Conferences Steering Committee, 2018. [160] Joshua A Tucker, Andrew Guess, Pablo Barber´ a, Cristian Vaccari, Alexandra Siegel, Sergey Sanovich, Denis Stukal, and Brendan Nyhan. Political polarization, and polit- ical disinformation: a review of the scientific literature. Hewlett Foundation, 2018. [161] SanderVanderLinden. Theconspiracy-effect: Exposuretoconspiracytheories(about globalwarming)decreasespro-socialbehaviorandscienceacceptance. Personality and Individual Differences , 87:171–173, 2015. [162] Onur Varol, Emilio Ferrara, Clayton Davis, Filippo Menczer, and Alessandro Flam- mini. Online human-bot interactions: Detection, estimation, and characterization. In Proceedings of the International AAAI Conference on Web and Social Media, vol- ume 11, 2017. [163] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N 174 Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InNeurIPS, pages 5998–6008, 2017. [164] Alejandro Veen and Frederic P Schoenberg. Estimation of space–time branching pro- cess models in seismology using an em–type algorithm. Journal of the American Sta- tistical Association, 103(482):614–624, 2008. [165] Victor Veitch, Dhanya Sridhar, and David Blei. Adapting text embeddings for causal inference. In Conference on Uncertainty in Artificial Intelligence , pages 919–928. PMLR, 2020. [166] Svitlana Volkova and Jin Yea Jang. Misleading or falsification: Inferring deceptive strategies and types in online news and social media. In Companion of the The Web Conference 2018 on The Web Conference 2018, pages 575–583. International World Wide Web Conferences Steering Committee, 2018. [167] Soroush Vosoughi, Deb Roy, and Sinan Aral. The spread of true and false news online. Science, 359(6380):1146–1151, 2018. ISSN 0036-8075. doi: 10.1126/science.aap9559. URL https://science.sciencemag.org/content/359/6380/1146. [168] Gang Wang, Xinyi Zhang, Shiliang Tang, Haitao Zheng, and Ben Y Zhao. Unsuper- vised clickstream clustering for user behavior analysis. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pages 225–236, 2016. [169] Senzhang Wang, Xia Hu, Philip S Yu, and Zhoujun Li. Mmrate: inferring multi- aspect diffusion networks with multi-pattern cascades. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1246–1255. ACM, 2014. [170] William Yang Wang. “liar, liar pants on fire”: A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for 175 Computational Linguistics (Volume 2: Short Papers), pages 422–426. Association for Computational Linguistics, 2017. doi: 10.18653/v1/P17-2067. URL http://aclweb .org/anthology/P17-2067. [171] Claire Wardle. Fake news. It’s complicated, 2017. URL https://firstdraftnews.o rg/fake-news-complicated/. [172] Samuel C Woolley and Philip N Howard. Computational propaganda: political parties, politicians, and political manipulation on social media. Oxford University Press, 2018. [173] Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems, 33, 2020. [174] Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Self- attention with functional time representation learning. In NeurIPS, 2019. [175] Hongteng Xu. Poppy: A point process toolbox based on pytorch. arXiv preprint arXiv:1810.10122, 2018. [176] Shuang-Hong Yang and Hongyuan Zha. Mixture of mutually exciting processes for viral diffusion. In International Conference on Machine Learning, pages 1–9, 2013. [177] Xiaoxin Yin, Jiawei Han, and S Yu Philip. Truth discovery with multiple conflict- ing information providers on the web. IEEE Transactions on Knowledge and Data Engineering, 20(6):796–808, 2008. [178] YouGov. C4 study reveals only 4% surveyed can identify true or fake news. re- trieved. http://www.channel4.com/info/press/news/c4-study-reveals-only-4- surveyed- can-identify-true-or-fake-news, 2017. [179] Rose Yu, Xinran He, and Yan Liu. Glad: group anomaly detection in social media analysis. ACM Transactions on Knowledge Discovery from Data, 10(2), 2015. 176 [180] Sixie Yu, Yevgeniy Vorobeychik, and Scott Alfeld. Adversarial classification on social networks. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 211–219. International Foundation for Autonomous Agents and Multiagent Systems, 2018. [181] Savvas Zannettou, Michael Sirivianos, Jeremy Blackburn, and Nicolas Kourtellis. The web of false information: Rumors, fake news, hoaxes, clickbait, and various other shenanigans. arXiv preprint arXiv:1804.03461, 2018. [182] Savvas Zannettou, Tristan Caulfield, Emiliano De Cristofaro, Michael Sirivianos, Gi- anluca Stringhini, and Jeremy Blackburn. Disinformation warfare: Understanding state-sponsored trolls on twitter and their influence on the web. In Companion pro- ceedings of WebConf, pages 218–226, 2019. [183] Savvas Zannettou, Tristan Caulfield, William Setzer, Michael Sirivianos, Gianluca Stringhini, and Jeremy Blackburn. Who let the trolls out? towards understanding state-sponsored trolls. In ACM WebSci, pages 353–362, 2019. [184] Qiang Zhang, Aldo Lipani, Omer Kirnap, and Emine Yilmaz. Self-attentive hawkes process. In ICML, 2020. [185] Yizhou Zhang, Karishma Sharma, and Yan Liu. Vigdet: Knowledge informed neural temporal point process for coordination detection on social media. Advances in Neural Information Processing Systems, 34:3218–3231, 2021. [186] ZheZhao,PaulResnick,andQiaozhuMei. Enquiringminds: Earlydetectionofrumors insocialmediafromenquiryposts. InProceedings of the 24th International Conference on World Wide Web, pages 1395–1405. International World Wide Web Conferences Steering Committee, 2015. [187] Ke Zhou, Hongyuan Zha, and Le Song. Learning social infectivity in sparse low-rank networks using multi-dimensional hawkes processes. In AISTATS, 2013. 177 [188] Xinyi Zhou, Apurva Mulay, Emilio Ferrara, and Reza Zafarani. Recovery: A multimodal repository for covid-19 news credibility research. arXiv preprint arXiv:2006.05557, 2020. [189] Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propagation, 2002. [190] Xiaojin Zhu and Andrew B Goldberg. Introduction to semi-supervised learning. Syn- thesis lectures on artificial intelligence and machine learning , 3(1):1–130, 2009. [191] Melissa Zimdars. False, Misleading, Clickbait-Y, and Satirical ‘News’ Sources, 2016. URL https://21stcenturywire.com/wp-content/uploads/2017/02/2017-DR-ZIM DARS-False-Misleading-Clickbait-y-and-Satirical-%E2%80%9CNews%E2%80%9 D-Sources-Google-Docs.pdf. [192] Arkaitz Zubiaga, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Peter Tolmie. Analysing How People Orient to and Spread Rumours in Social Media by Looking at Conversational Threads. PLOS ONE, 11(3):e0150989, March 2016. ISSN 1932-6203. doi: 10.1371/journal.pone.0150989. URL https://journals.plos.org/ plosone/article?id=10.1371/journal.pone.0150989. [193] Arkaitz Zubiaga, Ahmet Aker, Kalina Bontcheva, Maria Liakata, and Rob Procter. Detection and resolution of rumours in social media: A survey. ACM Computing Surveys (CSUR), 51(2):32, 2018. [194] Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. Transformer hawkes process. In NeurIPS, 2020. 178 Appendix A Disinformation Labeling for Dataset and Analysis A.1 Annotation Guidelines A.1.1 Guidelines for Disinformation Labeling In Fig A.1, the instructions and guidelines specified for annotators is included. The anno- tators are asked to label in the context of when the tweet is posted, with examination of facts from high-factual, low bias news article sources, fact-checking resources, and official information sources. The annotators are provided tweets with the screen name, news source domain, news source label, full tweet text (including the news URL hyperlink), tweet times- tamp are provided to aid the annotator. The article URL provides context to the tweet content, and is needed at times to understand the tweet’s claim. Typical and trick examples with remarks in each category were provided to review and revisit while annotating, which is a useful guide to provide the distinctions between label types. A.1.2 Additional Analysis of COVID-19 Vaccine Dataset Communities Feature Distributions The feature distributions of accounts in the most significant com- munities (circled in Fig 4.4) are compared in Fig. A.4 and Fig. A.5. In Fig. A.4, we compare Followings (accounts followed by accounts in the community), Followers, Listed (accounts mentioned in topical lists created by other accounts on Twitter), Favourites (number of tweetsliked/favouritedbytheaccountsinthecommunity),Tweets(whichisthetotaltweets posted by the account in its lifetime). These features are available in the account metadata obtainedusingtheTwitterAPI,andweusetheaccount’sstatisticsatitslastobservedtweet in the dataset. The main insights about account characteristics in each community are, The interquartile range for distribution of followers and followings is significantly higher 179 Figure A.1: Labeling guidelines provided to annotators for weak label refinement Figure A.2: Labeling guidelines provided to annotators for weak label refinement. fortheFar-rightconspiracycommunity(C8)(Fig.A.4). Thedistributionacrossothergroups is similar, with the Anti-vaccine community (C1) having slightly lower upper quartile. This suggests more interconnected accounts in the Far-right conspiracy group, which likely ac- tively follow each other. More accounts appear in Twitter Lists (Listed) from Left (C0), Mainstream (C2), Right (C3), and Far-right communities, compared to Anti-vaccine misin- formation and Spanish-En conspiracy community (C14). Yet, the anti-vaccine community do have many Listed accounts, suggesting that such content does have a sizable audience. InFigA.4wealsohavetheTweets(totaltweetspostedbytheaccountinitslifetime). To 180 Figure A.3: Labeling guidelines provided to annotators for weak label refinement. (a) Anti-vax misinformation/ conspiracy (b) Far-right leaning conspira- cies (c) Spanish-En anti-vax con- spiracies (d) Left-Leaning (e) Mainstream News (f) Right-leaning Figure A.4: Account statistics boxplot for (mis)informed communities better understand the accounts activities, in Fig A.5, we additionally compare the Account Age (Days between first observed tweet in the dataset and the account creation date), Vax Tweets (tweets posted by the account specific to vaccine related content, quantified by observedtweetsoftheaccountinthecollecteddataset),VaxTweetEngagements(numberof vaccinetweetsthatreferencei.e. mention,reply,retweet,quotetheaccountinthecommunity or any of its vaccine tweets, quantified by observed tweets in the collected dataset). The Total Tweets distribution is lowest for Anti-vaccine misinformation and Spanish-En conspiracy communities. Account Age distribution is the smallest for these two communi- ties, i.e., more recently created accounts. Account Age for Left and Mainstream are highest (older accounts), and Right and Far-right are in between. In the Vax Tweets, however, 181 Anti-vax Far-right Spanish Left Mainstr. 0 2000 4000 6000 Account Age Right Anti-vax Far-right Spanish Left Mainstr. 0 500 1000 1500 2000 Vax Tweets Right Anti-vax Far-right Spanish Left Mainstr. 0 2000 4000 6000 Vax Tweets Engagements Right Figure A.5: Account statistics for communities (a) account age, (b) number of collected vaccine tweets (original, retweet, reply) by account, and Engagements (account retweeted, replied, mentioned by others) in collected vaccine tweets the Anti-vaccine community is the most vocal with higher Vax Tweet distribution than other communities; Far-right and Right being the least. Interestingly, Anti-vaccine misin- formation, Far-right conspiracy, and Spanish-En conspiracy communities have the largest interquartile ranges compared to the other communities on Vax Tweets Engagements. That means most accounts in these communities receive non-negligible engagements (mentions from other accounts, or replies, retweets, quotes of its vaccine tweets), in contrast with the othercommunities. Itsuggeststhatdiscussionsaremoredecentralizedinthesecommunities. News Sources and Demographic Analysis We investigated statistics of the unreliable and conspiracy URL tweets alongside collected statistics about U.S. vaccination rates in different states or demographics, and side-effects reported in the CDC VAERS records. U.S.StatesVaccineUptakeInFig.A.6a,wecomparetheratioofmisinformation(un- reliable/conspiracy URLs) to reliable URLs observed in collected tweets for accounts with geolocation available extracted using [31]. The public records of how many people have been vaccinated in each state available through CDC is curated and maintained by the research community [108] (accessed June 6, 2021). The vaccine uptake i.e., percentage of vaccinated individuals as of June 6, 2021 per U.S. state is plotted against the rate of misinformation (ratio of unreliable/conspiracy URL tweets to reliable URL tweets) in Fig A.6a. The states political affiliation is designated based on the 2020 Election votes (Red states voted for Don- ald Trump and Blue States for Joe Biden in the 2020 Presidential Election). The analysis is state-wise, therefore, only account tweets with valid extracted geolocations of US States 182 0.1 0.2 0.3 0.4 0.5 0.6 Ratio: Unreliable/Conspiracy URLs to Reliable URLs 35 40 45 50 55 60 65 70 People Vaccinated per 100 California District of Columbia Florida Georgia Illinois Maryland Michigan Minnesota New York Oregon Texas Virginia Arizona Colorado North Carolina Ohio Massachusetts Washington Pennsylvania New Jersey Biden wins (2020) Trump wins (2020) (a) % Vaccinated individuals in each U.S. states vs. ratio of unreliable/conspiracy to reliable URL tweets (Pearson coefficient: - 0.731, p-val: 7.5e-11) extracted with geolo- cation over Red and Blue states in 2020 Elections(top-20statesbytweetvolumeare shown with state label) 0 10000 20000 30000 40000 50000 sputniknews.com swarajyamag.com gnews.org newsmax.com globalresearch.ca aubedigitale.com dailywire.com thenationalpulse.com infowars.com davidharrisjr.com vaccineimpact.com thehighwire.com reseauinternational.net fr.sputniknews.com westernjournal.com brighteon.com politicususa.com activistpost.com francesoir.fr dailycaller.com americasfrontlinedoctors.com humansarefree.com oann.com nationalfile.com greatgameindia.com breitbart.com thegatewaypundit.com theepochtimes.com truepundit.com express.co.uk zerohedge.com rt.com lifesitenews.com childrenshealthdefense.org dailymail.co.uk Extreme Pseudoscience & Conspiracy Promotion of Pseudoscience & Conspiracy Extreme Left/Right Propaganda Left/Right leaning Content Pending Investigation / Unknown Factual Reporting: MIXED Factual reporting: LOW Factual reporting: VERY LOW (b) Tweet volume for top-35 news sources are utilized in the plot. The estimated correlation results are that Pearson’s correlation co- efficient is -0.731 between vaccine uptake and misinformation rate. This confirms a negative correlation of % individuals vaccinated and the rate of online misinformation in states. News Sources Fig. A.6b presents categorization of the top news domains in unreli- able/conspiracy URLs tweets. The categorization is based on the Media Bias/Fact Check ratings of (i) degree of factual reporting (ii) political bias, and (iii) scientific reporting mea- sures. The factual reporting level from these sources is regarded as either Low, very Low, or Mixed on Media Bias/Fact Check. The top domains contain sources promoting both ex- treme pseudoscience and conspiracy (e.g. ChildrensHealthDefense.org), Left/Right political propaganda (e.g. dailymail.co.uk, rt.com, truepundit.com). Topic Modeling We use topic modeling on unreliable/conspiracy URLs tweets. The text is pre-processed by tokenization, punctuation removal, stop-word removal, and removal of URLs, hashtags, mentions, and special characters, and represented using pre-trained fastText word embeddings [12] 1 . The average of word embeddings in the tweet text is used to represent each tweet. Pre-trained embeddings trained on large English corpora en- 1 Pre-trained embeddings can be downloaded from https://fasttext.cc/docs/en/english-vectors.html 183 code more semantic information useful for short texts where word co-occurrence statistics are limited for utilizing traditional probabilistic topic models [94]. The text representations are clustered (k-means) to identify topics. Number of clusters is selected using silhouette and Davies-Bouldin measures of cluster separability between 3-35. Inspecting the word dis- tribution and top representative tweets, we label and merge over-partitioned clusters. The misinformation largely targeted seven forms of information manipulation about the COVID-19 vaccines, namely, manipulation of Scientific facts about the COVID-19 vaccines, misleading information about Side Effects, Deaths, Vaccine Effectiveness, and Vaccine Re- fusal,alongwithmisinformationrelatedtoVaccineRollout,andDehumanization,Depopula- tion, Great Reset, Bill Gates, or Pharmageddon conspiracies. Table. A.1 provides examples of top representative tweets and word distributions for each identified topic. Examination of thetweetssuggestspresenceofthefivetechniquesofmanipulationFLICC[92](fakeexperts, logicalfallacies,impossibleexpectations,cherry-picking,andconspiracytheoriescorrespond- ing to science denial). In part, for vaccine safety and effectiveness, not only are out-wright false claims about scientific facts and side-effects are observed, but also true reported side- effects were discussed with negative strong anti-vaccine sentiment or missing or misleading contexts, including misleading expectations about vaccine effectiveness by suggesting that since vaccines cannot prevent the infection, then its ineffective or not useful, as presented. Social Media Discussion of Vaccination Effects We examine vaccine side-effects discussed on social media in misinformation narratives. We study whether the discussion of vaccine side-effects on social media differs from the CDC VAERS (accessed June 10, 2021) recorded side-effects obtained from healthcare providers and public reports. We explore the correlation between frequency of the discussed side-effects on social media and their frequency in the VAERS records. To measure the frequency on Twitter, we first extract the medicalconceptsfromthetweetsviatextmatchingbasedonalargemedicalconceptcorpus: AskAPatient [98]. This corpus provides us with common medical concepts on social media in different forms, such as abbreviations, misspelling. We use the number of tweets in which 184 0 10000 20000 30000 40000 Frequency in VAERS records 0 100000 200000 300000 400000 Frequency in Tweets illness allergic reaction worried pain common cold infection malignant neoplastic disease influenza asthenia nausea chill fear fever tired arthralgia disability pressure headache fatigue dizziness myalgia rash erythema itching of skin (a) All tweets in dataset (Pearson coeff. - 0.250, p-val 0.239). Tweet frequency vs. VAERSrecordsfrequencyforreportedmed- ical concepts post vaccination 0 10000 20000 30000 40000 Frequency in VAERS records 0 1000 2000 3000 4000 5000 Frequency in Tweets asthenia influenza infection allergic reaction common cold illness malignant neoplastic disease swelling fear paralysis pain headache abdominal pain migraine facial swelling slurred speech allergic condition facial palsy nausea rash fatigue dizziness chill arthralgia (b) Unreliable/conspiracy URLs tweets (Pearson coeff. -0.358, p-val 0.079). Tweet vs. VAERS record frequency for reported medical concepts post vaccination Figure A.7: Frequency correlation of side-effects discussed on Twitter compared with that recorded in VAERS a medical concept appears to represent its frequency on Twitter. To measure the frequency in VAERS records, we conduct medical concept extraction in the same way and count the number of individuals whose medical records mention a concept. In Fig. A.7, we plot the medical concept frequencies in all collected tweets (Fig. A.7a) and in unreliable/conspiracy URLs tweets (Fig. A.7b) against corresponding frequencies in VAERS records. From the visualization, we can see that the concepts that are widely dis- cussed on social media are the ones that are rarer in the medical records. More frequent effects like pain, fever, headache are less frequent in tweets. In misinformation URL tweets, rarer reported concepts such as “paralysis”, “allergic reaction”, “malignant neoplastic dis- ease” (cancer) are more frequently referenced. The biased discussion of vaccine side-effects or adverse effects falls under cherry-picking, one of the five science denial techniques. This nuanced distortion of the true facts for misleading narratives is especially challenging in COVID-19 vaccine misinformation and harder to mitigate. 185 A.2 U.S. 2020 Election Dataset Analysis A.2.1 CSI Validation Tweets Human validation of inferred CSI [138] ensemble model label provided in multiple tables. 186 Table A.1: Misinformation topic clusters word distribution with highest TF-IDF scores Top Words (by TF-IDF scores) Representative Tweets (nearest distance to cluster centroid) 1 Scientific facts: mrna, pfizer, cells, moderna, human, via, system, new, operating, evidence, experimen- tal, pathogenic, trial, priming, dna, virus,alarming,adults,analysis,study, hiv, correlation, older, fetal, flu “#Nuremberg #CrimesAgainstHumanity #Plandemic #CovidHOAX #GreatReset #Event201 #Agenda2030 #NoMasks #NoForcedVac- cines #Pharmageddon Mainstream science admits COVID-19 vaccines contain mRNA “nanoparticles” that trigger severe allergic reactions https://t.co/KAYqrHRK8Y” 2SideEffects: allergic, reaction, pfizer, fda, severe, adverse, serious, worker, health, healthcare, hospitalized, ef- fects, moderna, news, workers, side, threatening, intubated, rate, doctor, people, boston, alaska, facial, higher, shot, suffering, paralysis, uk “Cardiothoracic surgeon warns FDA, Pfizer on immunological danger of COVID vaccines in recently convalescent and asymptomatic carriers — Opin- ion — LifeSite https://t.co/yPn3rSYxjy” “SAFE AND EFFECTIVE! IS IT? TIME TO FACE THE TRUTH! FDA In- vestigates Allergic Reactions to Pfizer COVID Vaccine After More Healthcare Workers Hospitalized • Children’s Health Defense https://t.co/HHfv3sbfGF” 3 Effectiveness: news, pfizer, dr, says, world, via, fauci, bill, gates, kennedy, new, jr, robert, moderna, take, people, get, doctors, uk, coronavirus, rollout, desantis, video, taking, trial, pharma, african, us, medical, lifesite “Can this this be? Then Its Not a Vaccine: Crazy Dr. Fauci Said in October Early COVID Vaccines Will Only Prevent Symptoms and NOT Block the Infection What? #FREESPEECH #WALKA- WAY#DEPLORABLE#DrainTheSwamp#FakeNews#Trump2020#Israel #StopTheSteal https://t.co/4l9yGMIKM6” 4 Deaths: pfizer, dies, receiving, home, nurse, nursing, days, getting, health, doctor, portuguese, worker, die, shot, taking,residents,first,old,two,miami, died, weeks, healthy “46 Nursing Home Residents in Spain Die Within 1 Month of Getting Pfizer COVID Vaccine! @ScottMorrisonMP @GregHuntMP Are you still proceeding with the #Pfizer #vaccine rollout which could kill older people? #BREAK- ING #BreakingNews #auspol #COVID19 https://t.co/32CppATPpz” 5 Vaccine Refusal: workers, pfizer, health, refuse, emergency, coronavirus, people, care, refusing, get, getting, use, hundreds, take, fda, staff, us, uk, room, says, passports, receiving, hospi- tal, news, line, healthcare, one, report, doses “Asmanyas60%ofhealthcareworkersarerefusingtogetthe#Covidvaccine. There’s an overwhelming lack of trust. Why? https://t.co/5OCa5X5AHJ #COVID19 #vaccine #CovidVaccine” “Start a DEMONSTRATION AND BOYCOTT CAMPAIGN against the VACCINE TYRANTS! COVID vaccines disaster of Adverse reports to CDC....look it up! NYC Waitress Fired For Refusing COVID Vaccine Over Fertility Concerns https://t.co/4vKDS8QjHw” 6 Rollout: trump, biden, joe, uns, fetus, praises, gavi, playing, owns, million, funding, male, funded, gave, gates, stopped, billion, us, plan, ad- min, distribution, rollout, administra- tion, google, jill, president “Joe Biden Struggles to Read Teleprompter as He Trashes Trump Adminis- tration’s Covid-19 Vaccine Distribution Efforts https://t.co/C1YOoU7XXH @gatewaypundit” “We all knew all along that Trump would botch the initial vaccine roll- out, and that vaccinations wouldn’t properly get underway until Biden takes office. Just one more thing for Trump to screw up on his way down. https://t.co/V9jMVdcx04” 7 Dehumanization: takedowntheccp, yanlimeng, wipe, drlimengyan, weapon, bio, white, ccpvirus, war, made, plan, part, ccp, world “@Newsweek World War V5.0: Covid virus is a bio weapon made by CCP. Vaccine is a part of plan to wipe out all white people. @DrLiMengYAN1 #DrLiMengYan1 #YanLiMeng #CCPVirus #Covid19 #TakeDown- TheCCP #COVID19 https://t.co/i6P9uTe11F. https://t.co/4PYB8dxe2v https://t.co/j3oKSTlpa1” 187 Table A.2: CSI label and human validation label with unreliable (1), and reliable (0) CSI label Tweet Validation Label 1 “Silence is Violence.” That’s what we have heard from the democrat party. Well your silence on the rioting and de- struction is violent as hell, @JoeBiden, @BarackObama, @HillaryClinton, @SenSchumer, @SpeakerPelosi, @NYGov- Cuomo 1 (Source: Reuters Fact- check) 0 It took the US 98 days to reach 1 million COVID-19 cases. Then 43 days after that to reach 2 million cases Then 28 days after that to reach 3 million cases. This is not a good trend, @realDonaldTrump 0 1 I don’t think for ONE minute that @JoeBiden watches these riots ; lootings on TV. “Joe Biden clearly rattled af- ter MONTHS of hiding in his basement; failing to stand up to the radical leftist MOBS taking over his campaign.” AND whose taken over his running mate #KamalaHarris https://t.co/SVtfopXtlN 1 (Source: Reuters fact- check) 0 Democracy as we know it is on the line. Now is the time to raiseyourvoices—tonight, tomorrow, thenextday, andevery day after that. 0 1 I bet @donlemon is a closet @realDonaldTrump supporter. I’d be willing to bet that he even VOTED for Trump 1(Suspendedac- count) 1 #TakeTheOath #Q #Qanon #QArmyJapanFlynn #WWG1WGA @okabaeri9111 @intheMatrixxx @pray- ingmedic @realDonaldTrump @GenFlynn #Japan 1(Suspendedac- counts, conspir- acy) 0 TheTrumpAdmin’sthreattodeportinternationalstudentsif theiruniversitiesmovetoonlinelearninghasnopublichealth justification. It’s just cruel. I joined @SenWarren, @Sen- Markey, @RepPressley and others in urging ICE ; DHS to withdraw it. https://t.co/zVNzGqagyk 0 1 RT @SPICYMOOSEBALLS: @MilitaryStart4 @banana- phone001 @HeARTofGod99 @GODISBACK5 @LLDMim @toddhassinger11 @RarnToGoHome @Dingleberries71 @sam... 1 (Suspended account, follow train) 1 RT @RaeAnon: @OrtaineDevian @Maca691 @Corp125Vet @mclanelfn @ZacharyMcbrien @NannyMcTrump @OBXRe- alty @BreyndaMcoy @MissesJ3 @MEMcCaffrey1 @... 1 (Suspended account, follow train) 1 BREAKING Real News LEADER? of #COVID #coro- navirus task force @VP Incompetent as 100’s of #USA Real Doctors Warn “Do NOT wear a Mask” hurts our im- mune system. Our Great @realDonaldTrump putU inCharge toLead. Uget F- Help @IngrahamAngle @DrKellyVictory https://t.co/fw9FLPwxqZ 1 (mask hurts immune system) 1 More than one-sixth of the mail-in ballots sent to voters in Nevada’s largest county during the 2020 primary went to outdated addresses, according to a new watchdog report. https://t.co/wfq6Nq52C2 1 (unreliable: voting fraud) 1 @realDonaldTrump is 100% correct on Mail In Voting ! CBS Shows How Mail In Votes would work; Ends up Being An Epic Disaster ! @RudyGiuliani @Potus @Lrihendry @madis- ongesiotto @ChanelRion https://t.co/wO7bJi3S9U 1 (voter fraud) 188 Table A.3: Model label and human validation label with unreliable (1), and reliable (0) CSI label Tweet Validation Label 0 In the 2016 election, there were 57.2 MILLION votes cast by mail. Trump raised no objections. Instead, he falsely claimed that 3 mil- lion undocumented immigrants voted. Then Trump himself voted by mail in subsequent elections. https://t.co/yCNcGES3Vi 0 0 Wisconsin farmers: Are Trump tariffs helping? Milwaukee: With unemployment at 8.4%, is the economy working for you? Wiscon- sin: Are 843 new Covid cases yesterday worrying you? Trump’s plans don’t work. Time for @JoeBiden #DemCastWI #ONEV1 https://t.co/YXqZyNNKP3 0 1 Voting starts earlier than you might realize. It has Trump bat- tling the clock in some battleground states Guess who is first? NorthCarolinabeginssendingmailballotstovoterswhorequested them on Sept. 4th. Georgia will do the same on Sept. 15th. https://t.co/dWO7S8E1lZ 1(Suspendedac- count) 1 BOOM!! THANK YOU PRESIDENT TRUMP FOR PUTTING AN END TO MORE OBAMA-BIDEN REGULATIONS THAT HURT THE AMERICAN WORKERS! Obama-Biden shipped mil- lions of American jobs overseas-millions of our jobs! @realDon- aldTrump https://t.co/OMl3Qb4vjR 1 1 Just a reminder that @JoeBiden served in the Senate for 37 years and V.P. for 8 years ; never once lifted a finger to help black Amer- icans. Not once did he pass a bill to help black Americans. Trump has done more for black Americans in 3 years than every other president combined 1 1 Thanks to @realDonaldTrump leadership. Tel Aviv on the left, Dubai on the right. Mazal tov and Saalam Aleikum #StandWith- Israel https://t.co/qjPVBCsa21 1(Suspendedac- count) 1 Right now the GOP is hoping; wishing BLM goes away while they urge .@realDonaldTrump to wear a face mask ; fall 4 the Dems latest Russian hoax. Can America please stand up ; support a who stands by @POTUS rough thick thin. Plz to https://t.co/iGp2WFfHqI https://t.co/ANBMhN3307 1(Russianhoax) 0 In the first few minutes of his first RNC speech, Trump falsely claimed Obama and Biden personally spied on his campaign, that it’s not possible to tabulate a lot of mail ballots, and that women had been at a record unemployment rate pre-pandemic (close but no). 0 0 Donald trump is LITERALLY telling his supporters to break the law and VOTE TWICE – by mail-in ballot AND in person. He is destroying our democracy. 0 (Source: Reuters, BBC) 0 On Thursday, July 23 at 4:30pm ET, @SenKamalaHarris will hold a North Carolina Biden for President roundtable to discuss Joe Biden’s “Build Back Better” plan to build a robust 21st century caregiving and education workforce. https://t.co/CeZdXxVI5G https://t.co/MG2DCBWtTN 0 1 What’s all the fuss about the Post Office? Like many government systems, it’s the slowest and least efficient option, which is why we have private options like UPS and FedEx. Nobody gave a damn about the Post Office until Democrats wanted mailmen to become ballot collectors. 0 1 “#Obama,#JoeBiden,; theirtopintelofficershuddledintheOval Office shortly before @realDonaldTrump was inaugurated to dis- cuss what they would do about this new President they despised,” @TomFitton in Breitbart. Read:https://t.co/21EYhZhIp8 #Oba- magate 1 (Conspiracy) 1 RT @kimallthat: @KohanimKnew @mizdonna @LisaMei62 @Ma- gaKarma1 @JupiterWalls @TruthIsLight5 @realDonaldTrump @RonColeman @RoscoeBDavis1 @Daw... 1 (Suspended account, follow train) 189 Table A.4: Model label and human validation label with unreliable (1), and reliable (0) CSI label Tweet Validation Label 0 Kanye West will not get a spot on the November ballot in his home state of Illinois after the Board of Elections ruled he didn’t gather enough qualified signatures. https://t.co/bkAcxCMxvC 0 1 This Is The @realDonaldTrump Haters Vision Of #America. Under No Circumstances Can You Convince Freedom Loving Americans Anything Close To This Behavior Is Acceptable. #BidenHarris2020 And The DC Establishment Pander To These Groups For Votes. November 3rd We #DrainTheSwamp https://t.co/o66fjdnEri 1(Suspendedac- count, conspir- acy) 1 @baalter @realDonaldTrump He will be truly missed! R.I.P. patriot! https://t.co/QU1eUkerIu 1(Suspendedac- count) 1 Now I got over 8,000! Don’t worry ‘bout it Sweetheart!. 24 days and a waketilltheRunOff-I’msotired-Iwakeupeverymorningknowingthe sacrifice is worth it. We MUST elect strong men of courage and humility to lead and help our GREAT POTUS @realDonaldTrump 1 (Tweet re- moved, unreli- able or conspir- acy group) 1 #NeverForget 9-11 NEVER FORGET @TheDemocrats are defunding the same #HEROS that risk their lives 2 save ours REMEMBER THIS VIDEO WHEN #VOTING @realDonaldTrump @TeamTrump @heytootssweet #MAGAMusician host of @PatriotsInTune w/ @Jew- elsJones1 https://t.co/NdTEwrmRwx 1 (Defunding: false) 0 With the jobs report set to come out at week’s end, @JoeBiden will deliver remarks Friday in Wilmington, DE “on the economic crisis that has been worsened by Trump’s failure to get the virus under control,” his campaign says. 0 1 @JoeBidenSelectionof@KamalaHarrisISAHugeStepBackForWomen https://t.co/YBBFVO08ru 1 (Unreliable) 0 RT @MotherEarthHH: @JoeBiden 0 0 NEW VIDEO: @realDonaldTrump’s botched response to the pan- demic tanked the economy and his continued failure to get #coro- navirus under control is preventing any economic recovery. Bot- tom line: Trump made America’s Greatest Depression. #COVID19 https://t.co/CvPO7YkKOM 0 0 @robertjdenault @jwgop @joebiden A Trump Goon? Figures. Hope he is the first of many of Trump’s administration to go to trial. 0 1 President@realDonaldTrumphasbeenTOUGHonChinatoputAMER- ICANS first!!! @PeterNavarro45 https://t.co/UJFPJtL6HI 0 1 VENEZUELA SECUESTRADA POR CRIMINALES SOCIALISTAS @realDonaldTrump #YoNoTeApoyoGuaido #RatasDelG4 #MUDyP- SUVsocialistas @ImBaack2 https://t.co/2M4eYX9AXY 1 (Unreliable: foreign interfer- ence) 1 ThepeacedealbetweenIsraelandtheUAEmarksahistoricsteptowards tremendous improvement in the region. Thank you @realDonaldTrump and @jaredkushner for your work in securing peace and fighting the in- fluences from Iran and Hezbollah. 1 (Tweet re- moved, unreli- able) 0 ”The words of a president matter – no matter how incompetent that president is,” says @joebiden on Joy Reid’s MSNBC show. 0 1 IfcompanybuildingisonFire,willthecompanysaveemployeesorassets inside the building . It seems current administration is only worried about assets , human beings don’t hold any value 144k died in Pandemic @TheDemocrats @JoeBiden @KamalaHarris @SpeakerPelosi 1(Suspendedac- count) 0 Wow. The @ProjectLincoln folks already have an ad slam- ming @realDonaldTrump over the “Russian bounties” matter. https://t.co/U1oUveMndH 0 1 Entire thing was offensive: @SenatorTimScott urges @realDonaldTrump to take down retweet of ’white power’ video https://t.co/HSkrZAuU92 0 0 TwooftheprogressivecandidatesIendorsedwillhavetheirprimarieson Tuesday! Here’s my interview with @JENFL23 who is running against Debbie Wasserman Schultz, a perfect symbol for the old guard. Jen has done a great job and I wish her well. https://t.co/HdvlcQVor8 0 190 Appendix B Supplementary Analysis of U.S. 2020 Election Dataset B.1 Regression Statistics In Table B.1, goodness-of-fit tests are provided for the RDD regression function (illustrated examples in Fig 8.5 in the main part). When comparing different degree polynomials as chosen functions for RDD, we found AIC and BIC were lowest for degree-1 (we measured up to degree-4, since the results are similar we show only degree-1 and degree-2 in the table). Degree-1 is the model used in the RDD analysis y =mx+b+β I(x>x 0 ) [90]. Adjusted R 2 , f-statisticandp-valuesofthecoefficientsalsoindicatedegree-1ispreferred, sincethep-value of coefficient m 2 for x 2 term is not significant at 0.05. The results were also consistent when separate coefficients of dependent variables were used on either side of the intervention. B.2 List of Right-Leaning and Left-Leaning Hashtags Right-Leaning (88 categorized from top-3000) trump2020landslidevictory, trump- landslide, bestpresidentever, armyfortrump, gaysfortrump, trumpwon2020, trumpforever, lovemypresident, trump2020, kag, kag2020, maga2020, kaga, keepamericagreat, makeam- ericagreatagain, maga2024, magaforever, kaga2020, kag2024, prouddeplorable, democrat- shateamerica, democratsaredestroyingamerica, rednationrising, redwave2020, votered, red- wave,voteredtosaveamerica,keeptexasred,redwaverising,buildthewall,buildthatwall,buildthe- wallnow, resistbiden, draintheswamp, walkaway, walkedaway, walkawayfromdemocrats, for- merdemocrat, redpilled, bidenisnotmypresident, joebidenisnotmypresident, bidennotmypres- ident,notmypresidentbiden,illegitimatepresident,bidencheated2020,bidencheated,impeach- biden, impeach46, impeachbidennow, trumplandslide2020, trumptrain, trump2024, trump- won, trump2020landslide, istandwithtrump, trumpsupporter, trumppence2020, veterans- 191 Table B.1: Regression stats. and goodness of fit for different degree polynomials. Note: *(p-value <= 0.05). AIC, BIC were lower for degree-1 compared to higher degree curves Hashtag R 2 R 2 adj F-stat AIC BIC β m m 2 wwg1wga 0.72 0.71 93.64 ∗ 229.71 236.7 -1.51 ∗ -0.05 ∗ 0.72 0.71 62.87 ∗ 230.57 239.89 -1.5 ∗ -0.11 0.06 qanon 0.68 0.67 78.22 ∗ 172.67 179.66 -0.90 ∗ -0.03 ∗ 0.68 0.67 51.81 ∗ 174.29 183.61 -0.89 ∗ -0.06 0.03 kag 0.51 0.49 37.37 ∗ 1.68 8.67 -0.4q ∗ -0.0 0.51 0.49 24.6 ∗ 3.64 12.96 -0.4 ∗ -0.0 0.0 q 0.67 0.67 75.77 ∗ -90.17 -83.17 -0.23 ∗ -0.0 ∗ 0.68 0.66 50.24 ∗ -88.6 -79.27 -0.22 ∗ -0.01 0.0 qarmy 0.58 0.57 50.28 ∗ -4.88 2.11 -0.21 ∗ -0.01 ∗ 0.58 0.56 33.16 ∗ -3.01 6.31 -0.21 -0.01 0.0 qanons 0.58 0.56 49.57 ∗ -119.35 -112.36 -0.11 ∗ -0.0 ∗ 0.58 0.57 33.73 ∗ -118.86 -109.54 -0.11 ∗ -0.01 0.01 patriotstriketeam 0.45 0.44 30.39 ∗ -183.19 -176.2 -0.09 ∗ -0.0 0.45 0.43 19.98 ∗ -181.19 -171.87 -0.08 ∗ -0.0 0.0 deepstate 0.76 0.75 113.22 ∗ -284.36 -277.36 -0.08 ∗ -0.0 ∗ 0.76 0.75 74.52 ∗ -282.41 -273.08 -0.08 ∗ -0.0 -0.0 walkawayfromdemocrats 0.36 0.34 20.67 ∗ -174.36 -167.37 0.14 ∗ -0.0 0.36 0.34 13.71 ∗ -172.61 -163.28 0.14 ∗ -0.0 0.0 vote 0.19 0.17 8.79 ∗ -229.96 -222.96 0.07 ∗ -0.0 0.2 0.17 6.07 ∗ -228.7 -219.38 0.07 ∗ -0.0 0.0 warroompandemic 0.11 0.08 4.3 ∗ -229.15 -222.15 0.06 ∗ -0.0 0.11 0.07 2.84 ∗ -227.19 -217.86 0.07 ∗ -0.0 -0.0 th3d3n 0.15 0.13 6.41 ∗ -259.06 -252.07 0.06 ∗ -0.0 ∗ 0.18 0.14 5.22 ∗ -259.73 -250.41 0.06 ∗ -0.0 0.0 hcqworksfauciknewin2005 0.38 0.36 22.45 ∗ -380.48 -373.49 0.06 ∗ -0.0 ∗ 0.38 0.36 15.0 ∗ -378.94 -369.62 0.06 ∗ -0.0 0.0 nomailinvoting 0.41 0.4 25.48 ∗ -351.56 -344.57 0.05 ∗ -0.0 0.41 0.39 16.78 ∗ -349.6 -340.28 0.05 ∗ -0.0 0.0 mo03 0.06 0.04 2.42 -230.76 -223.77 0.05 ∗ -0.0 0.09 0.05 2.28 -230.79 -221.46 0.05 ∗ -0.01 0.0 trump2020victory 0.18 0.15 7.78 ∗ -303.01 -296.01 0.05 ∗ -0.0 0.18 0.14 5.23 ∗ -301.3 -291.98 0.05 ∗ -0.0 0.0 hermancain 0.12 0.09 4.91 ∗ -270.0 -263.01 0.06 ∗ -0.0 ∗ 0.12 0.09 3.42 ∗ -268.52 -259.2 0.06 ∗ 0.0 -0.0 bigpharma 0.31 0.29 16.61 ∗ -371.12 -364.12 0.05 ∗ -0.0 ∗ 0.31 0.28 10.94 ∗ -369.17 -359.85 0.05 ∗ -0.0 -0.0 bidenisapedo 0.09 0.07 3.71 ∗ -265.18 -258.19 0.05 ∗ -0.0 0.11 0.08 3.06 ∗ -264.93 -255.61 0.05 ∗ -0.0 0.0 hcqzinc4prevention 0.24 0.22 11.58 ∗ -358.27 -351.28 0.05 ∗ -0.0 ∗ 0.24 0.21 7.62 ∗ -356.27 -346.95 0.05 ∗ -0.0 0.0 fortrump,trumparmy,blackvoicesfortrump,trumppence,womenfortrump,trumpsarmy,trump- 2020nowmorethanever,hispanicsfortrump,latinasfortrump,trumpismypresident,trump2016, teamtrump, latinosfortrump, protrump, fightfortrump, trumptrain2020, trump2021, blacks- fortrump, trumpgirl, lovetrump, trumpnowmorethanever, trump4eva, trusttrump, bestpres- identever45, trump4ever, standwithtrump, women4trump, votetrump, fourmoreyears, al- waystrump, istandwithpresidenttrump Left-Leaning (217 categorized from top-3000) votejoe, gojoe2020, americastronger- withbiden, khiveforbiden, bidenisyourpresident, votebidenharris2020, votebidenharris, vi- 192 cepresidentharris, kamalaharris, bidenharriswon, bidenharris2020landslide, bidenismypresi- dent,bidenharris2024,votebiden,bidenharristosaveamerica,teambidenharris,votebiden2020, bidenharrislandslide2020, vpkamalaharris, bidenkamala2020, veteransforbiden, kamala202x, kamala2020,bidenforpresident,joeandkamala,mvpharris,vetsforbiden,vicepresidentkamala- harris, madamvicepresident, vets4biden, teampeteforjoe, petetobiden, presidentbidenvphar- ris, ridinwithbiden2020, womenforbiden, presidentjoebiden, votejoebiden, joebidenismypres- ident, presidentelectjoebiden, madamvicepresidentharris, bidenharrislandslide, khive4ever, khiveforever, khiveforjoe, bidenandharris, ridinwithbiden, bidenharris, biden2020, bidenhar- ris2020, presidentbiden, joebiden2020, dogs4biden, ridenwithbiden, gojoe, vpharris, biden- won,biden2024,teambiden,bidenharris2021,joebidenkamalaharris2020,joe2020,byedon2020, byedon,bidencalm,presidentelectbiden,bluetsunami2018,voteblue2020,bluewave2020,blue- wave, voteblue, votebluenomatterwho, votebluetosaveamerica, voteblue2022, 2020bluewave, bluewave2022,bluetsunami,thebluewave,bluenomatterwho,votebluenomatterwho2020,blue- wave2018,georgiablue,votedblue,blueinaredstate,votebluetosaveamerica2020,bluetsunami2022, votebluein22,blue2020,votebluedownballot,voteblue22,wtpblue,wtp2020,bluetsunami2020, votebluetoendthisnightmare, staybluein2022, flipitblue, turnfloridablue, flipthesenateblue, flipthesenate,votebluein2022,votebluealways,votebluetosavedemocracy,anyonebuttrump2020, worstpresidentever,votefordout2022,votetrumpout,dumptrump2020,dumptrump,removetrump- now,byetrump,byebyetrump,trumpmustgo,removetrump,remove45,americaortrump,vote- trumpout2020,demvoice1,demvoice1b,republicansaretheproblem,prouddemocrat,goptraitors, gopbetrayedamerica, demswork4usa, votegopout, dems4usa, demforce, demcast, demcastca, fuckthegop, demcastpa, gopcorruption, gopcorruptionovercountry, lifelongdemocrat, vote- outgop, corruptgop, nevergop, gopcowards, gopcomplicittraitors, putinsgop, fuckgop, vote- dem, endthegop, gopgenocide, gopcomplicit, onevoice1, obamagreat, fbr, resister, resistor, resisters, fbrparty, resistthegop, resisttrumpism, proudresister, followbackresistance, vetsre- sist, resistersunite, globalresistance, resist45, resisttyranny, resisttrump, iresist, resisting, resistersister, resistancetaskforce, resistanceunited, iamtheresistance, keepresisting, theresis- 193 Figure B.1: Cascade propagation comparison only on news source labeled cascades set tancerises, veteransresist, resisted, fbresist, paresists, resistors, resisttogether, fbresistance, persister, neverthelessshepersisted, wehaveherback, imwithher, imstillwithher, alwayswith- her, stillwithher, makeamericakindagain, makeamericasmartagain, makeamericasaneagain, buildbackbetter, ridingwithbiden, bernie2020, warrendemocrat, teampete, teamjoe, team- pelosi, teampeteforever, yang2024, yanggang2024, warren2020, teamkamala, aoc2024, fore- verteampete,bernieorbust,hillary2016,bernie2016,warren2024,bernieforever,buttigiegdemo- crat,yanggangforlife,warrendem,yang2020,yanggangforever,feelthebern,berniewouldhave- won,peteforamerica,warrendemocrats,yanggang2020,bernie2024,teamwarren,berniesanders2020 B.3 Cascade Propagation Additional Analysis To confirm that our findings are not biased by CSI predictions, in Fig B.1, we compare only labeled set of unreliable/conspiracy vs. reliable cascades (labeled by the news source, used in training CSI). We find the conclusions are robust with similar trends for all propaga- tion properties. CSI uses engagements, but also text, temporal and account suspiciousness features, making predictions more robust, and verified to be within reasonable error rates. 194 Appendix C Proof and Analysis in Coordinated Campaigns Detection C.1 Proof of Joint Optimization in AMDN-HAGE Here we provide the proof to Theorem for joint optimization of AMDN-HAGE. Theorem 5. Our proposed optimizing algorithm will converge at a local minimum or a saddle point if in any iteration i the optimizer opt satisfies following conditions: • Given the frozen θ (i) g acquired by EM algorithm in iteration i, opt converges at a local minimum or or a saddle point (θ (i) a ,E (i) ). • L(θ (i) a ,E (i) ,θ (i) g )≤ L(θ (i− 1) a ,E (i− 1) ,θ (i) g ), where θ (i− 1) a and E (i− 1) are the starting points in iteration i. Proof. We first prove that the training algorithm converges. In EM, each step increases the likelihood function of a mixture model. We have: logP(E (i− 1) |θ (i) g )≥ logP(E (i− 1) |θ (i− 1) g ) (C.1) Thus, we obtain L(θ (i− 1) a ,E (i− 1) ,θ (i) g )≤ L(θ (i− 1) a ,E (i− 1) ,θ (i− 1) g ). (C.2) Then, from the second condition, we know that: L(θ (i) a ,E (i) ,θ (i) g )≤ L(θ (i− 1) a ,E (i− 1) ,θ (i) g ) (C.3) Therefore, we have: L(θ (i) a ,E (i) ,θ (i) g )≤ L(θ (i− 1) a ,E (i− 1) ,θ (i− 1) g ) (C.4) 195 which means the loss function monotonically decreases. Since we constraint the variance of the mixture model in both point processing model and social group model larger than a constant ϵ , the loss function is bounded by a constant C on a given activity trace set: L(θ (i) a ,E (i) ,θ (i) g )≥ C (C.5) Thus the loss function converges when i increases. Then we prove that the loss function converges at a local minimum or a saddle point. First when the parameters converges, we have: L(θ (i) a ,E (i) ,θ (i) g ) =L(θ (i+1) a ,E (i+1) ,θ (i+1) g ). (C.6) Thus: L(θ (i+1) a ,E (i+1) ,θ (i) g ) =L(θ (i+1) a ,E (i+1) ,θ (i+1) g ) (C.7) logP(E (i) |θ (i+1) g ) = logP(E (i) |θ (i) g ). (C.8) In EM, if logP(E (i) |θ (i+1) g ) = logP(E (i) |θ (i) g ) then θ (i) g = θ (i+1) g . Since (θ (i) a ,E (i) ) is a local minimum or a saddle point, we have: ∂L(θ (i) a ,E (i) ,θ (i) g ) ∂θ (i) a = ∂L(θ (i) a ,E (i) ,θ (i) g ) ∂E (i) = 0 (C.9) Since EM is known to converge to a local minimum, we have: ∂logP(E (i) |θ (i+1) g ) ∂θ (i+1) g = ∂logP(E (i) |θ (i) g ) ∂θ (i) g = 0 (C.10) Therefore, (θ (i) a ,E (i) ,θ (i) g ) is a local minimum or a saddle point. C.2 Proof of Lower Bound in M-Step For VigDet, we derive the lower bound used to approximate the M-step optimization. 196 Theorem6. Given a fixed inference of Q and a pre-defined ϕ G , we have following inequality: E Y∼ Q logP(Y|E,G)≥ E Y∼ Q X u∈V log exp{φ θ (y u ,E u )} P 1≤ m ′ ≤ M exp{φ θ (m ′ ,E u )} +const = X u∈V X 1≤ m≤ M Q u (y u =m)log exp{φ θ (m,E u )} P 1≤ m ′ ≤ M exp{φ θ (m ′ ,E u )} +const (C.11) Proof. To simplify the notation, let us apply following notations: Φ θ (Y;E) = X u∈V φ θ (y u ,E u ), Φ G (Y;G) = X (u,v)∈E ϕ G (y u ,y v ,u,v) (C.12) Let us denote the set of all possible assignment asY, then we have: E y∼ Q logP(y|E,G) =E Y∼ Q log exp(Φ( Y;E,G)) P Y ′ ∈Y exp(Φ( Y ′ ;E,G)) =E y∼ Q Φ( Y;E,G)− log X Y ′ ∈Y exp(Φ( Y ′ ;E,G)) =E y∼ Q (Φ θ (Y;E)+Φ G (Y;G))− log X Y ′ ∈Y exp(Φ( Y ′ ;E,G)) (C.13) Because ϕ G is pre-defined, Φ G (Y;G) is a constant. Thus, we have E y∼ Q logP(y|E,G) =E y∼ Q Φ θ (Y;E)− log X Y ′ ∈Y exp(Φ( Y ′ ;E,G))+const (C.14) Now, let us consider the log P Y ′ ∈Y exp(Φ( Y ′ ;E,G)). Since ϕ G is pre-defined, there must be an assignment Y max that maximize Φ G (Y;G). Thus, we have: log X Y ′ ∈Y exp(Φ( Y ′ ;E,G))≤ log X Y ′ ∈Y exp(Φ θ (Y;E)+Φ G (Y max ;G)) = logexp(Φ G (Y max ;G)) X Y ′ ∈Y exp(Φ θ (Y;E)) = Φ G (Y max ;G)+log X Y ′ ∈Y exp(Φ θ (Y;E)) (C.15) 197 Since ϕ G is pre-defined, Φ G (Y max ;G)) is a constant during the optimization. Note that P Y ′ ∈Y exp θ (Φ( Y ′ ;E)) sums up over all possible assignments Y ′ ∈ Y. Thus, it is actually the expansion of following product: Y u∈V X 1≤ m ′ ≤ M exp(φ θ (m ′ ,E u )) = X Y ′ ∈Y Y u∈V exp(φ θ (y ′ u ,E u )) = X Y ′ ∈Y exp(Φ θ (Y ′ ;E)) (C.16) Therefore, for Q which is a mean-field distribution and φ θ which model each account’s assignment independently, we have: E Y∼ Q logP(y|E,G)≥ E y∼ Q Φ θ (Y;E)− log X Y ′ ∈Y exp(Φ θ (Y ′ ;E))+const =E Y∼ Q Φ θ (Y;E)− log Y u∈V X 1≤ m ′ ≤ M exp(φ θ (m ′ ,E u ))+const =E Y∼ Q Φ θ (Y;E)− X u∈V log X 1≤ m ′ ≤ M exp(φ θ (m ′ ,E u ))+const =E Y∼ Q X u∈V log exp{φ θ (y u ,E u )} P 1≤ m ′ ≤ M exp{φ θ (m ′ ,E u )} +const = X u∈V X 1≤ m≤ M Q u (y u =m)log exp{φ θ (m,E u )} P 1≤ m ′ ≤ M exp{φ θ (m ′ ,E u )} +const (C.17) C.3 VigDet Training The E-step and M-step form a closed loop. To create a starting point, we initialize E with the embedding layer of a pre-trained neural temporal process model (in this paper we apply AMDN-HAGE) and initialize φ θ via clustering learnt on E (like fitting the φ θ to the prediction of k-Means). After that we repeat E-step and M-step to optimize the model. The pseudo code of the training algorithm is presented in Alg. 5. 198 Algorithm 5 Training Algorithm of VigDet. Require: DatasetS and pre-defined G and ϕ G Ensure: Well trained Q, E and φ θ 1: Initialize E with the embedding layer of AMDN-HAGE pre-trained on S. 2: Initialize φ θ on the initialized E. 3: while not converged do 4: Acquire Q by repeating Eq. C.18 with E, φ θ and ϕ G until convergence.{E-step} 5: φ θ ,E← argmax φ θ ,E logp(S|E)+E Y∼ Q P u∈V log exp{φ θ (yu,Eu)} P 1≤ m ′ ≤ M exp{φ θ (m ′ ,Eu)} . {M-step} 6: end while C.3.1 Detailed Justification to E-step In the E-step, to acquire a mean field approximation Q(Y) = Q u∈V Q u (y u ) that minimize the KL-divergence between Q and P, denoted as D KL (Q||P), we repeat following belief propagation operations until the Q converges: Q u (y u =m) = ˆ Q u (y u =m) Z u = 1 Z u exp{φ θ (m,E u )+ X v∈V X 1≤ m ′ ≤ M ϕ G (m,m ′ ,u,v)Q v (y v =m ′ )} (C.18) Here, we provide a detailed justification based on previous works [82, 84]. Let us recall the definition of the potential function Φ( Y;E,G) and the Gibbs distribution defined on it P(Y|E,G): Φ( Y;E,G) = X u∈V φ θ (y u ,E u )+ X (u,v)∈E ϕ G (y u ,y v ,u,v) (C.19) P(Y|E,G) = 1 Z exp(Φ( Y;E,G)) (C.20) where Z = P Y exp(Φ( Y;E,G)). With above definitions, we have the following theorem: Theorem 7. (Theorem 11.2 in [82]) D KL (Q||P) = logZ− E Y∼ Q Φ( Y;E,G)− H(Q) (C.21) where H(Q) is the information entropy of the distribution Q. A more detailed derivation of the above equation can be found in the appendix of 199 [84]. Since Z is fixed in the E-step, minimizing D KL (Q||P) is equivalent to maximizing E Y∼ Q Φ( Y;E,G)+H(Q). For this objective, we have following theorem: Theorem 8. (Theorem 11.9 in [82]) Q is a local maximum if and only if: Q u (y u =m) = 1 Z u exp(E Y−{ yu}∼ Q Φ( Y −{ y u };E,G|y u =m)) (C.22) where Z u is the normalizer and E Y−{ yu}∼ Q Φ( Y −{ y u };E,G|y u = m) is the conditional ex- pectation of Φ given that y u =m and the labels of other nodes are drawn from Q. Meanwhile, notethattheexpectationofalltermsinΦthatdonotcontain y u isinvariant to the value of y u . Therefore, we can reduce all such terms from both numerator (the exponential function) and denominator (the normalizer Z u ) of Q u . Thus, we have following corollary: Corollary 9. Q is a local maximum if and only if: Q u (y u =m) = 1 Z u exp{φ θ (m,E u )+ X v∈V X 1≤ m ′ ≤ M ϕ G (m,m ′ ,u,v)Q v (y v =m ′ )} (C.23) where Z u is the normalizer A more detailed justification of the above corollary can be found in the explanation of Corollary 11.6 in the Sec 11.5.1.3 of [82]. Since the above local maximum is a fixed point of D KL (Q||P), fixed-point iteration can be applied to find such local maximum. More details such as the stationary of the fixed points can be found in the Chapter 11.5 of [82]. C.3.2 Details of Experiments Implementation details on IRA dataset Wesplitthesequencesetto75/15/15fractionsfortraining/validation/testsets. Fortheset- ting of AMDN and AMDN-HAGE we use the described setting for AMDN-HAGE including 200 2 3 4 5 6 7 Cluster Number 0.015 0.020 0.025 0.030 0.035 0.040 0.045 Silhouette Score 0.043 0.033 0.03 0.025 0.02 0.019 Figure C.1: The silhouette scores of different group number activity sequences of maximum length 128 (we split longer sequences), batch size of 256 (on 1 NVIDIA-2080Ti gpu), embedding dimension of 64, number of mixture components for the PDF in the AMDN part of 32, single head and single layer attention module, component number in the HAGE part of 2. We use PyTorch, Adam optimizer with 1e-3 learning rate and 1e-5 regularization. For VigDet, the number of outer loops in the EM algorithm is picked up from {1,2,3} based on the validation account set performance. In each E-step, we repeat the belief propagation until convergence (within 10 iterations) to acquire the final inference. In each M-step, we train the model for max 50 epochs with early stopping based on validation objective function. The validation objective function is computed from the sequence likelihood on the 15% held-out validation sequences, and KL-divergence on the whole account set based on the inferred account embeddings in that iteration. C.3.3 Additional Evaluation We also consider a semi-supervised setting so that we can compare other ways to combine data-driven learning and prior knowledge graphs. In this setting, an additional 100 accounts are removed and used for training (provided as ground-truth group assignments) to the VigDet and the baselines. The baselines variants considered for comparison are Semi-NN, 201 Table C.1: Results on semi-supervised coordination detection (IRA) in 2016 U.S. Election Method AP AUC F1 Prec Rec MaxF1 MacroF1 LPA(HP) .633± .09 .768± .04 .681± .05 .762± .06 .618± .06 .716± .05 .815± .03 LPA(TL) .697± .04 .859± .02 .623± .06 .885± .03 .486± .08 .661± .05 .786± .03 LPA(CF) .711± .04 .853± .02 .608± .04 .665± .03 .564± .07 .683± .06 .772± .02 A-H + Semi-NN .771± .04 .878± .03 .705± .04 .766± .06 .655± .04 .723± .04 .828± .02 A-H + GNN (HP) .755± .06 .84± .05 .72± .07 .83± .14 .651± .08 .766± .05 .837± .04 A-H + GNN (CF) .806± .06 .895± .04 .73± .07 .863± .06 .637± .09 .764± .06 .845± .04 A-H + GNN (TL) .813± .05 .902± .03 .736± .06 .782± .06 .702± .09 .772± .06 .846± .03 VigDet-PL(TL) .877± .04 .955± .01 .739± .08 .942± .04 .614± .09 .80± .06 .851± .04 VigDet-E(TL) .881± .04 .881± .04 .881± .04 .957± .01 .957± .01 .957± .01 .734± .08 .946± .04 .604± .09 .808± .05 .808± .05 .808± .05 .848± .04 VigDet(TL) .88± .04 .957± .01 .957± .01 .957± .01 .736± .08 .942± .04 .609± .09 .808± .05 .808± .05 .808± .05 .849± .04 VigDet-PL(CF) .851± .03 .953± .01 .697± .06 .934± .03 .559± .07 .79± .04 .828± .03 VigDet-E(CF) .871± .04 .952± .01 .744± .06 .928± .03 .624± .08 .797± .05 .853± .04 VigDet(CF) .876± .03 .956± .01 .761± .06 .761± .06 .761± .06 .872± .07 .681± .09 .681± .09 .681± .09 .798± .05 .862± .04 .862± .04 .862± .04 2 4 6 8 10 12 14 16 18 Num of Clusters 0.48 0.49 0.50 0.51 0.52 0.53 0.54 Silh Score: Covid-19 data −0.05 0.00 0.05 0.10 0.15 Silh Score: IRA data 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 Account Creation Date (Year) 0 5 10 15 % Accounts Created / Year C0 C1 C2 Figure C.2: Left. Selection of number of clusters based on silhouette scores in COVID-19 and IRA datasets. Right. Distribution of account creation years in COVID-19 dataset for each identified accounts group or cluster a semi-supervised feed-forward neural network without requiring additional graph structure information. It is trained with self-training algorithm [190, 133]. Label Propagation Algo- rithm (LPA) [189] and Graph Neural Network (GNN) [81] are two baselines incorporated with graph structure. In LPA and GNN, for the graph structures (edge features), we use the CF and TL based prior knowledge graphs (similarly used in VigDet), as well as the graph learned by HP model as edge features. For the node features in GNN, we provide the account embeddings learned with AMDN-HAGE. The results are reported in Table C.1. C.4 Additional Analysis in COVID-19 Dataset Also, we present some numeric results and statistic properties we found on the COVID-19 dataset. WecomparethedistributionofaccountcreationyearsinCOVID-19datasetforeach identified accounts group or cluster. As we can see, the ratio of accounts created in recent 202 CoronaVirusOutbreak IR TousEnsemble BoycottCAC40 Raoult StayAtHomeSaveLives DesobeissanceCivile leadership IndiaFightsCoronavirus GeniouxMG MeaningfulGrowth DigitalTransformation GiletsJaunes CautionYesPanicNo 9News Japan Germany Management HealthForAll OMS SwasthaBharat PaliardFranco GreveGenerale facemasks ENDIRECTE HerdImmunity CoVId19 SputnikUpdates TestTraceIsolate CoronaAlert CoronaCrisis Ordnungsamt_Wolfsburg Ortsrat_Detmerode Polizei_Suedstadt WVG Nötigung Körperverletzung RAEconsultas QAnon Detmerode RB Entérate Perú TrumpLiesAmericansDie love Texas Israel WashYourHands NuevaNormalidad Auspol Resist EEUU 0 1000 2000 3000 4000 Figure C.3: Hashtag distribution for cluster C1 (COVID-19 dataset) years is higher in the two detected anomalous clusters, which is consistent to the reality that coordinatedcampaignsareraisedinrecentyears. Also,wepresentthehashtagdistributionin theC1clusterinFigureC.3. Surprisingly, althoughtheratioofsuspendedcoordinatedusers in this cluster is significantly higher than normal, the hashtags look not like disinformation contents or topics. Upon further inspection, we find the political hashtags like “Resist” or “Resistance” in the top-50 unique hashtags of this cluster, refer to U.S. liberal political movement against the reigning president, and have not been linked to disinformation or conspiracies. This phenomenon suggests coordination may not be specific to disinformation spreading. Innormalinformation,therecouldbetheinfluencefromcoordinatedcampaigns. 203 Appendix D Proof of PAC-Learnability and Runtime Analysis of MIC D.1 PAC-Learnability of Diffusion Mixture Model (MIC) Theorem4. Givenamixtureofunlabeledcascadeswithcompletelyobservedlive-edgegraphs, with diffusion parameters θ M ={p M e |e∈E} with M ∈{T,F} and any ϵ,δ > 0, with mixing weight π M ≥ ϵ mn we can recover in time poly (m 2 n/ϵ )· log(1/δ ), a list of poly (m 2 n/ϵ ) many candidates, at least one of which satisfies the following bound on the influence function σ θ M (S) and its estimate ˆ σ θ M (S) learned from the observed cascades for seed set S drawn from any distribution P over nodes in G, P S∼ P (|ˆ σ θ M (S)− σ θ M (S)|>ϵ )≤ δ with sample complexityO ( n 4 m 8 ϵ 4 ) 3 ln m δ (Proof in Appendix D.1). Proof. Each edge (coordinate) j has associated bernoulli variables x i j with parameter p i j for component i in the k-component mixture distribution. The pairwise coordinate means then are defined as follows, corr(j,j ′ ) =E[x j x j ′] = k X i=1 π i p i j p i j ′, 1≤ j <j ′ ≤ m (D.1) The sample estimate of corr(j,j ′ ) can be obtained directly from the observed live-edge graphs of unlabeled cascades. By the reduction to learning mixtures of discrete product distributions, given the sample estimates of the pairwise coordinate means, the parameters p i j and π can be estimated using algorithm Weights and Means (WAM) [39] for learning mixture distributions. We restate lemmas in [39, 21] used in the proof for completeness, with notations used in the reduction. 204 Lemma10 ([39]). Fork =O(1) and anyϵ ′ ,δ ′ > 0, WAM runs in time poly (m/ϵ ′ )· log(1/δ ′ ) and outputs a list of poly (m/ϵ ′ ) many candidates, at least one of which (with probability at least 1− δ ′ ) satisfies the following, |ˆ π i − π i |≤ ϵ ′ ,∀i and |ˆ p i e − p i e |≤ ϵ ′ ,∀π i ≥ ϵ ′ Lemma 11 (Lemma 4 in [21]). Given graph G and parameter space ϑ such that∀θ 1 ,θ 2 ∈ϑ ,||θ 1 − θ 2 || ∞ ≤ ϵ 0 , then,∀S⊆ V, |σ θ 1 (S)− σ θ 2 (S)|≤ mnϵ 0 Using the above lemmas and setting ϵ 0 = ϵ mn , δ ′ =δ and ϵ ′ = ϵ mn , the sample complexity for the desired influence function estimate is obtained. WAM requires sample estimates for E[x j x j ′] for all 1 ≤ j < j ′ ≤ m to be within an additive accuracy of ϵ matrix = ϵ ′2 m 2 (k+1) . x j x j ′ ∈{0,1} and therefore is Bernoulli distributed with some parameter say p jj ′ equal to E[x j x j ′]. Let ˆ p jj ′ be the sample estimate forE[x j x j ′] calculated from the observed cascades. Since each observed cascade is independently generated, we can compute the sample com- plexity of estimating E[x j x j ′] = p jj ′ within additive accuracy of ϵ matrix given the observed cascades. Applying chernoff bounds, we get P(|ˆ p jj ′ − p jj ′| ≥ ϵ matrix ) ≤ δ matrix with num- ber of observed samples being at least 2+ϵ matrix ϵ 2 matrix ln 2 δ matrix . Applying union bound, we get P(|ˆ p jj ′− p jj ′|≥ ϵ matrix )≤ δ matrix m(m− 1)/2 for all j,j ′ ∈ [m]. Setting δ matrix = 2δ m(m− 1) , we get with probability at least 1− δ , |ˆ p jj ′− p jj ′| is within additive accuracy of ϵ matrix for all j,j ′ and the sample complexity isO ( n 4 m 8 ϵ 4 ) k+1 ln m δ . D.2 Runtime and Convergence Analysis of MIC In Fig D.1, the runtime analysis of MIC vs. baseline SEIZ (SZ) on Twitter-2 are provided. The baseline SEIZ is run with time interval of 24 hours and cut-off time of 10K hours, and it runs differential equation solvers for each cascade, to fit the data with parameters 205 (a) Runtime (Twitter 2) (b) EM iterations (NLL) Figure D.1: MIC: runtime and convergence analysis specific to each cascade. The runtimes are evaluated and compared on Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz on single thread in python. Multi-threading, parallelization is left to future implementations. The runtime analysis is conducted on Twitter-2, since is the larger of the two datasets (with more users and more cascades), so that runtime can be analyzed with respect to different user sizes. We implemented vectorized computations and pre-computed users and cascades needed in the likelihood computation at the start of the EM iterations which improves computational efficiency, and reduces impact of number of cascades on runtime. TheEMestimationinAlgorithm1istrainedtillconvergence,i.e. thechangeinlikelihood is smaller than 0.01. The lookback window W, discussed in paragraph on Relaxation under Parameter Estimation, is set to 10 past events. The value of W impacts computational time and should be set to a constant smaller than V, that is the number of users. In the experiments, we set W with line search in the range{5,10,15} based on cross validation for computational efficiency. The EM converges within few iterations. The worst-case runtime complexity per EM iteration isO(k|C|V 2 ) for V users and C cascades and k is the number of components, and by setting W to a constant smaller than V, reducing toO(k|C|VW). 206
Abstract (if available)
Abstract
The proliferation of false and misleading information on social media has greatly reduced trust in online systems. Increasing reliance on social media, combined with sophistication in malicious operations to promote disinformation as a tool to influence public opinion and social outcomes has become a significant threat to society. In this thesis, we address challenges in disinformation mitigation by leveraging the diffusion or propagation dynamics of disinformation on social media, using diffusion network inference and analysis.
We consider two aspects in disinformation mitigation 1) Detection of disinformation and malicious efforts 2) Interventions to limit disinformation. In this thesis, for the first aspect we focus on timely detection of disinformation. We address the challenge of disinformation labeling in new and evolving domains in a timely and scalable manner. We propose to weakly-label social media posts using news-source credibility analysis, and leverage model-guided refinement of weak labels for disinformation labeling, by modeling instance credibility jointly with user credibility or stance from the content and social context of a post. Furthermore, we propose a disinformation detection model for early detection, i.e., before the content propagation. In order to improve early detection, we learn a generative model of social media responses conditioned on the content, leveraging historical social media responses to disinformation contents to enrich semantic understanding of why a content is labeled as disinformation, and thereby improve early detection when only contents without the social media responses are available. Secondly, we investigate how disinformation spreads and propose an unsupervised, generative model for detection of malicious coordinated campaigns employed for opinion manipulation and amplifying the spread of disinformation. The proposed model detects malicious groups by learning to infer unobserved or latent influence between accounts' activities, and their collective group anomalous behaviors from observed activities. The data-driven estimation of latent influence and group behaviors provides large improvements over state-of-the-art methods based on predefined coordination patterns or individual behaviors. We can also incorporate domain knowledge with data-driven learning by encouraging consistency between the group assignments using variational inference.
For the second part, we focus on interventions to limit disinformation, and characterize disinformation engagement to further inform detection and mitigation strategies. We address the problem of learning network interventions to limit disinformation propagation and prevent viral cascades by proposing a mixture model to infer diffusion dynamics of disinformation and legitimate contents from observed, unlabeled diffusion cascades. In addition, we use data-driven analysis to characterize engagements and platform interventions for social and political events, i.e., the pandemic, vaccines, and U.S. Election discourse. Using our disinformation labeling and detection methods, we examine disinformation and uncover suspicious coordinated groups in the pandemic and vaccine data. Furthermore, we investigate engagement with disinformation and conspiracies in the U.S. 2020 Election, and the effect of Twitter's ban and restrictions on the QAnon conspiracy group. We examine causal changes in content-posting strategies and sustained engagement with a regression discontinuity design. Our findings suggest that conspiracy groups' sustained activity and alternate content posting strategies pose challenges to mitigation measures. The outcome of this thesis is to improve the characterization of disinformation engagement and manipulation, and leverage diffusion inference to inform timely detection and improve mitigation interventions.
Linked assets
University of Southern California Dissertations and Theses
Conceptually similar
PDF
Understanding diffusion process: inference and theory
PDF
Modeling information operations and diffusion on social media networks
PDF
Disentangling the network: understanding the interplay of topology and dynamics in network analysis
PDF
Physics-aware graph networks for spatiotemporal physical systems
PDF
Computing cascades: how to spread rumors, win campaigns, stop violence and predict epidemics
PDF
Leveraging programmability and machine learning for distributed network management to improve security and performance
PDF
Towards combating coordinated manipulation to online public opinions on social media
PDF
Socially-informed content analysis of online human behavior
PDF
Measuing and mitigating exposure bias in online social networks
PDF
Fair Machine Learning for Human Behavior Understanding
PDF
Deep generative models for time series counterfactual inference
PDF
Modeling social and cognitive aspects of user behavior in social media
PDF
Graph embedding algorithms for attributed and temporal graphs
PDF
Deep learning models for temporal data in health care
PDF
Deriving real-world social strength and spatial influence from spatiotemporal data
PDF
Controlling information in neural networks for fairness and privacy
PDF
Sharpness analysis of neural networks for physics simulations
PDF
Alleviating the noisy data problem using restricted Boltzmann machines
PDF
Improving machine learning algorithms via efficient data relevance discovery
PDF
Robust causal inference with machine learning on observational data
Asset Metadata
Creator
Sharma, Karishma Rajesh
(author)
Core Title
Diffusion network inference and analysis for disinformation mitigation
School
Viterbi School of Engineering
Degree
Doctor of Philosophy
Degree Program
Computer Science
Degree Conferral Date
2022-12
Publication Date
09/01/2022
Defense Date
08/29/2022
Publisher
University of Southern California
(original),
University of Southern California. Libraries
(digital)
Tag
disinformation,fake news,information diffusion,machine learning,OAI-PMH Harvest,social network analysis
Format
application/pdf
(imt)
Language
English
Contributor
Electronically uploaded by the author
(provenance)
Advisor
Liu, Yan (
committee chair
), Ferrara, Emilio (
committee member
), Morstatter, Fred (
committee member
)
Creator Email
karish.sharma24@gmail.com,krsharma@usc.edu
Permanent Link (DOI)
https://doi.org/10.25549/usctheses-oUC111645414
Unique identifier
UC111645414
Legacy Identifier
etd-SharmaKari-11171
Document Type
Dissertation
Format
application/pdf (imt)
Rights
Sharma, Karishma Rajesh
Type
texts
Source
20220906-usctheses-batch-977
(batch),
University of Southern California
(contributing entity),
University of Southern California Dissertations and Theses
(collection)
Access Conditions
The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the author, as the original true and official version of the work, but does not grant the reader permission to use the work if the desired use is covered by copyright. It is the author, as rights holder, who must provide use permission if such use is covered by copyright. The original signature page accompanying the original submission of the work to the USC Libraries is retained by the USC Libraries and a copy of it may be obtained by authorized requesters contacting the repository e-mail address given.
Repository Name
University of Southern California Digital Library
Repository Location
USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA
Repository Email
cisadmin@lib.usc.edu
Tags
disinformation
fake news
information diffusion
machine learning
social network analysis