University of Southern California Dissertations and Theses

Identifying Social Roles in Online Contentious Discussions

(USC Thesis Other)

Identifying Social Roles in Online Contentious Discussions

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Identifying Social Roles in Online Contentious Discussions
by
Siddharth Jain
A Dissertation Presented to the
FACULTY OF THE GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulllment of the
Requirements for the Degree
DOCTOR OF PHILOSOPHY
(Computer Science)
August 2018
Copyright 2018 Siddharth Jain
Acknowledgements
This dissertation research would not have been possible if there were not the contributions of many hearts
and minds over years. First and foremost, I thank my advisor, Dr. Ron Artstein, for his invaluable
guidance, support and encouragement. Both his diligence and insightfulness in academic research and his
personality will always be an exemplary role model in my life. He taught me a lot of fundamental lessons
about research, either through his excellent advice, or through personal examples. With Dr. Artstein's
mentoring, I have learned to explore research topics, to design efficient solutions, and to write papers for
reporting research results, all of which are essential for a successful researcher.
I would also like to thank professors Paul Rosenbloom, Morteza Dehghani, Kallirroi Georgila, and Elsi
Kaiser for serving on my qualication exam and dissertation committees. Their constructive feedback and
suggestions greatly improved this dissertation.
Iamgratefultomypreviousadvisor,Prof. EduardHovy,forguidingmyentrancetotheresearchworld
and for his valuable discussions during the initial phase of this research. I also thank the Department of
Computer Science for the T.A. support they provided during my tenure at USC.
Finally, from my heart, I would like to thank my parents, my brother, and sister-in-law for their love
and encouragement. They have inspired me to always strive for excellence. I also want to thank my wife,
Priyal for her unconditional support during these challenging years.
ii
Table of Contents
Acknowledgements ii
List Of Tables vii
List Of Figures ix
Abstract xi
Chapter 1: Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Corpus for Social Roles (Chapter 3) [Published LREC'14] . . . . . . . . . . . . . . . . . . . 3
1.3 Leadership Models (Chapter 4) [Published SMMR'13] . . . . . . . . . . . . . . . . . . . . . 5
1.4 Claims and Argumentation Structure (Chapter 5) [Published as part of SocialNLP'16] . . . 6
1.5 Social Roles Model (Chapter 6) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.6 Social Roles Model (Neural Networks) (Chapter 7) . . . . . . . . . . . . . . . . . . . . . . . 7
1.7 Participants' Behavioral Analysis using Social Roles model (Chapter 8) . . . . . . . . . . . 8
Chapter 2: Literature Review 9
2.1 Functional Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 Structural Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3 Identifying Specic Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3.1 Experts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3.2 In
uencers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.4 Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Chapter 3: Corpus for Social Roles 23
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.2 Corpus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.2.1 Wikipedia: Articles for Deletion (AfD) . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.2.2 4forums.com . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.3 Annotations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.4 Behaviors and Social Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.4.1 Participation Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.4.1.1 Stubbornness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.4.1.2 Sensibleness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.4.2 Attendedness Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.4.2.1 Ignored-ness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.4.3 In
uence Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.4.3.1 In
uence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
iii
3.5 Social Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.5.1 Leader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.5.2 Follower . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.5.3 Rebel** . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.5.4 Voice in Wilderness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.5.5 Idiot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.5.6 Nothing and Nothing (Sensible) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.5.7 Other . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.6 An Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Chapter 4: Leadership Models 44
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.3 Corpus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.3.1 Contentious Discussions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.4 Leadership Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.4.1 Content Leader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.4.1.1 Attract Followers (AF) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.4.1.2 Counterattack (CA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.4.2 SilentOut Leader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.4.2.1 Factual Arguments (FA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.4.2.2 Small Wins (SW) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.5.1 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.5.2 Content Leader calculation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.5.3 SilentOut Leader calculation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.6 Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.6.1 Correlation between Content Leaders and SilentOut Leaders . . . . . . . . . . . . . 54
4.6.2 Predicting outcome of the discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.6.3 Verifying authenticity of the leaders . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
Chapter 5: Claims and Argumentation Structure 60
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
5.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
5.3 Corpus and Annotation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
5.3.1 Claims . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
5.3.2 Claim-Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5.4 Model and Implementation Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5.4.1 Claim Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.4.2 Claim Delimitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.4.3 Claim-Link Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
5.5 Experiments and Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5.5.1 Claim Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5.5.2 Claim Delimitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
5.5.3 Claim-Link Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
iv
Chapter 6: Social Roles Model 74
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
6.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
6.2.1 Stubbornness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
6.2.2 Sensibleness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
6.2.3 Ignored-ness. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
6.2.4 In
uence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
6.3 Experiments and Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
6.3.1 Characteristics and Social Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
6.3.2 Feature Ablation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
6.3.2.1 Stubbornness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
6.3.2.2 Sensibleness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
6.3.2.3 Ignored-ness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
6.3.2.4 In
uence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
6.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
Chapter 7: Social Roles Models (Neural Networks) 85
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
7.2 Word Embeddings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
7.2.1 Word Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
7.2.2 Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
7.3 Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
7.3.1 CNN with user utterances (CNN-user) . . . . . . . . . . . . . . . . . . . . . . . . . . 89
7.3.2 Bidiractional LSTM with user utterances (BiLSTM-user) . . . . . . . . . . . . . . . 90
7.3.3 Bidirectional LSTM with all utterances (BiLSTM-all) . . . . . . . . . . . . . . . . . 92
7.3.3.1 Vector representation of utterances . . . . . . . . . . . . . . . . . . . . . . 92
7.3.3.2 BiLSTM-all . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
7.4 Experiments and Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
7.4.1 Tuning Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
7.4.2 Characteristics and Social Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
7.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Chapter 8: Participants' Behavioral Analysis using Social Roles model 99
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
8.2 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
8.3 Participants' Behavioral Diversity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
8.4 Behavioral diversity vs Topic of discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
8.4.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
8.5 Behavioral Diversity vs Contention Observed . . . . . . . . . . . . . . . . . . . . . . . . . . 109
8.5.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
8.6 Diversity vs Other Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
8.6.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
8.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
Chapter 9: Conclusion 117
9.1 Framework for Social Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
9.2 Leadership Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
9.3 Argumentation Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
9.4 Social Roles models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
9.5 Social Roles model as Analytic Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
v
Appendix A
Social Roles Annotation Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
A.1 Stubbornness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
A.2 Sensibleness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
A.3 Ignored-ness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
A.4 In
uence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
Appendix B
Neural Network Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
B.1 CNN-user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
B.2 BiLSTM-user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
B.3 BiLSTM-all . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Reference List 125
vi
List Of Tables
3.1 Corpora stats. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.2 Social roles distribution in corpora. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.3 Cohen's Kappa score for social roles' annotation. . . . . . . . . . . . . . . . . . . . . . . . . 30
3.4 The relationship between the values of behavioral characteristics and social roles corre-
sponding to them. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.5 Participant roles for the discussion in Figure-3.9. . . . . . . . . . . . . . . . . . . . . . . . . 43
4.1 Stance majority as factor for outcome of discussion. . . . . . . . . . . . . . . . . . . . . . . 47
4.2 Majority vs Average participant comments. . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.3 Spearman correlation between Leadership models. . . . . . . . . . . . . . . . . . . . . . . . 54
4.4 Comparison of discussion outcome prediction accuracy. . . . . . . . . . . . . . . . . . . . . . 56
4.5 Comparison of accuracy for different coefficient values for Content Leader model. . . . . . . 57
4.6 Comparison of accuracy for different coefficient values for SilentOut Leader model. . . . . . 57
4.7 Accuracy for authenticity of leaders identied by models for Wikipedia: AfDs. . . . . . . . 58
4.8 Accuracy for authenticity of leaders identied by models for 4forums.com. . . . . . . . . . . 58
5.1 Examples of sentences with and without claims. . . . . . . . . . . . . . . . . . . . . . . . . . 64
5.2 Cohen's Kappa score for claim annotation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5.3 Examples of claims linked with each other. . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5.4 Claim Detection results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5.5 Claim Delimitation results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
5.6 Claim-Link Detection results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
vii
5.7 Mis-classied claims. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
6.1 Results for characteristics and social roles models. . . . . . . . . . . . . . . . . . . . . . . . 80
6.2 Feature ablation results for stubbornness. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
6.3 Feature ablation results for sensibleness. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
6.4 Feature ablation results for ignored-ness. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
6.5 Feature ablation results for in
uence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
7.1 Hyperparameters for word embeddings and tested values. Default values shown in bold. . . 88
7.2 Hyperparameters for models and tested values. . . . . . . . . . . . . . . . . . . . . . . . . . 96
7.3 Weighted F1-score for all the models. For each model, 2 scores are reported corresponding
to 80 and 160 discussions respectively. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
8.1 Participation distribution in contentious discussions. . . . . . . . . . . . . . . . . . . . . . . 100
8.2 Topic models for Wikipedia discussions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
B.1 Hyperparameter values for custom word embeddings for CNN-user . . . . . . . . . . . . . . 122
B.2 Hyperparameter values for CNN-user with custom embedding and word2vec respectively . . 122
B.3 Hyperparameter values for custom word embeddings for BiLSTM-user . . . . . . . . . . . . 123
B.4 Hyperparameter values for BiLSTM-user with custom embedding and word2vec respectively123
B.5 Hyperparameter values for doc2vec for BiLSTM-all . . . . . . . . . . . . . . . . . . . . . . . 123
B.6 Hyperparameter values for autoencoder for BiLSTM-all . . . . . . . . . . . . . . . . . . . . 123
B.7 Hyperparameter values for BiLSTM-all with doc2vec and autoencoder respectively . . . . . 124
viii
List Of Figures
3.1 An example discussion on Wikipedia: AfD forum.. . . . . . . . . . . . . . . . . . . . . . . . 25
3.2 An example discussion on 4forums.com forum. . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3 \Threaded view" of a discussion on 4forums.com . . . . . . . . . . . . . . . . . . . . . . . . 28
3.4 Examples of non-stubborn comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.5 Examples of sensible comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.6 Examples of non-sensible comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.7 Examples of not-ignored comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.8 Examples of in
uence indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.9 An example discussion from Wikipedia corpus . . . . . . . . . . . . . . . . . . . . . . . . . . 41
7.1 CBOW and Skip-gram models, taken from [Mikolov et al., 2013] . . . . . . . . . . . . . . . 87
7.2 Convolution Neural Network (CNN), taken from [Kim, 2014a] . . . . . . . . . . . . . . . . . 89
7.3 Bidirectional LSTM network (BiLSTM), taken from [Graves, 2012] . . . . . . . . . . . . . . 91
7.4 doc2vec models, taken from [Le and Mikolov, 2014] . . . . . . . . . . . . . . . . . . . . . . 93
7.5 Autoencoder model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
8.1 Social Roles distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
8.2 Average Stubbornness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
8.3 Average Sensibleness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
8.4 Average Ignored-ness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
8.5 Average Leadership . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
ix
8.6 Average Followership . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
8.7 Behavioral diversity vs Topic of discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
8.8 Sensibleness diversity based on topic of the discussion . . . . . . . . . . . . . . . . . . . . . 108
8.9 Stubbornness diversity based on topic of the discussion . . . . . . . . . . . . . . . . . . . . . 109
8.10 Observed contention distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
8.11 Behavioral diversity vs Observed contention . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
8.12 Leadership diversity based on contention observed in the discussion . . . . . . . . . . . . . . 113
8.13 Sensibleness diversity based on contention observed in the discussion . . . . . . . . . . . . . 113
8.14 Behavioral diversity vs Other participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
8.15 Leader/Follower diversity based on other participants in the discussion . . . . . . . . . . . . 115
8.16 Stubbornness diversity based on other participants in the discussion . . . . . . . . . . . . . 116
x
Abstract
Themaingoalofthisdissertationistoidentifysocialrolesofparticipantsinonlinecontentiousdiscussions,
dene these roles in terms of behavioral characteristics participants show in such discussions, and develop
methods to identify these participant roles automatically. As social life becomes increasingly embedded in
onlinesystems,theconceptofsocialrolebecomesincreasinglyvaluableasatoolforsimplifyingpatternsof
action, recognizing distinct participant types, and cultivating and managing communities. In contentious
discussions, the roles may exert a major in
uence on the course and/or outcome of the discussions. The
existingworkonsocialrolesmostlyfocusesoneitherempiricalstudiesornetworkbasedanalysis. Whereas
thisdissertationpresentsamodelofsocialrolesbyanalyzingthecontentoftheparticipants'contribution.
In the rst portion of this dissertation, I present the corpus of participant roles in online discussions
from Wikipedia: Articles for Deletion and 4forums.com discussion forums. A rich set of annotations of
behavioral characteristics such as stubbornness, sensibleness, in
uence, and ignored-ness, which I believe
all contribute to the identication of roles played by participants, is created to analyze the contribution
of the participants. Using these behavioral characteristics, Participant roles such as leader, follower,
rebel, voice in wilderness, idiot etc. are dened, which re
ect these behavioral characteristics. In the
second part of this dissertation I present the methods used to identify these participant roles in online
discussions automatically using the contribution of the participants in the discussion. First, I develop two
models to identify leaders in online discussions that quantify the basic leadership qualities of participants.
Then, I present the system for analyzing the argumentation structure of comments in discussions. This
analysis is divided in three parts: claim detection, claim delimitation, and claim-link detection. Then,
the dissertation presents the social roles model to identify the participant roles in discussions. I create
xi
classicationmodelsandneuralnetworkstructuresforeachbehavioralcharacteristicusingasetoffeatures
based on participants' contribution to the discussion to determine the behavior values for participants.
Using these behavioral characteristic values the roles of participants are determined based on the rules
determined from the annotation scheme. I show that for both, the classication models and neural
networks, the rule based methods perform better than the model that identies the participant roles
directly. This signies that the framework of breaking down the problem of identifying social roles to
determining values of specic behavioral characteristics make it more intuitive in terms of what we expect
from participants who assume these roles. Although the neural network methods perform worse than
their traditional classication method counterparts, when provided with additional training data, neural
network structures improve at a much higher rate.
In the last part of the dissertation, I use the social roles model as a tool to analyze participants'
behavior in large corpus. Social roles are automatically tagged in Wikipedia corpus containing 26000
discussions. Thisallowsdeterminingparticipants'rolesovertimeinordertoidentifywhethertheyassume
different roles in different discussions, and what factors may affect an individual's role in such discussions.
I investigate three factors: topic of the discussion, amount of contention in the discussion, and other
participants in the discussion. The results show that participants behave similarly in most situations.
However, the social roles model is able to identify instances where participants' behavior patterns are
different than their own typical behavior. In doing so, the model provides useful context regarding the
reason behind these behavioral patterns by identifying specic behaviors affected by the situation.
xii
Chapter 1
Introduction
1.1 Motivation
Participants in online social environment adopt various roles. The concept of social roles has long been
used in social science to describe the intersection of behavioral, meaningful, and structural attributes that
emerges regularly in particular settings. People's behavior in social situations is not random and com-
pletely unpredictable, nor is it uniformly identical in each situation. Rather, people act differently toward
different people, and depending on the circumstances. The reason is that, besides having personalities, by
being part of a social group, people occupy positions in the social structures of groups that allow them to
do and say certain things, as well as constrain them from saying and doing other things. This mixture of
allowance and constraints, combined with the choices the individual makes given this mixture, constitutes
a social role. Understanding that people have roles in their communities allows others to contextualize
their behavior. The number of such roles, and what exactly denes them, is however not easy to specify.
In everyday life, many of the roles people take on are well-dened and unambiguous; these tend to
be the roles to which titles can be attached. Some are dened by legal and government means, like the
President and police officer. Others are roles held by virtue of employment, like teacher and taxi driver,
or engineer and manager. Family roles have both biological and legal foundations and include husband,
wife, daughterandparent. Byknowingthetitleassociatedwithsomeone'srole, onecanhaveareasonably
1
good understanding of that person's skills, privileges and responsibilities. In sharp contrast to roles such
as these, other roles are not so well dened and are far more ambiguous and
exible, constituted through
repeated interactions and mutually agreed upon practices. These tend to be the roles that are unspoken
and yet are demonstrated by patterns in behavior. For example, in a group of friends one may emerge as
the \group planner" or \decision-maker". Similarly, another member of the group may grow to be seen
as a trustworthy condant, counselor or arbitrator within the group. In the same way that titled roles
implysetsofexpectations, implicitrolesgrowasaresultoftheexpectationsassociatedwithanindividual.
Expectations about others' behavior are useful because they give us some knowledge about how to act
toward them. Roles, in turn, are useful because, when they comprise sets of expectations, they allow us
to generalize across people and have some a priori knowledge about the categories of people, how to act
toward them and what to expect in their actions toward us.
Online discussions are a particularly interesting environment in which to study social roles, because all
therolesandallthesocialstructurethatarepresentinthecorrespondingcommunitiesarisesolelythrough
the behavior of the participants. There are no roles that are technologically imposed from the outset, yet
participants emerge as leaders, motivators, gatekeepers, etc., shaping their communities. Different genres
of discussion communities develop their own social norms regarding what kinds of conversations to have
and what kind of speech is acceptable. Some communities are welcoming to newcomers, some are not.
Some enjoy rigorous debate, other shy away from controversy. Whatever a community's standards are
for discussions, they are the result of the evolving tastes of the participants in those discussions. These
standards are continually enforced and challenged by the behaviors of those participants. Therefore
identifying what role someone is playing can be helpful in many ways. For example, the roles can be
used to identify authoritative sources in social media, nd in
uential people in the community, detect
subgroups, mine attitude towards events and topics, measure public opinion and the controversiality of
different topics, summarize different viewpoints with respect to some topic or entity, and many other
similar applications.
2
In general, the study of social roles seeks to understand the social structure of the community and
their members. According to the Dictionary of Psychology [Corsini, 2002], a role is \a set of behaviors
expected of a person possessing a certain social status". Accordingly, a role is an upper-level concept
that can comprise a set of behaviors. The most general approach to nding roles consists of two main
stages [Welser et al., 2007]: an in-depth understanding of the community in order to identify roles which
may be detected, and then creation of a role with observed characteristics and rules that will allow the
classication of individuals into the pre-dened roles. The role one plays in a discussion is re
ected by
the behaviors one shows in one's social interactions. These behaviors can be indicators of a participant's
personal goals, their ability to convince others, their attitude towards one another, etc.
In this dissertation, I present a framework that analyzes the contribution and interaction patterns
of participants in online contentious discussions to identify their social roles. I focus my analysis on
discussions on Wikipedia: Articles for Deletion and 4forums.com discussion forums where participants
have different goals based on their opinion. This encourages them to exhibit different behaviors to express
their own opinion or to challenge someone else's to achieve their goals. Since what roles persist in such
discussions are not known a priori, I rst analyze the discussions to identify pertinent behaviors of the
participantsandcreateataxonomyofsocialrolesfromthecombinationsofthesebehaviors. Ithendevelop
methodsbasedontheanalysisofthecontributionofparticipantsinthediscussiontocreatevariousmodels
to identify these social roles automatically. In the end, I use the social roles model as a tool to analyze
participants' behavior over time in a large corpus in order to identify whether they assume different roles
in different discussions and investigate factors that affect participants' behavior in such discussions.
1.2 Corpus for Social Roles (Chapter 3) [Published LREC'14]
I create the corpus for social roles by identifying the following behavioral characteristics of participants
in online contentious discussions based on their characteristic language use, patterns of communication,
attitude towards one another, etc.
3
Stubbornness - Stubbornness is an indicator of the participation level of a participant and the
intransigencetowardshis/herstance. Itvarieswidely,fromuserswhopostmanydozensofcomments,
to those who post very few. It is often observed that very small number of participants write a
disproportionately large percentage of comments. The unequal levels of participation underscore
the fact that participants are not all the same.
Sensibleness - The level of sensibleness of the contribution by a participant is very important for
measuringhisorherimpactonthediscussion. Thiscontributestowardstheimpressionaparticipant
makes on others. The notion of what can be considered as sensible often depends on the domain
and nature of discussion.
Ignored-ness - For all participants in discussions, the responses they receive from others provide
valuablecluesabouthowwellreceivedtheircontributionswere. Whenaparticipantreceivespositive
feedback, his behavior is rewarded; when he receives negative feedback, or none at all, his behavior
is punished.
In
uence - This aspect helps in identifying leaders and followers among the participants. It helps
answer these questions: (i) in
uence on others: was a participant able to in
uence another par-
ticipants through their contribution?, and (ii) endorsement: did a participant acknowledge another
participant's in
uence?
Not all kinds of behaviors contribute directly towards identifying the social role. For example, a partici-
pant'sbehaviorofnotfollowingtherulesandregulationsofthediscussioncommunitycontributestowards
his nonsensical behavior. Also, some behaviors can have different level of impact. For example, deviating
from the discussion can be considered acceptable in some domains whereas it can be unacceptable for
others. Using these behavioral characteristics I create the taxonomy of social roles. Here is a brief de-
scription of the roles dened in contentious discussions. A participant can have more than one role based
on his/her behavioral characteristics.
Leader - Participants who in
uence other participants.
4
Follower-Participantsthatarein
uencedbyotherparticipantsorwhodon'thavetheirownclaims.
Rebel - Sensible participants who are devoted to the discussion.
VoiceinWilderness-Sensibleparticipantswithsubstantialcontributionwhoareignoredbyother
participants.
Idiot - Nonsensical participants whose contribution towards the discussion is not constructive.
Nothing/Nothing (Sensible) - Participants who make minimal contribution to the discussion.
1.3 Leadership Models (Chapter 4) [Published SMMR'13]
I present two models to identify leaders in online contentious discussions based on their contribution to
the discussion. The models quantify various intuitive leadership qualities to assign leadership points to
participants.
ContentLeader-Thismodelisbuiltupontwobasiccharacteristicsofleaders: 1)Encourageothers
tofollowtheirarguments(Attractfollowers)and2)Countertheargumentsfromtheopposinggroups
(Counterattack).
SilentOutLeader -Thismodelquantiestheabilityofleaderstosilenceoutopposingparticipants
with their arguments. This is modeled by two attributes: 1) Presenting arguments that cannot be
countered (Factual Arguments) and 2) Winning the small battles in the discussion (Small Wins).
I use these models as part of the analysis to identify the in
uence value for participants in the social roles
model.
5
1.4 ClaimsandArgumentationStructure(Chapter5)[Published
as part of SocialNLP'16]
Analyzingtheargumentationstructureofparticipants'commentsisanimportantaspectofthesocialroles
analysis. I develop supervised machine learning methods to identify claims and links between claims in
comments made by participants to create the argumentation structure of their comments. This analysis
is divided into three parts:
Claim Detection - Each sentence is classied as having claim or not.
Claim Delimitation - Specic claims inside the sentences are identied.
Claim-Link Detection - Determine whether two claims are linked to each other.
The argumentation structure of the comments is an important aspect in determining sensibleness and
parts of it are also used in determining stubbornness of participants.
1.5 Social Roles Model (Chapter 6)
The social roles of participants in contentious discussions are identied by quantifying their behavioral
characteristics in such discussions. I develop methods to identify values for each characteristic for partic-
ipants by analyzing their contributions and then determine their social roles using the rules described in
the taxonomy of social roles.
Stubbornness - The stubbornness value of a participant is classied by analyzing the amount of
participation and interaction pattern of the participant.
Sensibleness - The sensibleness value of a participant is classied by leveraging features obtained
through argumentation mining, determining relevance of the contribution to the discussion, and
analyzing the attitude of other participants towards his/her contributions.
6
Ignored-ness - The ignored-ness value of a participant is determined using features that indicate
whether a participant is ignored by other participants.
In
uence-Thein
uencevalueofaparticipantisclassiedbyidentifyingendorsementbyortowards
the participant, calculating leadership points based on the leadership models, and determining the
novelty of the contribution by the participant.
The results show that identifying social roles through behavioral characteristics perform better than
identifying the social roles directly. This shows that breaking down the problem of identifying social roles
to determining values of specic behavioral characteristics makes it more intuitive in terms of what we
expect from participants who assume these roles and also performs better than identifying the social roles
directly.
1.6 Social Roles Model (Neural Networks) (Chapter 7)
In order to test whether there is any need of feature engineering to identify the social roles, I develop
neural network structures to identify these roles using the words used by the participants. I develop
neural network structures to identify the social roles directly and also to identify values for each behav-
ioral characteristic of the participants and then determine their social roles using the rules described in
the taxonomy of social roles. I compare the following neural networks for identifying the social role or
behavioral characteristic of the participant:
Convolution Neural Network (User): A Convolution Neural Network that uses the utterances
madebyaparticipantinthediscussion. CNN-useridentiespatternsinalltheparticipantutterances
combined to identify the social role or the behavioral characteristic.
Bidirectional LSTM (User): A Bidirectional Long Short Term Memory (BiLSTM) network
that uses the utterances made by a participant in the discussion. BiLSTM-user uses time-series
information of participant's utterances to identify the social role or the behavioral characteristic.
7
Bidirectional LSTM (All): A two-layer network which incorporates all the utterances in the dis-
cussiontocapturethecontextaroundparticipant'sutterances. Therstlayerofthenetworkcreates
a vector representation of each utterance in the discussion. The second network is a Bidirectional
LSTM that uses these vector representations to identify the social role or behavioral characteristic
of the participant.
I show that for all the neural network structures identifying social roles by determining behavioral charac-
teristicvaluesperformbetterthanidentifyingsocialrolesdirectly. Althoughtheneuralnetworkstructures
performworsethantheircounterpartfromChapter-6,whenprovidedwithadditionaltrainingdata,neural
network structures improve at a much higher rate.
1.7 Participants' Behavioral Analysis using Social Roles model
(Chapter 8)
IusesocialrolesmodelasatooltoautomaticallyidentifysocialrolesforparticipantsinalargeWikipedia:
ArticleforDeletioncorpusof26071discussions. Thebehavioralanalysispresentedinthischapterisaimed
tounderstandhowdifferentfactorsaffect individual behaviorsobservedonline. In particular, Iinvestigate
3 factors that can affect participants' behaviors in such discussions. 1) the topic of the discussion, 2)
the amount of contention observed by the participant in the discussion, and 3) other participants present
in the discussion. I show that the social roles model provides useful context regarding the reason for
participants' behavioral diversity by identifying specic behavioral patterns in certain situations.
Chapters 3, 4, and 5 were written originally as independent papers; in adapting them for inclusion in
the dissertation I kept the changes minimal, so each of them can be read independently of the rest of the
dissertation. Chapter 5 has been published as part of a paper for Sensibleness analysis model presented
in chapter 6, but since the work presented in Chapter 5 is used by other components of the social roles
model as well, I present it as a separate chapter.
8
Chapter 2
Literature Review
The concept of social roles has long been a topic of discussion, and different authors have proposed
various denitions and approaches to study them [Biddle, 1986]. Looking at the different denitions of
the term, researchers often associate social roles with social statuses, social categories, or social parts
([Biddle, 1986]; [Markel, 1998]; [Merton, 1968]; [Zurcher, 1983]). Furthermore, enactors of social roles
adapt their behavior to suit both their own preferences (Markel, 1998) and the expectation of others
([Biddle, 1986]; [Merton, 1968]). Thus people expect certain behaviors, attitudes, and values from people
enacting certain social roles ([Biddle, 1986]; [Markel, 1998]; [Merton, 1968]; [Zurcher, 1983]). In recent
years, a great deal of interest in researching social roles in online communities has emerged. Different
authors have used different approaches or techniques to model and explain certain social roles that are
present in such communities. To create typology of social roles in online communities, certain social role
theories seem more appropriate than others. To give an overview of different approaches to research this
area, two social role theories are categorized that are applicable and important for researching social roles
in online communities: the functional roles and the structural roles.
2.1 Functional Roles
The functional role theory focuses on the characteristic behavior of persons who occupy social positions
withinastablesocialsystem[Biddle, 1986]. Asthenamesuggests,thetheoryismainlyconcernedwiththe
9
function of participants in different positions, and it has been criticized for being rigid and not accounting
for the different personalities of people enacting the roles [Biddle, 1986]. Quantitative evaluation of the
types of content posted by online community members is used to distinguish between social roles. The
focusofthesestudiesisonthefunctionsthesemembersperformandtheeffecttheyhaveonthecommunity.
This approach tries to group online community members based on their qualitative behavior, the type of
content they produce, or the part they perform in the community.
Earlier work on functional social roles has focused on empirical studies to analyze online communi-
ties to create an ontology of social roles and explain how these roles re
ect the behaviors present in the
corresponding communities. [Masolo et al., 2004] establish a general formal framework for developing a
foundational ontology of socially constructed entities, in the broadest sense of this notion and further
contribute to understanding the ontological nature of roles. [Gliwa et al., 2013] introduces different roles
of users in social media, taking into consideration their strength of in
uence and different degrees of co-
operativeness. The most prominent method for creating a taxonomy of social roles has been based on
analyzingthebehaviorsofparticipantsinthecommunitiesandthengroupingtheparticipantsthatexhibit
similar behavior to identify their role. [Golder, 2003] uses a combination of qualitative and quantitative
methods to develop a typology of social roles in Usenet. The typology explains participation inequal-
ity; users have different needs and goals, as well as different abilities and privileges, all of which require
participants to behave differently from one another. It also explains the changes in online communities
through the changes in the roles the participants enact in the newsgroup community. Building on this,
[Golder and Donath, 2004]developedatypologyofsocialrolesinUsenet. Todothis, theyobservedactive
members and grouped them, based on the assumption that observed behavior denes a member's social
role. They found and described seven roles: the newbie, the celebrity, the elder, the lurker, the
amer,
the troll, and the ranter. [Pfeil et al., 2011] study an online support community for older people with
the aim of developing a taxonomy of social roles based on content analysis and social network analysis.
They identify a set of six social roles; moderating supporter, central supporter, active member, passive
10
member, technical expert, and visitor. [Welser et al., 2011] analyze qualitative comments posted on com-
munity oriented pages, wiki project memberships, and user talk pages in order to identify a sample of
editors who represent four key roles: substantive experts, technical editors, vandal ghters, and social
networkers. [Fller et al., 2014] investigate the heterogeneous roles of contest participants based on an
international innovation-contest community. They identify six user types associated with various behav-
ioral contribution patterns by using cluster and social network analysis. The six user types further differ
in their communicative content and contribution quality. Some researchers have tried to use the social
roles they nd in their study to analyze the community further. [Chan et al., 2010] present an empiri-
cal analysis of user communication roles in a medium-sized bulletin board and analyze the composition
of several forums in terms of these roles, demonstrating similarities between forums based on underly-
ing user behavior rather than topic. [Wever et al., 2010] examine the social knowledge construction in
e-discussions and introduces ve roles: starter, summarizer, moderator, theoretician, and source searcher.
They show a positive effect of role assignment on students' social knowledge construction at the start of
the discussions. This implies that roles should be introduced at the start of the discussions and can be
faded out towards the end. Rather than looking at online communities, some researchers focus on real-life
groups to analyze the social roles. [Yeh, 2010] presents the ndings of a study of 32 preservice teachers
participating in an 18-week instruction program. The analysis of online group discussions revealed: (a) of
thirteen identied online behaviors, the most common were constructing a positive atmosphere, providing
opinions for group assignments, and providing reminders of assignment-related work; (b) of eight online
roles identied within a group, the most common roles were information providers, opinion providers,
and troublemakers; (c) four online learning communities based on `collaboration' and `participation' were
identied. [Jones et al., 2011] present the ndings of an exploratory study within a real-life context that
investigates participant behavior and emergent user roles in asynchronous distributed collaborative idea
generation by a dened community of users. The study revealed ve user roles; contributor, encourager,
social loafer, harvester, and absentee.
11
Although researchers have proposed many theories dening social roles in online communities, very
few of them have proposed methods identifying these roles automatically based on the analysis of the type
of content provided by participants. People have used various natural language processing methods to
analyze the content and identify the roles. [Leuski, 2004] argues that an individual's role can be detected
by analyzing the patterns of the speech acts in his/her incoming and outgoing emails. [Garg et al., 2008]
propose an approach for automatic recognition of roles in meetings by combining two sources of infor-
mation: the lexical choices made by people playing different roles, and the Social Networks describing
the interactions between the meeting participants. [Sapru and Valente, 2012] investigate the automatic
recognition of speaker role in meeting conversations from AMI corpus. They identify two types of roles:
formal roles, xed over the meeting duration and recognized at recording level, and social roles related
to the way participants interact between themselves, recognized at speaker turn level. [Du et al., 2016]
propose a Dirichlet Process Mixture Model to automatically optimize the number of roles and integrate
features mined from text data to analyze user roles. Others have tried to take advantage of the meta-data
oftheconversationsinordertogetsomeinsightoftheparticipants'behaviors. [Lee et al., 2013]proposea
method to use the content-based behavioral features extracted from user generated content and behavior
patterns to identify users' roles and to explore role change patterns in social networks. They use fea-
tures based on users' personality, behavior, action sequence, affectivity, and recognition to determine their
membership in various role-sets. [Chen et al., 2016] study the problem of inferring social roles of mobile
users from users' communication behaviors. They propose Mobile Communication Behaviors framework
to infer social roles and show that it solves the difficulties of inferring such roles with few labeled users,
inaccurate label information, and few users' feature information.
2.2 Structural Roles
Theemphasisinthestructuralroletheoryisonmemberswithcertainsocialpositionswhosharethesame,
patterned behaviors (roles) that are directed towards other sets of persons in the structure [Biddle, 1986].
12
This approach analyses the system as a whole and looks at communication
ows to explain the roles
in a system. Since the emergence of online social networks, there have been several studies looking at
social roles from a structural perspective, applying social network analysis. The social network analysis
approachfocusesonsocialinteractionwithinacommunitytogroupmemberswhodisplaysimilarpatterns
toothermembers. Differentgroupsare assumed toindicatedifferentsocialpositions. Although structural
role theory has gained much popularity in online studies, it has been criticized for not trying to explain
members' behavior [Biddle, 1986]. Traditionally, structural roles have attracted more researchers than
functional roles. Blockmodeling is one of the basic approach for modeling structural roles. It is an
algebraic framework that deals with various issues of social networks such as identication of communities
and their in-between relations, roles, etc. Blockmodeling focuses mainly on the network structure, but
it can also deal with node attributes and multiple relations. Another modern approach to role analysis
uses unsupervised hierarchical bayesian models, mainly on textual datasets. The authors argue that the
relational structure is not enough when analyzing textual datasets, such as emails, blogs, scientic papers.
Theideaistousethetextualcontentassociatedtothegraphnodesinadditiontotherelationalstructure.
One of the most prominent approach for role analysis for structure roles is to use the connectivity
between the users in some form and then dene the roles based on those metrics. [Welser et al., 2007]
use visualization methods to reveal these structural signatures and regression analysis to conrm the rela-
tionship between these signatures and their associated roles in Usenet newsgroups. Their analysis focuses
on distinguishing the signatures of one role from others, the role of \answer people". Answer people are
individuals whose dominant behavior is to respond to questions posed by other users. [Kelly et al., 2006]
explores three social roles: the ghters, the friendlies, and the fringe. The role of ghters represents the
great majority of actors who are the ones that respond more often to opponents rather than to allies. The
role of friendlies refers to a smaller group of actors who respond to allies more often than to opponents.
And, nally, the fringe represents a marginal group that raises interesting questions for qualitative study.
They focus on the in-degree and out-degree egocentric networks with each node containing the actor's
politicalaffiliation. [Chou and Suzuki, 2010]proposeanewmethodforidentifyingtheroleofavertexina
13
socialnetwork. Theyproposethreecommunityorientedroles,bridges,gateways,andhubs,withoutknowl-
edgeonthecommunitystructure,forrepresentingverticesthatbridgecommunities. [Hecking et al., 2015]
explore network analysis methods for the analysis of emergent themes as well as types of users in discus-
sion forums. They extract keywords from forum threads and then link to the forum users resulting in a
bipartite network based on their activity in discussion threads. [Zohrabi Aliabadi et al., 2013] study the
Enron dataset for classifying organizational roles by creating a feature vector containing a set of social
network metrics and interaction-based features re
ecting users' engaging-ness and responsiveness in their
community. [Choobdar et al., 2015] argue that the structural patterns in the neighborhood of nodes may
already contain enough information to infer users' roles, independently from the information
ow in itself.
They examine how network characteristics of users affect their actions in the cascade. [Tinati et al., 2012]
develop a model based upon the Twitter message exchange which enables to analyze conversations around
specic topics and identify key players in a conversation. This helps categorize Twitter uses by specic
roles based on their dynamic communication behavior rather than an analysis of their static friendship
network. [Labatut et al., 2014] study the position of social capitalists on Twitter with respect to the
communitystructure of the network. They use an unsupervised approachto distinguish the roles, in order
to provide more
exibility relatively to the studied system. [Zhao et al., 2013] investigate the social roles
and statuses that people act in online social networks in the perspective of network structures, since the
uniqueness of social networks is connecting people. They qualitatively analyze a number of key social
principles and theories that correlate with social roles and statuses. [Doran, 2015] present a data-driven
approach for the discovery of social roles in large scale social systems. The method discovers roles by
the conditional triad censuses of user ego-networks, which is a promising tool because they capture the
degree to which basic social forces push upon a user to interact with others. Clusters of censuses, inferred
from samples of large scale network carefully chosen to preserve local structure properties, dene the
social roles. [Revelle et al., 2016] analyze dynamic networks from two datasets (Facebook and Scratch)
to nd roles which dene users' structural positions. Each dynamic network is partitioned into snapshots
and social roles are found in each network snapshot independently. [Rossi et al., 2012] propose a scalable
14
non-parametric approach to automatically learn the structural dynamics of the biological or technological
network and individual nodes. Their approach learns the appropriate structural role dynamics for any
arbitrary network and tracks the changes over time.
Another popular approach is to create clusters of users based on some pre-dened criteria and then
label the users by characterizing them. [Smith Risser and Bottoms, 2014] explore the relationship be-
tween how a member participates in virtual blog network and the role of that member in the network.
They utilized cluster analysis to combine behavior and structural information in detecting roles: newbie,
inbound participants, full participants, celebrities, and peripheral participants. They show how an indi-
vidual who participates in the network has an in
uence not only on their current role in the network,
but also in how and how quickly their role in the community changes. [Stadtfeld, 2012] propose a new
method to identify latent behavioral roles in dynamic social networks. The approach analyzes the in-
dividual choice patterns of actors and clusters them based on their behavioral similarity in a cell phone
communicationnetwork. [Maia et al., 2008]proposeamethodologyforcharacterizingandidentifyinguser
behaviors in online social networks. They use a clustering algorithm to group users that share similar
behavioral patterns. Next, they show that attributes that stem from the user social interactions, in con-
trast to attributes relative to each individual user, are good discriminators and allow the identication of
relevant user behaviors. [White et al., 2012] develop a methodology to cluster together users with similar
ego-centric network structure. They use a mixed membership formulation which allows for the fact that
differentgroupsofusersmayhavecharacteristicsincommon. Someresearchersdeneparticipants'behav-
iorbasedontheiractivityandthenusenetworkbasedanalysisofthesebehaviorstodenethesocialroles.
[Viol et al., 2016] derive 16 metrics characterizing the participation behavior, message content and struc-
tural position of Enterprise Social Network users of an Australian professional service rm. They identify
four distinct dimensions of user behavior: Contribution & networking, information provision, contact dis-
persion, andinvisibleusage. [Buntain and Golbeck, 2014] exploreusers' postingbehavioron reddit. They
demonstrate that the well-known \answer-person" role is present in the reddit community, provide an
exposition on an automated method for identifying this role based solely on user interactions and show
15
thatusersrarelyexhibitsignicantparticipationinmorethanonecommunity. [Angeletou et al., 2011]use
statistical analysis, combined with a semantic model and rules for representing and computing behavior
in online communities. They categorize behavior of community members over time, and report on how
different behavior compositions correlate with positive and negative community growth in these forums.
[Fuger et al., 2017] investigate the network structure of crowdsourcing community and detect behavioral
patterns and user roles based on participation behavior that aims to enhance conditions in low income
communities. They illustrate that context and purpose of crowdsourcing initiatives may in
uence the
behavioral pattern of users. A few researchers have used the fuzzy theory to create classication models
for identifying the roles. [Wang et al., 2010] adopts fuzzy classication method and construct a hierarchy
for role classication. They advocate of utilization of social roles by considering their identiable features
at different levels. [Fazeen et al., 2011] present two methods for classication of different social network
actors such as leaders, lurkers, spammers, and close associates. They propose a two-stage process with
fuzzy-set theoretic approach to evaluation of the strengths of network links followed by a simple linear
classier to separate the actor classes. They also present a method that performs actor classication by
matchingtheirshorttermtweetpatternswiththegenerictweetpattersoftheprototypeactorsofdifferent
classes.
Some of the studies use non-conventional approaches to use the social networks' structure to dene
theirsocialroles. [Vega et al., 2016]extendtheconceptsofpositionandroleinanetwork, basingthemon
various well-known measures such as geodesic distance and modularity. [Mathioudakis and Koudas, 2009]
formalize notions of `starters' and `followers' in social media. They focus on the development of random
samplingapproachesallowingthemtoachievesignicantefficiencywhileidentifyingstartersandfollowers.
[Gilpin et al., 2013] provide an alternative least squares framework that allows convex constraints to be
placed on the role discovery problem, which can provide useful supervision. They explore supervision to
enforcesparsity,diversity,andalternativenessintheroles. [Koch et al., 2013]investigatetheheterogeneity
of community participants, by deducing typical roles, the development over time and possible in
uences
on the overall community building process. They nd different user roles to differ in kind and quality of
16
their contributions in creating, shaping, and disseminating Open Government activities. [Sun et al., 2016]
present a transfer learning approach to network role classication based on feature transformations from
each network's local feature distribution to a global feature space.
2.3 Identifying Specic Roles
Severalworksdealwiththeidenticationofaspecicroleinsideasocialnetwork. Theseworksusevarious
metrics in order to detect certain criteria satised by some users. Here is an overview of two specic roles:
Experts, and In
uencers.
2.3.1 Experts
Experts are the people to whom the other social network members will go to in order to seek advice or
help. The need for experts is often seen in forums dealing with technical or even health issues. Within
suchonlinecommunities,thepostingsarequestionsoranswersonacertainsubjects. Knowingtheexperts
facilitates the identication of the answers that are more likely to be correct and/or complete. Moreover,
differentiating the quality replies amongst hundreds of other postings allows a reader to quickly nd out
the postings worth being read.
Reseachers have used both content and network driven methods to identify the experts in online com-
munities. [Zhang et al., 2007] test a set of network-based ranking algorithms, including PageRank and
HITS, on a large size social network in order to identify users with high expertise. They then use sim-
ulations to identify a small number of simple simulation rules governing the question-answer dynamic in
the network. [Worsley and Blikstein, 2011] report on an exploratory analysis of learning through mul-
tiple modalities: speech, sentiment and drawing. A rich set of features is automatically extracted from
the data and used to identify emergent markers of expertise. Some of the most prominent markers of
expertise include: user certainty, the ability to describe things efficiently and a disinclination to use un-
necessary descriptors or qualiers. [Li et al., 2015] present an approach to discover expertise network
17
in online communities based on textual information and social links. In addition to computing docu-
ments' topic-focus degree, the approach measures the quality of documents according to users' feedback
behaviors and topic-specic in
uence of users who give feedback. [Hennis, 2011] proposes a method to en-
able evaluation of contributions in online knowledge-based communities by using authority and specifying
reputation on the keyword-level. [Liu et al., 2014] introduce a three-stage framework that automatically
generates expertise proles of online community members. In the rst two stages, document-topic rel-
evance and user-document association are estimated for calculating users' expertise levels on individual
topics. Inthethirdstagetheytestwhetheralteringstrategycanimprovetheperformanceofexpertpro-
ling. Recommendation systems are one of the primary platforms to use methods to identify the experts.
[Budalakoti et al., 2009]presentsagenerativemodelforcharacterizingtheproductionofcontentinanon-
line question and answer forum and a decision theoretic framework for recommending expert participants
while maintaining questioner satisfaction and distributing responder load. [Kumar and Pedanekar, 2016]
introducetheideaofminingshapesofuserexpertiseinatypicalonlinesocialQuestionandAnswer(Q&A)
community where expert users often answer questions posed by other users. They report observations on
distribution of different shapes of expertise in a StackExchange community called Super User.
2.3.2 In
uencers
The in
uencers are the right people to market to, the \market-movers". They are the ones that can
accelerate the diffusion of innovation whether this involves the launching of new products or novel mar-
keting, social, and political ideas. Knowing the in
uencers can lead to the reduction of the lag between
knowing (be informed) and doing (accept and apply a new idea), and, thus, the spread of new ideas
becomes quicker and more efficient. [Rosenthal, 2014] explores in
uence in discussion forums, web-blogs,
andmicro-blogsusingcomponentslikeauthortraits, agreement, claims, argumentation, persuasion, credi-
bility, and certain dialogue patterns. She classies in
uencers across ve online genres and analyzes which
features are most indicative of in
uencers in each genre. The most popular type of in
uencers researched
18
is the concept of an opinion leader. [Hung and Yeh, 2014] propose a text mining-based approach to eval-
uate features of expertise, novelty and richness of information from contents of posts for identication
of opinion leaders. [Shaikh et al., 2012] present a novel approach towards the detection and modeling of
complex social phenomena in multi-party discourse, including leadership, in
uence, pursuit of power and
group cohesion. They develop a two-tier approach that relies on observable and computable linguistic
features of conversational text to make predictions about sociolinguistics behaviors such as Topic Con-
trol and Disagreement, that speakers deploy in order to achieve and maintain certain positions and roles
in a group. [Towhidi and Sinha, 2015] argue that textual characteristics of reviews, along with reviewer
characteristics, could be used to predict opinion leadership. They propose a predictive model to iden-
tify opinion leaders in online word-of-mouth communities using both review and reviewer characteristics.
Some researchers use variations of page rank algorithm to identify in
uencers. [Song et al., 2007] propose
anovelalgorithmcalledIn
uenceRanktoidentifyopinionleadersintheblogosphere. Thealgorithmranks
blogs according to not only how important they are as compared to other blogs, but also how novel the
information they can contribute to the networks. [Chen et al., 2014] propose a novel method to identify
opinion leaders from online comments based on both positive and negative opinions. In this method, they
rst construct a signed network from online comments, and then design a new model based on PageTrust,
called TrustRank, to identify opinion leaders from the signed network. [Aleahmad et al., 2016] propose
an algorithm, OLFinder, that detects the main topics of discussion in a given domain, calculates a com-
petency and a popularity score for each user in the given domain, then calculates a probability for being
an opinion leader in that domain by using the competency and the popularity scores and nally ranks the
users of the social network based on their probability of being an opinion leader.
Some researchers rely on the network of the community to identify the in
uencers. [Song et al., 2012b]
focus on modeling comment network with explicit and implicit links for detecting the most in
uential
comment dynamically, and modeling user network and clustering users for detecting the most in
uential
user. They propose an approach with sentiment analysis, explicit and implicit link mining for modeling
comment network in Chinese news comments. [Volpentesta and Felicetti, 2012] model a time-dependent
19
commercial social network as a time-varying weighted directed graph. They also propose an approach to
determine opinion leaders and their contributions to a temporal business value, by taking into account
behavioral and structural aspects of the commercial social network. Few studies combine the content and
network based approaches to identify the in
uencers. [Duan et al., 2014] propose a method to recognize
opinion leaders in Web-based stock message boards by combining clustering algorithm and sentiment
analysis. [Jing and Lizhen, 2014] propose a hybrid data mining approach based on user feature and
interaction network, which includes three parts: a way to analyze users' authority, activity and in
uence,
a way to consider the orientation of sentiment in interaction network and a combined method based on
HITS algorithm for identifying micro blog opinion leaders. [Zhang et al., 2013] extract the communities
by analyzing the replies of each post in the bulletin board system. Then, an opinion leader community
mining method is proposed based on the level structure. Thus, the communities have better overlaps and
multiple relations.
2.4 Neural Networks
Deep learning methods are becoming important due to their demonstrated success at tackling complex
learning problems. At the same time, increasing access to high-performance computing resources and
state-of-the-art open-source libraries are making it more and more feasible for everyone to use these meth-
ods. ANeuralNetworkisabiologicallyinspiredprogrammingparadigmwhichenablesacomputertolearn
from observed data. It is composed of a large number of interconnected processing elements, neurons,
workinginunisontosolveaproblem. AnArticialNeuralNetworkisconguredforaspecicapplication,
such as pattern recognition or data classication, through a learning process. To date there hasn't been
anyresearchfocusedonusingneuralnetworkstoidentifysocialroles. However, therearemanyworksthat
present neural network models that can be used to analyze participants' contribution to identify specic
characteristics of the participants and in turn their roles. [Kalchbrenner et al., 2014] use the Dynamic
Convolutional Neural Network (DCNN) for the semantic modeling of sentences. The network handles
20
input sentences of varying length and induces a feature graph over the sentence that is capable of explic-
itly capturing short and long-range relations. They experiment their network with various settings like
predictingsentimentofmoviereviews, categorizingquestionsinsixquestiontypes, andpredictingthesen-
timentoftwitterposts. [Zhang et al., 2016]presentDependencySensitiveConvolutionalNeuralNetworks
(DSCNN) as a general-purpose classication system for both sentences and documents. DSCNN hierar-
chically builds textual representations by processing pre-trained word embeddings via Long Short-Term
Memory networks and subsequently extracting features with convolution operators. [Kim, 2014b] report
on a series of experiments with Convolutional neural networks (CNN) trained on top of pre-trained word
vectors for sentence-level classication tasks. They show that a simple CNN with little hyperparameter
tuning and static vectors achieves excellent results on multiple benchmarks. [Lai et al., 2015] introduce a
recurrent convolutional neural network for text classication without human-designed features. In their
model, they applyarecurrentstructureto capturecontextualinformationasfar aspossiblewhenlearning
word representations, which may introduce considerably less noise compared to traditional window-based
neural networks. [Amir et al., 2016] introduce a deep neural network for automated sarcasm detection.
They propose to automatically learn and then exploit user embeddings, to be used in concert with lexi-
cal signals to recognize sarcasm. [Lee and Dernoncourt, 2016] present a model based on recurrent neural
networks and convolutional neural networks that incorporates the preceding short texts for sequential
short-text classication, and evaluate it on the dialog act classication task. [Cocarascu and Toni, 2017]
propose a deep learning architecture with two Long Short-Term Memory networks to capture argumenta-
tive relations of attack and support from one piece of text to another, of the kind that naturally occur in
a debate.
2.5 Summary
To summarize, each social role has a set of expected behaviors and attitudes, and the functional role
theory techniques dene social roles through the characteristic behavior of participants. Furthermore,
21
socialrolescanbedirectlyrelatedtosocialpositions, assuggestedbystructuralroletheory, byassociating
participants'structuralpositionswithroles. Incontrasttopriorwork,Iformalizeaframeworkforassigning
social roles to participants in online discussions by identifying behavioral characteristics that dene these
social roles. In addition, I also develop methods to automatically identify the social roles by quantifying
these behavioral characteristics using the participants' contributions in such discussions. This allows to
provide a detailed analysis of the behavioral diversity of the participants over time along with the factors
that affect their behaviors.
22
Chapter 3
Corpus for Social Roles
The ndings of this chapter have been published at LREC 2014: [Jain et al., 2014]. The corpus is freely
available at http://www-scf.usc.edu/
~
siddhajj/. The original paper consisted of 80 Wikipedia dis-
cussions. The stats for Wikipedia corpus in this chapter have been updated to incorporate the additional
annotationof80moreWikipediadiscussionsthatwascarriedoutlaterfortheneuralnetworkexperiments
presented in Chapter 7.
3.1 Introduction
The previous chapter introduced the idea and importance of Social Roles in online environments, and
providedanoverviewofmethods,conceptsandtechnologiesthathavebeentraditionallyusedandrecently
implemented in this area towards the goal of identifying Social Roles.
This chapter provides an overview of the corpus used to analyze and discover the behavioral char-
acteristics and social roles of participants in online contentious discussions. This chapter describes the
annotation procedure adopted to annotate the corpus for social roles. Then it presents the resulting
model explaining different behavioral characteristics that help distinguish social roles from one another
along with all the social roles. In the end, an example discussion is presented and the social roles for all
the participants are identied.
23
3.2 Corpus
The corpus presented here consists of two sets of online contentious discussions. The rst set consists
of discussions from Wikipedia: Articles for Deletion (AfD)
1
forum. Wikipedia, being a very large peer
production system, has its own decentralized governance system to maintain quality of articles created by
the users. AfD is a forum where Wikipedians discuss and debate about whether an article should be kept
on Wikipedia. The second set consists of discussions from 4forums.com
2
discussion forum. This forum
hosts a variety of political and controversial topics for discussion where anyone can express their views
and debate about the topic in hand.
3.2.1 Wikipedia: Articles for Deletion (AfD)
Wikipedia is a free-access, free content internet encyclopedia. Except for particularly sensitive and/or
vandalism-prone pages that are \protected" to some degree, Wikipedia visitors can edit articles even if
they do not have a Wikipedia account. Therefore it becomes necessary for Wikipedia to keep check on the
kind of content put on the website and also verify its authenticity. Over years Wikipedia has developed
guidelines and policies to help editors collectively decide whether topics are suitable for inclusion or not.
All articles, especially new ones, are reviewed by the community to determine if they meet Wikipedia's
notability guidelines
3
. Wikipedia: Articles for Deletion serves the purpose of keeping Wikipedia \clean"
by allowing any Wikipedia user to nominate any article on the website for deletion. If the nomination
is legitimate, a community discussion takes place where any fellow editors have the opportunity to make
their voices heard. The usual process is to have a week long discussion during which community members
can discuss in favor or against keeping the article. Participants traditionally declare their stance in their
rst contribution to the discussion. A participant can take any of the following stances:
Keep: suggesting that the article should be kept.
1
Wikipedia: Articles for Deletion: http://en.wikipedia.org/wiki/Wikipedia:Articles_for_deletion
2
4forums.com: http://www.4forums.com/political/
3
Wikipedia's notability guidelines: http://en.wikipedia.org/wiki/Wikipedia:Notability
24
Delete: suggesting that the article should be deleted.
Merge: suggesting that the article should be merged with another existing article.
Redirect: suggesting that the article should be redirected to another existing article.
Transwiki: suggesting that the article should be moved to another Wikipedia project.
Intheend,aWikipediauserwithadministrativeauthorityreviewsthediscussionanddeclaresthedecision
thatre
ectstheconsensusoftheusersparticipatingin thediscussion. Theresultofthediscussionscanbe
one of Keep,Delete,Merge,Redirect,Transwiki,Withdrawn (suggesting the nominator withdrew
his nomination), or No Consensus (suggesting that no consensus can be reached as per the discussion).
Figure 3.1 shows an example discussion Wikipedia discussion.
Figure 3.1: An example discussion on Wikipedia: AfD forum.
25
AfD discussions are public, well written with good language, well-structured, and freely available. I
collected all AfD discussions that took place in the period of January 1, 2009 to June 30, 2012. It contains
92066discussionswith811047distinctcommentsand47066distinctparticipants. Eachdiscussionincludes
the title of the article in question, the nominator of the discussion, all the comments in the discussion,
and the nal outcome alone with the admin who imposed it. Each comment in the discussion includes
the participant who posted it, the stance of the participant if specied, the time of the comment, and
the level of the comment. The level refers to information about whether the comment is a new thread or
a reply to some previous comment/reply. The data extracted was veried by cross checking 25 random
discussions with the original discussions on the website.
The dataset consists of a total of 457668 distinct instances of stance. Since merge, redirect, and
transwiki combine to a total of only 7.16% and as they rarely occur together in the same discussion, I
simplied the data by combining them to the single stance compromise. Similarly, the outcome of the
discussion was also converted to compromise if appropriate. Also, as an outcome, withdraw implicitly
means keeping the article. Therefore, it was converted to keep. I create a timeline for each discussion
based on the chronological order of each comment. Whenever participants state their stance for the rst
time in the discussion, it is propagated to their subsequent comments unless they explicitly state a change
in stance.
3.2.2 4forums.com
4forums.comisawebsiteforpoliticaldebateanddiscourse. Thesiteisafairlytypicalinternetforumwhere
people post some discussion topic, other people post responses and a conversation ensues. Each discussion
is a tree structure enabling people to respond to comments out of order and engage in conversations. The
forum software facilitates this by providing an option to view the discussion in \threaded mode". One
important feature in this forum is a mechanism for quoting other participant. A participant may decide
to link to and replicate a previous comment in whole or in part. This establishes very precise context
which is very helpful for forum participants and NLP applications alike.
26
I extracted the discussions from an already constituted Internet Argument Corpus
4
. The dataset
consists of 390704 posts in 11800 discussions by 3317 authors. Each discussion includes the title of the
topic under debate, the initiator of the discussion, and all the comments in the discussion. Each comment
in the discussion includes the participant who posted it, the time of the comment, and the quotes from
other comments.
Figure 3.2: An example discussion on 4forums.com forum.
Figure 3.2 shows part of the a discussion on 4forums.com. One of the differences between Wikipedia:
AfD discussions and discussions on 4forums.com is that the participant doesn't have to state his stance
regarding the topic. Participants can choose to post a new comment or they can reply to a previous
4
Internet Argument Corpus: http://nlds.soe.ucsc.edu/software
27
comment by another participant. Figure 3.3 shows the \threaded view" of the same debate. We can see
the comment-reply structure of the debate in the gure.
Figure 3.3: \Threaded view" of a discussion on 4forums.com
Iselectedtheseratherdifferentcorporainordertoensurethatthemodelofsocialrolesandtheirsignals
holds up in general. Although both sets consist of contentious discussions, the nature of the discussions is
very different. Where the AfD discussions have a measured and polite tone, the 4forums.com discussions
can become quite heated and ad hominem. Participants on the Wiki forum are goal-oriented i.e. they
wanttheirstancetobethenalconsensusofthediscussions, whileparticipantsonthe4forums.comforum
are opinion-oriented i.e. they are primarily focused on presenting their own viewpoint. This is a classic
example of argumentation by reason vs. argumentation by insistence.
3.3 Annotations
The annotations began with a training annotation set consisting of 10 AfD articles. Three annotators;
one undergraduate student, one post-graduate student, and one PhD student; were asked to identify
two basic social roles performed by participants (Leaders and Rebels, where Rebels were described as
the participants who make \enough" contributions but are still unable to exercise any kind of in
uence
28
towards other participants or the outcome of the discussion). Also, they were asked to assign any other
role that would identify a participant with characteristics different from the two given roles.
After the completion of the initial task, the annotators agreed upon a set of social roles for the initial
coding manual. The annotators came up with a set of behavioral characteristics that dene each role and
alsothecriteriatoassignvaluestoeachcharacteristicforeachparticipant. Usingtheinitialcodingmanual,
the annotators were asked to annotate 8 more sets, each consisting of 10 AfD articles. After annotating
each set, the annotators discussed the annotations and rened and/or added any roles, characteristics,
or criteria that were agreed to be helpful. After completing all 8 sets, the annotators re-annotated all
the discussions again using the nal coding manual described in Appendix A. The nal coding manual
included 4 characteristics and 8 social roles, described later in the chapter. Later, one of the annotator
annotated 80 more Wikipedia discussions for the neural network experiments presented in Chapter-7.
Wikipedia 4forums.com
#Discussions 160 10
#Participants 1668 174
#Comments 2982 624
#Words 203143 51659
Table 3.1: Corpora stats.
The same annotators started the annotations for the 10 discussions from the 4forums.com forum using
thesame codingmanual. Some ofthecriteria for assigningvaluestocharacteristicsweremodiedinorder
to adapt to the different style of 4forums.com discussions but the set of characteristics and social roles
remained the same. In the annotation process, each annotator determines the value of each characteristic
for each participant in the discussion, and subsequently assigns the corresponding role. Table 3.1 provides
some statistics for the corpora and Table 3.2 presents the social roles distribution in the same.
29
Social Role Wikipedia: Afd 4forums.com
Leader 187 25
Follower 278 13
Rebel 344 49
Idiot 32 8
Voice in Wilderness 65 1
Nothing (Sensible) 950 89
Nothing 200 23
Other 63 4
Table 3.2: Social roles distribution in corpora.
Theinter-annotatoragreementforannotated social rolesiscomputedusing Cohen'sKappacoefficient.
Cohen'sKappameasurestheagreementbetweentwoannotatorswhoeachclassifyNitemsintoCmutually
exclusive categories.
Kappa ()
Annotator1 - Annotator2 0.74
Annotator2 - Annotator3 0.71
Annotator1 - Annotator3 0.69
Table 3.3: Cohen's Kappa score for social roles' annotation.
The equation for is:
=
Pr(a)Pr(e)
1Pr(e)
(3.1)
where, Pr(a) is the relative observed agreement among raters, and Pr(e) is the hypothetical probability
of chance agreement, using the observed data to calculate the probabilities of each annotator randomly
30
saying each category. If the annotators are in complete agreement then = 1. If there is no agreement
among the raters other than what would be expected by chance (as dened by Pr(e)), = 0. Table 3.3
shows the agreement score between the annotators for both corpus together. A kappa value of anything
above 0.8 is usually considered \good agreement" in NLP.
3.4 Behaviors and Social Roles
As annotation progressed, the annotators increased the number of roles and rened their denitions and
behavioralcharacteristicstheyarecomprisedof. Eventually,theannotatorsproducedthefollowingprinci-
pal roles that accommodated for all types of contributions in such discussions: Leader,Follower,Rebel,
Idiot,Voice in Wilderness,Nothing (Sensible),Nothing, andOther. These roles are dened later
inthechapter. Diggingdeeper, theannotatorsidentiedthreeaspectsofparticipant'overallcontribution,
further subdivided into four identiable characteristics, which in various combinations re
ect the behav-
iorsofthesesocialroles: ParticipationType(Stubbornness(St), Sensibleness(Se)),Attendedness
Value (Ignored-ness(Ig)), and In
uence Value(In
uence(In)).
3.4.1 Participation Type
This aspect helps determine the type and amount of contribution the participants make and what their
style of participation is.
3.4.1.1 Stubbornness
Stubbornness captures the intransigence of a participant in the discussion. This characteristic has two
components: theamountofparticipationandthedegreeofunwillingnesstochangeopinionorstance. This
characteristic differentiates between participants who form the heart of the discussion from participants
who may comment only once or twice, or in minor ways only. A combination of the following criteria
31
determines participation: the number of comments by the participant, the arguments/claims presented
by the participant, and the level of engagement with other participants.
The annotators compare the number of comments by each participant against the average number of
comments for each discussion, if the number of comments is higher than the average, the participant is
considered stubborn. However, note that while calculating the average, the annotators do not consider
the outliers. For example, if most of the participants comment between 1-5 times, but there is a partic-
ipant who comments 15 times, then while calculating the average, the participant with 15 comments is
considered an outlier. When counting the number of comments by a participant, the annotators do not
accountforcommentsthatarenotconsideredstubborn,i.e.,thecommentsthatdonotpresent/strengthen
their arguments. Some examples of these may be a query asking for information, notes stating changes in
the Wikipedia article, a comment just endorsing some other participant, or a comment having no argu-
ments/claims. The rst two comments in Figure 3.4 show such examples from the Wikipedia corpus and
the third comment shows the same from 4forums.com.
Figure 3.4: Examples of non-stubborn comments
The rst comment in the gure by andy asking someone to clarify something. This does not fortify his
stanceinanymannerinthediscussionandthereforethiscommentisconsideredanon-stubborncomment.
Similarly, the second comment by llairs is a note to notify everyone in the discussion about the change
in the article. And the last comment by fallingupwards is just a word which has no arguments/claims in
32
it. Thus, all the comments that doesn't have arguments/claims that forties the stance of the participant
are considered non-stubborn comments.
If a participant's number of comments is lower than the average, but the length of the comment(s) is
large enough and their arguments/claims either strengthen the case for their own stance or weaken the
case of the opposite stance, then they are considered stubborn. This is because this captures the level
of commitment from the participant towards his/her own stance. Alternatively, if most of the comments
made by a participant are just responses to other participants' questions and do not really support their
own stance, then those comments are not considered as contributing towards the stubbornness of the
participant. The annotators also consider whether the participant only presents his/her own arguments
or he/she also replies to others' comments to further support his/her stance.
Thepossiblevaluesforstubbornnessaredenedtobe+1,0,or-1,where+1meansthattheparticipant
is stubborn, -1 means that the participant is not stubborn and 0 means that the stubbornness level for
the participant cannot be determined. A very coarse-grained coding scheme is chosen specically to avoid
complexities introduced by more nuance and more potential for disagreement. This three-way distinction
is quite enough to determine roles accurately.
3.4.1.2 Sensibleness
The level of sensibleness of the arguments/claims presented by a participant is very important for measur-
ing his or her impact on the discussion. Sensibleness analysis is dependent on the domain or nature of the
discussion as well. Therefore the annotators dene somewhat different criteria for assigning sensibleness
values in the two corpora. As mentioned, the AfD discussions are goal-oriented: each participant tries to
sway the decision of the discussion in favor of their own stance. Also, since Wikipedia pages should meet
the requirements stated in Wikipedia policies, as one would expect discussions on this forum sometimes
revolve around such policies. Therefore the main criterion for someone to be sensible in such discussions
is that they appeal to authority in support of their arguments/claims. Examples of such authority can
33
be Wikipedia policies, links to external sources of recognized expertise, etc. Figure 3.5 shows some exam-
ples of sensible comments from Wikipedia: AfD discussions. The rst comment in the gure refers to a
Wikipedia policy WP:PROF which contributes to sensibleness whereas the second comment is considered
sensible because it questions the sources of the article and notability which in turn refers to Wikipedia
policies.
Figure 3.5: Examples of sensible comments
Another criterion is to evaluate whether the type of argumentation used is reasonable, logical, expands
ordevelopsonthepreviousarguments,re
ectsavision;orwhetheritisanonsequiturorstartsatangential
discussion. Theannotatorsalsotrytodecidewhethertheyknowthedomain: dotheyseemknowledgeable?
Also, since the discussions on AfD include a wide range of topics, many of which the annotator may not
be familiar with, participants' analysis of arguments/claims presented by other participants, i.e., peer
review, also plays an important role in determining the sensibleness level. Further, if it seems that the
participant's contribution is guided more by his or her emotional and often inappropriate response to
others, e.g., if they directly attack a participant or show that they are hurt by others not adopting their
position or agreeing with them, these participants are considered not sensible.
In contrast, the discussions on 4forums.com are opinion-oriented, where participants primarily focus
on presenting their own opinions and reasoning, and do not seriously consider that of others except to
dispute it. In this domain the sensibleness analysis differs from the AfD forum in some ways. The
introduction of tangential discussions, common in 4forums.com discussions, does not have an effect on
the sensibleness analysis if the introduced tangential discussion does attract other participants. Figure
34
3.6 presents examples of non-sensible comments. The rst comment seems illogical whereas the second
comment is more of a personal attack towards some participant in the discussion.
Figure 3.6: Examples of non-sensible comments
Since 4forums.com has no argumentation policies to refer to and the topics for the discussions are
controversial socio-political issues, the sensibleness analysis is heavily dependent on the type of issue.
One of the criteria is to distinguish arguments that present personal preference as opposed to ones that
presentglobaleffects. Forexample,participants'argumentsmayre
ectlocal-levelreasoning(e.g.,personal
preferences such as participants wanting their children to have Christian education) as opposed to global-
level reasoning (e.g., deforestation having a major impact on global warming). This depends on the
nature of the issue: local-level reasoning about local-level issues (e.g., whether home schooling is good)
are considered sensible. However, participants' personal preferences stated for global-level issues (e.g.,
whether evolution and religion can coexist) are considered low on sensibleness. The annotation judgments
in this forum are perforce more intuitive than for the AfD corpus.
The possible values for sensibleness are +1, 0 or -1, where +1 means that the participant is sensible,
-1 means that the participant in not sensible and 0 means that the sensibleness level of the participant
cannot be determined.
35
3.4.2 Attendedness Value
This aspect is concerned with how other participants attend to a participant's contribution.
3.4.2.1 Ignored-ness
Ignored-ness captures the attitude of participants towards each other. This attitude can be an indicator
of the importance or relevance of a comment. Generally, for example, participants ignore spammers. The
main criterion for ignored-ness analysis is to see whether the participant receives any replies to his or her
comments. Therefore, when someone mentions a participant in his/her comment then that participant
is considered not ignored. Wikipedia:AfD and 4forums.com discussions are stored in a structured way
(repliestoacommentareindented),makingdeterminationofreplieseasy. Also,indiscussionsparticipants
quote arguments presented by another participant, or name them explicitly. These also indicate that the
latter is not ignored. Figure 3.7 shows some examples that indicate that a participant is not ignored:
participants Boston and Crim are being replied to in the comments shown.
Figure 3.7: Examples of not-ignored comments
The possible values for ignored-ness are +1 or -1, where +1 means that the participant is ignored and
-1 means that the participant is not ignored.
36
3.4.3 In
uence Value
This aspect helps in mainly to identify leaders and followers among the other roles. This has two aspects:
(i)in
uenceonothers: wasaparticipantabletoin
uenceanotherparticipantthroughtheircontribution?,
and (ii) endorsement: did a participant acknowledge another participant's in
uence explicitly? This
aspect helps mainly to identify leaders among the other roles. The primary characteristic of leaders in
contentious discussions is that they are able to in
uence others by their actions or arguments/claims.
Therefore in
uence analysis becomes a key part of identifying leaders and followers in such discussions.
3.4.3.1 In
uence
This characteristic deals with the question if the contributor is in
uential to other participants. The most
observable indication of in
uence occurs when another participant changes his/her stance during the dis-
cussion and acknowledges the in
uential participant for in
uencing him/her for the change. In such cases,
the participant who was in
uential in engendering the change is considered a leader and the participant
who changes his/her stance or endorses other participants is considered a follower. Another example of
indication of in
uence is when other participants acknowledge the in
uential participant through expres-
sions such as \according to ...", or \as per ...", etc. Participants who repeat arguments presented by other
participants and don't have any of their own arguments are also considered to be in
uenced by others.
Figure 3.8 shows some examples of in
uence indicators.
Figure 3.8: Examples of in
uence indicators
37
The possible values for in
uence are +1, 0, or -1, where +1 means that the participant was able to
in
uence another participant, -1 means that the participant got in
uenced by another participant, and 0
means that the participant was neither in
uential nor got in
uenced by others.
3.5 Social Roles
Asmentioned,thenalcodingmanualcomprises8principalrolesdistinguishedby4characteristics. Table
3.4showstherelationshipbetweenthebehavioralcharacteristics'valuesandthecorrespondingsocialroles.
Note that the roles Leader and Follower have a special status in that all the characteristics' values except
in
uence areunspecied. Thismeansthatanyparticipant,irrespectiveoftheirsensibleness, stubbornness
etc., may be seen as a Leader/Follower as long as they have the appropriate in
uence value. Hence any
other roles could additionally acquire the qualities of being a Leader or Follower.
Stubbornness Sensibleness Ignored-ness In
uence Social Role
x x x +1 Leader
x x x -1 Follower
+1 +1 -1 x Rebel
+1 +1 +1 0/-1 Voice in Wilderness
+1 -1 x x Idiot
-1 +1 x x Nothing (Sensible)
-1 -1 x x Nothing
Table3.4: Therelationshipbetweenthevaluesofbehavioralcharacteristicsandsocialrolescorresponding
to them.
Any combination of characteristics not specied in the table is annotated as the role Other. An \x" in
the table indicates that the corresponding characteristic for the role can take any possible value. As one
38
can notice, participants can thus have multiple roles. A Rebel can become Rebel-Leader if the participant
has in
uence value +1. Similarly, a Nothing can become Nothing-Follower if in
uence value is -1.
3.5.1 Leader
ALeaderisdenedasaparticipantwhomanagestoin
uenceanotherparticipanttochangehis/herstance
or in
uence them to follow him by endorsing through his/her actions or arguments/claims. The dening
characteristicofaLeaderistheamountofin
uencehe/sheisabletoinduceregardlesstheamountortype
ofcontribution. ThecharacteristiccombinationforaLeaderis(x,x,x,+1)forstubbornness,sensibleness,
ignored-ness, and in
uence respectively, where `x' means any value for that characteristic is allowed.
3.5.2 Follower
As opposed to a Leader, a Follower is dened as a participant who is in
uenced by other participants and
changes his/her stance or endorses other participants for their actions or arguments/claims. A Follower
is also a participant who doesn't have his/her own arguments/claims, but instead re-states arguments/
claims made by other participants. These participants provide support to leaders by endorsing them or
by re-stating the same arguments/claims. Therefore, the contribution amount or type doesn't matter for
such participants to dene them as a Follower, making the characteristic combination for a Follower (x,
x, x, -1) for stubbornness, sensibleness, ignored-ness, and in
uence respectively.
3.5.3 Rebel**
A Rebel is a participant who forms the heart of the discussion and drives it in some direction. One of the
main characteristics of a rebel is his/her devotion to the discussion, based on the amount of contribution
and their level of engagement with other participants. The arguments/claims presented by a rebel are
sensible and he/she is not ignored by other participants, which provides justication for the importance
**The original idea behind the label `Rebel' was to tag participants who are not leaders but have a signicant amount
of contribution. But later, in order to generalize the label of `Leader' across other labels (e.g., an Idiot-Leader or Nothing-
Leader), separate behavioral category for leader/follower was created and the label `Rebel' was used to refer to anyone who
has a signicant amount of contribution in the discussion.
39
of his/her presence in the discussion. The characteristic combination for a rebel is (+1, +1, -1, x) for
stubbornness, sensibleness, ignored-ness, and in
uence respectively.
3.5.4 Voice in Wilderness
A Voice in the Wilderness is very similar to a Rebel in the amount and type of contribution which forms
the heart of the discussion. The arguments/claims presented by a voice in wilderness are sensible as well.
The only difference between a Voice in the Wilderness and a Rebel is that the former is ignored by other
participants for some reason. Therefore the sensibleness value for a Voice in the Wilderness is important
to distinguish them from potential spammers, since spammers are never regarded as sensible contribu-
tors. The characteristic combination for a Voice in the Wilderness is (+1, +1, +1, x) for stubbornness,
sensibleness, ignored-ness, and in
uence respectively.
3.5.5 Idiot
An idiot is a participant whose contribution to the discussion does not serve the purpose of it. He/she
doesparticipatealotinthediscussionbutthecontentmaybeemotional, illogical, orcompletelyoff-topic.
The ignored-ness characteristic has no signicance in dening the role of an idiot since the main criterion
for labeling a participant an idiot is the non-sensible contribution by himself/herself. The characteristic
combination for an idiot is (+1, -1, x, x) for stubbornness, sensibleness, ignored-ness, and in
uence
respectively.
3.5.6 Nothing and Nothing (Sensible)
These are participants who make minimal contribution to the discussion and hence cannot be considered
stubborn enough to stick to their arguments/claims. As a result, they may not have a major in
uence
on the course or the outcome of the discussion. Nothing and Nothing (Sensible) are distinguished based
on the amount of sensible arguments/claims. The reason of having these two separate roles is to be able
to distinguish between potential spammers or trolls from those who may have minimal but legitimate
40
contribution. The characteristic combination for a nothing is (-1, -1, x, x) for stubbornness, sensibleness,
ignored-ness, and in
uence respectively, and the characteristic combination for a nothing (sensible) is (-1,
+1, x, x) for stubbornness, sensibleness, ignored-ness and in
uence respectively.
3.5.7 Other
The Other roleis dened for participantsfor whom valuesof certain characteristicscannot be determined.
These include combinations of characteristics where either the stubbornness value is 0 or the sensibleness
value is 0. The most common case where stubbornness cannot be determined occurs when a participant
changes his/her stance during the discussion. The most common case where sensibleness cannot be
determined occurs when there is not enough information in the contribution of the participant to perform
a sensibleness analysis.
Figure 3.9: An example discussion from Wikipedia corpus
41
3.6 An Example
I present an example discussion in gure 3.9 from Wikipedia dataset and analyze the role for each partic-
ipant contributing to the discussion in Table 3.5.
Participant Social Role Analysis
Timneu22 Nothing (Sensi-
ble)
- Not stubborn as has only minimal contribution
-SensibleasmentionsWikipediapolicyfordeletionargument
Shep Leader-Rebel - Leader as Vegaswikian endorses him
- Sensible as presents facts which are not disputed by anyone
else and gives a logical suggestion about reviving article if
needed later
Vegaswikian Follower-Nothing - Follower as he endorses Shep
- Not sensible as he doesn't have any of his own claims
John Idiot - Stubborn as he keeps emphasizing his point by repeating it
in multiple comments
- Not sensible as he keeps repeating the same point multiple
times and suggests to alter the guidelines to make his point
stand correct
Dennis The Tiger Nothing (Sensi-
ble)
- Not stubborn as has minimal contribution
-SensibleasmentionsWikipediapolicyfordeletionargument
ChildofMidnight Other -Stubbornnesscannotbedeterminedashechangeshisstance
in the end
- Sensible as backs his claims with guidelines
42
B Rebel - Stubborn as he replies to the participants who counter him
to fortify his claims
- Sensible as he provides proper reasoning for his claims
Scapler Nothing (Sensi-
ble)
- Not stubborn as has minimal contribution
- Sensible as mentions Wikipedia policy in his comment
Table 3.5: Participant roles for the discussion in Figure-3.9.
43
Chapter 4
Leadership Models
The ndings of this chapter have been published at SMMR 2013: [Jain and Hovy, 2013]. The leadership
modelspresentedinthischapterareusefulforanalyzingparticipants'in
uentialbehaviorinthediscussion
towards the other participants. The results of the model are used for in
uence model presented in chapter
6.
4.1 Introduction
Whatisaleader? Associalbeings,humansareadeptatrecognizingandrespondingtoleadershipingroup
settings. Yet leadership is surprisingly hard to dene or recognize computationally. Participants in online
decision making environment assume different roles. Especially in contentious discussions, the outcome
often depends critically on the decision leaders. I also want to monitor the structure and evolution of
groups, and leaders are important. Also, I want to build proles of individual people, and seeing how
they act in group situations is important. Recent work on automated leadership analysis has focused
on collaborations where all the participants have the same goal. In this chapter we focus on contentious
discussions, in which the participants have different goals based on their opinion, which makes the notion
of leader very different. The explosive growth of social multimedia, whether targeted or un-targeted (for
example, personal email and tweets respectively), has made available a wealth of material that can be
used for study. I use discussions on the Wikipedia: Articles for Deletion (AfD) forum for creating the
44
models and then use the annotated AfDs and 4forums.com discussions presented in Chapter 3 to test the
models. This chapter:
denes two complementary models of leadership, Content Leader and SilentOut Leader, in online
contentious discussions,
presents methods to automatically identify and quantify various characteristics of leaders,
merges these results to identify discussion leaders,
measures the correlation between leaders identied by the models,
proposes a method to verify the quality of leaders identied by the models,
veries the authenticity of identied leaders against the annotated corpus presented in Chapter 3.
4.2 Related Work
Identifyingleadersindifferentonlinesocialenvironmentshasattractedmuchresearchinrecenttimes. The
typesofOpinionLeadersthathavebeenexploredthemostcanbecategorizedas: leadersinsocialnetworks
and leaders in online discussion groups. [Lu et al., 2011] propose the LeaderRank algorithm to identify
leaders in social network by quantifying the user in
uence over network. [Clemson and Evans, 2012]
study a network version of minority games to identify the followers in the network and in turn identify the
leaders by determining the users who follows the smallest number of users. [Fazeen et al., 2011] present
context dependent and independent models to identify leaders, lurkers, spammers and group associates in
social networks. [Tsai et al., 2012] use probabilistic time based graph propagation model to build action
specic in
uence chains to determine leaders in social communities. [Song et al., 2012a] propose a data
structure called the Longest Sequence Phrase tree to measure similarities between comments made by
different users in order to identify positive opinion leader group in discussion forums. Another method
to identify groups and leaders in discussions is by [Arvapally et al., 2012] using the K-means clustering
45
algorithm, which uses users' provided weights for support/opposition to different opinions to formulate
factiongroupsandthenidentifyleadersineachgroupastheoneshavingthemostsupportfortheirgroup's
opinion. [Catizone et al., 2012] use social network analysis methods to infer social and instrumental roles
and relationships in online discussions.
The work that comes closest to the work presented in this chapter is by [Bracewell et al., 2012], who
dene psychologically motivated acts to determine social goals of individuals based on discourses' inten-
tional structure. Apart from network analysis methods, all the methods discussed here use annotators
to characterize user behavior in the social network or discussion forums and use that to train learning
systems. The models presented in this chapter are the rst models to quantify leadership automatically
and identify the leaders in an unsupervised way.
4.3 Corpus
In the analysis of leadership, I focus on the contentious discussions from Wikipedia: Articles for Deletion
(AfD) forum. The structure and purpose of AfDs make them ideal for leadership analysis as every
participant wants their stance to be the outcome of the discussion which motivates them to contribute to
the discussions as well as engage more often with the participants having a different stance.
4.3.1 Contentious Discussions
The outcomes of discussions on AfD are not decided based on the number of votes for each stance as
stated in the guidelines for the deletion process, but depend on the nature of the discussion. Hence the
content,
ow, and structure of the discussion become very signicant. I dene a contentious discussion as
any discussion containing at least one participant each for at least two stances. But not all contentious
discussions are suited for leadership analysis. If a stance shows an overwhelming majority, then there is
no incentive for a participant to take up the leadership role, as he would be sure that his goal would be
achieved.
46
Table 4.1 shows the relationship between the degree of majority for any stance in a discussion and the
outcome being the same as the majority for all the discussions where some consensus was concluded by
the admin for the discussion (i.e., neglecting the discussions where the outcome was no consensus). The
majority cannot be less than 1/3 as there are only 3 possible stances. Also, to resolve ties, I select the
majority stance as the one with the larger length of content in support of it. Evident from the table, we
can see that when the majority is over 60%, the outcome of the discussion is very much in favor of the
majority. This reduces the incentive for leadership behavior of any participant in the discussion.
Majority # of discussions % Accuracy
30-40% 463 38.23
40-50% 996 29.21
50-60% 6833 48.48
60-70% 11487 70.00
70-80% 9557 86.73
80-90% 10467 93.77
90-100% 43646 97.69
Table 4.1: Stance majority as factor for outcome of discussion.
Tofortify this claim, I analyze the average contribution by participantsin discussions. Table 4.2 shows
the average number of comments by a participant in discussions versus the majority of any stance in it.
It shows that when the discussion is very contentious, participants contribute more in order to sway the
outcome in their favor.
As described earlier, a participant can reply to a comment/reply from another participant. I analyze
the degree to which participants get direct replies from other participants having a different stance in the
discussion. This shows the amount of direct opposition to participants' arguments, which signies the
levelofcontentioninthediscussion. Fordiscussionshavingmajoritymorethan60%, 88.65%ofcomments
47
didn't get any direct opposing reply. However, for discussions whose majority comments number less than
or equal to 60%, only 60.46% of the comments didn't get any direct opposing reply. This shows that in
contentious discussions participants are not only focused on proving their arguments, but also are equally
focused on disproving others' arguments, which indicates potential leadership.
Majority Average participant comments
30-40% 2.23
40-50% 1.95
50-60% 2.11
60-70% 1.65
70-80% 1.52
80-90% 1.41
90-100% 1.32
Table 4.2: Majority vs Average participant comments.
The analysis shown here suggests that leadership incentive is highly in
uenced by the degree of con-
tention in the discussion. Therefore, for all the experiments described below I only consider discussions
in which the majority stance forms less than or equal to 60% of the total.
4.4 Leadership Models
I dene two principal leadership models of participants based on the basic qualities that re
ect the lead-
ership behavior of a person in contentious discussions. The idea is to quantify the degree of leadership for
each participant by assigning leadership points for such behavior instances.
48
4.4.1 Content Leader
Language can be a great indicator to recognize leadership qualities of a person. As suggested by the name
of the model, the Content Leader model quanties the language use of a participant to assign leadership
points. This model is built upon two basic characteristics of a leader: 1) Encourage others to follow his
arguments (Attract Followers) and 2) Counter the arguments from the opposing groups (Counterattack).
4.4.1.1 Attract Followers (AF)
One of the prime qualities of a leader is to attract followers by convincing them with his arguments.
This can be very useful in building majority in contentious discussions and thus swaying the outcome of
the discussion in his/her favor. I model this attribute by matching the n-grams (word sequences) of a
participant against the n-grams used by another participant having the same stance. The participant who
encourages other participants to support his argument by making them reuse the same phrases acquires
leadership points.
4.4.1.2 Counterattack (CA)
This attribute of the model quanties the quality to stand up against opponents and try to nullify the
arguments presented in oppose to the participant's stance. I model this attribute by matching the n-
grams of a participant against the n-grams of participants holding a different stance. The participant who
counterattacksacquires leadershippoints. In addition, the participantwho isgetting counterattackedalso
gets some leadership points. This is because of his/her ability to attract the attention of opposing groups,
which implies a signicant contribution in regards to the discussion.
Both these attributes implicitly quantify another ability of a leader, which is having command over
the course of the discussion. The fact that other participants having the same or different stances are
49
using the same argumentation words implies that the leader is guiding the discussion. The equation for
leadership points for Content Leader for a participant in discussions is dened as:
AF +CA
oppose
+
CA
opposed
(4.1)
AF = n-gram weight for n-grams used by other participants having the same stance, found in word n-
grams produced by the participant
CA
oppose
= n-gram weight for n-grams used by participant found also in n-grams of participants with
different stance
CA
opposed
= n-gram weight for n-grams used by other participants having different stance, found in n-
grams of participant
;;
= weights of corresponding attributes
4.4.2 SilentOut Leader
One of the techniques to win a discussion is to present arguments or facts that the opposition cannot
counter. One can also argue that countering arguments or facts presented by opposing participants so
that they cannot hold on to the same can also be a strategy to have a signicant impact in the discussion.
The SilentOut Leader model quanties the ability of a leader to silence opposing participants out with his
arguments. This is modeled by two attributes: 1) Giving arguments that cannot be countered (Factual
Arguments) and 2) Winning the small battles in the discussion (Small Wins).
4.4.2.1 Factual Arguments (FA)
The analysis presented earlier showed that participants counter the arguments given by opposing stance
participants more often in highly contentious discussions. So, if a participant presents an argument that
none of the participants from the opposing stances attack, it shows the quality of the argument which
silences out the opposing participants. This contributes to the leadership qualities of the participant in
50
the sense that he can give factual arguments that relate to the discussion and can have an impact on the
outcome of the discussion. I model this attribute by giving a constant amount of leadership points to
participants for each comment that elicits no reply from any opposing stance participant.
4.4.2.2 Small Wins (SW)
This attribute of the model refers to the ability of a leader to silence out other participants by countering
theirindividualargumentsandthuswinningasmallbattleoverotherparticipants. Thiscounterargument
not only nullies the original argument from the opposing participant but also strengthens the arguments
for the leader's own stance. To model this attribute, I divide the discussion into small argument sections.
Each argument section contains an original argument and at least one reply from a participant from an
opposing stance. For each such argument section, the participant who has the last say (i.e., whose reply
gets no counterargument) acquires a constant amount of leadership points.
TheequationfortheleadershippointsforSilentOutLeadermodelforanyparticipantinanydiscussion
is dened as:
FA+SW (4.2)
FA = # of comments from participant which didn't have any reply from any participant from opposing
stance
SW = # of small battles participant won
; = weights of corresponding attributes
4.5 Experiments
4.5.1 Setup
I create a timeline for each discussion based on the chronological order of comments. Each discussion
is processed comment by comment, so at any step, all the participants who have participated in the
51
discussion so far, their current stance, the n-grams that they have used so far, and the nesting level for
each comment (which denes argument sections) are known. To determine the weight of any n-gram, we
use the Inverse Document Frequency (idf)
1
of that n-gram across all the discussions in the whole corpus.
The idf value of an n-gram denoted by t is given by:
idf(t)=log
jNj
jn2N :t2nj
(4.3)
jNj = # of discussions in the corpus
jn2N :t2nj = # of discussions where the n-gram t appears
Iuseunigrams,bigrams,andtrigramsaspartofourn-gramsandcalculatetheweightsacrossthewhole
corpus, not only the contentious discussions. To deal with closed class words that are not important in
calculating the leadership points, I ignore all n-grams which appear in at least 25% of the discussions.
The motivation behind using idf values is to assign more importance to topic words that may be more
relevant to the discussion. I also maintain a window to determine a list of active participants for Content
Leader analysis. Any participants who haven't contributed in a long span in the discussion shouldn't
receive leadership points for new comments. Thus, if a participant hasn't contributed in the window of
consideration, he/she won't get any leadership points for any of the new comments. As the average length
of the discussions in consideration is 13.25, the window size is set to 10.
4.5.2 Content Leader calculation
Discussions are processed comment by comment, extracting n-grams from the text and storing them in
the bag of n-grams for the participant. I also keep track of the stance of each participant as the discussion
progresses. Given a new comment by participant B, the n-grams in this comment are extracted. For
each participant A, who has already participated in the discussion, the n-grams used by participant B are
matched to the n-grams in the bag of participant A. If participant B has the same stance as participant A,
1
Inverse Document Frequency (idf): http://en.wikipedia.org/wiki/Tf-idf
52
thenforeachn-gramofparticipantBthatmatchessomen-graminthebagofparticipantA,participantA
gets times the idf weight of the n-gram amount of leadership points, per the AF formula. If participant
B has a stance different from participant A, then for each n-gram of participant B that matches some
n-graminthebagofparticipantA,participantAgets
timestheidfweightofthen-gramandparticipant
B gets times the idf weight of the n-gram as their respective leadership points, per the CA formula.
While matching the n-grams with each other, I look for synonyms of words that might have been used
insteadoftheoriginalwords. IuseWordNet[Miller et al., 1999]toexpandtheoriginalwordwhichistobe
matchedandaddthesynonymsofthatwordinthelistwhichthenwouldbematchedwiththetargetword
which is to be matched with. Here, the absolute values of ;;
are not signicant. But their relative
values are important because that re
ects their relative importance to the corresponding attributes of the
model. Table 4.5 presents the analysis of different coefficients values along with the results.
4.5.3 SilentOut Leader calculation
To calculate the SilentOut Leader in a discussion, the whole discussion is divided into argument sections
based on the nesting level of each comment in the discussion. Thus, each comment and the replies to
that comment form one argument section. Now for each argument section, the participant who initiated
the argument section is identied, say participant A. If an argument section contains no replies from any
participant with a stance different from participant A (i.e., the argument section shows no contention),
thenparticipantAacquiresleadershippoints,pertheFAformula. Andifanargumentsectiondoeshave
replies from any participant with a stance different from participant A (i.e., the argument section shows
contention), then the participant who replied last in the chronological order gets leadership points, per
the SW formula.
Similar to the Content Leader model, the absolute values of and are not signicant. But their
relative values are important because they re
ect the relative importance of the corresponding attributes
of the model. Table 4.6 presents the analysis of different coefficients values along with the results.
53
4.6 Results
4.6.1 Correlation between Content Leaders and SilentOut Leaders
Using the two leadership models, I calculate leadership points for each participant in each of the 8292
contentious Wikipedia AfD discussions. The leadership points for each participant are then aggregated
across all the discussions and average leadership points per discussion is calculated. Using the average
score, I create a ranked list of leaders for both the models. The ranked lists are compared using the
Spearman's rank correlation coefficient
2
. For any two ranked lists, the coefficient is calculated as:
=16
∑
d
2
N(N
2
1)
(4.4)
d = difference in statistical rank of corresponding participant N = total number of participants in con-
sideration
Min participation # of participants
1 9489 0.23
5 2166 0.30
10 1144 0.46
20 533 0.48
50 183 0.64
Table 4.3: Spearman correlation between Leadership models.
Table 4.3 shows the Spearman's rank correlation between the Content Leader and SilentOut Leader
models. The rst column limits the participants by minimum number of different discussions they must
have participated in to be considered for correlation calculation. The second column shows the number
of participants satisfying the criteria for minimum number of discussions and the last column shows
2
Spearman's rank correlation coefficient: http://en.wikipedia.org/wiki/Spearman's_rank_correlation_coefficient
54
the correlation coefficient for those participants. Positive correlation coefficient implies that the models
complement each other for identifying same participants as leaders.
4.6.2 Predicting outcome of the discussion
This work presents a possible method to test the quality of leaders identied by each model. The criterion
toqualifyasaqualityleaderistobeabletoturntheoutcomeofthediscussionintoone'sfavor. Icompare
our models with the baseline models presented below:
Majority Stance: The outcome of the discussion is predicted as the stance with the majority votes.
To resolve ties, the majority stance is selected as the one which has larger length of content in
support of it.
Majority Content Stance: The outcome of the discussion is predicted as the stance which has the
largestnumberof wordsin support. Toresolveties, thestance withthe majorityofvotesisselected.
If a tie cannot be resolved, the stance is predicted with equal probability.
Talkative Leader: The participant who has contributed the most number of words in the discussion
is chosen as the leader of the discussion. The outcome of the discussion is predicted as the stance of
that leader.
Content Leader: The participant with the most leadership points per the Content Leader model is
chosen as the leader of the discussion. The outcome of the discussion is predicted as the stance of
that leader.
SilentOut Leader: The participant with the most leadership points per the SilentOut Leader model
is chosen as the leader of the discussion. To resolve ties, for leadership points, I choose as leader the
participant who has contributed a larger length of content in the discussion. The outcome of the
discussion is predicted as the stance of that leader.
55
Content and SilentOut Leader: Icalculatetheleadershippointsforeachparticipantinthediscussion
using both the models separately and then combine the leadership points of the model using the
following equation:
CL+SL (4.5)
CL = leadership points from Content Leader model
SL = leadership points from SilentOut Leader model
After experimenting with values of and , they were set to 1 and 2 respectively. The participant
withthehighestcombinedleadershippointsischosenastheleaderofthediscussion. Toresolveties,
the participant having greater SilentOut leadership points is chosen to be the leader because of the
higher individual accuracy of the model individually. The outcome of the discussion is predicted as
the stance of that leader.
Model % Accuracy
Majority Stance 45.59
Majority Content Stance 42.56
Talkative Leader 40.28
Content Leader 60.90
SilentOut Leader 65.01
Content and SilentOut Leader 68.34
Table 4.4: Comparison of discussion outcome prediction accuracy.
Table 4.4 shows the comparison of accuracies for predicting the outcome of Wikipedia AfD contentious
discussions based on the different models described above. Table 4.5 and Table 4.6 show an analysis of
relative values of model coefficients on the outcome prediction accuracy for Content Leader and SilentOut
Leader models respectively. I choose the coefficient values related to the best accuracy for each model to
56
create the leadership rankings, which are then used to calculate Spearman correlation between the two
models.

% Accuracy
1 1 1 53.59
1 1 2 52.88
1 2 1 59.60
1 2 2 55.33
2 1 1 54.68
2 1 2 53.48
2 2 1 60.90
Table 4.5: Comparison of accuracy for different coefficient values for Content Leader model.
% Accuracy
1 1 64.05
1 2 65.01
2 1 62.25
Table 4.6: Comparison of accuracy for different coefficient values for SilentOut Leader model.
4.6.3 Verifying authenticity of the leaders
I use the annotated corpus for social roles for Wikipedia: AfD forum and 4forums.com forum described in
Chapter 3 to verify the authenticity of the leaders identied by the models described earlier. The models
described in the chapter are not binary models where they make participants with leadership qualities as
leaders. Therefore almost every participant has some leadership points. So, verifying whether annotated
leaders from the corpus are leaders or not according to the models becomes meaningless. I verify the
57
authenticity of the leaders by the rank the models give them based on the leadership points they get. For
example, if a discussion has n numbers of annotated leaders, I say that only rst n participants in the
ranked list produced by the models are leaders according to the models and rest of the participants are
marked as non-leaders. Now for each annotated leader, I verify whether that participant is marked as
leader in the modied list. Thus, if an annotated leader is within top n ranks of the list produced by the
model, then I say that the model identies that participant as a leader as well.
Table 4.7 shows the accuracy for the described verication method for AfD discussions and Table
4.8 shows the accuracy for 4forums.com discussions. For each model in the table, I experimented with
different values for the coefficients for the corresponding models. I achieved the best accuracies with the
same coefficient values with which best accuracies for predicting outcome of the discussions are achieved.
Model % Accuracy
Content Leader 78.61
SilentOut Leader 71.66
Content and SilentOut Leader 80.75
Table 4.7: Accuracy for authenticity of leaders identied by models for Wikipedia: AfDs.
Model % Accuracy
Content Leader 83.65
SilentOut Leader 74.19
Content and SilentOut Leader 84.73
Table 4.8: Accuracy for authenticity of leaders identied by models for 4forums.com.
58
4.7 Conclusion
In this study, I construct two models to nd leaders in contentious discussions. The models quantify
the basic qualities of the leader behavior and assign leadership points to users participating in the dis-
cussions for each instance of such behavior. The results show promise for identifying users who succeed
in in
uencing the outcome of the discussions. Also, the similarity measure between the models implies
that the models complement each other to identify leaders based on different leadership qualities. The
authenticityoftheleaderstestedagainsttheannotatedcorporashowsthatthemodelscanidentifyleaders
in contentious discussions successfully.
59
Chapter 5
Claims and Argumentation Structure
The ndings of this chapter have been published as part of the sensibleness model paper published at
SocialNLP2016: [Jain, 2016]. Argumentationanalysisisoneofthemostimportantaspectfordetermining
sensibleness of participants in discussions. This analysis aims to identify whether the participants have
any claims in their assertions and how these claims are related to one another to present a structured and
coherent argument by the participants to strengthen their stance. The results of this analysis are used for
stubbornness and sensibleness models presented in chapter 6.
5.1 Introduction
In online discussions participants often try to persuade others. Persuasion and sensibleness play a ma-
jor role in social media discussions, in which people often try to convince others through their argu-
ments. Since antiquity, the discipline of rhetoric has studied the nature of argumentation and persuasion
[Aristotle, 1954]. Argumentation constitutes a major component of human intelligence. The ability to
engage in argumentation is essential for humans to understand new problems, to perform scientic rea-
soning, to express, to clarify and to defend their opinions. Modern social media discussions offer a rich
area for study. Participants in such discussions identify pros and cons of arguments presented by others
and analyze situations prior to presenting some information and prior to making some decision. Similarly,
these arguments can also help persuade others to change their stance towards something.
60
An argument is a set of premises, pieces of evidence (e.g. facts), offered in support of a claim. The
claim is a proposition, an idea which is either true or false, put forward by somebody as true. The claim
of an argument is normally called its conclusion. Argumentation may also involve chains of reasoning,
where claims are used as premises for deriving further claims. Therefore, analyzing argument structure
becomes crucial for identifying its impact on the target audience. For example, consider the sentences:
Impose a tax on fast foods.
Impose a tax on fast foods because obesity-related diseases burden our healthcare system.
Both sentences present a claim about imposing a tax on fast foods, but the second one provides the ratio-
nale for the claim. Therefore, the second argument becomes more sensible than the rst. Automatically
identifying argumentation structure, in this case recognizing that the second sentence contains another
claim which acts as a reason that justies the main claim, is required for various aspects of discourse anal-
ysis, including identifying arguments and their authors, nding which claims function as justications,
and for which others, which claims contradict others, etc.
In this chapter, I explore the problem of automatically identifying the argumentation structure of the
contributionbyparticipantsinonlinediscussions: howpeoplestateandconnectinanattempttopersuade
others. We use the word claim to refer to any assertion made in discussion, which the author intends
the reader to believe to be true, and that can be disputed. The Oxford Dictionary
1
denes claim as \An
assertion of the truth of something, typically one that is disputed or in doubt". I use the word claim-link
for the causal/conditional dependency between claims. As of now, I do not differentiate betweenthe types
of links from an argumentation structure.
DatadrawnfromWikipedia: AfDforum, differentfromthesocialrolescorpus, isannotatedtoidentify
claims and link between claims. Using this I analyze the argumentation structure in these discussions and
build a model of the nature of online argumentation that can be used in identifying roles that participants
1
Oxford Dictionary: https://en.oxforddictionaries.com/
61
play. I use supervised algorithms that automatically produce the argumentation structure. To identify
the argumentative structure, the whole problem is divided into 3 sub-parts:
Claim Detection: Detect sentences that express claim
Claim Delimitation: Identify specic claims within the sentence
Claim-Link Detection: Discover links between claims if any
In the rest of this chapter, I rst describe the related work. Next the corpus and annotation scheme for
theexperimentispresented. Thechapterthendescribesthemodelsandimplementationmethodsfollowed
by experiments and results. In the end, I conclude with some discussion of difficult examples and error
analysis.
5.2 Related Work
While there has been a lot of work in the eld of Rhetorics [Perelman, 1970]; [Ross Winterowd, 1975];
[Toulmin, 2003]; [Toulmin et al., 1979], computational discourse analysis, and specically the structure
of argumentation, has just recently attracted many research communities. [Marcu, 1997] produced the
rst discourse structure parser, using Rhetorical Structure Theory relations [Mann and Thompson, 1988].
Other RSTworkanalyzed discourse structure, but little of it performed automated analysis or arguments.
Work on other discourse relation sets (e.g., the Penn Discourse Treebank [Miltsakaki et al., 2004]) also
focus on the kinds of links rather than on the automated discovery of the argument structure itself.
[Spertus, 1997] describes a method to identify hostile messages automatically. A lot of computational
work focuses on the automated classication of sentiment in text; see for example [Kim and Hovy, 2004];
[Popescu and Etzioni, 2005]. Thisisnothoweverourprincipalfocus; weconsiderreasonedargumentation,
not pure like/dislike judgments. [Kwon et al., 2007] identify and classify subjective claims in order to
identifyconclusivesentencesshowingauthor'sattitudetothemaintopic. [Rosenthal and McKeown, 2012]
detect opinionated claims in online discussions in which an author expresses a belief. They investigate the
62
impact of features such as sentiment and committed belief on their system. [Biran and Rambow, 2011a]
identify justications in written text by classifying text as argumentative. They hypothesize that a text
segment is argumentative if it contains at least one discourse relation which aims to increase the reader's
willingness to accept a proposition. [Palau and Moens, 2009] present their research on argumentation
mining. They formulate the problem of identifying argumentation structure as identifying the two basic
components of the arguments i.e. a premise and a conclusion. They use machine learning approach to
rst detect an argument segment in the text and then they classify the argument segment into one of the
two basic components of the argument.
5.3 Corpus and Annotation
For this experiment, the corpus consists of 70 discussions from Wikipedia: Articles for Deletion (AfD)
discussion forum; different from the discussions used for social roles analysis. The dataset consists of 2568
sentences each ranging from 3 to 132 characters.
5.3.1 Claims
Dening what assertions can be considered to be claims is often very difficult. For example, consider the
following:
Non-notable academic as far as I can tell
The author of the article should have listed them
These are straightforward cases; the former sentence is a claim because it directly and openly states the
author'sbelief.,andthelattersentenceisnotaclaimssinceitdescribesapersonalsuggestionbytheauthor.
Thus sentence types can be an indicator for whether sentence is a claim; for example, questions cannot
contain claims. In contrast, the sentence \The article's creator complained loudly" is much harder to
classify: it is not clear whether the sentence is a claim (about which reasonable disagreement is possible)
63
or not a claim (which cannot reasonably be disputed), or both. In this study, I do not differentiate
between facts and claims. The main purpose of this study is to create an argumentation structure for
the contribution made by participants in online discussions. Therefore, it is more important to identify
whether claims are backed by other claims/facts rather than distinguishing between claims and facts.
Hence, in this study, I treat facts as claims having a dened truth value. Table 5.1 gives examples of
sentences with and without claims.
Sentences with claims Sentences without claims
He is a major gure. Can you explain to me what postmodernist
means?
Not notable enough for an article. I abstain until further evidence crops up.
The search result list Jo7hs2 provided doesn't es-
tablish this.
I can't believe we are having this discussion.
Table 5.1: Examples of sentences with and without claims.
The coding manual for annotating claims was created by the same annotators that annotated the
social roles corpora. The annotators were initially told that a claim is any assertion by a user that can be
disputed. The annotators were also asked to annotate facts as claims. Rather than annotating whether
a sentence is a claim or not, the annotators were asked to identify specic claims within the sentences,
using XML tags. This enables treatment of the many sentences containing more than one claim, and
helps in identifying specic claims automatically as well. The rst annotation round consisted of 10 AfD
discussions (313 sentences). After the rst round, the annotators discussed results, and the annotators
created the coding manual. Using this coding manual, the annotators then annotated another 10 AfD
discussions (422 sentences).
64
Kappa ()
Annotator1 - Annotator2 0.72
Annotator1 - Annotator2 0.68
Annotator1 - Annotator2 0.67
Table 5.2: Cohen's Kappa score for claim annotation.
To calculate agreement scores, two annotators are said to agree when a part of a sentence is marked
as a claim and the starting word of the claim by annotator1 is within 2 words of the starting word of
the claim by annotator2, with the same criteria holding for the end word of the claim. The agreement
scores using Cohen's Kappa coefficient () are presented in Table 5.2 As the agreement scores for these
annotations were satisfactory, this coding manual was accepted as the nal one. Using this nal coding
manual,oneoftheannotatorsannotated70moreAfDdiscussions, whichwereusedforalltheexperiments
described in the chapter.
5.3.2 Claim-Links
Once claims are identied, discovering the links between them is not so difficult. One can more easily
identifycausal/conditionalrelationshipsbetweenclaims. Forexample, thesentence \It seems that most of
the books that mention this award or site are self-published authors, so there goes that claim to notability"
consistsof twoclaims, oneither side of the comma. It is evidentthat the latter isa conclusion drawnfrom
the former, where the link between the claims is the word \so". The claim links always connect exactly
one claim to exactly on other. Table 5.3 gives examples of some claims linked with each other.
65
Claims linked with each other Link
Plenty of references, but not for the thesis of the article. but
Notability appears unprovable, so fails WP:Notability so
If he is a major gure, you should have no trouble nding sources that satisfy WP:N. If
Table 5.3: Examples of claims linked with each other.
The coding manual for annotating links between claims was created by two annotators. The anno-
tators were asked to annotate for links between claims that may establish relations such as conclusion,
justication, contradiction, etc. The annotators were also asked to annotate the direction of the relation
between (i.e., whether claim1 is dependent on claim2 or vice versa). Initially, the annotators were given
5 AfD discussions (201 sentences). The Cohen's Kappa agreement score for these annotations was 0.84.
Since this value is acceptable, the annotators created the nal coding manual based on these annotations,
and one of the annotators annotated the same 70 AfD discussions mentioned above for links between the
claims.
In the corpus, 72.7% of the sentences contained at least 1 claim. The average number of claims in
these sentences was 1.36. Out of all the annotated claims, 34.1% of the claims were linked to other claims.
76.1% of the claim-links had both claims in the same sentence and 18.3% of the claim-links had claims in
adjacent sentences. The remainder had longer-distance links.
5.4 Model and Implementation Methods
I use a supervised machine learning approach for detecting sentences that express claims, identifying
claims in the sentences, and establishing links between claims. I use the Stanford Parser
2
to preprocess
the discussions, add part-of-speech (POS) tags to the sentences, and extract dependency triples for the
sentences.
2
The Stanford Parser: A statistical parser: http://nlp.stanford.edu/software/lex-parser.shtml
66
5.4.1 Claim Detection
Eachsentenceisclassiedaseitherhavingornothavingaclaimusingseverallexicalfeatures. Thefeatures
include word n-grams(1-3), POS tag n-grams(1-3), and dependency triples [Marneffe et al., 2006]. The
dependencyparseforagivensentenceisasetoftripleswhichiscomposedofagrammaticalrelationandthe
two words from the sentence for which that relation holds (rel
i
,w
j
,w
k
). The classier also uses generalized
back-offfeaturesforn-gramsanddependencytriplesasproposedby[Joshi and Penstein-Ros e, 2009]. The
idea behind these features is to create more generalizable features by backing off to the POS tag of either
word in the relation. The two types of composite back-off features are:
h-bo: Features of the form rel
i
POS
j
w
k
m-bo: Features of the form rel
i
w
j
POS
k
Similarly back-off features for lexical bigrams and trigrams are used. Here all possible combinations of
words in the n-gram are replaced by their respective POS tags. This creates features of the form w
i
POS
j
and POS
i
w
j
for each bigram and similarly for each trigram. The motivation behind these features is
the diversity of the topics that prevails in the discussions, which causes data sparsity with specic word
combinations, which occur very infrequently.
5.4.2 Claim Delimitation
Claim delimitation is useful since a sentence may contain multiple claims. I use a CRF tagger to identify
claims within the sentences. The annotated sentences are pre-processed to add B C, I C, and O C tags
to each word, where B C indicates a word starting a claim, I C indicates a word inside a claim and O C
indicates a word outside any claim. The following features are used to tag each word w
i
automatically:
word unigrams: All the words w
j
(j=i-2:i+2)
word bigrams: Combinations of the form w
j-1
|w
j
(j=i-1:i+2)
word trigrams: Combinations of the form w
j-2
|w
j-1
|w
j
(j=i:i+2)
67
POSunigrams, bigramsandtrigramscombinationssimilartowordunigrams, bigrams, andtrigrams
Questionmark(?): Questionsrarelycontainanyclaims,thereforabinaryfeaturesindicatingwhether
the sentence the word is in ends with a question mark symbol is used.
5.4.3 Claim-Link Detection
For claim-link detection, claim pairs are formed and determined whether they are linked. For each claim
c
i
in sentence s
k
, pairs (c
i
, c
j
) are created for all claims c
j
in sentences s
k
, s
k+1
, and s
k+2
. Separate entry
for the reversed pair (c
j
, c
i
) is not created. Each pair (c
i
, c
j
) is classied as either having no link between
them, c
i
is dependent on c
j
or c
j
is dependent on c
i
. For each claim pair (c
i
, c
j
) the following features are
used:
word and POS n-grams(1-3) for words in c
i
word and POS n-grams(1-3) for words in c
j
word and POS unigrams for at most 5 words preceding and succeeding c
i
word and POS unigrams for at most 5 words preceding and succeeding c
j
# of similar words in c
i
and c
j
Claim distance: distance between c
i
and c
j
in terms of how many claims are between them
Sentence distance: distance between c
i
and c
j
in terms of how many sentences they are apart from
each other
A word w can have 4 features for itself based on where it appears (and similarly for POS unigrams). All
the features are represented in standard bag-of-words type binary representation.
68
5.5 Experiments and Results
Data mining software Weka
3
is used for the Claim Detection and Claim-Link Detection classication
experiments. In that, I use Support Vector Machine (SVM)
4
with radial basis function and J48 Decision
Tree
5
classiers. CRFsuite
6
for the Claim Detection and Claim Identication experiments. For each
experiment, I train the model on 80% of the data and test it on the remaining 20% of the data.
5.5.1 Claim Detection
IuseSVMwithradialbasisfunctionandJ48DecisionTreeclassiersaswellasaCRFtaggerfordetecting
sentences expressing claims. I compute the accuracy of the classiers using F1-score
7
. I report the F1-
score from Weka as results for SVM and J48 Decision Tree. To compute F1-score for CRF, a sentence is
said to express a claim if the CRF tagger tags at least one word of the sentence as B C, otherwise I mark
that sentence as without claim. I compare the results with the following baselines:
Model F1-Score
Majority 0.72
Question (?) 0.76
SVM 0.83
J48 0.82
CRF 0.85
Table 5.4: Claim Detection results.
Majority: Each sentence is classied with the class having majority.
3
Weka 3: Data Mining Software in Java: http://www.cs.waikato.ac.nz/ml/weka/
4
Support Vector Machine: http://en.wikipedia.org/wiki/Support_vector_machine
5
J48 Decision Tree: http://en.wikipedia.org/wiki/C4.5_algorithm
6
CRFsuite: A fast implementation of Conditional Random Fields: http://www.chokkan.org/software/crfsuite/
7
F1-score: http://en.wikipedia.org/wiki/F1_score
69
Question (?): Each sentence having `?' symbol is classied as a sentence without claim, otherwise
it is marked as a sentence with claim
Table 5.4 presents the best results for each experiment on test data.
5.5.2 Claim Delimitation
I use a CRF tagger to identify claims in a sentence. I also try to combine Claim Detection and Claim
Identication to see whether it gets better result. To accomplish this, I use the result of SVM Claim
Detection. For all the sentences marked as without claim by SVM, all the words in those sentences are
assigned tag O C. Now to tag the sentences marked as with claim, I use a CRF tagger trained only on
sentences with claim in the training data. I report F1-score for B C, I C and O C tags as the result of
the experiment. I compare the results with the following baseline:
Question (?): All the words in sentences having `?' symbol are tagged as O C. For all the other
sentences, the rst alphanumeric word is marked as B C tag, all the words followed by that until the
last non-symbolic words are marked as I C tag, and the rest of the words are marked as O C tag.
I report the best result for each experiment on test data in Table 5.5
F1-Score
Model B C I C O C
Question (?) 0.60 0.82 0.49
CRF 0.79 0.76 0.77
SVM-CRF 0.80 0.74 0.72
Table 5.5: Claim Delimitation results.
70
5.5.3 Claim-Link Detection
I use SVM with radial basis function and J48 Tree classiers for detecting links between the claims.
F1-score from Weka is reported as results for SVM and J48. I compare our results with the following
baselines:
Adjacent: For each adjacent claim pair (c
i
, c
j
), if they both are in the same sentence, they are
marked as linked.
Same sentence: For each claim pair (c
i
, c
j
), if they both are in the same sentence, they are marked
as linked.
Table 5.6 reports the best results for each experiment on test data.
Model F1-Score
Adjacent 0.70
Same Sentence 0.77
SVM 0.86
J48 0.89
Table 5.6: Claim-Link Detection results.
5.6 Discussion
Notallsentencescanclearlybeclaims. Itisoftendifficulttodistinguishwhenaclaimisexpressedagainst
when assertion is made. Table 5.7 gives some examples of mis-classied claims by the model described in
this chapter.
71
Actual Claims Detected Claims
Unlesscleanedupandimproved,Iwouldn'tmind
its deletion.
Unlesscleanedupandimproved, Iwouldn'tmind
its deletion.
Find references, and I'll reconsider, until then,
this violates WP:V.
Find references, and I'll reconsider, until then,
this violates WP:V.
I have removed references to the Barons
Yarbourgh, because the surname is different.
I have removed references to the Barons
Yarbourgh, because the surname is different.
And it is to the domesday book that I must refer
in the Yarbourgh case.
And it is to the domesday book that I must refer
in the Yarbourgh case.
Table 5.7: Mis-classied claims.
The actual claims inside the sentences are marked in blue and the predicted claims for the same
sentences are marked in red. We can see from sentence 1 & 3 that the system is confused in terms of
distinguishing between claims and personal feelings/actions. This may occur because personal actions are
marked as claims where as personal feelings are not and there is not much difference between them in
terms of lexical analysis of the text in those cases. In sentence 2, the author makes a promise to reader
which is not a claim. And in sentence 4, the author is asserting an action for himself rather than an
action that he may have performed. For the Claim-Link identication model, the most common error
occurs when there is a hidden link between two claims. For example, the sentence \The references are not
enough, it violates WP:N." contains two separate claims \The references are not enough" and \it violates
WP:N". These claims are linked to one another by a causal relationship but there the link-word is hidden.
5.7 Conclusion
In this chapter, I present methods to identify claims and links between claims using basic lexical features.
The resulting system performs quite better for identifying the underlying argumentation structure in the
72
contribution of participants. Earlier work in automated discourse structure analysis suggested that it is a
very difficult problem. But breaking down the procedure into a series of smaller steps, rst focusing just
on identifying claims, makes the task more intuitive.
73
Chapter 6
Social Roles Model
The sensibleness model presented in this chapter has been published at SocialNLP 2016: [Jain, 2016].
The original results presented in the paper have been updated to incorporate the additional Wikipedia
annotation.
6.1 Introduction
This chapter presents the Social Roles model to automatically identify participant roles described in
Chapter 3. I develop methods to identify values for each behavioral characteristic for a participant and
then identify the social role using the rules dened by the annotators. The Social Roles model is trained
on Wikipedia discussions and tested on both Wikipedia and 4forums.com discussions.
6.2 Methods
I use machine learning methods for identifying the values of all the characteristics except for ignored-ness.
After experimenting with several classiers, SVM classier with radial basis function is chosen for all
classication tasks using the features described later in this section. Parameter estimation for all SVM
classiers is done by performing grid-search, which is simply an exhaustive searching through a manually
specied subset of the parameter space.
74
The results for all characteristics and social roles model are compared with two baselines: Random
and Bag-of-Words. For Random models, the value for any characteristic is chosen randomly from the
probability distribution of the values for the corresponding characteristic. The Bag-of-Words model for
each characteristic is created using an SVM classier with radial basis function. The weight for each
word is calculated using tf-idf measure. To identify the social role using the baseline models, rule based
approach is used by identifying the characteristic values using their respective characteristic model.
To determine whether the rule based approach is useful, another model is created for identifying social
roles using all the features for all the behavioral characteristics. As described previously in Chapter 3, it
is possible for participants to have multiple roles. Therefore two sets of mutually exclusive social roles are
createdandthenanSVMclassierwithradialbasisfunctioniscreatedforeachusingallthefeatures. The
rst set contains roles Leader, Follower, and None (not a Leader or a Follower). The second set contains
roles Rebel, Voice in Wilderness, Idiot, Nothing, Nothing (Sensible), and Other. Participants are assigned
a role for each set using the respective classier and the nal role is decided based on them.
The remaining of this section presents the features used to create models for each characteristics.
6.2.1 Stubbornness
Stubbornness value for each participant is classied by analyzing the amount of participation and inter-
action pattern of the participants using the features listed below.
# of comments: This represents the total contribution by the participant in the discussion.
# of comments/Avg number of comments: This represents the amount of contribution by the par-
ticipant compared to average participation in the discussion.
Avg. length of comments: This feature gives an indication of how verbose the participant is.
% of comments as reply to others: This feature indicates the commitment of the participant to
defend his/her stance or counter the opposite stance.
75
Change in stance: A binary feature for explicit change in stance by the participant.
% of comments with no claims: Amount of contribution that doesn't argue in favor of the partic-
ipant's stance. Claim detection model described in Chapter 5 is used to detect comments with no
claims.
6.2.2 Sensibleness
The sensibleness model leverages features obtained through argumentation mining and analyzing the
content provided by the participant throughout the discussion. Sensibleness value for participants is
classied using the following features.
% of sentences containing claims: This feature indicates the amount of contribution by the partici-
pant supporting his/her stance or arguing against that of others. Claim detection model described
in Chapter 5 is used to detect sentences containing claims.
% of claims linked to other claims: The amount of claims linked to other claims signies that the
participant is backing his claims with other claims or facts, which indicates good argumentation on
from the participant. Claim-Link detection model described in Chapter 5 is used to detect claims
linked together.
% of comments as tangential comments: Participants who tend to de
ect from the main subject of
the discussion are considered to be non-sensible. For each participant, each of his/her comments
is categorized as tangential to the discussion or not. To quantify this, itf-ipf, a slightly modied
version of tf-idf, is used to approximate tangentiality of any comment. For any tangential comment,
the words used in the comment would be used relatively less than other words overall and would
be used by relatively fewer participants. tf (term frequency) and pf (participant frequency: total
76
number of participants who used the word in the discussion) are calculated and the itf-ipf value for
each word w in a comment is computed as:
w
itfipf
=
1
w
tf
log
N
w
pf
(6.1)
N = total number of participants in discussion.
The total itf-ipf value is divided by the total number of words to nullify the effect of the length
of the comment. For Wikipedia discussions, if the value of TQ
C
for a comment is more than 1.3
standarddeviationsfromtheaveragetangentialquotientofthediscussion(μ+1.3s), thecommentis
considered tangential. Similarly, for 4forums.com discussions, if the value of TQ
C
for a comment is
more than 1.5 standard deviations from the average tangential quotient of the discussion (μ+1.5s),
the comment is considered tangential.
# of positive and # of negative reviews for participant: Peer reviews provide an external opinion on
the sensibleness of a participant. They therefore play a signicant part in determining sensibleness
of a participant, as a system with no domain knowledge of the discussion topic cannot verify the
validity of their claims. For this analysis, all sentences that contain references to other participants
are identied using NLTK
1
toolkit's NER (Named Entity Recognition) module. Second person
pronouns in replies to other participants as reference are also identied. Next, the sentences that
contain the reference are analyzed using NLTK's sentiment analysis module. If the sentence has
non-neutral sentiment, then the polarity of the sentence is checked. If the polarity of the sentence
is positive, then it is considered a positive review towards the participant who is referenced in the
sentence. Similarly, if the polarity is negative, then it is considered a negative peer review.
% of sentences as questions: It can be a good strategy to ask questions related to the discussion,
but asking too many questions can be considered as non-sensible.
1
http://www.nltk.org/
77
% of comments as personal attacks: Thisfeatureisusefulforidentifyingparticipantswhoconstantly
attackothersratherthanpresentingtheirownarguments. Asimilarmethodtothatforpeerreviews
isusedtoidentifycommentsthataretargetedtowardsotherparticipantsandhavenegativepolarity.
% of comments mentioning Wikipedia policy (for Wikipedia discussions only): For Wikipedia dis-
cussions, a domain specic feature of \Policy" is also incorporated based on the intuition that
participants who mention Wikipedia policies in their comments are considered following the norms
of the Wikipedia and hence can be considered as sensible. A small vocabulary is used to detect
policy mentions in any comment.
6.2.3 Ignored-ness
A rule based method is used to identify the ignored-ness value for a participant in the discussion. The
following binary features are used to determine whether the participant is ignored by others. If any of the
featuresbelowistrueforaparticipantthentheparticipantisconsiderednotignoredbyotherparticipants.
Participant referred/mentioned by others: A binary feature to indicate whether other participants
refer/mention the participantin their comments. NER and second person nouns are used to identify
mentions in comments.
Participant replied to by others: A binary feature to indicate whether the participant's comments
get direct reply by other participants.
6.2.4 In
uence
In
uence value for participants is determined by identifying endorsements in the discussions. I also use
leadershipmodelspresentedinChapter4tocalculateleadershippointsofparticipantsasdescribedbelow.
The in
uence value for participants is classied using the following features.
# of comments endorsing the participant: Comments referenced towards the participant are identi-
ed and are determined to be an endorsement or not using a small dictionary.
78
# of comments by the participant endorsing other participants: Comments referenced to other par-
ticipants are identied and are determined to be an endorsement or not using a small dictionary.
Content leadership points: Quanties the language use of a participant to assign leadership points.
Participantsgainpointsforfollowersusingsimilarlanguageastheparticipantandalsoforcountering
participants having different stance.
SilentOut leadership points: Quanties the ability of a leader to silence opposing users out with
his/her arguments. Participants gain points for having comments that is not countered by others
and also for having the last say in a argument section in the discussion.
% of words already used by other participants: Followers are more likely to use words already used
by others in the discussion.
6.3 Experiments and Results
Stanford Parser is used for preprocessing the discussions. Weka is used for all classication tasks. NLTK
toolkit's NER (Named Entity Recognition) module is used for identifying sentences referenced toward
other participants. The remaining of this section rst presents the results for identifying characteristics
and social roles of participants followed by an analysis of the importance of individual features for the
social roles model.
6.3.1 Characteristics and Social Roles
The social roles of participants are identied based on the value of their characteristics as classied
using their corresponding models. All classiers are trained on Wikipedia discussions and tested on both
Wikipedia and 4forums.com discussions. 10-fold cross validation is used for testing Wikipedia models.
The weighted precision, recall, and F1-score for the best performing classier after parameter estimation
is reported in Table-6.1.
79
Wikipedia 4forums
Model Precision Recall F1-score Precision Recall F1-score
Random 0.53 0.53 0.54 0.50 0.49 0.50
Stubbornness Bag-of-Words 0.67 0.68 0.67 0.65 0.63 0.64
Stubbornness Model 0.89 0.88 0.88 0.82 0.80 0.81
Random 0.72 0.70 0.71 0.69 0.68 0.69
Sensibleness Bag-of-Words 0.73 0.80 0.76 0.65 0.73 0.68
Sensibleness Model 0.88 0.90 0.89 0.78 0.79 0.78
Random 0.48 0.45 0.46 0.34 0.35 0.35
Ignored-ness Bag-of-Words 0.61 0.64 0.62 0.60 0.58 0.59
Ignored-ness Model 0.94 0.91 0.92 0.91 0.90 0.90
Random 0.57 0.56 0.56 0.63 0.66 0.64
In
uence Bag-of-Words 0.70 0.63 0.65 0.68 0.61 0.63
In
uence Model 0.88 0.86 0.87 0.79 0.78 0.78
Random 0.46 0.41 0.42 0.44 0.37 0.39
Social Bag-of-Words 0.64 0.63 0.63 0.58 0.61 0.60
Roles All features 0.78 0.75 0.77 0.71 0.65 0.69
Rule based 0.84 0.83 0.83 0.76 0.72 0.75
Table 6.1: Results for characteristics and social roles models.
TheresultspresentedinTable-6.1showthattheindividualcharacteristicmodelsandsocialrolesmodel
perform better than the baseline models. The rule-based model for social roles also performs better than
the classication model using all features directly. This signies that intuitive approach of breaking down
the problem of identifying social roles to identifying individual characteristics is better than identifying
the social roles directly.
80
6.3.2 Feature Ablation
Feature ablation experiments are performed to analyze signicance of individual features, used for identi-
fying the value of characteristics, for the overall social roles model. In this experiment, one of the features
for one of the characteristics is removed from the full feature set at a time. The result for this modied
rule based model is then compared with the rule based model with all the features to calculate the sta-
tistical signicance of the feature removed from the set. McNemar's test
2
is used to measure statistical
signicance. A signicance difference of of loss in performance for p < 0.01 is depicted with
▼
and for p
< 0.05 is depicted with
▽
(loss).
6.3.2.1 Stubbornness
Table 6.2 shows the feature ablation results for stubbornness characteristic. `# of comments' and `Avg.
length of comments' are the most signicant features for identifying stubbornness for both Wikipedia and
4forums discussions. `# of comments/Avg # of comments' feature is important to distinguish stubborn-
ness of participants with similar amount of contribution in discussions having different amount of total
contribution. `% of comments as reply' is more signicant for Wikipedia than 4forums because of the
inherent difference between the two discussion domains. On Wikipedia discussions participants have more
incentive to counter participants having opposing stance by directly replying to them so that the nal
outcome of the discussion is declared in their favor. Whereas, there is no nal outcome of the discus-
sion on 4forums and therefore participant are more focused on presenting their own opinion rather than
countering others'. `Change in stance' doesn't contribute signicantly in identifying stubbornness. `%
of comments with no claims' is signicant in identifying non-stubborn contribution by participant that
doesn't contribute towards his/her stance.
2
McNemar's test: https://en.wikipedia.org/wiki/McNemar's_test
81
F1-score
Features Wikipedia 4forums
All 0.83 0.75
# of comments 0.77
▼
0.70
▼
# of comments/Avg # of comments 0.81
▽
0.72
▽
Avg. length of comments 0.80
▼
0.73
▽
% of comments as reply 0.81
▽
0.74
Change in stance 0.82 0.75
% of comments with no claims 0.81
▽
0.73
▽
Table 6.2: Feature ablation results for stubbornness.
6.3.2.2 Sensibleness
Table 6.3 shows the feature ablation results for sensibleness characteristic. Argumentation structure
features `Claims' and `Claim-Links' combined are most signicant for the sensibleness model. `Tangential'
is more signicant for Wikipedia for identifying non-sensible participants because Wikipedia discussions
mostly revolve around Wikipedia policies and therefore any participant swaying away from the same are
identiedmorereliably. `Policy'featureissignicantforWikipediaforthesamereason. `Peerreviews'are
important for both Wikipedia and 4forums whereas `Questions' and `Personal attacks' don't contribute
signicantly towards identifying sensibleness.
F1-score
Features Wikipedia 4forums
All 0.83 0.75
Claims 0.80
▼
0.71
▼
Claim-Links 0.82 0.74
Claims+Links 0.78
▼
0.70
▼
82
Tangential 0.80
▼
0.73
▽
Peer reviews 0.81
▽
0.73
▽
Questions 0.83 0.75
Personal attacks 0.82 0.74
Policy 0.81
▽
NA
Table 6.3: Feature ablation results for sensibleness.
6.3.2.3 Ignored-ness
Table 6.4 shows the feature ablation results for ignored-ness characteristic. Both features `referred by
others' and `replied to by others' are very signicant for determining whether the participant is ignored
by other participants.
F1-score
Features Wikipedia 4forums
All 0.83 0.75
referred by others 0.80
▼
0.71
▼
replied to by others 0.76
▼
0.70
▼
Table 6.4: Feature ablation results for ignored-ness.
6.3.2.4 In
uence
Table6.5showsthefeatureablationresultsforin
uencecharacteristic. Endorsementsareaprimarywayof
expressingleadershipqualitiesofothers. Thereforeboth`#ofcommentsendorsingtheparticipant'and`#
of comments by participant endorsing others' are signicant in identifying in
uence value for participants.
`Content leadership points' is signicant for identifying leaders by quantifying implicit endorsements in
terms of followers using similar arguments as leaders. Similarly, `% of words already used by others'
83
is signicant for Wikipedia for identifying followers by quantifying implicit endorsements. `SilentOut
leadership points' doesn't contribute signicantly in identifying in
uence.
F1-score
Features Wikipedia 4forums
All 0.83 0.75
# of comments endorsing the participant 0.80
▼
0.71
▼
# of comments by participant endorsing others 0.80
▼
0.73
▽
Content leadership points 0.81
▽
0.72
▽
SilentOut leadership points 0.82 0.75
% of words already used by others 0.81
▽
0.74
Table 6.5: Feature ablation results for in
uence.
6.4 Conclusion
This chapter tests a model for identifying social roles in online discussions by analyzing the contribution
made by participants in the discussion to quantify behavioral characteristics that dene these roles. Both
rule based and machine learning methods are used for creating models used to determine values for the
characteristics and in turn the social roles. The models trained on Wikipedia discussions outperform the
baseline models for both Wikipedia and 4forum.com discussions. The analysis presented in the chapter
shows that breaking down the problem of identifying social roles to determining values of specic charac-
teristics make it more intuitive in terms of what we expect from participants who assume these roles. The
rule based method to identify social roles performs better than the model created using all the features.
This strengthens the intuition for using the bottom up approach of identifying characteristics and then
the social roles.
84
Chapter 7
Social Roles Models (Neural Networks)
7.1 Introduction
There is a recent surge of interest in neural networks, which are based on continuous-space representation
of the input and non-linear functions. Hence, neural networks are capable of modeling complex patterns
in data. Moreover, since these methods do not depend on manual engineering of features, they can be
applied to solve problems in an end-to-end fashion.
In this chapter I present an approach on how deep learning methods can be used to train neural
networks which can encapsulate information useful for identifying participants' behavioral characteristics
and social roles. One of the advantages of using neural networks for NLP tasks is that it doesn't require
feature engineering as the network tries to learn the features on its own using the raw text. Hence, one
of the goals for this experiment is to assess whether the features presented in the previous chapters to
train the SVM network are required for identication of the behavioral characteristics and social roles.
Therefore, all the neural networks presented in this chapter are provided with the raw text from the par-
ticipantsastheirinput, andareexpectedtolearntheunderlyingstructuresrequiredtoidentifybehavioral
characteristics and social roles automatically. I develop 3 different neural network structures to identify
participants' social roles and behavioral characteristics. Using behavioral characteristics, the social role
of the participant is then determined using the rules dened by the social roles framework. I also create
85
custom word embeddings trained on Wikipedia discussions in order to capture domain specic dependen-
cies and compare the performance of these word vectors against state-of-the-art word embedding tools
using extrinsic evaluation. The results show that using neural networks to identify participants' behav-
ioral characteristics and then the social roles performs signicantly better than identifying the social roles
directly. Using extrinsic evaluations I show that the custom word embeddings perform at least as well as
the state-of-the-art word embeddings.
The results show that the SVM models presented in the previous chapter performs better than the
neural networks for identifying participants' behavioral characteristics and social roles. Traditionally,
neural networks require signicant amount of data to learn meaningful representation of the underlying
structure of the data in order to perform the desired task. In order to compare how additional data affects
the performance of both SVM and neural networks, I annotate more data for participants' behavioral
characteristics and social roles. The results on the bigger dataset show that neural networks improve at
a much higher rate compared to SVM network.
7.2 Word Embeddings
Although word embeddings have been studied extensively in recent work (e.g., [Lapesa and Evert, 2014]),
most such studies only involve general domain texts and evaluation datasets. State-of-the-art word em-
bedding tools like word2vec
1
and Global Vectors (GloVe)
2
have been shown to achieve state-of-the-art
performance for various NLP tasks. In order to capture domain specic dependencies, I train the custom
wordembeddingsusingmodelsimplementedin word2vecwithdifferenthyperparametersusingWikipedia
discussions. I then use these custom word embeddings for behavioral characteristic identication and
compare its performance with word2vec embeddings. The results show that the custom word embeddings
perform at least as well as word2vec embeddings.
1
word2vec: https://code.google.com/archive/p/word2vec/
2
GloVe: https://nlp.stanford.edu/projects/glove/
86
7.2.1 Word Vectors
Factors that affect the performance of word representations include the training corpora, the model ar-
chitectures, and the hyperparameters. For training corpora, I use the Wikipedia: AfD discussions from
January 1, 2009 to June 30, 2012 consisting of 92364 discussions with 811047 comments. The discussions
are preprocessed to preserve the comments made by the participants in lower-cased text and remove any
additionalinformationfromthetextsuchasname of theparticipantwhopostedthecomment, timestamp
of the comment, etc.
Figure 7.1: CBOW and Skip-gram models, taken from [Mikolov et al., 2013]
I generate two sets of word vectors by applying the CBOW (Continuous Bag-of-Words) and the skip-
gram model. CBOW model learns to predict the word given its context, whereas skip-gram model learns
to predict the context given the word. I then build additional word vectors for these model architectures
by varying values of one of the hyperparameters shown in Table 7.1 and keeping others as default.
87
7.2.2 Hyperparameters
I experiment with the following hyperparameters:
Minimum-count (min): The minimum count denes the minimum number of occurrences required for
a word to be included in the word vectors.
Vector dimension (dim): The vector dimension is the size of the learned word vector.
Context window size (win): The size of the context window denes the range of words to be included
as the context of a target word.
Negative sample size (neg): The representation of a word is learned by maximizing its predicted
probability to co-occur with its context words, while minimizing the probability for others, which is
computedusingco-occurrencesbetweenwordsandalltheircontextsinthecorpus,whichistime-consuming
to compute. To address this issue, negative sampling only calculates the probability with reference to a
set number of other randomly chosen negative words.
Learning rate (alpha): The learning rate controls the frequency of updating weight vectors for the
neural network for minimizing the objective function.
Parameter Values
min 1 / 2 / 5 / 10 / 20
dim 50 / 100 / 200 / 400
win 1 / 2 / 5 / 10
neg 1 / 2 / 5 / 10
alpha 0.01 / 0.025 / 0.05 / 0.1
Table 7.1: Hyperparameters for word embeddings and tested values. Default values shown in bold.
I compare the vectors created using different combinations of the hyperparameters against the state-
of-the-art word2vec word embeddings using extrinsic evaluation, i.e., analyzing their performance on the
nal task of identifying the behavioral characteristics of the participants.
88
7.3 Models
I develop and compare 3 neural network structures for identifying participants' social roles and their
behavioral characteristics.
7.3.1 CNN with user utterances (CNN-user)
The rst model I create is a Convolution Neural Network (CNN) that uses the utterances made by a
participant in the discussion to identify the social role/behavioral characteristic of the participant. The
model architecture is a slight variantof the CNN architecture of of [Collobert et al., 2011]. The CNN-user
architecture has the following layers:
Figure 7.2: Convolution Neural Network (CNN), taken from [Kim, 2014a]
Input layer: The input to the network is all the utterances of a participant concatenated as one
utterance. Each word in each utterance is represented as a d-dimensional word vector corresponding to
thewordinthewordembeddings. Hence,theconcatenatedutteranceoflengthn(paddedwherenecessary)
is represented as a concatenation of the word vectors for the words present in the utterance.
Convolution layer: In this layer, a set of m lters is applied to a sliding window of length h over
each concatenated utterance to produce a feature map. The lters are learned during the training phase
of the neural network.
89
Max pooling layer: The output of the convolution layer is passed through a non-linear activation
function, before entering a pooling layer. The latter aggregates feature map elements by taking the
maximum over a xed set of non-overlapping intervals.
Hidden layer: A fully connected hidden layer computes the transformation (W x+b), where W
is the weight matrix, b the bias, and the activation function.
Softmax layer: Finally, the output of the nal pooling/hidden layer are fully connected to a softmax
regression layer, which returns the class with the largest probability. For regularization I employ dropout
on the penultimate layer. Dropout prevents co-adaptation of hidden units by randomly dropping out, i.e.
setting to zero, a proportion of the hidden units during forward propagation.
7.3.2 Bidiractional LSTM with user utterances (BiLSTM-user)
The second model I create is a Bidirectional Long Short Term Memory (BiLSTM) network that uses the
utterances made by a participant in the discussion to identify the social role/behavioral characteristic of
the participant. Recurrent Neural Network (RNN) is an extension of conventional feed-forward neural
network. However, standard RNN has the gradient vanishing or exploding problems. The Long Short
Term Memory (LSTM) addresses the problem by re-parameterizing the RNN model. The core idea of
LSTMisintroducingthe\gates"tocontrolthedata
owintherecurrentneuralunit. TheLSTMstructure
ensures that the gradient of the long-term dependencies cannot vanish. Brie
y speaking, in LSTM, the
hidden states h
t
and memory cell c
t
is a function of their previous c
t-1
and h
t-1
and input vector x
t
. The
hidden state h
t
denotes the representation of position t while also encoding the preceding contexts of the
position. The BiLSTM-user architecture has the following layers:
Input layer: The input to the network is a sequence of utterances made by a participant in the
discussion in the order as they appear in the discussion. Each word in each utterance is represented as a
d-dimensional word vector corresponding to the word in the word embeddings.
BiLSTM layer: In LSTM, the hidden state of each position h
t
only encodes the prex context in
the forward direction while the backward context is not considered. Bidirectional LSTM exploits two
90
parallel passes forward and backward of the input and concatenates hidden states of the two LSTMs.
At each position t, the new representation of the input is the concatenation of the hidden states of the
forward LSTM and backward LSTM. In this way, the forward and backward contexts can be considered
simultaneously.
Figure 7.3: Bidirectional LSTM network (BiLSTM), taken from [Graves, 2012]
Hidden layer: A fully connected hidden layer computes the transformation (W x+b), where W
is the weight matrix, b the bias, and the activation function.
Softmaxlayer: Finally,theoutputofthenalBiLSTM/hiddenlayerarefullyconnectedtoasoftmax
regression layer, which returns the class with the largest probability. For regularization I employ dropout
on the penultimate layer.
91
The advantage of BiLSTM-user network over CNN-user network is that it preserves the number and
order of utterances by the participant in the discussion. This extra information may help the network
learn better structure from the input data to identify the behavioral characteristics of participants.
7.3.3 Bidirectional LSTM with all utterances (BiLSTM-all)
One major shortcoming of the BiLSTM-user and CNN-user networks is that they only consider the
utterances of the participant whose social role/behavioral characteristic is to be identied. It does not
incorporate the information regarding the utterances by other participants in the discussion which can be
pivotal in determining characteristics such as ignored-ness, in
uence which are dependent on how other
participants react to the participant's contribution to the discussion.
Inordertoovercomethisshortcoming,thenalmodelIcreateisatwo-layernetworkwhichincorporates
all the utterances in the discussion. The rst layer of the network creates a vector representation of
each utterance in the discussion. I then add an additional dimension to this vector to incorporate the
information about the speaker of the utterance. The additional dimension can be 1 (representing the
participant whose social role/behavioral characteristic is to be identied) or -1 (representing any other
participant). In the end, the sequence of the vector representation of the utterances in the discussions is
used to train the Bidirectional LSTM network.
7.3.3.1 Vector representation of utterances
I experiment with the following 2 techniques to create the vector representation of the utterances in the
discussion:
doc2vec: doc2vec is an extension of the word2vec word embeddings. The goal of doc2vec is to create
a numeric representation of a document, regardless of its length. doc2vec essentially uses a word2vec
model and an additional vector (Paragraph ID) to represent the numeric representation of the document.
There are two approaches within doc2vec: dbow and dmpv. dbow works in the same way as skip-gram,
except that the input is replaced by a special token representing the document. In this architecture, the
92
order of words in the document is ignored; hence the name distributed bag of words. dmpv works in a
similar way to cbow. For the input, dmpv introduces an additional document token in addition to multiple
target words. Unlike cbow, however, these vectors are not summed but concatenated. The objective is
again to predict a context word given the concatenated document and word vectors. Once the model is
trained, embeddings of new/unseen documents can be inferred from the pre-trained model efficiently.
Figure 7.4: doc2vec models, taken from [Le and Mikolov, 2014]
Itrainthedoc2vecvectorsontheWikipedia: AfDdiscussionsfromJanuary1,2009toJune30,2012. I
trainthevectorsontheutterancesofparticipantsinthediscussion,andnotthewholediscussion. Ithenuse
thetraineddoc2vecencodertocreatethevectorrepresentationoftheutterancesoftheparticipantsforthe
Bidirectional LSTM network. I experiment with the following hyperparameters for training the doc2vec
model: vectorsize(200/400),contextwindowsize(5/10/15),minimumcount(2/5/10),andnegativesample
size(3/5).
Autoencoder: An autoencoder is an unsupervised neural network which is trained to reconstruct a
giveninputfromitslatentrepresentation. Itconsistsoftwoparts,namelyencoderanddecoder. Whilethe
encoder aims to compress the original input data into a low-dimensional representation using a non-linear
activation function, the decoder tries to reconstruct the original input data based on the low-dimension
representation generated by the encoder. The training objective is to learn network parameters that
minimize the average reconstruction error over a set of input vectors. Autoencoders are a means to learn
93
representations of some input by retaining useful features in the encoding phase which help to reconstruct
the input, whilst discarding useless or noisy ones.
Figure 7.5: Autoencoder model
I train the autoencoder on the Wikipedia: AfD discussions from January 1, 2009 to June 30, 2012. I
train the autoencoder on the utterances of participants in the discussion, and not the whole discussion. I
use the custom word embeddings as look-up for words in the utterances and then use the average word
vector as representation of the utterance to train the autoencoder model. I then use the encoder part of
the learned autoencoder model to create the vector representation of the utterances of the participants
for the Bidirectional LSTM network. I experiment with the following hyperparameters for training the
autoencoder network: input layer dimension (200/400), activation function(relu/tanh), and hidden layer
dimension (50/100/200).
94
7.3.3.2 BiLSTM-all
Once I create the vector representation of the utterances of participants in the discussion, I add one
additional dimension to the vector representation to indicate the speaker of the utterance. The value 1
for the additional dimension represents the participant whose social role/behavioral characteristic is to be
determined and the value -1 represents any other participant in the discussion. The BiLSTM-all model
hasthesameBiLSTM layer, Hiddenlayer, and Softmax layerastheBiLSTM-usermodel. Theinputlayer
to the network is a sequence of the vector representation of the utterances by all the participants in the
discussion in the order as they appear in the discussion.
7.4 Experiments and Results
NLTK's word tokenizer is used for preprocessing the Wikipedia discussions. Each model created for
identifying the social role/behavioral characteristic is trained on 90% of the annotated Wikipedia data
and tested on the remaining 10% of the annotated data. For validation, I select 10% of the data from the
training data. Tensor
ow implementations of CNN, BiLSTM, and Autoencoder of the Keras
3
module are
used for training the neural networks. I initialize my own word embeddings and word2vec embeddings
for both CNN-user and BiLSTM-user models. Unknown words from the pre-compiled embeddings are
initialized randomly in the range of [-0.25, 0.25]. I also update all the embedding vectors during the
training. For doc2vec training I use the implementation of the gensim
4
module.
7.4.1 Tuning Hyperparameters
I manually explore the hyperparameter space for both CNN and BiLSTM networks by performing grid
search. The hyperparameters for each model are tuned using the validation data. The hyperparameters
and their possible values are presented in Table 7.2.
3
Keras: https://keras.io/
4
gensim doc2vec: https://radimrehurek.com/gensim/models/doc2vec.html
95
Parameter Values
batch-size 50 / 100
dropout rate 0.3 / 0.5
epochs 4 / 8
learning rate 0.0001 / 0.001
kernel size 3 / 4 / 5
number of lters 100 / 200
hidden dimensions 50 / 100
activation functions relu / tanh
Table 7.2: Hyperparameters for models and tested values.
7.4.2 Characteristics and Social Roles
Thesocialrolesofparticipantsfortherulebasedmodelareidentiedbasedonthevalueoftheirbehavioral
characteristics as identied by the corresponding models. For identifying social roles directly, two sets of
mutually exclusive social roles are created and then the neural network is created for each set. The rst
set contains roles Leader, Follower, and None (not a Leader or a Follower). The second set contains roles
Rebel, Voice in Wilderness, Idiot, Nothing, Nothing Sensible, and Other. Participants are assigned a role
for each set using the respective network output and then the nal role is decided based on them. For all
themodels,thebestperformingmodelonvalidationdataisthenusedtoidentifythesocialrole/behavioral
characteristics for participants.
TheweightedF1-score forall thebest performing neuralnetworkmodels isreportedin Table7.3 along
with the SVM model. In order to test the changes in performance of all the models with additional data,
I report the F1-score for all the models using 80 discussions and 160 discussions respectively. I keep the
test set the same while testing the models with additional data. The hyperparameter settings for all the
best performing neural networks are reported in Appendix B.
96
Model
CNN-user
w/ custom
embedding
CNN-user
w/
word2vec
BiLSTM-user
w/ custom
embedding
BiLSTM-user
w/ word2vec
BiLSTM-all
w/ doc2vec
BiLSTM-all
w/
autoencoder
SVM
Stubbornness 0.63/0.65 0.62/0.64 0.68/0.72 0.68/0.72 0.78/0.82 0.76/0.81 0.88/0.88
Sensibleness 0.75/0.77 0.75/0.76 0.78/0.82 0.77/0.81 0.80/0.83 0.79/0.82 0.89/0.89
Ignored-ness 0.54/0.55 0.54/0.55 0.60/0.63 0.60/0.63 0.76/0.79 0.77/0.79 0.92/0.92
In
uence 0.67/0.71 0.67/0.70 0.72/0.76 0.71/0.75 0.76/0.80 0.75/0.79 0.86/0.87
Social Roles
(Directly)
0.50/0.53 0.50/0.53 0.55/0.58 0.55/0.58 0.59/0.64 0.59/0.63 0.76/0.77
Social Roles
(Rule based)
0.63/0.65 0.62/0.64 0.65/0.68 0.65/0.68 0.71/0.74 0.70/0.73 0.83/0.83
Table 7.3: Weighted F1-score for all the models. For each model, 2 scores are reported corresponding to
80 and 160 discussions respectively.
7.5 Conclusion
This chapter presents how deep learning can be utilized to train neural networks in order to identify
participants' behavioral characteristics and social roles in online discussions. Out of the three neural
network models presented in this chapter, Bidirectional LSTM model with doc2vec embeddings trained
on all the utterances in the discussion performs better than CNN and Bidirectional LSTM models trained
only on the participant's utterance. This implies that the neural network model benets from the context
provided to the network for identifying behavioral characteristics and social roles. For all the neural
network models, the rule based approach performs better than the models identifying the social roles
directly. One of the reason for this is data sparsity problem. Given the same amount of data, the models
identifying the roles directly have higher number of target values to distinguish from than the models
trying to identify behavioral characteristics. The rule based models benets from the knowledge of the
97
relationship between the behavioral characteristics and social roles, whereas the direct models have to
learn the structure of the social roles directly from the data.
The results also show that the SVM models perform better than their corresponding neural network
counterpart models. The amount of data available for training plays a major role for the difference in
the performance. Traditionally, neural network models require much more data to learn the network
parameters robustly from the data compared to SVM models to learn from the features provided to them
for the same task. When provided with more data, neural network models show higher improvement in
their performance compared to the SVM models. This implies that neural network models improve at
a much higher rate compared to SVM model when provided with more data. One important difference
between the neural network models and the SVM models is what kind of data the models are provided to
learn from. The neural network models use only words used by the participants whereas the SVM models
use features derived from words as well as discussion structure. SVM uses features like % of comments
as replies, % participant replied to by others, etc., that can be quite difficult for neural network models
to learn based on words only. Although, neural networks do outperform their counterpart bag-of-words
SVM models which use the same input data, i.e. words only.
Finally, the custom word embeddings presented in this chapter perform at least as well as word2vec
embeddings and in some cases better than them. This signies that the domain specic dependencies
among the words learned by the custom embeddings assist the neural network models for identifying the
participants' behavioral characteristics and social roles.
98
Chapter 8
Participants' Behavioral Analysis using Social Roles model
8.1 Introduction
In online environments, participants exhibit behavior based on their interest, knowledge, and how they
perceive the situation. Therefore, every individual behaves differently under different circumstances. The
social roles model presented in this dissertation is useful in understanding this diversity of participants'
behavior by analyzing specic behaviors. This in turn provides us with information that can be used to
devise strategies regarding how to behave towards these participants and what to expect from them.
In this chapter, I present the analysis of participants' behavior diversity in contentious online discus-
sions using the social roles model. Behavior is the action or reaction of an entity, human or otherwise,
to situations or stimuli in its environment. It is a key entity in understanding the driving forces and
cause-effects of many issues. However, behaviors in social media are only observed by the traces they
leave in social media. We rarely observe the driving factors that cause these behaviors; nor can we in-
terview individuals regarding their behaviors. I use the social roles model presented in Chapter 6 to tag
participants on Wikipedia: AfD discussions that took place between January 1, 2009 and June 30, 2012
for their social roles and analyze the behaviors they exhibit in these discussions. This behavior analysis
is aimed to understand how different factors affect individual behaviors observed online. In particular, I
investigate 3 potential factors that can affect participants' behavior in such discussions. 1) the topic of
99
the discussion, 2) the amount of contention observed by the participant in the discussion, and 3) other
participants present in the discussion. I identify participants' behavior in specic situation dened by the
individual factors mentioned above and compare them with their average behavior. The results show that
in most situations participants behave similarly. However, in scenarios where participants show substan-
tial change in their behavior, social roles model provide context regarding specic behavior affected by
the situation.
8.2 Data
# of discussions participated in # of participants
0-9 30572
10-25 1808
26-50 778
51-100 395
101-200 246
201-500 123
501-1000 33
1000+ 11
Table 8.1: Participation distribution in contentious discussions.
I collected all Wikipedia: AfD discussions that took place in the period of January 1, 2009 to June 30,
2012 using a web crawler. The data contains 92364 discussions with 811047 comments and 47638 distinct
participants. For the purpose of the analysis proposed in this chapter, the focus is limited to contentious
discussions only as the social roles model presented earlier identies the roles in such discussions. A
contentious discussion is dened as any discussion with at least 7 comments with at least 2 comments for
at least 2 of the stances a participant can take on Wikipedia discussions. Using this criteria for ltering
100
non-contentious discussions, the remaining data contains 26071 discussions with 470901 comments and
33966 distinct participants. Table 1 presents the distribution of participation in these discussions.
The purpose of the analysis presented in this chapter is to identify factors that contribute towards
participants' behavioral diversity in different discussions. Therefore, all the analysis presented in the
remainder of this chapter is focused on participants who have participated in at least 10 discussions in
the corpus.
8.3 Participants' Behavioral Diversity
I preprocess and tag the 26071 Wikipedia contentious discussions for social roles of participants in these
discussions using the social roles model presented in Chapter 6. These discussions consist of 3393 distinct
participants that participate in at least 10 of these discussions. The total participation from these partic-
ipants is 197361. I now present some statistics for these participants from the social roles analysis of the
tagged data.
Figure 8.1 presents the social role distribution in the contentious discussions incorporating all the
non-unique participants in all the discussions. As we can see, most of the participation is non-stubborn
and sensible, similar to the human annotated corpus presented earlier. We can also observe the relative
decreaseinin
uentialbehavior. Thisismainlyduetothefactthatthediscussionsinthehumanannotated
corpus are more contentious than the overall discussions in the behavior analysis corpus, which would
encourage participants to exhibit more leadership qualities to have an impact on the course/outcome of
the discussion. Figure 8.2-8.6 present the individual participants' average behavior in the discussions
they participate in. For all the participants, their average behavior is determined by calculating the %
of discussions where they exhibit certain behavior. For example, a participant can be stubborn in 40%
of the discussions he/she participates in. We can see from the distribution of the behavior diversity that
majority of the participants exhibit behavioral diversity under different situations. Very limited number
of participants are consistent with their behaviors in all discussions they participate in.
101
Figure 8.1: Social Roles distribution
Figure 8.2: Average Stubbornness
102
Figure 8.3: Average Sensibleness
Figure 8.4: Average Ignored-ness
103
Figure 8.5: Average Leadership
Figure 8.6: Average Followership
104
8.4 Behavioral diversity vs Topic of discussion
The rst factor I investigate for participants' behavioral diversity is the topic of the discussion they par-
ticipate in. People have different level of expertise/familiarity and interest in various elds/topics. This
may affect how they express themselves in discussions. In order to analyze participants' behavior in dif-
ferent topics, I rst use topic modeling to create clusters of discussions having similar topic. Although the
discussions under consideration for topic modeling are for some Wikipedia pages, many of the discussions
do not have their target Wikipedia page if the outcome of the discussion was to delete the Wikipedia page
under discussion. Therefore, the information I use for topic modeling of these discussions is the title of
the Wikipedia page under discussion and the content of the discussion.
K-means
1
algorithm is used to cluster the discussions. It assigns each discussion to one of K clusters
according to which centroid the discussion is close to by the similarity function. The clustering algorithm
stops when the centroid of each cluster does not change or the iteration of the algorithm exceeds a pre-
dened threshold. I use tf-idf values of words as feature and cosine distance as similarity function for
clustering. Estimating the number of the induced clusters, K, is difficult for clustering problems. I use
Silhouette [Rousseeuw, 1987] analysis to determine the number of clusters for the K-means algorithm.
Silhouette is used to validate the consistency within the clusters. It measures how close each discussion
in one cluster is to discussions in the neighboring clusters. This measure has a range of [-1, 1]. Silhouette
coefficients (as these values are referred to as) near +1 indicate that the sample is far away from the
neighboring clusters. A value of 0 indicates that the sample is on or very close to the decision boundary
betweentwoneighboringclustersandnegativevaluesindicatethatthosesamplesmighthavebeenassigned
tothewrongcluster. Iexperimentwith10-40clustersusingtheK-meansalgorithm. Astheinitialcentroid
of each cluster is assigned randomly, the K-means algorithm is run 10 times for each number of clusters.
The nal Silhouette coefficient value for each K2[10,40] number of clusters is calculated by averaging the
Silhouette coefficient value over the 10 runs for that particular K value. I then use the top 3 K values
1
K-means implementation of scikit-learn: http://scikit-learn.org/stable/modules/generated/sklearn.cluster.
KMeans.html
105
based on the average Silhouette coefficient to compute the corresponding clusters. In the end, I pick the
best K value for clusters by manually looking at the resulting clusters and choosing the one which has the
least ratio number of ambiguous clusters to the total number of clusters. Table 8.2 presents the resulting
clusters for K=20 along with the number of discussions in each cluster and their top 5 topic words.
Cluster No. # of discussions Top topic words
1 1104 news, event, story, notnews, media
2 372 relation, bilateral, countries, embassies, diplomatic
3 743 category, list, group, party, organization
4 557 award, pornbio, people, person, work
5 735 game, video, information, content, circball
6 804 local, news, politician, national, city
7 805 company, press, business, product, software
8 1908 information, content, character, material, history
9 4834 news, google, search, mention, hits
10 807 lm, movie, news, imdb, review
11 3360 information, people, research, original, content
12 605 school, high, primary, elementary, district
13 678 series, character, episode, show, plot
14 1086 book, author, published, work, review
15 1210 music, band, song, released, album
16 1786 person, work, news, prof, name
17 431 university, journal, academia, major, college
18 1083 term, word, dictionary, concept, denition
19 2435 people, time, point, content, case
20 727 league, professional, football, athlete, team
Table 8.2: Topic models for Wikipedia discussions.
106
Once the discussions are clustered, I analyze participants' behaviors in these clusters and compare
them with their average behaviors in all the discussions they participate in. I identify instances where any
participant appears in at least 5 discussions from any of the clusters. I then compute the participant's
behavior in that particular cluster by averaging the participant's behavior in each of those discussions.
In the end, for each such instance, I calculate the absolute difference between the participant's average
behavior in the cluster and the participant's average behavior in all the discussions he/she participated in.
For example, if a participant is 90% sensible in all the discussions he/she participated in and is only 40%
sensible in all the discussions he/she participated in that belong to cluster-x, then the absolute difference
for sensibleness for the participant for cluster-x is 50. Using the topic model presented above, I identied
9490 instances where a participant participated in at least 5 discussions belonging to one cluster. Figure
8.7 presents the distribution for the absolute behavior difference for all the behaviors in each of the 9490
instances.
Figure 8.7: Behavioral diversity vs Topic of discussion
Based on the distribution shown above, we can observe that there are <50% instances where partici-
pants show a change of >10% in behavior while participating in discussions within a topic cluster. This
follows the norm that most of the participants behave similarly regardless the topic they are discussing.
107
Excluding in
uence, the rest of the behavior have >10% instances where participants show a change of
>20% in discussions within a topic cluster. Here are some examples of notable change in certain behavior
for participants within specic topic clusters.
8.4.1 Examples
\llywrch" participated in 24 discussions within cluster-2 showing 37% sensibleness compared to 76%
sensibleness overall in 182 discussions. This implies that although the participant is sensible in
most discussions he/she participates in; i.e. the participant's comments help the discussion in a
positive manner to move forward; within cluster-2, the participant's contributions were considerably
less helpful for the discussion. Within cluster-2, the participant was 17% stubborn, 58% ignored,
0% leader, and 25% follower. Figure 8.8 shows some examples of non-sensible comments from the
participantinthediscussionswithincluster-2wheretheparticipantdoesn'thaveanyrelevantclaims
of his/her own and often indulges in irrelevant discussions.
Figure 8.8: Sensibleness diversity based on topic of the discussion
\paulmcdonald"participatedin17discussionswithincluster-5showing82%stubbornnesscompared
to 48% stubbornness overall in 191 discussions. This implies that the participant was considerably
more stubborn within cluster-5; i.e. the participant had higher amount of participation and was
unwilling to change the stance; compared to his/her other discussions. Within cluster-5, the par-
ticipant was 88% sensible, 35% ignored, 18% leader, and 0% follower. Figure 8.9 shows one such
instance of stubbornness shown by the participant in the discussions within cluster-5.
108
Figure 8.9: Stubbornness diversity based on topic of the discussion
8.5 Behavioral Diversity vs Contention Observed
The second factor I investigate is the amount of contention towards the stance participants take in the
discussion. AswesawintheanalysisshowninChapter4, differentamountofcontentionattractsdifferent
amount of participation and hence different behaviors from participants. In this analysis, I use the con-
tention observed by the participants towards their stance in the discussion as the criteria for determining
behavioraldiversity. Theamountofcontentioninadiscussiontowardsastanceisprimarilydenedbythe
ratio of the number of comments supporting the stance and the number of comments opposing it in the
discussion. Forexample, ifadiscussionhas5commentssupportingstanceAand10commentssupporting
stance B, then the amount of contention towards participants with stance A will be 2:1 and similarly
the amount of contention towards participants with stance B will be 1:2. But, this ratio is insufficient in
determining the actual contention observed by the participants as it doesn't take into account the actual
strength of any of the stance in the discussion. For example, consider discussion-1 with 5 comments
supporting stance A & 10 comments supporting stance B and discussion-2 with 50 comments supporting
stance A & 100 comments supporting stance B. In both discussions, the relative amount contention for
109
both stances is same, but participants in discussion-2 will observe higher amount of contention than par-
ticipants in discussion-1 because of the actual strength of each stance in discussion-2. Taking this into
consideration, I dene the contention observed by any participant towards his/her stance in a discussion
using the following components:
Stance ratio: Ratio of number of opposing stance comments to number of supporting stance
comments. This component calculates the tilt in the discussion towards the opposing stance.
Opposing stance strength: Number of opposing stance comments. This component incorporates
how strong the tilt towards the opposing stance is.
The nal equation for the contention observed towards an stance in a discussion is:
Contentionobserved =
(#ofopposingstancecomments)
2
(#ofsupportingstancecomments)
(8.1)
Figure 8.10: Observed contention distribution
110
Note that the amount of contention observed by participants having the same stance in a discussion
is same regardless the number of comments by the individual participants. I calculate the amount of
contention in each discussion for each of the 3 possible stances discussed in Section 3.2.1: Delete, Keep,
andOther(Merge,Redirect,orTranswiki). Figure8.10showsthedistributionoftheamountofcontention
for any particular stance versus the number of discussions that fall into that category. Discussions missing
any of the 3 stances are not incorporated for the corresponding stance distribution.
In order to analyze how participants behave differently while observing different amount on contention
in the discussion, for each participant, I divide the discussions into two categories: discussions where
participant observes low contention and discussions where participant observes high contention. Keep in
mind that all the discussions under the analysis are contentious based on the selection criteria mentioned
previously. As there is no principled way of deciding the threshold between low and high contentious
discussions, I determine the threshold such that both the categories have enough discussions for the
analysis. I set the threshold of 20 between the two categories of contention, i.e. discussions having
contention of 20 or lower towards any stance are categorized as low contentious discussion for that stance
and the remaining are categorized as high contentious discussions for that stance.
Figure 8.11: Behavioral diversity vs Observed contention
111
Nowforeachparticipant,Idividetheparticipant'sdiscussionsintothetwocontentioncategoriesbased
on the stance the participant takes in the corresponding discussions. For fair comparison, I consider the
participantswhohaveatleast4discussionsinboththecategories. Ithencalculatetheaveragebehaviorof
the participant in both categories and determine the behavioral diversity of the participant by calculating
the absolute difference between the two averages. Figure 8.11 presents the distribution for the absolute
behavior difference for the threshold of 20 amount of contention with 1504 participants who satisfy the
minimum participation criteria.
The behavioral diversity for participants based on the amount of contention observed by them in the
discussion show that <50% participants show almost similar behavior (<10% absolute difference) for low
and high contention discussions for all behaviors except for in
uence. This implies that majority of the
participants show diverse behavior when the amount of contention in the discussion varies. Here are some
examples of notable change in certain behavior based on the amount of contention observed.
8.5.1 Examples
\sawomir biay" participated in 9 high contentious discussions and 23 low contentious discussions.
The participant has 55% leadership, 78% stubbornness, and 22% ignored-ness in high contentious
discussions compared to 4% leadership, 39% stubbornness, and 65% ignored-ness in low contentious
discussions. The participant seems to put in a lot more effort in high contentious discussions and
is able to in
uence others through his contributions. The absolute difference for the remaining
behaviors for the participant between high and low contentious discussions is 5% and 7% for sen-
sibleness and followership respectively. Figure 8.12 shows an example of participant's contribution
to a high contentious discussion and another example of the ability of the participant to in
uence
others through the contribution.
112
Figure 8.12: Leadership diversity based on contention observed in the discussion
\2"participatedin8highcontentiousdiscussionsand23lowcontentiousdiscussions. Theparticipant
has25%sensiblenessinhighcontentiousdiscussionscomparedto78%sensiblenessinlowcontentious
discussions. This implies that in high contentious discussions, the participant's contribution was not
as helpful for the discussion as it was in low contentious discussions. The absolute difference for the
remaining behaviors for the participant between high and low contentious discussions is 10%, 10%,
0%, and 8% for stubbornness, ignored-ness, leadership, and followership respectively. Figure 8.13
shows examples of participant's contribution in high contentious discussions where the participant
often just follows others rather than presenting any additional arguments to support the stance.
Figure 8.13: Sensibleness diversity based on contention observed in the discussion
8.6 Diversity vs Other Participants
The last factor I investigate is whether participants' behavior depends on their fellow participants in the
discussions. Participants may behave differently with different individuals but it is likely that they behave
113
similarlywiththesameindividuals. Alliesarelikelytosupporteachotherwhenevertheyaretogetherina
discussion and similarly, adversaries are likely to attack each other when facing off in the same discussion.
Figure 8.14: Behavioral diversity vs Other participants
I identify pairs of participants who participated together in at least 5 discussions. For each such
pair, I compute the average behavior of each participant in these common discussions. I then calculate
the absolute difference between the participant's average behavior in the common discussions and the
participant's average behavior in all the discussions he/she participated in. Figure 8.14 presents the
distribution of the absolute difference between all the behaviors for 62990 instances for the 31495 pairs of
participants.
The behavioral diversity for participants based their fellow participants in the discussions show that
in >50% of instances participants show almost similar behavior (<10% absolute difference) when they
appearwithsomeotherparticipantinatleast5differentdiscussions. Thisimpliesthatthemajorityofthe
participants don't get bothered with who they are discussing with. But there are a few instances where
the behavior of participants change signicantly in discussions when they are paired with some particular
114
individuals. Here are some examples of notable change in certain behavior based on the other participants
in the discussions.
8.6.1 Examples
\hasteur"and\hullaballoowolfowitz"participatedtogetherin29discussions. Onaverage,\hasteur"
is 27% follower and \hullaballoo wolfowitz" is 14% leader in all the discussions they participated
in. But, within the 29 discussions that they appeared together, \hasteur" is 100% follower and
\hullaballoo wolfowitz" is 100% leader. This tells us that whenever these two participants appear
together in any discussions, \hasteur" always support any contribution made by \hullaballoo wol-
fowitz" in the discussion. Also, within these 29 discussions, \hasteur" was 0% stubborn and 0%
sensible, whereas \hullaballoo wolfowitz" was 7% stubborn and 100% sensible. Figure 8.15 shows a
couple of examples of the endorsements by \hasteur" towards \hullaballoo wolfowitz".
Figure 8.15: Leader/Follower diversity based on other participants in the discussion
\ikip"and\thuranx"participatedtogetherin24discussions. Theyalwayshadoppositestances. On
average,\ikip"and\thuranx"are41%and39%stubbornrespectively. But,withinthe24discussions
thattheyappeartogether, \ikip"is92%stubbornand\thuranx"is100%stubborn. Also, theyboth
are 100% sensible in all these discussions. This signies that the participants had formed a rivalry
toward each other by strongly supporting their own stance with substantial contribution towards
the discussion. Figure 8.16 shows a couple of examples where the participants interact with each
other in these discussions.
115
Figure 8.16: Stubbornness diversity based on other participants in the discussion
8.7 Conclusion
This chapter investigates three factors that may affect participants' behavior in online contentious dis-
cussions using the social roles model. The analysis presented in this chapter in large follows the common
norms that participants in online discussions behave in similar manner for the majority of the discussions
they participate in. However, using the social roles model we can identify certain situations where the
participants show considerable diversity in their behavior. In particular, the social roles model provides
useful context regarding the rationale for such a change in behavior by identifying the specic behaviors
that are affected in such situations. This can be useful in targeted analysis of participants in specic
situations and it also provides with some insights regarding what to expect from them in those situations.
Oneofthemajorshortcomingsoftheanalysispresentedinthischapteristhatit'sdifficulttoverifythe
validity of these behavioral patterns based on the contributing factors. The behavioral patterns present in
the discussions can be due to some other factors that cannot be observed in the discussions. For example,
some participant being stubborn in a discussion can be due to the fact that the discussion is directly
related to the participant (e.g., author of the Wikipedia page under discussion).
116
Chapter 9
Conclusion
In this dissertation, I have developed a theory and have proposed and implemented a framework for the
identication of social roles in online contentious discussions. The framework allows for participants'
social roles to be dened based on the behaviors they exhibit in the discussions. The results show that
using the same set of features, identifying social roles through participants' behaviors perform better than
identifying the social roles directly. In this chapter, I summarize the contributions of this dissertation and
discuss future directions.
9.1 Framework for Social Roles
I developed a framework for dening social roles in online communities by analyzing the behaviors of
the members of the community. Using this framework, I created a corpus for social roles in online
discussions. Todate,nosuchcorpusisavailableforanalyzingparticipantsinsuchenvironment. Thecorpus
is annotated with 8 distinct participant roles comprising of 4 behavioral characteristics in two different
discussion domains. In future, the social roles framework and the annotation manual for the corpora can
be used as a reference for developing similar corpora in other online communities like healthcare support
forums or question/answer forums.
117
9.2 Leadership Models
Ideveloptwomodelstoidentifyleadersinonlinecontentiousdiscussionsbyquantifyingvariousleadership
qualitiesshownbytheparticipantsinsuchdiscussions. The\ContentLeader"modelquantiesleadership
qualities like attracting followers and counterattacking your opponents through your contribution. The
\SilentOut Leader" model quanties leadership qualities like presenting factual arguments and winning
small battles within the discussion. In future, these leadership models can be used to develop automated
online recommendation systems. A newcomer in an online community can be recommended to be paired
with a leader in the community to enhance the experience of the newcomer. An inquirer on a Q/A
forum can be recommended to contact someone who has shown leadership qualities in answering similar
questions.
9.3 Argumentation Structure
The Argumentation structure analysis presented in this dissertation identies specic claims within com-
ments and then links these claims based on the causal/conditional dependencies between them. The
analysis is divided into 3 sub-parts: detect sentences that express claims, identify specic claims within
such sentences, and discover links between the claims if any. In future, the argumentation structure anal-
ysis can be extended into a fact-checking system. An argument is a set of premises, pieces of evidence,
offered in support of a claim. The argumentation structure is useful for differentiating the claim from its
supporting pieces of evidences. Once we determine the pieces of evidence, we can use the knowledge base
to verify the evidence and in turn the truth value of the claim. The argumentation structure can also be
used in developing a system that matches claims with their counterclaims.
118
9.4 Social Roles models
I develop the social roles model to automatically determine participants' roles in online contentious dis-
cussions using traditional classication techniques and neural network structures. The subcomponents
of the model identies the values of the behavioral characteristics of the participants by analyzing their
contribution in the discussion. Then, using these values, the social roles are determined based on the
social roles framework. In future, the social roles model can be used to summarize the discussions or
to predict the outcome of the discussion. The participants' contributions can be given weight based on
the role they exhibit in the discussion. Using these weights, a summary can be created for each stance
in the discussion, which can be further used in summarizing the discussion. Similarly, the summary for
each stance can also be used to determine what in
uence they might have on the nal outcome of the
discussion.
9.5 Social Roles model as Analytic Tool
I use the social roles model as a tool to analyze participants' behavioral diversity over time in online
discussions. I tag a large corpus of online discussions and investigate factors that may affect participants'
behavior in such discussions. I investigate three factors: 1) topic of the discussion, 2) contention observed
by the participant in the discussion, and 3) other participants in the discussion. In future, this kind of
analysis can be used for user proling. Based on this analysis we learn important information about the
participants regarding how they behave and respond to specic situations.
Thesocialrolesmodelpresentinthisdissertationcanbeusedasananalytictoolforvarioussocialmedia
environments to understand the participants' behavior in order to further develop tools to enhance the
user experience of these participants. The framework for the social roles can be generalized to incorporate
different behaviors and roles prevalent in other online communities.
119
Appendix A
Social Roles Annotation Manual
The annotation manual for Social Roles for contentious discussions was created by three annotators after
several weeks of discussions while annotating different contentious discussions from Wikipedia: Articles
for Deletion and 4forums.com forums. In the nal annotation manual, the problem of identifying Social
Roles was broken down to identifying the values for certain behavioral characteristics of the participants
and then deciding the social role based on those values. The annotators identied four principal charac-
teristics(Stubbornness, Sensibleness, Ignored-ness, andIn
uence)andeightsocialroles(Leader, Follower,
Rebel, Voice in Wilderness, Idiot, Nothing, Nothing Sensible, and Other). The relationship between the
social roles and their corresponding characteristic values are described in Table 3.4. Determining some
of the behavioral characteristics is subjective and therefore the annotators dene some criteria to gather
knowledge about the participant and then determine the value for the characteristic based on the overall
information. I now present the rules for determining the values for each behavioral characteristic.
A.1 Stubbornness
Stubbornnessofaparticipantisdeterminedbasedonhowdevotedhe/sheistothediscussionandtoargue
for or defend his/her stance. The annotators seek to answer the following questions and determine the
stubbornness value based on the overall information. Each question has value attached to the possible
answers which points in the direction of the nal stubbornness value.
\Does the participant have more than average comments?" fYes)Stubborn, No)Not-Stubborng
\If the participant has less than average comments, does he/she have substantial contribution is
his/her comments? fYes)Stubborn, No)Not-Stubborng
\Does the participant have his/her own claims/arguments?" fYes)Stubborn, No)Not-Stubborng
\Doestheparticipantchangehis/herstanceduringthediscussion?" fYes)Indeterminable,No)Stubborng
A.2 Sensibleness
Sensiblenessanalysisofaparticipantishighlysubjectiveanddomaindependent. Thereforetheannotators
dene questions that are domain independent and determine the sensibleness value based on the overall
information and the domain the participant belongs to. Each question has value attached to the possible
answers which points in the direction of the nal sensibleness value.
\Doestheparticipantsoundreasonableandknowledgeable?" fYes)Sensible,No)Not-Sensible/Indeterminableg
\How many positive/negative responses does the participant have?" (Sometimes due to lack of the
domain knowledge about the topic of discussion, the annotators rely on how participants respond
to one another.) fMore Positive ) Sensible, More negative ) Not-Sensible, 0 ) Indeterminableg
120
\Does the participant start or get involved in tangential discussion?" fYes ) Not-Sensible, No )
Sensibleg
\How much emotion does the participant express and what is the tone of it?" fSensibleness value
for this question is quantied based on the amount of emotions compared with the amount of
argumentationg
\Does the participant mention Wikipedia policies?" (For Wikipedia discussions only) fYes ) Sen-
sible, No ) Not-Sensible/Indeterminableg
A.3 Ignored-ness
Ignored-ness value is determined based on how other participants responds to a particular participant.
The annotators seek to answer the following questions and determine the ignored-ness value based on
the overall information. Each question has value attached to the possible answers which points in the
direction of the nal stubbornness value.
\Does the participant have direct response to his/her comments?" fYes ) Not-Ignored, No )
Ignoredg
\Does any other participant mention this participant in his/her comments?" fYes ) Not-Ignored,
No ) Ignoredg
A.4 In
uence
In
uencevalueisdeterminedbasedonwhethertheparticipantisabletoin
uenceothersortheparticipant
isin
uencedbyothers. Theannotatorsseektoanswerthefollowingquestionsanddeterminethein
uence
value based on the overall information. Each question has value attached to the possible answers which
points in the direction of the nal stubbornness value.
\Does any other participant endorse this participant for his/her contribution?" fYes ) Leader, No
) Nothingg
\Does this participant endorse other participants for their contribution?" fYes ) Follower, No )
Nothingg
121
Appendix B
Neural Network Hyperparameters
B.1 CNN-user
Parameters Values
min 5
dim 200
win 5
neg 10
alpha 0.05
Table B.1: Hyperparameter values for custom word embeddings for CNN-user
Model
Parameter Stubbornness Sensibleness Ignored-ness In
uence
Social Roles
(Directly)
batch-size 50/50 50/50 50/50 50/50 50/50
dropout rate 0.5/0.5 0.5/0.5 0.3/0.3 0.5/0.5 0.3/0.3
epochs 4/4 4/4 4/4 4/4 4/4
learning rate 0.001/0.001 0.001/0.001 0.001/0.001 0.001/0.001 0.0001/0.0001
kernel size 5/5 5/5 5/5 5/5 5/5
number of lters 100/100 100/100 100/100 100/100 100/100
hidden dimensions 100/100 100/100 100/100 100/100 100/100
activation functions relu/relu relu/relu relu/relu relu/relu relu/relu
Table B.2: Hyperparameter values for CNN-user with custom embedding and word2vec respectively
122
B.2 BiLSTM-user
Parameters Values
min 5
dim 200
win 5
neg 5
alpha 0.025
Table B.3: Hyperparameter values for custom word embeddings for BiLSTM-user
Model
Parameter Stubbornness Sensibleness Ignored-ness In
uence
Social Roles
(Directly)
batch-size 50/50 50/50 50/50 50/50 50/50
dropout rate 0.5/0.5 0.5/0.5 0.5/0.5 0.5/0.5 0.3/0.3
epochs 4/4 4/4 4/4 4/4 4/4
learning rate 0.001/0.001 0.001/0.001 0.001/0.001 0.001/0.001 0.0001/0.0001
hidden dimensions 100/100 100/100 100/100 100/100 100/100
activation functions relu/relu relu/relu relu/relu relu/relu relu/relu
Table B.4: Hyperparameter values for BiLSTM-user with custom embedding and word2vec respectively
B.3 BiLSTM-all
Parameters Values
Vector size 200
Context window size 10
Minimum count 5
Negative sample size 10
Table B.5: Hyperparameter values for doc2vec for BiLSTM-all
Parameters Values
Input layer dimension 400
Hidden layer dimension 200
Activation function relu
Table B.6: Hyperparameter values for autoencoder for BiLSTM-all
123
Model
Parameter Stubbornness Sensibleness Ignored-ness In
uence
Social Roles
(Directly)
batch-size 50/50 50/50 50/50 50/50 50/50
dropout rate 0.5/0.5 0.5/0.5 0.3/0.3 0.5/0.5 0.5/0.5
epochs 4/4 4/4 4/4 4/4 4/4
learning rate 0.001/0.001 0.001/0.001 0.0001/0.0001 0.001/0.001 0.001/0.001
hidden dimensions 100/100 100/100 100/100 100/100 100/100
activation functions relu/relu relu/relu relu/relu relu/relu relu/relu
Table B.7: Hyperparameter values for BiLSTM-all with doc2vec and autoencoder respectively
124
Reference List
[Abbott et al., 2011] Abbott, R., Walker, M., Anand, P., Fox Tree, J. E., Bowmani, R., and King, J.
(2011). How can you say such things?!?: Recognizing disagreement in informal political argument. In
Proceedings of the Workshop on Languages in Social Media, LSM '11, pages 2{11, Stroudsburg, PA,
USA. Association for Computational Linguistics.
[Aharoni et al., 2014] Aharoni, E., Polnarov, A., Lavee, T., Hershcovich, D., Levy, R., Rinott, R., Gutfre-
und, D., and Slonim, N. (2014). A benchmark dataset for automatic detection of claims and evidence.
In in the Context of Controversial Topics", in Proceedings of the First Workshop on Argumentation
and Computation, ACL 2014.
[Aleahmad et al., 2016] Aleahmad, A., Karisani, P., Rahgozar, M., and Oroumchian, F. (2016). Olnder.
J. Inf. Sci., 42(5):659{674.
[Amir et al., 2016] Amir, S., Wallace, B. C., Lyu, H., Carvalho, P., and Silva, M. J. (2016). Modelling
contextwithuserembeddingsforsarcasmdetectioninsocialmedia. In Proceedings of the 20th SIGNLL
Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11-
12, 2016, pages 167{177.
[Angeletou et al., 2011] Angeletou, S., Rowe, M., and Alani, H. (2011). Modelling and Analysis of User
Behaviour in Online Communities, pages 35{50. Springer Berlin Heidelberg, Berlin, Heidelberg.
[Aral and Walker, 2012] Aral, S. and Walker, D. (2012). Identifying in
uential and susceptible members
of social networks. Science, 337(6092):337{341.
[Argamon et al., 2009] Argamon,S.,Koppel,M.,Pennebaker,J.W.,andSchler,J.(2009). Automatically
proling the author of an anonymous text. Commun. ACM, 52(2):119{123.
[Aristotle, 1954] Aristotle (1954). The Thetorics and Poetics. Translated by W. Rhys Roberts, Modern
Library.
[Arvapally et al., 2012] Arvapally, R.S., Liu, X.F., andJiang, W.(2012). Identicationoffactiongroups
and leaders in web-based intelligent argumentation system for collaborative decision support. In Pro-
ceedings of 6th Annual ISC Graduate Research Symposium ISC-GRS.
[Bakshy et al., 2011] Bakshy, E., Hofman, J. M., Mason, W. A., and Watts, D. J. (2011). Everyone's an
in
uencer: Quantifyingin
uenceontwitter. InProceedingsoftheFourthACMInternationalConference
on Web Search and Data Mining, WSDM '11, pages 65{74, New York, NY, USA. ACM.
[Baldi, 2012] Baldi, P. (2012). Autoencoders, unsupervised learning, and deep architectures. In Unsu-
pervised and Transfer Learning - Workshop held at ICML 2011, Bellevue, Washington, USA, July 2,
2011, pages 37{50.
[Biddle, 1986] Biddle, B. J. (1986). Recent development in role theory. Annual Review of Sociology,
12:67{92.
125
[Biran and Rambow, 2011a] Biran, O. and Rambow, O. (2011a). Identifying justications in written
dialogs. In Proceedings of the 2011 IEEE Fifth International Conference on Semantic Computing,
ICSC '11, pages 162{168, Washington, DC, USA. IEEE Computer Society.
[Biran and Rambow, 2011b] Biran, O. and Rambow, O. (2011b). Identifying justications in written
dialogs. In Proceedings of the 2011 IEEE Fifth International Conference on Semantic Computing,
ICSC '11, pages 162{168, Washington, DC, USA. IEEE Computer Society.
[Bracewell et al., 2012] Bracewell, D. B., Tomlinson, M., and Wang, H. (2012). A motif approach for
identifying pursuits of power in social discourse. In Proceedings of the 2012 IEEE Sixth International
Conference on Semantic Computing, ICSC '12, pages 1{8, Washington, DC, USA. IEEE Computer
Society.
[Budalakoti et al., 2009] Budalakoti, S., DeAngelis, D., andBarber, K.S.(2009). Expertisemodelingand
recommendation in online question and answer forums. In Proceedings of the 12th IEEE International
Conference on Computational Science and Engineering, CSE 2009, Vancouver, BC, Canada, August
29-31, 2009, pages 481{488.
[Buntain and Golbeck, 2014] Buntain, C. and Golbeck, J. (2014). Identifying social roles in reddit using
network structure. In Proceedings of the 23rd International Conference on World Wide Web, WWW
'14 Companion, pages 615{620, New York, NY, USA. ACM.
[Catizone et al., 2012] Catizone, R., Guthrie, L., Thomas, A., and Wilks, Y. (2012). Lie: Leadership,
in
uence and expertise. In Proceedings of the Eight International Conference on Language Resources
and Evaluation (LREC'12), Istanbul, Turkey. European Language Resources Association (ELRA).
[Chan et al., 2010] Chan, J., Hayes, C., and Daly, E. (2010). Decomposing discussion forums and boards
using user roles.
[Chen et al., 2016] Chen, Y., Li, H., Zhang, J., and Miao, G. (2016). Inferring Social Roles of Mobile
Users Based on Communication Behaviors, pages 402{414. Springer International Publishing, Cham.
[Chen et al., 2014] Chen, Y., Wang, X., Tang, B., Xu, R., Yuan, B., Xiang, X., and Bu, J. (2014).
Identifying Opinion Leaders from Online Comments,pages231{239. SpringerBerlinHeidelberg,Berlin,
Heidelberg.
[Choobdar et al., 2015] Choobdar, S., Ribeiro, P., Parthasarathy, S., and Silva, F. (2015). Dynamic
inference of social roles in information cascades. Data Mining and Knowledge Discovery, 29(5):1152{
1177.
[Chou and Suzuki, 2010] Chou, B.-H. and Suzuki, E. (2010). Discovering community-oriented roles of
nodes in a social network. In Proceedings of the 12th International Conference on Data Warehousing
and Knowledge Discovery, DaWaK'10, pages 52{64, Berlin, Heidelberg. Springer-Verlag.
[Clemson and Evans, 2012] Clemson, T. and Evans, T. (2012). The emergence of leadership in social
networks. Physica A: Statistical Mechanics and its Applications, 391(4):1434 { 1444.
[Cocarascu and Toni, 2017] Cocarascu, O. and Toni, F. (2017). Identifying attack and support argu-
mentative relations using deep learning. In Proceedings of the 2017 Conference on Empirical Methods
in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages
1374{1379.
[Collobert et al., 2011] Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., and Kuksa,
P.P.(2011). Naturallanguageprocessing(almost)fromscratch. JournalofMachineLearningResearch,
12:2493{2537.
126
[Corsini, 2002] Corsini, R. (2002). The Dictionary of Psychology. Brunner/Routledge, New York, N.Y.
[Danescu-Niculescu-Mizil et al., 2013] Danescu-Niculescu-Mizil, C., Sudhof, M., Jurafsky, D., Leskovec,
J., and Potts, C. (2013). A computational approach to politeness with application to social factors.
CoRR, abs/1306.6078.
[Dong et al., 2007] Dong, W., Lepri, B., Cappelletti, A., Pentland, A. S., Pianesi, F., and Zancanaro, M.
(2007). Using the in
uence model to recognize functional roles in meetings. In Proceedings of the 9th
International Conference on Multimodal Interfaces, ICMI '07, pages 271{278, New York, NY, USA.
ACM.
[Doran, 2015] Doran, D. (2015). Triad-Based Role Discovery for Large Social Systems, pages 130{143.
Springer International Publishing, Cham.
[Du et al., 2016] Du, F., Liu, Y., Liu, X., Sun, J., and Jiang, Y. (2016). User role analysis in online social
networks based on dirichlet process mixture models. In 2016 International Conference on Advanced
Cloud and Big Data (CBD), pages 172{177.
[Duan et al., 2014] Duan, J., Zeng, J., and Luo, B. (2014). Identication of opinion leaders based on user
clustering and sentiment analysis. In 2014 IEEE/WIC/ACM International Joint Conferences on Web
Intelligence (WI) and Intelligent Agent Technologies (IAT), volume 1, pages 377{383.
[Ennals et al., 2010] Ennals, R., Trushkowsky, B., andAgosta, J.M.(2010). Highlightingdisputedclaims
ontheweb. InProceedings of the 19th International Conference on World Wide Web, WWW'10, pages
341{350, New York, NY, USA. ACM.
[Fazeen et al., 2011] Fazeen, M., Dantu, R., and Guturu, P. (2011). Identication of leaders, lurkers,
associates and spammers in a social network: context-dependent and context-independent approaches.
Social Network Analysis and Mining, 1(3):241{254.
[Fuger et al., 2017] Fuger, S., Schimpf, R., Fuller, J., and Hutter, K. (2017). Network structure and user
roles of a crowdsourcing community - the context of social innovations for a development project. In
HICSS.
[Fller et al., 2014] Fller, J., Hutter, K., Hautz, J., and Matzler, K. (2014). User roles and contributions
in innovation-contest communities. Journal of Management Information Systems, 31(1):273{308.
[Garg et al., 2008] Garg, N. P., Favre, S., Salamin, H., Hakkani T ur, D., and Vinciarelli, A. (2008). Role
recognition for meeting participants: An approach based on lexical information and social network
analysis. In Proceedings of the 16th ACM International Conference on Multimedia, MM '08, pages
693{696, New York, NY, USA. ACM.
[Gilpin et al., 2013] Gilpin, S., Eliassi-Rad, T., and Davidson, I. (2013). Guided learning for role dis-
covery (glrd): Framework, algorithms, and applications. In Proceedings of the 19th ACM SIGKDD
International Conference on Knowledge Discovery and Data Mining, KDD '13, pages 113{121, New
York, NY, USA. ACM.
[Gliwa et al., 2013] Gliwa, B., Zygmunt, A., and Ko zlak, J. (2013). Analysis of Roles and Groups in
Blogosphere, pages 299{308. Springer International Publishing, Heidelberg.
[Golder, 2003] Golder, S. (2003). A Typology of Social Roles in Usenet.
[Golder and Donath, 2004] Golder, S. A. and Donath, J. (2004). Social roles in electronic communities.
In in Association of Internet Researchers (AoIR) conference Internet Research 5.0.
[Graves, 2012] Graves, A. (2012). Supervised Sequence Labelling with Recurrent Neural Networks, volume
385 of Studies in Computational Intelligence. Springer.
127
[Habernal and Gurevych, 2016] Habernal, I. and Gurevych, I. (2016). What makes a convincing argu-
ment? empirical analysis and detecting attributes of convincingness in web argumentation. In Pro-
ceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016,
Austin, Texas, USA, November 1-4, 2016, pages 1214{1223.
[Hecking et al., 2015] Hecking, T., Chounta, I.-A., and Hoppe, H. U. (2015). Analysis of user roles and
the emergence of themes in discussion forums. In Proceedings of the 2015 Second European Network
Intelligence Conference, ENIC '15, pages 114{121, Washington, DC, USA. IEEE Computer Society.
[Hennis, 2011] Hennis, T. (2011). Monitoring Contributions Online: A Reputation System to Model Ex-
pertise in Online Communities, pages 422{425. Springer Berlin Heidelberg, Berlin, Heidelberg.
[Hung and Yeh, 2014] Hung,C.andYeh,P.-W.(2014). Identicationofopinionleadersusingtextmining
technique in virtual community. In SIMBig.
[Jain, 2016] Jain, S. (2016). Identifying sensible participants in online discussions. In Proceedings of The
Fourth International Workshop on Natural Language Processing for Social Media, pages 41{47, Austin,
TX, USA. Association for Computational Linguistics.
[Jain et al., 2014] Jain, S., Bhatia, A., Rein, A., and Hovy, E. (2014). A corpus of participant roles in
contentious discussions. In Proceedings of the Ninth International Conference on Language Resources
and Evaluation (LREC'14), Reykjavik, Iceland. European Language Resources Association (ELRA).
[Jain and Hovy, 2013] Jain, S.andHovy, E.H.(2013). Determiningleadershipincontentiousdiscussions.
In ICME Workshops.
[Jing and Lizhen, 2014] Jing, L. and Lizhen, X. (2014). Identication of microblog opinion leader based
on user feature and interaction network. In 2014 11th Web Information System and Application Con-
ference, pages 125{130.
[Jones et al., 2011] Jones, S., Poulsen, A., Maiden, N., and Zachos, K. (2011). User roles in asynchronous
distributed collaborative idea generation. In Proceedings of the 8th ACM Conference on Creativity and
Cognition, pages 349{350, New York, NY, USA. ACM.
[Joshi and Penstein-Ros e, 2009] Joshi, M. and Penstein-Ros e, C. (2009). Generalizing dependency fea-
turesforopinionmining. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, ACLShort
'09, pages 313{316, Stroudsburg, PA, USA. Association for Computational Linguistics.
[Junquero-Trabado and Dominguez-Sal, 2012] Junquero-Trabado, V. and Dominguez-Sal, D. (2012).
Building a role search engine for social media. In Proceedings of the 21st International Conference
on World Wide Web, WWW '12 Companion, pages 1051{1060, New York, NY, USA. ACM.
[Kalchbrenner et al., 2014] Kalchbrenner, N., Grefenstette, E., and Blunsom, P. (2014). A convolutional
neural network for modelling sentences. In Proceedings of the 52nd Annual Meeting of the Association
for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, Volume 1: Long
Papers, pages 655{665.
[Kelly et al., 2006] Kelly, J. W., Fisher, D., and Smith, M. A. (2006). Friends, foes, and fringe: norms
and structure in political discussion networks. In DG.O.
[Kim and Hovy, 2004] Kim, S.-M. and Hovy, E. (2004). Determining the sentiment of opinions. In Pro-
ceedings of the 20th International Conference on Computational Linguistics,COLING'04,Stroudsburg,
PA, USA. Association for Computational Linguistics.
128
[Kim, 2014a] Kim, Y. (2014a). Convolutional neural networks for sentence classication. In Proceedings
of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October
25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1746{
1751.
[Kim, 2014b] Kim, Y. (2014b). Convolutional neural networks for sentence classication. In Proceedings
of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October
25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1746{
1751.
[Koch et al., 2013] Koch, G., Hutter, K., Decarli, P., Hilgers, D., and Fller, J. (2013). Identifying partic-
ipants' roles in open government platforms and its impact on community growth. In 2013 46th Hawaii
International Conference on System Sciences, pages 1900{1910.
[Kumar and Pedanekar, 2016] Kumar, V. and Pedanekar, N. (2016). Mining shapes of expertise in online
social q&a communities. In Proceedings of the 19th ACM Conference on Computer Supported
Cooperative Work and Social Computing Companion, CSCW '16 Companion, pages 317{320, New
York, NY, USA. ACM.
[Kwon et al., 2007] Kwon, N., Zhou, L., Hovy, E., and Shulman, S. W. (2007). Identifying and classifying
subjective claims. In Proceedings of the 8th Annual International Conference on Digital Government
Research: Bridging Disciplines & Domains, dg.o '07, pages 76{81. Digital Government Societyof North
America.
[Labatut et al., 2014] Labatut, V., Dugu, N., and Perez, A. (2014). Identifying the community roles of
social capitalists in the twitter network. In 2014 IEEE/ACM International Conference on Advances in
Social Networks Analysis and Mining (ASONAM 2014), pages 371{374.
[Lai et al., 2015] Lai, S., Xu, L., Liu, K., and Zhao, J. (2015). Recurrent convolutional neural networks
for text classication. In Proceedings of the Twenty-Ninth AAAI Conference on Articial Intelligence,
January 25-30, 2015, Austin, Texas, USA., pages 2267{2273.
[Lapesa and Evert, 2014] Lapesa, G. and Evert, S. (2014). A large scale evaluation of distributional
semantic models: Parameters, interactions and model selection. Transactions of the Association for
Computational Linguistics, 2:531{545.
[Le and Mikolov, 2014] Le, Q. V. and Mikolov, T. (2014). Distributed representations of sentences and
documents. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014,
Beijing, China, 21-26 June 2014, pages 1188{1196.
[Lee et al., 2013] Lee,A.J.T.,Yang,F.-C.,Tsai,H.-C.,andLai,Y.-Y.(2013). Discoveringcontent-based
behavioral roles in social networks. Decis. Support Syst., 59:250{261.
[Lee and Dernoncourt, 2016] Lee, J. Y. and Dernoncourt, F. (2016). Sequential short-text classication
with recurrent and convolutional neural networks. In NAACL HLT 2016, The 2016 Conference of the
North American Chapter of the Association for Computational Linguistics: Human Language Technolo-
gies, San Diego California, USA, June 12-17, 2016, pages 515{520.
[Leuski, 2004] Leuski, A. (2004). Email is a stage: Discovering people roles from email archives. In
Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development
in Information Retrieval, SIGIR '04, pages 502{503, New York, NY, USA. ACM.
[Li and Gillet, 2013] Li, N. and Gillet, D. (2013). Identifying in
uential scholars in academic social media
platforms. In Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social
Networks Analysis and Mining, ASONAM '13, pages 608{614, New York, NY, USA. ACM.
129
[Li et al., 2015] Li, Y., Ma, S., and Huang, R. (2015). Social Context Analysis for Topic-Specic Expert
Finding in Online Learning Communities, pages 57{74. Springer Berlin Heidelberg, Berlin, Heidelberg.
[Liu et al., 2014] Liu, X., Wang, G. A., Johri, A., Zhou, M., and Fan, W. (2014). Harnessing global
expertise: A comparative study of expertise proling methods for online communities. Information
Systems Frontiers, 16(4):715{727.
[Lu et al., 2011] Lu, L., Zhang, Y.-C., Yeung, C. H., and Zhou, T. (2011). Leaders in social networks, the
delicious case.
[Maia et al., 2008] Maia, M., Almeida, J., and Almeida, V. (2008). Identifying user behavior in online
social networks. In Proceedings of the 1st Workshop on Social Network Systems, SocialNets '08, pages
1{6, New York, NY, USA. ACM.
[Mann and Thompson, 1988] Mann, W. C. and Thompson, S. A. (1988). Rhetorical Structure Theory:
Toward a functional theory of text organization. Text, 3(8):243{281.
[Marcu, 1997] Marcu, D. (1997). The rhetorical parsing of natural language texts. In Proceedings of the
Eighth Conference on European Chapter of the Association for Computational Linguistics, EACL '97,
pages 96{103, Stroudsburg, PA, USA. Association for Computational Linguistics.
[Markel, 1998] Markel, N. (1998). Semiotic psychology: Speech as an index of emotions and attitudes.
[Marneffe et al., 2006] Marneffe, M., Maccartney, B., and Manning, C. (2006). Generating typed depen-
dency parses from phrase structure parses. In Proceedings of the Fifth International Conference on
Language Resources and Evaluation (LREC-2006), Genoa, Italy. European Language Resources Asso-
ciation (ELRA).
[Masolo et al., 2004] Masolo, C., Vieu, L., Bottazzi, E., Catenacci, C., Ferrario, R., Gangemi, A., and
Guarino, N. (2004). Social roles and their descriptions. In Proceedings of the Ninth International
Conference on Principles of Knowledge Representation and Reasoning, KR'04, pages 267{277. AAAI
Press.
[Mathioudakis and Koudas, 2009] Mathioudakis, M. and Koudas, N. (2009). Efficient identication of
startersandfollowersinsocialmedia. In Proceedings of the 12th International Conference on Extending
Database Technology: Advances in Database Technology, EDBT '09, pages 708{719, New York, NY,
USA. ACM.
[Merton, 1968] Merton, R. K. (1968). Social theory and social structure (enlarged ed.).
[Mikolov et al., 2013] Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of
word representations in vector space. CoRR, abs/1301.3781.
[Miller et al., 1999] Miller, G. A., Beckwith, R., Fellbaum, C., Gross, D., and Miller, K. (1999). Wordnet:
An electronic lexical database. The Library Quarterly, 69(3):406{408.
[Miltsakaki et al., 2004] Miltsakaki, E., Prasad, R., Joshi, A., andWebber, B.(2004). Thepenndiscourse
treebank. InProceedings of the Fourth International Conference on Language Resources and Evaluation
(LREC'04), Lisbon, Portugal.
[Mohamed and Awadallah, 2011] Mohamed, A. and Awadallah, H. (2011). Identifying roles in social
networks using linguistic analysis.
[Palau and Moens, 2009] Palau, R. M. and Moens, M.-F. (2009). Argumentation mining: The detection,
classication and structure of arguments in text. In Proceedings of the 12th International Conference
on Articial Intelligence and Law, ICAIL '09, pages 98{107, New York, NY, USA. ACM.
130
[Perelman, 1970] Perelman, C. (1970). The New Rhetoric: A Theory of Practical Reasoning, pages 1{42.
Springer Netherlands, Dordrecht.
[Pfeil et al., 2011] Pfeil, U., Svangstu, K., Ang, C. S., and Zaphiris, P. (2011). Social roles in an online
support community for older people. International Journal of HumanComputer Interaction, 27(4):323{
347.
[Popescu and Etzioni, 2005] Popescu, A.-M. and Etzioni, O. (2005). Extracting product features and
opinionsfromreviews. In Proceedings of the Conference on Human Language Technology and Empirical
Methods in Natural Language Processing, HLT '05, pages 339{346, Stroudsburg, PA, USA. Association
for Computational Linguistics.
[Revelle et al., 2016] Revelle, M., Domeniconi, C., and Johri, A. (2016). Persistent Roles in Online Social
Networks, pages 47{62. Springer International Publishing, Cham.
[Rosenthal, 2014] Rosenthal, S. (2014). Detecting in
uencers in social media discussions. ACM Cross-
roads, 21(1):40{45.
[Rosenthal and McKeown, 2012] Rosenthal, S. and McKeown, K. (2012). Detecting opinionated claims
in online discussions. In Proceedings of the 2012 IEEE Sixth International Conference on Semantic
Computing, ICSC '12, pages 30{37, Washington, DC, USA. IEEE Computer Society.
[Ross Winterowd, 1975] Ross Winterowd, W. (1975). Contemporary Rhetoric. Harcourt Brace Jo-
vanovich.
[Rossi et al., 2012] Rossi, R., Gallagher, B., Neville, J., and Henderson, K. (2012). Role-dynamics: Fast
mining of large dynamic networks. In Proceedings of the 21st International Conference on World Wide
Web, WWW '12 Companion, pages 997{1006, New York, NY, USA. ACM.
[Rousseeuw, 1987] Rousseeuw,P.(1987). Silhouettes: Agraphicalaidtotheinterpretationandvalidation
of cluster analysis. J. Comput. Appl. Math., 20(1):53{65.
[Sapru and Valente, 2012] Sapru, A. and Valente, F. (2012). Automatic speaker role labeling in AMI
meetings: Recognition of formal and social roles. In 2012 IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), pages 5057{5060. IEEE.
[Shaikh et al., 2012] Shaikh, S., Strzalkowski, T., Stromer-Galley, J., Broadwell, G. A., Taylor, S., Liu,
T., Ravishankar, V., Ren, X., and Boz, U. (2012). Modeling in
uence in online multi-party discourse.
In 2012 Second International Conference on Cloud and Green Computing, pages 515{522.
[Smith Risser and Bottoms, 2014] Smith Risser, H. and Bottoms, S. (2014). \newbies" and \celebrities":
Detecting social roles in an online network of teachers via participation patterns. International Journal
of Computer-Supported Collaborative Learning, 9(4):433{450.
[Song et al., 2012a] Song, K., Wang, D., Feng, S., Wang, D., and Yu, G. (2012a). Detecting positive
opinion leader group from forum. In WAIM.
[Song et al., 2012b] Song, K., Wang, D., Feng, S., and Yu, G. (2012b). Detecting Opinion Leader Dynam-
ically in Chinese News Comments, pages 197{209. Springer Berlin Heidelberg, Berlin, Heidelberg.
[Song et al., 2007] Song, X., Chi, Y., Hino, K., and Tseng, B. (2007). Identifying opinion leaders in
the blogosphere. In Proceedings of the Sixteenth ACM Conference on Conference on Information and
Knowledge Management, CIKM '07, pages 971{974, New York, NY, USA. ACM.
[Spertus, 1997] Spertus, E. (1997). Smokey: Automatic recognition of hostile messages. In Proceedings
of the Fourteenth National Conference on Articial Intelligence and Ninth Conference on Innovative
Applications of Articial Intelligence, AAAI'97/IAAI'97, pages 1058{1065. AAAI Press.
131
[Stadtfeld, 2012] Stadtfeld, C. (2012). Discovering latent behavioral roles in dynamic social networks. In
Proceedingsofthe2012ASE/IEEEInternationalConferenceonSocialComputingand2012ASE/IEEE
International Conference on Privacy, Security, Risk and Trust, SOCIALCOM-PASSAT '12, pages 629{
635, Washington, DC, USA. IEEE Computer Society.
[Sun and Ng, 2012] Sun, B. and Ng, V. T. (2012). Identifying in
uential users by their postings in social
networks. In Proceedings of the 3rd International Workshop on Modeling Social Media, MSM '12, pages
1{8, New York, NY, USA. ACM.
[Sun et al., 2016] Sun, J., Kunegis, J., and Staab, S. (2016). Predicting user roles in social networks using
transfer learning with feature transformation.
[Tang and Yang, 2012] Tang, X. and Yang, C. C. (2012). Ranking user in
uence in healthcare social
media. ACM Trans. Intell. Syst. Technol., 3(4):73:1{73:21.
[Tinati et al., 2012] Tinati, R., Carr, L., Hall, W., and Bentwood, J. (2012). Identifying communicator
roles in twitter. In Proceedings of the 21st International Conference on World Wide Web, WWW '12
Companion, pages 1161{1168, New York, NY, USA. ACM.
[Toulmin, 2003] Toulmin, S. (2003). The Uses of Argument. Cambridge University Press.
[Toulmin et al., 1979] Toulmin, S. E., Rieke, R., and Janik, A. (1979). An Introduction to Reasoning.
Macmillan.
[Towhidi and Sinha, 2015] Towhidi, G. and Sinha, A. P. (2015). Predicting opinion leaders in word-of-
mouth communities. In AMCIS.
[Tsai et al., 2012] Tsai, M.-F., Tzeng, C.-W., and Chen, A. L. P. (2012). Discovering leaders from social
network by action cascade. In Proceedings of the Fifth Workshop on Social Network Systems, SNS '12,
pages 12:1{12:2, New York, NY, USA. ACM.
[Tye, 2015] Tye, E. (2015). Forecasting user roles in online communities.
[Vega et al., 2016] Vega, D., Magnani, M., Montesi, D., Meseguer, R., and Freitag, F. (2016). A new
approach to role and position detection in networks. Social Network Analysis and Mining, 6(1):39.
[Viol et al., 2016] Viol,J.,Bernsmann,R.,andRiemer,K.(2016). Behaviouraldimensionsfordiscovering
knowledge actor roles utilising enterprise social network metrics. CoRR, abs/1606.00883.
[Volpentesta and Felicetti, 2012] Volpentesta,A.P.andFelicetti,A.M.(2012). Identifying Opinion Lead-
ersinTime-DependentCommercialSocialNetworks,pages571{581. SpringerBerlinHeidelberg,Berlin,
Heidelberg.
[Wang et al., 2010] Wang, M.-F., Kuo, Y.-L., Tsai, M.-F., Tang, C.-H., and Huang, K. (2010). Hier-
archical role classication based on social behavior analysis. In Proceedings of the 8th International
Conference on Advances in Mobile Computing and Multimedia, MoMM '10, pages 425{429, New York,
NY, USA. ACM.
[Welser et al., 2011] Welser, H. T., Cosley, D., Kossinets, G., Lin, A., Dokshin, F., Gay, G., and Smith,
M. (2011). Finding social roles in wikipedia. In Proceedings of the 2011 iConference, iConference '11,
pages 122{129, New York, NY, USA. ACM.
[Welser et al., 2007] Welser, H. T., Gleave, E., Fisher, D., and Smith, M. (2007). Visualizing the Signa-
tures of Social Roles in Online Discussion Groups. The Journal of Social Structure, 8.
132
[Wever et al., 2010] Wever, B. D., Keer, H. V., Schellens, T., and Valcke, M. (2010). Roles as a struc-
turing tool in online discussion groups: The differential impact of different roles on social knowledge
construction. Computers in Human Behavior, 26(4):516 { 523.
[White et al., 2012] White, A. J., Chan, J., Hayes, C., and Murphy, T. B. (2012). Mixed membership
models for exploring user roles in online fora. In ICWSM.
[Worsley and Blikstein, 2011] Worsley, M. and Blikstein, P. (2011). What's an expert? using learning
analytics to identify emergent markers of expertise through automated speech, sentiment and sketch
analysis. In EDM.
[Yeh, 2010] Yeh, Y.-C. (2010). Analyzing online behaviors, roles, and learning communities via online
discussions. Educational Technology and Society, 13(1):140{151.
[Yu et al., 2016] Yu, L., Kittur, A., and Kraut, R. E. (2016). Encouraging \outside- the- box" thinking in
crowd innovation through identifying domains of expertise. In Proceedings of the 19th ACM Conference
on Computer-Supported Cooperative Work & Social Computing, CSCW '16, pages 1214{1222, New
York, NY, USA. ACM.
[Zhang et al., 2007] Zhang, J., Ackerman, M. S., and Adamic, L. (2007). Expertise networks in online
communities: Structure and algorithms. In Proceedings of the 16th International Conference on World
Wide Web, WWW '07, pages 221{230, New York, NY, USA. ACM.
[Zhang et al., 2016] Zhang, R., Lee, H., and Radev, D. R. (2016). Dependency sensitive convolutional
neural networks for modeling sentences and documents. In NAACL HLT 2016, The 2016 Conference
of the North American Chapter of the Association for Computational Linguistics: Human Language
Technologies, San Diego California, USA, June 12-17, 2016, pages 1512{1521.
[Zhang et al., 2013] Zhang, W., He, H., and Cao, B. (2013). Identifying and evaluating the internet
opinion leader community based on k-clique clustering. Neural Computing and Applications, 25:595{
602.
[Zhao et al., 2013] Zhao, Y., Wang, G., Yu, P. S., Liu, S., and Zhang, S. (2013). Inferring social roles
and statuses in social networks. In Proceedings of the 19th ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining, KDD '13, pages 695{703, New York, NY, USA. ACM.
[Zohrabi Aliabadi et al., 2013] Zohrabi Aliabadi, A., Razzaghi, F., Madani Kochak, S. P., and Ghorbani,
A. A. (2013). Classifying Organizational Roles Using Email Social Networks, pages 301{307. Springer
Berlin Heidelberg, Berlin, Heidelberg.
[Zurcher, 1983] Zurcher, L. A. (1983). Social roles: conformity, con
ict, and creativity.
133

Abstract (if available)

Abstract The main goal of this dissertation is to identify social roles of participants in online contentious discussions, define these roles in terms of behavioral characteristics participants show in such discussions, and develop methods to identify these participant roles automatically. As social life becomes increasingly embedded in online systems, the concept of social role becomes increasingly valuable as a tool for simplifying patterns of action, recognizing distinct participant types, and cultivating and managing communities. In contentious discussions, the roles may exert a major influence on the course and/or outcome of the discussions. The existing work on social roles mostly focuses on either empirical studies or network based analysis. Whereas this dissertation presents a model of social roles by analyzing the content of the participants’ contribution. ❧ In the first portion of this dissertation, I present the corpus of participant roles in online discussions from Wikipedia: Articles for Deletion and 4forums.com discussion forums. A rich set of annotations of behavioral characteristics such as stubbornness, sensibleness, influence, and ignored-ness, which I believe all contribute in the identification of roles played by participants, is created to analyze the contribution of the participants. Using these behavioral characteristics, Participant roles such as leader, follower, rebel, voice in wilderness, idiot etc. are defined which reflect these behavioral characteristics. In the second part of this dissertation I present the methods used to identify these participant roles in online discussions automatically using the contribution of the participants in the discussion. First, I develop two models to identify leaders in online discussions that quantify the basic leadership qualities of participants. Then, I present the system for analyzing the argumentation structure of comments in discussions. This analysis is divided in three parts: claim detection, claim delimitation, and claim-link detection. Then, the dissertation presents the social roles model to identify the participant roles in discussions. I create classification models and neural network structures for each behavioral characteristic using a set of features based on participants’ contribution to the discussion to determine the behavior values for participants. Using these behavioral characteristic values the roles of participants are determined based on the rules determined from the annotation scheme. I show that for both, the classification models and neural networks, the rule based methods perform better than the model that identifies the participant roles directly. This signifies that the framework of breaking down the problem of identifying social roles to determining values of specific behavioral characteristics make it more intuitive in terms of what we expect from participants who assume these roles. Although the neural network methods perform worse than their traditional classification method counterparts, when provided with additional training data, neural network structures improve at a much higher rate. ❧ In the last part of the dissertation, I use the social roles model as a tool to analyze participants’ behavior in large corpus. Social roles are automatically tagged in Wikipedia corpus containing ∼26000 discussions. This allows determining participants’ roles over time in order to identify whether they assume different roles in different discussions, and what factors may affect an individual’s role in such discussions. I investigate three factors: topic of the discussion, amount of contention in the discussion, and other participants in the discussion. The results show that participants behave similarly in most situations. However, the social roles model is able to identify instances where participants’ behavior patterns are different than their own typical behavior. In doing so, the model provides useful context regarding the reason behind these behavioral patterns by identifying specific behaviors affected by the situation.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Behavior understanding from speech under constrained conditions: exploring sparse networks, transfer and unsupervised learning

PDF

Computational narrative models of character representations to estimate audience perception

PDF

Incrementality for visual reference resolution in spoken dialogue systems

PDF

Modeling social and cognitive aspects of user behavior in social media

PDF

Learning distributed representations from network data and human navigation

PDF

Neural creative language generation

PDF

Automatic evaluation of open-domain dialogue systems

PDF

Human behavior understanding from language through unsupervised modeling

PDF

Socially-informed content analysis of online human behavior

PDF

Building and validating computational models of emotional expressivity in a natural social task

PDF

Predicting and modeling human behavioral changes using digital traces

PDF

Graph embedding algorithms for attributed and temporal graphs

PDF

Spoken language processing in low resource scenarios with applications in automated behavioral coding

PDF

Modeling dyadic synchrony with heterogeneous data: validation in infant-mother and infant-robot interactions

PDF

Towards social virtual listeners: computational models of human nonverbal behaviors

PDF

Designing neural networks from the perspective of spatial reasoning

PDF

Bound in hatred: a multi-methodological investigation of morally motivated acts of hate

PDF

Automatic quantification and prediction of human subjective judgments in behavioral signal processing

PDF

Generating psycholinguistic norms and applications

PDF

Not a Lone Ranger: unleashing defender teamwork in security games

Asset Metadata

Creator Jain, Siddharth (author)

Core Title Identifying Social Roles in Online Contentious Discussions

School Viterbi School of Engineering

Degree Doctor of Philosophy

Degree Program Computer Science

Publication Date 07/26/2018

Defense Date 04/23/2018

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag behavioral characteristics,classification,corpus,natural language processing,neural networks,,OAI-PMH Harvest,online discussions,Social roles

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Artstein, Ron (committee chair), Dehghani, Morteza (committee member), Georgila, Kallirroi (committee member), Kaiser, Elsi (committee member), Rosenbloom, Paul (committee member)

Creator Email sid.jain250888@gmail.com,siddhajj@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c89-23778

Unique identifier UC11668945

Identifier etd-JainSiddha-6495.pdf (filename),usctheses-c89-23778 (legacy record id)

Legacy Identifier etd-JainSiddha-6495.pdf

Dmrecord 23778

Document Type Dissertation

Format application/pdf (imt)

Rights Jain, Siddharth

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA