SCAN: Socio-Cultural Attitudinal Networks
Summer 2020 Webinar Series
RegisterSpeakers
| V.S. Subrahmanian is the Dartmouth College Distinguished Professor in Cybersecurity, Technology, and Society and Director of the Institute for Security, Technology, and Society at Dartmouth. He previously served as a Professor of Computer Science at the University of Maryland from 1989-2017 where he created and headed both the Lab for Computational Cultural Dynamics and the Center for Digital International Government. He also served for 6+ years as Director of the University of Maryland's Institute for Advanced Computer Studies. Prof. Subrahmanian is an expert on big data analytics including methods to analyze text/geospatial/relational/social network data, learn behavioral models from the data, forecast actions, and influence behaviors with applications to cybersecurity and counter-terrorism. He has written five books, edited ten, and published over 300 refereed articles. He is a Fellow of the American Association for the Advancement of Science and the Association for the Advancement of Artificial Intelligence and received numerous other honors and awards. His work has been featured in numerous outlets such as the Baltimore Sun, the Economist, Science, Nature, the Washington Post, American Public Media. He serves on the editorial boards of numerous journals including Science, the Board of Directors of the Development Gateway Foundation (set up by the World Bank), SentiMetrix, Inc., and on the Research Advisory Board of Tata Consultancy Services. He previously served on DARPA's Executive Advisory Council on Advanced Logistics and as an ad-hoc member of the US Air Force Science Advisory Board. |
| Judee K. Burgoon is Director of Research for the Center for the Management of Information at the University of Arizona, where she holds titles as professor of communication, family studies and human development. She has authored or edited 17 books and monographs (Detecting Trust and Deception in Group Interaction, Social Signal Processing, Discovering Hidden Temporal Patterns in Behavior and Interaction, Nonverbal Communication, Interpersonal Adaptation, Small Group Communication, Mexican Americans and the Mass Media) and over 300 articles, chapters and reviews. Her current research on deception, dyadic interaction, and technologies for automated analysis of nonverbal and verbal communication has been funded by the National Science Foundation, Department of Defense Air Force, Army, Navy and ODNI, Department of Homeland Security, and Gannett Foundation, among others. She is the recipient of the highest honors given by the International Communication Association and National Communication Association and has held several offices in NCA. |
| Norah E. Dunbar is a Professor of Communication at University of California Santa Barbara. She teaches courses in nonverbal and interpersonal communication, communication theory, and deception detection. She is also Affiliate Faculty in the Center for Information, Technology & Society; the Center for Digital Games Research; and the Quantitative Methods in the Social Sciences program. She has received over $13 Million in research funding from agencies such as the Intelligence Advanced Research Projects Activity, the National Science Foundation, the Department of Defense, and the Center for Identification Technology Research. She has published over 65 peer-reviewed journal articles and book chapters and has presented over 100 papers at National and International conferences. Her research has appeared in top journals in her discipline including Communication Research, Communication Monographs, and Journal of Computer-Mediated Communication as well as interdisciplinary journals such as Journal of Management Information Systems and Computers in Human Behavior. She has served on the editorial board of over a dozen disciplinary journals and as the Chair of the Nonverbal Division of the National Communication Association in 2014-2016. She is the current Chair of the Communication Department at UCSB. |
| Dr. Jure Leskovec is an associate professor of Computer Science at Stanford University where he is a member of the InfoLab
and the AI lab. He joined the department in September 2009. He is also Chief Scientists at Pinterest and an investigator at Chan Zuckerberg Biohub,
where he focuses on developing new methods for analysis of biomedical data. His general research area is applied machine learning and data science for large interconnected systems. Focuses on modeling complex, richly-labeled relational structures, graphs, and networks for systems at all scales, from interactions of proteins in a cell to interactions between humans in a society. Applications include commonsense reasoning, recommender systems, computational social science, and computational biology with an emphasis on drug discovery. This research has won several awards including a Lagrange Prize, Microsoft Research Faculty Fellowship, the Alfred P. Sloan Fellowship, and numerous best paper and test of time awards. It has also been featured in popular press outlets such as the New York Times and the Wall Street Journal. He received my bachelor's degree in computer science from University of Ljubljana, Slovenia, PhD in machine learning from Carnegie Mellon University and postdoctoral training at Cornell University. He also founded ASEF: American Slovenian Education Foundation which unites over 50 Slovenian professors all over the world and provides fellowships for students to further their education and research. |
| Dr. Pan Li is a postdoctoral researcher in the SNAP group lead by Prof. Jure Leskovec at Stanford University. He will join the
Department of Computer Science at Purdue University as an assistant professor in Fall, 2020. Previously, he earned his master’s degree from Tsinghua
University in 2015 and his PhD from the University of Illinois at Urbana-Champaign in 2019. He holds broad research interests on topics on graph computation, graph-based machine learning, network analysis etc. He has established some fundamental theory for spectral methods over higher-order graphs. He is currently working on fundamental understanding and optimization of graph neural networks, scalable algorithms for complex graph/network computation. |
| Chongyang Bai is a final-year Ph.D. candidate at Dartmouth College advised by Prof. V.S. Subrahmanian. He obtained a B.S. in Computational Mathematics and B.Eng in Computer Science from University of Science and Technology of China in 2016. He spent one year as a research intern at Microsoft Research Asia and will spend this summer internship at Google Research. His research interests are multi-modal representation learning and prediction in videos. Specifically, he has studied the problems of predicting human social behaviors (e.g. persuasion, dominance, nervousness and deception) with the focus on group interaction network effects and individual audio-visual attributes. He has published several papers in top-tiered conferences and journals and served as reviewers in related AI conferences (e.g. ICME, ICMI, ICWSM) and journals (e.g. IEEE Intelligent Systems). |
| Dr. Dimitris Metaxas is a Distinguished Professor in the Department of Computer Science at Rutgers University,
and director of the Center for Computational Biomedicine, Imaging and Modeling and the NSF IUCRC CARTA Center. Dr. Metaxas has been conducting research towards the development of formal Data Science methods upon which AI, machine learning, computer vision, medical image analysis and computer graphics can advance synergistically. In AI, machine learning and computer vision, new methods have been developed for understandable machine learning, real time data analytics, dynamic data driven application systems, 3D human motion analysis, human behaviors and intent recognition, scene understanding and segmentation, surveillance, object recognition, sparsity and biometrics in the wild. In medical and biological image analysis new data science methods have been developed for material modeling and shape estimation of internal body parts (e.g., lungs) from MRI, SPAMM and CT data, a pioneering framework for cardiac motion analysis and for linking the anatomical and physiological models of the human body, cancer diagnosis, histopathology, cell tracking, cell type analysis and mouse behavior analysis. In computer graphics, the Navier-Stokes methodology for Fluid animations in the mid 90s was introduced, based on which the water scenes in the movie “Antz” were created in 1998. Since then, new dynamic data science techniques for modeling fluid phenomena, and control theoretic techniques for automating and improving the animation of articulated (e.g., humans) objects. Dr. Metaxas has published over 700 research articles in these areas and has graduated 56 PhD students. Dr. Metaxas has received 8 patents and numerous best paper awards for his work on vision, medical imaging and fluid modeling. The above research has been funded by NSF, NIH, ONR, AFOSR, DHS, DARPA and the ARO. Dr Metaxas was awarded a Fulbright Fellowship in 1986, is a recipient of an NSF Research Initiation and Career awards, an ONR YIP, is a Fellow of IEEE, a Fellow of the MICCAI Society and a Fellow of the American Institute of Medical and Biological Engineers. |
| Jay F. Nunamaker, Jr. is Regents and Soldwedel Professor of MIS, Computer Science and Communication. He was Director of the Center for the Management of Information and the Director of the National Center for Border Security and Immigration from 2008-2015 at the University of Arizona, funded by the Department of Homeland Security (DHS) Center of Excellence program. Dr. Nunamaker was inducted into the Design Science Hall of Fame, May 2008. Dr. Nunamaker received the LEO Award for Lifetime Achievement from the Association of Information Systems (AIS) at ICIS in Barcelona, Spain, December 2002. He was elected a fellow of the AIS in 2000. He was featured in the July 1997 Forbes Magazine issue on technology as one of eight key innovators in information technology. He is widely published with an H index of greater than 84. He has produced over 400 journal articles, book chapters, books and refereed proceedings and has been a major professor for 102 Ph.D. students. His specialization is in the fields of system analysis and design, collaboration technology and deception detection. He has co-founded five spin-off companies based on his research: (1) Combinatorics which was acquired by Mathematica, 1972, (2) PLEX Corporation acquired by AT&T, 1981 (3) ULTA Systems, 1986, (4) GroupSystems Inc., 1989 and (5) Discern Science, Inc., 2011. The commercial product, GroupSystems, ThinkTank based upon Nunamaker’s research, is often referred to as the gold standard for structured collaboration systems. He was a research assistant funded by the ISDOS project in industrial engineering at the University of Michigan and an associate professor of computer science and industrial administration at Purdue University. In his career he has received 100+ million dollars as the PI or Co-PI on sponsored research at the University of Arizona, Purdue University and the University of Michigan. He founded the MIS department at the University of Arizona in 1974 and served as department head for 18 years. From 1976-1991, Nunamaker served as chairman of the ACM Curriculum Committee on Information Systems and as a committee member from 2009-2014. Dr. Nunamaker received his Ph.D. in operations research and systems engineering from Case Institute of Technology, an M.S. and B.S. in engineering from the University of Pittsburgh, and a B.S. from Carnegie Mellon University. He received his professional engineer’s license in 1965. |
| Miriam J. Metzger (Ph.D. University of Southern California) is a Professor in the Department of Communication at the University of California at Santa Barbara. Her research lies at the intersection of digital technology and trust, centering on how information and communication technologies alter our understandings of credibility and force us to confront new challenges in protecting our privacy. She has also published work examining the theoretical and regulatory changes brought about by emerging information and communication technologies. Her work has been published in the fields of communication, psychology, information science, computer science. Outside her home department, Dr. Metzger serves as Director of Education for the Center for Information, Technology & Society (CITS) at UCSB. |
Talk abstracts
| Introduction to the SCAN Project and Deception Detection from Online Videos V.S. Subrahmanian, Dartmouth College The talk will start with a brief introduction to the Socio-Cultural Attitudinal Networks (SCAN) project – a joint effort by Dartmouth College, Rutgers University, Stanford University, University of Arizona, University of Maryland, and University of California Santa Barbara. The goal of the SCAN project is to understand the link between non-verbal expressions (facial expressions, speech patterns, language) and relationships amongst people in a group setting. A team of social scientists and computer scientists jointly explore the non-verbal factors related to: who likes/dislikes who? Who is deferential to or dominates who? Who trusts who? Who is being deceptive. This portion of the talk will describe the background of the SCAN project and set the stage for the SCAN Summer 2020 Webinar Series. The deception detection part of this talk will consist of two parts. First, we will describe a system called DARE (Deception Analytics Reasoning Engine) for unobtrusive automated deception detection using information available in a video. We study the importance of different modalities like vision, audio and text for this task. On the vision side, our system uses classifiers trained on low level video features which predict human micro-expressions. We show that predictions of high-level micro-expressions can be used as features for deception prediction. We use a combination of IDT (Improved Dense Trajectory) features and fuse the scores of classifiers trained on IDT features and high-level micro-expressions, MFCC (Mel-frequency Cepstral Coefficients) and text transcripts to obtain an AUC of 0.877 (10-fold cross-validation) when evaluated on subjects who were not part of the training set. Subsequently, we extend some of these ideas to predict deception in group videos. We introduce a novel class of features called LiarRank that build upon ideas of PageRank. In particular, we show that using facial micro-expressions, MFCCs, emotions, together with the LiarRank features, we can obtain an AUC of over 0.7 to predict deceivers in a fully automated manner. |
| A Novel Approach to Investigating Deception during Group Interaction Judee Burgoon, University of Arizona This webinar presents the role of relational communication in group interaction and in predicting deception. Relational communication refers to the implicit ways in which people define their interpersonal relationships with one another. Usually through nonverbal signals, they express who is dominant and who is submissive, who is liked and who is disliked, who is relaxed or nervous and tense, and who is trusted or distrusted. These implicit messages which are gathered from group members in turn can predict who is being deceptive and may serve as surrogates for directly measuring deception. I present data from our SCAN project showing how powerful these perceptions are in distinguishing truth tellers from liars in group interactions. |
| Persuasive Deception and Dyadic Power Theory Norah E. Dunbar, Department of Communication, University of California Santa Barbara Power is a critical factor in how people relate to one another in group dynamics and thus may be influential during attempts at deception. Deception in persuasive contexts differs from deception in other contexts, where concealment or equivocation may be the preferred tactic. Deceivers who are attempting to persuade others face the intertwined tasks of appearing credible and making convincing arguments in favor of their position while simultaneously attempting to avoid detection. We label this type of lying persuasive deception. In this presentation, Dunbar will discuss how we expanded her Dyadic Power Theory to examine the dominance behavior of liars and truth-tellers in cross-cultural groups. We examined 95 groups playing a popular party card game called Mafia in which some players were randomly assigned the role of “Spies,” and others, the role of “Villagers.” Spies concealed their identity and deceived the other naïve players (the Villagers), who were assumed to be truthful. Data were collected from University students around the world, including in Israel, Singapore, Hong Kong, Fiji, Zambia, and three locations within the U.S. The results revealed differences in the way that players from those different cultural locations exhibited dominance and rated the trustworthiness of other players. In general, more dominant players were seen as more trustworthy, although that was moderated by the cultural location. Spies (deceivers) were viewed as less dominant than Villagers (truth-tellers). Males were rated as more dominant, especially when in the Spy role. Implications for the theory and other applications of dyadic power theory will be discussed. |
| Dynamic Embeddings of Temporal Interaction Networks Jure Leskovec, Stanford University Modeling sequential interactions between a group of people is crucial in domains such as e-commerce, social networking, and education. Representation learning presents an attractive opportunity to model the dynamic evolution of each individuals, where each of them can be embedded in a Euclidean space and its evolution can be modeled by an embedding trajectory in this space. However, existing dynamic embedding methods generate embeddings only when someone takes actions and do not explicitly model the future trajectory of him/her in the embedding space. In this talk, we introduce a coupled recurrent neural network model that learns the embedding trajectories of individuals based on their interactive behaviors. Our approach employs recurrent neural networks to update the embedding of each individual at every interaction. Crucially, the approach also models the future embedding trajectory of each individual. To this end, it introduces a novel projection operator that learns to estimate the embeddings at any time in the future. These estimated embeddings are then used to predict future interactions. To make the method scalable, we develop a t-Batch algorithm that creates time-consistent batches and leads to 9× faster training. We conduct six experiments to validate our method on two prediction tasks— future interaction prediction and state change prediction—using four real-world datasets. We show that the method outperforms six state- of-the-art algorithms in these tasks by at least 20% in predicting future interactions and 12% in state change prediction. |
| An Interpretable Representation Learning Framework for Dynamic Social Interaction Networks Pan Li, Stanford University In this talk, we will discuss our recent progress on make inference over dynamic social interaction networks, including our MURI SCAN dataset. We propose a neural-network-based model, temporal network-diffusion convolution networks, that learns node representations of this dynamic social interaction network to build informative profile for each individual. These representations naturally fit into the prediction tasks including deception detection, dominance identification and nervousness detection. The first part of our model is network diffusion of node attributes that naturally captures the interweaving between highly dynamic node attributes and interactions. Graph diffusion procedure allows us to consider long paths and also improves the model interpretation which is often important for social scientists to understand social patterns. The second part of our model is a multi-layer temporal convolution network (TCN) accompanied with set pooing to aggregate representations of nodes over a long time span. Due to the locality of temporal convolution kernels, TCN is able to extract complex patterns from interactions with various durations as these interactions may appear alternatively across multiple consecutive snapshots, and set-pooling is useful to collect scattered subtle patterns over a long-time span. Moreover, our model is end-to-end trainable, and therefore shows a new opportunity of neural networks for computational social scientists to automatically process dynamic social interactions and obtain social insights simultaneously. We evaluate this model for deception detection, dominance identification, nervousness detection over the MURI SCAN dataset. From the perspective of making inferences, our model significantly outperforms all previous benchmarks that are either based on neural networks for generic dynamic networks or based on feature engineering designed for certain tasks. From the perspective of model interpretation, our model allows for in-depth analysis of its learnt coefficients and yield insights on the patterns of social interactions. Results show that involvement in direct interactions (looking, speaking and listening) between individuals is more informative for dominance identification and nervousness detection while avoiding direct interactions between individuals provides strong signals for deception detection. This observation coincides with previous findings in psychology via extensive statistical analysis, while our model is able to detect it automatically. Our model also shows that difference between the quantified attributes of one individual and those of his/her interacted neighbors is a stronger signal to indicate one’s dominance and nervousness than one’s own attributes, which again demonstrates the significance to understand people’s social behavior via dynamic social interaction networks. |
| Dominance Detection in Group Interaction Videos Chongyang Bai, Dartmouth College This talk will start from a novel method for building face-to-face interaction networks in videos, then focus on predicting dominance from the networks and videos. To build the interaction networks, we propose ICAF (Iterative Collective Attention Focus), a collective classification model to jointly learn the visual focus of attention of all people (i.e. who look at whom). ICAF models the people collectively—the predictions of all other people’s classifiers are used as inputs to each person’s classifier. This explicitly incorporates interdependencies between all people’s behaviors. ICAF outperforms the strongest baseline by 1%–5% accuracy in two datasets. For dominance prediction, we consider the problems of predicting (i) the most dominant person in a group of people, and (ii) the more dominant of a pair of people. We introduce a novel family of features called Dominance Rank from the interaction networks. We combine features not previously used for dominance prediction (e.g., facial action units, emotions), with a novel ensemble-based approach to solve these two problems. We test our models against four competing algorithms in the literature on two datasets. Our models achieve 2.4%--16.7% improvement in AUC on one dataset, and 0.6%--8.8% in accuracy on the other. |
| Detection and Tracking of Humans and Faces using Machine Learning Dimitris Metaxas, Rutgers University The human face has evolved to express many types of important information in human communication and interaction such as emotions, deception and intent. The face has several global degrees of freedom such as translation and rotations, as well as local ones related to expressions. To estimate those degrees of freedom there are two approaches. The first are traditional model-based machine learning methods and more recently deep machine learning methods. In this talk we will present both and discuss the their advantages and disadvatages in 2D and 3D tracking of faces from video. |
| Video-based Deception Detection and Corresponding Feature Discovery Dimitris Metaxas, Rutgers University The development of robust automated deception detection (ADD) systems is a long sought-after goal. In this talk, we introduce current approaches to video-based deception detection and discuss their limitations. We will present a novel framework that detects deception in videos using facial signals (Action Units and Gaze Signals). By using a higher-level input and not the raw video, we are able to train a simple, modular and powerful model that achieves state-of-the-art performance in video-based deception detection. Finally, we propose a novel approach to interpret our model’s predictions, by computing the attention of the neural network in the time domain. This method can enable domain scientists perform retrospective analysis of deceptive behavior. The approach is general and can be applied to the recognition of other video-based events. |
| Going the Last Mile for SCAN Transition Jay F. Nunamaker, Jr, University of Arizona The SCAN program originated from a 2015 Department of Defense (DOD) solicitation to advance research in the social sciences. The DOD requested research to evolve the insights of underlying communication messages within small groups, specifically the use of signals in group activities that explain the links between actors, their intentions, and context. Following a successful proposal win, the University of Maryland (UM) and its research partners Dartmouth, University of Arizona, UC Santa Barbara, Rutgers, and Stanford engaged in a multi-year, multi-country experiment to collect data on group behavior. Details of this experiment and the method for the collection of behavioral data is presented in Dorn et al. (in press). From our desk chairs, it may be tempting to work up an idea, build a quick prototype, test it in a lab, and say, "Our work here is done; the rest is merely details." However, more scholarly knowledge awaits discovery by researchers who shepherd a solution through The Last Research Mile. That is, taking a solution through a successful transition to the workplace. Going the Last Research Mile means using scientific knowledge and methods to address important unsolved classes of problems for real people with real stakes in the outcomes. The Last Research Mile proceeds in three stages: Proof-of-Concept (POC) research to demonstrate the functional feasibility of a solution (Does it work?); Proof-of-Value (POV) research to investigate whether a solution can create value across a variety of conditions (How is value created?); and Proof-of-Use (POU) research to address complex issues of operational feasibility (Is it useful?). The Last Research Mile ends only when practitioners are routinely using a solution in the field. Systems researchers who take their solutions through the Last Research Mile may ultimately have the greatest impact on science and society. We demonstrate the Last Research Mile with cases from the MURI Scan project with examples of POC, POV, and POU. The road of the Last Research Mile in the MURI Scan project, takes us through three steps. In this project, we layout a systems design and an analysis model for creating behavioral features from videos of interpersonal interactions (group or dyadic), analyze those features, and produce an inferencing engine for the prediction of complex attribution for individuals within the videos. Proof-of-Concept demonstrates the feasibility of the approach. Machine Learning techniques are used to evaluate detailed face and head kinesic information for their power to predict the perceived judgments of dominance, trust, and nervousness. Next, a model is created to predict deception using these perceptual judgments. Proof-of-Value verifies the approach contains sufficient analysis and testing to create useful and valid predictions. The model from the POC stage is extended in a generalizable way to show the ability to predict small group leader elections and other game outcomes using perceptual judgments. Ground truthed data is employed to validate the approach. Proof-of-Use will deploy a practical application. The application may look very different from the algorithms and software employed to establish POC and POV. Experience, real problems, and real users in the field will drive adaptions to the system for predicting dominance, trustworthiness, nervousness, and deceptiveness of individuals and groups that may be encountered in the field. |
| International Data Collection using Human Subjects: Logistics and Challenges Miriam Metzger, University of California, Santa Barbara Collecting data in international locations presents special challenges, especially when human subjects are involved. There is a lot more to consider than when conducting research domestically. This training will discuss how to plan, find, select, and secure international partners and venues to conduct research, tips for navigating the necessary approval process(es) and logistics, and once "on the ground" what procedures to put in place to help ensure a smooth and successful data collection effort. The training will draw upon our team’s experience conducting research for the SCAN project in six countries across the globe. |
Register
All registered attendees will receive an email with a link for the Zoom meeting before each webinar in the series.| Participant name | |
| Participant email | |
| Participant affiliation | |