Mathematical Models Of Social Systems Pdf Semantic Scholar

Saif M. Mohammad (he/him)

Email: saif.mohammad@nrc-cnrc.gc.ca

Picture: Palau De La Música Catalana (Palace of Catalan Music). Architect: Antoni Gaudi.
A postcard from my first academic conference, ACL-2004.

Research Focus: Emotions and Language, Lexical Semantics, Computational Creativity, Natural Language Processing.

BIO

Dr. Saif M. Mohammad is a Senior Research Scientist at the National Research Council Canada (NRC). He received his Ph.D. in Computer Science from the University of Toronto. Before joining NRC, Saif was a Research Associate at the Institute of Advanced Computer Studies at the University of Maryland, College Park. His research interests are in Computational Linguistics and Natural Language Processing (NLP), especially Lexical Semantics, Emotions in Language, Sentiment Analysis, Computational Creativity, Fairness in NLP, Psycholinguistics, and Information Visualization. He has published over 100 scientific articles (journal articles, book chapters, and conference papers). He has served in various capacities at prominent journals and conferences, including: action editor for Computational Linguistics, chair of the Canada--UK symposium on Ethics in AI, co-chair of SemEval 2017-19 (the largest platform for semantic evaluations), workshops co-chair for ACL 2020, co-organizer of WASSA 2017 and 2018 (a sentiment analysis workshop), and area chair for ACL, NAACL, and EMNLP (in the areas of sentiment analysis, lexical semantics, and fairness in NLP). His team developed a sentiment analysis system which ranked first in shared task competitions. His word--emotion resources, such as the NRC Emotion Lexicon, are used for analyzing affect in text. His work has garnered media attention, including articles in Time, SlashDot, LiveScience, io9, The Physics arXiv Blog, PC World, and Popular Science.

RECENT NEWS AND PUBLICATIONS

Invited Talks

Emotions in Text: The Power and Versatility of Large Emotion Lexicons. Canadian Digital Services. January 13, 2022.

Judging AI: How to Develop a Healthy Mistrust of AI. The Canadian Appellate Courts Seminar, Halifax, Canada. Nov 25, 2021.

Papers

What Makes Sentences Semantically Related: A Textual Relatedness Dataset and Empirical Study.Mohamed Abdalla, Krishnapriya Vishnubhotla, Saif Mohammad. arXiv:2110.04845. Oct 2021.
Paper (pdf) BibTeX Data and Code

Ethics Sheet for Automatic Emotion Recognition and Sentiment Analysis. Saif M. Mohammad.arXiv:2109.08256. Sep 2021.
Paper (pdf) BibTeX (click on Export Bibtex Citation on the right of the arxiv page)

Ethics Sheets for AI Tasks. Saif M. Mohammad.arXiv preprint arXiv:2107.01183. July 2021. (Last updated Sep 2021)
Paper (pdf) BibTeX (click on Export Bibtex Citation on the right of the arxiv page)

Blog Posts

Ethics Sheets for AI Tasks. July 5, 2021.
Video

Ethics Sheet for Automatic Emotion Recognition and Sentiment Analysis. July 5, 2021.

Invited Talks

Ethics Sheets for AI Tasks and a Case Study for Automatic Emotion Recognition. The University of British Columbia Language Sciences Talks, Vancouver, Canada. July 15, 2021.
Slides Video

Emotion Dynamics of Fictional Characters using Large Emotion Lexicons. The 17th IGEL Conference. June 23, 2021.
Slides

Ethics Sheets for AI Tasks and a Case Study for Automatic Emotion Recognition. The Alan Turing Institute, London, UK. May, 2021.

Paper

Ruddit: Norms of Offensiveness for English Reddit Comments. Rishav Hada, Sohi Sudhir, Pushkar Mishra, Helen Yannakoudakis, Saif M. Mohammad, and Ekaterina Shutova. In Proceedings of the 59th Annual Meeting of the Association of Computational Linguistics (ACL-2021), August 2021.
Paper (pdf) BibTeX Code and Data

Journal Papers

Emotion Dynamics in Movie Dialogues. Will E. Hipson and Saif M. Mohammad. PLoS ONE 16(9): e0256153, September, 2021.
Paper (pdf) BibTeX Code

Examining the Language of Solitude vs. Loneliness in Tweets. Will E. Hipson, Svetlana Kiritchenko, Robert J. Coplan, Saif M. Mohammad. Journal of Social and Personal Relationships. March 2021.
Paper (pdf) BibTeX

Book Chapter

Sentiment Analysis: Automatically Detecting Valence, Emotions, and Other Affectual States from Text. Saif M. Mohammad, Emotion Measurement (Second Edition), Elsevier, 2021.
PDF (arxiv preprint arXiv:2005.11882) BibTeX

Talk

What is a Research Ethics Statement and Why does it Matter? Saif M. Mohammad.
Slides

I presented some of these ideas at the EMNLP 2020 Plenary Panel on Publishing in the Era of Responsible AI: How Can we be Proactive? Considerations and Implications. Panelists: Emily M. Bender, Rosie Campbell, Allan Dafoe, Pascale Fung, Meg Mitchell, Saif M. Mohammad.

Paper

Practical and Ethical Considerations in the Effective use of Emotion and Sentiment Lexicons. Saif M. Mohammad. arXiv preprint arXiv:2011.03492. December 2020.
Paper (pdf) BibTeX

Recognition

Nominated for the outstanding acheivement in research excellence award by National Research Council Canada. November 2020. (1 of 6 nominees)

Invited Talk

Gender Gap in Natural Language Processing Research: Disparities in Authorship and Citations. Women+@DCS Seminar Series, University of Sheffield, October 28 2020, Sheffield, UK.
Slides Video

Papers

Lexichrome: Text Construction and Lexical Discovery with Word-Color Associations Using Interactive Visualization. Chris K. Kim, Uta Hinrichs, and Saif M. Mohammad, and Chris Collins. In Proceedings of DIS 2020: Conference on Designing Interactive Systems, July 2020, Eindhoven, Netherlands.
Paper (pdf) BibTeX Video Interactive Visualization

Examining Citations of Natural Language Processing Literature. Saif M. Mohammad. In Proceedings of the 58th Annual Meeting of the Association of Computational Linguistics (ACL-2020), July 2020, Seattle, USA.

Examines nine questions pertaining to broad trends in citations of NLP papers (across time, across venue types, across paper types, across areas, etc.).

Paper (pdf) BibTeX Presentation   Project Home Page Interactive Visualizations Medium Blog Posts

Gender Gap in Natural Language Processing Research: Disparities in Authorship and Citations. Saif M. Mohammad. In Proceedings of the 58th Annual Meeting of the Association of Computational Linguistics (ACL-2020). July 2020. Seattle, USA.

Summary: Examines eight questions pertaining to disparities across male and female NLP researchers (in authorship and citations).

Paper (pdf) BibTeX Video Presentation   Project Home Page Medium Blog Posts

PoKi: A Large Dataset of Poems by Children. Will E. Hipson, and Saif M. Mohammad. In Proceedings of the 12th Language Resources and Evaluation Conference (LREC-2020), May 2020, Marseille, France.
Paper (pdf) BibTeX Project Home Page and Data

NLP Scholar: An Interactive Visual Explorer for Natural Language Processing Literature. Saif M. Mohammad. In Proceedings of the 58th Annual Meeting of the Association of Computational Linguistics (ACL-2020), July 2020, Seattle, USA.

Presents an interactive visualization tool to help users find (related) work published in the ACL Anthology.

Paper (pdf) BibTeX Presentation Project Home Page Interactive Visualizations Medium Blog Posts

SOLO: A Corpus of Tweets for Examining the State of Being Alone. Svetlana Kiritchenko, Will Hipson, Robert Coplan, and Saif M. Mohammad. In Proceedings of the 12th Language Resources and Evaluation Conference (LREC-2020), May 2020, Marseille, France.
Paper (pdf)    BibTeX       Data

WordWars: A Dataset to Examine the Natural Selection of Words. Saif M. Mohammad. In Proceedings of the 12th Language Resources and Evaluation Conference (LREC-2020), May 2020, Marseille, France.
Paper (pdf)    BibTeX       Data      Project Home Page and Visualizations

NLP Scholar: A Dataset for Examining the State of NLP Research. Saif M. Mohammad. InProceedings of the 12th Language Resources and Evaluation Conference (LREC-2020), May 2020, Marseille, France.

Presents the NLP Scholar Dataset -- a single unified source of information from both the ACL Anthology (AA) and Google Scholar for tens of thousands of NLP papers.Presents initial work on analyzing the volume of research in NLP over the years, identifies some the most cited papers in AA, as well as outlines a list of potential applications of the dataset.

Paper (pdf) BibTeX Project Home Page and Data Interactive Visualizations Medium Blog Posts

The State of NLP Literature: A Diachronic Analysis of the ACL Anthology. Saif M. Mohammad. arXiv preprint arXiv:1911.03562. November 2019.

A manuscript that brings together the analyses of NLP papers first presented in the four State of NLP blog posts.

Paper (pdf) BibTeX Project Home Page and Data Interactive Visualizations Medium Blog Posts

Applied AI Ethics. Report on Canada-United Kingdom Symposia on Ethics in AI in Ottawa, Canada and London, UK. de Bruijn, B., Désillets, A., Fraser, K., Kiritchenko, S., Mohammad, S., Vinson, N., Bloomfield, P., Brace, H., Brzoska, K., Elhalal, A., Ho, K., Kinsey, L., McWhirter, R., Nazare, M., and Ofuri-Kuragu, E. Digital Catapult, London, UK / NRC, Ottawa, Canada, 2019.
Paper (pdf)     Symposium Website

Recognition

Awarded a "Top Writer for Science on Medium" status. November 2019. (One of 50 writers who had this tag at the time. There were about 63K stories in the Science category on Medium at the time.)

Invited Talks

The Search for Emotions, Creativity, and Fairness in Language. Data Systems Seminar Series, University of Waterloo, October 28 2019, Waterloo, Canada.
Slides Video

Fairness and Emotions in Language. The Globe and Mail. October 29, 2019, Toronto, Canada.

Hosting the Responsible AI Summit, October 24 2019, Montreal, Canada.

Creativity and Emotions in Language. Invited talk and panel on AI and Creativity in Government at the Creative Marketplace Lab on Data, Skills and Technology, Department ofCanadianHeritage, September 30 2019, Gatineau, Canada.

The Search for Emotions, Creativity, and Fairness in Language. Mila - Quebec AI Institute, August 2019, Montreal, Canada.
Slides

The Search for Emotions, Creativity, and Fairness in Language. Keynote speech at the 8th KDD Workshop on Issues of Sentiment Discovery and Opinion Mining (WISDOM-2019), August 2019, Anchorage, Alaska.
Slides

Web/Press Mentions

Gender and Racial Bias in Cloud NLP Sentiment APIs, Aug 21, 2019. Article looking into race and gender biases in the Google and AWS cloud sentiment analysis APIs using the Equity Evaluation Corpus and the techniques we published in 2018.

Journal Paper

AffectiveTweets: a Weka Package for Analyzing Affect in Tweets. Felipe Bravo-Marquez, Eibe Frank, Bernhard Pfahringer, Saif M. Mohammad. Journal of Machine Learning Research, 20(92):1−6, 2019.
Paper (pdf) BibTeX Code

Papers

Big Bird: A Large, Fine-Grained, Bigram Relatedness Dataset for Examining Semantic Composition. Shima Asaadi, Saif M. Mohammad, and Svetlana Kiritchenko. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL-2019), June 2019, Minnesota, USA.
Paper (pdf) BibTeX Poster Data Project Home Page and Visualizations Code

How do we feel when a robot dies? Emotions expressed on Twitter before and after hitchBOT's destruction. Kathleen C. Fraser, Frauke Zeller, David Harris Smith, Saif M. Mohammad, and Frank Rudicz. In Proceedings of the NAACL workshop on computational approaches to subjectivity, sentiment, and social media analysis (WASSA-19), June 2019, Minneapolis, USA.
Paper (pdf) BibTeX Slides Visualizations

Invited Talks

Examining Fairness in Language Through Emotions. Keynote speech at the 2019 Canada--UK Workshop on Practical AI Ethics, March 12, 2019, London, UK.
Slides

The Search for Emotions, Creativity, and Fairness in Language. The Alan Turing Institute. March 11, 2019. London, UK.
Slides Video

Examining Fairness in Language Through Emotions. The 2019 Canada--UK Symposium on Ethics in AI, Feb 21, 2019, Ottawa, Canada.
Slides Video

Journal Paper

The Natural Selection of Words: Finding the Features of Fitness. Peter D. Turney and Saif M. Mohammad. PLoS One, 14 (1):e0211512. January 2019.
Paper (pdf) BibTeX Code

Professional Community Involvement

Chair of the 2019 Canada--UK Symposium on Ethics in AI, Feb 21--22, Ottawa, Canada.

Area chair for Sentiment Analysis and Argumentation Mining, EMNLP-IJCNLP-2019, Hong Kong.

Area Chair for ACL 2019 in the area of Sentiment Analysis and Argument Mining.

Area chair for Semantics in NLP Applications, *Sem-2019, Minneapolis, Minnesota, USA.

Area Chair for Ethics, Bias and Fairness at NAACL-2019, Minneapolis, USA.

NAACL-2019 has a special theme encouraging papers on ethics, bias, and airness.

Co-chair of SemEval 2019, Minneapolis, Minnesota, USA.

Mentor, Student Research Workshop, ACL-2019, Florence, Italy

Mentor, Student Research Workshop, NAACL-2019, Minneapolis, Minnesota, USA.

Invited Talks

The Search for Emotions, Creativity, and Fairness in Language. Venue: Keynote speech at the 6th evaluation campaignEVALITA 2018. December 12, 2018. Turin, Italy.

The Search for Emotions, Creativity, and Fairness in Language. Venue: Department of Computational Linguistics, Heidelberg University. December 13, 2018. Heidelberg, Germany.

The Search for Emotions, Creativity, and Fairness in Language. Venue: Invited Speaker Series. Pickering Centre for Human Development, Carleton University. November 16, 2018. Ottawa, Canada.

The Search for Emotions, Creativity, and Fairness in Language. Short talk to 9th graders on the occasion of Bring-Your-Ninth-Grader-to-Work Day at the National Research Council Canada. November 14, 2018. Ottawa, Canada

The Search for Emotions, Creativity, and Fairness in Language. At the Natural Language Processing Seminar and AI Seminar, University of Michigan, Ann Arbor. October 2, 2018. Ann Arbor, Michigan.

Journal Paper

Data and systems for medication-related text classification and concept normalization from Twitter: Insights from the Social Media Mining for Health (SMM4H)-2017 shared task. Abeed Sarker, Maksim Belusov, Jasper Friedrichs, Kai Hakala, Sifei Han, Svetlana Kiritchenko, Farrokh Mehryary, Anthony Rios, Tung Tran, Berry de Bruijn, Filip Ginter, Ramakanth Kavuluru, Debanjan Mahata, Saif M. Mohammad, Goran Nenadic, Graciela Gonzalez-Hernandez. Journal of the American Medical Informatics Association (JAMIA), 25(10):1274--1283, October 2018.

Official Rankings : Our team (NRC-Canada) ranked first in the AMIA Shared Task on detecting adverse drug reactions in tweets.

Invited Talks

The Search for Emotions, Creativity, and Fairness in Language. Department of Computer Science, University of Utah. August 6, 2018. Salt Lake City, Utah.

The Search for Emotions, Creativity, and Fairness in Language. Keynote speech at the ACL Workshop on Natural Language Processing for Social Media. July 20, 2018. Melbourne, Australia.

The Search for Emotions, Creativity, and Fairness in Language. Department of Computer Science, University of Melbourne, July 20, 2018. Melbourne, Australia.

The Search for Emotions, Creativity, and Fairness in Language. The University of Waikato, Computing & Mathematical Sciences. July 10, 2018. Hamilton, New Zealand.

Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems. Invited talk at the Second ACL Workshop on Ethics in Natural Language Processing, New Orleans, LA, USA, June 2018.

The Search for Emotions in Language. Google Research. May 25, 2018. Montreal, Canada.

Professional Community Involvement

Co-organizer of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, to be held in conjunction with EMNLP-2018, Oct 31–Nov 1, Brussels, Belgium.

Papers

Obtaining Reliable Human Ratings of Valence, Arousal, and Dominance for 20,000 English Words. Saif M. Mohammad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia, July 2018.
Paper (pdf)    BibTeX    Project Page and Data     Presentation    Video    Poster
Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems. Svetlana Kiritchenko and Saif M. Mohammad. In Proceedings of *Sem, New Orleans, LA, USA, June 2018.
Paper (pdf)    BibTeX    Project Page and Data     Presentation

Agree or Disagree: Predicting Judgments on Nuanced Assertions. Michael Wojatzki, Torsten Zesch, Saif M. Mohammad, and Svetlana Kiritchenko. In Proceedings of *Sem, New Orleans, LA, USA, June 2018.
Paper (pdf)    BibTeX        Project Page and Data        Presentation

Semeval-2018 Task 1: Affect in tweets. Saif M. Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. In Proceedings of International Workshop on Semantic Evaluation (SemEval-2018), New Orleans, LA, USA, June 2018.
Paper (pdf)    BibTeX       Data and Visualization     Presentation
SemEval-2018 Task 1: Affect in Tweets Webpage

75 teams and about 200 participants.

DeepMiner at SemEval-2018 Task 1: Emotion Intensity Recognition Using Deep Representation Learning. Habibeh Naderi, Svetlana Kiritchenko, Saif M. Mohammad, and Stan Matwin. In Proceedings of International Workshop on Semantic Evaluation (SemEval-2018), New Orleans, LA, USA, June 2018.
Paper (pdf)    BibTeX

WikiArt Emotions: An Annotated Dataset of Emotions Evoked by Art. Saif M. Mohammad and Svetlana Kiritchenko. InProceedings of the 11th Edition of the Language Resources and Evaluation Conference (LREC-2018), May 2018, Miyazaki, Japan.
Paper (pdf)    BibTeX       Poster       Project Page and Data

Word Affect Intensities. Saif M. Mohammad. InProceedings of the 11th Edition of the Language Resources and Evaluation Conference (LREC-2018), May 2018, Miyazaki, Japan.
Paper (pdf)    BibTeX       Presentation       Project Page and Data

Understanding Emotions: A Dataset of Tweets to Study Interactions between Affect Categories. Saif M. Mohammad and Svetlana Kiritchenko. InProceedings of the 11th Edition of the Language Resources and Evaluation Conference (LREC-2018), May 2018, Miyazaki, Japan.
Paper (pdf)    BibTeX       Presentation       Shared Task Page and Data

Quantifying Qualitative Data for Understanding Controversial Issues. Michael Wojatzki, Saif M. Mohammad, Torsten Zesch, and Svetlana Kiritchenko. InProceedings of the 11th Edition of the Language Resources and Evaluation Conference (LREC-2018), May 2018, Miyazaki, Japan.
Paper (pdf)    BibTeX        Presentation      Project Page and Data

Invited Talks

The Search for Emotions in Language. Featured speaker at the Distinguished Computational Linguistics Lecture at Rochester Institute of Technology. March 1, 2018. Rochester, NY, USA.

The Search for Emotions in Language. University of Toronto. Feb 27, 2018. Toronto, Canada.

Finding Emotions in Language. Information Systems Sprott School of Business, Carleton University. November 13, 2017. Ottawa, Canada.

Emotion-Aware Machines. Short talk to 9th graders on the occasion of Bring-Your-Ninth-Grader-to-Work Day at the National Research Council Canada. November 1, 2017. Ottawa, Canada.

Professional Community Involvement

Co-chair for SemEval-2018.

Organizer of SemEval-2018 Task 1: Affect in Tweets.
75 teams and about 200 participants.

Papers

NRC-Canada at SMM4H Shared Task: Classifying Tweets Mentioning Adverse Drug Reactions and Medication Intake. Svetlana Kiritchenko, Saif M. Mohammad, Jason Morin, and Berry de Bruijn (2017). In Proceedings of the Social Media Mining for Health Applications Workshop at AMIA-2017, Washington, DC, USA, 2017.
Paper (pdf)   BibTeX    Our System Homepage

Official Rankings : Our team (NRC-Canada) ranked first in the AMIA Shared Task on detecting adverse drug reactions in tweets.

WASSA-2017 Shared Task on Emotion Intensity. Saif M. Mohammad and Felipe Bravo-Marquez. In Proceedings of the EMNLP 2017 Workshop on Computational Approaches to Subjectivity, Sentiment, and Social Media (WASSA), September 2017, Copenhagen, Denmark.
Paper (pdf)    BibTex     Data and Shared Task    Presentation

Emotion Intensities in Tweets. Saif M. Mohammad and Felipe Bravo-Marquez. In Proceedings of the Sixth Joint Conference on Lexical and Computational Semantics (*Sem), August 2017, Vancouver, Canada.
Paper (pdf)    BibTex     Data and Shared Task    AffcetiveTweets package    Presentation

Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation. Svetlana Kiritchenko and Saif M. Mohammad. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL-2017), Vancouver, Canada, 2017.
Paper (pdf)    BibTeX       Data      Poster

Word Affect Intensities. Saif M. Mohammad. arXiv preprint arXiv:1704.08798, April 2017.
Paper (pdf)

Invited Talks

Affect Associations in Creative Language. Seminar Series for the Doctorate in Cognitive Informatics at Université du Québec à Montréal (UQAM), April 13, 2017, Montreal, Canada.
Presentation

Affect Associations in Creative Language. Keynote speech at the COLING 2016 workshop on Computational Modeling of People's Opinions, Personality, and Emotions in Social Media (PEOPLES 2016), December 12, 2016, Osaka, Japan.
Presentation

Affect Associations: The Building Blocks of Sentiment Analysis. Keynote speech at the 20th International Conference on Asian Language Processing (IALP 2016), November 21-23, 2016, National Cheng Kung University, Tainan, Taiwan.
Presentation

Affect Associations in Creative Language. Keynote speech at the 2016 IR Workshop at the Academia Sinica, November 24, 2016, Taipei, Taiwan.
Presentation

Emotion-Aware Machines. Short talk to 9th graders on the occasion of Bring-Your-Ninth-Grader-to-Work Day at the National Research Council Canada. November 1, 2016. Ottawa, Canada.

Capturing Reliable Fine-Grained Sentiment Associations by Crowdsourcing and Best–Worst Scaling. Invited talk at the TAMALE seminar series, September 15, 2016, University of Ottawa. (Joint work with, and talk given by, Svetlana Kiritchenko.)
Presentation

Metaphor as a Medium for Emotion: An Empirical Study. Invited talk at the TAMALE seminar series, September 15, 2016, University of Ottawa. (Joint work with Ekaterina Shutova and Peter Turney.)
Presentation

Professional Community Involvement

Co-chair for SemEval-2017.

Co-organizer of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis (WASSA-2017), to be held in conjunction with EMNLP-2017, Copenhagen, Denmark.

Organizer of shared task on detecting emotion intensity at WASSA-2017.

Area chair for Sentiment Analysis and Opinion Mining at ACL-2017, Vancouver, Canada.

Area chair for Sentiment Analysis and Opinion Mining at EMNLP-2017, Copenhagen, Denmark.

Event

A symphony orchestra performed music composed using the NRC Emotion Lexicon under the glass of the Louvre museum in Paris on Sept. 20, 2016. Click here for a video of the performance.
Articles published in the Washington Post, CBS News, Columbia Tribune, and others.

Journal Paper

Stance and Sentiment in Tweets. Saif M. Mohammad, Parinaz Sobhani, and Svetlana Kiritchenko. Special Section of the ACM Transactions on Internet Technology on Argumentation in Social Media,2017, 17(3).
Paper (pdf) BibTeX Data and Visualization

Papers

Metaphor as a Medium for Emotion: An Empirical Study. Saif M. Mohammad, Ekaterina Shutova, and Peter Turney. In Proceedings of the Joint Conference on Lexical and Computational Semantics (*Sem), August 2016, Berlin, Germany.
Paper (pdf)    BibTeX     Presentation       Data and Interactive Visualization

Detecting Stance in Tweets And Analyzing its Interaction with Sentiment. Parinaz Sobhani, Saif M. Mohammad, and Svetlana Kiritchenko. In Proceedings of the Joint Conference on Lexical and Computational Semantics (*Sem), August 2016, Berlin, Germany.
Paper (pdf)    BibTeX    Presentation       Data and Visualization

Determining Word-Emotion Associations from Tweets by Multi-Label Classification. Felipe Bravo-Marquez, Eibe Frank, Saif Mohammad, and Bernhard Pfahringer. In Proceedings of the 2016 IEEE/WIC/ACM International Conference on Web Intelligence (WI'16), October 2016, Omaha, Nebraska, USA.
Paper (pdf)    BibTeX    Data (scroll to section on this paper)

Invited Talk

Sentiment and Emotions in Tweets. Department of Geography & Geoinformation Science at George Mason University. May 29, 2016, FairFax, Virginia.
Presentation

Professional Community Involvement

Publicity Chair for the 2016 meeting of the 2016 Conference on Empirical Methods on Natural Language Processing. (EMNLP-2016).

Area Chair for Sentiment and Opinion Mining at EMNLP 2016.

Book Chapter

Challenges in Sentiment Analysis. Saif M. Mohammad, A Practical Guide to Sentiment Analysis, Springer, 2016.
Pre-print version (pdf) BibTeX

Papers

Capturing Reliable Fine-Grained Sentiment Associations by Crowdsourcing and Best-Worst Scaling. Svetlana Kiritchenko and Saif M. Mohammad. In Proceedings of the 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, June 2016, San Diego, California.
Paper (pdf)    BibTeX Presentation Data

Sentiment Composition of Words with Opposing Polarities. Svetlana Kiritchenko and Saif M. Mohammad. In Proceedings of the 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, June 2016, San Diego, California.
Paper (pdf)    BibTeX Poster Data: Opposing Polarity Sentiment Lexicon    Interactive Visualization

The Effect of Negators, Modals, and Degree Adverbs on Sentiment Composition. Svetlana Kiritchenko and Saif M. Mohammad, In Proceedings of the NAACL 2016 Workshop on Computational Approaches to Subjectivity, Sentiment, and Social Media (WASSA), June 2016, San Diego, California.
Paper (pdf)    BibTeX Presentation Data (Sentiment Composition Lexicon of Negators, Modals, and Adverbs (SCL-NMA)) and Visualization

Semeval-2016 Task 6: Detecting Stance in Tweets. Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. In Proceedings of the International Workshop on Semantic Evaluation (SemEval '16). June 2016. San Diego, California.
Paper (pdf)    BibTeX    Presentation    Task Website

Semeval-2016 Task 7: Determining Sentiment Intensity of English and Arabic Phrases. Svetlana Kiritchenko, Saif M. Mohammad, and Mohammad Salameh. In Proceedings of the International Workshop on Semantic Evaluation (SemEval '16). June 2016. San Diego, California.
Paper (pdf)    BibTeX    Presentation    Task Website

A Practical Guide to Sentiment Annotation: Challenges and Solutions. Saif M. Mohammad, In Proceedings of the NAACL 2016 Workshop on Computational Approaches to Subjectivity, Sentiment, and Social Media (WASSA), June 2016, San Diego, California.
Paper (pdf)    BibTeX Presentation

Sentiment Lexicons for Arabic Social Media. Saif M. Mohammad, Mohammad Salameh, and Svetlana Kiritchenko. In Proceedings of the 10th edition of the Language Resources and Evaluation Conference, May 2016, Portoro� (Slovenia).
Paper (pdf)    BibTeX Presentation    Video        Data: Arabic Sentiment Lexicons

A Dataset for Detecting Stance in Tweets. Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. In Proceedings of the 10th edition of the Language Resources and Evaluation Conference, May 2016, Portoro� (Slovenia).
Paper (pdf)    BibTeX Presentation Data: Stance Dataset    Interactive Visualization

Happy Accident: A Sentiment Composition Lexicon for Opposing Polarities Phrases. Svetlana Kiritchenko and Saif M. Mohammad. In Proceedings of the 10th edition of the Language Resources and Evaluation Conference, May 2016, Portoro� (Slovenia).
Paper (pdf)    BibTeX Poster    Data: Opposing Polarity Sentiment Lexicon    Interactive Visualization

Book Chapter

Sentiment Analysis: Detecting Valence, Emotions, and Other Affectual States from Text. Saif M. Mohammad, Emotion Measurement, 2016.
Pre-print version (pdf) BibTeX
This is a survey on automatic methods for affect analysis.

Interactive Visualization and Paper

Imagisaurus: An Interactive Visualizer of Valence and Emotion in the Roget's Thesaurus. Saif M. Mohammad. In Proceedings of the EMNLP 2015 Workshop on Computational Approaches to Subjectivity, Sentiment, and Social Media (WASSA), September 2015, Lisbon, Portugal.
Paper (pdf) BibTeX Interactive Visualization

Data

The NRC Emotion Lexicon is now available in over 20 languages.

Journal Paper

How Translation Alters Sentiment. Saif M. Mohammad, Mohammad Salameh, and Svetlana Kiritchenko, Journal of Artificial Intelligence Research , January 2016, 55:95-130.
Paper (pdf) BibTeX Data: Arabic Sentiment Lexicons

Professional Community Involvement

Organizing these shared task competitions under the aegis of SemEval-2016 (see webpage for schedule):

Detecting Stance in Tweets (new task). Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry.

Determining sentiment intensity of English and Arabic phrases. Svetlana Kiritchenko, Saif M Mohammad, and Mohammad Salameh. This is an expansion of the SemEval-2015 Task 10 subtask E - Determining strength of association of Twitter terms with positive sentiment (or, degree of prior polarity).

Journal Paper

Developing a Successful SemEval Task in Sentiment Analysis of Twitter and Other Social Media Texts.Preslav Nakov, Sara Rosenthal, Svetlana Kiritchenko, Saif M. Mohammad, Zornitsa Kozareva, Alan Ritter, Veselin Stoyanov, and Xiaodan Zhu. Language Resources and Evaluation. January 2016, 50(1):35-65.
Paper (pdf) Preprint Version (pdf) BibTeX

Tutorial

Computational Analysis of Affect and Emotion in Language. Saif M. Mohammad and Cecilia Ovesdotter Alm. Tutorial at the 2015 Conference on Empirical Methods on Natural Language Processing, September 2015, Lisboa, Portugal.
Presentation Annotated Bibliography Extended Bibliography Proposal

Papers

SemEval-2015 Task 10: Sentiment Analysis in Twitter. Sara Rosenthal, Preslav Nakov, Svetlana Kiritchenko, Saif M Mohammad, Alan Ritter, and Veselin Stoyanov. In Proceedings of the ninth international workshop on Semantic Evaluation Exercises (SemEval-2015), June 2015, Denver, Colorado.
Paper (pdf) BibTeX

Sentiment After Translation: A Case-Study on Arabic Social Media Posts. Mohammad Salameh, Saif M. Mohammad, and Svetlana Kiritchenko, In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL-2015), June 2015, Denver, Colorado.
Paper (pdf) BibTeX Data: rabic Sentiment Lexicons

Visualization

Explore the interactive visualization for the NRC Word-Emotion Association Lexicon.

Symposium

My N is Ten Million: Using Social Media to Track Emotion, Mental Health, and Measure Personality Across Entire Populations. Gregory J Park, Saif M Mohammad, and Johannes C Eichstaedt. A symposium at the International Convention of Psychological Science (ICPS), March 2015, Amsterdam, The Netherlands.

Tutorial

Sentiment Analysis of Social Media Texts. Saif M. Mohammad and Xiaodan Zhu. Tutorial at the 2014 Conference on Empirical Methods on Natural Language Processing, October 2014, Doha, Qatar.
Presentation Video Proposal

Professional Community Involvement

I am serving as Publicity Chair for the 2015 meeting of the North American Association for Computational Linguistics (NAACL-2015). I am also the Area Chair for Sentiment and Opinion Mining.

Invited Talk

The Words are Alive: Associations with Sentiment, Emotions, Colours, and Music. Invited talk at Language Technology Institute, Carnegie Mellon University, September 2014, Pittsburgh, PA.
Presentation Video

Journal Paper

Sentiment, Emotion, Purpose, and Style in Electoral Tweets. Saif M. Mohammad, Svetlana Kiritchenko, Xiaodan Zhu, and Joel Martin. Information Processing and Management, Volume 51, Issue 4, July 2015, Pages 480–499.
Paper (pdf) BibTeX AnnotatedData

Invited Talk

Words: Evaluative, Emotional, Colourful, Musical! Keynote speech at the ACL 2014 Workshop on Computational Approaches to Subjectivity, Sentiment, and Social Media (WASSA), June 27, 2014, Baltimore, MD.
Presentation

Journal Papers

Experiments with Three Approaches to Recognizing Lexical Entailment. Peter D. Turney, Saif M. Mohammad, Natural Language Engineering, 21(3):437-476, May 2015.
Paper (pdf) BibTeX

Using Hashtags to Capture Fine Emotion Categories from Tweets. Saif M. Mohammad, Svetlana Kiritchenko, Computational Intelligence, 31( 2):301-326, May 2015.
Paper (pdf) BibTeX

Professional Community Involvement

One of the organizers of SemEval-2015 Task 10 - Sentiment Analysis in Twitter. Of special interest to me is subtask E - Determining strength of association of Twitter terms with positive sentiment (or, degree of prior polarity). Task description, trial data, test data, and other details available here.

Journal Paper

Papers

NRC-Canada-2014: Detecting Aspects and Sentiment in Customer Reviews, Svetlana Kiritchenko, Xiaodan Zhu, Colin Cherry, and Saif M. Mohammad. In Proceedings of the eighth international workshop on Semantic Evaluation Exercises (SemEval-2014), August 2014, Dublin, Ireland.
Paper (pdf)    BibTeX     Poster

Official Rankings: Our team (NRC-Canada) ranked first in three of the six subtasks. About 30 teams participated.

NRC-Canada-2014: Recent Improvements in the Sentiment Analysis of Tweets, Xiaodan Zhu, Svetlana Kiritchenko, and Saif M. Mohammad. In Proceedings of the eighth international workshop on Semantic Evaluation Exercises (SemEval-2014), August 2014, Dublin, Ireland.
Paper (pdf)    BibTeX     Poster

Official Rankings: Our team (NRC-Canada) ranked first in five of the ten subtask-domain combinations. About 40 teams participated.

An Empirical Study on the Effect of Negation Words on Sentiment. Xiaodan Zhu, Hongyu Guo, Saif Mohammad and Svetlana Kiritchenko. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, June 2014, Baltimore, MD.
Paper (pdf)    BibTeX

Semantic Role Labeling of Emotions in Tweets. Saif M. Mohammad, Xiaodan Zhu, and Joel Martin, In Proceedings of the ACL 2014 Workshop on Computational Approaches to Subjectivity, Sentiment, and Social Media (WASSA), June 2014, Baltimore, MD.
Paper (pdf)    BibTeX     AnnotatedData

Generating Music from Literature. Hannah Davis and Saif M. Mohammad, In Proceedings of the EACL Workshop on Computational Linguistics for Literature, April 2014, Gothenburg, Sweden.
Paper (pdf)    BibTeX    TransProse Website

Notable Press Mentions: The Physics arXiv Blog, March 20, 2014, TIME, May 7, 2014, PC World, May 15, 2014, Popular Science, May 14, 2014, io9, May 12, 2014, LiveScience, May 11, 2014.

QUICK LINKS

Presence in Research Networks:

ACL Anthology

dblp

Google Scholar page of Saif

Google Scholar pages of NLP Researchers at NRC

LinkedIn

NRC

Research Gate

Semantic Scholar

Twitter

Lexicons:

the NRC Emotion Lexicon

the NRC Affect / Emotion Intensity Lexicon

the NRC Valence, Arousal, and Dominance Lexicon

the NRC Word-Colour Association Lexicon

all lexicons

Other:

Best-Worst Scaling Research, Scripts, and Data

2021 Survey on Emotion Detection from Text and Sentiment Analysis

The NRC-Canada Sentiment Anaysis system

Pinned Blog Posts

March 2020, Ten Years of the NRC Word-Emotion Association Lexicon

Ten years back, with some anticipation and lots of excitement, Peter Turney and I introduced the NRC Word-Emotion Association Lexicon. So grateful that, over the years, so many people have put their hopes and trust in it. Such joy to see them boldly shine a light on the human condition — the positives and the negatives; not shying away even from sadness, fear, and anger. This blog bost puts a spotlight on ten favorites (includes fun video and audio clips as well as links to papers and popular press articles):

October 2019, The State of NLP Literature

This series of posts presents a diachronic analysis of the ACL Anthology — Or, as I like to think of it, making sense of NLP Literature through pictures.

PROFESSIONAL ACTIVITIES

2012--ongoing

Standing reviewer for Transactions of ACL (TACL) journal

2022

Senior Area Chair for Ethics and NLP, ACL-2022.

2021

Action Editor, Computational Linguistics

Action Editor, ACL Rolling Review

Senior Area Chair for Sentiment Analysis, Stylistic Analysis, and Argument Mining, ACL-IJCNLP-2021.

2020

Action Editor, Computational Linguistics

Senior Area Chair for Sentiment Analysis, Stylistic Analysis, and Argument Mining, EMNLP-2020, Punta Cana, Dominican Republic

Area Chair for Sentiment Analysis, Stylistic Analysis, and Argument Mining, StarSem-2020, Barcelona, Spain

Reviewer for Nature

Workshops Co-chair ACL-2020, Seattle, USA

Area chair for Lexical Semantics, ACL-2020, Seattle, USA

Member of the COLING 2020 Ethics Advisory Group

2019

Area chair for Sentiment Analysis and Argumentation Mining, EMNLP-IJCNLP-2019, Hong Kong, China

Area chair for Sentiment Analysis and Argumentation Mining, ACL-2019, Florence, Italy

Area chair for Ethics, Bias and Fairness, NAACL-2019, Minneapolis, Minnesota, USA

Area chair for Semantics in NLP Applications, *Sem-2019, Minneapolis, Minnesota, USA

General Chair of the 2019 Canada–UK Symposium on Ethics in AI, Ottawa, Canada

Co-chair of SemEval 2019, Minneapolis, Minnesota, USA

Mentor, Student Research Workshop, ACL-2019, Florence, Italy

Mentor, Student Research Workshop, NAACL-2019, Minneapolis, Minnesota, USA

2018

Co-chair of SemEval-2018, New Orleans, Louisiana, USA

Co-organizer of WASSA-2018, Brussels, Belgium

Mentor, Student Research Workshop, NAACL-2018, New Orleans, Louisiana, USA

Mentor, Student Research Workshop, ACL-2018, Melbourne, Australia.

2017

Co-chair of SemEval-2017, Vancouver, Canada

Co-organizer of WASSA-2017, Copenhagen, Denmark

Area chair for Sentiment and Opinion Mining, ACL-2017, Vancouver, Canada

Area chair for Sentiment and Opinion Mining, EMNLP-2017, Copenhagen, Denmark

Mentor, Student Research Workshop, ACL-2017, Vancouver, Canada.

2016

Area chair for Sentiment and Opinion Mining, EMNLP-2016, Austin, Texas, USA

Publicity chair, EMNLP-2016, Austin, Texas, USA

2015

Area chair for Sentiment and Opinion Mining, NAACL-2015, Denver, Colorado

Publicity chair, NAACL-2015, Denver, Colorado

Mentor, Student Research Workshop, NAACL-2015, Denver, Colorado

2014, 2012, 2011

Mentor, Student Research Workshops, ACL-2014, NAACL-2012, ACL-2011

Most years 2008–present

Program committee member for top NLP journals and conferences, including TACL, ACL, NAACL, EMNLP, CL, CI, NLE, JAIR, TSLP, IJCNLP, ICWSM, and IJCAI.

NOTABLE WEB/PRESS MENTIONS

CBC Radio, Feb 12, 2021. Automatic emotion detection: benefits and potential harms.

The Economist, Jan 12, 2021. How Donald Trump evolved into a prolific, angry Twitter user. Use of the NRC Emotion lexicon to Analyze emotions in tweets by Donald Trump.

AdNews, April 20, 2020. Coronavirus: Analysing stories from the Twitter frontline. Use of the NRC Emotion lexicon to Analyze emotions in tweets during the Covid-19 pandemic.

Tech Xplore, February 26, 2020. A language generation system that can compose creative poetry. The NRC Emotion Lexicon is used along with a deep learning system for automatic poetry generation. Original paper: Introducing Aspects of Creativity in Automatic Poetry Generation.

USA Today, Dec 24, 2019. Analysis: Trump more negative, prolific on Twitter amid Democratic impeachment inquiry. Use of the NRC Emotion lexicon to Analyze emotions in tweets by Donald Trump.

Universo Online, Dec 5, 2019. Gender bias in sentiment analysis systems and gender disparities in language.

Politico.mx, August 28, 2019. Enojo o sorpresa: emociones en Informes de López Portillo a Calderó. The NRC Emotion Lexicon is used to analyze the speeches of the Mexican President and other politicians.

Gender and Racial Bias in Cloud NLP Sentiment APIs, Aug 21, 2019. Article looking into race and gender biases in the Google and AWS cloud sentiment analysis APIs using the Equity Evaluation Corpus and the techniques we published in 2018.

The New York Times, February 26, 2019. Extreme Weather Can Feel 'Normal' After Just a Few Years, Study Finds. Brief interview on my opinion on work on perceptions of climate change.

Psychology Today, December 27, 2018. Emotional Contagions Can Spread Like Wildfire Via YouTube. The NRC Emotion Lexicon is used to analyze the emotions in user comments on vlogs. Original Paper: Multilevel emotion transfer on YouTube: Disentangling the effects of emotional contagion and homophily on video audiences.

InformationWeek, November 15, 2018. Decoding Programmers: How Emotions Can Change Code. Analyzing developer comments to show that their emotions impact their code.

Medium, DAIR, Artificial Intelligence Research, Perspectives, and Technologies, Aug 2, 2018. Examining Gender and Race Bias in Sentiment Analysis Systems.

SeekingAlpha, Apr. 30, 2018. A Sentiment Analysis Approach To Predicting Stock Returns. The NRC Emotion Lexicon is used to analyze 10-K reports of S&P 500 companies to predict future stock returns.

Technical.ly, February 1, 2018. Who's the angriest character on 'Seinfeld'? The NRC Emotion Lexicon is used to analyze the language of the characters in Seinfeld.

January 24, 2018. An examination of the Twitter habits of Kentucky lawmakers. The NRC Emotion Lexicon is used to analyze the tweets of Democrats and Republicans.

April 13, 2017. Faster but less furious: why the Fast and Furious franchise is a box office goldmine. The NRC Emotion Lexicon is used to analyze the emotions in the Fast and Furious series movie scripts.

The Crosstab, March 2, 2017. Trump's SOTU vs. the Past — Sentiment Analysis and Topic Modeling.

Esquire, February 23, 2017. All Radiohead Songs Are Sad, but This Graph Shows Which Are the Saddest. The NRC Emotion Lexicon is used to compute the degree of sadness for Radiohead songs. Interactive Chart.
Also picked up in:

A Journal of Musical Things, February 24, 2017. Science Discovers Which Radiohead Song is the Saddest

Thomas Park, February 28. Where I End and You Begin: Finding the Most Depressing Radiohead Songs using Crowd Data from SongMeanings.

Religion in Public, February 13, 2017. Evangelical leaders are as negative on immigration and Islam as Donald Trump.

February 11, 2017. Text Analysis of NHL Hockey Coach Interviews.

Washington Post, October 22, 2016. Donald Trump and Hillary Clinton took to the debate stage and made sweet, sweet music. The article mentions NRC in relation to the lexicons we created, which were used to generate music from the Trump-Hillary debate text.

Washington Post, CBS News, Columbia Tribune, and others, September 23, 2016. This symphony had both human and computer composers. Articles about a symphony orchestra performed music composed using the NRC Emotion Lexicon under the glass of the Louvre museum in Paris on Sept. 20, 2016. Click here for a video of the performance.

Washington Post, August 12, 2016. Two people write Trump's tweets. He writes the angrier ones.

BGR, August 11, 2016. Donald Trump's angriest tweets are sent from his Android while the nice ones are sent from an iPhone.

NYC Data Science Academy, August 7, 2016. Twitter Analysis of Presidential Candidates 2016.

Variance Explained, August 6, 2016. Text analysis of Trump's tweets confirms he writes only the (angrier) Android half.

News also picked up by NPR, Los Angeles Times, Scientific American, The Verge, and others.

The Telegraph, June 15, 2016: EU referendum: Remain uses Project Fear more in tweets than Leave, analysis shows. [Use of the NRC Emotion Lexicon, aka EmoLex, to track sentiment in EU referendum tweets (Brexit).]

An art project, the Wishing Wall, that uses the NRC Emotion lexicon was displayed in:

Tekniska Museet, Stockholm, Sweden (Oct 2014 – Aug 2015)

Onassis Cultural Centre, Athens (19th Oct 2015 – 10th Jan 2016)

Zorlu Centre in Istanbul (16th Feb – 12th Jun 2016)

Fast Company, March 25, 2016: An Emotional Map Of The City, As Captured Through Its Sounds. [Use of the NRC Emotion Lexicon, aka EmoLex to create Chatty Maps.]

Press release stating the use of EmoLex to create Chatty Maps.

PC World, May 15, 2014: AI System Reads Novels, Writes Music for Them.

Billboard, May 19, 2014. Computers, Classics and Cadenzas: Making Math Music From Literature. On our work on automatically generating music that captures the emotions in a novel.

Popular Science, May 14, 2014: Robot Reads Novels, Writes Songs about Them.

io9, May 12, 2014: Researchers Train Computers to Manipulate Human Emotions with Art.

LiveScience, May 11, 2014: 'TransProse' Software Creates Musical Soundtracks from Books.

TIME, May 7, 2014: This Is What Classic Novels Sound Like When a Computer Turns Them Into Piano Music.

SlashDot, March23, 2014: Algorithm Composes Music By Text Analyzing the World's Best Novels.

The Physics arXiv Blog, March 20, 2014: The Music Composed By An Algorithm Analysing The World's Best Novels.

Glass Hammer, December 3, 2013: Are Your Emails Communicating a Lack of Confidence?

Singularity Hub, November 10, 2013: Algorithm Tracks Literary Emotion in Shakespeare, the Brothers Grimm.

The Physics ArXiv, October 4, 2013: Data Mining Reveals the Emotional Differences in Emails Written by Men and Women.

SlashDot, October 4, 2013: Data Mining Reveals the Emotional Differences In Emails From Men and Women.

The Physics ArXiv, October 1, 2013: Text Analyser Reveals Emotional Temperature of Novels and Fairy Tales.

Also in EduBits

SlashDot, October 1, 2013: Text Analyzer Reveals Emotional 'Temperature' of Novels and Fairy Tales

The New Scientist, September 27: What your email style says about your personality

Also in Times of India, MSN, Pharmacon, Galileo, Amic, and others

Article in MIT Technology Review, September 5, 2013: How Mechanical Turkers Crowdsourced a Huge Lexicon of Links Between Words and Emotion.

Also in its Spanish website and in crowdsourcing.org.

TIME, August 14, 2013: Main Tweet: Researchers Dig Into The Intersection of Politics and Twitter.

Organization of International Shared Task Competitions

Co-chair of SemEval 2019, Minneapolis, Minnesota, USA

Co-chair of SemEval-2018, New Orleans, Louisiana, USA
(12 tasks, 196 papers, ~1000 participants)

Organizer of SemEval-2018 Task 1: Affect in Tweets (75 teams and about 200 participants)

Co-chair of SemEval-2017, Vancouver, Canada
(12 tasks, 181 papers, ~900 participants)

Organizer of shared task on detecting emotion intensity at WASSA-2017

Co-organizer of Semeval-2016 Task 6: Detecting Stance in Tweets

Co-organizer of Semeval-2016 Task 7: Determining Sentiment Intensity of English and Arabic Phrases

Co-organizer of SemEval-2015 Task 10: Sentiment Analysis in Twitter

Co-organizer of SemEval-2012 Task 2: Measuring Degrees of Relational Similarity