Publications

DADIT: A Dataset for Demographic Classification of Italian Twitter Users and a Comparison of Prediction Methods

May, 2024
Social scientists increasingly use demographically stratified social media data to study the attitudes, beliefs, and behavior of the …

Emotion Analysis in NLP: Trends, Gaps and Roadmap for Future Directions

March, 2024
Emotions are a central aspect of communication. Consequently, emotion analysis (EA) is a rapidly growing field in natural language …

Angry Men, Sad Women: Large Language Models Reflect Gendered Stereotypes in Emotion Attribution

March, 2024
Large language models (LLMs) reflect societal norms and biases, especially about gender. While societal biases and stereotypes have …

Subjective isms? On the Danger of Conflating Hate and Offence in Abusive Language Detection

March, 2024
Natural language processing research has begun to embrace the notion of annotator subjectivity, motivated by variations in labelling. …

Impoverished Language Technology: The Lack of (Social) Class in NLP

March, 2024
Since Labov’s (1964) foundational work on the social stratification of language, linguistics has dedicated concerted efforts …

Classist Tools: Social Class Correlates with Performance in NLP

March, 2024
Since the foundational work of William Labov on the social stratification of language (Labov, 1964), linguistics has made concentrated …

Mirages. On Anthropomorphism in Dialogue Systems

December, 2023
Automated dialogue or conversational systems are anthropomorphised by developers and personified by users. While a degree of …

The Empty Signifier Problem: Towards Clearer Paradigms for Operationalising 'Alignment' in Large Language Models

November, 2023
In this paper, we address the concept of ‘alignment’ in large language models (LLMs) through the lens of post-structuralist …

SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Models

November, 2023
The past year has seen rapid acceleration in the development of large language models (LLMs). For many tasks, there is now a wide range …

XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models

October, 2023
Without proper safeguards, large language models will readily follow malicious instructions and generate toxic content. This risk …

The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values

October, 2023
Human feedback is increasingly used to steer the behaviours of Large Language Models (LLMs). However, it is unclear how to collect and …

Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions

September, 2023
Training large language models to follow instructions makes them perform better on a wide range of tasks, generally becoming more …

Wisdom of Instruction-Tuned Language Model Crowds. Exploring Model Label Variation

July, 2023
Large Language Models (LLMs) exhibit remarkable text classification capabilities, excelling in zero- and few-shot learning (ZSL and …

What about ''em''? How Commercial Machine Translation Fails to Handle (Neo-)Pronouns

July, 2023
As 3rd-person pronoun usage shifts to include novel forms, e.g., neopronouns, we need more research on identity-inclusive NLP. …

The State of Profanity Obfuscation in Natural Language Processing Scientific Publications

July, 2023
Work on hate speech has made considering rude and harmful examples in scientific publications inevitable. This situation raises various …

The Ecological Fallacy in Annotation: Modeling Human Label Variation goes beyond Sociodemographics

July, 2023
Many NLP tasks exhibit human label variation, where different annotators give different labels to the same texts. This variation is …

Temporal and Second Language Influence on Intra-Annotator Agreement and Stability in Hate Speech Labelling

July, 2023
Much work in natural language processing (NLP) relies on human annotation. The majority of this implicitly assumes that annotator’s …

Respectful or Toxic? Using Zero-Shot Learning with Language Models to Detect Hate Speech

July, 2023
Hate speech detection faces two significant challenges: 1) the limited availability of labeled data and 2) the high variability of hate …

MilaNLP at SemEval-2023 Task 10: Ensembling Domain-Adapted and Regularized Pretrained Language Models for Robust Sexism Detection

July, 2023
We present the system proposed by the MilaNLP team for the Explainable Detection of Online Sexism (EDOS) shared task. We propose an …

A Multi-dimensional study on Bias in Vision-Language models

July, 2023
In recent years, joint Vision-Language (VL) models have increased in popularity and capability. Very few studies have attempted to …

Leveraging Social Interactions to Detect Misinformation on Social Media

June, 2023
Detecting misinformation threads is crucial to guarantee a healthy environment on social media. We address the problem using the data …

Computer says “No”: The Case Against Empathetic Conversational AI

June, 2023
Emotions are an integral part of human cognition and they guide not only our understanding of the world but also our actions within it. …

A Cross-Lingual Study of Homotransphobia on Twitter

May, 2023
We present a cross-lingual study of homotransphobia on Twitter, examining the prevalence and forms of homotransphobic content in tweets …

Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale

May, 2023
Machine learning models are now able to convert user-written text descriptions into naturalistic images. These models are available to …

Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP)

May, 2023
Natural Language Processing has seen impressive gains in recent years. This research includes the demonstration by NLP models to have …

ferret: a Framework for Benchmarking Explainers on Transformers

May, 2023
As Transformers are increasingly relied upon to solve complex NLP problems, there is an increased need for their decisions to be …

Can Demographic Factors Improve Text Classification? Revisiting Demographic Adaptation in the Age of Transformers

May, 2023
Demographic factors (e.g., gender or age) shape our language. Previous work showed that incorporating demographic factors can …

Know Your Audience: Do LLMs Adapt to Different Age and Education Levels?

April, 2023
Large language models (LLMs) offer a range of new possibilities, including adapting the text to different audiences and their reading …

Beyond Digital 'Echo Chambers': The Role of Viewpoint Diversity in Political Discussion

February, 2023
Increasingly taking place in online spaces, modern political conversations are typically perceived to be unproductively …

Viewpoint: Artificial Intelligence Accidents Waiting to Happen?

January, 2023
Artificial Intelligence (AI) is at a crucial point in its development: stable enough to be used in production systems, and increasingly …

Twitter-Demographer: A Flow-based Tool to Enrich Twitter Data

December, 2022
Twitter data have become essential to Natural Language Processing (NLP) and social science research, driving various scientific …

It's Not Just Hate: A Multi-Dimensional Perspective on Detecting Harmful Speech Online

December, 2022
Well-annotated data is a prerequisite for good Natural Language Processing models. Too often, though, annotation decisions are governed …

SocioProbe: What, When, and Where Language Models Learn about Sociodemographics

December, 2022
Pre-trained language models (PLMs) have outperformed other NLP models on a wide range of tasks. Opting for a more thorough …

Bridging Fairness and Environmental Sustainability in Natural Language Processing

December, 2022
Fairness and environmental impact are important research directions for the sustainable development of artificial intelligence. …

Measuring Harmful Representations in Scandinavian Language Models

December, 2022
Scandinavian countries are perceived as role-models when it comes to gender equality. With the advent of pre-trained language models …

Data-Efficient Strategies for Expanding Hate Speech Detection into Under-Resourced Languages

October, 2022
Hate speech is a global phenomenon, but most hate speech datasets so far focus on English-language content. This hinders the …

Is It Worth the (Environmental) Cost? Limited Evidence for the Benefits of Diachronic Continuous Training

October, 2022
Language is constantly changing and evolving, leaving language models to quickly become outdated, both factually and linguistically. …

Welcome to the Modern World of Pronouns: Identity-Inclusive Natural Language Processing beyond Gender

October, 2022
The world of pronouns is changing – from a closed word class with few members to an open set of terms to reflect identities. However, …

Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design

September, 2022
Over the last several years, end-to-end neural conversational agents have vastly improved their ability to carry unrestricted, …

Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Models

July, 2022
Hate speech detection models are typically evaluated on held-out test sets. However, this risks painting an incomplete and potentially …

HATE-ITA: Hate Speech Detection in Italian Social Media Text

July, 2022
Online hate speech is a dangerous phenomenon that can (and should) be promptly counteracted properly. While Natural Language Processing …

Hard and Soft Evaluation of NLP models with BOOtSTrap SAmpling - BooStSa

May, 2022
Natural Language Processing (NLP) ‘s applied nature makes it necessary to select the most effective and robust models. Producing …

MilaNLP at SemEval-2022 Task 5: Using Perceiver IO for Detecting Misogynous Memes with Text and Image Modalities

April, 2022
In this paper, we describe the system proposed by the MilaNLP team for the Multimedia Automatic Misogyny Identification (MAMI) …

Language Invariant Properties in Natural Language Processing

April, 2022
Meaning is context-dependent, but many properties of language (should) remain the same even if we transform the context. For example, …

XLM-EMO: Multilingual Emotion Prediction in Social Media Text

April, 2022
Detecting emotion in text allows social and computational scientists to study how people behave and react to online events. However, …

Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks

April, 2022
Labelled data is the foundation of most natural language processing tasks. However, labelling data is difficult and there often are …

Pipelines for Social Bias Testing of Large Language Models

April, 2022
The maturity level of language models is now at a stage in which many companies rely on them to solve various tasks. However, while …

Measuring Harmful Sentence Completion in Language Models for LGBTQIA+ Individuals

April, 2022
Current language technology is ubiquitous and directly influences individuals' lives worldwide. Given the recent trend in AI on …

Benchmarking Post-Hoc Interpretability Approaches for Transformer-based Misogyny Detection

April, 2022
Transformer-based Natural Language Processing models have become the standard for hate speech detection. However, the unconscious use …

Fair and Argumentative Language Modeling for Computational Argumentation

April, 2022
Although much work in NLP has focused on measuring and mitigating stereotypical bias in semantic spaces, research addressing bias in …

DS-TOD: Efficient Domain Specialization for Task Oriented Dialog

April, 2022
Recent work has shown that self-supervised dialog-specific pretraining on large conversational datasets yields substantial gains over …

SAFETYKIT: First Aid for Measuring Safety in Open-domain Conversational Systems

March, 2022
The social impact of natural language processing and its applications has received increasing attention. In this position paper, we …

Entropy-based Attention Regularization Frees Unintended Bias Mitigation from Lists

March, 2022
Natural Language Processing (NLP) models risk overfitting to specific terms in the training data, thereby reducing their performance, …

Text Analysis in Python for Social Scientists – Prediction and Classification

January, 2022
Text contains a wealth of information about about a wide variety of sociocultural constructs. Automated prediction methods can infer …

Learning from Disagreement: A Survey

December, 2021
Many tasks in Natural Language Processing (NLP) and Computer Vision (CV) offer evidence that humans disagree, from objective tasks such …

Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence

August, 2021
Topic models extract groups of words from documents, whose interpretation as a topic hopefully allows for a better understanding of the …

On the Gap between Adoption and Understanding in NLP

August, 2021
There are some issues with current research trends in NLP that can hamper the free development of scientific research. We identify five …

Five sources of bias in natural language processing

August, 2021
Recently, there has been an increased interest in demographically grounded bias in natural language processing (NLP) applications. Much …

Exposing the limits of Zero-shot Cross-lingual Hate Speech Detection

August, 2021
Reducing and counter-acting hate speech on Social Media is a significant concern. Most of the proposed automatic methods are conducted …

'We will Reduce Taxes' - Identifying Election Pledges with Language Models

August, 2021
In an election campaign, political parties pledge to implement various projects–should they be elected. But do they follow …

The Importance of Modeling Social Factors of Language: Theory and Practice

June, 2021
Natural language processing (NLP) applications are now more powerful and ubiquitous than ever before. With rapidly developing (neural) …

HONEST: Measuring Hurtful Sentence Completion in Language Models

June, 2021
Language models have revolutionized the field of NLP. However, language models capture and proliferate hurtful stereotypes, especially …

Language in a (Search) Box: Grounding Language Learning in Real-World Human-Machine Interaction

June, 2021
We investigate grounded language learning through real-world data, by modelling a teacher-learner dynamics through the natural interactions occurring between users and search engines.

MilaNLP @ WASSA: Does BERT Feel Sad When You Cry?

May, 2021
The paper describes the MilaNLP team’s submission (Bocconi University, Milan) in the WASSA 2021 Shared Task on Empathy Detection and …

FEEL-IT: Emotion and Sentiment Classification for the Italian Language

May, 2021
Sentiment analysis is a common task to understand people’s reactions online. Still, we often need more nuanced information: is …

Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning

May, 2021
Supervised learning assumes that a ground truth label exists. However, the reliability of this ground truth depends on human …

Universal Joy A Data Set and Results for Classifying Emotions Across Languages

April, 2021
While emotions are universal aspects of human psychology, they are expressed differently across different languages and cultures. We …

BERTective: Language Models and Contextual Information for Deception Detection

April, 2021
Spotting a lie is challenging but has an enormous potential impact on security as well as private and public safety. Several NLP …

Cross-lingual Contextualized Topic Models with Zero-shot Learning

March, 2021
We introduce a novel topic modeling method that can make use of contextulized embeddings (e.g., BERT) to do zero-shot cross-lingual topic modeling.

Text Analysis in Python for Social Scientists – Discovery and Exploration

December, 2020
Text is everywhere, and it is a fantastic resource for social scientists. However, because it is so abundant, and because language is …

“You Sound Just Like Your Father” Commercial Machine Translation Systems Include Stylistic Biases

July, 2020
The main goal of machine translation has been to convey the correct content. Stylistic considerations have been at best secondary. We …

Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview

July, 2020
An increasing number of natural language processing papers address the effect of bias on predictions, introducing mitigation techniques …

Visualizing Regional Language Variation Across Europe on Twitter

March, 2020
Geotagged Twitter data allows us to investigate correlations of geographic language variation, both at an interlingual and intralingual …

What the [MASK]? Making Sense of Language-Specific BERT Models

March, 2020
Recently, Natural Language Processing (NLP) has witnessed an impressive progress in many areas, due to the advent of novel, pretrained …

Helpful or Hierarchical? Predicting the Communicative Strategies of Chat Participants, and their Impact on Success

March, 2020
When interacting with each other, we motivate, advise, inform, show love or power towards our peers. However, the way we interact may …

Fake opinion detection: how similar are crowdsourced datasets to real data?

January, 2020
Identifying deceptive online reviews is a challenging tasks for Natural Language Processing (NLP). Collecting corpora for the task is …

A Case for Soft Loss Functions

January, 2020
Recently, Peterson et al. provided evidence of the benefits of using probabilistic soft labels generated from crowd annotations for …

Identifying Linguistic Areas for Geolocation

November, 2019
Geolocating social media posts relies on the assumption that language carries sufficient geographic information. However, locations are …

Hey Siri. Ok Google. Alexa: A topic modeling of user reviews for smart speakers

November, 2019
User reviews provide a significant source of information for companies to understand their market and audience. In order to discover …

Geolocation with Attention-Based Multitask Learning Models

November, 2019
Geolocation, predicting the location of a post based on text and other information, has a huge potential for several social media …

Dense Node Representation for Geolocation

November, 2019
Prior research has shown that geolocation can be substantially improved by including user network information. While effective, it …

Women’s Syntactic Resilience and Men’s Grammatical Luck: Gender-Bias in Part-of-Speech Tagging and Dependency Parsing

July, 2019
Several linguistic studies have shown the prevalence of various lexical and grammatical patterns in texts authored by a person of a …

Peer networks and entrepreneurship: A Pan-African RCT

January, 2019
Can large-scale peer interaction foster entrepreneurship and innovation? We conducted an RCT involving almost 5,000 entrepreneurs from …

Predicting News Headline Popularity with Syntactic and Semantic Knowledge Using Multi-Task Learning

October, 2018
Newspapers need to attract readers with headlines, anticipating their readers’ preferences. These preferences rely on topical, …

Comparing Bayesian Models of Annotation

October, 2018
The analysis of crowdsourced annotations in natural language processing is concerned with identifying (1) gold standard labels, (2) …

Capturing Regional Variation with Distributed Place Representations and Geographic Retrofitting

October, 2018
Dialects are one of the main drivers of language variation, a major challenge for natural language processing tools. In most languages, …

The Social and the Neural Network: How to Make Natural Language Processing about People again

June, 2018
Over the years, natural language processing has increasingly focused on tasks that can be solved by statistical models, but ignored the …