Until recently, the dominant paradigm in natural language processing (and other areas of artificial intelligence) has been to resolve observed label disagreement into a single “ground truth” or “gold standard” via aggregation, adjudication, or statistical means. However, in recent years, the field has increasingly focused on subjective tasks, such as abuse detection or quality estimation, in which multiple points of view may be equally valid, and a unique ‘ground truth’ label may not exist. At the same time, as concerns have been raised about bias and fairness in AI, it has become increasingly apparent that an approach which assumes a single “ground truth” can erase minority voices.
Strong perspectivism in NLP (Basile et al., 2021a) pursues the spirit of recent initiatives such as Data Statements (Bender and Friedman, 2018), extending their scope to the full NLP pipeline, including the aspects related to modelling, evaluation and explanation.
The “Perspectivist Approaches to Disagreement in NLP” (NLPerspectives) workshop will explore current and ongoing work on the collection and labelling of non-aggregated datasets, and approaches to modelling and including these perspectives, as well as evaluation and applications of multi-perspective Machine Learning models. We also welcome opinion pieces and literature reviews, e.g. in the context of fairness and inclusion.
A key outcome of the workshop will be to build on the work begun at https://pdai.info/ to create a repository of perspectivist datasets with non-aggregated labels for use by researchers in perspectivist NLP modelling.
Authors are therefore invited to share their LRs. In particular, when submitting a paper from the START page, authors will be asked to provide essential information about resources (in a broad sense, i.e. also technologies, standards, evaluation kits, etc.) that have been used for the work described in the paper or are a new result of your research.
Moreover, ELRA encourages all LREC authors to share the described LRs (data, tools, services, etc.) to enable their reuse and replicability of experiments (including evaluation ones).
Submissions
We accept two types of submissions, long papers and short papers (e.g., demonstration papers, dataset and resource papers, short focused contributions), all following the LREC template: https://lrec2022.lrec-conf.org/en/submission2022/authors-kit/
Long papers may consist of up to 8 pages of content, plus unlimited references. Short papers may consist of up to 4 pages of content; shorter versions are also welcome. Final versions will be given one additional page of content so that reviewers’ comments can be taken into account. Submissions should be sent in electronic forms, using the Softconf START conference management system. The submission site is at this link:
https://www.softconf.com/lrec2022/NLPerspectives/user/scmd.cgi?scmd=submitPaperCustom&pageid=0
We invite original research papers from a wide range of topics, including but not limited to:
- Non-aggregated data collection and annotation frameworks
- Descriptions of corpora collected under the perspectivist paradigm
- Multi-perspective Modelling and Machine Learning
- Evaluation of multi-perspective models/ models of disagreement
- Multi-perspective disagreement as applied to NLP evaluation
- Fairness and inclusive modelling
- Applications of multi-perspective modelling
- Computing with (dis)agreement
- Perspectivist Natural Language Generation
- Foundational aspects of perspectivism
- Opinion pieces and reviews on perspectivist approaches to NLP
Submissions are open to all, and are to be submitted anonymously. All papers will be refereed through a double-blind peer review process by at least three reviewers with final acceptance decisions made by the workshop organizers.
NEW The proceedings of NLPerspectives will be published on the LREC 2022 website. We will be flexible in order to accommodate requests, e.g. for non-archival submission.
The workshop is scheduled for June 20th afternoon. Contact us at g.abercrombie@hw.ac.uk if you have any questions.
Important Dates
- Friday April 15, 2022: Paper submission
- Friday May 6, 2022: Notification of acceptance
- Friday May 20, 2022: Camera-ready papers due
- Monday June 20, 2022: Workshop