Creating a cross-referenced multi-linguistic dataset to investigate semantic priming

Erin M. Buchanan, Kelly Cuccolo, Savannah Lewis, & The Psychological Science Accelerator

The Psychological Science Accelerator

The PSA is a CERN for psychological science
Globally distributed network of researchers with more than 1000 members in 82 countries
Open science principles and practices
PSA007: Semantic Priming Across Many Languages

Semantic priming has a rich history in cognitive psychology
Semantic priming occurs when response latencies are facilitated (faster) for related word-pairs than unrelated word-pairs
Usually measured with the lexical decision or naming task
The Semantic Priming Project (Hutchison et al., 2013) provided priming values for 1661 English word-pairs

What about other datasets?
What about other languages?
Recent increase in publication of linguistic norms has lead to the availability of these norms for many variables
Multi-linguistic overlap is poor

Outcomes:
- Create an online framework for data collection, modeling after the success of the Small World of Words (De Deyne et al., 2019)
- Produce a large, multi-linguistic semantic priming dataset complete with other cognitive variables
- Provide computational packages for researchers to explore and use the datasets, modeled after LexOPS (Taylor et al., 2020)
Learn more: https://psysciacc.org/psacon2020-videos/
Join us! Email 007spaml@gmail.com

How do you create stimuli for a priming study?
Similarity: shared meaning between concepts
- Defined by face value DOG-CAT versus DOG-SPOON
- Number of shared features using feature production norms
- Association strength using free association norms
- Co-occurrence using computational models and text corpora

Where do you get large, open text corpora that are comparable for calculating similarity?
The Open Subtitles Corpus (http://opus.nlpl.eu/OpenSubtitles-v2018.php) provides linguistic data for 50+ languages
Subtitles have shown to be critically useful datasets for word frequency calculation (New et al., 2007; Brysbaert & New, 2009; Keuleers et al., 2010; Cuetos et al., 2012; Van Heuven et al., 2014; Mandera et al., 2015; and more)
The corpora are freely available to download for use in linguistic projects

First, we decided to select nouns, verbs, adjectives, and adverbs for potential inclusion in the study
Therefore, we needed to be able to use part of speech tagging on our potential languages
udpipe is a language package in R that provides part of speech tagging in many languages

##     token  upos
## 1    This   DET
## 2 package  NOUN
## 3      is   AUX
## 4   great   ADJ
## 5       ! PUNCT

Each language corpus with an available udpipe model was examined for corpus size and the Wikipedia corpus for that language was added to small corpus languages (Afrikaans, Hindi, Armenian, Tamil, Urdu)
All stop words and numbers were removed
All words with less than three characters were removed
The words were filtered for nouns, verbs, adjectives, and adverbs
Using word frequency, the top 10,000 words in each language were selected
Forty-four languages in total are included!

Similarity was calculated by using a FastText model based on the language subtitles and/or Wikipedia combination
- The subs2vec project was used for initial calculations (van Paridon & Thompson, 2021)
- Vector space model of 300 dimensions with 5 count window size
- Other ongoing word is comparing different dimensions and window sizes for models
Cosine is a distance measure of vector similarity, similar to correlation
Top five cosine values for each word were calculated from the FastText model

We then used translation to convert each language’s stimuli back to English
These data were merged together to create a dataset of possible stimuli across all languages
1208416 number of pairs were found across the forty-four languages with an average overlap of 3.23% (2.70 to 70.27)
The pairs were sorted by language overlap to final selection

1000 pairs were selected:
- Each word was only used once
- Words were not different forms of the same word (RUNNING - RUN)
- Limit use of proper names

The overlap between languages is still difficult:
- Mean overlap: 28%
- Average words for translation: 710

Multi-linguistic research is often in a few selected languages
Even with advances in published datasets, cross referencing is difficult
However, by focusing on the available data, we can be guided by the language, rather than selecting English words and simply translating
Join the project! Data collection and translation labs are needed