The Psychological Science Accelerator

SPAML

• Semantic priming has a rich history in cognitive psychology
• Semantic priming occurs when response latencies are facilitated (faster) for related word-pairs than unrelated word-pairs
• Usually measured with the lexical decision or naming task
• The Semantic Priming Project (Hutchison et al., 2013) provided priming values for 1661 English word-pairs

SPAML

• Items are considerably variable … and current sample sizes may not tell us true population scores
  
• This data is only in English

SPAML

• Outcomes:

  • Create an online framework for data collection, modeling after the success of the Small World of Words (De Deyne et al., 2019)
  • Produce a large, multi-linguistic semantic priming dataset complete with other cognitive variables
  • Provide computational packages for researchers to explore and use the datasets, modeled after LexOPS (Taylor et al., 2020)

• Learn more: https://psysciacc.org/psacon2020-videos/

• Join us! Email 007spaml@gmail.com

Adaptive Algorithms for Sampling

• We want to create a large dataset
• But participant attention is a finite resource
• So, we can create a large set of stimuli and sample
• But should all stimuli have the same sample size?

Accuracy in Parameter Estimation

• AIPE: Accuracy in Parameter Estimation

• Sample data until confidence intervals that are “sufficiently narrow”

• Multistep procedure:

  • Define a minimum acceptable sample size
  • Define a stopping rule
  • Define a maximum sample size

• Kelley, 2007; Kelley et al., 2018; Maxwell et al., 2008

Defining “Sufficiently Narrow”

• The English Lexicon Project: lexical decision and naming response latencies for over 40,000 words
• Data provides a good metric for base response latencies for words

##    Trial Type Accuracy  RT      Stimulus  Participant     ZScore
## 3      3    1        1 526 philosophical participant1 -1.0256075
## 5      5    1        1 512     belonging participant1 -1.1129066
## 6      6    1        1 626      lowliest participant1 -0.4020424
## 8      8    1        1 846         yacht participant1  0.9698008
## 9      9    1        1 434       warmish participant1 -1.5992874
## 10    10    1        1 552      splendor participant1 -0.8634806

Defining the Stopping Rule

• Define a stopping rule
• If we want to know when our stimuli is measured “accurately”, we might expect our confidence interval to be small.
• By examining previous response latency data, we can determine what that might look like
• What is the average standard error for our standardized response latencies?

Defining the Stopping Rule

Defining the Minimum Sample Size

• If I assume these data to be representative, what actual sample size might approximate SE = 0.16?
• Simulation of 100 randomly sampled words with sample sizes ranging from 5 to 200.
• Approximately 80% of samples hit the criteria at sample sizes of 25

Defining the Maximum Sample Size

• You could simply define the maximum sample size by time, money, or effort
• You could use a larger power criteria: 90% at n = 35, and 95% at n = 50
• In our study, we used the Semantic Priming Project data to define a maximum sample size based on priming scores (n = 320)

Stimulus Sampling

• Now, we can apply these rules to our study
• We have selected 1000 stimuli, only show each participant a smaller portion of stimuli
• At the start of the study, all stimuli have equal probability of being selected
• After each participant, sample size for each stimulus is calculated
• Once the stimulus has achieved the minimum sample size, calculate standard error
• Continue sampling until the item either achieves the goal standard error or maximum sample size
• Set the probability of selection for that stimulus to floor
• Continue sampling until complete with all stimuli

Study Flow

Conclusions

• Previous data and other large norming studies provide quality estimates for this procedure
• We can collect data on multiple stimuli by optimizing our sample sizes
• By focusing on accuracy in parameter estimation, we provide quality data:
  • That adequately answers hypothesis questions
  • That can be reused
  • That maximizes experimenter/participant time and effort

Questions

• Thank you for listening!
• Interested in the code? Check out https://github.com/SemanticPriming/SPAML
• All PSA collaborators are listed with their author information online