WEIRD ICWSM: How Western, Educated, Industrialized, Rich, and Democratic is Social Computing Research?
Summary
Paper digest
What problem does the paper attempt to solve? Is this a new problem?
The paper aims to investigate the extent to which social computing research focuses on WEIRD populations, which stands for Western, Educated, Industrialized, Rich, and Democratic . This study explores the representation of WEIRD samples in social computing papers, particularly analyzing datasets from ICWSM papers published between 2018 and 2022 . The research questions formulated in the paper address the WEIRD-ness of datasets in social computing papers, the diversity of datasets in poster and dataset papers, and the impact of cross-country authorships on the representation of WEIRD samples . This study sheds light on the under-representation of non-WEIRD populations in social computing research, highlighting the need for broader inclusivity in dataset selection and analysis . The problem addressed in the paper is not entirely new, as it builds upon previous works on WEIRD populations in different research domains like psychology and human-computer interaction .
What scientific hypothesis does this paper seek to validate?
This paper seeks to validate the hypothesis related to the extent of reliance on WEIRD (Western, Educated, Industrialized, Rich, and Democratic) populations in social computing research presented at the ICWSM conference . The study aims to explore the representation of WEIRD samples in research papers, focusing on the diversity of datasets used, the geographic origins of the datasets, and the influence of cross-country authorships on the WEIRD scores of the papers . The research questions formulated in the paper aim to investigate the WEIRD-ness of datasets in social computing papers, the relationship between author affiliations and dataset WEIRD scores, and the impact of cross-country authorships on the WEIRD nature of datasets .
What new ideas, methods, or models does the paper propose? What are the characteristics and advantages compared to previous methods?
The paper "WEIRD ICWSM: How Western, Educated, Industrialized, Rich, and Democratic is Social Computing Research" proposes several new ideas, methods, and models based on its analysis of 420 ICWSM papers published between 2018 and 2022 . Here are some key points from the paper:
-
WEIRD Scores and Metrics: The paper introduces a set of five WEIRD metrics adopted from previous literature and extends them to be applicable for social media data . These metrics are used to assess the representation of WEIRD samples in social computing research, particularly focusing on Western populations .
-
Diversity in Dataset Papers: The analysis reveals that dataset and poster papers in ICWSM tend to draw from populations of less Educated and Democratic countries compared to full research papers . This suggests a need to broaden the diversity of datasets used in social computing research.
-
Responsible AI Statements: The paper emphasizes the importance of incorporating responsible AI statements in research to address potential harms and risks associated with datasets from specific countries . This aligns with the broader context of developing fair, transparent, and accountable AI.
-
Author Diversity and Mentorship Programs: The paper advocates for championing author diversity and implementing shadow mentoring programs to promote collaborations and career development opportunities for underrepresented groups . These initiatives aim to enhance the visibility of authors who may not frequently publish in main conference proceedings.
-
Reducing Reliance on WEIRD Populations: Practical strategies are suggested to reduce the reliance on WEIRD populations in social computing research, including broadening paper checklists, promoting author diversity, and including responsible AI statements . These strategies aim to foster more inclusive research practices.
-
Future Research Directions: The paper acknowledges several limitations that call for future research efforts, such as exploring alternative ways to quantify the Democratic variable and examining broader concepts of inclusivity beyond WEIRD populations . These limitations highlight areas for further investigation and development in the field of social computing research.
Overall, the paper contributes to the discourse on the representation of WEIRD samples in social computing research, proposes strategies for diversifying datasets and authorship, and advocates for responsible and inclusive research practices in the field . The paper "WEIRD ICWSM: How Western, Educated, Industrialized, Rich, and Democratic is Social Computing Research" introduces several characteristics and advantages compared to previous methods based on its analysis of 420 ICWSM papers published between 2018 and 2022 . Here are the key points:
-
WEIRD Metrics Extension: The paper extends a set of five WEIRD metrics from previous literature to be applicable for social media data, allowing for a more comprehensive assessment of the representation of WEIRD samples in social computing research .
-
Dataset Diversity Analysis: Unlike previous analyses on CHI and FAccT, the paper focuses on ICWSM and reveals a surprising trend in the representation of WEIRD samples. It found that only 37% of ICWSM papers were centered on Western populations, indicating a broader range of datasets from both Western and non-Western regions .
-
Focus on Affluent Countries: The research presented at ICWSM tends to examine populations from more affluent, industrialized countries compared to studies in FAccT. Dataset and poster papers in ICWSM scored lower on the "EIRD" metric, suggesting a focus on populations from less Educated and Democratic countries .
-
Author Affiliation Diversity: The paper explores the relationship between the diversity of author affiliations by country and the WEIRD scores. It found a statistically significant negative correlation between the diversity of countries authors are affiliated with and both the Educated and Democratic scores, indicating that papers authored by individuals from multiple countries tend to focus on less Educated or Democratic countries .
-
Practical Strategies for Inclusivity: The paper concludes with practical strategies to reduce reliance on WEIRD populations in social computing research. These strategies include broadening the scope of paper checklists, including responsible AI statements, and promoting author diversity to foster more inclusive research practices .
Overall, the paper's analysis provides valuable insights into the representation of WEIRD samples in social computing research, offers extensions to existing metrics, and suggests practical strategies for enhancing inclusivity and diversity in research practices .
Do any related researches exist? Who are the noteworthy researchers on this topic in this field?What is the key to the solution mentioned in the paper?
Several related research studies exist in the field of social computing research focusing on the representation of WEIRD populations in academic conferences. Noteworthy researchers in this field include Ali Akbar Septiandri, Marios Constantinides, and Daniele Quercia from Nokia Bell Labs . Other researchers who have contributed to this area of study include Nourian, Shinohara, Tigwell, Oleson, Xie, Salac, Everson, Kivuva, Ko, Oliveira, de Melo, Amaral, Pinho, Olteanu, Castillo, Diaz, Kıcıman, van Berkel, Sarsenbayeva, Goncalves, Wilkinson, Knijnenburg, Kennedy, Muller, Stumpf, Liao, Baeza-Yates, Aroyo, Holbrook, Luger, Madaio, Blumenfeld, De-Arteaga, Vitak, and many others .
The key solution mentioned in the paper involves evaluating the dependence on WEIRD populations in research presented at the AAAI ICWSM conference. The study analyzed 494 papers published from 2018 to 2022, focusing on full research papers, dataset papers, and posters. The research aimed to adapt existing WEIRD metrics to be applicable for social media data and found that 37% of the papers focused solely on data from Western countries. This percentage is significantly less than those observed in research from other conferences, indicating a greater diversity of dataset origins within ICWSM. However, the studies at ICWSM still predominantly examine populations from countries that are more Educated, Industrialized, and Rich compared to those in other conferences, with a special note on the 'Democratic' variable reflecting political freedoms and rights .
How were the experiments in the paper designed?
The experiments in the paper were designed as follows:
- Participants were presented with five papers on a custom-designed HTML page, including research papers and a Quality Control (QC) paper to assess the accuracy of their annotations .
- Two basic attention checks were included where participants had to choose 'Casablanca' as their favorite city and 'Colombia' as their favorite country to pass the checks .
- The task involved two steps: first, participants familiarized themselves with the task using example papers, and second, they pinpointed and extracted specific variables from the papers .
- The estimated time for task completion was around 23 minutes, with participants compensated at a rate of £9 per hour .
- A crowdsourcing study with 188 participants was conducted to extract information from the papers for computing the WEIRD scores, ensuring data quality by extracting variables twice .
- The study analyzed 420 ICWSM papers published between 2018 and 2022, focusing on the representation of WEIRD samples in social computing research .
- The research questions formulated for the study included assessing the WEIRD-ness of datasets in social computing papers and exploring the impact of cross-country authorships on dataset diversity .
- The study collected and analyzed papers from the ICWSM conference, including full research papers, dataset papers, and posters, to calculate the WEIRD scores .
What is the dataset used for quantitative evaluation? Is the code open source?
The dataset used for quantitative evaluation in the study is a collection of 494 ICWSM papers published between 2018 and 2022, including full research papers, dataset papers, and posters . The code used for the evaluation is not explicitly mentioned as open source in the provided context. However, the study conducted a crowdsourcing study with manual validation to extract information from the papers for computing the WEIRD scores .
Do the experiments and results in the paper provide good support for the scientific hypotheses that need to be verified? Please analyze.
The experiments and results presented in the paper provide strong support for the scientific hypotheses that needed verification. The study conducted a chi-squared contingency test to explore the relationship between the Western score and the type of paper, finding no statistically significant difference in the Western-centric focus of studies across different paper types . Additionally, the study examined the relationship between the diversity of author affiliations by country and the WEIRD scores, revealing a statistically significant negative correlation between the diversity of countries authors are affiliated with and both the Educated and Democratic scores . These findings indicate a clear link between author diversity and the focus on less Educated or Democratic countries in research papers.
Furthermore, the study analyzed 494 papers from the ICWSM conference and computed a set of five WEIRD metrics to assess the representation of Western, Educated, Industrialized, Rich, and Democratic countries in social computing research . The analysis revealed that papers with cross-country authorship tend to focus on samples from countries with lower levels of democracy, highlighting the impact of author diversity on research focus . This aligns with the hypothesis that cross-country authorships are associated with datasets from less WEIRD countries, providing empirical evidence to support this hypothesis.
Overall, the experiments and results presented in the paper offer robust support for the scientific hypotheses that needed verification. The study's methodology, including crowdsourcing data collection and statistical analyses, effectively examined the relationship between author diversity, country affiliations, and the focus on WEIRD populations in social computing research . The findings contribute valuable insights into the impact of author diversity on research perspectives and the representation of diverse populations in academic studies.
What are the contributions of this paper?
The paper makes three main contributions:
- Collection and analysis of 494 ICWSM papers published between 2018 and 2022, including full research papers, dataset papers, and posters, with a focus on WEIRD populations .
- Computation of WEIRD metrics from previous literature and their extension to be applicable for social media data, revealing that ICWSM research tends to include datasets from more Educated, Industrialized, and Rich countries compared to FAccT studies .
- Identification of practical strategies to reduce reliance on WEIRD populations, such as broadening paper checklists, including responsible AI statements, promoting author diversity, and implementing shadow mentoring programs .
What work can be continued in depth?
Further research in the field of social computing can be continued in depth by addressing several key areas highlighted in the study:
- Diversifying Research Perspectives: Future studies could focus on understanding the extent to which published papers across scientific communities rely on WEIRD samples and emphasize the importance of diversifying research perspectives .
- Reducing Reliance on WEIRD Populations: Initiatives can be developed to reduce the reliance on WEIRD populations in social computing research. This can include broadening the scope of paper checklists, incorporating responsible AI statements, and promoting author diversity .
- Exploring Alternative Concepts of Inclusivity: Future endeavors should examine alternative concepts such as the WILD (Worldwide, In-situ, Local, and Diverse) to embrace a broader spectrum of inclusivity beyond the WEIRD framework .
- Studying Cultural Differences: Research can delve deeper into exploring cultural differences in product design, minority views in computing education, and digital accessibility in various regions to ensure a more inclusive approach in social computing research .
- Enhancing Author Diversity and Mentorship: Championing author diversity and implementing shadow mentoring programs can help promote collaborations, diversity among researchers, and provide career development opportunities for underrepresented groups in social computing research .
By focusing on these areas, researchers can contribute to a more inclusive, diverse, and comprehensive understanding of social computing that goes beyond the limitations of WEIRD populations.