FORK: A Bite-Sized Test Set for Probing Culinary Cultural Biases in Commonsense Reasoning Models
Shramay Palta, Rachel Rudinger
Abstract
It is common sense that one should prefer to eat a salad with a fork rather than with a chainsaw. However, for eating a bowl of rice, the choice between a fork and a pair of chopsticks is culturally relative. We introduce FORK, a small, manually-curated set of CommonsenseQA-style questions for probing cultural biases and assumptions present in commonsense reasoning systems, with a specific focus on food-related customs. We test several CommonsenseQA systems on FORK, and while we see high performance on questions about the US culture, the poor performance of these systems on questions about non-US cultures highlights systematic cultural assumptions aligned with US over non-US cultures.
Topics & Concepts
Fork (system call)Set (abstract data type)Commonsense reasoningTest (biology)Computer scienceTuning forkStyle (visual arts)Commonsense knowledgePsychologyArtificial intelligenceHistoryEcologyArchaeologyBiologyOperating systemVibrationProgramming languageKnowledge-based systemsQuantum mechanicsPhysicsNatural Language Processing TechniquesTopic ModelingAdvanced Text Analysis Techniques