Methods
Preregistration and ethics
The experimental design was preregistered at https://osf.io/3nygq on December 13, 2023 prior to data collection; and the hypotheses and analyses were preregistered at https://osf.io/8ajsg/overview on April, 11 2025 prior to data analysis, using pilot dyad. We report hypotheses, predictions, outcome variables and analyses as preregistered unless otherwise indicated. All preregistered scripts are available at a frozen repository (https://github.com/sarkadava/FLESH_Effort/tree/preregistered) and changes between the pre-registered and final processing pipelines are documented in a dedicated change-tracking report, which provides a line-by-line comparison of the two versions. Any deviations reflect improvements to data quality and processing robustness rather than changes motivated by the results, and are individually motivated in the Supplementary Material. All participants provided informed consent and were reimbursed by course credit or monetary reward. This study was approved by the Ethics Committee Social Sciences (ECSS) of Radboud University (reference number 22N.004687). All materials, data and code are publicly available on Github, with reproducible analysis code provided in the electronic supplementary material (https://sarkadava.github.io/FLESH_Effort/).
Participants
Participants were 144 native Dutch speakers (mostly bilingual with a high proficiency in English) recruited from a Dutch university for course credit or monetary reward. They were able to sign up as a team or alone through the university recruitment system. We collected data about their personality traits (Denissen et al., 2008), handedness (Oldfield, 1971), and demographics. Additionally, we asked to assess how familiar and comfortable they feel with their experimental partner. In total, we collected data from 72 dyads, a number covering for exclusions (outlined below) and based on pre-registered power analysis.
After exclusion, our pool of participants consisted of 61 dyads (i.e., 122 participants). Of these, 99 were females, 20 were males, 1 non-binary, 1 other and 1 preferred not to say, with a mean age of 19.9 years (SD = 1.9, range = 17–27). The majority were right-handed (108), with 14 left-handed participants. The final count for trials is 1244 for the gesture condition, 1933 for the vocal condition, and 1251 for the multimodal condition.
Data exclusions
Before data processing, we excluded one dyad due to consent withdrawal. During data processing, we excluded nine dyads due to recording errors. Further, following preregistered exclusion criteria, we removed one dyad due to persistent high residual error (larger than 25 mm) in pose estimation. Additionally, we excluded 628 trials that had high error for inverse kinematics (larger than 40 mm). Lastly, we excluded 165 trials in which participants violated the condition rule (e.g., not using both modalities in the combined condition, using sound in the gesture condition, etc.) to ensure that effort is strictly unimodal or multimodal, and 12 trials in which participants used speech for concept explanation.
Go to script: Exclusion I: adherence to condition
Go to script: Exclusion II: speech presence
Materials
Lab equipment
Cameras dedicated for motion tracking with computer vision. For video recording, we use three Elgato Facecam cameras located at a movable arch. The Elgato settings are set at a relatively low resolution of 940x540 to increase mobility of the data, a shutter speed of 1/200s to decrease motion blur increasing motion tracking performance (and an ISO 354, to stabilize contrast due to lower exposure time). We use a custom Python script to capture and write videos, using the opencv and ffmpegcv packages (Kadavá et al., 2024). We write the videos in raw video codec with a frame rate of 60 fps. After each session, we compress each video to the XVID codec. At the beginning of each session, we record 1-minute calibration videos with a checkerboard and charuco board, which will allow us to estimate the intrinsic and extrinsic angles of the cameras needed for 3D pose estimation Matthis et al. (2022). The experimenter is following a protocol for uniform camera calibration (e.g., covering a vertical plane with small circular motions with the outermost corners defined by the edges of the balance board).
Balance board. For postural sway analysis, we use a balance board that has been designed by the Technical Support Group of Donders Institute. The board incorporates sensors adapted from the Wii Balance Board, which have been redesigned. The system is synchronized in time with an accuracy of one millisecond and in space with an accuracy of several submillimeters. An A/D conversion is performed by a National Instruments card, specifically the USB-62221, which is connected to the PC via USB. This card collects four signals at a sampling rate of 400 Hz.
Audio recording. For audio recording, we use a C520 head-mounted condenser microphone, with a low-noise 2-channel DAP PRE-202 microphone amplifier (with +48 V phantom power and low-cut function turned on) of which the gain was set at a constant level (25%) over the entire experiment. The volume level in the computer’s sound panel setting is set to 100. The audio signal was split using an XLR splitter, allowing us to record the audio signal at 16 kHz as an LSL stream transmitted via a Linux-based Minux system and a higher quality 48 kHz via another recording PC using Audacity. The 48 kHz audio recording will be used for extraction of acoustic features (e.g., fundamental frequency, amplitude envelope, spectral flux, etc.) Additionally, the audio signal leading to this PC is split once more, allowing us to connect noise-canceling Sony headphones (WH-1000XM5). These are used by the guesser. The motivation was to make sure that the audio signal from the performer could be smoothly perceived by the guesser as the black curtain separating the two participants hinders the intensity of the audio signal in the room. The guesser also uses a second pair of identical headphones in gesture-only condition: these do not transmit any audio signal as they are intended to ensure that the performances are perceived only via visual modality.
One-way screen. We use a one-way screen that has been designed by the Technical Support Group at the Max Planck Institute for Psycholinguistics. The motivation for the one-sided view is to limit any spill-over feedback or communication from the guesser through gestures or facial expressions. This enables maximal control of the communicative flow and feedback while sustaining the co-presence of two participants in one room.
Experiment software. The experiment is managed by a custom Python script, using the PsychoPy package, and a buttonbox that is implemented in the script with the RuSocSci package. It outputs a CSV file that stores information about accuracy for each trial (i.e., concept).
All specifications regarding laboratory equipment are described in detail in the method preregistration. All scripts for recording are available at https://github.com/sarkadava/FLESH_Effort/tree/main/00_labscripts.
Stimuli
The stimuli were selected from a list of 206 concepts. This list included 100 Leipzig-Jakarta List concepts (Tadmor et al., 2010) reflecting concepts that are common across a large range of languages and are most resistant to borrowing from other languages. Additionally, 100 concepts were included varying in sensory expressibility (Lynott et al., 2020; Lynott & Connell, 2009). In a previous experiment (Ćwiek et al., 2026), 227 native Dutch participants rated the current concepts on a continuous scale for how well they could be communicated without language across three modalities: gesture, vocal, and combined. From the 206-item list, we excluded concepts with any expressibility value below a threshold (mean expressibility – 1 SD) to avoid low-expressible concepts. Using a custom Python script, we constructed three modality-specific 28-item lists of top-ranked expressible concepts, ensuring no overlaps between them. Body-related concepts (e.g., ‘tooth,’ ‘ear,’ ‘tongue’) were excluded to prevent indexical reference to the body itself. They were replaced with taste-related concepts that are generally low in expressibility, aligning with secondary research interests. The final 84-item list maintains expressibility statistics comparable to the original list. The final concepts included nouns, verbs and adjectives. Stimuli lists were pre-randomized to ensure balanced occurrences across sessions (each concept appears at least 10 times, or 12 if additional dyads are included). However, the occurrences are not precisely balanced due to a processing mistake.
Procedure
Participants engaged in a referential, charade-like game in a lab-based setting in which one person (the performer) expresses a concept without using language, and the other (the guesser) guesses its meaning. This paradigm has been widely used to explore the process of grounding the meaning of novel expressions built on various modalities such as drawing (Hawkins et al., 2023), gestures and vocalizations (Ćwiek et al., 2021; Fay et al., 2014; Fay et al., 2022; Macuch Silva et al., 2020), or other, more artificial systems (Selten & Warglien, 2007). Importantly, we adapted the referential game to include feedback. After each guesser’s answer, the performer saw the exact response, and if it did not match the target concept, they had to repair their original performance. Each participant performed novel expressions of 21 concepts, for a total of 42 concepts per dyad, across three within-subject modality conditions: vocal, gesture, and combined (i.e., 7 concepts per condition). This design allowed us to assess how effort is affected by and distributed across modalities. The order of modality conditions was randomized. Each modality was introduced with instructions, followed by two practice concepts and seven trials. After each performance, the guesser typed their answer on a keyboard. If correct, they proceeded to the next concept. If incorrect, both saw the incorrect response, and the performer repeated the concept while the guesser made another attempt. Up to two repair attempts were allowed before moving on. For our purposes, the first performance of a concept established a baseline, and the potential repairs that followed are what we called first and second corrections. Correctness was judged by the experimenter, who checked only for typos; synonyms were not accepted to ensure consistency across sessions. Participants switched roles within and between conditions.
Instructions, stimuli, and feedback were presented on a screen via a custom PsychoPy script. Performers indicated the start and end of each production with a predefined gesture, crossing their arms in front of their body. The guesser was seated at a table with a computer screen to present instructions and feedback. The guessers used a keyboard to type in the answers, which was possible only after the performers indicated the end of a production.
All specifications regarding experimental setup are described in detail in the method preregistration.
Data processing
Pre-processing
Each session produced several recorded streams (i.e., video frame stream, balance board stream, and audio stream), which was read and processed via custom Python scripts. These streams were natively synchronized using LabStreamingLayer (Kothe et al., 2025), which captured all streams in an XDF formatted file. We used buttonbox timestamps to isolate trial segments in each stream and associate them with metadata (e.g., condition, participant number, etc.). For each trial, the output of interest includes a 60 fps video, 48 kHz audio, and balance board data. The audio and video were visually inspected to ensure they start and end when the performer signals the beginning and the end. Trials that started late or ended earlier were adjusted.
Go to script: Pre-Processing I: from XDF to raw files
Go to script: Pre-Processing II: Aligning 16 kHz audio with 48 khZ audio from a second source
Go to script: Pre-Processing III: Correcting trials
Go to script: Pre-Processing IV: Final checks
Motion tracking
We used OpenPose (Cao et al., 2019), specifically its 135-marker model, to obtain a 2D skeleton with 135 body keypoints (per camera), including hands and face, with a sampling rate of 60 Hz.
Go to script: Motion tracking I: Preparation of videos
Go to script: Motion tracking II: 2D pose estimation via OpenPose
To convert multiple 2D data streams into 3D positional data, we used triangulation based on calibrated cameras. Triangulation was performed using Pose2Sim (version 0.10.39) (Pagnon et al., 2022b). To triangulate all 2D skeleton data, we first calibrated the cameras using a checkerboard to determine the intrinsic and extrinsic parameters (Supplementary Material section B, Fig.1-2). For intrinsic calibration, we obtained an error of 0.24 pixels for each camera (recommended below 0.5 pixels). Residual (extrinsic) calibration errors across all cameras were 0.96 cm. The mean reprojection error for all points across all frames and trials was 1.49 cm (values below 1 cm are recommended, but up to 2.5 cm is acceptable). The triangulated data were directly smoothed within the Pose2sim pipeline, which was set as a 4th-order, 10Hz low-pass, zero-phase Butterworth filter; however, this still yielded jittery trajectories, so we smoothed the coordinates further with a Savitzky-Golay 3rd-order polynomial filter with a span of 167 ms which is useful for removing high-frequency jitter. Before obtaining inverse kinematics, we further processed the data to align coordinates with OpenSim requirements (e.g., the feet need to touch the floor).
We further used Pose2sim’s implementation to scale a skeletal model to each participant’s body (i.e., height and weight). The values for weighting the importance of each keypoint were kept at default values. We then used the scaled model to calculate joint angles, i.e., angles between the line of the proximal and distal segment of a joint, for each trial (represented by the coordinates obtained in the previous step).
Joint angles were then used to obtain generalized net joint torques via the OpenSim package (Seth et al., 2018). Joint torque is a measure of rotatory force with which a segment moves, and can be estimated by inverse dynamic analysis. To prevent amplification of noise when solving inverse dynamics, we first smoothed the joint-angle data using a Savitzky-Golay filter with a span of 560 ms and a polynomial order of 3. The average root mean square error was 0.03 m.
Go to script: Motion tracking III: Triangulation via Pose2sim, Inverse kinematics & dynamics via OpenSim
Processing of key time series
Amplitude envelope. We used the high-sampling 48 kHz audio data to extract the amplitude envelope of the acoustic signal. We followed a method by Tilsen and Arvaniti (Tilsen & Arvaniti, 2013), implemented in Python by Pouw (Pouw, 2024). We used a bandpass and a 2nd-order, 10Hz low-pass, zero-phase Butterworth filter. Finally, the data were normalized to the range 0 to 1 within each participant.
Go to script: Processing II: Acoustics
Net joint moment. Joint torque data were smoothed with a Savitzky-Golay filter with a span of 560 ms and polynomial order of 1. Further, we obtained the time derivative of the joint forces, namely, the torque change (in Nm/s). We smoothed the torque change with a Savitzky-Golay filter with a span of 560 ms and polynomial order of 1. To create an aggregated derivative measure for the whole arm, we computed an Euclidean sum over the torque change of all key points belonging to the arm.
Go to script: Processing I: Motion tracking and balance
Balance. We computed the change in 2D magnitude in the center of pressure and smoothed it using the Savitzky-Golay filter with a span of 102 ms and polynomial order of 5.
Go to script: Processing I: Motion tracking and balance
All time series were merged on a common sampling rate of 500 Hz.
Go to script: Processing III: Merging multimodal data
Extraction of key variables
From the processed time series, we derived six effort variables across three channels listed below. To fully capture the effort dynamics, we considered both cumulative and instantaneous effort for all these three measures. Cumulative effort reflects the total exertion over time, which is particularly relevant for sustained actions and overall physical demand. We quantified this by integrating each feature across time, providing a measure of accumulated intensity, torque change, and postural shifts. Instantaneous effort, on the other hand, highlights peak exertion at specific moments. By analyzing peak values of the signals, we can identify the local physical demands on the body. This dual approach allowed us to differentiate between continuous exertion and sudden bursts of effort, offering a more comprehensive perspective on physical cost across communicative attempts.
Amplitude envelope (vocal effort). For vocal effort, we computed the cumulative and instantaneous features of the amplitude envelope as a continuous index of respiratory(-vocal) engagement. We opted for intensity over f0 or formants because non-linguistic vocalizations do not always contain phonation, making spectral measures unreliable across our stimulus set.
Torque change (upper-limb effort). For upper-limb effort, we computed cumulative and instantaneous features of aggregated net joint torque change, which accounts for segment masses the inertial properties of moving limb segments.
Change in center of pressure (postural effort). For postural effort, we computed cumulative and instantaneous features of change in the center of pressure (COPc), capturing fluctuations in postural control, which will be very much driven by upper-limb movement and whole-body engagement when present.
Go to script: Extraction of effort-related features
Degree of misunderstanding. We operationalized a degree of misunderstanding as a semantic distance between the guesser’s answer and the performer’s target concept, computed as the cosine similarity of the ConceptNet word embeddings (Speer et al., 2018). For each target–answer pair from the experiment, we computed cosine similarity using Dutch word embeddings from ConceptNet (numberbatch version 19.08, see preregistration for validation). For pairs not represented in the embeddings (e.g., two-word answers), we conducted an online rating study, collecting data from 16 Dutch native speakers who were asked to rate how similar the word pairs felt.
Go to script: Computing concept similarity using ConceptNet word embeddings
Data analysis
Confirmatory analysis
Confirmatory analyses concern pre-registered analysis and hypotheses. All statistics were performed using R (Team, 2025). We fit Bayesian mixed effects models, using the brms package (Bürkner, 2017), to test two preregistered hypotheses:
H1: Correction recruits more physical effort than the baseline performance.
H2: A higher degree of misunderstanding will require a performer to engage in more effortful correction.
For H1, we fitted six sets of models for the six investigated dependent variables: 1) arm torque integral (cumulative upper-limb effort), 2) envelope integral (cumulative vocal effort), 3) COPc integral (cumulative postural effort), 4) arm torque peak mean (instantenous upper-limb effort), 5) envelope peak mean (instantenous vocal effort), 6) COPc peak mean (instantenous postural effort). The primary predictor of interest was correction. To identify the most appropriate model specification, we fitted four variants for each dependent variable, increasing in complexity.
Model 0 included only the primary predictor as a fixed effect.
Model 1 additionally included covariates: familiarity between the guesser and the performer, extraversion of the performer (BFI), expressibility of the concept, modality, and trial number. These were included not as predictors of primary interest but as theoretically motivated adjustment variables. To guide covariate selection, we constructed a directed acyclic graph (DAG) reported in full in the Supplementary Material (section E). The DAG formalized our assumptions about the causal structure of the data – specifically, which variables influence both the likelihood of receiving a correction and the effort expended during it. A variable that affects both the treatment (correction) and the outcome (effort) without lying on the causal path between them constitutes a confounder; failing to adjust for it would bias estimates of the correction effect. In our DAG, familiarity, extraversion, expressibility, modality, and trial number all plausibly met this criterion: for example, more expressive concepts are easier to communicate and therefore less likely to require correction, while also requiring less physical effort – leaving expressibility uncontrolled would therefore inflate the apparent effect of correction on effort. All fixed effect predictors were pre-registered before data analysis. Model 1 included random intercepts and slopes for all predictors that vary within each grouping factor (participants, dyads, concepts). Although dyad-level random effects were not formally pre-registered, we include them because participants are nested within dyads by design – dyad partners share the same interactional context, and ignoring this clustering would lead to underestimation of uncertainty in fixed-effects estimates.
Model 2 extended Model 1 by estimating the full covariance structure between random intercepts and slopes for each grouping factor.
Model 3 was identical to Model 2 but did not estimate correlations between random effects. This simplification was motivated empirically: in Model 2, most correlation parameters had credible intervals spanning the full [-1, 1] range, indicating the data did not contain sufficient information to estimate them reliably, and their inclusion caused sampling inefficiency (low effective sample sizes).
The reporting model was selected from Models 1–3 based on convergence quality (Rhat < 1.01, bulk ESS > 1000) and, among converged models, predictive performance (ELPD-LOO). All models were fit with weakly informative priors that are unbiased with respect to H0 and H1. All categorical covariates were contrast-coded, and all continuous covariates were z-scored or centered.
Model 4 extended the best converging model from Models 1–3 by adding interaction terms between the primary predictor and theoretically motivated covariates. Specifically, the fixed effects included: correction × modality (testing whether effort escalation at corrections differs by communicative channel), correction × expressibility (testing whether the correction effect depends on concept expressibility), correction × familiarity (testing whether familiar dyads escalate effort differently at corrections), correction × extraversion (testing whether personality modulates communicative effort responsiveness), and modality × expressibility (testing whether the expressibility-effort relationship is modality-specific). These interactions were pre-registered. The random-effects structure retained the specification of the best converging base model. Model 4 was fit for exploratory purposes to identify theoretically motivated modulations of the correction effect, and results are interpreted accordingly.
For H2, we selected the best-performing model from H1 for each dependent variable and extended it with two additional predictors: the cosine similarity of the previous answer to the target concept, and the effort expended in the preceding communicative attempt. Previous effort was included as a statistical control: effort levels may vary across trials due to differences between concepts, dyads, and participants. Including previous effort accounts for this trial-level continuity, ensuring that any effect of answer similarity on current effort reflects genuine calibration to the previous answer rather than between-trial differences in effort. Note that this is a deviation from the preregistered analysis, where we intended to model change in effort directly. However, effort change scores – computed as the difference between consecutive attempts – produced distributions with substantial positive and negative values that proved difficult to model reliably, explaining less than 5% across all six dependent variables. Modeling current effort with previous effort as a covariate recovered the same inferential target while avoiding these estimation problems. The preregistered change-score models are reported in full in the Supplementary Material (section E) for transparency.
Across the four models we fit (Models 1–4), we adopted the model comparison framework articulated in McElreath McElreath (2018): rather than treating model fitting as a selection problem in which one model is chosen and the rest discarded, we treated the set of models as complementary lenses on the same data, each estimating a different combination of parameters. Model 3 served as our primary reporting model, on the grounds described above (best balance of convergence diagnostics and ELPD-LOO among Models 1–3). Estimates of the primary effects were stable across Models 1–4, indicating that our inferences did not depend on a particular random-effects specification. Where parameters of interest were not estimated in Model 3, we drew on the models that did: Model 2 for correlations between random intercepts and slopes, and Model 4 for theoretically motivated interactions.
Exploratory analysis
Beyond the preregistered analyses, we also investigated questions that emerged after fitting the models of primary interest. In the first set of exploratory models, we asked whether cumulative effort can be explained simply by duration – that is, whether more time merely accumulates more movement – or whether it also reflects modulation of amplitude. Trial duration was fitted as one additional predictor for each of the best-performing models from H1 for each cumulative dependent variable.
In the second set of exploratory models, we asked whether the effort dedicated to a correction depends on the participant’s average baseline level of effort at first performance. We fitted the best-performing models from H1 for each dependent variable with an additional predictor: the effort of the first performance, entered in interaction with the correction predictor. This allowed us to test whether correction-related effort escalation is uniform across performers or modulated by their initial effort level.
In the third set of exploratory models, we asked whether effort invested across the correction sequence predicts communicative success. We examined four complementary outcomes: whether misunderstanding was resolved on the first attempt (Bernoulli model), how semantically similar the guesser’s following answer was to the target concept (zero-one inflated Beta model), how quickly understanding was reached across the full sequence (cumulative ordinal model), and whether understanding was ever achieved (Bernoulli model). For each outcome, effort predictors - arm torque, amplitude envelope, and COPc, both cumulative and instantaneous - were entered alongside modality as interaction terms, allowing us to test whether the effort-success relationship differs by communicative channel. All effort predictors were log-transformed and centered on the active-channel mean; in inactive channels, values were attenuated by a constant factor to reflect noise rather than signal.
Go to script: Analysis: Modeling the effect of correction on physical effort