Social inclusion from sensor data

Overview

In inclusive classrooms, the physical presence of children with hearing loss (HL) does not guarantee social inclusion with their typically hearing (TH) peers. Self-report is biased, and manual observation cannot cover a full school day at fine temporal resolution. Inclusion also varies with what the class is doing, and a daily average can hide context-specific effects of opposite sign. The thesis therefore labels every child-minute with an activity context discovered from wearable sensors, and compares HL and TH children within each context.

Data

One inclusive preschool classroom with 13 children (6 HL, 7 TH), recorded over 13 days for about 164 minutes per child per day. Each child wears two UWB tags, giving position and shoulder orientation at 10 Hz, and a LENA recorder whose audio is classified into speaker-type segments. After multimodal fusion, 21,847 attendance-confirmed child-minutes enter the analysis.

Pipeline

UWB (10 Hz) -> displacement, teacher distance, spatial groups (Dominant Sets) --+
                                                                              +-> join per (child, minute)
LENA audio  -> adult words, overlap, utterances per minute ---------------------+
   -> room-level mean (≥ 3 children) -> Yeo-Johnson -> GMM, stability-aware K -> LMM per context

Position is needed to measure co-presence and grouping, and audio to measure vocal participation, so the two streams are joined on (child, minute) as a complete case. LENA segments that straddle a minute boundary are split proportionally, which conserves totals. Displacement is summed per minute rather than converted to a velocity, which would amplify position noise.

Method

Spatial groups

Following an earlier pipeline for the same cohort, each frame gets a pairwise affinity A_ij = exp(−d_ij² / 2σ²) × max(0, u_ij·d_i) × max(0, −u_ij·d_j), the product of a Gaussian proximity kernel (σ = 0.5 m, personal distance scaled to preschoolers) and a mutual-orientation term. The product form means that children who are close but back to back, or facing each other from several metres away, score zero. Groups are extracted as dominant sets of the affinity matrix by replicator dynamics, preferred over partitioning methods such as k-means because groups vary in size and not every child must belong to one. Each group is labelled HL-only, TH-only or mixed.

Context features

Each minute is described by four room-level features averaged over the children present: adult word count, auditory overlap, cumulative displacement and teacher distance. They cover the moderators of communicative demand identified in the literature, are shared by everyone in the room, and exclude the behavioural outcomes to avoid circularity. Time of day is excluded to avoid leaking the routine into the clusters, and minutes with fewer than three children are dropped. Because the features are strongly right-skewed and contain zeros, a per-feature Yeo-Johnson transformation is applied rather than Box-Cox or a fixed log transform.

Context discovery

A full-covariance Gaussian Mixture Model is used rather than k-means because it gives posterior memberships, suited to gradual classroom transitions, and models correlated features. It also has a lower BIC than diagonal and spherical covariance. Hard assignment costs little: the median posterior of the assigned component is 0.90.

Since BIC alone tends to settle on shallow minima that reflect fitting noise, K is selected with a stability-aware rule. For each K from 2 to 20, twenty GMMs are fitted with independent seeds. A K is admissible only if its partitions agree across seeds (mean pairwise adjusted Rand index of at least 0.80), and the admissible K with the lowest mean BIC is chosen. This gives K = 6, and the choice holds for any threshold between 0.70 and 0.85.

Statistical inference

Each child contributes many correlated minutes, so each outcome is modelled with a linear mixed model: cluster, hearing status and their interaction as fixed effects and a random intercept per child, with per-context Wald contrasts between HL and TH children. Random slopes would be under-identified with 13 children and six clusters. The homophily index, one value per child per context, is tested with Mann-Whitney U. Benjamini-Hochberg false discovery rate correction is applied within each marker's six contrasts, as a family-wise correction would cost too much power in an exploratory study of this size.

Results

Recovered activity contexts

The labels are post-hoc descriptions of the sensor profiles. All tests are run on cluster numbers. The contexts are qualitatively consistent with an independent manual coding of the daily routine, consulted only after K was fixed.

Peer co-presence

The difference changes sign between contexts. HL children are in groups more than TH children during independent work (+7.8% of the minute, d = +1.59, q < 0.0001), and less during peer-driven activity (−4.3%, q = 0.02) and seated guided work (−5.3%, q = 0.002). Pooled over the day the two groups look the same (38.8% against 39.9%), because these effects cancel out.

Vocal participation

HL children speak less where fast peer talk dominates and adult scaffolding is reduced: peer-driven activity (q = 0.034) and adult-scaffolded peer activity (q = 0.048). Where adult speech dominates or the room is quiet, the gap is small and not significant.

Peer affiliation

This is the most consistent asymmetry. TH children's grouped time is concentrated in TH-only groups, while HL children spend more grouped time in mixed groups than TH children in all six contexts, significantly in four. Class composition alone gives TH children a homophily advantage of about 0.08. In the four significant contexts the observed gap is 0.12 to 0.19, and it is as large in teacher-organised contexts as in child-chosen ones.

Interpretation

Conditioning on context reveals effects that a whole-day comparison averages away. The per-child random intercept matters for the same reason: a simpler ANOVA would report no effect of hearing status on vocal participation, while the mixed model shows that the signal is concentrated in specific contexts. The results were presented to developmental psychologists.

Key takeaways

• Analyses should be conditioned on context, since effects of opposite sign can cancel out in a daily average.
• Peer affiliation was the most consistent difference, with the same direction in all six contexts.
• Selecting the number of clusters for stability rather than by BIC alone yielded contexts that were reproducible across random seeds.

The recordings of minors were collected under a research-ethics protocol that does not permit public release.