• What kinds of cooperative structures are DNNs inherently capable or incapable of representing? • How do differences in these cooperative structures shape macroscopic behaviors such as generalization, fitting ability, adversarial robustness, and structural modeling capacity? Multi-order interactions for characterizing cooperative structure. These questions call for a mechanistic investigation of interactions inside deep networks, as interactions constitute the most basic units of such cooperative structure. To formally characterize them, we employ the Shapley Interaction Index (SII) I(i, j) [7], * Equal contribution. † Corresponding author.

• Huiqi Deng, Zhuofan Chen, Zhenyuan Cui, Peng Zhang, Hongbin Pei are with Xi’an Jiaotong University, Xi’an, China. • Qihan Ren and Quanshi Zhang are with Shanghai Jiao Tong University, Shanghai, China. • Wen Shen is with Tongji University, Shanghai, China.

which quantifies the nonlinear collaboration between two variables by measuring how the inclusion or exclusion of one variable changes the attributed importance of the other across different contexts (Fig. 1(a)).

While SII captures whether two variables interact, it does not indicate how much contextual information the interaction relies on-an essential dimension of representation complexity. To capture this dimension, we decompose each bivariate interaction I(i, j) into multi-order components I (m) (i, j) [57], where the order m denotes the size of contextual set accompanying (i, j). Each I (m) (i, j) isolates the interaction utility within contexts containing exactly m variables, yielding a stratified view of cooperative structure: loworder interactions reflect simple, weakly contextual patterns; high-order interactions correspond to complex, strongly context-dependent relations; and midorder interactions lie between them (Fig. 1(b)).

Prior work [57] shows that a DNN’s prediction can be represented as a weighted sum over all multi-order interaction utilities across variable pairs. Thus, multiorder interactions provide a natural bridge for connecting microscopic interaction patterns with macroscopic representational behavior such as generalization, adversarial robustness.

Discovering a universal interaction bottleneck. Using this multi-order decomposition, we observe a striking and previously overlooked regularity: across Multi-order interaction: 𝐼 (%) 𝑖, 𝑗 = 𝔼 𝑺 #𝒎 Impor 𝑖 𝑗 present, 𝑆 -Impor(𝑖|𝑗 masked, 𝑆)

Decompose by contextual complexity |𝑺| j Fig. 1. (a) Shapley interaction index and multi-order interactions. The bivariate interaction I(i, j) quantifies the influence of the presence/absence of variable j on the importance of variable i across varying contexts. By decomposing I(i, j) by the contextual complexity, the m-th order interaction I (m) (i, j) represents the average interaction utility within contexts containing exactly m variables (|S| = m). (b) Illustration of low-order and highorder interactions: low-order interactions arise in simple, local contexts, whereas high-order interactions emerge when many contextual regions jointly influence the relationship between i and j. low-order and high-order interactions but exhibit markedly weaker representation of mid-order ones. This interaction bottleneck manifests as a characteristic U-shaped trend in the magnitudes of I (m) (i, j) across interaction orders (Fig. 2(a)).

This bottleneck reveals an inherent representation bias: DNNs easily model simple, local low-order interactions and complex, global high-order interactions, while mid-order interactions, neither clearly local nor fully global, receive much weaker representation.

Proving the interaction bottleneck. To uncover the theoretical origin of this bottleneck, we analyze how the learnability of interactions varies across orders. We prove that the effective gradient signal for I (m) (i, j) is inversely related to its contextual variability n-2 m . Consequently, low-order and high-order interactions arise under limited or nearly fixed contextual configurations, leading to low contextual variability and stable gradient directions, and are therefore readily learned. While mid-order interactions, correspond to maximal contextual variability, causing large gradient variance that hinder stable optimization. The resulting theoretical learning-strength curve closely matches the empirical multi-order interaction distribution (Fig. 2(b)).

Modulating the interaction bottleneck. Given the existence of this interaction bottleneck, a key question is whether DNNs’ interaction distributions can be explicitly modulated. To this end, we introduce two complementary loss functions that respectively encourage or suppress interactions of designated orders. Both theoretical analysis and empirical evidence show that these losses effectively reshape the distribution of multi-order interactions, enabling DNNs to learn representations with targeted interaction distribution.

From micro interactions to macro performance. Using this modulation mechanism, we construct models that predominantly learn low-order, mid-order, or high-order interactions and systematically evaluate their macroscopic representation capacities.

Low-order-biased models exhibit strong generalization and adversarial robustness but limited structural expressiveness and fitting capacity, whereas highorder-biased models offer richer structural modeling and fitting ability but show weaker generalization and robustness. Mid-order-biased models fall between these extremes. Together, these findings reveal a fundamental trade-off governed by interaction complexity: (i) different interaction orders preferentially support different macroscopic capabilities, and (ii) practical DNNs benefit from implicitly combining contributions across orders, thereby achieving a more balanced representational capacity.

Contributions. Our contributions are as follows:

• We discover a universal interaction bottleneck phenomenon across architectures and modalities: DNNs strongly represent low-order and high-order interactions but consistently underrepresent mid-order ones. • We uncover the theoretical origin of this bottleneck by showing that mid-order interactions maximize contextual variability, resulting in maximal gradient variance and reduced learnability. • We introduce two complementary loss functions and demonstrate that they provide explicit, orderlevel modulation of interaction distributions. • We establish a micro-macro connection demonstrating how interaction order governs macroscopic capabilities such as fitting, generalization, robustness, and structural expressiveness.

Research related to our work spans three areas: the representational capacity of DNNs, the role of interactions in understanding model behavior, and the methodological extensions introduced in this journal version beyond our prior conference paper.

Evaluating and interpreting the representation capacity of DNNs has been a long-standing research topic and has attracted extensive attention since the emergence of deep learning.

Early studies primarily focused on expressivity, which refers to the DNN’s ability to approximate complex function distributions. This line of research explored the problem from multiple theoretical perspectives, including analyzing the family of functions that DNNs can represent [1], the maximum number of linear response regions in multilayer perceptrons (MLPs) [23], [26], and the complexity of ReLU transformations [32]. In addition, some studies examined how architectural parameters influence expressivity, revealing that the expressive power of DNNs grows exponentially with network depth [30], where in recurrent neural networks (RNNs), it depends on factors such as recurrent depth, feedforward depth, and skip coefficients [59]. These works advanced the understanding of DNNs’ theoretical capacity, but they only characterized the optimal representation space under idealized assumptions, rather than the actual representation structures that emerge during training.

Beyond expressivity, subsequent studies have shifted toward analyzing generalization and adversarial robustness. Researchers have sought to explain why deep models often achieve remarkably low generalization errors despite heavy overparameterization [2], [52], and why they remain vulnerable to adversarial perturbations [12]. A growing body of evidence indicates that generalization is positively correlated with the flatness of the loss landscape [11], [13], [19], [24]. Furthermore, metrics such as stiffness [6], sensitivity [25], and the CLEVER score [49] have been proposed to quantify generalization and robustness. However, these works still focus predominantly on macroscopic performance indicators, capturing the external behavior of DNNs rather than revealing their internal representation mechanisms.

In summary, most prior studies have characterized the representation capacity of DNNs from a macroscopic perspective, emphasizing theoretical capacity, generalization, and robustness. In contrast, this work re-examines representation capacity from a microscopic viewpoint by modeling how DNN encode interaction structures, investigating how DNNs learn interactions of varying complexity, and establishing a micro-macro connection between interaction complexity and macroscopic performance indicators.

Interactions among input variables encoded by DNNs have received growing attention in neural network interpretability. A variety of metrics have been proposed to characterize how input variables collaborate or depend on each other, including the Shapley Interaction Index [7], Shapley-Taylor Interaction Index [40], multi-order interaction [57], and the Harsanyi dividend [31]. These studies use interactions to explain the internal decision-making behavior of DNNs.

Beyond explanation, interactions have also been used to study how DNNs’ representational capacity relates to interaction strength or interaction complexity. For example, theoretical work interprets dropout as suppressing high-order interactions [18], [57], and interaction metrics have been applied to analyze model generalization [54], [61], adversarial transferability [46], [58], adversarial robustness [33], [45], and improving feature disentanglement [27], [43].

Whereas prior works primarily use interactions to interpret model decisions, we uncover an interaction bottleneck that identifies the interaction structures DNNs are intrinsically able or unable to represent.

A preliminary version of this work appeared at ICLR [3]. While the conference version introduced the interaction bottleneck and provided initial theoretical analysis and empirical evidence, this journal version substantially expands and deepens the study in conceptual framing, theoretical grounding, empirical breadth, and overall presentation. The major extensions are summarized below.

(i) Extensive cross-domain validation of the interaction bottleneck (Section 4). We greatly broaden the empirical evidence for the interaction bottleneck by evaluating Vision Transformers on two datasets with five architectures (2×5), 3D point-cloud models on two datasets with five architectures (2×5), and NLP Transformers on three datasets with four architectures (3×4). We further validate the bottleneck throughout training dynamics, showing that the pattern consistently emerges during learning. This breadth of evidence goes far beyond the ICLR version, which examined only CNNs and MLPs.

(ii) New analysis on fitting vs. generalization (Section 6.3). We extend the empirical study to fundamental macroscopic capacities, investigating how interaction complexity (order) shapes fitting and generalization, which were not examined in the ICLR version. We show that low-order-biased models generalize well but fit poorly, whereas high-order-biased models fit strongly but generalize weakly, revealing an intrinsic trade-off governed by interaction order.

(iii) Introduction of contextual variability as the underlying mechanism (Section 5.2). We introduce the concept of contextual variability, which quantifies how many distinct contextual configurations each interaction order accounts for. This provides the missing physical and intuitive mechanism behind the interaction bottleneck: mid-order interactions have the highest contextual variability (Fig. 6), yielding maximally unstable learning signals. This offers a clear conceptual advance over the ICLR version, which presented algebraic derivations but did not identify the fundamental cause of the bottleneck.

(iv) Conceptual reframing and semantic clarification (Section 3.1 and 3.2). We reorganize and rewrite the introduction to multi-order interactions so that they arise naturally from the Shapley interaction framework, making the formulation more intuitive and theoretically grounded. We further provide a semantic characterization of different interaction orders and clarify the physical meaning of the interaction bottleneck. These conceptual enhancements were ab-TABLE 1 Main notations used in this paper.

Model output on the full input x v(S)

Model output on the masked input x S I(i, j)

Bivariate interaction defined in Eq. ( 2) I (m) (i, j) Multi-order interaction defined in Eq. ( 3)

sent from the ICLR version and substantially improve clarity and interpretability.

(v) Substantial improvements in exposition and organization. Beyond isolated section updates, we restructure the entire paper-refining the motivation, clarifying the problem formulation, strengthening theoretical explanations, adding an overview and illustrative figures, and improving narrative flow. These revisions make the journal version markedly more accessible and coherent than the conference version.

This section introduces the notion of multi-order interactions and explains how different interaction orders characterize distinct forms of variable collaboration. (i) We formally define multi-order interactions, which measure the cooperative effect between two variables under a fixed contextual complexity (Sec. 3.1);

(ii) We then analyze the semantic implications of different interaction orders, clarifying their meanings and functional roles in the decision process (Sec. 3.2).

Understanding DNNs internally organize the relationships among input variables is essential for interpreting their decision processes. This requires moving beyond independent attribution and examining how input variables interact with one another.

Consider a pre-trained DNN v(•) and an input sample x = [x 1 , . . . , x n ] T whose variables are indexed by N = {1, . . . , n} (e.g., an image decomposed into n patches). We denote by v(N ) the network output on the full input x 1 . To formally capture how a

Originally introduced in cooperative game theory [7], the Shapley bivariate interaction index has been adapted to deep learning as a way to quantify how two input variables collaborate. Specifically, an interaction utility I(i, j) between variables i and j measures how the presence of variable j changes i’s contribution to the network output. For instance, I(i, j) = 0.1 indicates that masking variable j reduces the contribution of i by 0.1.

The main challenge lies in how to measure the contribution of i under the presence or absence of j. A naive formulation compares the marginal effect v({i, j})-v({j}) with v({i})-v(∅), where v(S) denotes the model output on a masked input x S with variables in N \ S replaced by a baseline value 2 . However, this approach ignores contextual dependencies: the interaction between i and j may vary substantially depending on other involved contextual variables.

A more accurate measure of the interaction between i and j must account for the fact that their joint effect can vary across different contextual subsets. Thus, for any contextual subset S ⊆ N \ {i, j}, we measure how the marginal contribution of i changes when j is added to the context S:

i’s contribution when j is absent given S

The overall Shapley bivariate interaction I(i, j) is then obtained as the expectation of ∆v(i, j, S) over all contextual subsets S, grouped by their subset size:

Intuitively, I(i, j) captures the average collaboration effect between i and j across different contextual conditions. This pairwise formulation naturally serves as the basis for defining multi-order interactions, introduced next.

To characterize how the interaction utility between variables i and j changes with contextual complexity, Zhang et al. [57] introduced the multi-order interaction I (m) (i, j), which isolates their interaction utility under contexts of fixed size m. Here, the order m reflects the contextual complexity involved in the interaction. Formally, the m-th order interaction is defined as:

The Shapley bivariate interaction index I(i, j) can 2. Typically chosen as the empirical mean of each variable. then be written as the aggregation of multi-order interactions over orders:

As Figure 3 shows, low-order interactions (small m) reflect how variables i, j collaborate under simple, limited contexts; whereas high-order interactions (large m) reflect their joint effect under complex and massive contexts.

The multiorder interaction I (m) (i, j) satisfies five fundamental axioms, including linearity, nullity, commutativity, symmetry, and efficiency, establishing it as a mathematically well-founded measure [56]. (ii) Connections to established metrics: Prior work [33], [57] has shown that multi-order interactions are theoretically linked to several widely used explanation metrics, including the Shapley value [21], [36], Shapley interaction index [7], Harsanyi interaction index [9], [31]. These connections indicate that I (m) (i, j) is consistent with, rather than isolated from, established explanatory principles. Details are provided in the Supplementary Materials. These groundings give I (m) (i, j) a solid theoretical basis, lending credibility to the subsequent analyses.

Building on the formal definition of multi-order interactions, we analyze their semantic roles. For any variable pair (i, j), multi-order interactions characterize how the contribution of (i, j) cooperation evolves as the contextual complexity m varies. Thus, loworder, mid-order, and high-order interactions do not correspond to different variable pairs; rather, they reflect how the same pair (i, j) behaves under minimal, partial, and near-global contextual conditions. (i) Low-order interactions: local semantics under minimal context. Low-order interactions I (m) (i, j) are computed from extremely small contextual sets S (very small |S|). With almost no contextual cues available, the interaction is dominated by the intrinsic cooperative structure of (i, j). As shown in Figure 3, the (i, j) region retains stable, easily recognizable semantics under weak contexts, revealing that low-order interactions encode local, fundamental, and contextindependent semantic cues. (ii) High-order interactions: global semantics under near-complete context. High-order interactions I (m) (i, j) use a nearly complete contextual set (|S| ≈ n). As Figure 3 shows, when global structure becomes available-such as body, pose, and background-it imposes strong semantic constraints that resolve the ambiguity of the local region (i, j) (the bird head). Accordingly, high-order interactions capture global, long-range semantic relations. (iii) Mid-order interactions: intermediate semantics under partial context. Mid-order interactions I (m) (i, j) rely on large yet incomplete contextual sets (intermediate |S|). Such partial contexts introduce more structural cues than low-order settings but still fail to reconstruct the global semantics, placing (i, j) in an intermediate semantic regime. As shown in Figure 3, combinations such as “head + partial body” provide supplementary semantic hints, yet these cues vary markedly across different partial contexts, resulting in high semantic instability. Therefore, midorder interactions reflect partially structured semantic relations whose cooperative effects fluctuate strongly with the specific contextual composition.

Summary. Low-order interactions yield stable local semantics, high-order interactions produce global semantics, and mid-order interactions lie between them and exhibit strong dependence on contextual changes. These three semantic regimes, induced by differing contextual complexities, provide a principled lens for understanding how interaction orders shape the semantic organization of DNNs.

Building on the above formulation, this section (i) establishes that a DNN’s prediction admits an exact decomposition over multi-order interactions, linking microscopic interaction patterns with macroscopic outputs (Sec. 4.1); (ii) presents extensive cross-domain experiments discovering a universal interaction bottleneck: DNNs model low-order and high-order interactions far more strongly than mid-order ones (Sec. 4.2); and (iii) interprets this phenomenon through the intrinsic learning preferences of DNNs (Sec. 4.3).

Prior work [57] has shown that the prediction of a DNN can be decomposed exactly into the collective contributions of multi-order interaction patterns. Formally, by the efficiency property of multi-order interactions, the output on the full input v(N ) admits the following decomposition:

where

captures the individual contribution of variable i, and I (m) = i,j⊆N I (m) (i, j) aggregates the interaction effects of all variable pairs under contextual complexity m, weighted by the coefficient w (m) .

This decomposition establishes a clear micro-macro correspondence: each term I (m) serves as a microscopic reasoning component of order m, and the final prediction is obtained by their compositional integration across all orders. Moreover, macroscopic representational behaviors-such as generalization, robustness-are influenced by how the model organizes interaction contributions across different orders. In this sense, multi-order interactions provide a principled microscopic foundation for understanding the macroscopic representation capabilities of DNNs.

The efficiency property expresses the network output as a sum of interaction contributions across all orders. Building on this decomposition, we examine how strongly a neural network represents interactions at each order and uncover a consistent bias in how these interactions are learned.

Quantifying interaction strength across orders. To compare how strongly a network represents interactions of different complexities, we define the interaction strength at order m for a given sample x as

which summarizes the magnitude of all pairwise interactions at that order. A model-level measure J (m) is obtained by averaging over samples and normalizing across orders:

Here, Ω denotes the set of all input samples. This measure allows us to directly compare the relative representation strength across interaction orders.

Discovering the interaction bottleneck. By examining the distribution of interaction strength J (m) , we uncover a striking and universal phenomenon across a wide range of architectures and tasks: Fig. 4. Distributions of interaction strength J (m) across a wide range of DNNs, datasets, and architectures. All curves consistently exhibit a characteristic interaction bottleneck : DNNs emphasize low-order and high-order interactions while systematically downweighting mid-order ones.

• DNNs consistently encode stronger low-order and high-order interactions, while encode significantly weaker mid-order interactions. As illustrated in Figure 4, the relative interaction strength J (m) typically shows a characteristic U-shape trend: it is high when the interaction order m is either small (e.g., m < 0.1n) or large (e.g., m > 0.9n), but drops substantially around the mid-range (e.g., m ≈ 0.5n). We refer to this distinctive U-shaped distribution as the interaction bottleneck.

We consistently observe this phenomenon across diverse architectures, modalities, and datasets:

• Vision CNNs: five representative CNN models (AlexNet [15], VGG-16 [38], ResNet-18/20/50/56 [10], GoogLeNet [41]) trained on CIFAR-10 [14], Tiny-ImageNet [17], and ImageNet [34].

Transformer models, including ViT [5], DeiT [42], PVT [44], TNT [8], and Swin Transformer [20], trained on the Tiny-ImageNet, and ImageNet. • 3D Point Cloud Models: five widely adopted ar-chitectures, i.e., DGCNN [47], GCNN [37], Pointconv [50], PointNet [28], and PointNet++ [29], trained on the ModelNet10 [51] and ShapeNet [55] benchmarks for 3D shape classification.

• NLP Transformers: four popular Transformerbased NLP models, including BERT [4], Distil-BERT [35], XLnet [53], and ALBERT [16], on SST-2 [39], AG News [60], and CoLA [48].

• Tabular MLPs: two multilayer perceptrons (MLPs) with 5 and 8 layers, trained on the UCI census and UCI commercial datasets. Across all these settings, the interaction bottleneck appears consistently, demonstrating its remarkable generality and robustness.

Moreover, the interaction bottleneck is not exclusive to fully trained models, but rather persists consistently throughout the entire training process. As shown in Figure 5, the characteristic U-shaped distribution of J (m) emerges early (e.g., by epoch 10) and remains highly stable over time, up to final epochs. This suggests that the Strength Fig. 5. The distribution of interaction strength J (m) across different training epochs, showing that the interaction bottleneck persists throughout training.

interaction bottleneck is an intrinsic property of DNN learning dynamics, rather than a byproduct of convergence. Implementation details are provided in the supplementary materials.

The interaction bottleneck reveals an inherent inductive bias in how DNNs utilize interactions of different orders. Specifically, (i) DNNs preferentially capture low-order interactions, which reflect simple and context-light dependencies that generalize reliably across samples. (ii) DNNs also reliably learn high-order interactions, which arise under rich contexts and encode coherent global relationships. (iii) In contrast, DNNs downweight mid-order interactions, whose semantic roles fluctuate across partially informative contexts.

In this section, we uncover the theoretical origin of the universally observed interaction bottleneck from a learning perspective: Why do DNNs reliably learn low-order and high-order interactions, yet systematically underrepresent mid-order ones?

We identify that this bottleneck originates from the systematic differences in contextual variability across interaction orders, with the mid-order interactions exhibiting the highest variability and consequently the greatest learning difficulty.

To theoretically explain the interaction bottleneck, we quantify how strongly a DNN learns interactions of different orders during training.

To this end, we decompose the parameter update in each training step into components associated with interactions of different orders. Specifically, let W ∈ R K denote the parameters of the network. The parameter update ∆W is given by:

where L is the loss and η is the learning rate.

Using the efficiency property in Eq. ( 5), the network output v(N ) can be decomposed into a sum of multiorder interaction terms I (m) (i, j). Consequently, the parameter update ∆W can be decomposed into gradient components associated with these multi-order interactions:

where

with U = v(∅) + i∈N µ i denoting the aggregated independent effect of all input variables. According to the above decomposition, the parameter update ∆W separates into two distinct learning contributions: ∆W U captures the update associated with learning the independent effect U , whereas ∆W (m) (i, j) captures the update associated with learning each multi-order interaction I (m) (i, j) between variable pairs (i, j). Then, we define the learning strength of m-order interactions as the expected L 2 norm of the corresponding gradient component:

A larger F (m) indicates that m-order interactions exert a stronger influence on the parameter update.

In this subsection, we derive a closed-form expression for the learning strength of multi-order interactions and show that it closely follows a characteristic U-shaped curve, thereby theoretically confirming the interaction bottleneck in DNNs. We further demonstrate that this U-shaped behavior arises fundamentally from the uneven distribution of contextual variability across interaction orders.

Definition of contextual variability. For an m-order interaction, the contextual variability is defined as the

High order (Low contextual variability)

Fig. 6. Contextual variability across interaction orders for a fixed variable pair (i, j) (highlighted in red box). Loworder and high-order interactions occur under contexts with limited variation, whereas mid-order interactions arise under diverse and heterogeneous partial contexts (e.g., tail-only regions, partial-body fragments, or background patches), resulting in much higher variability. This disparity in contextual variability explains why DNNs struggle to learn mid-order interactions.

number of possible contextual sets at that order:

This quantity measures how many distinct subsets of the remaining (n -2) variables can serve as context. Because u(m) peaks at m ≈ (n -2)/2, mid-order interactions exhibit the highest contextual variability, whereas low-order and high-order interactions correspond to much lower variability.

Proof of the interaction bottleneck. We now derive a closed-form expression for the learning strength of multi-order interactions, from which the interaction bottleneck follows immediately.

Theorem 1: (Proof in supplementary materials) Assume that for any order m, E i,j E |S|=m [ ∂∆v(i,j,S) ∂W ] = 0 3 . Let σ 2 denote the variance of each coordinate of ∂∆v(i,j,S) ∂W . Under this assumption, the learning strength of order-m interactions satisfies

where C = Kη ∂L ∂v(N ) is a term independent of the order m, and K denotes the dimensionality of W .

Corollary 1: The learning strength of multi-order interactions decreases as contextual variability increases.

Theorem 1 and Corollary 1 show that the learning strength of m-order interactions, F (m) , is inversely proportional to their contextual variability u(m). Consequently, low-order and high-order interactions, where contextual variability is small, receive stronger learning signals, while mid-order interactions-where contextual variability attains its maximum-receive 3. The zero-mean assumption is discussed and empirically validated in the supplementary materials.

substantially weaker learning signals. This inverse dependence yields a characteristic U-shaped learningstrength curve, formally establishing the interaction bottleneck phenomenon.

Intuitive explanation. Fig. 6 provides intuition for why different interaction orders exhibit different levels of learnability by illustrating how contextual variability systematically changes with m, and how this variability affects the stability of the gradients associated with each order.

• Low-order and high-order interactions are evaluated under either very small contexts or nearly complete ones, where the set of admissible conditioning configurations is inherently limited. As shown in Fig. 6, these contexts vary only minimally across conditioning sets, and such stability translates into consistent gradient directions that the model can aggregate reliably. • Mid-order interactions, in contrast, arise under an extremely large number of possible partial contexts, reaching up to n-2 n/2 . As illustrated in Fig. 6, these partial contexts differ widely-highlighting background-only regions, partial body segments, or mixtures of unrelated patches. This high contextual variability leads to highly inconsistent gradient directions across conditioning sets, substantially weakening the learnability of mid-order interactions.

Thus, the interaction bottleneck arises naturally from the uneven distribution of contextual variability across interaction orders: DNNs learn interactions most effectively when their contextual conditions are stable (low orders and high orders) and struggle particularly when the contextual space becomes highly variable (mid orders).

To validate our theory, we use the analytically derived learning strength F (m) to simulate the empirical interaction strengths J (m) observed in real-world DNNs, showing strong agreement between the two.

To enable a fair comparison, we normalize both quantities as F (m) = F (m) /F (0) and Ĵ(m) = J (m) /J (0) , providing a common reference point with F (0) = Ĵ(0) = 1. Moreover, as DNNs usually learn highly redundant feature representations, their effective latent dimensionality n ′ is typically much smaller than the number of input variables n. We thus compute F (m) using n ′ (with n ′ « n), which yields a theory curve that better aligns with the behavior of practical DNNs.

As shown in Figure 7(a), the theoretical curve F (m) closely matches the empirical distribution Ĵ(m) , confirming that the derived theory faithfully captures the interaction bottleneck observed in practice.

Having uncovered and theoretically explained the ubiquitous interaction bottleneck, we now ask whether a model’s learning emphasis across interaction orders can be explicitly controlled. To this end, we introduce two modulation losses (Sec. 6.1) and demonstrate their effectiveness in steering a model toward desired interaction structures (Sec. 6.2). Using these losses, we instantiate three representative DNN variants-favoring low-order, mid-order, and highorder interactions, and examine how these distinct interaction regimes influence macroscopic properties such as generalization, robustness, and structural modeling capability (Sec. 6.3).

We introduce two simple yet effective losses that explicitly modulate the interaction structure-one that encourages interactions within a targeted order range and another that suppresses them.

The key idea is to construct a controllable signal, ∆u(r 1 , r 2 ), which aggregates interactions within the order range [r 1 n, r 2 n]. The encouraging loss forces the model to classify using this signal, whereas the suppressing loss discourages reliance on it.

Core component ∆u(r 1 , r 2 ). To isolate the contribution of specific interaction orders, we define the output-difference term:

where the expectation is taken over subset pairs

Intuitively, the difference between v(S 2 ) and the rescaled v(S 1 ) cancels out all interactions shared by both subsets, leaving only the interactions that are activated when context grows from the size r 1 n to r 2 n. This intuition is formalized in the following theorem. Theorem 2: (Proof in supplementary materials) The output change ∆u(r 1 , r 2 ) can be expressed as follows:

where

Theorem 2 shows that w(m) concentrates around m = r 1 n and vanishes beyond m = r 2 n, meaning that ∆u(r 1 , r 2 ) primarily captures interactions up to order r 2 n, with emphasis around r 1 n (See Figure 7(b)). This provides the theoretical basis for selectively promoting or penalizing specific interaction orders.

Losses for encouraging and suppressing specific interaction orders. To regulate how a DNN utilizes interactions in a chosen order range, we design two complementary losses: an encouraging loss L + (r 1 , r 2 ) and a suppressing loss L -(r 1 , r 2 ).

Encouraging loss. The encouraging loss L + (r 1 , r 2 ) trains the model to perform classification using only the interaction signal contained in the order range [r 1 n, r 2 n], thereby amplifying the learning of interactions within this interval:

For each sample x and class c, we define an interactionbased logit ∆u c (r 1 , r 2 | x) that aggregates only those interaction contributions whose contextual sizes fall within [r 1 n, r 2 n]. Following Eq. ( 14), it is computed by averaging differences of masked-subset logits v c (S|x):

This ∆u c (r 1 , r 2 | x) serves as the effective logit when the model is restricted to the specified interaction interval. The predicted probability P (y

is then obtained by applying a softmax over the classwise values {∆u c (r 1 , r 2 | x)} C c=1 . Suppressing loss. The suppressing loss L -(r 1 , r 2 ) reduces the model’s reliance on interactions within the range [r 1 n, r 2 n] by maximizing the entropy of the prediction based solely on the interaction signal:

The negative entropy term penalizes confident predictions derived from ∆u(r 1 , r 2 )thereby reducing their influence in the learned representation.

Overall loss. By combining the two modulation losses, we can obtain a model biased toward any desired interaction-order distribution.

where L cls is the standard classification cross-entropy, and λ 1 , λ 2 ≥ 0 control the modulation strength.

We now empirically validate the effectiveness of the proposed loss, i.e., they indeed provide fine-grained modulation over interaction orders.

Single loss effect. We first examine the individual effects of the encouraging and suppressing losses. For L + (r 1 , r 2 ), we enable only the encouraging term (λ 1 = 1, λ 2 = 0), and train three AlexNet models on Tiny-ImageNet using (r 1 , r 2 ) ∈ {(0.2, 0.5), (0.3, 0.7), (0.6, 0.9)}. For L -(r 1 , r 2 ), we activate only the suppressing term (λ 1 = 0, λ 2 = 1), and train two additional AlexNet models with (r 1 , r 2 ) = (0, 0.2) and (0, 0.5).

As shown in Fig. 8(b), L + (r 1 , r 2 ) markedly boosts the interaction strength J (m) within the target order interval [r 1 n, r 2 n], whereas L -(r 1 , r 2 ) effectively suppresses interactions in the corresponding range. These results demonstrate that each loss term modulates interaction orders exactly as intended.

Effect of combining multiple losses. We next test whether combining multiple terms L + and L -yields more flexible interaction control. Five configurations are evaluated by training AlexNet models on CIFAR-10, using either pairs of encouraging losses or mixed encouraging-suppressing combinations: (i) L + (0, 0.3) and L + (0.7, 1.0); (ii) L + (0.3, 0.7) and L + (0.7, 1.0); (iii) L + (0.2, 0.4) and L + (0.6, 0.8); (iv) L + (0.6, 1.0) and L -(0, 0.5); (v) L + (0, 0.5) and L -(0.6, 1.0).

The resulting distributions of J (m) are shown in Fig. 8(a). These results demonstrate that combining multiple losses enables substantially more flexible and fine-grained control of interaction orders. For instance, L + (0, 0.3) with L + (0.7, 1.0) boosts both low and high orders, while L + (0.6, 1.0) with L -(0, 0.5) enhances higher orders and suppresses lower ones. Notably, multi-loss configurations can generate interaction distributions unattainable via singleloss setups-for example, combining L + (0.2, 0.4) and L + (0.6, 0.8) produces a clear bimodal pattern, whereas single losses typically yield unimodal curves.

Effect of hyperparameters. Finally, we analyze how λ 1 and λ 2 influence interaction modulation. As shown in Fig. 8(b), increasing λ 1 progressively amplifies interactions within the target order range, while increasing λ 2 produces correspondingly stronger suppression. This confirms that the magnitudes of λ 1 and λ 2 directly govern the intensity of modulation.

In the previous subsection, we introduced modulation losses that allow DNNs to be trained with an intended 2.

Training four representative types of DNNs. To enable a systematic comparison, we construct four representative DNN variants, each biased toward a different interaction regime. A conventionally trained model serves as the baseline, while three additional variants are trained using our modulation losses to predominantly encode low-order, mid-order, or highorder interactions. We refer to these four types as normal DNNs, low-order DNNs, mid-order DNNs, and high-order DNNs. Specifically, the normal DNNs are trained without any modulation term, i.e., λ 1 = λ 2 = 0. The loworder DNNs suppress high-order interactions (within [0.7n, n]) by applying the suppressing loss L -(0.7, 1.0) with λ 1 = 0, λ 2 = 1. The mid-order DNNs enhance interactions in the middle range ([0.3n, 0.7n]) using the encouraging loss L + (0.3, 0.7) with λ 1 = 1, λ 2 = 0. The high-order DNNs suppress low-order interactions (within [0, 0.5n]) by applying L -(0, 0.5) with λ 1 = 0, λ 2 = 1. This setup allows a controlled and direct comparison of DNNs dominated by different interaction orders.

Figure 9 verifies that the four DNN variants indeed acquire the intended interaction structures. Low-order DNNs markedly suppress interactions in [0.7n, n], yielding distributions dominated by low-order terms. Mid-order DNNs strongly amplify interactions in [0.3n, 0.7n], producing a peak centered within this in-terval. High-order DNNs suppress low-order interactions in [0, 0.5n], leading to distributions concentrated at higher orders.

High-order interactions naturally encode global, multi-part dependencies, whereas low-order interactions primarily capture local, part-level cues. Thus, we hypothesize that high-order DNNs should possess the strongest structural modeling ability, while low-order DNNs should rely minimally on global structure. To test this hypothesis, we train VGG-16 and AlexNet on CIFAR-10 and Tiny-ImageNet under each interaction configuration.

To assess structural sensitivity, we design two controlled masking settings (Fig. 10): (i) Random Masking: randomly occludes image regions, disrupting global spatial organization and largely preserving only local, unstructured evidence. (ii) Surrounding Masking: masks only peripheral regions, preserving the central region that maintains the object’s overall contour and global layout.

If a model performs substantially better under surrounding masking than random masking, it indicates stronger reliance on global structural cues. Accordingly, the performance gap between the two settings directly quantifies structural modeling capacity. In Fig. 11, this gap is visualized as the area between the blue (random masking) and orange (surrounding masking) accuracy curves: larger areas correspond to greater dependence on global structure and thus richer structural representations. More experimental results are provided in supplementary materials.

The results reveal three clear trends: (i) Low-order DNNs show minimal structural dependence. Their accuracy under random masking is com-90% 80% 70% 60% 50% 40% 90% 80% 70% 60% 50% 40% masked parable to-or even exceeds-that under surrounding masking, indicating little reliance on global structure. (ii) High-order DNNs encode the richest structural information. They achieve substantially higher accuracy under surrounding masking, confirming that high-order interactions capture global spatial cues. (iii) Mid-order and normal DNNs fall in between. Their accuracy gaps are moderate, reflecting partial but not dominant use of global structural information.

We next examine how different interaction orders affect a model’s fitting and generalization capacity. Across all datasets and architectures, we observe a strikingly consistent pattern: low-order DNNs fit less but generalize better, whereas high-order DNNs fit nearly perfectly but generalize poorly.

To rigorously validate this phenomenon, we conduct controlled experiments using three-layer MLPs trained on the UCI Census and Commercial tabular datasets. For each dataset, we construct four variants-normal, low-order, mid-order, and high-order DNNs-and compare their training-testing dynamics. DNNs lie between the extremes. (iii) Trade-off in test accuracy: Despite the striking differences in fitting and generalization behaviors across the four models, their final test accuracies remain surprisingly close. This occurs because interaction orders drive the model toward opposite inductive biases: high-order interactions amplify fitting but weaken generalization, whereas low-order interactions strengthen generalization but limit fitting. The test accuracy therefore emerges as the equilibrium point of these competing forces, with each model occupying a different position along the same fitting-generalization trade-off.

We further validate these findings on AlexNet and VGG-16 using CIFAR-10 and Tiny-ImageNet. As shown in Table 3, the same pattern robustly reappears across architectures and datasets: low-order DNNs consistently exhibit much higher training losses but the smallest loss gaps, indicating weak fitting yet strong generalization. High-order DNNs show the opposite trend: their training losses are nearly zero but their loss gaps are the largest, reflecting severe overfitting. Taken together, these results identify interaction order as a dominant factor governing the trade-off between fitting and generalization.

We further examine how interaction order affects adversarial robustness. Experiments are conducted on the Census and Commercial tabular datasets using MLP-5 and MLP-8 architectures. For each dataset, we obtain four variants-normal, low-order, mid-order, and high-order DNNs.

Adversarial examples are generated using the untargeted PGD attack [22] under the L ∞ constraint |∆x| ∞ ≤ ϵ. We adopt ϵ = 0.6 (100 steps) for Census and ϵ = 0.2 (50 steps) for Commercial, with step size fixed at 0.01.

Table 4 reports the resulting adversarial accuracies and reveals three consistent observations: (i) Low-order DNNs exhibit the strongest adversarial robustness: They surpass normal DNNs across all settings, showing that suppressing high-order interactions enhances resistance to adversarial perturbations. (ii) High-order DNNs are the most vulnerable. Their adversarial accuracies drop sharply, indicating that reliance on high-order interactions substantially increases attack susceptibility. (iii) Mid-order and normal DNNs lie between these two extremes. Their robustness levels lie between those of low-order and high-order DNNs, aligning with their intermediate structural modeling and generalization behavior.

These results support a key conclusion: adversarial robustness is largely supported by low-order interactions, whereas adversarial vulnerability primarily stems from high-order interactions. This aligns with Ren et al. [33], who observed that adversarial attacks predominantly disrupt high-order interactions.

Taken together, comparative analyses reveal a clear micro-macro linkage: the micro-level interaction patterns that a DNN encode directly shape its macroscopic capabilities, from structural expressiveness to fitting, generalization, and adversarial robustness.

As summarized in Table 2, each interaction regime yields a distinct and complementary capability profile:

• Low-order DNNs rely on simple, localized interactions, which yield strong generalization and adversarial robustness, but limited structural expressiveness and fitting ability. • High-order DNNs depend on complex, contextdependent interactions, and exhibit strong structural expressiveness and fitting capacity, yet make the model fragile, thereby producing poor generalization and weak adversarial robustness. • Mid-order DNNs fall between these two extremes. Their reliance on moderately complex interactions leads to intermediate levels of structure modeling, fitting, generalization, and robustness. Importantly, this interaction-order perspective also explains why standard DNNs work well in practice: although not explicitly optimized for any specific order, they naturally combine the stability of loworder interactions with the expressiveness of highorder ones, leading to a balanced capability profile.

This work establishes interaction complexity as a fundamental perspective for understanding and shaping the behavior of DNNs. We uncover a universal interaction bottleneck shared across architectures and tasks: DNNs naturally prioritize low-order and high-order interactions while systematically underrepresenting mid-order ones. Our theoretical analysis attributes this phenomenon to the high contextual variability of mid-order interactions, which yields unstable gradients and weak learning strength.

To overcome this bottleneck, we introduce two loss functions that explicitly modulate interaction orders. Furthermore, by training models dominated by loworder, mid-order, and high-order interactions, we reveal a clear micro-macro linkage: different interaction regimes give rise to distinct macroscopic capability profiles. Low-order DNNs generalize and defend against adversarial perturbations well but have limited structural modeling and fitting capacity; highorder DNNs excel at structural modeling and fitting but exhibit poor generalization and robustness; midorder DNNs lie between these extremes. Together, these results show that interaction order is a key factor that shapes how a network balances fitting ability, generalization, robustness, and structural modeling.