Introduction

Imbalanced datasets are a pervasive challenge in machine learning and statistical analysis, where one class (the minority class) is significantly underrepresented compared to others (the majority class). The imbalance often leads to biased predictive models that perform poorly on the minority class , which may represent critical cases such as fault detection , medical diagnoses , network security , or computer vision . To mitigate this issue, various data augmentation techniques have been developed to generate synthetic samples of the minority class, thereby balancing the dataset and improving model performance.

One of the most widely used augmentation methods is the Synthetic Minority Over-sampling Technique (SMOTE), introduced by Chawla et al. . SMOTE generates new synthetic samples by interpolating between existing minority class instances and their nearest neighbors. Specifically, for a given minority class sample, synthetic points are created along the line segments joining it with its $`k`$-nearest neighbors. This approach has been empirically successful and has been integrated into numerous applications and extended in various ways .

Despite its widespread adoption and a plethora of empirical studies that analyze its properties, the theoretical aspects of SMOTE remain relatively unexplored. In particular, there is limited understanding of how the synthetic samples generated by SMOTE relate to the original data distribution as the sample size increases. The question of convergence is central to understanding the effectiveness of the SMOTE algorithm. The gap in the literature is significant because a theoretical foundation would provide insights into the algorithm’s effectiveness, inform parameter selection, and guide the development of improved augmentation methods.

In this paper, we fill the gap in analytical framework by providing a theoretical analysis of the convergence of SMOTE-generated samples to the original data distribution. Our study considers a continuous random variable $`X`$ and examines the behavior of the synthetic random variable $`Z`$, which is generated from an independent and identically distributed (i.i.d.) sample of $`X`$.

Our main contributions are as follows:

1. Convergence in Probability: We prove that the synthetic random variable $`Z`$ converges in probability to the original random variable $`X`$ as the sample size $`n`$ approaches infinity (Theorem 1). This result establishes that, with sufficiently large samples, the distribution of SMOTE-generated samples approximates the original data distribution.

2. Impact of Nearest Neighbor Rank $`k`$: We analyze the effect of the choice of $`k`$ on the convergence rate of $`Z`$ to $`X`$. Our analysis reveals that selecting lower values of $`k`$ leads to faster convergence. Specifically, we demonstrate through Equation [compk] that the expected distance between a sample point and its $`k`$-th nearest neighbor increases with $`k`$. Consequently, synthetic samples generated using higher values of $`k`$ are more likely to be farther from the original data points, potentially slowing down convergence.

3. Empirical Validation: We support our theoretical findings with simulation studies using uniform, Gaussian, and exponential distributions. The simulations illustrate the convergence behavior of $`Z`$ for different values of $`k`$ and sample sizes $`n`$ (Figures 5–8). We observe that lower values of $`k`$ result in faster convergence of $`Z`$ to the distribution of $`X`$, as measured by the Kullback-Leibler (KL) divergence (Figure 12).

4. Convergence in Mean: Under the additional assumption that $`X`$ has compact support, we establish that $`Z`$ converges to $`X`$ in mean as $`n \rightarrow \infty`$ (Theorem 3). This stronger form of convergence provides a deeper understanding of the relationship between the synthetic and original data distributions.

Our results have practical implications for the application of SMOTE in data augmentation. They suggest that using the nearest neighbor ($`k = 1`$) in the SMOTE algorithm is preferable for achieving faster convergence to the original distribution. This insight can guide practitioners in selecting appropriate parameters for SMOTE and potentially improve the performance of models trained on augmented data.

The remainder of the paper is organized as follows. In Section 3, we present the main theoretical results, including proofs of convergence in probability and mean. Section 4 details the simulation studies that corroborate our theoretical findings and explore the effects of different choices of $`k`$ and sample sizes $`n`$. Finally, in Section 5, we discuss the implications of our work and suggest directions for future research.

We believe that this work contributes significantly to the theoretical understanding of SMOTE and opens avenues for further exploration of data augmentation techniques in machine learning.

Background

Handling imbalanced data remains a dynamic area of research. Although numerous sampling algorithms have been developed , the SMOTE algorithm is arguably the most widely recognized method in the literature. Initially introduced by Chawla et al. in 2002 , SMOTE has been successfully applied across a variety of applications. There are numerous extensions of SMOTE that aim to enhance the original algorithm’s sampling performance. For example, the Adaptive Synthetic (ADASYN) algorithm is similar to SMOTE but varies the number of generated samples based on the estimated local distribution of the class being oversampled. Another widely used variant, Borderline SMOTE , identifies borderline samples and utilizes them to create new synthetic instances. Additionally, SVM-SMOTE was developed in by incorporating support vector machines into the SMOTE framework. More recently, several other extensions have been proposed, including Center Point SMOTE, Inner and Outer SMOTE , DeepSMOTE , and Deep Attention SMOTE , among others, further expanding the applicability and effectiveness of SMOTE in various contexts.

Despite its popularity and wide-spread use, there exist very few studies that analyze theoretical properties of SMOTE. In early attempt to understand theoretical aspects of sampling strategies, King and Zeng studied the random undersampling strategy in the context of the logistic regression. In , Elreedy and Atiya derived the expectation and covariance matrix of the SMOTE generated patterns. More recently, Elreedy et al. developed the theoretical distribution of $`Z`$ described in Equation [eq:elreedy]. While enlightening, the expression in Equation [eq:elreedy] is not practical as it is expressed in the form of a double integral. In , Sakho et al. proposed an alternative derivation of Equation [eq:elreedy] using random variables instead of geometrical arguments. Sakho et al. also studied the convergence of $`Z`$ to $`X`$ and showed that if $`X`$ has a bounded support, then $`Z_{K,n}\mid X_c = x_c\to x_c`$ in probability (Theorem 3.3 ). In , the authors improve the convergence result of by showing that $`Z`$ converges to $`X`$ conditionally in mean, when $`X`$ is left-bounded.

Our research advances existing literature in two significant ways. First, we demonstrate that $`Z`$ converges to $`X`$ in probability without requiring any assumptions about the boundedness of the support. Second, we show that when the support is bounded, $`Z`$ converges to $`X`$ in mean, which is a stronger form of convergence than convergence in probability.

Main Results

Let $`X`$ be a real-valued random variable with cumulative distribution function $`F`$ and probability density function $`f`$. Consider an independent and identically distributed (i.i.d.) sample $`X_1, X_2, \dots, X_n`$ drawn from $`X`$. Let $`Z`$ be the random variable generated via the SMOTE-$`k`$ procedure, as described in Algorithm 1. We will show that $`Z`$ converges to $`X`$ in probability as $`n\rightarrow \infty`$.

We note that there exists a more general version of the SMOTE-$`k`$ algorithm which will be discussed in Section 3.1. Specifically, all the results established for the SMOTE-$`k`$ algorithm are also applicable to this broader algorithm.

\quad \lim_{n \to \infty} P\left( \left| Z - X_1 \right| > \epsilon \right) = 0, \,\forall \epsilon > 0.

P\left( \left| Z - X_1 \right| > \epsilon \right) \leq P\left( \left| X_{1,(k)} - X_1 \right| > \epsilon \right).

\quad \lim_{n \to \infty} P\left( \left| X_{1,(k)} - X_1 \right| > \epsilon \right) = 0,  \,\forall \epsilon > 0.

D_i = |X_i - X_1|.

D_{(1)} \leq D_{(2)} \leq \dots \leq D_{(n - 1)},

Y = \sum_{i=2}^{n} \mathbf{1}_{\{ D_i \leq \varepsilon \}},

p(x_1) = P(D_i \leq \varepsilon \mid X_1 = x_1) = \int_{x_1 - \varepsilon}^{x_1 + \varepsilon} f(x) \, dx.

Y \mid X_1 = x_1 \sim \text{Binomial}(n - 1, p(x_1)).

\begin{equation}
\label{distk}
P(D_{(k)} \geq \varepsilon \mid X_1) =  P(Y \leq k - 1 \mid X_1).
\end{equation}

\begin{equation}
\label{uncond}
P(D_{(k)} \geq \varepsilon) = E_{X_1}\left[ P(Y \leq k - 1 \mid X_1) \right].
\end{equation}

\begin{equation}
\label{bound}
\int_{S^c} f(x_1) \, dx_1 < \delta.
\end{equation}

\begin{equation}
\label{comp}
 p(x_1) \geq p_{\min}, \,\forall x_1 \in S.
\end{equation}

\begin{equation}
\label{break}
P(D_{(k)} \geq \varepsilon) = P(D_{(k)} \geq \varepsilon, X_1 \in S) + P(D_{(k)} \geq \varepsilon, X_1 \in S^c).
\end{equation}

\begin{aligned}
&P(D_{(k)} \geq \varepsilon, X_1 \in S) = E_{X_1}\left[ P(Y \leq k - 1 \mid X_1  \in S) \right]\\
&= \int_{S}  P(Y \leq k - 1) f(x_1) \, dx_1\\
&=\sum_{j=0}^{k-1} \int_{S} \binom{n-1}{j} [p(x_1)]^j [1 - p(x_1)]^{n-1-j} f(x_1) \, dx_1\\
&\leq\sum_{j=0}^{k-1} (n-1)^{j} [1 -  p_{\min}]^{n-1-j} \cdot \int_{S} [p(x_1)]^j  f(x_1) \, dx_1\\
&\leq \sum_{j=0}^{k-1} (n-1)^{j} [1 -  p_{\min}]^{n-1-j}
\end{aligned}

\lim_{n\to \infty} \sum_{j=0}^{k-1} (n-1)^{j} [1 -  f_{\min}]^{n-1-j}=0.

\begin{align*}
  P(D_{(k)} \geq \varepsilon, X_1 \in S^c) &\leq P(X_1 \in S^c)\\
  &= \int_{S^c} f(x_1) \, dx_1 < \delta.
\end{align*}

\lim_{n \to \infty} P(D_{(k)} \geq \varepsilon) = 0.

P(|Z - X_1| > \varepsilon) \leq P(D_{(k)} \geq \varepsilon) \to 0 \quad \text{as} \quad n \to \infty.

\lim_{n \to \infty} P(|Z - X| > \varepsilon) = 0.

To validate Theorem 1 on real-world data, we analyzed carbon monoxide readings from the UCI Air Quality dataset , treating the full series as the ground truth $`X`$. We generated synthetic samples $`Z`$ from subsets of varying sizes $`n`$ using neighbor ranks $`k=1`$ and $`k=5`$, measuring the Kolmogorov-Smirnov (KS) distance to $`X`$. As shown in Figure 1, the error decreases monotonically as $`n`$ increases, confirming convergence in probability. Moreover, $`k=1`$ yields consistently lower KS statistics than $`k=5`$ at smaller sample sizes, empirically supporting our theoretical conclusion that lower neighbor ranks lead to faster convergence.

In the next theorem, we consider a continuous random variable with compact support. We show that the expected difference between any two consecutive order statistics tends to zero uniformly as the sample size increases. As a consequence (Theorem 3), we will obtain that $`Z`$ converges to $`X`$ in mean. Before proceeding further, we need the following auxiliary lemma whose proof is trivial.

E[X] = \sum_y E[X \mid Y = y] \cdot f_Y(y).
```*

</div>

<div id="meanConverge" class="theorem">

**Theorem 2**. *Let $`X`$ be a continuous random variable with finite
support $`[a, b]`$. Let $`X_1, X_2, \dots, X_n`$ be independent and
identically distributed (i.i.d.) samples from $`X`$, and let
$`X_{(1)} \leq X_{(2)} \leq \dots \leq X_{(n)}`$ denote their order
statistics. Then $`\lim_{n\to\infty}E[X_{(k+1)} - X_{(k)}] =0`$
uniformly.*

</div>

<div class="proof">

*Proof.* Assume without the loss of generality that the cumulative
distribution function $`F`$ is continuous and strictly increasing on
$`[a, b]`$. Our goal is to show that for any $`\varepsilon > 0`$, there
exists an $`N \in \mathbb{N}`$ such that for all $`n > N`$ and all
$`1 \leq k \leq n - 1`$,
``` math
E[X_{(k+1)} - X_{(k)}] < \varepsilon.

U_i = F(X_i), \quad \text{for } i = 1, 2, \dots, n.

\begin{equation}
\label{expectedDiff}
E[U_{(k+1)} - U_{(k)}] = \frac{1}{n + 1}, \quad \text{for } k = 1, 2, \dots, n - 1.
\end{equation}

|u - v| < \delta \implies |F^{-1}(u) - F^{-1}(v)| < \varepsilon.

\begin{align*}
E[X_{(k+1)} - X_{(k)}] &= E\left[ (X_{(k+1)} - X_{(k)}) \cdot \mathbb{I}\left( U_{(k+1)} - U_{(k)} < \delta \right) \right] \\
&\quad + E\left[ (X_{(k+1)} - X_{(k)}) \cdot \mathbb{I}\left( U_{(k+1)} - U_{(k)} \geq \delta \right) \right],
\end{align*}

|X_{(k+1)} - X_{(k)}| = |F^{-1}(U_{(k+1)}) - F^{-1}(U_{(k)})| < \varepsilon.

(X_{(k+1)} - X_{(k)}) \cdot \mathbb{I}\left( U_{(k+1)} - U_{(k)} < \delta \right) \leq \varepsilon \cdot \mathbb{I}\left( U_{(k+1)} - U_{(k)} < \delta \right).

\begin{align*}
    &E\left[ (X_{(k+1)} - X_{(k)}) \cdot \mathbb{I}\left( U_{(k+1)} - U_{(k)} < \delta \right) \right]\\
    &\leq \varepsilon \cdot P\left( U_{(k+1)} - U_{(k)} < \delta \right)\\
    &\leq \varepsilon.
\end{align*}

0 \leq X_{(k+1)} - X_{(k)} \leq b - a.

\begin{align*}
    &E\left[ (X_{(k+1)} - X_{(k)}) \cdot \mathbb{I}\left( U_{(k+1)} - U_{(k)} \geq \delta \right) \right]\\
    &\leq (b - a) \cdot P\left( U_{(k+1)} - U_{(k)} \geq \delta \right).
\end{align*}

P(X \geq t) \leq \frac{E[X]}{t}.

P\left( U_{(k+1)} - U_{(k)} \geq \delta \right) \leq \frac{E[U_{(k+1)} - U_{(k)}]}{\delta}.

P\left( U_{(k+1)} - U_{(k)} \geq \delta \right) \leq \frac{1}{n \delta}.

\frac{1}{n \delta} \leq \frac{\varepsilon}{2(b - a)}.

(b - a) \cdot P\left( U_{(k+1)} - U_{(k)} \geq \delta \right) \leq \frac{\varepsilon}{2}.

\begin{align*}
&E[X_{(k+1)} - X_{(k)}]\\
&= E\left[ (X_{(k+1)} - X_{(k)}) \cdot \mathbb{I}\left( U_{(k+1)} -
 U_{(k)} < \delta \right) \right] \\
&\quad + E\left[ (X_{(k+1)} - X_{(k)}) \cdot \mathbb{I}\left( U_{(k+1)} - U_{(k)} \geq \delta \right) \right] \\
&\leq \varepsilon + (b - a) \cdot \frac{\varepsilon}{2(b - a)} = \varepsilon + \frac{\varepsilon}{2} = \frac{3\varepsilon}{2},
\end{align*}

We can generalize Theorem 2 to the expected difference between any order statistics.

E[X_{(k+1)} - X_{(k)}] < \frac{\varepsilon}{m}.

\begin{align*}
E[X_{(k+m)} - X_{(k)}] &=\sum_{i=k}^{m-1} E[X_{(i+1)} - X_{(i)}]\\
&\leq m\cdot  \frac{\varepsilon}{m} = \varepsilon.
\end{align*}

In Theorem 1, we showed that $`X`$ converges to $`Z`$ in probability. Now, under additional condition of bounded support of $`X`$, we obtain a stronger result that $`X`$ converges to $`Z`$ in mean.

\lim_{n\to\infty}E\left[ \left|Z - X_{1}\right| \right] =0.

E[X_{(k+m)} - X_{(k)}] < \varepsilon.

\begin{align*}
E[|Z - X_{1}|] &\leq E[|X_{1,(m)} - X_{1}|]\\
&\leq \sup_{1\leq k \leq n-m} E[X_{(k+m)} - X_{(k)}] \\
&\leq \varepsilon
\end{align*}

To verify Theorem 3, we analyzed the “median income” feature from the California Housing dataset . The data is normalized to $`[0, 1]`$ to satisfy the assumption of compact support. We calculated the Wasserstein distance as suitable proxy for convergence in mean between the synthetic samples $`Z`$ and the true population $`X`$. As illustrated in Figure 2, the distance decreases monotonically as the sample size $`n`$ increases. It empirically confirms that the synthetic distribution approaches the original distribution in mean.

General SMOTE

In this section, we broaden our main results related to converegence to a more general version of SMOTE. The general SMOTE algorithm is an extension of Algorithm [alg:smote1]. The details of the general procedure are presented in Algorithm [alg:smotek]:

Let $`Z_k`$ and $`Z_K`$ denote the SMOTE variables generated via SMOTE-$`k`$ and SMOTE-$`K`$ procedures, respectively. Then,

Z_K = \frac{1}{K}\sum_{k=1}^K Z_k.

First, we extend the result about the convergence in probability from $`Z_k`$ to $`Z_K`$.

\begin{align*}
P\left( \left| Z_K - X_1 \right| > \delta \right)
&=P\left( \left| \left(\frac{1}{K}\sum_{k=1}^K Z_k\right) - X_1 \right| > \delta \right)\\
&\leq P\left( \frac{1}{K}\sum_{k=1}^K \left| Z_k - X_1 \right| > \delta \right)\\
&= \frac{1}{K}\sum_{k=1}^K P\left(  \left| Z_k - X_1 \right| > \delta \right)\\
&\leq\varepsilon,
\end{align*}

Next, we extend the result about the convergence in mean from $`Z_k`$ to $`Z_K`$.

\begin{align*}
E\left[ \left|Z_K - X_{1}\right| \right]
&=E\left[ \left| \left(\frac{1}{K}\sum_{k=1}^K Z_k\right) - X_{1}\right| \right]\\
&\leq \frac{1}{K}\sum_{k=1}^K E\left[ \left|Z_k - X_{1}\right| \right] \\
&\leq\varepsilon,
\end{align*}

Optimal rank $`k`$

In the standard SMOTE algorithm, the rank $`k`$ of the nearest neighbor is typically set to $`k=5`$ . However, Equation [distk] demonstrates that selecting lower values of $`k`$ is preferable. Concretely, since $`P(Y \leq k - 1 \mid X_1) \leq P(Y \leq k \mid X_1)`$, it follows that

\begin{equation}
\label{compk} 
P(D_{(k)} \geq \varepsilon \mid X_1) \leq P(D_{(k+1)} \geq \varepsilon \mid X_1).
\end{equation}

The above inequality implies that the expected distance between $`X_1`$ and its $`k`$-th nearest neighbor $`X_{1,(k)}`$ increases with $`k`$. Since the synthetic sample $`Z`$ is generated between $`X_1`$ and $`X_{1,(k)}`$, a larger $`k`$ results in $`Z`$ being farther from $`X_1`$. To illustrate this effect, we compare the distribution of $`Z`$ to that of $`X`$ for various values of $`k`$.

Figure 5 displays the distribution of $`Z`$ simulated for sample sizes $`n = 8, 20, 70`$ based on the uniform random variable $`X \sim U(0,1)`$. In Figure 3, we used the nearest neighbor of rank $`k=1`$, whereas in Figure 4, we used $`k=5`$. As shown in the figures, for both $`k=1`$ and $`k=5`$, the distribution of $`Z`$ converges to that of $`X`$ as $`n`$ increases. However, the convergence is significantly faster in the case of $`k=1`$ compared to $`k=5`$, especially for lower values of $`n`$.

Similarly, we have conducted simulations using Gaussian ($`X \sim \mathcal{N}(0,1)`$) and exponential ($`X \sim \mathrm{Exp}(1)`$) random variables. The results, presented in Figure 8, show that the distribution of $`Z`$ approaches to the original distribution $`X`$ faster for $`k=1`$ than for $`k=5`$.

To further assess the rate of convergence of the synthetic sample $`Z`$ to the original distribution $`X`$, we calculated the Kullback-Leibler (KL) divergence between the simulated distribution of $`Z`$ and the theoretical distribution of $`X`$ for nearest neighbor ranks $`k = 1`$ and $`k = 5`$ over a range of sample sizes $`n`$. As illustrated in Figure 12, the KL divergence for both the uniform and Gaussian random variables decreases to zero as the sample size $`n`$ increases. Notably, the KL divergence is consistently smaller for $`k = 1`$ than for $`k = 5`$ across all values of $`n`$, particularly for smaller sample sizes. In the case of the exponential random variable, the KL divergence for $`k = 1`$ approaches zero, whereas for $`k = 5`$, it initially increases before starting to decrease around $`n = 55`$.

Conclusion

In this study, we conducted a thorough theoretical examination of the convergence behavior of the SMOTE algorithm. Our analysis demonstrated that the synthetic samples $`Z`$ generated by SMOTE converge in probability to the original data distribution $`X`$. In addition, under the assumption of compact support for $`X`$, we established that $`Z`$ also converges to $`X`$ in mean, providing a stronger form of convergence. Thus, our results offer a theoretical validation that the SMOTE algorithm accurately approximates the original data distribution for large sample sizes.

The practical implications of our findings are significant for those applying SMOTE to address imbalanced data challenges. Our main results provide theoretical support to the validity of SMOTE, allowing researchers and practitioners to use the SMOTE algorithm with confidence. By recommending smaller $`k`$ values, our work provides actionable guidance that can lead to more accurate and representative synthetic samples, thereby improving the performance and fairness of machine learning models in critical applications such as healthcare, finance, and security. Additionally, similar techniques can be applied to analyze convergence in higher dimensions. Future research could extend the theoretical framework to other data augmentation techniques, further enhancing our understanding and application of methods for handling imbalanced data.