Multichannel signal detection in interference and noise when signal mismatch happens

Multichan nel signal detection in interference and no ise when signal mismatch happens W eijian Liu a , Jun Liu b, ∗ , Y ongchan Gao c , Guoshi W ang a , Y ong-Liang W ang a a W uhan Electr on ic Information Institute, W uhan 430019 , China b Department of Electr on ic Engineering and Information Science, Univers ity of Science and T echnology of China, Hefei 230027, China c Xidian University , Xi’an 710071, China Abstract In this paper , we consider the problem of detecti n g a multi channel signal i n in - terference and noise when si gnal m ismatch happens. W e first propose two selec- tiv e detectors, si nce their st rong selectivity is preferred in some situ ations. How- e ver , these two detectors would not be sui table candidat es if a robust detector i s needed. T o overcome this shortcoming, we then devise a tunable d et ector , which is parametrized by a non -negative scaling factor , referred to as the tunable pa- rameter . By adjustin g the t unable p arameter , the proposed detector can smoothly change its capability in rejecting or robustly detecting a m ismatch signal. More- over , one selectiv e detector and the tunable detector with an appropriate tunable parameter can provide n early the same detection performance as existing detec- tors in th e absence of signal m ismatch. W e o b tain analytical expressions for the probabilities of detection (PDs) and probabilities of false alarm (PF As) of the t hree proposed detectors, which are verified by Monte Carlo si m ulations. K eyw or ds: Adaptive detection, constant false alarm rate, multichannel signal, ∗ Correspon d ing author . This pa p er was accepted by Sign a l Proc e ssing (Elsevier) on Au gust 24, 2019, with DOI : 10.10 16/j.sigpr o.201 9.107268. E-mail address: liuvjian @163.com (W . Liu), junliu@ustc.ed u.cn (J. Liu), ycga o@xidian.ed u.cn (Y . Gao) , 147 61646 9@qq.co m (G. W an g), and ylwangkjld@16 3.com (Y .- L. W an g). Pr eprint submitted to Signal Pr ocessing August 27, 2019 signal mismatch, subs pace signal. 1. Intr oduction Detection of a m u ltichannel signal is a basic problem i n signal processing. Many well -known detectors were proposed in the li terature, such as Kelly’ s gen- eralized likelihood ratio test (KGLR T) [1], adapt iv e matched filter (AMF) [2], adaptiv e coh erent estim at o r (A CE) [3], and their s ubspace generalizations [4–7], etc. The above detectors were d esi gned without taking into account the i nterfer - ence, w h i ch usually exists and can significantly degrade the detection performance of a detector . In [8], it is assu med t h at there exists interference which lies i n a subspace, l inearly ind epend ent of the signal subspace. This kind of interference is often referred to as subspace interference. Se veral detectors were proposed i n [8] i n subspace interference based on the GL R T criterion. Recently , many other related detectors were proposed for t he case of subspace interference, such as the ones in [9] and the references t herein. It is worth poi n ting out that in the above refere nces, the signal is assumed to h ave an exactly known steering vector or completely li e in a given subspace. Howe ver , in practice there are many factors (e.g., not perfectly calibrated array , pointing error , and mult i-path effects [10, 11]) leading to si g nal mismatch, for which th e actual signal steering vector m ay not b e aligned with the nominal one or not completely li e in the presumed si g nal subspace. Seldom work was done for the si gnal detection in t he presence of i nterference when s ignal mismatch happens. A related work is [12], whi ch analysed the statistical performance of the GLR T - based detector in [8] in the presence of signal m ismatch. Howe ver , t o the best of our knowledge, no detector is specifically designed for the detection problem in interference when sign al mi smatch arises. 2 In this paper , we propose two selective (less tol erant to signal mism atch) 2 detectors for m ultichannel signal detection in the presence o f interference wh en signal mismatch occurs. Both selecti ve d etectors ha ve improved detection per - formance in rejecting mismatched signals. Howev er , when a robust detector is needed, neither o f these two detectors is a good choice. T o overc ome this draw- back, we then design a tunable detector , which is parametrized by a non-negati ve scaling factor , called the tu n able parameter . By adjus t ing the tun able parame- ter , the proposed tunable detector can fle xibly control the directivity property (the capability of selectivity or robustness to mism atched signal). In particular , the tunable detector with a sm all tunable p arameter can b e much more rob ust to m is- matched si gnals than existing detectors, while it, with a moderately lar ge t unable parameter , can be more selectiv e e ven than the two proposed selective d etectors. W e deriv e analytical expressions for the probabilit ies of detectio n (PDs) and prob- abilities of false alarm (PF As) of the three detectors, confirmed by Monte Carlo simulatio n s . The rest of the paper is organized as follows. Section 2 formulates the detec- tion problem to be solved. Section 3 gives the p roposed detectors , whose statis- tical properties are in vestigated in Section 4 . Section 5 ill u s trates the numerical example. Finall y , Section 6 concludes the paper . 2 In some p ractical application s, a selecti ve detector would be p referred rather than a robust detector, because sign al mismatch m ay be caused by sidelobe targets or jamming signal. More in-depth analysis ca n b e fo und in [1 3]. 3 2. Prob lem formulation a nd r elated detectors For an N × 1 test data vector x , 3 under signal-absent hypot hesis, it consi s ts of noise n and i nterference j . In contrast, under signal-present hypothesis, x con- tains noise n , interference j , and useful s ignal s . The interference j and signal s are assumed to li e in known linearly independent su bspaces but with unknown coordinates. Precisely , j and s can be expressed as j = J φ and s = H θ , respec- tiv ely . The N × p full -column-rank mat ri x H spans the signal subspace, while the N × q full-colum n-rank m atrix J spans the interference subspace. The q × 1 vector φ and p × 1 vector θ denote the interference and s i gnal coordinates, respecti vely . Note that p + q ≤ N , due to the assum ption of linear independence of t h e inter- ference subspace and signal subspace. Th e noise n is Gaussian d istributed, with a zero mean and a covariance matrix R , which is usually unknown in practice. T o estimate R , i t is assum ed that there are L noise-onl y independent and identically distributed (IID) traini ng data, denoted as x l , l = 1 , 2 , · · · , L , sharing the same cov ariance matrix wit h the test data. Thus, the binary hypothesis test to be solved is summ arized as    H 0 : x = J φ + n , x l = n l , l = 1 , 2 , · · · , L, H 1 : x = H θ + J φ + n , x l = n l , l = 1 , 2 , · · · , L, (1) 3 Scalars are den oted by lightfaced lowercase letters, vector s by bo ld faced lowercase letters, and m atrices b y bo ldfaced uppe r case letters, r espectively . min { a, b } choo ses the m inimum value between real nu mbers a and b . | h | denotes the mo dulus of the complex number h . Pr [ · ] is the probab ility o f an ev ent. A H stands for the conjug ate tr anspose of the matrix A . < A > stand s for the subspace spanned by th e columns of A . The symbol “ ∼ ” denotes “ be distributed as”. C F M ,N ( ξ ) and C B M ,N ( δ 2 ) den ote a comp lex noncen tral F-distribution w ith M and N d egrees o f freedom (DOFs) and a co mplex noncen tral Beta-distribution with M an d N DOFs, respecti vely , and ξ and δ 2 are the cor respond in g noncentrality parameters. When ξ = δ 2 = 0 , the two statistical distributions become central ones and written as C F M ,N and C B M ,N , respectively . Finally , I N is the N × N identity matrix and 0 p × q is th e p × q nu ll matrix . 4 where n l is the noise in the l th train i ng data vector x l . For the detection problem in (1), the GLR T and two-step GLR T (2S–GLR T) are [8] t GLR T –I = ˜ x H P P ⊥ ˜ J ˜ H ˜ x 1 + ˜ x H P ⊥ ˜ J ˜ x − ˜ x H P P ⊥ ˜ J ˜ H ˜ x (2) and t 2S–GLR T –I = ˜ x H P P ⊥ ˜ J ˜ H ˜ x , (3) respectiv ely , where ˜ x = S − 1 / 2 x , ˜ J = S − 1 / 2 J , ˜ H = S − 1 / 2 H , (4) P ⊥ ˜ J = I N − P ˜ J , P ˜ J = ˜ J ( ˜ J H ˜ J ) − 1 ˜ J H , (5) P P ⊥ ˜ J ˜ H = P ⊥ ˜ J ˜ H ( ˜ H H P ⊥ ˜ J ˜ H ) − 1 ˜ H H P ⊥ ˜ J , (6) and S = P L l =1 x l x H l is L ti mes t he sample covariance matrix (SCM). For con ve- nience, the detectors in (2) and (3) are referre d to as the GLR T with interference rejection (GLR T –I) and 2S–GLR T with interference rejection (2S–GLR T –I), re- spectiv ely . T o the best of our knowledge, no detector is specifically desi gned for th e d e- tection problem in (1) when signal mismatch happens. 3. Prop osed detectors In this section we first prop ose two sel ective detectors for mismatched sig- nals, and then propose a tunable detector , which can smoothly adjust its det ecti on performance for mis matched signals. It i s observed th at (2) and (3) have similar forms as the subspace-based GLR T (SGLR T) [4, 5] and subs pace-based AMF (SAMF) [14], respectively 4 . The SGLR T 4 This would be more o bvious if we intro duce th e qu antities ˜ z = P ⊥ ˜ J ˜ x a nd ˜ A = P ⊥ ˜ J ˜ H an d substituting them into (2) and (3). 5 and SAMF were designed withou t taking the possi bility of signal mi smatch, and they ha ve poor detection performance in terms of rejecting mismatched signals. T wo well-known selectiv e detectors for mism atched sig n als in the absence of in- terference are t he adaptive beamformer orthogonal rejection test (ABOR T) [13 ] and whit ened ABOR T (W –ABOR T) [15]. According to the detectio n statistics of the ABOR T and W –ABOR T , we can analogously design the following two selec- tiv e detectors in the presence of interference t ABOR T –I = 1 + ˜ x H P P ⊥ ˜ J ˜ H ˜ x 1 + ˜ x H P ⊥ ˜ J ˜ x − ˜ x H P P ⊥ ˜ J ˜ H ˜ x (7) and t W –ABOR T –I = 1 + ˜ x H P ⊥ ˜ J ˜ x (1 + ˜ x H P ⊥ ˜ J ˜ x − ˜ x H P P ⊥ ˜ J ˜ H ˜ x ) 2 , (8) which, for con venience, are referred to as th e ABOR T with interference rejection (ABOR T –I) and W –ABOR T with interference rejectio n (W –ABOR T –I), respec- tiv ely . It is expected that t he propos ed ABOR T –I and W –ABOR T –I can provide better performance i n terms of rejecting mismatched signals. In fact, this is indeed the case, as shown in Section 4 below . Howe ver , th ey suffer from performance loss if a robust detector is needed. T o cope wit h t his problem, we introduce t h e following tunable detector t T –W –ABOR T –I = 1 + ˜ x H P ⊥ ˜ J ˜ x (1 + ˜ x H P ⊥ ˜ J ˜ x − ˜ x H P P ⊥ ˜ J ˜ H ˜ x ) κ , (9) which is named as the tunable W – A BOR T –I (T –W –ABOR T –I). The non-negative factor κ is taken as the tunable parameter . Roughly speaking, the numerator of (9) col lects the total energy of the quasi- whitened test data ˜ x after interference sup pression 5 . In contrast, th e denominator 5 Quasi-whitenin g is done by multiplying th e test data x with S − 1 2 , and interfer ence su ppression is owing to m ultiplying th e q uasi-whitened test data ˜ x with the ortho g onal pro je c tion matrix P ⊥ ˜ J . 6 of (9) gathers the ener gy of t h e quasi-whitened test data ˜ x projected onto the sub- space orthogonal to the s i gnal-plus-interference 6 . Hence, by adjustin g the tun abl e parameter κ , one can control the directivity property of the T –W –ABOR T –I for mismatched signals. Increasing κ will make the T –W –ABOR T –I more and more selectiv e, while decreasing κ wil l m ake the T –W –ABOR T –I more and more ro- bust. In particular , the T –W –ABOR T –I with κ = 0 is most robust to signal m is- match, and in thi s case the T –W –ABOR T –I reduces t W –ABOR T –I ,κ =0 = ˜ x H P ⊥ ˜ J ˜ x , (10) where the constant is i g nored. Equation (10) as be recast as t ′ W –ABOR T –I ,κ =0 = ˜ z H ˜ z , (11) which has the same form as the adaptive ener gy detector (AED) in [16 ]. In (11), ˜ z = P ⊥ ˜ J ˜ x . When κ = 2 , the T –W –ABOR T –I reduces to the W –ABOR T –I. When κ = 1 , the T –W –ABOR T –I reduces to t W –ABOR T –I ,κ =1 = 1 + ˜ x H P ⊥ ˜ J ˜ x 1 + ˜ x H P ⊥ ˜ J ˜ x − ˜ x H P P ⊥ ˜ J ˜ H ˜ x , (12) which is equi v alent to the GLR T –I, since t W –ABOR T –I ,κ =1 = 1 / (1 − t GLR T –I ) can serve as a monotonically increasing function of t GLR T –I . 4. Statistical perf ormance of the proposed detect ors in the presence of signal mismatch When signal mismat ch happens, the actual si gnal, denoted as s 0 , will not be- long to the nominal signal subspace < H > . T o facilitate the deri va tions of the 6 This is mor e evident if we rewrite ˜ x H P ⊥ ˜ J ˜ x − ˜ x H P P ⊥ ˜ J ˜ H ˜ x as ˜ x H P ⊥ ˜ B ˜ x , where ˜ B = [ ˜ J , ˜ H ] . 7 statistical properties of the p rop osed detector , a lo s s factor i s introduced β = (1 + ˜ x H P ⊥ ˜ J ˜ x − ˜ x H P P ⊥ ˜ J ˜ H ˜ x ) − 1 . (13) Using (2) and (13), we can rewrite (7), (8), and (9) as t ABOR T –I = t GLR T –I + β , (14) t W –ABOR T –I = (1 + t GLR T –I ) β , (15) and t T –W –ABOR T –I = β κ − 1 (1 + t GLR T –I ) , (16) respectiv ely . Using (14)-(16), we can readily o b tain the expressions for the conditional PDs and PF As of the three propo s ed detectors, conditioned on β . Precisely , the condi- tional PDs of th e ABOR T –I, W –ABOR T –I, and T –W –ABOR T –I can be expressed as PD ABOR T –I | β = Pr [ t GLR T –I + β > η a ; H 1 ] = 1 − P 1 ( η a − β ) , (17) PD W –ABOR T –I | β = Pr [(1 + t GLR T –I ) β > η w ; H 1 ] = 1 − P 1  η w β − 1  , (18) and PD T –W –ABOR T –I | β = Pr [ β κ − 1 (1 + t GLR T –I ) > η t ; H 1 ] = 1 − P 1 ( η t β 1 − κ − 1) , (19) respectiv ely , where η a , η w , and η t are the detection thresho l ds for the ABOR T –I, W – ABOR T –I, and T –W –ABOR T –I, respectively , P 1 ( η ) is the cumu l ativ e dis t ribu- tion function (CDF) of t GLR T –I in (2) under hypothesis H 1 conditioned on β , given by P 1 ( η ) = Pr [ t GLR T –I ≤ η | β ; H 1 ] . (20) 8 Cautions must be taken when aver aging the conditional PDs over β . In (17)- (19), to ensure that the CDF is meaningful, the following constraints are needed: β ≤ η a , β ≤ η w , and β 1 − κ > η − 1 t , (21) respectiv ely . Consequently , together with the fact 0 < β < 1 , the expressions for the PDs of the ABOR T –I and W –ABOR T –I can be calculated as PD ABOR T –I = Z min(1 ,η a ) 0 [1 − P 1 ( η a − β )] f 1 ( β ) d β , (22) and PD W –ABOR T –I = Z min(1 ,η w ) 0  1 − P 1  η w β − 1  f 1 ( β ) d β , (23) respectiv ely . In (22) and (23), f 1 ( β ) i s the p robability density function (PDF) of β defined in (13) under hypothesis H 1 . The calculations of the PD of the T –W – ABOR T –I are divided into the following four cases: i) 0 ≤ κ ≤ 1 and η t ≤ 1 PD T –W –ABOR T –I = 1 , (24) ii) 0 ≤ κ ≤ 1 and η t > 1 PD T –W –ABOR T –I = Z 1 η − 1 / (1 − κ ) t [1 − P 1 ( η t β 1 − κ − 1)] f 1 ( β ) d β (25) iii) κ > 1 and η t ≤ 1 PD T –W –ABOR T –I = Z η − 1 / (1 − κ ) t 0 [1 − P 1 ( η t β 1 − κ − 1)] f 1 ( β ) d β , (26) iv) κ > 1 and η t > 1 PD T –W –ABOR T –I = Z 1 0 [1 − P 1 ( η t β 1 − κ − 1)] f 1 ( β ) d β . (27) 9 In the presence of signal mism atch, t GLR T –I in (2), with a fixed β under hy poth- esis H 1 , is distributed as [12] t GLR T –I | [ β , H 1 ] ∼ C F p,L − N + q +1 ( ρ ef f β ) , (28) where ρ ef f = ¯ s H 0 P ⊥ ¯ J ¯ H ( ¯ H H P ⊥ ¯ J ¯ H ) − 1 ¯ H H P ⊥ ¯ J ¯ s 0 (29) is referred t o as the ef fecti ve sign al -t o -noise ratio (eSNR). In (29), ¯ s 0 = R − 1 / 2 s 0 , ¯ J = R − 1 / 2 J , ¯ H = R − 1 / 2 H , P ⊥ ¯ J = I N − P ¯ J , and P ¯ J = ¯ J ( ¯ J H ¯ J ) − 1 ¯ J H . The statistical distri bution of t GLR T –I in (2) under hypothesi s H 0 is [12] t GLR T –I ∼ C F p,L − N + q +1 , (30) Moreover , in t he presence of s i gnal mismatch, β in (13) under hypoth eses H 1 and H 0 is distributed as [12] β | H 1 ∼ C B L − N + p + q +1 ,N − p − q ( δ 2 ) (31) and β | H 0 ∼ C B L − N + p + q +1 ,N − p − q , (32) respectiv ely , where δ 2 = ¯ s H 0 P ⊥ ¯ J P ⊥ P ⊥ ¯ J ¯ H P ⊥ ¯ J ¯ s 0 . (33) with P ⊥ P ⊥ ¯ J ¯ H = I N − P P ⊥ ¯ J ¯ H and P P ⊥ ¯ J ¯ H = P ⊥ ¯ J ¯ H ( ¯ H H P ⊥ ¯ J ¯ H ) − 1 ¯ H H P ⊥ ¯ J . According to (A2-29) in [17], the CDF in (20) can be calculated as P 1 ( η ) = L − N + q X k =0 C k + p L − N + p + q η k + p (1 + η ) L − N + p + q IG k +1  ρ ef f β 1 + η  , (34) where C m n = n ! m !( n − m )! is the binominal coeffi cient and IG k +1 ( a ) = e − a P k m =0 a m m ! is th e i ncomplete Gamma function. Moreover , according to (A2-23) in [17], t he 10 PDF of β in (13) und er hypothesis H 1 is f 1 ( β ) = f 0 ( β ) e − δ 2 β L − N + p + q +1 X k =0 C k L − N + p + q +1 ( N − p − q − 1)! ( N − p − q + k − 1)! δ 2 k (1 − β ) k , (35) where f 0 ( β ) = β L − N + p + q (1 − β ) N − p − q − 1 B ( L − N + p + q + 1 , N − p − q ) . (36) is the PDF of β under hypothesi s H 0 . In (36), B ( m, n ) = ( m − 1)!( n − 1)! ( m + n − 1)! is the Beta function. T aking (34) and (35) into (22)–(27), we can o btain the final expression for the PDs of the ABOR T –I, W – ABOR T –I, and T –W –ABOR T –I. Setting ρ ef f = 0 in (34) results in the CDF of t GLR T –I under hypothesis H 0 , i.e., P 0 ( η ) = C p L − N + p + q η p (1 + η ) L − N + p + q . (37) The PF As of the ABOR T –I, W –ABOR T –I, and T –W –ABOR T –I can be obtained by replacing P 1 ( · ) and f 1 ( β ) by P 0 ( · ) and f 0 ( β ) , respecti vely , i n (22)–(27). Some remarks on the influence of signal mismatch on the detection perfor - mance of the detectors are given below . The eSNR in (29) can be recast as [12] ρ ef f = ρ SNR sin 2 ψ cos 2 ϑ, (38) where ρ SNR = ¯ s H 0 ¯ s 0 (39) is the con ventional SNR for multichannel signal detection in the absence of inter- ference, sin 2 ψ = ¯ s H 0 P ⊥ ¯ J ¯ s 0 ¯ s H 0 ¯ s 0 , (40) and cos 2 ϑ = ¯ s H 0 P P ⊥ ¯ J ¯ H ¯ s 0 ¯ s H 0 P ⊥ ¯ J ¯ s 0 . (41) 11 The quantity cos 2 ϑ in (41) s erves as t he metric of sig nal mismatch in the presence of interference. If si gnal mismatch does not occur , there exists a p × 1 vector θ 0 such that s 0 = H θ 0 . Using this result, we can verify that cos 2 ϑ = 1 . For comparison purpos es, a well-known m etric of si g nal mismatch in the ab- sence of interference is list ed belo w [18] cos 2 φ = ¯ s H 0 P ¯ H ¯ s 0 ¯ s H 0 ¯ s 0 . (42) Some preliminary analysis is sum m arized in the following proposi t ion. Propositi on 1. i). cos 2 φ = 1 results i n cos 2 ϑ = 1 , b ut not vi ce versa. i i) cos 2 φ = 0 does not necessarily lead to cos 2 ϑ = 0 , and vice versa. Pr oof. See the appendix A.  Before proceeding, we would like to p oint out that the three prop o sed detectors can successfully sup press t he interference, since the p ower of the in t erference does not impact the PDs and PF As. The interference affects the detection performance through sin 2 ψ and the DOFs of the statis t ical dis tributions. More analysi s of the influence of interference on t he detection performance can be found in [12]. 5. Numerical examples In this section, we ev aluate t h e detection performance of the proposed ABOR T – I, W –ABOR T –I, and T –W –ABOR T –I for t he case of no si gnal mismatch and the case o f signal mismatch. Both t heoretical and M onte Carlo simulation results are provided. The noise is modelled as exponentially correlated random vector with one-lag correlation coef ficient. Hence, the ( i, j ) th element of R is R ( i, j ) = ǫ | i − j | , i, j = 1 , 2 , · · · , N , and ǫ is chosen to be 0 . 9 . The i nterference-to-noise ratio (INR) is defined as INR = φ H J H R − 1 J φ . (43) T o reduce the running time of Monte Carlo simulations , t h e PF A is chosen as PF A = 10 − 3 . 10 5 Monte Carlo si mulations are used to generate a detection 12 threshold, while 10 4 Monte Carlo sim u lations are carried out to generate a PD. Moreover , the fol lowing parameters are adopted throughout this section: N = 12 , L = 2 N , p = 1 , q = 2 , and INR = 10 dB. Fig. 1 shows the PDs of the proposed detectors under di ff erent SNRs, com- pared with the existing GLR T –I and 2 S–GLR T –I. In the legend, “TH” indi cates theoretical results, while “MC” stands for Mon t e Carlo simulation results . It is seen that the t heoretical results m atch the Monte Carlo simulat i on result s pretty well. For the chosen parameters, the ABOR T –I, T –W –ABOR T –I with κ = 0 . 8 , GLR T –I, and 2S–GLR T –I roughly have t he sam e PDs. The W –ABOR T –I and T –W –ABOR T –I with κ = 2 . 5 suffer from certain performance loss for matched signals, compared wit h th e other detectors. Howe ver , the W –ABOR T –I and T – W – ABOR T –I with κ = 2 . 5 exhibit satis fied detection performance in terms of rejecting mism atch si gnals, as shown in Fig . 3 belo w . Fig. 2 pl ots t he PDs of the detectors under different sin 2 ψ . The tunable pa- rameter for the T –W –ABOR T -I i s κ = 0 . 8 . The PD curve of the W –ABOR T –I is not given, since it suf fers from certain detection performance los s , compared with the other detectors. The results sho w that all the PDs of the detectors in- crease when sin 2 ψ i n creases. Th i s is because the increase of sin 2 ψ results in the increase of the eSNR, defined in (29), w h i ch leads to the im provement in the PD. Fig. 3 disp l ays the PDs of the T –W –ABOR T –I with different t unable parame- ters κ . It i s shown that when there is no signal mismatch, i.e., cos 2 ϑ = 1 . 0 , the PD of the T –W –ABOR T –I first increases and th en decreases as the tunable parameter κ increases. In contrast, when signal mismatch occurs, i.e., cos 2 ϑ = 0 . 3 , the PD of the T –W –ABOR T –I decre ases directly as the tunable parameter κ i ncreases. This is due to the fact that the selectivity o f the T –W –ABOR T –I increases as the increase of the t unable parameter . Specifically , in the range o f 0 . 6 ≤ κ ≤ 1 . 0 , the T –W –ABOR T –I can provide roughly the same PD as th e GLR T –I (a special case 13 of the T –W –ABOR T –I with ≤ κ = 1 . 0 ) in the case of no sign al mismatch. Fig. 4 depicts th e contours of the PDs of the detectors under di f ferent de- grees of signal mismatch and di fferent SNRs. Thi s t ype of figure is usuall y called mesa p l ot. Th e sol i d lines denote theoretical results, which are consistent wi th the Monte Carlo resul ts indicated by the dotted lines. It is shown that the ABOR T –I and W –ABOR T -I have better detection performance than the GLR T –I and 2S– GLR T –I in terms of mi smatched signal rejection. T aking the ABOR T –I for ex- ample. When cos 2 ϑ < 0 . 5 , it cannot provide a PD greater th an 0.5, no matter how high the SNR is. In other words, the ABOR T –I and W –ABOR T -I do not take a largely m ismatched signal as a desired target. Moreover , the T –W –ABOR T –I is very flexible in controll ing the detection performance for m i smatched signals . W ith a large tun abl e p arameter , i.e., κ = 2 . 5 , the T –W –ABOR T –I possesses t he best selectivity property . On the other hand, the T –W –ABOR T –I, with a small tun- able parameter , is very robust to signal mismatch. For the chosen parameters, the T –W –ABOR T –I with κ = 0 . 8 roughly has the same rob ustness as the 2S–GLR T – I. In fact, the T –W –ABOR T –I wi th a smal l er tunable parameter can become much more robust than t he 2S–GLR T –I. Gathering t he results in Figs. 1, 2, and 4, we can conclu de that: 1) The ABOR T –I has s lightly better selectivity property t han the GLR T –I. Howe ver , t h e former suffers from sli g htly performance loss com p ared wi th the later in the case of no si gnal mismatch. 2) The T –W –ABOR T –I, with a proper tun abl e parameter less than u nity , can provide better robustness than the GLR T –I and 2S–GLR T – I. The T –W –ABOR T –I, with the same tunable p arameter , suffers from a slightl y performance loss for matched signals, compared with the GLR T –I. 3 ) The W – ABOR T –I and T –W –ABOR T –I wit h a proper tu n able parameter greater t han two are m uch more selective t han the other detectors. Howe ver , these two detectors suffe r from non-negligibl e loss in the case of no signal mi s match. 14 6. Conclusions In this paper , we considered the problem of detecti n g a multichannel signal in the presence of interference when signal mi smatch happens. T wo selecti ve detec- tors, namely , the ABOR T –I and W –ABOR T –I, and a tunable detector , namely , T – W – ABOR T –I were p rop osed, and th e corresponding analy t ical expressions for the PDs and PF As were gi ven. Numerical examples show that the ABOR T –I and W – ABOR T –I exhibit better detection performance in terms of rejecting mismatched signals, and the T –W –ABOR T –I h as the flexibility in governing the detection per- formance for m ismatched signals. The T –W –ABOR T –I, wit h a lar ge tunable pa- rameter , is very selective, whil e it becomes robust wi th a moderately small tunable parameter . In addition, in the case of n o signal m ismatch, the ABOR T –I and T – W – ABOR T –I with a s u i table t u nable parameter , say , 0 . 6 ≤ κ ≤ 1 . 0 , can provide nearly the same detectio n performance as the GLR T –I. Ap pendix A. Pr oof of Prop osition 1 i) If cos 2 φ = 1 , then there exists a p × 1 vector θ 0 such that ¯ s 0 = ¯ H θ 0 . T aking this result int o (41) yields that cos 2 ϑ = 1 . On the other hand, if cos 2 ϑ = 1 , then we ha ve P P ⊥ ¯ J ¯ H ¯ s 0 = P ⊥ ¯ J ¯ s 0 , (A.1) which can be recast as P P ⊥ ¯ J ¯ H P ⊥ ¯ J ¯ s 0 = P ⊥ ¯ J ¯ s 0 . (A.2) It follows t h at P ⊥ ¯ J ¯ s 0 lies in the subspace < P ⊥ ¯ J ¯ H > . Hence, there exists a p × 1 vector θ 1 such that P ⊥ ¯ J ¯ H θ 1 = P ⊥ ¯ J ¯ s 0 . (A.3) 15 Using the matrix ¯ J , we can obt ain an N × ( N − q ) semi-unit ary matrix ¯ J ⊥ such that P ⊥ ¯ J = ¯ J ⊥ ¯ J H ⊥ , (A.4) ¯ J H ⊥ ¯ J ⊥ = I N − q , and ¯ J H ⊥ ¯ J = 0 ( N − q ) × q . (A.5) Then (A.3) can be rewritten as ¯ J ⊥ ¯ J H ⊥ ¯ H θ 1 = ¯ J ⊥ ¯ J H ⊥ ¯ s 0 . (A.6) According to (A.5), (A.6) can be rewritten as ¯ J ⊥ ¯ J H ⊥ ( ¯ H θ 1 + ¯ J φ 1 ) = ¯ J ⊥ ¯ J H ⊥ ¯ s 0 , (A.7) where φ 1 is an arbitrary q × 1 vector . It foll ows from (A.7) that if t he whit ened signal component ¯ s 0 can be expressed as s 0 = ¯ H θ 1 + ¯ J φ 1 , (A.8) then cos 2 ϑ = 1 . It is known from (A.8) that ¯ s 0 may not com pletely lie in < ¯ H > when cos 2 ϑ = 1 . In this case, we have cos 2 φ < 1 . ii) If cos 2 φ = 0 , then ¯ H H ¯ s 0 = 0 p × 1 , or equiv alently , ¯ H H // ¯ s 0 = 0 p × 1 , (A.9) where ¯ H // = ¯ H ( ¯ H H ¯ H ) − 1 2 . Using ¯ H // we can cons truct an N × N unitary matrix U = [ ¯ H // , ¯ H ⊥ ] , which can be take n as a basic matrix of the N × N complex space C N × N . Hence, there exists an N × 1 vector θ such that ¯ s 0 = U θ . (A.10) W e can partition θ as θ = [ θ T // , θ T ⊥ ] T , where t h e dim ens ions of θ // and θ ⊥ are p × 1 and ( N − p ) × 1 , respectiv ely . According to the definitions of U and θ , we hav e ¯ s 0 = ¯ H // θ // + ¯ H ⊥ θ ⊥ . (A.11) 16 Pre-multiplying (A.11) with ¯ H H // yields that θ // = 0 p × 1 . (A.12) Substitutin g (A.12) into (A.11) leads to ¯ s 0 = ¯ H ⊥ θ ⊥ . (A.13) Substitutin g (A.4) and (A.13) into (41) leads to cos 2 ϑ = θ H ⊥ ¯ H H ⊥ ¯ J ⊥ ¯ J H ⊥ ¯ H // ( ¯ H H // ¯ J ⊥ ¯ J H ⊥ ¯ H // ) − 1 ¯ H H // ¯ J ⊥ ¯ J H ⊥ ¯ H ⊥ θ ⊥ θ H ⊥ ¯ H H ⊥ ¯ J ⊥ ¯ J H ⊥ ¯ H ⊥ θ ⊥ , (A.14) which is generally not equal to zero. For example, a specific form of ¯ J ⊥ , for a g iv en nomi nal signal matrix H , is ¯ J ⊥ = [ ¯ H // , ¯ H ⊥ , 1 ] , where ¯ H ⊥ , 1 is the first N − p − q columns of ¯ H ⊥ . If cos 2 ϑ = 0 , then ¯ H H P ⊥ ¯ J ¯ s 0 = 0 p × 1 . (A.15) In a manner sim i lar to the deriv ations of (A.10)-(A.13), ¯ s 0 can be expressed as ¯ s 0 = ¯ V θ 2 , (A.16) where ¯ V i s an N × ( N − p ) matrix such that th e augmented m atrix [ P ⊥ ¯ J ¯ H ( ¯ H H P ⊥ ¯ J ¯ H ) − 1 2 , ¯ V ] is an N × N unitary matrix, and θ 2 is an ( N − p ) × 1 vector . Substi tuting (A.16) into (42) results in cos 2 φ = θ H 2 ¯ V H P ¯ H ¯ V θ 2 θ H 2 ¯ V H ¯ V θ 2 . (A.17) Another form of (A.17) is given below . It is known from (A.15) that ¯ H H ¯ s 0 = ¯ H H P ¯ J ¯ s 0 . (A.18) Substitutin g (A.18) into (42), after so me algebra, leads to cos 2 φ = ¯ s H 0 P ¯ J P ¯ H P ¯ J ¯ s 0 ¯ s H 0 ¯ s 0 . (A.19) From (A.17) and (A.19), we know that cos 2 φ is not generally equal t o zero. 17 Ap pendix B. The method to generate J , j , sin 2 ψ , and cos 2 ϑ with specific values f or Monte Carlo simulations There are th ree main steps to generate J , sin 2 ψ , and cos 2 ϑ wi th s p ecific v al- ues: 1) Generate an arbitrary N × q matrix J . 2) Generate the actual sig n al steering vector s 0 satisfying a specific value of sin 2 ψ . Precisely , s 0 is generated by select- ing a properly scalar 0 ≤ r ≤ 1 such that s 0 = R 1 2 ¯ s 0 and ¯ s 0 = r ¯ j 0 + (1 − r ) ¯ j 1 satisfying a specific sin 2 ψ , with ¯ j 0 being an arbitrary colu mn of ¯ J and ¯ j 1 being the last colum n of A . A is th e matrix contain i ng the left s ingular-v ectors of ¯ J . 3) Generate t he nominal s ignal matrix H s at i sfying specific value of cos 2 ϑ . Pre- cisely , H can be generated by cho o sing an appropriate scalar 0 ≤ α ≤ 1 such that H = R 1 2 ¯ H and ¯ H = α ¯ H 0 + (1 − α ) ¯ H 1 satisfies a sp ecific cos 2 ϑ , with ¯ H 0 = [ ¯ s 0 , ¯ H r ] and ¯ H 1 = W 1 . ¯ H r is an arbitrary N × ( p − 1) matrix, and W 1 is the last p column s of W , with W containi ng the left singular-ve ctors o f P ⊥ ¯ J ¯ s 0 . Moreover , for a given INR defined in (43), we ca n generate th e interfer - ence j as j = c J φ n , wh ere φ n is an arbitrary q × 1 column vector and c = p φ H n J H R − 1 J φ n . Acknowledgeme nts This work was supported in p art by National Natural Science Foundation of China under Contracts 61501505, 61501351, and 61871 4 69, in part by the Natural Science Foundation of Hubei Province u nder Contract 2017CFB589, in part by the National Natural Science F oundation of China and Ci v il A viatio n Adm inistration of China under Grant U1733116, and in part by the Y outh Innovation Promotion Association CAS under Grant CX21000 6 0053. 18 Refer ences Refer ences [1] E. J. Kelly , “ An adaptive detection al g orithm, ” IEEE T ransactions on Aer ospa ce and Electr onic Systems , v ol. 22, no. 1, pp. 115–127 , 1986. [2] F . C. Robey , D. R. Fuhrm ann, E. J. K elly , and R. Nitzberg, “ A CF AR adaptive matched filter detector , ” IEEE T ransactions on Aerospace and Electr oni c Systems , vol. 28, no. 1, pp. 208–216, 1992. [3] S. Kraut and L. L. Scharf, “The CF AR adaptiv e subspace detector is a scale- in variant GLR T, ” IEEE T ransactio ns on Sig nal P r ocessing , vol. 47, no . 9, pp. 2538–25 4 1, 1 9 99. [4] R. S. Raghav an, N. Puls o ne, and D. J. McLaughlin, “Performance o f the GLR T for adaptive vector subspace detection, ” IEEE T ransactions on Aer ospa ce and Electr onic Systems , v ol. 32, no. 4, pp. 1473–14 87, 1996. [5] D. P astina, P . Lombardo, and T . Bucciarelli, “ Adaptive pol arim etric tar- get detectio n with coherent radar part I: Detection again s t Gaussian back- ground, ” IEEE T ransactions on Aer ospa ce and Electr onic Syst ems , vol. 37, no. 4, pp. 1194– 1206, 2001. [6] S. Kraut and L. L. Scharf, “ Adaptive subspace detectors, ” IEEE T ransactions on Signal Pr ocessin g , vol. 49, no. 1, pp. 1–16, 2001 . [7] W . Liu, W . Xie, J . Liu, and Y . W ang, “ Adaptive doubl e subspace signal de- tection in Gaussian background–part I: Homogeneous en vironments, ” IEE E T ransactions on S ignal Pr ocessing , vol. 62, no. 9, pp. 2345–2357, 2014. [8] F . Bandiera, A. De Maio, A. S. Greco, and G. Ricci, “ Adaptive radar de- tection of distributed targets in homogeneous and partiall y homogeneous 19 noise plus subs pace interference, ” IEEE T ransaction s on Signal Pr ocessing , vol. 55, no. 4, pp. 1223–1237, 2007. [9] W . Liu, J. Liu, H. Li , Q. Du, and Y . W ang, “Multichannel signal detec- tion based on W ald t est in subspace interference and Gaussian nois e, ” IEEE T ransactions on Aer os pace and Electr onic Systems , v ol. 55, no. 3, pp. 1370– 1381, 2019. [10] D. Orlando and G. Ricci, “ A Rao test wi th enhanced selectivity properties in homogeneous s cenarios, ” IEEE T ransactions on S ignal Pr ocessing , v ol. 58, no. 10, pp. 5385–5390 , 2010. [11] C. Hao, B. Liu, and L. Cai, “Performance analysis of a two-stage Rao detec- tor , ” Signal Pr ocessing , vol. 91, no. 8, pp. 2141–2146 , 2011. [12] W . Li u, J. Liu, H. Lei, C. Hao, and Y .-L. W ang, “Per formance analysis of adaptiv e d etectors for p oint tar gets in subspace interference and Gauss ian noise, ” IEEE T ransactions on Aer ospace and Electr onic Systems , vol. 5 4, no. 1, pp. 429–4 41, 2018. [13] N. B. Puls o ne and C. M. Rader , “ Adaptive beamformer orthogonal rejection test, ” IEEE T ransactions o n Signal Processing , vol. 49, no. 3, pp. 521–5 2 9, 2001. [14] J. Liu, Z.-J. Zhang, and Y . Y ang, “Optimal wa veform design for general- ized likelihood ratio and adaptive matched filter d etectors using a div ersely polarized antenna, ” Signal Pr ocessing , v ol. 92, no. 4, pp. 1126–11 3 1, 2 0 12. [15] F . Bandiera, O. Besson , and G. Ricci, “ An ABOR T-like detector with im - proved mism atched sign al s re jection capabilities, ” IEEE T ransactions on Signal Pr o cessing , v ol. 56, no. 1, pp. 14–25, 2007. 20 [16] R. S. Ragha van, H. F . Qiu, and D. J. M claughlin, “CF A R d etecti on i n clu tter with unknown correlatio n properti es, ” IEEE T ransactions on Aer ospace and Electr oni c Systems , v ol. 31, no. 2, pp. 647–657, 1995. [17] E. J. K elly and K. M. Forsythe, “ Adaptive detection and parameter estim a- tion for mul tidimensi o nal signal models, ” Lincoln Laboratory , Lexingto n , T echnical Report, 19 89. [18] W . Liu, J. Liu, C. Zhang, and H. Li , “Performance prediction o f s ubspace- based adaptive d et ectors wi t h signal mismatch, ” Si gnal Pr o cessing , v ol. 123, pp. 122–126 , 2016. 21 5 10 15 20 SNR (dB) 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 PD ABORT-I, TH W-ABORT-I, TH T-W-ABORT-I, TH, κ =0.8 T-W-ABORT-I, TH, κ =2.5 ABORT-I, MC W-ABORT-I, MC T-W-ABORT-I, MC, κ =0.8 T-W-ABORT-I, TH, κ =2.5 GLRT-I, TH 2S-GLRT-I, TH Fig.1. PD versus SNR in the absence of s ignal mismatch. cos 2 ϑ = 1 and sin 2 ψ = 0 . 8 . 22 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 sin 2 ψ 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 PD ABORT-I, TH T-W-ABORT-I, TH GLRT-I, TH 2S-GLRT-I, TH Fig. 2. PD versus sin 2 ψ in the absence of si gnal mismatch. cos 2 ϑ = 1 and SNR = 17 dB. 23 0 0.5 1 1.5 2 2.5 κ 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 PD T-W-ABORT-I, TH, cos 2 ϑ =1.0 T-W-ABORT-I, MC, cos 2 ϑ =1.0 T-W-ABORT-I, TH, cos 2 ϑ =0.3 T-W-ABORT-I, MC, cos 2 ϑ =0.3 GLRT-I, TH Fig. 3. PD of the T –W –ABOR T –I versus κ . sin 2 ψ = 0 . 8 and SNR = 17 dB. 24 5 10 15 20 25 30 SCNR (dB) 0 0.2 0.4 0.6 0.8 1 cos 2 ϑ ABORT-I, PD=0.1 ABORT-I, PD=0.5 ABORT-I, PD=0.9 GLRT-I, PD=0.1 GLRT-I, PD=0.5 GLRT-I, PD=0.9 5 10 15 20 25 30 SCNR (dB) 0 0.2 0.4 0.6 0.8 1 cos 2 ϑ WABORT-I, PD=0.1 WABORT-I, PD=0.5 WABORT-I, PD=0.9 T-WABORT-I, κ =2.5, PD=0.1 T-WABORT-I, κ =2.5, PD=0.5 T-WABORT-I, κ =2.5, PD=0.9 5 10 15 20 25 30 SCNR (dB) 0 0.2 0.4 0.6 0.8 1 cos 2 ϑ T-WABORT-I, κ =0.8, PD=0.1 T-WABORT-I, κ =0.8, PD=0.5 T-WABORT-I, κ =0.8, PD=0.9 2S-GLRT-I, PD=0.1 2S-GLRT-I, PD=0.5 2S-GLRT-I, PD=0.9 Fig. 4. Contours of PDs vs SNR and cos 2 ϑ . sin 2 ψ = 0 . 8 . The s olid lines with symbols denote theoretical results, while the d o tted lines stand for the Mont e Carlo results. 25