Off-Grid DOA Estimation Using Sparse Bayesian Learning in MIMO Radar With Unknown Mutual Coupling

A CCEPTED BY IEEE TRANSA CTIONS ON SIGNAL PR OCESSING 1 Of f-Grid DO A Estimation Using Sparse Bayesian Learning in MIMO Radar W ith Unkno wn Mutual Coupling Peng Chen, Member , IEEE , Zhenxin Cao, Member , IEEE , Zhimin Chen, Member , IEEE , Xianbin W ang, F ellow , IEEE Abstract —In the practical radar with multiple antennas, the antenna imperfections degrade the system performance. In this paper , the problem of estimating the direction of arrival (DO A) in multiple-input and multiple-output (MIMO) radar system with unknown mutual coupling effect between antennas is in vesti- gated. T o exploit the target sparsity in the spatial domain, the compressed sensing (CS)-based methods have been proposed by discretizing the detection area and f ormulating the dictionary matrix, so an off-grid gap is caused by the discretization pro- cesses. In this paper , different fr om the present DO A estimation methods, both the off-grid gap due to the sparse sampling and the unknown mutual coupling effect between antennas ar e consider ed at the same time, and a no vel sparse system model for DOA estimation is formulated. Then, a novel sparse Bayesian learning (SBL)-based method named sparse Bayesian learning with the mutual coupling (SBLMC) is proposed, where an expectation- maximum (EM)-based method is established to estimate all the unknown parameters including the noise variance, the mutual coupling vectors, the off-grid vector and the variance vector of scattering coefficients. Additionally , the prior distrib utions for all the unknown parameters are theoretically derived. With regard to the DO A estimation performance, the proposed SBLMC method can outperform state-of-the-art methods in the MIMO radar with unknown mutual coupling effect, while keeping the acceptable computational complexity . Index T erms —Compressed sensing, DO A estimation, MIMO radar , sparse Bayesian learning, mutual coupling. I . I N T R O D U C T I O N U NLIKE the traditional phased-array radar, multiple-input and multiple-output (MIMO) radar systems can transmit correlated or uncorrelated signals and improv e the degree of freedom, so the recent adv ancement of radar technology has directly led to MIMO radar systems. Usually , the MIMO radar systems can be classified into the colocated radar and dis- tributed radar . In the colocated MIMO radar , the space between antennas is comparable with the wavelength of transmitted signals [ 1 ]–[ 3 ], such that the wa veform div ersity can be used This work was supported in part by the National Natural Science Foundation of China (Grant No. 61801112, 61471117, 61601281), the Natural Science Foundation of Jiangsu Province (Grant No. BK20180357), the Open Program of State Key Laboratory of Millimeter W aves at Southeast University (Grant No. Z201804). (Corr esponding author: P eng Chen) P . Chen and Z. Cao are with the State Key Laboratory of Millimeter W av es, Southeast Univ ersity , Nanjing 210096, China (email: {chenpengseu, caozx}@seu.edu.cn). Z. Chen is with the School of Electronic and Information, Shanghai Dianji Univ ersity , Shanghai 201306, China (email: chenzm@sdju.edu.cn). X. W ang is with the Department of Electrical and Computer Engineering, W estern Uni versity , Canada (e-mail: xianbin.wang@uwo.ca). to impro ve the target estimation performance. In the distrib uted MIMO radar , the distances between antennas are significant, so the spatial di versity of target’ s radar cross section (RCS) provided by the different view-angles of antennas can be used to improv e the target detection performance [ 4 , 5 ]. In general, the operation of distributed MIMO radar could be challenging due to the coordination and signal exchange among dif ferent antennas. Therefore, in this paper , a colocated MIMO radar system is in vestigated to estimate the directions of arriv al (DO As) for multiple targets. T raditionally , the DO A estimation can be achie ved based on the discrete Fourier transform (DFT) of the receiv ed signal in the spatial domain [ 6 ], but the resolution of such technique is too low to estimate multiple targets using one beam. The max- imum likelihood-based and the subspace-based methods hav e been proposed to improve the DO A estimation performance, including multiple signal classification (MUSIC) method [ 7 , 8 ], Root-MUSIC method [ 9 ], and estimating signal parameters via rotational in variance techniques (ESPRIT) method [ 10 ]. Additionally , the beamspace-based methods hav e also been proposed for DO A estimation [ 11 ]. For e xample, a beamspace design method is proposed in [ 12 ] for the DOA estimation in the MIMO radar with colocated antennas; a two-dimensional joint transmit array interpolation and beamspace design for planar array mono-static MIMO radar is proposed in [ 13 ] for DO A estimation via tensor modeling; a transmit beamspace energy focusing method is proposed in [ 14 ] for MIMO radar with application to direction finding. Moreover , a combined Capon and approximate maximum likelihood (CAML) method is proposed in [ 15 ] for the estimation of target locations and amplitudes in MIMO radar . The tensor algebra and multidi- mensional harmonic retriev al are inv estigated for the DOA estimation of MIMO radar [ 16 ], and an iterati ve adaptive Kronecker beamformer for MIMO radar is proposed in [ 17 ]. Howe ver , in the subspace-based DOA estimation methods, only the po wer of recei ved signals from targets are exploited to establish the target and noise subspaces. T o exploit the target sparsity in the spatial domain, com- pressed sensing (CS)-based methods are utilized to estimate DO A [ 18 ]–[ 25 ]. F or example, in [ 11 ], an iterative adaptiv e approach (IAA), maximum likelihood-based IAA (IAA-ML) and multi-snapshot sparse Bayesian learning (M-SBL) are giv en for the beamforming design based on the sparsity . The Bayesian approach together with expectation-maximization (EM) is used in M-SBL to realize the user parameter-free A CCEPTED BY IEEE TRANSA CTIONS ON SIGNAL PR OCESSING 2 method. In [ 26 ], both the SBL and the relev ance vector machine (R VM) are proposed, and the sparse reconstruction theory based on SBL is dev eloped. In [ 27 ], Bayesian com- pressiv e sensing (BCS) is dev eloped for the sparse signal reconstruction with the CS measurements. In the CS-based method, the DO A estimation performance can be improved by the dense sampling grids. Howe ver , both the computational complexity and the mutual coherence between the columns in the dictionary are increased by the dense sampling grids. T o improve the DOA estimation performance without the dense sampling grids, the of f-grid DOA estimation method is proposed in [ 28 ]. T o further impro ve the sparse estimation performance, an off-grid sparse Bayesian inference (OGSBI) method is first proposed in [ 29 ] for the DOA estimation. Then, by solving a specific polynomial in the off-grid DOA esti- mation problem, a Root-SBL method with low computational complexity is proposed in [ 30 ]. In [ 31 ], the perturbed SBL- based algorithm is proposed for the DOA estimation. A dic- tionary learning algorithm for off-grid sparse reconstruction is proposed in [ 32 ]. In [ 33 ], a grid ev olution method is proposed to refine the grids for the SBL-based DO A estimation. Howe ver , in the practical MIMO radar system, the mutual coupling effect between antennas cannot be ignored [ 34 , 35 ]. Therefore, the DO A estimation methods with the unkno wn mutual coupling ef fect ha ve been proposed [ 36 ]–[ 38 ]. Usu- ally , the mutual coupling ef fects among the antennas can be characterized by a mutual coupling matrix, which is a symmetric T oeplitz matrix [ 39 ]–[ 41 ]. Ho we ver , in the present papers, the effects of both off-grid in the CS-based method and the mutual coupling among antennas hav e not been considered simultaneously , especially , for the methods based on the Bayesian theory . In this paper , the DO A estimation problem in the MIMO radar system with unknown mutual coupling effect is in ves- tigated. Different from the present methods, a novel sparse- based system model considering both the off-grid gap and the unknown mutual coupling ef fect is formulated. Then, a nov el estimation method named SBL with the mutual cou- pling (SBLMC) is proposed, where an EM-based method is established to iterativ ely estimate all the unknown parameters including the noise variance, the mutual coupling vectors, the off-grid vector and the v ariance v ector of scattering coef fi- cients. Additionally , we theoretically derive the prior distri- butions for all the unknown parameters including the target scattering coefficients, the mutual coupling v ectors, the of f- grid v ector and the noise v ariance. Then, the proposed SBLMC is compared with state-of-the-art methods. T o summarize, we make the contributions as follows: • The Sparse-based model for MIMO radar with un- known mutual coupling effect: Considering both the off-grid effect and mutual coupling effect, a nov el system model of MIMO radar is formulated by e xploiting the target sparsity in the spatial domain, so the DOA esti- mation problem is con verted into a sparse reconstruction problem. • The SBL-based method for DOA estimation with unknown mutual coupling effect: A novel SBL-based method (SBLMC) is proposed for the DOA estimation in T argets Colocated ΜΙΜΟ radar system ! d T θ κ ! θ κ d R Transmitter Receiver Fig. 1. The MIMO radar system for DOA estimation. the MIMO radar system with unknown mutual coupling effect and off-grid effect. By estimating all the unkno wn parameters iterati vely , the better estimation performance can be achiev ed than state-of-the-art methods. • The theoretical expr essions for all unknown parame- ters in the SBL-based method: In the proposed SBL- based method (SBLMC), the estimation expressions for all unknown parameters including the noise variance, the mutual coupling vectors, the variance vector of scattering coefficients and the off-grid vector are all theoretically deriv ed. The remainder of this paper is organized as follows. The MIMO radar model for DO A estimation is elaborated in Sec- tion II . The proposed DOA estimation method with unknown mutual coupling, i.e., SBLMC, is presented in Section III . Section IV gives the simulation results. Finally , Section V concludes the paper . Notations: Matrices are denoted by capital letters in bold- face (e.g., A ), and vectors are denoted by lowercase letters in boldface (e.g., a ). I N denotes an N × N identity matrix. E {·} denotes the expectation operation. C N ( a , B ) denotes the complex Gaussian distribution with the mean being a and the variance matrix being B . k · k F , k · k 2 , ⊗ , T r {·} , v ec {·} , ( · ) ∗ , ( · ) T and ( · ) H denote the Frobenius norm, the ` 2 norm, the Kronecker product, the trace of a matrix, the vectorization of a matrix, the conjugate, the matrix transpose and the Hermitian transpose, respecti vely . C M × N denotes the set of M × N matrices with the entries being complex numbers. R{ a } denotes the real part of complex value a . For a vector a , [ a ] n denotes the n -th entry of a , and diag { a } denotes a diagonal matrix with the diagonal entries from a . For a matrix A , [ A ] n denotes the n -th column of A , and diag { A } denotes a vector with the entries from the diagonal entries of A . I I . M I M O R A DA R M O D E L F O R D OA E S T I M A T I O N As shown in Fig. 1 , we consider a colocated MIMO radar system, where M transmitting antennas and N receiving an- tennas are adopted. In the MIMO radar system, the orthogonal signals are transmitted by the antennas, and the wa veform in the m -th ( m = 0 , 1 , . . . , M − 1 ) transmitting antenna is s m ( t ) . Assuming that K far-field point targets in the same range cell are detected, we will consider the DO A estimation problem for these targets. The angle of the k -th ( k = 0 , 1 , . . . , K − 1 ) A CCEPTED BY IEEE TRANSA CTIONS ON SIGNAL PR OCESSING 3 target is denoted as θ k . Therefore, under the assumption of narro wband signals, the receiv ed signals during the p -th ( p = 0 , 1 , . . . , P − 1 , and P denotes the number of pulses) pulse can be expressed as y p ( t ) = K − 1 X k =0 γ k,p C R b ( θ k ) [ C T a ( θ k )] T s ( t − τ T − τ R ) + v p ( t ) ( τ T + τ R ≤ t ≤ τ T + τ R + T P ) , (1) where T P denotes the pulse duration, v p ( t ) ,  v p, 0 ( t ) , v p, 1 ( t ) , . . . , v p,N − 1 ( t )  T denotes the additiv e white Gaussian noise (A WGN), and γ k,p denotes the scattering coefficient of the k -th target during the p -th pulse. τ T denotes the signal propagation time delay between the transmitter and the range cell, and τ R denotes the delay between the range cell and the receiv er . The receiv ed signals and transmitted signals are respectiv ely defined as y p ( t ) ,  y 0 ( t ) , y 1 ( t ) , . . . , y N − 1 ( t )  T , (2) s ( t ) ,  s 0 ( t ) , s 1 ( t ) , . . . , s M − 1 ( t )  T . (3) The steering vectors of the transmitter and receiv er are respec- tiv ely denoted as a ( θ ) , h 1 , e j 2 π d T λ sin θ , . . . , e j 2 π ( M − 1) d T λ sin θ i T , (4) b ( θ ) , h 1 , e j 2 π d R λ sin θ , . . . , e j 2 π ( N − 1) d R λ sin θ i T , (5) where d T and d R denote the distance separation between two neighboring antennas in the transmitter and receiver , respectiv ely , and λ denotes the wa velength. C T ∈ C M × M and C R ∈ C N × N denotes the mutual coupling matrices in the transmitter and recei ver , respectiv ely . The mutual coupling ma- trix C T is a symmetric T oeplitz matrix, and can be expressed as [ 41 ] C T =      1 c T , 1 . . . c T ,M − 1 c T , 1 1 . . . c T ,M − 2 . . . . . . . . . . . . c T ,M − 1 . . . c T , 1 1      , (6) where c T ,m denotes the m -th entry of a vector c T ,  c T , 0 , c T , 1 , . . . , c T ,M − 1  T . Alternativ ely , the entry of C T at the m -th row and m 0 -th column can be also written as C T ,m,m 0 = ( 1 , m = m 0 c T , | m − m 0 | , otherwise . (7) Using the same method, we can obtain the expression of C R . Since the orthogonal signals are adopted in the transmitting antennas, we can use M matched filters corresponding to the M orthogonal signals to distinguish the orthogonal signals. W e estimate the parameters of targets at a specific range cell, so the delays τ T and τ R are omitted. Therefore, passing the m -th matched filter (designed for the m -th signal) [ 1 ], the receiv ed signals r p ( t ) from the same range cell are sampled at T P and obtained as r p,m , K − 1 X k =0 γ k,p C R b ( θ k ) [ C T a ( θ k )] T         R s 0 ( t ) s H m ( t ) dt . . . R s m ( t ) s H m ( t ) dt . . . R s M − 1 ( t ) s H m ( t ) dt         | {z } e M m +         R v p, 0 ( t ) s H m ( t ) dt . . . R v p,m ( t ) s H m ( t ) dt . . . R v p,M − 1 ( t ) s H m ( t ) dt         | {z } n p,m (8) = K − 1 X k =0 γ k,p C R b ( θ k ) [ C T a ( θ k )] T e M m + n p,m = K − 1 X k =0 γ k,p [ C T a ( θ k )] m C R b ( θ k ) + n p,m , where e M m is a M × 1 vector with the m -th entry being 1 and others entries being zeros, and n p,m is the additiv e noise. Collect r p,m into a matrix, and we can obtain R p ,      r T p, 0 r T p, 1 . . . r T p,M − 1      = K − 1 X k =0 γ k,p C T a ( θ k ) [ C R b ( θ k )] T + N p (9) where the noise matrix is defined as N p ,  n p, 0 , n p, 1 , . . . , n p,M − 1  T . V ectorizing the receiving signal matrix into a vector r p , v ec { R p } , we can obtain r p = K − 1 X k =0 γ k,p v ec n C T a ( θ k ) [ C R b ( θ k )] T o + n p (10) = K − 1 X k =0 γ k,p [ C R b ( θ k )] ⊗ [ C T a ( θ k )] + n p , where n p , v ec { N p } . Alternativ ely , the recei ved signal r p can be also re written into a matrix form r p = ∆ γ p + n p , (11) where γ p ,  γ p, 0 , γ p, 1 , . . . , γ p,K − 1  T , ∆ ,  δ 0 , δ 1 , . . . , δ K − 1  , and δ k , [ C R b ( θ k )] ⊗ [ C T a ( θ k )] (12) = [ C R ⊗ C T ] [ b ( θ k ) ⊗ a ( θ k )] . By defining C , C R ⊗ C T and d ( θ k ) = b ( θ k ) ⊗ a ( θ k ) , we hav e ∆ = C  d ( θ 0 ) , d ( θ 1 ) , . . . , d ( θ K − 1 )  = C D , (13) A CCEPTED BY IEEE TRANSA CTIONS ON SIGNAL PR OCESSING 4 where D ,  d ( θ 0 ) , d ( θ 1 ) , . . . , d ( θ K − 1 )  . Therefore, the re- ceiv ed signal with mutual coupling ef fect can be formulated by the following model r p = C D γ p + n p . (14) T o simplify the formula with the mutual coupling matrix in ( 14 ), we will use the follo wing lemma: Lemma 1. F or complex symmetric T oeplitz matrix A = T o eplitz { a } ∈ C M × M and comple x vector c ∈ C M × 1 , we have [ 41 ]–[ 43 ] Ac = Qa , (15) wher e a is a vector formed by the first r ow of A , and Q = Q 1 + Q 2 with the p -th ( p = 0 , 1 , . . . , M − 1 ) r ow and q -th ( q = 0 , 1 , . . . , M − 1 ) column entries being [ Q 1 ] p,q = ( c p + q , p + q ≤ M − 1 0 , otherwise , (16) [ Q 2 ] p,q = ( c p − q , p ≥ q ≥ 1 0 , otherwise . (17) Based on Lemma 1 , δ k can be rewritten as δ k = [ C R b ( θ k )] ⊗ [ C T a ( θ k )] (18) = [ Q b ( θ k ) c R ] ⊗ [ Q a ( θ k ) c T ] = [ Q b ( θ k ) ⊗ Q a ( θ k )] c , where c , c R ⊗ c T , and the m -th entry of c T and the n -th entry of c R respectiv ely are [ c T ] m = ( 1 , m = 0 c T ,m , otherwise , (19) [ c R ] n = ( 1 , n = 0 c R ,n , otherwise . (20) Q a ( θ k ) and Q b ( θ k ) can be obtained as Q a ( θ k ) = Q a 1 ( θ k ) + Q a 2 ( θ k ) , (21) Q b ( θ k ) = Q b 1 ( θ k ) + Q b 2 ( θ k ) , (22) where the p -th row and q -th column entries of Q a 1 ( θ k ) , Q a 2 ( θ k ) , Q b 1 ( θ k ) and Q b 2 ( θ k ) respectiv ely are [ Q a 1 ] p,q = ( [ a ( θ k )] p + q , p + q ≤ M − 1 0 , otherwise , (23) [ Q a 2 ] p,q = ( [ a ( θ k )] p − q , p ≥ q ≥ 1 0 , otherwise , (24) [ Q b 1 ] p,q = ( [ b ( θ k )] p + q , p + q ≤ N − 1 0 , otherwise , (25) [ Q b 2 ] p,q = ( [ b ( θ k )] p − q , p ≥ q ≥ 1 0 , otherwise . (26) Therefore, we have ∆ = Q [ I K ⊗ c ] , (27) where Q ,  Q b ( θ 0 ) ⊗ Q a ( θ 0 ) , . . . , Q b ( θ K − 1 ) ⊗ Q a ( θ K − 1 )  . (28) Then, the receiv ed signal in ( 14 ) can be rewritten as r p = Q ( I K ⊗ c ) γ p + n p = Q  γ p ⊗ c  + n p . (29) Collect the P pulses into a matrix, and the receiv ed signal can be finally obtained as R = Q  γ 0 ⊗ c , γ 1 ⊗ c , . . . , γ P − 1 ⊗ c  + N = Q ( Γ ⊗ c ) + N , (30) where R ,  r 0 , r 1 , . . . , r P − 1  , N ,  n 0 , n 1 , . . . , n P − 1  , Γ ,  γ 0 , γ 1 , . . . , γ P − 1  , and the u -th row and p -th column of Γ is denoted as Γ u,p . In this paper, we will estimate the DO As from R with the unknown mutual coupling vector c , the target scattering coef ficients Γ and the noise variance σ 2 n . I I I . D O A E S T I M AT I O N M E T H O D W I T H U N K N O W N M U T U A L C O U P L I N G A. The Off-Grid Sparse Model Discretize the angle of detection area into U grids ζ ,  ζ 0 , ζ 1 , . . . , ζ U − 1  , and the u -th discretized angle are denoted as ζ u . Then, a dictionary matrix can be formulated as Ψ ,  Φ ( ζ 0 ) , Φ ( ζ 1 ) , . . . , Φ ( ζ U − 1 )  ∈ C M N × U M N , (31) where Φ ( ζ u ) , Q b ( ζ u ) ⊗ Q a ( ζ u ) , and the space between discretized angles, also kno wn as grid size, is δ , | ζ u +1 − ζ u | . For the dictionary matrix Ψ , the restricted isometry property (RIP) with constant δ $ is defined as [ 44 ] (1 − δ $ ) k x k 2 2 ≤ k Ψ x k 2 2 ≤ (1 + δ $ ) k x k 2 2 , (32) for all $ -sparse v ector x ( $ = K M N ). If x is $ -sparse and Ψ satisfies δ 2 $ + δ 3 $ < 1 , then x is the unique ` 1 minimizer [ 45 ]. Howe ver , the tight RIP constant of a giv en matrix Ψ is difficult to compute, so we calculate the minimum DO A separation for multiple targets. As described in [ 46 ], in our scenario (the system parameters are given in Section IV ), the minimum DOA separation can be obtained as 9 . 67 °. Howe ver , for the k -th target, the DO A is θ k and is not at the discretized grids exactly , so the sub-matrix for the k -th target can be approximated by Φ ( θ k ) = Φ ( ζ u k + ( θ k − ζ u k )) ≈ " Q b ( ζ u k ) + ( θ k − ζ u k ) ∂ Q b ( ζ ) ∂ ζ     ζ = ζ u k # ⊗ " Q a ( ζ u k ) + ( θ k − ζ u k ) ∂ Q a ( ζ ) ∂ ζ     ζ = ζ u k # ≈ Φ ( ζ u k ) + ( θ k − ζ u k ) Ω ( ζ u k ) , (33) A CCEPTED BY IEEE TRANSA CTIONS ON SIGNAL PR OCESSING 5 a b α n c d β X e 1 f 1 υ T e 2 f 2 υ R c T c R δ ν R Fig. 2. Graphical model of SBLMC (rectangles are the hyperparameters, circles are the radar parameters and signals). where ζ u k is the discretized grid angle nearest to the target DO A θ k , and we define the first order of deriv ativ e as Ω ( ζ u k ) , Q b ( ζ u k ) ⊗ ∂ Q a ( ζ ) ∂ ζ     ζ = ζ u k + ∂ Q b ( ζ ) ∂ ζ     ζ = ζ u k ⊗ Q a ( ζ u k ) . (34) By formulating a sparse matrix X ∈ C U × P with the columns x p ( p = 0 , 1 , . . . , P − 1 ) having the same support set, i.e., X ,  x 0 , x 1 , . . . , x P − 1  , the recei ved signal can be approximated by a sparse-based model R ≈ [ Ψ + Ξ (diag { ν } ⊗ I M N )] ( X ⊗ c ) + N , (35) where Ξ ,  Ω ( ζ 0 ) , Ω ( ζ 1 ) , . . . , Ω ( ζ U − 1 )  , and the u -th sub- matrix can be also written as Ξ u , Ω ( ζ u ) to simplify the notation. The u -th row and p -th column of sparse matrix X ∈ C U × P is X u,p = ( Γ u k ,p , u = u k 0 , otherwise , (36) and the u -th entry of the off-grid vector ν ∈ R U × 1 is ν u = ( θ k − ζ u k , u = u k 0 , otherwise . (37) Finally , by absorbing the approximation into the additiv e noise, the off-grid sparse model in the MIMO radar with unknown mutual coupling effect can be described by a sparse model R = Υ ( ν )( X ⊗ c R ⊗ c T ) + N , (38) where Υ ( ν ) , Ψ + Ξ (diag { ν } ⊗ I M N ) . With the re- ceiv ed signal R , we can estimate the target DO As θ k ( k = 0 , 1 , . . . , K − 1 ) with the unkno wn parameters including the sparse matrix X , the off-grid vector ν , and the mutual coupling vectors c T and c R . The DO As can be obtained from the support sets of X , the tar get scattering coef ficients are obtained from the nonzero entries of X , and the mutual coupling matrices can be obtained from c T and c R . B. Sparse Bayesian Learning-Based DO A Estimation Method In this paper , we propose an SBL-based method to estimate the target DOAs with unknown mutual coupling effect, and the proposed method is named as SBL with the mutual coupling (SBLMC). The graphical model of SBLMC is gi ven in Fig. 2 , where the unkno wn parameters are determined by the hyperparameters, and the recei ved signal S is determined by radar parameters and signals. T o realize the SBLMC algorithm, the distribution assumptions are given as follows. W e assume that the additiv e noise is white (circularly symmetric) Gaussian noise with the noise v ariance being σ 2 n , and the distribution of noise can be expressed as p ( N | σ 2 n ) = U − 1 Y u =0 C N ( n p | 0 M N × 1 , σ 2 n I M N ) , (39) where the complex Gaussian distribution is defined as C N ( x | a , Σ ) = 1 π N det( Σ ) e − ( x − a ) H Σ − 1 ( x − a ) . (40) When the noise v ariance σ 2 n is unkno wn, by defining a hyperparamter , i.e., the precision , α n , σ − 2 n , a Gamma distribution can be adopted to describe the inv erse of noise variance p ( α n ) = G ( α n ; a, b ) , (41) where a and b are the hyperparameters for α n , and G ( α n ; a, b ) , Γ − 1 ( a ) b a α a − 1 n e − bα n , (42) Γ( a ) , Z ∞ 0 x a − 1 e − x dx. (43) Note that the Gamma distribution α n ∼ G ( α n ; a, b ) is a conjugate prior of the Gaussian distrib ution given mean with unknown v ariance x ∼ N ( x | 0 , α − 1 n ) , so the posterior distri- bution p ( α n | x ) also follows a Gamma distribution. Therefore, the assumption of Gamma distribution for the pr ecision α n can simplify the following analysis. When the scattering coef ficients Γ are independent among snapshots, we can also assume that the sparse matrix X follows a Gaussian distribution p ( X | Λ x ) = P − 1 Y p =0 C N ( x p | 0 U × 1 , Λ x ) , (44) where Λ x ∈ R U × U is a diagonal matrix with the u -th diagonal entry being σ 2 x,u . Usually , the sparseness prior is the Laplace density function [ 26 , 27 ], but the Laplace prior is not conjugate to the Gaussian likelihood. Therefore, for simplification, we use the Gaussian prior for the sparse matrix X and obtain the estimation expressions in closed form. Then, by defining the pr ecision β ,  β 0 , β 1 , . . . , β U − 1  T and β u , σ − 2 x,u , we have the following Gamma prior for β p ( β ; c, d ) = U − 1 Y u =0 G ( β u ; c, d ) , (45) where c and d are the hyperparmaters for β . A CCEPTED BY IEEE TRANSA CTIONS ON SIGNAL PR OCESSING 6 Similarly , when the mutual coupling coefficients are inde- pendent with antennas, we can also assume that the mutual coupling vectors c T and c R follow Gaussian distributions p ( c T | Λ T ) = M − 1 Y m =0 C N ( c T ,m | 0 , σ 2 T ,m ) , (46) p ( c R | Λ R ) = N − 1 Y n =0 C N ( c R ,n | 0 , σ 2 R ,n ) , (47) where Λ T ∈ R M × M is a diagonal matrix with the m -th diagonal entry being σ 2 T ,m , and Λ R ∈ R N × N is a diagonal matrix with the n -th diagonal entry being σ 2 R ,n . Define the pr ecisions ϑ T ,  ϑ T , 0 , ϑ T , 1 , . . . , ϑ T ,M − 1  T ( ϑ T ,m , σ − 2 T ,m ) and ϑ R ,  ϑ R , 0 , ϑ R , 1 , . . . , ϑ R ,N − 1  T ( ϑ R ,n , σ − 2 R ,n ). Then, we can hav e the following Gamma distrib utions p ( ϑ T ; e 1 , f 1 ) = M − 1 Y m =0 G ( ϑ T ,m ; e 1 , f 1 ) , (48) p ( ϑ R ; e 2 , f 2 ) = N − 1 Y n =0 G ( ϑ R ,n ; e 2 , f 2 ) , (49) where both e 1 and f 1 are the hyperparameters for ϑ T , and both e 2 and f 2 are the hyperparameters for ϑ R . Usually , we can choose the follo wing values a = b = c = d = e 1 = f 1 = e 2 = f 2 = 10 − 2 as the hyperparameters. As shown in [ 26 ], the small values for hyperparameters are chosen and not sensitiv e to specific values [ 47 ]. The of f-grid parameter ν follows a uniform prior distribu- tion, and the distribution of the u -th entry ν u can be expressed as p ( ν u ; δ ) = U ν u  − 1 2 δ, 1 2 δ  , (50) where we have U x ([ a, b ]) , ( 1 b − a , a ≤ x ≤ b 0 , otherwise . (51) The relationships between parameters are shown in Fig. 2 . T o estimate the DOAs, we can formulate the following prob- lem to maximize the posterior probability with the received signal ˆ X = arg max X p ( X | R ) , (52) where we use a set X ,  X , ν , c T , c R , σ 2 n , β  to contain all the unkno wn parameters. Howe ver , the problem of posterior probability cannot be solv ed directly , so an EM method is adopted to realize the sparse Bayesian learning. T o obtain the posterior distribution of X , we first calculate the joint distribution for all the parameters p ( R , X ) = p ( R | X ) p ( X | β ) p ( c T | ϑ T ) p ( c R | ϑ R ) p ( α n ) p ( β ) p ( ϑ T ) p ( ϑ R ) p ( ν ) . (53) Therefore, with the parameters α n , β , ϑ T , ϑ R , ν , c T and c R , the posterior for X can be obtained as p ( X | R , ν , c T , c R , α n , β , ϑ T , ϑ R ) = p ( R , X ) p ( R , ν , c T , c R , α n , β , ϑ T , ϑ R ) = p ( R | X ) p ( X | β ) p ( R | ν , c T , c R , α n , β , ϑ T , ϑ R ) , (54) where p ( R | X ) and ( X | β ) can be calculated as p ( R | X ) = P − 1 Y p =0 C N ( r p | Υ ( ν )( x p ⊗ c ) , α − 1 n I M N ) = P − 1 Y p =0 α M N n π M N e − α n k r p − Υ ( ν )( x p ⊗ c ) k 2 2 , (55) p ( X | β ) = P − 1 Y p =0 C N ( x p | 0 U × 1 , diag { β } − 1 ) = P − 1 Y p =0 U − 1 Y u =0 β u ! 1 π U e − x H p diag { β } x p . (56) Since the denominator in ( 54 ) is not a function of X , the posterior distribution of X can be simplified as p ( X | R , ν , c T , c R , α n , β , ϑ T , ϑ R ) ∝ p ( R | X ) p ( X | β ) . (57) Both p ( R | X ) and p ( X | β ) are Gaussian functions, so the posterior for X can be also expressed as a Gaussian function p ( X | R , ν , c T , c R , α n , β , ϑ T , ϑ R ) ∝ p ( R | X ) p ( X | β ) ∝ P − 1 Y p =0 e − α n k r p − Υ ( ν )( I U ⊗ c ) x p k 2 2 − x H p diag { β } x p , P − 1 Y p =0 C N ( x p | µ p , Σ X ) , (58) where the mean µ p and cov ariance matrix Σ X are µ p = α n Σ X ( I U ⊗ c ) H Υ H ( ν ) r p , (59) Σ X =  α n ( I U ⊗ c ) H Υ H ( ν ) Υ ( ν )( I U ⊗ c ) + diag { β }  − 1 . (60) and we use µ p,u to denote the u -th entry of µ p . T o calculate Σ X and µ p , we need to estimate the mu- tual coupling vectors c T and c R , the off-grid parameter ν , and the precisions α n and β . W e can use the max- imum posterior probability (MAP) method to maximize p ( ν , c T , c R , α n , β , ϑ T , ϑ R | R ) . W e hav e p ( ν , c T , c R , α n , β , ϑ T , ϑ R | R ) p ( R ) = p ( ν , c T , c R , α n , β , ϑ T , ϑ R , R ) , (61) so maximizing p ( ν , c T , c R , α n , β , ϑ T , ϑ R | R ) is equiv alent to maximizing p ( ν , c T , c R , α n , β , ϑ T , ϑ R , R ) . The EM method can be used to solve the MAP estimation by treating X as a hidden variable. Before estimating the parameters, we will A CCEPTED BY IEEE TRANSA CTIONS ON SIGNAL PR OCESSING 7 first obtain the likelihood function under the expectation with respect to the posterior of X L ( ν , c T , c R , α n , β , ϑ T , ϑ R ) , E X | R , ν , c T , c R ,α n , β , ϑ T , ϑ R { ln p ( X , ϑ T , ϑ R , R ) } . (62) T o simplify the notation, we just use E {·} to represent E X | R , ν , c T , c R ,α n , β , ϑ T , ϑ R {·} , so the likelihood function can be simplified as L ( ν , c T , c R , α n , β , ϑ T , ϑ R ) = E  ln p ( R | X ) p ( X | β ) p ( c T | ϑ T ) p ( c R | ϑ R ) p ( α n ) p ( β ) p ( ϑ T ) p ( ϑ R ) p ( ν )  . (63) In the follo wing contents, we will gi ve the expressions for all the remaining unknown parameters. 1) For the mutual coupling vector c T , ignoring terms inde- pendent thereof, we can obtain the following likelihood function L ( c T ) = E { ln p ( R | X , ν , c T , c R , α n ) p ( c T | ϑ T ) } = E ( ln P − 1 Y p =0 C N ( r p | Υ ( ν )( x p ⊗ c ) , α − 1 n I M N ) ) + ln M − 1 Y m =0 C N ( c T ,m | 0 , ϑ − 1 T ,m ) ∝ − α n P T r  ( I U ⊗ c ) H Υ H ( ν ) Υ ( ν )( I U ⊗ c ) Σ X  − P − 1 X p =0 α n k r p − Υ ( ν )( µ p ⊗ c ) k 2 2 − M − 1 X m =0 ϑ T ,m | c T ,m | 2 . (64) In Appendix A , the details about calculating ∂ L ( c T ) ∂ c T are giv en. By setting ∂ L ( c T ) ∂ c T = 0 , c T can be obtained as c T = H − 1 T z T , (65) where H T = P − 1 X p =0 α n T H T Υ H ( ν ) Υ ( ν )( µ p ⊗ c R ⊗ I M ) + α n P G H T U − 1 X p =0 U − 1 X k =0 Υ H p ( ν ) Υ k ( ν )Σ X ,k,p ! H ( c R ⊗ I M ) + diag { ϑ T } , (66) and z T = P − 1 X p =0 α n T H T Υ H ( ν ) r p , (67) T T ,  µ p ⊗ c R ⊗ e M 0 , . . . , µ p ⊗ c R ⊗ e M M − 1  , (68) G T ,  c R ⊗ e M 0 , c R ⊗ e M 1 , . . . , c R ⊗ e M M − 1  . (69) 2) For the mutual coupling vector c R , using the same method with c T , we can obtain c R = H − 1 R z R , (70) where H R = P − 1 X p =0 α n T H R Υ H ( ν ) Υ ( ν )( µ p ⊗ I N ⊗ c T ) + α n P G H R U − 1 X p =0 U − 1 X k =0 Υ H p ( ν ) Υ k ( ν )Σ X ,k,p ! H ( I N ⊗ c T ) + diag { ϑ T } , (71) and z R = P − 1 X p =0 α n T H R Υ H ( ν ) r p , (72) T R ,  µ p ⊗ e N 0 ⊗ c T , . . . , µ p ⊗ e N N − 1 ⊗ c T  , (73) G R ,  e N 0 ⊗ c T , e N 1 c T , . . . , e N N − 1 c T  . (74) 3) For the pr ecision β of scattering coef ficients, ignoring terms independent thereof, we can obtain the likelihood function L ( β ) = E { ln p ( X | β ) p ( β ) } = E ( ln P − 1 Y p =0 C N ( x p | 0 U × 1 , Λ x ) ) + ln U − 1 Y u =0 G ( β u ; c, d ) . (75) By setting ∂ L ( β ) ∂ β = 0 , the u -th entry of β can be obtained as β u = P + c − 1 d + P Σ X ,u,u + P P − 1 p =0 | µ u,p | 2 . (76) 4) For the pr ecision α n of noise, ignoring terms indepen- dent thereof, we can obtain the likelihood function L ( α n ) = E { ln p ( R | X , ν , c T , c R , α n ) p ( α n ) } = E ( ln P − 1 Y p =0 C N  r p | Υ ( ν )( x p ⊗ c R ⊗ c T ) , σ 2 n I  ) + ln G ( α n ; a, b ) . (77) By setting ∂ L ( α n ) ∂ α n = 0 , we can obtain α n = M N P + a − 1 P N 1 + N 2 + b , (78) where N 1 , T r { ( I U ⊗ c ) H Υ H ( ν ) Υ ( ν )( I U ⊗ c ) Σ X } , (79) N 2 , k R − Υ ( ν )( µ ⊗ c ) k 2 F , (80) µ ,  µ 0 , µ 1 , . . . , µ P − 1  . (81) 5) For the precision ϑ T of mutual coupling vector , ignoring terms independent thereof, we can obtain the likelihood function L ( ϑ T ) = E { ln p ( c T | ϑ T ) p ( ϑ T ) } = E ( ln M − 1 Y m =0 C N ( c T ,m | 0 , σ 2 T ,m ) ) + ln M − 1 Y m =0 G ( ϑ T ,m ; e 1 , f 1 ) . (82) A CCEPTED BY IEEE TRANSA CTIONS ON SIGNAL PR OCESSING 8 By setting ∂ L ( ϑ T ) ∂ ϑ T = 0 , we can obtain the m -th entry of ϑ T as ϑ T ,m = e 1 f 1 + c H T ,m c T ,m . (83) 6) For the pr ecision ϑ R of mutual coupling vector , using the same method, we can obtain the n -th entry of ϑ R as ϑ R ,n = e 2 f 2 + c H R ,n c R ,n . (84) 7) For off-grid ν , ignoring terms independent thereof, we can obtain the likelihood function L ( ν ) = E { ln p ( R | X , ν , c T , c R , α n ) p ( ν ) } . (85) By setting ∂ L ( ν ) ∂ ν = 0 , we can obtain ν = H − 1 z , (86) where the entry of the u -th ro w and m -column in H ∈ R U × U is H u,m = R ( P Σ X ,u,m + P − 1 X p =0 µ H p,m µ p,u ! c H Ξ H m Ξ u c ) , (87) and the u -th entry of z ∈ R U × 1 is z u = P − 1 X p =0 R n  r p − Ψ ( µ p ⊗ c )  H Ξ u µ u,p c o − U − 1 X m =0 R  P Σ X ,u,m c H Ψ H m Ξ u c  . (88) The details to obtain ν are gi ven in Appendix B . In Algorithm 1 , we show the details about the proposed method SBLMC to estimate the DO As with unknown mutual coupling effect. In the proposed SBLMC algorithm, after the iterations, we can obtain the spatial spectrum P X of the sparse matrix X from the receiv ed signal R . Then, by searching all the values of P X , the corresponding peak values can be found. By selecting positions of peak values corresponding to the K maximum v alues, we can estimate the DOAs of targets, where we use ζ + ν as the discretized angle vector . I V . S I M U L A T I O N R E S U L T S In this section, the simulation results about the proposed method for DOA estimation in the MIMO radar system are giv en, and the simulation parameters are giv en in T able I . For the proposed SBLMC algorithm, the maximum iteration is N iter = 10 3 and the stop threshold is λ = 10 − 3 . All experiments are carried out in Matlab R2017b on a PC with a 2.9 GHz Intel Core i5 and 8 GB of RAM. Matlab codes hav e been made a v ailable online at https://sites.google.com/ site/chenpengdsp/publications . First, we show the estimated spatial spectrum of 3 targets. As shown in Fig. 3 , 3 present methods including off-grid sparse Bayesian inference (OGSBI) [ 29 ], Bayesian compres- siv e sensing (BCS) [ 27 ] and MUSIC [ 8 ], hav e compared with the proposed SBLMC method. With the mutual coupling effect, the traditional MUSIC method cannot achiev e better Algorithm 1 SBLMC algorithm to estimate the DO As with unknown mutual coupling effect 1: Input: receiv ed signal R , dictionary matrix Ψ , the first order deri v ati ve of dictionary matrix Ξ , the number of pulses P , the maximum of iteration N iter , stop threshold λ th . 2: Initialization: c T = ϑ T = [1 , 0 1 × ( M − 1) ] T , c R = ϑ R = [1 , 0 1 × ( N − 1) ] T , α n = 1 , the hyperparameters a = b = c = d = e 1 = f 1 = e 2 = f 2 = 10 − 2 , ν = 0 U × 1 , β = 1 U × 1 , i iter = 1 , λ = k R k 2 F . 3: while i iter ≤ N iter or λ ≤ λ th do 4: Υ ( ν ) ← Ψ + Ξ (diag { ν } ⊗ I M N ) . 5: Obtain µ p ( p = 0 , 1 , . . . , P − 1 ) and Σ X from ( 59 ) and ( 60 ), respectiv ely . 6: Obtain the spatial spectrum P X = R { diag { Σ X }} + 1 P P − 1 X p =0 | µ p | 2 , (89) where | µ p | ,  | µ p, 0 | , | µ p, 1 | , . . . , | µ p,U − 1 |  T . 7: β 0 ← β , and update β from ( 76 ). 8: Update c T and c R from ( 65 ) and ( 70 ), respectively . 9: Update ϑ T and ϑ R from ( 83 ) and ( 84 ), respectively . 10: Estimate ν from ( 86 ). 11: Update α n from ( 78 ). 12: if i iter > 1 then 13: λ = k β − β 0 k 2 k β 0 k 2 . 14: end if 15: i iter ← i iter + 1 . 16: end while 17: Output: the spatial spectrum P X , and the DOAs ( ζ + ν ) can be obtained from the positions of peak values in P X . -80 -60 -40 -20 0 20 40 60 80 DOA (deg) 10 -4 10 -3 10 -2 10 -1 10 0 10 1 Spatial spectrum SBLMC Target DOA OGSBI BCS MUSIC Fig. 3. The spatial spectrum for DOA estimation. A CCEPTED BY IEEE TRANSA CTIONS ON SIGNAL PR OCESSING 9 -80 -60 -40 -20 0 20 40 60 80 DOA (deg) 10 -4 10 -3 10 -2 10 -1 10 0 10 1 Spatial spectrum SBLMC Target DOA Capon-like method MUSIC-like method CS-based method (L1 norm ) Fig. 4. The spatial spectrum of the proposed method is compared with that of the present methods in DO A estimation. -20 -10 0 10 20 SNR (dB) -80 -60 -40 -20 0 DOA estimation error (dB) SBLMC (2 o ) OGSBI BCS MUSIC CRLB SBLMC (1 o ) Fig. 5. The DO A estimation performance with different SNRs. -14 -12 -10 -8 -6 -4 -2 Mutual coupling between adjacent antennas (dB) -55 -50 -45 -40 -35 -30 -25 -20 -15 DOA estimation error (dB) SBLMC OGSBI BCS MUSIC Fig. 6. The DOA estimation performance with different mutual coupling effects. T ABLE I S I MU LAT I O N P AR AM E T E RS Parameter V alue The signal-to-noise ratio (SNR) of echo signal 20 dB The number of pulses P 100 The number of transmitting antennas M 10 The number of receiving antennas N 5 The number of targets K 3 The space between antennas d T = d R 0 . 5 wa velength The grid size δ 2 ° The detection DO A range [ − 80 ° , 80 ° ] The hyperparameters a, b, c, d, e 1 , f 1 , e 2 , f 2 10 − 2 The mutual coupling between adjacent antennas − 5 dB 2 4 6 8 10 Grid size (deg) -50 -40 -30 -20 -10 0 10 DOA estimation error (dB) SBLMC OGSBI BCS Fig. 7. The DO A estimation performance with different grid sizes. T ABLE II E S TI MAT E D D OA S Methods T arget 1 T arget 2 T arget 3 T arget DO As 4 . 3075 ° 27 . 0740 ° 49 . 3603 ° SBLMC 4 . 2746 ° 27 . 2441 ° 49 . 4521 ° OGSBI 1 . 8636 ° 25 . 6868 ° 50 . 7775 ° BCS 2 . 0000 ° 26 . 0000 ° 50 . 0000 ° MUSIC 2 . 9929 ° 25 . 7086 ° 50 . 1324 ° performance. Since the present Bayesian methods (OGSBI and BCS) hav e not considered the mutual coupling effect, the estimation performance cannot be further improv ed. The proposed SBLMC considers both off-grid and mutual coupling effects, can achiev e the best spatial spectrum and improv e the DO A estimation performance. In T able II , we gi ve the estimated DO As with different methods. W e use the follo wing expression to measure the estimation performance e , 10 log 10 k ˆ θ − θ k 2 2 ( dB ) , (90) where ˆ θ denotes the estimated DO A vector and θ is the target DO A vector . Both ˆ θ and θ are in rad. The estimation errors of OGSBI, BCS and MUSIC methods are − 25 . 20 dB, − 26 . 78 dB and − 28 . 94 dB, respectively . Since the mutual coupling effect has not been considered, the DO A estimation performance of A CCEPTED BY IEEE TRANSA CTIONS ON SIGNAL PR OCESSING 10 these 3 present methods almost the same. Howe ver , the DOA estimation error of the proposed SBLMC is − 49 . 31 dB, which is significantly better than the present methods. Additionally , the DO A estimation performance of the pro- posed SBLMC method is compared with present methods in Fig. 4 . The CS-based method (L1 norm) is proposed in [ 24 , 48 ], where the DO A estimation problem is con verted into a sparse reconstruction problem via ` 1 minimization. The MUSIC- like method is proposed in [ 41 ], where the mutual coupling effect is considered in the MUSIC-like method. The Capon- like is proposed in [ 15 ] for the DOA estimation in MIMO radar systems. As sho wn in this figure, the proposed method achiev es the best DO A estimation performance in the scenario with unkno wn mutual coupling effect by estimating all the unknown parameters iteratively in the SBL-based method. Howe ver , the disadvantage of the proposed method is higher computational complexity than these present methods. W e also show the DO A estimation performance with differ - ent signal-to-noise ratios (SNRs) in Fig. 5 . As sho wn in this figure, when SNR ≤ − 10 dB, all the methods cannot work well, and the performance is almost the same. When SNR > − 10 dB, the DOA estimation performance of present methods including OGSBI, BCS and MUSIC cannot be improved, and the estimation errors are around − 28 dB. Howe ver , with improving SNR, the estimation performance of SBLMC can also be improv ed, and the final estimation error can be lower than − 50 dB with SNR ≥ 5 dB. W ith the Cramér-Rao lower bound (CRLB) in [ 49 ], Fig. 5 also shows the corresponding CRLB of DO A estimation. As shown in this figure, when SNR > 5 dB, the proposed method can approach CRLB. In Fig. 5 , the curve “SBLMC ( 2 °)” is the SBLMC method with the grid size being δ = 2 ° and the curve “SBLMC ( 1 °)” is that with the grid size being δ = 1 °. From the curves, we can see that the estimation performance of SBLMC can approach the CRLB with the dense sampling grids ( δ = 1 °). Therefore, the reason why the estimation error cannot be further reduced (error floor) is that the spatial spectrum is discretized to find the peak values and the discretized grids cannot be infinitely small. The proposed SBLMC method can improv e the estimation performance, but the improv ement is limited by the grid size. Then, we also show the mutual coupling effect on the DO A estimation in Fig. 6 , where the mutual coupling effect between adjacent antennas is from − 15 dB to − 2 dB. With increasing the mutual coupling effect, the DOA estimation error of BCS is from − 34 dB to − 28 dB. Since the grid effect in the BCS method, decreasing the mutual coupling effect cannot further improv e the estimation performance when the mutual coupling between adjacent antennas is less than − 8 dB. Howe ver , for both OGSBI and MUSIC methods, decreasing the mutual coupling effect can decrease the estimation error from around − 25 dB to around − 50 dB. For the proposed methods, since the mutual coupling vectors c T and c R are estimated, the mutual coupling has limited ef fect on the DOA estimation performance, and the estimation error can be lower than − 50 dB when the mutual coupling between adjacent antennas is less than − 2 dB. W e show the DO A estimation performance with different T ABLE III C O MP UTA T I ON A L T I M E Methods Time (one iteration) Number of iterations T otal time SBLMC 4 . 12 s 139 537 . 71 s OGSBI 2 . 66 s 146 374 . 79 s BCS 0 . 17 s 147 17 . 23 s MUSIC – – 80 . 31 s grid sizes in Fig. 7 , where the space between adjacent dis- cretized angles δ is from 2 ° to 10 °. Since the BCS method has not considered the off-grid effect, the worst estimation performance is achieved in these 3 methods. Both BCS and OGSBI methods have not considered the mutual coupling, so when the grid size δ is less than 6 °, the estimation performance cannot be impro ved. Howe ver , for the proposed SBLMC, with decreasing the gird size δ from 10 ° to 2 °, the estimation error can be decreased from − 8 dB to − 50 dB. Finally , we compare the computational time of the pro- posed SBLMC method with that of the present 3 methods in T able III . All the methods ha ve not been further optimized to decrease the computational time. Without the additional simplifications, the computational complexities of SBLMC in Step 5 , Step 8 and Step 10 are O ( P U 2 M N + U M 2 N 2 + U 3 ) , O ( P U 2 M N + U M 2 N 2 + U 3 ) and O ( U 3 + P U 2 M N ) , so the computational complexities of SBLMC can be obtained as O ( U 3 + P U 2 M N + U M 2 N 2 ) per iteration and an ad- ditional computational workload of order O ( U 2 M 3 N 3 ) for initialization. T o simplify the representation, with U ≥ M N , the computational complexity of SBLMC can be approximated by O ( P U 3 ) . If U < M N the computational complexity of SBLMC can be approximated by O ( P U M 2 N 2 ) . Therefore, the proposed SBLMC algorithm has the same order of compu- tational complexity with OGSBI [ 29 ]. Since MUSIC algorithm is a continue domain method, we estimate the target DOAs by discretizing the detection range [ − 80 ° , 80 ° ] into 1 . 6 × 10 6 grids. As shown in this table, the BCS method is the fastest among all methods, since the detection angle is discretized by δ = 2 °. The proposed SBLMC is comparable with the OGSBI method, but the DOA estimation performance is much better . The computational time of MUSIC method is determined by the length of discretized angles, and usually is a method with higher computational complexity than BCS. Therefore, the proposed SBLMC method can significantly improv e the estimation performance in the MIMO radar system with both off-grid and mutual coupling effects with the acceptable com- putational complexity . V . C O N C L U S I O N S W e ha ve inv estigated the DOA estimation problem in MIMO radar system with unkno wn mutual coupling effect in this paper . The off-grid problem in the CS-based sparse reconstruction method has also been considered concurrently to improve the DO A estimation performance. The nov el sparse Bayesian learning with mutual coupling (SBLMC) method using EM has been proposed to estimate target DO As. Ad- ditionally , we hav e theoretically deriv ed the prior distributions A CCEPTED BY IEEE TRANSA CTIONS ON SIGNAL PR OCESSING 11 ∂ L ( c T ) ∂ c T = − α n P " c H U − 1 X p =0 U − 1 X k =0 Υ H p ( ν ) Υ k ( ν ) E x,k,p !  c R ⊗ e M 0 , . . . , c R ⊗ e M M − 1  # + P − 1 X p =0 α n [ r p − Υ ( ν )( µ p ⊗ c )] H Υ ( ν )  µ p ⊗ c R ⊗ e M 0 , . . . , µ p ⊗ c R ⊗ e M M − 1  − c H T diag { ϑ T } . (91) U − 1 X m =0 ν m R ( P Σ X ,u,m + P X p =0 µ H p,m µ p,u ! c H Ξ H m Ξ u c ) = P X p =0 R n  r p − Ψ ( µ p ⊗ c )  H Ξ u µ p,u c o − U − 1 X m =0 R  P Σ X ,u,m c H Ψ H m Ξ u c  . (92) for all the unknown parameters including the variance vector of target scattering coefficients, the mutual coupling vectors, the of f-grid v ector and the noise variance. Simulation results confirm that the proposed SBLMC method outperforms the present DOA estimation methods in the MIMO radar sys- tem with the unknown mutual coupling effect. Additionally , the computational complexity of SBLMC is also acceptable. Howe ver , with the same characteristic of the sparse-based super-resolution methods, the minimum DO A separation of the proposed SBLMC method is limited by the radar aperture. Future work will focus on the optimization of MIMO radar system using the SBLMC method for DOA estimation. A P P E N D I X A T H E D E R I V A T I O N O F L I K E L I H O O D F U N C T I O N L ( c T ) The likelihood function L ( c T ) can be rewritten as L ( c T ) ∝ − α n P G 1 ( c T ) − P − 1 X p =0 α n G 2 ( c T ) − G 3 ( c T ) , (93) where G 1 ( c T ) , T r  ( I U ⊗ c ) H Υ H ( ν ) Υ ( ν )( I U ⊗ c ) Σ X  , (94) G 2 ( c T ) , k r p − Υ ( ν )( µ p ⊗ c ) k 2 2 , (95) G 3 ( c T ) , M − 1 X m =0 ϑ T ,m | c T ,m | 2 . (96) W ith the deriv ations of complex vector and matrix, ∂ G 1 ( c T ) ∂ c T is a row vector , and the m -th entry can be calculated as  ∂ G 1 ( c T ) ∂ c T  m = T r  ∂ ( I U ⊗ c ) H Υ H ( ν ) Υ ( ν )( I U ⊗ c ) Σ X ∂ c T ,m  . (97) W e can calculate ∂ ( I U ⊗ c ) H Υ H ( ν ) Υ ( ν )( I U ⊗ c ) Σ X ∂ c T ,m = ∂ ( I U ⊗ c ) H ∂ c T ,m Υ H ( ν ) Υ ( ν )( I U ⊗ c ) Σ X + ( I U ⊗ c ) H Υ H ( ν ) Υ ( ν ) ∂ ( I U ⊗ c ) ∂ c T ,m Σ X = ( I U ⊗ c ) H Υ H ( ν ) Υ ( ν )  I U ⊗ ∂ c R ⊗ c T ∂ c T ,m  Σ X = ( I U ⊗ c ) H Υ H ( ν ) Υ ( ν )  I U ⊗ c R ⊗ e M m  Σ X , (98) where e M m is a M × 1 vector with the m -th entry being 1 and other entries being 0 . Therefore, the the m -th entry can be simplified as  ∂ G 1 ( c T ) ∂ c T  m = c H U − 1 X p =0 U − 1 X k =0 Υ H p ( ν ) Υ k ( ν )Σ X ,k,p ! ( c R ⊗ e M m ) , (99) and we finally have the deriv ation of G 1 ( c T ) as ∂ G 1 ( c T ) ∂ c T = c H U − 1 X p =0 U − 1 X k =0 Υ H p ( ν ) Υ k ( ν )Σ X ,k,p ! (100)  c R ⊗ e M 0 , c R ⊗ e M 1 , . . . , c R ⊗ e M M − 1  . ∂ G 2 ( c T ) ∂ c T can be simplified as ∂ G 2 ( c T ) ∂ c T = − [ r p − Υ ( ν )( µ p ⊗ c )] H Υ ( ν ) ∂ µ p ⊗ c ∂ c T = − [ r p − Υ ( ν )( µ p ⊗ c )] H Υ ( ν ) (101)  µ p ⊗ c R ⊗ e M 0 , . . . , µ p ⊗ c R ⊗ e M M − 1  . ∂ G 2 ( c T ) ∂ c T can be simplified as ∂ G 2 ( c T ) ∂ c T = c H T diag { ϑ T } . (102) Finally , with ∂ G 1 ( c T ) ∂ c T , ∂ G 2 ( c T ) ∂ c T and ∂ G 3 ( c T ) ∂ c T , the expression of ∂ L ( c T ) ∂ c T can be obtained in ( 91 ). A P P E N D I X B T H E D E R I V A T I O N O F L I K E L I H O O D F U N C T I O N L ( ν ) The likelihood function L ( c T ) can be rewritten as L ( ν ) ∝ P − 1 X p =0 T 1 ( ν ) + T 2 ( ν ) , (103) where T 1 ( ν ) , k r p − Υ ( ν )( µ p ⊗ c ) k 2 2 , (104) T 2 ( ν ) , T r { ( I U ⊗ c ) H Υ H ( ν ) Υ ( ν )( I U ⊗ c ) Σ X } . (105) A CCEPTED BY IEEE TRANSA CTIONS ON SIGNAL PR OCESSING 12 ∂ T 1 ( ν ) ∂ ν can be obtained as ∂ T 1 ( ν ) ∂ ν = − 2 R  [ r p − Υ ( ν )( µ p ⊗ c )] H ∂ Υ ( ν )( µ p ⊗ c ) ∂ ν  = − 2 R  [ r p − Υ ( ν )( µ p ⊗ c )] H Ξ (diag { µ p } ⊗ c )  . (106) ∂ T 2 ( ν ) ∂ ν ∈ R 1 × U is a row vector , and the u -th entry is  ∂ T 2 ( ν ) ∂ ν  u = T r  ∂ ( I U ⊗ c ) H Υ H ( ν ) Υ ( ν )( I U ⊗ c ) Σ X ∂ ν u  = T r  0 , ( I U ⊗ c H ) Υ H ( ν ) Ξ u c , 0  + T r n  0 , ( I U ⊗ c H ) Υ H ( ν ) Ξ u c , 0  H Σ X o = 2 R ( U − 1 X m =0 c H Υ H m ( ν ) Ξ u c Σ X ,u,m ) . (107) Therefore, ∂ T 2 ( ν ) ∂ ν can be simplified as ∂ T 2 ( ν ) ∂ ν = 2 R n diag  Σ X ( I U ⊗ c ) H Υ H ( ν ) Ξ ( I U ⊗ c )  T o . (108) Therefore, with ∂ L ( ν ) ∂ ν u = 0 , we can obtain the equation ( 92 ) to obtain ν . R E F E R E N C E S [1] J. Li and P . Stoica, “MIMO radar with colocated antennas, ” IEEE Signal Pr ocess. Mag. , v ol. 24, no. 5, pp. 106–114, Oct. 2007. [2] P . Chen, L. Zheng, X. W ang, H. Li, and L. W u, “Moving target detection using colocated MIMO radar on multiple distributed moving platforms, ” IEEE T rans. Signal Pr ocess. , vol. 65, no. 17, pp. 4670 – 4683, Jun. 2017. [3] M. Davis, G. Showman, and A. Lanterman, “Coherent MIMO radar: The phased array and orthogonal waveforms, ” IEEE Aerosp. Electron. Syst. Mag . , v ol. 29, no. 8, pp. 76–91, Aug. 2014. [4] A. M. Haimovich, R. S. Blum, and L. J. Cimini, “MIMO radar with widely separated antennas, ” IEEE Signal Process. Mag. , vol. 25, no. 1, pp. 116–129, Dec. 2008. [5] P . Chen, C. Qi, and L. W u, “ Antenna placement optimisation for compressed sensing-based distributed MIMO radar, ” IET Radar , Sonar & Navigation , v ol. 11, no. 2, pp. 285–293, Feb . 2017. [6] B. D. V . V een and K. M. Buckley , “Beamforming: A versatile approach to spatial filtering, ” IEEE ASSP Magazine , vol. 5, no. 2, pp. 4 – 24, Apr . 1988. [7] R. O. Schmidt, “Multiple emitter location and signal parameter estima- tion, ” IEEE T rans. Antennas Pr opag. , vol. 34, no. 3, pp. 276–280, Mar . 1986. [8] R. Schmidt, “A signal subspace approach to multiple emitter location spectrum estimation, ” Ph.D. dissertation, Stanford University , Stanford, CA, 1981. [9] M. Zoltowski, G. Kautz, and S. Silverstein, “Beamspace Root-MUSIC, ” IEEE T rans. Signal Process. , vol. 41, no. 1, pp. 344–364, Jan. 1993. [10] R. Roy and T . Kailath, “ESPRIT -estimation of signal parameters via rotational inv ariance techniques, ” IEEE T rans. Acoust., Speech, Signal Pr ocess. , vol. 37, no. 7, pp. 984–995, Jul. 1989. [11] L. Du, T . Y ardibi, J. Li, and P . Stoica, “Revie w of user parameter-free robust adaptiv e beamforming algorithms, ” Digital Signal Pr ocessing , vol. 19, no. 4, pp. 567 – 582, Jul 2009. [12] A. Khabbazibasmenj, A. Hassanien, S. A. V orobyov , and M. W . Morency , “Efficient transmit beamspace design for search-free based doa estimation in mimo radar, ” IEEE T rans. Signal Pr ocess. , vol. 62, no. 6, pp. 1490–1500, March 2014. [13] M. Cao, S. A. V orobyov , and A. Hassanien, “T ransmit array interpolation for DO A estimation via tensor decomposition in 2-D MIMO radar, ” IEEE T rans. Signal Pr ocess. , vol. 65, no. 19, pp. 5225–5239, Oct 2017. [14] A. Hassanien and S. A. V orobyov , “T ransmit energy focusing for DOA estimation in MIMO radar with colocated antennas, ” IEEE T rans. Signal Pr ocess. , vol. 59, no. 6, pp. 2669–2682, June 2011. [15] L. Xu, J. Li, and P . Stoica, “T arget detection and parameter estimation for MIMO radar systems, ” IEEE T rans. Aerosp. Electr on. Syst. , vol. 44, no. 3, pp. 927–939, July 2008. [16] D. Nion and N. D. Sidiropoulos, “T ensor algebra and multidimensional harmonic retriev al in signal processing for MIMO radar, ” IEEE T rans. Signal Pr ocess. , vol. 58, no. 11, pp. 5693–5705, Nov 2010. [17] Y . I. Abramovich, G. J. Frazer , and B. A. Johnson, “Iterative adaptiv e kronecker MIMO radar beamformer: Description and conver gence anal- ysis, ” IEEE T rans. Signal Pr ocess. , vol. 58, no. 7, pp. 3681–3691, July 2010. [18] Z. Y ang and L. Xie, “Exact joint sparse frequency recovery via opti- mization methods, ” IEEE T rans. Signal Process. , vol. 64, no. 19, pp. 5145 – 5157, Oct. 2016. [19] ——, “Enhancing sparsity and resolution via reweighted atomic norm minimization, ” IEEE T rans. Signal Pr ocess. , vol. 64, no. 4, pp. 995– 1006, Feb . 2016. [20] Y . Y u, A. P . Petropulu, and H. V . Poor, “Measurement matrix design for compressiv e sensing-based MIMO radar, ” IEEE Tr ans. Signal Process. , vol. 59, no. 11, pp. 5338 – 5352, Nov . 2011. [21] M. Carlin, P . Rocca, G. Oliveri, F . V iani, and A. Massa, “Directions- of-arriv al estimation through Bayesian compressive sensing strategies, ” IEEE T rans. Antennas Propa g. , vol. 61, no. 7, pp. 3828 – 3838, Jul. 2013. [22] ——, “Novel wideband DOA estimation based on sparse Bayesian learning with dirichlet process priors, ” IEEE T rans. Signal Pr ocess. , vol. 64, no. 2, pp. 275 – 289, Jan. 2016. [23] Q. Shen, W . Liu, W . Cui, and S. Wu, “Underdetermined DO A estimation under the compressive sensing frame work: A revie w, ” IEEE Access , vol. 4, pp. 8865 – 8878, Nov . 2016. [24] M. Rossi, A. M. Haimovich, and Y . C. Eldar, “Spatial compressi ve sensing for MIMO radar, ” IEEE T rans. Signal Process. , v ol. 62, no. 2, pp. 419–430, Jan 2014. [25] Z. T an, P . Y ang, and A. Nehorai, “Joint sparse recovery method for compressed sensing with structured dictionary mismatches, ” IEEE T rans. Signal Pr ocess. , vol. 62, no. 19, pp. 4997–5008, Oct 2014. [26] M. E. T ipping, “Sparse Bayesian learning and the relev ance vector machine, ” J. Mach. Learn. Res. , vol. 1, pp. 211–244, Sep. 2001. [27] S. Ji, Y . Xue, and L. Carin, “Bayesian compressiv e sensing, ” IEEE T rans. Signal Pr ocess. , vol. 56, no. 6, pp. 2346–2356, Jun. 2008. [28] H. Zhu, G. Leus, and G. B. Giannakis, “Sparsity-cognizant total least- squares for perturbed compressive sampling, ” IEEE T rans. Signal Pro- cess. , vol. 59, no. 5, pp. 2002–2016, May 2011. [29] Z. Y ang, X. Lihua, and Z. Cishen, “Off-grid direction of arriv al esti- mation using sparse Bayesian inference, ” IEEE T rans. Signal Pr ocess. , vol. 61, no. 1, pp. 38–43, Jan. 2013. [30] J. Dai, X. Bao, W . Xu, and C. Chang, “Root sparse Bayesian learning for off-grid DOA estimation, ” IEEE Signal Process. Lett. , vol. 24, no. 1, pp. 46–50, Jan. 2017. [31] X. Wu, W . Zhu, and J. Y an, “Direction of arriv al estimation for of f- grid signals based on sparse Bayesian learning, ” IEEE Sensor s Journal , vol. 16, no. 7, pp. 2004–2016, Apr . 2016. [32] H. Zamani, H. Zayyani, and F . Marvasti, “An iterati ve dictionary learning-based algorithm for DOA estimation, ” vol. 20, no. 9, pp. 1784– 1787, Sep. 2016. [33] Q. W ang, Z. Zhao, Z. Chen, and Z. Nie, “Grid ev olution method for DO A estimation, ” IEEE Tr ans. Signal Process. , vol. 66, no. 9, pp. 2474– 2383, May 2018. [34] Z. Zheng, J. Zhang, and J. Zhang, “Joint DOD and DOA estimation of bistatic MIMO radar in the presence of unknown mutual coupling, ” Signal Pr ocessing , vol. 92, pp. 3039 – 3048, Jun. 2012. [35] B. Clerckx, C. Craeye, D. V anhoenacker-Jan vier , and C. Oestges, “Impact of antenna coupling on 2 × 2 MIMO communications, ” IEEE T rans. V eh. T echnol. , vol. 56, no. 3, pp. 1009 –1018, May 2007. [36] J. Liu, Y . Zhang, Y . Lu, S. Ren, and S. Cao, “Augmented nested arrays with enhanced DOF and reduced mutual coupling, ” IEEE T rans. Signal Pr ocess. , vol. 65, no. 21, pp. 5549 – 5563, Nov . 2017. [37] P . Rocca, M. A. Hannan, M. Salucci, and A. Massa, “Single-snapshot DoA estimation in array antennas with mutual coupling through a multiscaling BCS strategy , ” IEEE T rans. Antennas Pr opag. , v ol. 65, no. 6, pp. 3203–3213, Jun. 2017. [38] M. Hawes, L. Mihaylova, F . Septer, and S. Godsill, “Bayesian compres- siv e sensing approaches for direction of arrival estimation with mutual coupling effects, ” IEEE T rans. Antennas Pr opag. , vol. 65, no. 3, pp. 1357–1367, Mar . 2017. [39] B. Liao, Z.-G. Zhang, and S.-C. Chan, “DOA estimation and tracking of ULAs with mutual coupling, ” IEEE T rans. Aer osp. Electron. Syst. , vol. 48, no. 1, pp. 891 – 905, Jan. 2012. [40] T . Basikolo, K. Ichige, and H. Arai, “A no vel mutual coupling com- pensation method for underdetermined direction of arrival estimation in A CCEPTED BY IEEE TRANSA CTIONS ON SIGNAL PR OCESSING 13 nested sparse circular arrays, ” IEEE T rans. Antennas Propag . , vol. 66, no. 2, pp. 909 – 917, Feb . 2018. [41] C. Zhang, H. Huang, and B. Liao, “Direction finding in MIMO radar with unknown mutual coupling, ” IEEE Access , vol. 5, pp. 4439 – 4447, Mar . 2017. [42] A. T ermos and B. M. Hochwald, “Capacity benefits of antenna cou- pling, ” in 2016 Information Theory and Applications (ITA) , La Jolla, CA, USA, Apr . 2004, pp. 1–5. [43] X. Liu and G. Liao, “Direction finding and mutual coupling estimation for bistatic MIMO radar, ” Signal Pr ocessing , vol. 92, no. 2, pp. 517 – 522, Feb . 2012. [44] E. J. Candès and T . T ao, “Decoding by linear programming, ” IEEE T rans. Inf. Theory , vol. 51, no. 12, pp. 4203–4215, Dec 2005. [45] ——, “Near-optimal signal recovery from random projections: Uni versal encoding strategies?” IEEE T rans. Inf. Theory , vol. 52, no. 12, pp. 5406– 5425, Dec 2006. [46] E. J. Candès and C. Fernandez-Granda, “T owards a mathematical theory of super-resolution, ” Communications on Pure and Applied Mathemat- ics , vol. 67, no. 6, pp. 906–956, Jun. 2014. [47] S. D. Babacan, M. Luessi, R. Molina, and A. K. Katsaggelos, “Sparse Bayesian methods for low-rank matrix estimation, ” IEEE T rans. Signal Pr ocess. , vol. 60, no. 8, pp. 3964–3977, Aug 2012. [48] Y . Y u, A. P . Petropulu, and H. V . Poor , “MIMO radar using compressiv e sampling, ” IEEE Journal of Selected T opics in Signal Pr ocessing , vol. 4, no. 1, pp. 146–163, Feb 2010. [49] H. Jiang, J. Zhang, and K. M. W ong, “Joint DOD and DO A estimation for bistatic MIMO radar in unknown correlated noise, ” IEEE T rans. V eh. T echnol. , vol. 64, no. 11, pp. 5113–5125, Nov 2015. Peng Chen (S’15-M’17) was born in Jiangsu, China in 1989. He received the B.E. degree in 2011 and the Ph.D. degree in 2017, both from the School of Information Science and Engineering, Southeast Univ ersity , China. From Mar . 2015 to Apr . 2016, he was a V isiting Scholar in the Electrical Engineering Department, Columbia Univ ersity , Ne w Y ork, NY , USA. He is now an associate professor at the State Ke y Laboratory of Millimeter W av es, Southeast Uni- versity . His research interests include radar signal processing and millimeter wave communication. Zhenxin Cao (M’18) was born in May 1976. He receiv ed the M. S. degree in 2002 from Nanjing Univ ersity of Aeronautics and Astronautics, China, and the Ph.D. degree in 2005 from the School of Information Science and Engineering, Southeast Univ ersity , China. From 2012 to 2013, he was a V isiting Scholar in North Carolina State Univ ersity . Since 2005, he has been with the State Key Laboratory of Millimeter W aves, Southeast Univ er- sity , where he is a Professor . His research interests include antenna theory and application. Zhimin Chen (M’17) was born in Shandong, China, in 1985. She received the Ph.D. degree in infor- mation and communication engineering from the School of Information Science and Engineering, Southeast Univ ersity , Nanjing, China in 2015. Since 2015, she has been with Shanghai Dianji University , Shanghai, China. Her research interests include array signal pro-cessing and Millimeter-W a ve communica- tions. Xianbin W ang (S’98-M’99-SM’06-F’17) is a Pro- fessor and Tier-I Canada Research Chair at W estern Univ ersity , Canada. He received his Ph.D. degree in electrical and computer engineering from National Univ ersity of Singapore in 2001. Prior to joining W estern, he was with Commu- nications Research Centre Canada (CRC) as a Re- search Scientist/Senior Research Scientist between July 2002 and Dec. 2007. From Jan. 2001 to July 2002, he was a system designer at STMicroelectron- ics, where he was responsible for the system design of DSL and Gigabit Ethernet chipsets. His current research interests include 5G technologies, Internet-of-Things, communications security , machine learn- ing and locationing technologies. Dr . W ang has over 300 peer-re viewed journal and conference papers, in addition to 26 granted and pending patents and sev eral standard contributions. Dr . W ang is a Fellow of Canadian Academy of Engineering, a Fellow of IEEE and an IEEE Distinguished Lecturer . He has received many awards and recognitions, including Canada Research Chair , CRC President’ s Excellence A ward, Canadian Federal Government Public Service A ward, Ontario Early Researcher A ward and five IEEE Best Paper A wards. He currently serves as an Editor/Associate Editor for IEEE Transactions on Communications, IEEE T ransactions on Broadcasting, and IEEE Transactions on V ehicular T echnology and He was also an Associate Editor for IEEE Transactions on Wireless Communications between 2007 and 2011, and IEEE W ireless Communications Letters between 2011 and 2016. Dr . W ang was inv olved in many IEEE conferences including GLOBECOM, ICC, VTC, PIMRC, WCNC and CWIT , in different roles such as symposium chair , tutorial instructor, track chair , session chair and TPC co-chair .