A Sample-Deficient Analysis of the Leading Generalized Eigenvalue for the Detection of Signals in Colored Gaussian Noise

This paper investigates the signal detection problem in colored Gaussian noise with an unknown covariance matrix. To be specific, we consider a sample deficient scenario in which the number of signal bearing samples ($n$) is strictly smaller than the dimensionality of the signal space ($m$). Our test statistic is the leading generalized eigenvalue of the whitened sample covariance matrix (a.k.a. $F$-matrix) which is constructed by whitening the signal bearing sample covariance matrix with noise-only sample covariance matrix. The whitening operation along with the observation model induces a single spiked covariance structure on the $F$-matrix. Moreover, the sample deficiency (i.e., $m>n$) in turn makes this $F$-matrix rank deficient, thereby {\it singular}. Therefore, a simple exact statistical characterization of the leading generalized eigenvalue (l.g.e.) of a complex correlated {\it singular} $F$-matrix with a single spiked associated covariance is of paramount importance to assess the performance of the detector (i.e., the receiver operating characteristics (ROC)). To this end, we adopt the powerful orthogonal polynomial technique in random matrix theory to derive a new finite dimensional c.d.f. expression for the l.g.e. of this particular $F$-matrix. It turns out that when the noise only sample covariance matrix is nearly rank deficient and the signal-to-noise ratio is $O(m)$, the ROC profile converges to a remarkably simple limiting profile.