Multi-Shot Person Re-Identification via Relational Stein Divergence

MUL TI-SHO T PERSON RE-IDENTIFICA TION VIA RELA TION AL STEIN DIVERGENCE Azadeh Alavi, Y an Y ang, Mehrtash Har andi, Conrad Sanderson NICT A, GPO Box 2434, Brisbane, QLD 4001, Australia Uni versity of Queensland, School of ITEE, QLD 4072, Australia Queensland Uni versity of T echnology , Brisbane, QLD 4000, Australia ABSTRA CT Person re-identification is particularly challenging due to significant appearance changes across separate camera views. In order to re- identify people, a representative human signature should effecti vely handle differences in illumination, pose and camera parameters. While general appearance-based methods are modelled in Euclidean spaces, it has been argued that some applications in image and video analysis are better modelled via non-Euclidean manifold geometry . T o this end, recent approaches represent images as cov ariance matri- ces, and interpret such matrices as points on Riemannian manifolds. As direct classification on such manifolds can be difficult, in this paper we propose to represent each manifold point as a vector of similarities to class representers, via a recently introduced form of Bregman matrix diver gence known as the Stein diver gence. This is followed by using a discriminati ve mapping of similarity vectors for final classification. The use of similarity vectors is in contrast to the traditional approach of embedding manifolds into tangent spaces, which can suf fer from representing the manifold structure inaccurately . Comparativ e ev aluations on benchmark ETHZ and iLIDS datasets for the person re-identification task show that the proposed approach obtains better performance than recent tech- niques such as Histogram Plus Epitome, Partial Least Squares, and Symmetry-Driv en Accumulation of Local Features. Index T erms — surveillance, person re-identification, manifolds. 1. INTR ODUCTION Person re-identification is the process of matching persons across non-ov erlapping camera views in div erse locations. W ithin the con- text of surveillance, re-identification needs to function with a large set of candidates and be robust to pose changes, occlusions of body parts, low resolution and illumination variations. The issues can be compounded, making a person difficult to recognise ev en by human observers (see Fig. 1 for e xamples). Compared to classical biometric cues (eg. face, gait) which may not be reliable due to non-frontality , low resolution and/or low frame-rate, person re-identification ap- proaches typically use the entire body . While appearance based person re-identification methods are generally modelled in Euclidean spaces [8, 11, 24], it has been ar- gued that some applications in image and video analysis are better modelled on non-Euclidean manifold geometry [28]. T o this end, recent approaches represent images as covariance matrices [3], and interpret such matrices as points on Riemannian manifolds [12, 28]. A popular way of analysing manifolds is to embed them into tangent spaces, which are Euclidean spaces. This process which can be in- terpreted as warping the feature space [27]. Embedding manifolds is not without problems, as pairwise distances between arbitrary points on a tangent space may not represent the structure of the manifold accurately [12, 13]. Fig. 1 . Examples of challenges in person re-identification, where each column contains images of the same person from two separate camera vie ws. Challenges include pose changes, occlusions of body parts, low resolution and illumination v ariations. In this paper we present a multi-shot appearance based person re-identification method on Riemannian manifolds, where embed- ding the manifolds into tangent spaces is not required. W e adapt a recently proposed technique for analysing Riemannian manifolds, where points on the manifolds are represented through their similar- ity vectors [2]. The similarity vectors contain similarities to class representers. W e obtain each similarity with the aid of a recently introduced form of Bregman matrix div ergence known as the Stein div ergence [13, 25]. The classification task on manifolds is hence con verted into a task in the space of similarity vectors, which can be tackled using learning methods de vised for Euclidean spaces, such as Linear Discriminant Analysis [5]. Unlike previous person re- identification methods, the proposed method does not require sep- arate settings for new datasets. W e continue the paper as follows. In Section 2 sev eral recent methods for person re-identification are briefly described. The pro- posed approach is detailed in Section 3. A comparati ve performance ev aluation on two public datasets is giv en In Section 4. The main findings are summarised in Section 5. 2. PREVIOUS WORK Giv en an image of an individual to be re-identified, the task of per- son re-identification can be categorised into two main classes. (i) Single-vs-Single (SvS), where there is only one image of each per- son in the gallery and one in the probe; this can be seen as a one- to-one comparison. (ii) Multiple-vs-Single (MvS), or multi-shot, where there are multiple images of each person av ailable in gallery and one image in the probe. Below we summarise sev eral person re-identification methods: Partial Least Squares (PLS) [24], Con- text based method [31], Histogram Plus Epitome (HPE) [4], and Symmetry-Driv en Accumulation of Local Features (SDALF) [8]. The PLS method [24] first decomposes a giv en image into over - lapping blocks, and extracts a rich set of features from each block. Three types of features are considered: textures, edges, and colours. The dimensionality of the feature space is then reduced by employ- ing Partial Least Squares regression (PLSR) [30], which models re- lations between sets of observed variables by means of latent vari- ables. T o learn a PLSR discriminatory model for each person, one- against-all scheme is used [9]. Nearest neighbour is then employed for classification. The Context-based method [31] enriches the description of a person by contextual visual knowledge from surrounding people. The method represents a group by considering two descriptors: (a) ‘center rectangular ring ratio-occurrence’ descriptor, which de- scribes the information ratio of visual words between and within v ar- ious rectangular ring regions, and (b) ‘block based ratio-occurrence’ descriptor , which describes local spatial information between visual words that could be stable. For group image representation only fea- tures extracted from foreground pixels are used to construct visual words. HPE [4] considers multiple instances of each person to create a person signature. The structural element (STEL) generativ e model approach [16] is employed for foreground detection. The combina- tion of a global (person le vel) HSV histogram and epitome re gions of foreground pixels is then calculated, where an image epitome [15] is computed by collapsing the gi ven image into a small collage of o ver- lapped patches. The patches contain the essence of textural, shape and appearance properties of the image. Both the generic epitome (epitome mean) and local epitome (probability that a patch is in an epitome) are computed. SD ALF [8] considers multiple instances of each person. Fore- ground features are used to model three complementary aspects of human appearance extracted from various body parts. First, for each pedestrian image, axes of asymmetry and symmetry are found. Then, complementary aspects of the person appearance are detected on each part, and their features are extracted. T o select salient parts of a given pedestrian image, the features are then weighted by exploiting perceptual principles of symmetry and asymmetry . The abov e methods assume that classical Euclidean geometry is capable of providing meaningful solutions (distances and statistics) for modelling and analysing images and videos, which might not be always correct [27]. Furthermore, they require separate parameter tuning for each dataset. 3. PR OPOSED APPRO A CH Our goal is to automatically re-identify a gi ven person among a large set of candidates in diverse locations ov er various non-ov erlapping camera views. The proposed method is comprised of three main stages: (i) feature extraction and generation of covariance descrip- tors, (ii) measurement of similarities on Riemannian manifolds via the Stein di vergence, and (iii) creation of similarity vectors and dis- criminativ e mapping for final classification. Each of the stages is elucidated in more detail in the following subsections. 3.1. F eature Extraction and Covariance Descriptors As per [4, 8], to reduce the effect of varying background, foreground pixels are extracted from each given image of a person via the STEL generativ e model approach [16]. W e note that it is also possible to use more advanced approaches, such as [21]. Based on preliminary experiments, for each each foreground pixel located at ( x, y ) , the following feature v ector is calculated: f = [ x, y , HSV xy , CIELAB xy , Λ xy , Θ xy ] T (1) where HSV xy = [ H xy , S xy , b V xy ] are the colour values of the HSV channels, employing histogram equalisation for channel V , CIELAB xy = [ L xy , a xy , b xy ] are the values of CIELAB colour space [1], while Λ xy = [ λ R xy , λ G xy , λ B xy ] and Θ xy = [ θ R xy , θ G xy , θ B xy ] indicate gradient magnitudes and orientations for each channel in RGB colour space. W e note that we hav e selected this relativ ely straightforward set of features as a starting point, and that it is cer- tainly possible to use other features. Ho wever , a thorough ev aluation of possible features is beyond the scope of this paper . Giv en a set F = { f i } N i =1 of extracted features, with its mean represented by µ , each image is represented as a cov ariance matrix: C = 1 N − 1 X N i =1 ( f i − µ )( f i − µ ) T (2) Representing an image with a cov ariance matrix has sev eral ad- vantages [3]: (i) it is a low-dimensional (compact) representation that is independent of image size, (ii) the impact of noisy samples is reduced via the av eraging during covariance computation, and (iii) it is a straightforward method of fusing correlated features. 3.2. Riemannian Manifolds and Stein Div ergence Cov ariance matrices belong to the group of symmetric positive def- inite (SPD) matrices, which can be interpreted as points on Rie- mannian manifolds. As such, the underlying distance and similarity functions might not be accurately defined in Euclidean spaces [23]. Efficiently handling Riemannian manifolds is non-trivial, due largely to two main challenges [26]: (i) as manifold curvature needs to be taken into account, defining div ergence or distance functions on SPD matrices is not straightforward; (ii) high computational require- ments, e ven for basic operations such as distances. For e xample, the Riemannian structure induced by considering the Af fine Inv ariant Riemannian Metric (AIRM) has been shown somewhat useful for analysing SPD matrices [14, 20]. For A , B ∈ S d ++ , where S d ++ is the space of positiv e definite matrices of size d × d , AIRM is defined as: δ R ( A , B ) : =    log  B − 1 2 AB − 1 2     F (3) where log( · ) is the principal matrix logarithm [25]. Howe ver , AIRM is computationally demanding as it essentially needs eigen- decomposition of A and B . Furthermore, the resulting structure has negati ve curvature which prev ents the use of con ventional learning algorithms for classification purposes. T o simplify the handling of Riemannian manifolds, they are of- ten first embedded into higher dimensional Euclidean spaces, such as tangent spaces [18, 19, 22, 29]. Howe ver , only distances between points to the tangent pole are equal to true geodesic distances, mean- ing that distances between arbitrary points on tangent spaces may not represent the manifold accurately . As an alternative to measuring distances on tangent spaces, in this work we use the recently introduced Stein div ergence, which is a version of the Bregman matrix diver gence for SPD matrices [25]. T o measure dissimilarity between two SPD matrices A and B , the Bregman di vergence is defined as [17]: D φ ( A , B ) , φ ( A ) − φ ( B ) − h∇ φ ( B ) , A − B i (4) where h A , B i = tr  A T B  and φ : S d ++ → R is a real-valued, strictly con vex and differentiable function. The div ergence in (4) is asymmetric which is often undesirable. The Jensen-Shannon sym- metrisation of Bregman di vergence is defined as [17]: D J S φ ( A , B ) , 1 2 D φ  A , A + B 2  + 1 2 D φ  B , A + B 2  (5) By selecting φ in (5) to be − log (det ( A )) , which is the bar- rier function of semi-definite cone [25], we obtain the symmetric Stein di vergence, also known as the Jensen Bregman Log-Det div er- gence [6]: J φ ( A , B ) , log  det  A + B 2  − 1 2 log (det ( AB )) (6) The symmetric Stein di vergence is inv ariant under congruence transformations and inv ersion [6]. It is computationally less expen- siv e than AIRM, and is related to AIRM in sev eral aspects which establish a bound between the div ergence and AIRM [6]. 3.3. Similarity V ectors and Discriminativ e Mapping For each query point (an SPD matrix) to be classified, a similarity to each training class is obtained, forming a similarity vector . W e ob- tain each similarity with the aid of the Stein div ergence described in the preceding section. The classification task on manifolds is hence con verted into a task in the space of similarity vectors, which can be tackled using learning methods devised for Euclidean spaces. Giv en a training set of points on a Riemannian manifold, X = { ( X 1 , y 1 ) , ( X 2 , y 2 ) , . . . , ( X n , y n ) } , where y i ∈  1 , 2 , . . . , m  is a class label, and m is the number of classes, we define the simi- larity between matrix X i and class l as: s i,l = 1 N l X j 6 = i J φ ( X i , X j ) δ ( y j − l ) (7) where δ ( · ) is the discrete Dirac function and N l =  n l − 1 if y i = l n l otherwise (8) where n l is the number of training matrices in class l . Using Eqn. (7), the similarity between X i and all classes is obtained, where i ∈  1 , 2 , . . . , n  . Each matrix X i is hence represented by a similarity vector: s i = [ s i, 1 , s i, 2 , . . . , s i,m ] T (9) Classification on Riemannian manifolds can now be reinter- preted as a learning task in R m . Giv en the similarity vectors of training data, S = { ( s 1 , y 1 ) , ( s 2 , y 2 ) , · · · , ( s n , y n ) } , we seek a way to label a query matrix X q , represented by a similarity vec- tor s q = [ s q, 1 , s q, 2 , . . . , s q,m ] T . As a starting point, we have chosen linear discriminant analysis [5], where we find a mapping W ∗ that minimises the intra-class distances while simultaneously maximising inter-class distances: W ∗ = argmax W trace  h W S W W T i − 1 h W S B W T i  (10) where S B and S W are the between class and within class scatter matrices [5]. The query similarity vector s q can then be mapped into the new space via: x q = W ∗ T s q (11) W e can now use a straightforward nearest neighbour classi- fier [5] to assign a class label to x q . W e shall refer to this approach as Relational Div ergence Classification (RDC). 4. EXPERIMENTS AND DISCUSSION In this section we e valuate the proposed RDC approach by providing comparisons ag ainst sev eral methods on two person re-identification datasets: iLIDS [31] and ETHZ [7, 24]. The VIPeR dataset [10] w as not used as it only has one image from each person in the gallery , and is hence not suitable for testing MvS approaches. Each dataset cov ers various aspects and challenges of the person re-identification task. The results are shown in terms of the Cumulativ e Matching Characteristic (CMC) curv es, where each CMC curv e represents the expectation of finding the correct match in the top n matches. In order to show the improvement caused by using similarity vectors in conjunction with linear discriminant analysis, we also ev aluate the performance of directly using the Stein di vergence in conjunction with a nearest neighbour classifier (ie. direct classifica- tion on manifolds, without creating similarity vectors). W e refer to this approach as the dir ect Stein method. 4.1. iLIDS Dataset The iLIDS dataset is a publicly av ailable video dataset capturing real scenarios at an airport arri val hall under a multi-camera CCTV net- work. From these videos a dataset of 479 images of 119 pedestrians was extracted and the images were normalised to 128 × 64 pixels (height × width) [31]. The extracted images were chosen from non- ov erlapping cameras, and are subject to illumination changes and occlusions [31]. W e randomly selected N images for each person to build the gallery set, while the remaining images form the probe set. The whole procedure is repeated 10 times in order to estimate an av erage CMC curve. W e compared the performance of the proposed RDC approach against the direct Stein method, as well as the algorithms described in Section 2 (SD ALF and Context based) for a commonly used setting of N = 3 . The results, sho wn in Fig. 2, indicate that the proposed method generally outperforms the other techniques. The results also show that the use of similarity vectors in conjunction with linear discriminant analysis is preferable to directly using the Stein div ergence. 20 30 40 50 60 70 80 90 100 1 2 3 4 5 6 7 Re c og ni tion a c c ura c y R DC Ste in SD AL F Context_Based Rank Fig. 2 . Performance on the iLIDS dataset [31] for N = 3 , using the proposed RDC method, the direct Stein method, SD ALF [8], context based method [31]. HPE results for N = 3 were not provided in [4]. 4.2. ETHZ Dataset The ETHZ dataset [7, 24] was captured from a moving camera, with the images of pedestrians containing occlusions and wide variations in appearance. Sequence 1 contains 83 pedestrians (4857 images), Sequence 2 contains 35 pedestrians (1936 images), and Sequence 3 contains 28 pedestrians (1762 images). W e downsampled all the images to 64 × 32 (height × width). For each subject, the training set consisted of N randomly selected images, with the rest used for the test set. The random selection of the training and testing data was repeated 10 times. Results were obtained for the commonly used setting of N = 10 and are sho wn in Fig. 3. On sequences 1 and 2, the proposed RDC method considerably outperforms PLS, SD ALF , HPE and the direct Stein method. On sequence 3, RDC obtains performance on par with SD ALF . Note that the random selection used by the RDC approach to create the gallery is more challenging and more realistic than the data selection strategy employed by SD ALF and HPE on the same dataset [4, 8]. SD ALF and HPE both apply clustering beforehand on the original frames, and then select randomly one frame for each cluster to build their gallery set. In this way they can ensure that their gallery set includes the k eyframes to use for the multi-shot signature calculation. In contrast, we hav en’t applied any clustering for the proposed RDC method in order to be closer to real life scenarios. 5. CONCLUSION W e hav e proposed a no vel appearance based person re-identification method comprised of: (i) representing each image as a compact cov ariance matrix constructed from feature vectors extracted from foreground pix els, (ii) treating cov ariance matrices as points on Rie- mannian manifolds, (iii) representing each manifold point as a vector of similarities to class representers with the aid of the recently intro- duced Stein div ergence, and (iv) using a discriminative mapping of similarity vectors for final classification. The use of similiarity vec- tors is in contrast to the traditional approach of analysing manifolds via embedding them into tangent spaces. The latter might result in inaccurate modelling, as the structure of the manifolds is only par- tially taken into account [12, 13]. Person re-identification experiments on the iLIDS [31] and ETHZ [7, 24] datasets sho w that the proposed approach outperforms sev eral recent methods, such as Histogram Plus Epitome [4], Partial Least Squares [24], and Symmetry-Driven Accumulation of Local Features [8]. 6. A CKNO WLEDGEMENTS NICT A is funded by the Australian Gov ernment as represented by the Department of Br oadband, Communications and the Digital Economy , as well as the Australian Research Council through the ICT Centr e of Excellence program. 75 80 85 90 95 100 1 2 3 4 5 6 7 R ecog n i ti o n accura cy RDC Stein H P E PLS SDAL F 75 80 85 90 95 100 1 2 3 4 5 6 7 R ec o g n iti o n ac cu r ac y R DC Ste in HP E PL S SD AL F 75 80 85 90 95 100 1 2 3 4 5 6 7 Re cogn iti on ac cura cy R D C Stein HP E PLS SDALF Rank Fig. 3 . Performance on the ETHZ dataset [24] for N = 10 , using Sequences 1 to 3 (top to bottom). Results are shown for the pro- posed RDC method, direct Stein method, HPE [4], PLS [24] and SD ALF [8]. 7. REFERENCES [1] T . Acharya and A. K. Ray . Imag e Pr ocessing: Principles and Applications . 2005. [2] A. Alavi, M. T . Harandi, and C. Sanderson. Relational div ergence based classification on Riemannian manifolds. In IEEE W orkshop on Applications of Computer V ision (W ACV) , pages 111–116, 2013. [3] C. Anoop, M. V assilios, and P . Nikolaos. Dirichlet process mixture models on symmetric positive definite matrices for ap- pearance clustering in video surv eillance applications. In IEEE Conf. Computer V ision and P attern Recognition (CVPR) , pages 3417–3424, 2011. [4] L. Bazzani, M. Cristani, A. Perina, M. Farenzena, and V . Murino. Multiple-shot person re-identification by HPE sig- nature. In Int. Conf. P attern Recognition (ICPR) , pages 1413– 1416, 2010. [5] C. M. Bishop. P attern Recognition and Machine Learning . Springer , 2006. [6] A. Cherian, S. Sra, A. Banerjee, and N. Papanik olopoulos. Ef- ficient similarity search for co variance matrices via the Jensen- Bregman LogDet di vergence. In Int. Conf. Computer V ision (ICCV) , pages 2399–2406, 2011. [7] A. Ess, B. Leibe, and L. V an Gool. Depth and appearance for mobile scene analysis. Int. Conf. Computer V ision (ICCV) , pages 1–8, 2007. [8] M. Farenzena, L. Bazzani, A. Perina, V . Murino, and M. Cristani. Person re-identification by symmetry-driv en ac- cumulation of local features. IEEE Conf. Computer V ision and P attern Recognition , pages 2360–2367, 2010. [9] P . Geladi and B. K owalski. Partial least-squares regression: a tutorial. Analytica Chimica Acta , 185:1–17, 1986. [10] D. Gray , S. Brennan, and H. T ao. Ev aluating appearance mod- els for recognition, reacquisition, and tracking. In Proc. IEEE International W orkshop on P erformance Evaluation for Tr ack- ing and Surveillance (PETS) , volume 3, page 5, 2007. [11] D. Gray and H. T ao. V iewpoint in variant pedestrian recognition with an ensemble of localized features. In Computer V ision– ECCV 2008, Lecture Notes in Computer Science , volume 5302, pages 262–275, 2008. [12] M. Harandi, C. Sanderson, A. W iliem, and B. Lovell. Kernel analysis over Riemannian manifolds for visual recognition of actions, pedestrians and textures. IEEE W orkshop on Applications of Computer V ision (W A CV) , pages 433–439, 2012. [13] M. T . Harandi, C. Sanderson, R. Hartley , and B. C. Lovell. Sparse coding and dictionary learning for symmetric positi ve definite matrices: A kernel approach. In Eur opean Confer ence on Computer V ision (ECCV), Lectur e Notes in Computer Sci- ence (LNCS) , volume 7573, pages 216–229, 2012. [14] T . Hou and H. Qin. Efficient computation of scale-space fea- tures for deformable shape correspondences. European Con- fer ence in Computer V ision (ECCV) , pages 384–397, 2010. [15] N. Jojic, B. J. Frey , and A. Kannan. Epitomic analysis of ap- pearance and shape. IEEE International Confer ence on Com- puter V ision , 1:34–41, 2003. [16] N. Jojic, A. Perina, M. Cristani, V . Murino, and B. Fre y . STEL component analysis: Modeling spatial correlations in image class structure. In IEEE Conference on Computer V ision and P attern Recognition , pages 2044–2051, 2009. [17] B. Kulis, M. Sustik, and I. Dhillon. Lo w-rank kernel learning with Bregman matrix div ergences. The Journal of Machine Learning Resear ch , 10:341–376, 2009. [18] Y . Lui. T angent bundles on special manifolds for action recog- nition. IEEE T rans. Cir cuits and Systems for V ideo T echnology , 22(6):930–942, 2011. [19] F . Porikli, O. T uzel, and P . Meer . Cov ariance tracking using model update based on Lie algebra. In IEEE Conf. Computer V ision and P attern Recognition (CVPR) , pages 728–735, 2006. [20] D. Ravi v , A. Bronstein, M. Bronstein, R. Kimmel, and N. Sochen. Affine-in variant geodesic geometry of deformable 3d shapes. Computers & Graphics , 35(3):692–697, 2011. [21] V . Reddy , C. Sanderson, and B. C. Lovell. Improv ed foreground detection via block-based classifier cascade with probabilistic decision integration. IEEE T ransactions on Cir cuits and Systems for V ideo T echnology , 23(1):83–93, 2013. [22] A. Sanin, C. Sanderson, M. T . Harandi, and B. C. Lovell. K-tangent spaces on Riemannian manifolds for improv ed pedestrian detection. In IEEE International Conference on Image Processing (ICIP) , pages 473–476, 2012. [23] A. Sanin, C. Sanderson, M. T . Harandi, and B. C. Lovell. Spatio-temporal covariance descriptors for action and gesture recognition. In IEEE W orkshop on Applications of Computer V ision (W ACV) , pages 103–110, 2013. [24] W . Schwartz and L. Davis. Learning discriminativ e appearance-based models using partial least squares. In Brazil- ian Symposium on Computer Graphics and Image Processing (SIBGRAPI) , pages 322–329, 2009. [25] S. Sra. Positiv e definite matrices and the symmetric Stein di- ver gence. Preprint: , 2012. [26] S. Sra and A. Cherian. Generalized dictionary learning for symmetric positi ve definite matrices with application to nearest neighbor retriev al. Machine Learning and Knowledge Discov- ery in Databases , 6913:318–332, 2011. [27] P . T uraga, A. V eeraraghav an, and R. Chellappa. Statistical analysis on Stiefel and Grassmann manifolds with applications in computer vision. In IEEE Conf. Computer V ision and P at- tern Recognition (CVPR) , pages 1–8, 2008. [28] P . T uraga, A. V eeraraghavan, A. Sriv astava, and R. Chellappa. Statistical computations on Grassmann and Stiefel manifolds for image and video-based recognition. IEEE T rans. P attern Analysis and Machine Intelligence , 33(11):2273–2286, 2011. [29] A. V eeraraghav an, A. Roy-Chowdhury , and R. Chellappa. Matching shape sequences in video with applications in hu- man movement analysis. IEEE. T rans. P attern Analysis and Machine Intelligence , 27(12):1896–1909, 2005. [30] H. W old, S. Kotz, and N. Johnson. Partial least squares. Ency- clopedia of Statistical Sciences , 6:581–591, 1985. [31] W . Zheng, S. Gong, and T . Xiang. Associating groups of peo- ple. In British Machine V ision Conference , volume 1, pages 1–11, 2009.