Exploiting unlabeled RSSI fingerprints in multi-building and multi-floor indoor localization through deep semi-supervised learning based on mean teacher

Kyeong Soo Kim, "Exploiting unlabeled RSSI fingerprints in multi-building and multi-floor indoor localization through deep semi-supervised learning based on mean teacher," Invited talk, 2023 International Workshop for Future Values of Mathematics and Its Applications, Chungnam National University (CNU), Daejoen, Korea, Aug. 7–9, 2023.

1. Exploiting Unlabeled RSSI Fingerprints in Multi-Building and Multi-Floor Indoor Localization through Deep Semi-Supervised Learning Based on Mean Teacher Kyeong Soo (Joseph) Kim (with Sihao Li, Zhe Tang, and Jeremy Smith) School of Advanced Technology Xi’an Jiaotong-Liverpool University (XJTLU) 2023 International Workshop for Future Values of Mathematics and Its Applications Chungnam National University August 7–9, 2023

2. Outline XJTLU Camus Information and Visitor Service System Wi-Fi Fingerprinting Semi-Supervised Learning (SSL) Indoor Localization Framework Based on Deep SSL Future Work 2

3. XJTLU Camus Information and Visitor Service System 3

4. Fingerprinting Server (SSID 1 , RSSI 1 ) (SSID 2 , RSSI 2 ) (SSID N , RSSI N ) RSS Measurements Estimated Location Location-Aware Services Client (User) XJTLU Intranet ICE ebridge portal  Front-end and Middleware  Service Request (RSS Measurements, …) 4

5. Engineering Building 3F Examples: Indoor Navigation and Location-Aware Service Lecture Theatre 5

6. Multi-Floor Indoor Localization with RSSI/Geomagnetic Field* 6 4 th Floor of IBSS Building 5 th Floor of IBSS Building * Z. Zhong et al., " XJTLUIndoorLoc : A new fingerprinting database for indoor localization and trajectory estimation based on Wi-Fi RSS and geomagnetic field ," Proc. 2018 CANDAR, Takayama, Japan, Nov. 27–30, 2018.

7. Wi-Fi Fingerprinting 7

8. Location Fingerprint EB306 8 A tuple of ( L, F ) L: Location information Geographic coordinates or a label (e.g., “EB306”) F : Vector/function of received signal strength Indicators (RSSIs) e.g., where is the RSSI from i th access point ( AP i ).  

9. Fingerprinting Server (MAC 1 , RSSI 1 ) (MAC 2 , RSSI 2 ) (MAC N , RSSI N ) Estimated Location Client (User)  RSS Measurements Fingerprint Database EB303, (x 2 , y 2 , z 2 ) {(9c:50:33:3f:98:50, -52), (9c:50:33:3f:98:51, -52), … }   EE202, (x 1 , y 1 , z 1 ) {(9c:50:33:3f:98:50, -50), (9c:50:33:3f:98:51, -55), … } 9

10. Two Phases of Indoor Localization 10 t … … RSSI Measurements at Reference Points ( Labeled Data ) DB Construction and Localization Model Building Location Estimation Based on Submitted RSSIs at Unknow Locations ( Unlabeled Data ) System Deployment Offline Phase Online Phase

11. Location Estimation Deterministic: Nearest Neighbour Methods. KNN or weighted KNN . Neural Network Methods. Deep neural networks (DNNs) enabled by deep learning. Probabilistic: Bayesian Inference. Support Vector Machine (SVM). Gaussian Process Latent Variable Model (GP-LVM). 11

12. Nearest Neighbour Methods* A simple approach based on the notion of distance in the signal space: Given a fingerprint of and an RSS measurement of , the Euclidean distance measure between them is defined as Then, we find a fingerprint providing a minimum distance, whose label L is the estimated location.   * P. Bahl and V. N. Padmanabhan, “ RADAR: An in-building RF-based user location and tracking system ,” Proc. of INFOCOM 2000, vol. 2, pp. 775-784, Mar. 2000. 12

13. Major Challenges in Large-Scale Implementation Scalability Localization models Fingerprint DB construction Localization accuracy Non-stationarity of location fingerprints Incremental/online learning algorithms with pruning/forgetting mechanisms* Passive vs. active location estimation Integration with other services Security/privacy issues * R. Elwell and R. Polikar, “ Incremental learning in nonstationary environments with controlled forgetting ,” Proc. IJCNN’09. 13

14. 2006 2017~ Changes in XJTLU Campuses 14

15. Floor (SSID, RSSI) Building Room (SSID, RSSI) = ? Hierarchical Multiclass Classifier with Flat Loss Function Flat Multiclass Classifier with Hierarchical Loss Function Building, Floor, Room 15

16. Semi-Supervised Learning (SSL) 16

17. Influence of Unlabeled Data in SSL * 17 * https://en.wikipedia.org/wiki/Weak_supervision#Semi-supervised_learning . Supervised Learning Semi-Supervised Learning

18. SL vs. SSL: Toy Examples* 18 * Y. Ouali et al., “ An Overview of Deep Semi-Supervised Learning ,” arXiv preprint arXiv:2006.05278 [ cs.LG ], 2006. ** Virtual adversarial training. **

19. Indoor Localization Framework Based on Deep SSL 19

20. Motivation To exploit unlabeled RSSI fingerprints in indoor localization through deep SSL based on the Mean Teacher * method. Conventional techniques under SL cannot exploit unlabeled RSSIs measured at unknown locations. Unlabeled RSSIs could be obtained as follows: Offline phase : Part of an initial, static fingerprint database, which are submitted by volunteers when the database is constructed. Online phase : Newly measured ones submitted by the users of an indoor localization system already deployed in the field. 20 * A. Tarvainen and H. Valpola , “ Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results ,” Proc. NIPS’17, Dec. 2018, pp. 1195-1204.

21. SSL train for   for   Teacher   Student   No Yes Start Pre-process Pre-train   Clone models Evaluation End Labeled Data   Unlabeled Data   Finished?   Label mask Test Data   Flowchart for SSL-Based Indoor Localization 21

22. Batch Training Procedure of The Proposed Framework 22 Insourcing         SAE         Voluntary Mask      

23. Data preparation Experiment Cases Original UJIIndoorLoc data Case 1 Four equal subsets Case 2 Case 3 Labeled Unlabeled Case 4 Experimental Results: Data Preparation Based on The UJIIndoorLoc Database* 23 * J. Torres- Sospedra et al., “ UJIIndoorLoc : A new multi-building and multi-floor database for WLAN fingerprint-based indoor localization problems ,” Proc. IPIN2014, Busan, Korea, Oct. 2014, pp. 261–270.

24. Experimental Results: Backbone Network Based on the scalable DNN model*― a benchmark in multi-building and multi-floor indoor localization ― with the following hyperparameters: 24 * K. S. Kim et al., “ A scalable deep neural network architecture for multi-building and multi-floor indoor localization based on Wi-Fi fingerprinting ,” Big Data Analytics, vol. 3, no. 4, Apr. 2018. Hyperparameter Value SAE Hidden Layers Nodes 256, 128 SAE Activation ReLU SAE Optimizer Adam SAE Loss Function MSE Classifier Hidden Layers Nodes 256, 118 Classifier Hidden Layers Activation ReLU Classifier Output Layers Activation Sigmoid Classifier Dropout Rate 0.5 Classifier Optimizer Adam Classifier Loss Function BCE

25. RSS 1 RSS 2 RSS 3  RSS N    RSS 1 RSS 2 RSS 3  RSS N    Encoder Decoder Stacked Autoencoder (SAE) for the reduction of feature space dimension 25

26. DNN Architecture for Scalable Building/Floor Classification and Floor-Level Coordinates Estimation based on Multi-Label Classifier 26 RSS 1 RSS 2 RSS 3  RSS N    B 1 B N B F 1    Encoder Classifier F max ( N F (1), … , N F ( N B ))   arg max(  ) arg max(  ) B i F j L 1 L max (…, N L ( i , j ), … )  Coordinates Estimation ( x , y )

27. Experimental Results: Minimum Weighted 3D localization Errors 27

28. Experimental Results: Improvement by The Proposed Framework over The Conventional One 28

29. Experimental Results: Improvement by The Proposed Framework w.r.t. Baseline 4* 29 * The experiment based on the conventional framework with 100% labeled data.

30. Experimental Results: Training Times 30 Experiment Time [min] Case 1 15.91 Case 2 16.29 Case 3 16.92 Case 4 17.41 Baseline 1 2.10 Baseline 2 4.25 Baseline 3 6.30 Baseline 4 8.10

31. Experimental Results: Summary The proposed framework can improve the localization performance of the adopted backbone network by up to 12.78% with only 25% of labeled RSSI fingerprints . The localization performance of this case is nearly equivalent to that of the scalable DNN model under the conventional framework with 100% labeled data. This implies that we could reduce the number of labeled fingerprints significantly (i.e., ∼25% of the total data in this case), which requires a time-consuming and labor-intensive process. 31

32. Future Work 32

33. Data Augmentation Based on Multi-Output Gaussian Process* 33 AP1 APN AP2 … SOGP SOGP SOGP …   …     AP1 AP2 APN MOGP …   * Z. Tang et al., " Multi-output Gaussian process-based data augmentation for multi-building and multi-floor indoor localization ," 2022 IEEE International Conference on Communications Workshops (ICC Workshops), pp. 361–366, May 2022.