Detecting visual-media-borne disinformation: a summary of latest advances at the IDT Lab of CERTH-ITI

We present very briefly some of the most important and latest (June 2024) advances in detecting visual-media-borne disinformation, based on the research work carried out at the Intelligent Digital Transformation Laboratory (IDT Lab) of CERTH-ITI.

1. 1 Detecting visual-media-borne disinformation: a summary of latest advances @ IDT Lab of CERTH-ITI Vasileios Mezaris Thessaloniki, June 2024

2. Verification Plug-in A verification “Swiss army knife” helping journalists, fact-checkers, and human rights defenders (>100.000 active users!) to verify videos and images 2 D. Teyssou , J.-M. Leung, E. Apostolidis , K. Apostolidis , S. Papadopoulos, M. Zampoglou , O. Papadopoulou , V. Mezaris , "The InVID Plug-in: Web Video Verification on the Browser", Proc. MuVer 2017 @ ACM Multimedia 2017, Mountain View, CA, USA, October 2017.

3. Reverse video search on the Web Debunking of fakes that rely on video re-use Latest advancements: Integrated an AI-based method for video thumbnail selection Automated retrieval of near-duplicate videos from the Web Minimized user’s interaction to a “one-click” process 3 E. Apostolidis , G. Balaouras , V. Mezaris , I. Patras, "Selecting a Diverse Set of Aesthetically-pleasing and Representative Video Thumbnails using Reinforcement Learning", Proc. IEEE ICIP 2023, Kuala Lumpur, Oct. 2023. DOI:10.1109/ICIP49359.2023.10222743

4. Image/video manipulation detection & localization  Detection: given an image or video frame, classify it as “manipulated” or “pristine” Localization: identify the manipulated regions within an image Latest advancements: Combination of several forensic filters to increase detection accuracy 4 Localization Detection Manipulated p = 87.23% K. Triaridis , V. Mezaris , "Exploring Multi-Modal Fusion for Image Manipulation Detection and Localization", Proc. MMM 2024, Amsterdam, NL, Springer LNCS vol. 14556, pp. 198–211, Jan.-Feb. 2024.  DOI:10.1007/978-3-031-53311-2_15

5. Explainable AI methods for deepfake detection Deepfakes are AI manipulated media in which, a person's face or body is digitally swapped to alter their identity or reenacted according to a driver video Detection: by careful human inspection, or/and using trained classifiers Latest advancements: AI Explainability : understand which image regions the classifier focused on to make it decision Image sources: https://malcomvetter.medium. com/deep-deep-fakes-d4507c735f44 https://bdtechtalks.com/2023/05/12/detect-deepfakes-ai-generated-media 5 K. Tsigos , E. Apostolidis , S. Baxevanakis , S. Papadopoulos, V. Mezaris , "Towards Quantitative Evaluation of Explainable AI Methods for Deepfake Detection", Proc. ACM MAD @ ICMR’24, Thailand, June 2024.  DOI:10.1145/3643491.3660292 M. Ntrougkas , N. Gkalelis , V. Mezaris , "T-TAME: Trainable Attention Mechanism for Explaining Convolutional Networks and Vision Transformers", IEEE Access, 2024.  DOI:10.1109/ACCESS.2024.3405788 Images source: FaceForensics ++ dataset

6. Detecting AV scene discrepancies in video Visual information can be manipulated, and audio (incl. but not limited to speech) too – but what if the visual scene and ambient audio information do not match? Latest advancements: New experimental protocol and benchmark dataset A baseline method that adapts visual- and audio-scene classification techniques to detect such discrepancies 6 K. Apostolidis , J. Abesser , L. Cuccovillo , V. Mezaris , "Visual and audio scene classification for detecting discrepancies in video: a baseline method and experimental protocol", Proc. ACM MAD @ ICMR’24, Thailand, June 2024.  DOI:10.1145/3643491.3660287 Voice clip #1 Voice clip #2 Processing point Ambient soundscape to mask edits

7. Questio ns ? Contact: Dr. Vasileios Mezaris Head of  Intelligent Digital Transformation Laboratory Information Technologies Institute / CERTH [email protected] ; http://www.iti.gr/~bmezaris ;  http://idt.iti.gr This work was supported by the EU Horizon Europe and Horizon 2020 programmes under grant agreements 687786 InVID , 951911 AI4Media , 101021866 CRiTERIA , 101070093 vera.ai , 101070190 AI4Trust . 7