Ai in cinematography and visual effects.

Technical seminar

1. AI IN CINEMATOGRAPHY AND VISUAL EFFECTS Date: 10 -02 -202 5 By OMKAR SHETTIGAR (4MW21AD034) Under the g uid ance of DR. VIGHNESH SHENOY DEPARTMENT OF ARTIFICIAL INTELLIGENCE AND DATA SCIENCE

2. Contents Abstract Introduction Literature Review Methodology Advantages Disadvantages Conclusion Future Work 2

3. 1. Abstract 3 Deep Learning Integration: Utilizes convolutional neural networks (CNNs) to perform advanced scene analysis and image enhancement, laying the groundwork for automated cinematography. Generative Modeling: Implements generative adversarial networks (GANs) to synthesise high-fidelity visual effects, including realistic textures, dynamic lighting, and environment augmentation. Computer Vision Techniques: Leverages object detection, semantic segmentation, and optical flow algorithms to enable precise shot composition, motion tracking, and automated camera calibration. Sequential Narrative Analysis: Incorporates recurrent neural networks (RNNs) and transformer architectures to analyse temporal sequences, facilitating automated storyboarding and continuity in shot planning. AI Pipeline Architecture: Establishes a multi-stage pipeline that includes data preprocessing, feature extraction, model inference, and post-processing for seamless integration with traditional cinematography tools. Efficiency and Creative Enhancement: Demonstrates how AI-driven methodologies enhance production efficiency and creative flexibility, setting a new paradigm for real-time rendering and post-production compositing.

4. 4

5. 5

6. 2. Introduction 6 AI Revolution in Film Production: Introduces the transformative impact of artificial intelligence, specifically through deep neural networks, on modern cinematography and the evolution of visual effects workflows. Advanced Scene Analysis: Explains the role of convolutional neural networks (CNNs) in enabling high-precision image segmentation and object detection, critical for automated framing and focus adjustments on set. Real-Time Visual Synthesis: Describes the use of generative adversarial networks (GANs) to generate realistic textures and simulate dynamic lighting conditions, thereby automating the creation of complex visual effects. Narrative Structure Processing: Details how recurrent neural networks (RNNs) and transformer models support the extraction and analysis of narrative elements, contributing to the development of automated storyboarding systems.

7. 7 Robust Motion and Camera Control: Highlights the integration of computer vision techniques such as optical flow estimation and feature matching to ensure accurate motion tracking and responsive camera planning in live shoots. Integrated AI Pipeline: Outlines a comprehensive AI pipeline—from data acquisition and preprocessing through model inference to final post-production rendering—that unifies these technologies into a cohesive system for enhanced cinematography and visual effects.

8. Literature Survey 8 Paper (Authors, Year) Approach Key Findings Limitations Saputra MRU, Markham A, Trigoni N (2018) - Visual SLAM and Structure from Motion in Dynamic Environments: A Survey Survey-based analysis of SLAM and SfM techniques in dynamic environments. Discussed challenges in visual SLAM when applied to dynamic scenes and suggested improvements using deep learning. Limited coverage of real-time performance and scalability of proposed techniques​ Gao K, Gao Y, He H, et al (2022) - NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review Analyzed recent developments in NeRF-based 3D reconstruction methods. Highlighted the strengths of NeRF for realistic 3D scene generation and its application in cinematography. High computational cost and inefficiency in real-time applications​ Ronneberger O, Fischer P, Brox T (2015) - U-Net: Convolutional Networks for Biomedical Image Segmentation Introduced a U-shaped convolutional neural network for biomedical image segmentation. Demonstrated superior accuracy in medical image segmentation tasks, influencing AI-based visual effects. Lack of robustness to complex background variations in images​ Siddique N, Paheding S, Elkin CP et al (2021) - U-Net and its Variants for Medical Image Segmentation: A Review of Theory and Applications Reviewed improvements and extensions of U-Net for different applications. Showcased U-Net’s adaptability beyond medical imaging to computer vision tasks in cinematography. Computationally intensive for large-scale applications​. Schonberger JL, Frahm JM (2016) - Structure-from-Motion Revisited Improved SfM techniques using better keypoint detection and multi-view stereo matching. Achieved enhanced accuracy in 3D reconstruction, crucial for virtual cinematography. High sensitivity to feature extraction quality and occlusions​ .complexity. Maiocchi (1990) – Pinocchio System Motion Capture Library and Retrieval Offered a practical method to reuse recorded articulated body movements, streamlining the animation generation process. Dependent on the availability and quality of motion capture data; less flexible for creating novel or highly stylized motions.

9. Methodology 1. Input Acquisition: Raw Data Collection: The system begins by gathering raw footage (video sequences, still images, sensor data, etc.). 2. Preprocessing: Data Enhancement: The input images and videos are preprocessed through noise reduction, segmentation, and normalization. Purpose: This step cleans the data and prepares it for deeper analysis. 3. AI Analysis & Processing: Deep Learning Modules: Convolutional Neural Networks (CNNs) analyze the visual content to recognize objects, scenes, and movements. Recurrent Neural Networks (RNNs) or transformers may be used to capture temporal or narrative context. Generative Models: Generative Adversarial Networks (GANs) or similar models are employed to generate new visual effects or enhance existing footage. 9

10. 10

11. 4. Effects Generation: Visual Effects Synthesis: Based on the analysis, the system produces visual enhancements, such as automated shot composition, style transfer, and the integration of computer-generated imagery (CGI). 5. Post-Processing & Integration: Final Output: The processed footage—with enhanced visuals and effects—is rendered and integrated with the original material to produce the final cinematic product. 11

12. Intelligent Cinematography 12

13. The diagram shows a typical computational cinematography pipeline. It starts with Previs (pre-visualization assets and constraints), then moves through a Sequence Planner (organizing shots and sequences), followed by a DGCL Compiler (translating cinematographic language rules into executable constraints), and a Heuristic Evaluator (scoring and selecting the best shot configurations). The final Outputs are the chosen camera setups or shot sequences ready for review or direct implementation. 13

14. Core Idea of Computational Cinematography This discusses how computational cinematography automates and optimizes traditional filmmaking tasks—such as camera management, shot segmentation, and visual analysis—using algorithms and structured “cinematographic languages.” This approach aims to streamline production workflows and assist creative decisions. Cinematographic Languages and Constraints These specialized “languages” define rules or constraints (e.g., camera angles, shot durations, scene composition) that guide automatic shot selection and arrangement. By encoding cinematic principles, the system can evaluate different shot configurations while respecting storytelling and aesthetic requirements. ECP (Event-Camera-Parameter) Framework Wu and Christie (2016) introduced ECP, which groups related shots into sub-sequences and employs a “double reverse depth search” algorithm to determine valid configurations. This two-stage process first identifies shot constraints (e.g., camera motion, framing) and then iterates over frames to label possible solutions at either the local (sub-sequence) or global (full sequence) level. Integration and Workflow The ECP framework can be integrated with 3D engines (e.g., Unity) to generate or preview camera setups in real-time. By combining automated analysis with user-driven inputs, filmmakers can quickly iterate on shot designs, leading to more efficient production planning. 14

15. Virtual Production Virtual production (VP) refers to the use of real-time computer graphics and digital environments to create cinematic content. Unlike traditional filmmaking, where VFX are added in post-production, virtual production integrates these elements during the shoot, enabling directors and cinematographers to see the final composite in real-time. 15

16. 16

17. Key Components LED Volumes/Stages : Large LED screens that display dynamic, high-resolution backgrounds and environments. These screens provide realistic lighting and reflections, enhancing the immersion for actors and reducing the need for post-production compositing. Real-Time Rendering Engines : Software like Unreal Engine is used to render digital environments in real-time, allowing for immediate feedback and adjustments. Motion Capture ( MoCap ) : Technology that captures the movements of actors and translates them into digital characters or elements within the virtual environment. In-Camera Visual Effects (ICVFX) : Visual effects that are captured directly in-camera, reducing the need for extensive post-production work. 17

18. 18

19. 19

20. Advantages: 20 Enhanced Creativity and Innovation. Increased Efficiency and Automation. Real-Time Processing and Feedback. Cost Reduction. Improved Consistency and Quality. Data-Driven Insights.

21. Disadvantages: High Computational Requirements. Integration Challenges. Potential Loss of the Human Touch. Technical Limitations and Artifacts. Ethical and Ownership Concerns. Steep Learning Curve. 21

22. Applications of AI in Cinematography and Visual Effects Automated Story Analysis and Storyboarding: Natural Language Processing (NLP) for Story Understanding. Storyboard Generation Virtual Cinematography and Camera Planning: Automated Camera Movement. Dynamic Framing and Composition Motion Capture and Animation Synthesis: Keyframe Interpolation and Inverse Kinematics. Automated Character Animation. Visual Effects (VFX) Enhancement: Image Processing and Style Transfer. Real-Time Effects Generation Music and Sound Design Integration: Automated Music Composition. Sound Effects Synchronization Real-Time Rendering and Editing: Instant Feedback. Enhanced Post-Processing 22

23. 5. Conclusion 23 Experimental productions offer real-world testing that overcomes simulation limits and promotes industry adoption. Advances in language models enable natural, language-driven automated camera control. New LED volume technologies revolutionise chroma-keying and roto-scoping, enhancing visual control despite alignment challenges. Innovations like production-ready NeRFs and Gaussian splatting provide faster, more accurate 3-D capture alternatives. These developments signal a transformative future for cinematography, driven by interdisciplinary research and ethical collaboration.

24. 6. Future Work Enhancement of Natural Language Understanding: Future work should focus on advancing the natural language processing modules to handle more complex and nuanced narratives. This includes deeper semantic disambiguation and integration of broader commonsense reasoning to improve the system’s ability to interpret varied storytelling styles. Refinement of Planning Algorithms: Both qualitative and quantitative planning methods require further refinement to enhance the robustness and flexibility of plot, act, and director planning. Developing adaptive algorithms that can dynamically adjust to changing narrative inputs and contextual cues will be essential for more natural story development. Advancement in Dynamic and Kinematic Modeling: Improving the dynamic and kinematic models used for character articulation and motion planning is critical. Future research should aim to incorporate more sophisticated physics-based and machine learning approaches to yield smoother and more realistic movements, addressing the current shortcomings in naturalness. Development of Real-Time Camera and Light Planning Systems: Although the current camera planning module provides a solid foundation, future work must focus on integrating real-time feedback mechanisms and adaptive adjustments. This will help address challenges such as occlusion avoidance and dynamically changing scene compositions during shooting. Automation of Music Composition and Synchronization: The music planning module, which in the prototype relied on manual composition, can be further automated by leveraging AI-driven music composition techniques. Integrating real-time audio analysis and synchronization with visual events will enhance the overall storytelling experience. 24

25. References Agarwal S, Furukawa Y, Snavely N et al (2011) Building rome in a day. Commun ACM 54(10):105–112. https://doi.org/10.1145/2001269.2001293 Balaji S, Karthikeyan S (2017) A survey on moving object tracking using image processing. In: 2017 11th international conference on intelligent systems and control (ISCO), IEEE, pp 469–474. https://doi.org/ 101109/ISCO20177856037 Chambers M, Israel J, Wright A (2017) Large scale vfx pipelines. In: ACM SIGGRAPH 2017 Talks. ACM, p 1–2. https://doi.org/10.1145/3084363.3085021 Dhariwal P, Nichol A (2021) Diffusion models beat gans on image synthesis. Adv Neural Inf Process Syst 34:8780–8794 Galvane Q, Ronfard R, Lino C, et al (2015) Continuity editing for 3d animation. In: Proceedings of the AAAI Conference on Artificial Intelligence. https://doi.org/10.1609/aaai.v29i1.9288 Helzle V (2023) Chapter 20 - immersive media productions involving light fields and virtual production led walls. In: Valenzise G, Alain M, Zerman E, et al (eds) Immersive Video Technologies. Academic Press, p 575–589. https://doi .org/10.101 6/B978-0-32 -391755- 1.00026-2 Jasińska A, Pyka K, Pastucha E et al (2023) A simple way to reduce 3d model deformation in smartphone photogrammetry. Sensors 23(2):728. https://doi.org/10.3390/s23020728 Kavakli M, Cremona C (2022) The virtual production studio concept - an emerging game changer in film making. In: 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp 29–37. https:// doiorg /101109/VR51125202200020 Ren S, He K, Girshick R, et al (2015b) Faster r- cnn : Towards real-time object detection with region proposal networks. Advances in neural information processing systems 28 Şah M, Direkoğlu C (2021) Review and evaluation of player detection methods in field sports. Multimedia Tools and Applications pp 1–25. https://doi.org/10.1007/s11042-021-11071-z 25

26. Thank You 26