Introduction to Matsuo-Iwasawa Lab (HP)

Introducing Matsuo-Iwasawa Laboratory of the University of Tokyo

1. Introduction to Matsuo- Iwasawa Lab July. 2024 Photography, video recording and disclosure to third parties without permissions are strictly prohibited.

2. Matsuo Lab startups Matsuo Lab Community Matsuo Lab community consists of 200+ members combined with Matsuo Lab and Matsuo Lab Startups Researchers (about 10 people) Staff (About 50 people) Assigned students (About 50 people) TAs (About 20 people) Prof. Matsuo Board members ＋ Matsuo Lab belongs to the Graduate School of Engineering at The University of Tokyo and specializes in Artificial Intelligence (AI) and Web Engineering research. We aim to create world-leading high-level results from our research. Matsuo Laboratory of the University of Tokyo graduates 88 (As of July, 2021) PhD: 14 M: 43 B: 31 And more

3. Prof. Yutaka Matsuo Professor, Technology Management for Innovation, School of Engineering, UTokyo E xpertise: AI, especially deep learning and web mining 1997 Graduated from the Faculty of Engineering at The University of Tokyo with a Bachelor’s degree in Information and Communication Engineering 2002 Completed a doctoral program and earned a doctorate in engineering at the Graduate School of Engineering at The University of Tokyo Became a researcher with the National Institute of Advanced Industrial Science and Technology (AIST) From Oct. 2005. Visiting Scholar , Stanford University From Oct. 2007 Associate Professor, Institute of Engineering Innovation / Center for Structuring of Knowledge / Department of Technology Management for Innovation, Graduate School of Engineering, The University of Tokyo From 2014. Joint Representative and Project Associate Professor, Chair for Global Consumer Intelligence, Department of Technology Management for Innovation , Graduate School of Engineering, The University of Tokyo 2012 – 14 Editor-in-Chief, Transactions of the Japanese Society for Artificial Intelligence , then Chair of Ethics Committee (present post) June 2017 Founder and Director , Japan Deep Learning Association (JDLA) From April 2019 Professor of Research Into Artifacts, Center for Engineering (RACE) / Department of Technology Management for Innovation, Graduate School of Engineering, The University of Tokyo From June 2019 Concurrent Outside Director, SoftBank Group Corp. From Oct. 2021 Member, PM’s Council of New Form of Capitalism Realization From May. 2023 Member, PM’s Council of AI Strategy Japan Deep Learning Association

4. Hongo Valley Vision | Make Hongo a center of i nnovation Aiming to build an ecosystem Where the results of research are not kept within academia, but widely shared in the form of startups and services. The benefits of those economic activities recirculate and promote further research. Where Japanese technological superpowers that can compete globally can be nurtured Joint research (Implementation) Fundamental research Advanced education Startups (eventually becoming big companies) Expertise, resources, etc. for success to be returned to the academia. Return the know-how and resources for success to the university Nurturing of technological “seeds” and let them in society Provide advanced education base on fundamental research Provide practical learning opportunities through classroom lectures, OJT and participation in joint researches Create a spiral of innovation

5. Overview 3. Social Implementation 1. Fundamental Research 2. Education 4. Incubation Conduct basic research on Deep Learning (especially world models) and produce academic results R&D in the area of Deep Learning through collaboration with private companies. Human resource development programs for students and adults (not limited to students and faculty members of the University of Tokyo) Entrepreneurship education, entrepreneurship support, startup support

6. Overview | Fundamental Research 3. Social Implementation 1. Fundamental Research 2. Education 4. Incubation Conduct basic research on Deep Learning (especially world models) and produce academic results R&D in the area of Deep Learning through collaboration with private companies. Human resource development programs for students and adults (not limited to students and faculty members of the University of Tokyo) Entrepreneurship education, entrepreneurship support, startup support

7. Fundamental Research | Mission Mission: Create Intelligence and Discover the Principle of Human Intelligence Matsuo Lab is conducting 2 types of research: Fundamental and Application Fundamental: Algorithm of Deep Learning Application of Deep Learning Natural Language Processing (NLP) Vision Robotic Reinforcement Learning (RL) Generative Models Transfer Learning )

8. Fundamental Research | World Models World Model is a technology to simulate the real world, which is key to the future development of Deep Learning and corresponds to the human 'imagination’. Leading AI companies and laboratories around the world are promoting research. Sources: https://deepmind.com/blog/article/neural-scene-representation-and-rendering , https://worldmodels.github.io Technology Overview Examples Humans can use imagination to compensate for gaps in information and to posit future conditions. E.g. Imagining future from the current state The key to future development is for AI to be able to efficiently learn and “imagine” the “common sense” of the outside world from experience. Its core technology is the World model. CRASH!! E.g. Looking at a part of an object and imagining the whole picture Leading companies and laboratories around the world are focusing on researching World models Example from DeepMind (Google): Reconstructing an entire object or group of objects from a series of limited views The object of this game is to avoid the bullets. The system’s ability to avoid being hit is improved by incorporating an efficient mechanism to imagine the future “Neural scene representation and rendering”, S. A. Eslami, et al., Science , 360(6394):1204–1210, 2018. Based on images from three different perspectives, AI reconstructs the 3D view. Example from Google Brain: Efficient prediction of future events “Recurrent world models facilitate policy evolution”. D. Ha, J. Schmidhuber, NeurIPS 2018, pp. 2455–2467, 2018.

9. Fundamental Research | Research on prompt engineering Matsuo Lab is also researching on large language models. Our research member, Takeshi Kojima, found prompt, “Let’s think step by step”, which elicits the logical knowledges and improve logical reasoning. Standard Prompting Proposed Prompting (Zero-Shot CoT ) LLM are typically give poor performance on multi-step reasoning (e.g. math) Internal working of the LLMs is also unclear Simply add a magical phrase (known as prompt), “Let’s think step by step” elicit logical knowledge Improve reasoning performance e.g., MultiArith (17.7% -> 78.7%) ”Large Language Models are Zero-Shot Reasonoers ”, NeurIPS2022, (2,000+ citations at 2024/05/28)

10. Matsuo Lab LLM “Weblab-10B” （ Aug. 2023) 10 Developed and publicly released a LLM with 10 billion parameters for Japanese and English by pre-training and post-training (fine tuning) Designed to increase the amount of training data by using not only Japanese but also English datasets for training, and to improve the accuracy of Japanese by transferring knowledge between languages The highest level of publicly available model in Japan at the time of release

11. Fundamental Research | Application (Robotics) Application of deep learning to develop intelligent robotics systems Our robot team, TRAIL won Robocup Japan Open (2023 & 2024) and nominated as 3 rd place in RoboCup World Championship 2023 Example: tidy-up robots, object recognition (example below)

12. Fundamental Research | Accepted conferences Deep Generative and World Models Escaping from Plato’s Cave with 2D Frozen Diffusion Model Given Sparse Views”, NeurIPS2023 ICASSP2024, ICML2023 ， ICLR2023 ， NeurIPS2022 etc. Reinforcement Learning and Robotics A total of 3 papers accepted at ICRA2024 including “Self-Recovery Prompting: Promptable General Purpose Service Robot System with Foundation Models and Self-Recovery ICLR2021, ICML2021, NeurIPS2021, ICLR2022 (Spotlight), ICLR2023 (Spotlight) etc. Transfer learning “Collective Intelligence for 2D push Manipulations with Mobile Robots”, RA-L, 2023 NeurIPS2021 (Spotlight), IJCAI2022 etc. Natural language processing (NLP) A total of 2 papers accepted at ICLR2024 including “A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis “ A total of 2 papers accepted at NAACL2024 including “ On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons “ “ Large-Language Models are Zero-Shot Reasoners”, NeurIPS2022 EMNLP2021×2, ACL2021, NAACL2022×2, EMNLP2022×2 ， ICLR2023 ， EMNLP2024 etc. Theory “ICML2023 ， ICLR2024 etc.

13. Overview | Education 3. Social Implementation 1. Fundamental Research 2. Education 4. Incubation Conduct basic research on Deep Learning (especially world models) and produce academic results R&D in the area of Deep Learning through collaboration with private companies. Human resource development programs for students and adults (not limited to students and faculty members of the University of Tokyo) Entrepreneurship education, entrepreneurship support, startup support

14. Education | Overview of lectures 30+ lectures a year in the 4 areas (Web, data science, deep learning, and entrepreneurship) Available for not only the students of Utokyo , but also students from other school Web Engineering and Business Model Web 工学 Fundamental project Web Engineering AI and Business M anagement Endowed Chair Web 工学 Global Consumer Intelligence (GCI) Endowed Course （ Data Scientist training ） Python Fundamental Data Science World Model Endowed Chair Web 工学 Data-driven Entrepreneurship Exercise Web 工学 Intense lecture （ Spring/Summer ） Deep generative modeling/Deep RL NLP/Image Recognition Lottery Hypothesis Financial Market Transactions and Machine Learning, LLM Deep Learning Fundamental Deep Learning Data-driven business planning exercise Web 工学 Invitation to DeepTech Entrepreneurs DeepTech Startup Visionary-Startup 14 Entrepreneurship

15. Education | Attendance The number of attendances has been increasing Increase in the attendance of junior high and high school students ※ 15 attendances 2024 (estimated) 25,000

16. Lecture | Global Consumer Intelligence (GCI) 16 An introductory AI course to learn fundamental data science and to acquire the problem-solving skill in business with a focus on marketing . Overview Practical online lectures and contents (exercises & competition) Launched in 2014 and taken by 15,000+ participants so far TAs’ office hours & Online community to deepen understanding Some startups are launched by students who took GCI Lecture 1 Introduction Lecture 2 Scientific Computing using Python （ Numpy ） Lecture 3 Fundamentals of Data Processing using Python （ Pandas ） Lecture 4 Fundamentals of Data Visualization using Python （ Matplotlib ） Lecture 5 Supervised learning Lecture 6 Unsupervised learning Lecture 7 SQL Lecture 8 Model validation and tuning methods Lecture 9 Feature engineering Lecture 10 Lecture on final assignment Lecture 11 Marketing fundamentals, some applications Lecture 12 Guest Lecturer （ TBA ） Lecture 13 Guest Lecturer （ TBA ） Curriculum （ 2024 Summer ）

17. Overview | Social Implementation 3. Social Implementation 1. Fundamental Research 2. Education 4. Incubation Conduct basic research on Deep Learning (especially world models) and produce academic results R&D in the area of Deep Learning through collaboration with private companies. Human resource development programs for students and adults (not limited to students and faculty members of the University of Tokyo) Entrepreneurship education, entrepreneurship support, startup support

18. Social Impact | Results and Case Studies Theme Industry Project description Image analysis Healthcare Development of image diagnosis algorithms to assist in the diagnosis of major dementias such as Alzheimer’s disease, by detecting micro-hemorrhages in MRI scans Behavior analysis Parts manufacturing Detection and visualization of people’s movement in the factory to analyze the cause of defective product rates and devise the transfer of skilled workers’ expertise Forecasting Chemicals Early detection of abnormalities in chemical plants and identification of their causes Deep Learning Medicine Increase the probability of success in clinical trials with low success rates and helping to deliver drugs to patients more quickly Deep Learning Information Optimization of the flow of goods and information to reduce waste loss, particularly of food products LLM Housing develop a platform (Minimum Viable Product) that connects home appliance users and home improvement contractors. (Photos are for illustrative purposes only.) Matsuo Lab is pursuing the application of research in a wide variety of topics and industries . In FY2023, 40 collaborative projects were conducted

19. Overview | Incubation 3. Social Implementation 1. Fundamental Research 2. Education 4. Incubation Conduct basic research on Deep Learning (especially world models) and produce academic results R&D in the area of Deep Learning through collaboration with private companies. Human resource development programs for students and adults (not limited to students and faculty members of the University of Tokyo) Entrepreneurship education, entrepreneurship support, startup support

20. Matsuo lab startups | 26 companies launched so far 20 Listed Matsuo Lab Startups = established by Matsuo Lab students + great technology and business capabilities + agree with Matsuo Lab’s philosophy on human resources development $1.5 billion+ Market cap combined M&A(2023) M&A(2024) * aim to Swing-by IPO

21. Incubation | Support for future entrepreneurs Education Lectures AI/Data Science web DeepTech Social Impact Joint research Startup-ready or Already Startup Wish to improve AI skills at real project or Immature to Startup Startup-ready AI skill training Business training Startup community Startup Invest ← 　　　　　　　　 Incubation 　　　　　　　　→