論文紹介：A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models

Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin, Volker Tresp, Philip Torr "A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models" arXiv2023

https://arxiv.org/abs/2307.12980

1. A Systematic Survey of Prompt
Engineering on Vision-Language
Foundation Models
Jindong Gu, Zhen Han, ShuoChen, Ahmad Beirami, BailanHe, GengyuanZhang,
RuotongLiao, Yao Qin, Volker Tresp, Philip Torr
arXiv2023
2024/6/19

2. ◼
•Vision-Language Prompt Engineering
◼Prompt
•AI
◼Prompt Engineering
•Prompt
ChatGPT
LLM 5
AI

3. ◼ / Prompt Engineering
◼ Prompt Engineering
◼ Prompt Engineering

4. ◼ / Prompt Engineering

5. /
◼LLM (
•
◼Prompt Engineering
•

6. LLM models
◼BERT [Devlin+, arXiv2018]
◼T5 [Raffel+, arXiv2019]
◼GPTs [Brown+, arXiv2020]
◼ViLBERT[Lu+, arXiv2019]
◼OFA [Wang+, arXiv2022]
◼Flamingo [Alayrac+, NeurIPS2022]
◼SimVLM[Wang+, arXiv2021]
◼PaLI[Chen+, arXiv2022]
◼MAGMA [Eichenberg+, arXiv2021]
◼BLIP2 [Li+, arXiv2023]
◼VL-T5 [Cho+, arXiv2021]
◼Frozen [Tsimpoukelli+, arXiv2021]
◼PICa[Yang+, arXiv2021]
◼FlanT5 [Chung+, arXiv2022]

7. Prompt
◼Hard Prompt
•Task instruction prompting
•In-context learning
•Retrieval-based prompting
•Chain-of-thought prompting
◼Soft Prompt
•Prompt tuning
•Prefix Token Tuning

8. Hard/Soft Prompt
◼Hard Prompt
• Prompt
CLIP
•‘ A photo of a ’text
◼Soft Prompt
•
Vision Transformer
• class token

9. Hard Prompt
◼Task instruction prompting
•[Efrat&Levy, arXiv2020]
◼Retrieval-based prompting
•PICa[Yang+, arXiv2021]
•[Rubin+, arXiv2021]
•[Li+, arXiv2023]
•[Ye+, arXIv2023]
◼In-context learning
•[Dong+, arXiv2022]
•GPTs [Brown+, arXiv2020]
◼Chain-of-thought prompting
•[Wei+, arXiv2022]
•[Zhang+, arXiv2022]
•[Qiao+, arXiv2022]

10. Hard Prompt
◼Task Instruction Prompting
• Propt
Prompt: Translate English to French
Prompt: Translate French to English
◼Retrieval-based prompting
•
[Rubin+, arXiv2021][Efrat+, arXiv2020]

11. Hard Prompt
◼In-context learning
•
◼Chain-of-thought prompting
• (=Prompt)
[Dong+, arXiv2022]
PositiveNegative

12. Soft Prompt
◼Prompt Tuning
• Prompt
•
◼Prefix Token Tuning
• Prompt
•
[Lester+, arXiv2021]
[Li+, arXiv2021]

13. Application
◼Visual Question Answering
•Flamingo [Alayrac+, NeurIPS2022]
•[Yang+, AAAI2022]
•[Tsimpoukelli+, NeurIPS2021]
◼Zero-shot Image Classification
•Kosmos [Huang+, arXiv2023]
◼Image Captioning
•PaLI[Chen+, arXiv2022]
•MAGMA [Eichenberg+, arXiv2021]
•SimVLM[Wang+, arXiv2021]
•OFA [Wang+, arXiv2022]
◼Chatbot
•Visual CahtGPT[Wu+, arXiv2023]
•ChatGPT
•GPT4 [Achiam+, arXiv2023]
•BiomedGPT[Zhang+, arXiv2023]

14. ◼ Prompt Engineering

15. ◼CLIP [Radford+,arXiv2021]
•Image-text matching
•Image, textEncoderTransformer

16. Prompt
◼Text Prompting
•Textprompt text encoder
◼Visual Prompting
•imageprompt image encoder
◼Unified Prompting
•Image, Text prompt encoder

17. Text Prompting
◼Hard Prompt
•“a photo of a” [Radford+,arXiv2021]
◼Soft Prompt
•Global Soft Prompt [Gao+, arXiv2020], [Shu+,arXiv2022], [Zhou+,arXiv2021]
• Prompt
•Group-specific Prompt [Ju+, arXiv2021],[Shen+,arXiv2022]
• Prompt
•Instance-specific Prompt [Zhou+,CVPR2022]
• Prompt

18. Text Prompting: Soft Prompt
◼Global Soft Prompt
◼Group-specific Prompt
◼Instance-specific Prompt

19. Image Prompting
◼Patch-wise Prompts
•
•
•[Jia+, ECCV2022]
•[Bahng+, arXiv2022]
•[Shen+, arXiv2022]
•[Wu+, arXiv2022]
•[Huang+, CVPR2023]
◼Annotation Prompts
•
•
•CPT [Yao+, arXiv2021]
•[Shtedritski+, ICCV2023]
•[Bar+,arXiv2022]

20. Image Prompting
◼Patch-wise Prompts
•[Jia+, ECCV2022]
•[Bahng+, arXiv2022]
◼Annotation Prompts
•[Shtedritski+, ICCV2023]

21. Unified Prompting
◼Coupled Unified Prompting
•Textimage Prompt
•
•[Zang+, arXiv2022]
◼Decoupled Unified Prompting
•Textimage Prompt
•
•[Shen+, arXiv2022]
•CPT [Yao+, arXiv2021]
•MaPLe[Khattak+, CVPR2023]

22. Unified Prompting
◼Coupled Unified Prompting
•[Zhang+, arXiv2022]
◼Decoupled Unified Prompting
•[Khattak+, CVPR2023]

23. Application
◼Image Classification
•CLIP [Radford+,arXiv2021]
•TPT [Shu+, arXiv2022]
◼Object Detection
•[Gu+, arXiv2021]
•[Guo+, arXiv2022]
•DualCoOp[Sun+, arXiv2022]
•[Du+, CVPR2022]
•PromptDet[Feng+, ECCV2022]
◼Semantic Segmentation
•Denseclip[Rao+, CVPR2022]
•[Kirillov+, arXiv2023]
◼Domain Adaptation
•[Ge+, arXiv2022]
•[Gao+, arXiv2022]
◼Continual Learning
•[Wang+, CVPR2022]
•Dualprompt[Wang+, ECCV2022]

24. ◼ Prompt Engineering

25. ◼
•DRAW [Gregor+, arXiv2015]
•GAN [Goodfellow+, arXiv2014]
•VAE [Kingma+, arXiv2019]
•DALL-E [Ramesh+, arXiv2021]
•Parti [Yu+, arXiv2022]
◼Prompt Engineering
•

26. Prompt
◼ Prompt
•[Witteveen+, arXiv2022]
◼AI Prompt
•DiffuMask[Wu+, ICCV2023]
•ImaginaryNet[Ni+, ICLR2022]
•[He+, ICLR2022]
•[Avrahami, arXiv2022]
•Gldie[Nichol, arXiv2021]
•OneWord[Gal+, arXiv2022]
•DreamBooth[Ruiz+, arXiv2022]
•Custom Diffusion [Kumari+, CVPR2023]
•[Feng+, arXiv2022]
•[Epstein+, arXiv2023]
•Imagic[Kawar+, CVPR2023]
•[Zhang+, ICCV2023]
•[Hertz+, arXiv2022]

27. Prompt
◼Investigating Prompt Engineering in Diffusion Models [Witteveen+, arXiv2022]
• (Prompt)
•
•
•
•
•

28. AI Prompt
Wu+, I CCV 2023
◼DiffuMask[Wu+, ICCV2023]
•Prompt Engineering
’photo of bird’
1.??????[sub class]Wiki
2.Clipretrieval ??????
Photo of bird
Photo of Duck bird Photo of Swan bird
…
??????
Photo of Duck bird
Photo of a big Duck bird Photo of a pretty Duck bird…
??????

29. AI Prompt
◼ImaginaryNet[Ni+, ICLR2022]
•
•Prompt Engineering
• 1 ”A photo of a [class]”
•
• Prompt

30. Application
◼
•DiffuMask[Wu+, NeurIPS2023]
•ImaginaryNet[Ni+, ICLR2022]
•[He+, arXiv2022]
◼
•Text to video
•Make-A-Video [Singer+, arXiv2022]
•Imagin video [Ho+, arXiv2022]
•FateZero[Qi+, arXiv2023]
•Tune-A-Video [Wu+, arXiv2023]
•[Ruan+, CVPR2023]
•MovieFactory[Zhu+, arXiv2023]
•Text to 3D
•DiffRF[Muller+, CVPR2023]
•DreamFusion[Poole+, arXiv2022]
•NeRF[Mildenhall+, ACM2021]
•Magic3D [Lin+, CVPR2023]
•Dream3D [Xu+, CVPR2023]
•[Lee+, arXiv2022]
•[Khalid+, arXiv2022]

31. ◼Prompt Engineering
•
•
•
•
◼
•
•
•

32. Prompting
◼ Prompt Engineering
•Text prompting
1.Hard Prompt
1.Task instruction prompting
2.In-context learning
3.Retrieval-based prompting
4.Chain-of-thought prompting
2.Soft Prompt
1.Prompt tuning
2.Prefix Token Tuning

33. Prompting
◼ Prompt Engineering
•Text Prompting
1.Hard Prompt
2.Soft Prompt
1.Global Soft Prompt
2.Group-specific Prompt
3.Instance-specific Prompt
•Visual Prompting
1.Patch-wise Prompts
2.Annotation Prompt
•Unified Prompting
1.Coupled Unified Prompting
2.Decoupled Unified Prompting

34. Prompting
◼ Prompt Engineering
• Prompt
•AI Prompt