Gen AI with LLM for construction technology

Gen AI with LLM for construction technology

1. 강태욱연구위원
공학박사
Ph.DTaewook, Kang
[email protected]
https://daddynkidsmakers.blogspot.com
건설AI 히치하이커를 위한가이드
GEN AI + LLM에초점을맞춘

2. KICT
Intro
Advanture. usecase from AI to Gen AI
My lab tools. open source
Jackpot and hallucination
Deep dive. Gen AI + LLM
Journey. Breadcrumbs
The Hitchhiker’s Guide to the Galaxy

3. KICT
Intro

4. Maker, Ph.D.
12 books author
BIM principle and Digital transformation: BIM principle & DX
사이트소개, Profile (dxbim.blogspot.com)

5. Study history
P4
Life cycle
“If it isn’t fun, you’re doing the wrong
technology.” -ivan sutherland

6. Study history
P4
Roadmap
Computer graphics, CAD
BIM-GIS + ISO (2012)
BIM + IoT + Scan (2014)
Scan to BIM + Robotics + AI (2017)
Digital Twin + AI (2021)BIM
VDCO
Mining
IoT
Scan
Virtual
Design
Construction
Operations
1D, 2D
3D, nD
Semantic
Knowledge
Sensors
Robotics

7. Usecase
advanture
CG
SCAN
ROBOT
AI
GEN AI
LLM
AEC

8. Generative AI identified by Deloitte
• Marketing content assistant
• Code assistant for developers
• Customer support on demand
• Product design assistant
• Research-based report generation
• Synthetic data generation
• Enterprise-wide data search and access
• Game content development
• Language translation
• Simulation of urban planning scenarios
• Hyper-personalised education
• Onboarding assistant
Usecase

9. Generative AI in AECO
• Multi Agent
• Anomaly detection in contract documents
• Reasoning in IoT dataset for O&M
• Risk management
• Optimization in Construction Management
• Concept design generation
• Report summarization and generation
• Training data generation
• Query using LLM
• Language translation
• Simulation of urban planning scenarios
• Personalised education and guidance system
Usecase

10. Scenario
CPS & DT
Digital
Twin
Structural
health
monitoring
Track and
trace
Remote
diagnosis
Remote
services
Remote
control
Condition
monitoring
Systems
health
monitoring
BIM
as i-DB
IoT…
AI
Sensor device
ICBM
SimulationRobotics
Scan-Vision
Smart contract
based on Blockchain
Gen AI
Multi Agent

11. Scenario
IoT
Big data
management
AI + Simulation using LLM
Cloud
platform
Machine control
Field monitoring
IoT
sensor
Usecase
for safety, accuracy, productivity
sensing
Data analysis
& prediction
GIS IoTbased monitoring
Field control
Infra IoT
service
connection
Plant control system (SCADA)
Field monitoring system
LoRA, BLE,
WiFi…
Layer 8 | IISL
(Infra IoTService Layer)Worker
Agency<device_definitionid=‘dd#1’>
<device id=‘T#1’ name=‘temp’ type=‘temperature’>
<maker name=‘CH korea’ email=‘[email protected]’ tel=‘82-0330-0802-1013’ location=‘…’/>
<specification>
<op_rangename=‘voltage’ unit=‘V’ type=‘real’ value=‘3.3’/>
<op_rangename=‘temperature’ unit=‘degree’ type=‘real’ begin=‘-10.0’ end=’60.0’/>
<op_rangename=‘humidity’ unit=‘%R.H’ type=‘real’ begin=‘0.0’ end=’50.0’/>
<op_rangename=‘GPS’ unit=‘WGS84’ type=‘vector2D’ begin=‘(0,0)’ end=‘(127, 32)’/>
<op_rangename=‘characteristic_curve’ unit1=‘temperature’ unit2=‘voltage’ type=‘vector2D’>
(0,0), (1.2, 2.4), (3.5, 6.2), (4.1, 7.2)
</op_range>
<op_rangename=‘period’ unit=‘year’ value=‘2’/>
</specification>
</device>
</device_definition>
Intelligent IoTsensor
•Self diagnose
•IISL protocol
•Security
•Availability
1
1
2
3
4
5
6
7
8
Plant sensing

12. Overview
AI ConTech $3B fundingfrom 2021 (Daniel Laboe, 2023.10)
2024년상반기스마트건설과BIM 기술동향(buildingSMART )
Machine
learning
Deep
learning
Gen AI
Multi-
agent
AGI

13. Usecase
AI 기반텍스트렌더링Revit애드인(VERAS, EnvolveLAB)

14. Usecase
https://daddynkidsmakers.blogspot.com/2023/04/ope
nai-for-grasshopper.html
OpenAI for Grasshopper

15. Usecase

16. Usecase
Technology to create 2D drawings with perfect dimensions from 3D
models with one click (Autodesk Fusion, November 2023)

17. Usecase
CreationAI-based design system AiCorb
(Obayashi Corporation)

18. Usecase
AI-based structural frame, automatic creation of member
cross sections SYMPREST (Shimizu Construction)

19. Usecase

20. Usecase
Funding $56.5M, 2021

21. Usecase
Pillar Technology

22. Usecase
Spot-R

23. Usecase
ANYboticsAG

24. Usecase
www.builtrobotics.com

25. Usecase
RebarTying로봇(FloridaDoT, 2020)

26. Usecase
Energy savings through
machine learning
AMR DNA, Energy savings through machine learning,
http://www.energyassets.co.uk/service/amr-dna

27. Usecase
Siemens Process Simulate (left) connects to NVIDIA Omniverse(right) to enable
a full-design-fidelity, photorealistic, real-time digital twin.
https://www.robotics247.com/article/siemens_xcelerator_nvidia_omniverse_accel
erate_digital_twins_manufacturing
NVIDIA Founderand CEO JensenHuang(left)
and SiemensCEO RolandBusch

28. My lab tools
open source

29. Open source
Alberto Rizzoli, 2022, 27+ Most Popular Computer Vision Applications and Use
Cases in 2022, V7

30. Open source
https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/controlnet.ipynb
https://github.com/mac999/gen_ai_gpt

31. Open source
GenerativeAdversarialNetworks
FakeObamacreatedusingAI videotool-
BBC News. Jul19, 2017
https://www.vegaitglobal.com/media-center/knowledge-
base/what-is-stable-diffusion-and-how-does-it-work

32. Open source CARLA -Open Urban Driving Simulator
https://github.com/carla-simulator/carla
YOUTUBE

33. Open source
Alberto Rizzoli, 2022, 27+ Most Popular Computer Vision Applications and Use
Cases in 2022, V7
YOLACT(You Only Look At CoefficienTs)
Detectron

34. Open source
Alberto Rizzoli, 2022, 27+ Most Popular Computer Vision Applications and Use
Cases in 2022, V7
Tesseract-OCR
UseGoogle CloudVisionAPI toprocess
invoicesand receipts

35. Open source
OpenNeuralNetwork eXchange

36. Open source

37. Cesium
http://cesiumjs.org/Seattle/
Open source
KICT

38. Open source

39. Open source
https://github.com/ZiwenZhuang/parkour?tab=readme-ov-file

40. Open source-github

41. Open source

42. Open source
http://guswnsxodlf.github.io/software-license
GNU General Public License(GPL) 2.0
–의무엄격. SW 수정및링크경우소스코드 제공의무
GNU Lesser GPL(LGPL) 2.1
–저작권표시. LPGL 명시. 수정한라이브러리 소스코드 공개
Berkeley Software Distribution(BSD) License
–소스코드 공개의무 없음. 상용SW 무제한사용가능
Apache License
–BSD와유사. 소스코드 공개의무 없음.
Mozilla Public License(MPL)
–소스코드 공개의무 없음. 수정코드는MPL에의해배포. 이외결합프로그램 코드는공개
필요없음
MIT License
–라이선스 / 저작권만 명시조건

43. Gym for AI Research

44. Gym for AI Research
Install pythonDownload Python | Python.org
Install anaconda Free Download | Anaconda
Install vscode Download Visual Studio Code -
Mac, Linux, Windows
Install sublime Download -Sublime Text
cmd*관리자권한으로실행해야함
cd \
mkdirprojects
cd projects
mkdirtest
pip install virtualenv
pip install jupyterlab
pip install ipywidgets

45. Gym for AI Research
1.pipinstalltensorflowpip로TensorFlow 설치
2.pip install keraskeras· PyPI
3.pipinstalltorchtorchvisiontorchaudioStart Locally | PyTorch

46. Gym for AI Research
Kaggle

47. Gym for AI Research

48. Gym for AI Research
pip install virtualenv
pip3 install virtualenv
.\venv\Scripts\activate
source ./venv/bin/activate

49. Gym for AI Research

50. Gym for AI Research
Install DockerGet Started | Docker
Creating a Simple Web Server with Docker: A Step-by-Step Guide to Running
Your Web Server as a Container | by SrijaAnaparthy| AWS Tip
Run cmd
docker run -d -p 8080:80 nginxdemos/hello nginxdemos/hello -Docker Image |
Docker Hub

51. Gym for AI Research

52. Gym for AI Research
1.Install Node.JSDownload | Node.js (nodejs.org)
2.Run cmd
3.cd c:\projects
4.mkdir test
5.cd test
6.npm install -g --unsafe-perm node-red
7.node-red

53. Gym for AI Research

54. Gym for AI Research
https://colab.research.google.com/

55. example
Trimble
2021.3
Deep Learning
IoT
GIS

56. example
Con Tech
2020.5

57. example
RANSAC
Dataset
= 730

58. example
Linear(4, out = 6)
ReLU()
BatchNorm1d(in = 6)
Linear(in = 6, out = 16)
ReLU()
BatchNorm1d(in = 16)
Linear(in = 16, out = 9)
ReLU()
…
ReLU()
BatchNorm1d(in = 3)
Linear(in = 3, out = 1))
epochs = 5000
learning_rate= 0.001
batch_size= 64
Loss=0.002. Train MAPE & Acc = (0.025, 95.30%). Test
MAPE & Acc = (0.292, 70.76%)

59. example
Trimble GPS
카메라
카메라
스캐너
IMU
DMI
KICT

60. example
https://github.com/mac999

61. Jackpot and hallucination

62. Gen AI core
Embedding

63. Gen AI core
https://daddynkidsmakers.b
logspot.com/2023/12/blog-
post.html

64. Gen AI core multi-modal
Variational autoencoder

65. Gen AI core
잠재공간에 맵핑된(인코딩된 ) 데이터(Alexej Klushyn, 2019.12, Learning Hierarchical Priors in VAEs)

66. Gen AI core multi-modal
CLIP(Contrastive
Language-Image Pre-
Training. Open AI. 2021
https://daddynkidsmakers.blogspot.com/2024/02/clip.html

67. Gen AI core
https://daddynkidsmakers.blogspot.
com/2024/02/llama-2.html

68. Gen AI core

69. Gen AI core
https://daddynkidsmakers.blogspot.
com/2024/02/llama-2.html

70. Gen AI core

71. Gen AI core

72. Gen AI core

73. Gen AI core
스테이블 디퓨전기술개발주역Computer Vision & Learning Group (ommer-lab.com)
Prof. Dr. Björn Ommer University of Munich

74. Jackpot & hallucination

75. Jackpot & hallucination

76. Jackpot & hallucination

77. Jackpot & hallucination

78. Jackpot & hallucination
output.append((transformer.generate(t
exts = ['bridge chair','computer table',
'chair table'] , temperature = 1) ))
output.append((transformer.generate(texts =
['sofa','bed', 'computer screen', 'bench', 'chair', 'table' ] ,
temperature = 0.0) ))
Not
practical.
MeshGPT

79. Jackpot & hallucination
Large Language Models as General Pattern Machines

80. Jackpot & hallucination
Large Language Models as General Pattern Machines

81. Deep Dive
Gen AI + LLM

82. Gen AI + LLM table data query

83. Gen AI + LLM LLAMA & Ollama & Langchain
A100 (80GB)GPU =
21 day (traning)
ollama run llava
"describe this
iamge: ./cat.jpg"
"테이블다리로보이는것옆에똑바로앉아있는얼룩무늬 고양이
의이미지입니다 . 고양이는 뚜렷한어두운줄무늬가 있는밝은주황
색털을가지고있는데, 이는흔히볼수있는패턴입니다 . 얼룩고
양이는눈을크게뜨고카메라를 정면으로 차분한태도로바라보고
있는모습입니다 .
배경에는 바닥에다음과같은패턴의러그가깔려있습니다 . 베이지
색과기타중성색이 포함됩니다 . 전경에테이블다리가있기때문에
다이닝룸과같은생활공간처럼 보입니다 ."

84. Gen AI + LLM LLAMA & Ollama & Langchain
Langchain CEO.2022.
30M$ in 2023.
https://daddynkidsmakers.blogsp
ot.com/2024/04/blog-
post_21.html

85. Gen AI + LLM
Chroma CEO. Jeff Huber,Anton Troynikov
18M$. 2023

86. Gen AI + LLM

87. Gen AI + LLM
class ChatPDF:
vector_store = None
retriever = None
chain = None
def __init__(self):
# OLLAMA의mistral 모델이용
self.model = ChatOllama(model="mistral")
# PDF 텍스트분할
self.text_splitter = RecursiveCharacterTextSplitter(chunk_size=1024, chunk_overlap=100)
self.prompt = PromptTemplate.from_template(
"""
<s> [INST] You are an assistant for question-answering tasks.
Use the following pieces of retrieved context
to answer the question. If you don't know the answer,
just say that you don't know. Use three sentences
maximum and keep the answer concise. [/INST] </s>
[INST] Question: {question}
Context: {context}
Answer: [/INST]
"""
)

88. Gen AI + LLM
class ChatPDF:
def ingest(self, pdf_file_path: str):
docs = PyPDFLoader(file_path=pdf_file_path).load()# 랭체인의 PDF 모듈이용해문서로딩
chunks = self.text_splitter.split_documents(docs) # 문서를청크로분할
chunks = filter_complex_metadata(chunks)
# 임메딩벡터저장소생성및청크설정
vector_store = Chroma.from_documents(documents=chunks, embedding=FastEmbedEmbeddings())
self.retriever = vector_store.as_retriever(search_type="similarity_score_threshold",
search_kwargs={
"k": 3,
"score_threshold": 0.5,
},
)# 유사도스코어기반벡터검색설정
self.chain = ({"context": self.retriever, "question": RunnablePassthrough()}| self.prompt| self.model|
StrOutputParser()) # 프롬프트 입력에대한모델실행, 출력파서방법설정
def ask(self, query: str):# 질문프롬프트 입력시호출
if not self.chain:
return "Please, add a PDF document first."
return self.chain.invoke(query)

89. Gen AI + LLM
https://daddynkidsmakers.blogspot.com/2024/
02/github-copilot-ai.html

90. Gen AI + LLM GeoLLM

91. Gen AI + LLM GeoLLM

92. Gen AI + LLM earthwork quantity takeoffH1 H1
H2
H1
H2
H3 H1
H2
H3
H4
H1
H2
H3
H4
L={H1, H2, H3, H4}
F1=L1
F2=L2
Step 1. Step 2.
Step 3. Step 4.
Step 5. Step 6.

93. Gen AI + LLM

94. Gen AI + LLM

95. Gen AI + LLM

96. Gen AI + LLM

97. Gen AI + LLMENA Model ID Batch
size
Epochs Layer architecture Normaliza
tion &
Activation
Learning
rate
Pre-trained model
embedding transformers
M1.1. MLP 32 150 [14]-[128,64,32]-[10] ReLU,
batch
normal,
dropout
0.001 - -
M1.2. MLP 32 150 [14]-[64,128,64]-[10] 0.001 - -
M1.3. MLP 32 150 [14]-[64,128,64,32]-[10] 0.001 - -
M1.4. MLP 32 150 [14]-[32,64,32]-[10] 0.001 - -
M2.1. LSTM 32 150 [14]-LSTM[128]-[10] dropout 0.001 - -
M2.2. LSTM 32 150 [14]-LSTM[128]-[64,32]-[10] 0.001 - -
M2.3. LSTM 32 150 [14]-LSTM[256]-[128,64]-[10] 0.001 - -
M3.1. Transformers
32 300 [320,512]-ENC[512]-[64]-[10]
multi-head
attention,
layer
normal,
dropout

1.00E-05
BERT-base-
uncased
-
M3.2.Transformers
64 300 [320,512]-ENC[512]-[64]-[10] 1.00E-05
BERT-base-
uncased
-
M3.3.Transformers
128 300 [320,512]-ENC[512]-[64]-[10] 1.00E-05
BERT-base-
uncased
-
M4.1. LLM
32 150 [293]-EMB-ENC-[10] 1.00E-05
BERT-base-
uncased
BERT
M4.2 LLM
32 300 [293]-EMB-ENC-[10]

1.00E-05
BERT-base-
uncased
BERT
1

98. Gen AI + LLMENA Model ID Train No Loss Accuracy Model size
(kb)
Time performance
(minutes)
M1.1. MLP 1650 0.0870 0.9494 61 0:02:34
M1.2. MLP 1714 0.0852 0.9519 84 0:02:36
M1.3. MLP 1716 0.0882 0.9544 93 0:02:42
M1.4. MLP 1718 0.1322 0.9507 30 0:02:25
M2.1. LSTM 1730 0.0889 0.9408 812 0:02:13
M2.2. LSTM 1732 0.0851 0.9420 850 0:02:17
M2.3. LSTM 1734 0.0886 0.9408 3,312 0:02:00
M3.1. Transformers 2003 0.3533 0.7744 74,557 0:11:06
M3.2.Transformers 2014 0.3551 0.7719 74,557 0:07:48
M3.3.Transformers 2021 0.3596 0.7423 74,557 0:06:25
M4.1. LLM 0103 0.0587 0.9507 427,783 3:12:05
M4.2. LLM 2334 0.0534 0.9531 427,783 7:59:00
1

99. Gen AI + LLM

100. Gen AI + LLMBIM RAG

101. Gen AI + LLMBIM RAG

102. Gen AI + LLM

103. Gen AI + LLM

104. Gen AI + LLM

105. Gen AI + LLMBIM RAG
Accuracy = 41% (7 / 17)
Domain specific RAG
challenge, direction
•Limited input context
for LLM
•No vocaboary
•Long token distance
•Hallucination

106. Gen AI + LLM how to make LLM dataset

107. Gen AI + LLM how to make LLM dataset

108. Gen AI + LLM how to make LLM dataset

109. Gen AI + LLM how to make LLM dataset

110. Gen AI + LLM how to make LLM dataset

111. Gen AI + LLMdomain knowledge fine tuning. ex) BIM-GIS
No fine tuning

112. Gen AI + LLMdomain knowledge fine tuning. ex) BIM-GIS

113. Gen AI + LLM

114. Gen AI + LLM
fine tuning. Epoch = 3 (90 min)

115. Gen AI + LLM
fine tuning. Epoch = 10 (over 5 hours)

116. Journey
breadcrumbs

117. guide
생성AI 시대BIM 기술동향과해외
스마트건설사례–이강, 정숭용
생성AI LLM과스테이블 디퓨전최신기술및
활용동향-최돈현, 김태영

118. guide
https://www.slideshare.net/laputa999

119. guide
https://github.com/mac999

120. guide
https://daddynkidsmakers.blogspot.com

121. Conclusion

122. conclusion
•Google, Meta, MS 등선진국테크기업 중심생성AI 잠재력테스트중
•파운데이션 모델개발은이미선진국테크기업 및중국중심
•AI 개발시빅테크기업 공개플랫폼&도구, Huggingface, W&B 등사용
은필수. 아키텍처 모델직접개발은극소수.
•LLM 은멀티모달 , 다중에이전트 플랫폼으로 발전
•GPU 리소스한계로RAG, finetuning 등다양한기술발전
•온라인문제로생성AI가스마트폰 , 노트북같은모바일장치에내장되
는추세
•Langchain과같은RAG, 에이전트 기술테크기업출현
•일부선진국중심으로 생성AI 기반건설테크기업출현
•생성AI가제대로동작될려면 챌린지와 연구가많이남아있음

123. conclusion

124. conclusion
Hitchhiker's Guide -Earth Destroyed and Guide
JoinGenAIworld