Master Thesis Slides: Topic Development of Methods for Deep Neural Network Architectures Optimization based on Tensor Factorization Algorithms

Current deep networks, having an excessive number of hyperparameters, have several optimums. They are trained until the value of the loss function is nearly zero (i.e., close to optimality), and training is considered successful if the optimal (or near-optimal) model thus found also works well on deferred data - i.e., it generalizes.
However, there is an opinion that the value of the loss function has nothing to do with the ability to generalize. Therefore, a wide range of compression methods have been developed, including pruning, quantization, distillation, and low-rank decomposition. In this work I look backwards to provide a comparative review of
what have been done previously to understand overall trends in deep learning, then discuss tensors and their decompositions that will help in developing an optimization method.

1. Development of Methods for Deep Neural
Network Architectures Optimization based on
Tensor Factorization Algorithms
Supervisor:Revin Ilya Evgenyevich, research associate,
Laboratory for Composite AI, Research Center “Strong AI in Industry”
Presented by: Zakharov Denis, J4232c

2. Pain: Current deep networks have excessive parameters which results in
long training time and expensive storing wights production.
Hypothesis: With the use of optimization method we can reduce size of
stored model, increase throughput.
Work: Development of Methods for Deep Neural Network Architectures
Optimization based on Tensor Factorization Algorithms
Problem
2

3. Purpose and objectives of study
Goal
Development of Methods for Deep Neural Network Architectures Optimization based on Tensor
Factorization Algorithms
Objectives
•Make a Literature review of related field to provide a background
•Make a scientific research of tensor Algorithms (Optimization, Operations)
•Make a scientific research of LoRA approach
•Perform experiment with TS Models
•Based on these findings propose a product that can be integrated into AutoML solution
•Develop Optimization Method
•Contribute to Fedot.Industrial framework
3

4. BERT 340MT5 11BGPT-3 175B
Megatron-Turing 530B
GPT-4 1.76T
Gemini Pro ≈30T
Gemini Ultra ≈60T
2018201920202021202220232024
ModelTrend
Modern Networks
4
Number
of parameters

5. Tensor Decomposition
5
!≈=
!!
"!
#!
"
!
"#$
$%
&
!≈
!!
!"!#
!!
!"
!#
!
!!≈
""
#!
$!!
!
"#$
!"# $!=∗ ∗
t-SVD (Singular Value Decomposition)CANDECOMP/PARAFAC
Tucker Block Term

6. Low-rank decomposition
6
! "!"#!≈×
!×#!×$
#×$

7. Statement of Experiment
7
Check performance of TS models NBEATS, Transformer, ARIMA on M4
dataset for checking how would they predict timeseries based on data.

8. Models
8
Transformer—state-of-the-art deep learning model introduced in
2017. It is an encoder-decoder architecture whose core feature is
the ‘multi-head attention’ mechanism
LinearLinearLinear
Scaled Dot-Product Attention
Concat
Linear
Multi-Head Attention
MatMul
SoftMax
Mask
Scale
MatMul
Scaled Dot-Product Attention
InOut
Multi-Head
Attention
Add & Norm
Feed
Forward
Add & Norm
Masked
Multi-Head
Attention
Add & Norm
Multi-Head
Attention
Add & Norm
Feed
Forward
Add & Norm
SoftMax
Linear
!
×
!
×

9. Models
9
NBEATS—deep learning-based approach for time series forecasting.
ARIMA—popular statistical model used to forecast future values in a time
series based on past values.
Block
FC Stack
FCFC
!!(#!)!"(#")
Backcast
Stack
Forecast
Stack
Block 1
Block 2
Block K
…
Forecast
Global
Forecast
Stack 1
Stack 2
Stack M
…

10. M4 Dataset
10
TheM4dataset is a collection of 100,000 time series used for the fourth
edition of the Makridakisforecasting Competition.
Consists of time series of:
•Yearly —63 avg training length
•Quarterly —125
•Monthly —302
•Weekly—2035
•Daily —475
•Hourly —682

11. Experiment - Monthly
11
ARIMANBEATS Transformer

12. Interpretation of results
12
On short series all three models performed well and suitable for predicting
However with longer range
•NBEATS perform better than others models
•Some models has a critical difference in predictions
•Longer Range all models not struggle
Training on just 50 epochs took:
•Almostan hour for NBEATS
•20 min for Transformer
•1 min for Arima

13. LoRA + rSVD
!∈#!×!
A = "(0,&!)
B = 0
r
x
h
!!
"×$
%
$×&
'
"×&
=
(
"×"
)
"×"
*!
"×&
SVD
recover
(
$×"
=
(+
"×"
!
$×"
13

14. Results NBEATS
Base Model
LoRA
Layers
Modellatencythroughput
No LoRA0.001084364808.0
LoRA Layer0.001664159606.0
MS Default0.001064748564.0
MS All0.001084482670.0
MS LoRA0.001055365210.0
14

15. Results Transformer
Modellatencythroughput
No LoRA0.000859306328.0
LoRA Layer0.000849782205.0
MS Default0.000859572622.0
MS All0.000859463282.0
MS LoRA0.000848586088.0
0
0,01
0,02
0,03
0,04
0,05
0,06
0,07
Epoch
1
Epoch
2
Epoch
3
Epoch
4
Epoch
5
Epoch
6
Epoch
7
Epoch
8
Model Train With Early Stopping
BaseLoRA
15

16. LoRA Implementation
14

17. Resume
In this work:
•Provide an Experiment for
research of performance:
•NBEATS
•ARIMA
•Transformer
•Logically assumed that layers in
models could be replaced using
LoRa approach
•Implement logic as a master
thesis
15
TN diagrams of some popular decompositions
!(")
!($)!(%)
"(")"($)"(%)"(&)
!!!"!#!$
!(",")
!(%,")
!(&,")
!(",%)
!(%,%)
!(&,%)
!(",&)
!(%,&)
!(&,&)
"""%"&
"'"(")
"*"+",
!(")
!"!#!$
…!($)!(%)%"%%&"%$
!(")
!"!#!$
…!($)!(%)%"%%&"%$
%'
!(")
"$
""
"%…
!(%)
!($)
!!
!"!#
#
!(")
"$
""
"%…
!(%)
!($)
#!!
!"!#
!(")
"$
""
"%…
!(%)
!($)
#!!
!"!#
#(")

18. THANK YOU
FOR YOUR TIME!
@misterzurg