Neural Texture Block Compression (MAM-MANER Workshop at EGSR 2024)

Block compression is a widely used technique to compress textures in real-time graphics applications, offering a reduction in storage size. However, their storage efficiency is constrained by the fixed compression ratio, which substantially increases storage size when hundreds of high-quality textures are required. In this paper, we propose a novel block texture compression method with neural networks, Neural Texture Block Compression (NTBC). NTBC learns the mapping from uncompressed textures to block-compressed textures, which allows for significantly reduced storage costs without any change in the this experiments show that NTBC can achieve reasonable-quality results with up to about 70% less storage footprint, preserving real-time performance with a modest computational overhead at the texture loading phase in the graphics pipeline.

https://arxiv.org/abs/2407.09543

1. 1|
Neural Texture Block Compression
Shin Fujieda, Takahiro Harada

2. 2|Neural Texture Block Compression | MAM-MANER 2024 | July 2, 2024
Motivation
•Textureis a key component to achieve high visual fidelity through material properties
•High-resolution textures (e.g., 4K) require a lot of storage
•Block compression (BC) is one of the most popular techniques to compress textures
•All variations, BC1 –BC7, compress each 4x4 texel block to a fixed number of bytes
•BC1 and BC4: 8 bytes, others: 16 bytes
•Consumes 8MB (= 4096*4096 / (4*4) * 8 bytes) for a single 4K texture with BC1
•To reduce the storage costs, we propose Neural Texture Block Compression (NTBC)
•Compress textures in BC1/BC4 formats using a multi-layer perceptron (MLP)
•Not require any change in the shader execution

3. 3|Neural Texture Block Compression | MAM-MANER 2024 | July 2, 2024
Pipeline -conventional
…
Uncompressed
textures
Disk VRAM
�
0�
1Indices
�
0�
1Indices
�
0�
1Indices
�
0�
1Indices
…
Block-compressed
data
Shader
fetch
texel
value
BC
�
0�
1Indices
�
0�
1Indices
�
0�
1Indices
�
0�
1Indices
…
Block-compressed
data
texture
loading
Pre-process
…
…
…
Mat A
Mat B
Mat C

4. 4|Neural Texture Block Compression | MAM-MANER 2024 | July 2, 2024
Pipeline - NTBC
…
Uncompressed
textures
Disk
VRAM
�
0�
1Indices
�
0�
1Indices
�
0�
1Indices
�
0�
1Indices
…
Block-compressed
data
Shader
fetch
Pre-process
…
…
…
Mat A
Mat B
Mat C
Neural models
……
……
…
model
training inference
……
texel
value

5. 5|Neural Texture Block Compression | MAM-MANER 2024 | July 2, 2024
Block Compression
•Block compression (BC) encodes 4x4 texel blocks into a fixed-size structure
•Each block contains a color palette with colors on a line segment in RGB space
•Two endpoints + linear-interpolations
R
B G
BC1
•RGB images
•2 RGB565 endpoints (4 bytes)
•Palette has 4 entries
•16 2-bit indices (4 bytes)
•0≤??????≤3
BC4
•Single-channel images
•2 8-bit endpoints (2 bytes)
•Palette has 8 entries
•16 3-bit indices (6 bytes)
•0≤??????≤7
How can we encode these block-compressed textures using neural networks?

6. 6|Neural Texture Block Compression | MAM-MANER 2024 | July 2, 2024
Naive approach
•Colors of the palette are evenly spaced on a line segment in RGB space:
•Encoding �
0,�
1,??????
?????? using NNs should be a straight-forward approach
… …
… …
1−??????
??????×??????
�+??????
??????×??????
�
Endpoint network
Weight network
??????
??????
??????
�
??????
�
??????
??????=1−??????
??????×??????
�+??????
??????×??????
�
??????
�,??????
�: two endpoints, ??????
??????: weights of
??????
3
(BC1),
??????
7
(BC4)
R
B G
�
0
�
1
�
0
�
1
�
2
�
3

7. 7|Neural Texture Block Compression | MAM-MANER 2024 | July 2, 2024
Naive approach
??????
�
… …
… …
??????
??????
Endpoint network
Weight network
??????
??????
Quantize
Texture coordinate
??????
�
??????
�
Multi-resolution feature grids
�
0
??????
�
0
�
�
0
??????
�
1
??????
�
1
�
�
1
??????
…
…
2D block index
…
…
Weights
0
3

8. 8|Neural Texture Block Compression | MAM-MANER 2024 | July 2, 2024
Neural Texture Block Compression
… …
??????
�
??????
�
�
0
??????
�
0
�
�
0
??????
�
1
??????
�
1
�
�
1
??????
…
…
…
…
Color network
R
G
B
… …
??????
??????
??????
??????
??????
Distances
�
??????
Uncompressed colors
R
B G
�
0
�
1
�
2
�
3
�
�
??????
Endpoint network
Texture coordinate
Multi-resolution feature grids
2D block index

9. 9|Neural Texture Block Compression | MAM-MANER 2024 | July 2, 2024
NTBC – model configuration
•Multi-resolution feature grids
Endpoint network
•7 levels
•Resolutions from 16 to 1024
•2D features per level
Color network
•8 levels
•Resolutions from 16 to 2048
•2D features per level
•Small MLPs with FP16
•64 x 3 hidden layers
•Adam optimizer

10. 10|Neural Texture Block Compression | MAM-MANER 2024 | July 2, 2024
NTBC – grid quantization
•Multi-resolution feature grids
Endpoint network
•7 levels
•Resolutions from 16 to 1024
•2D features per level
Color network
•8 levels
•Resolutions from 16 to 2048
•2D features per level
5.33 MB
21.33 MB
FP16
Quantization
2.67 MB
10.67 MB
FP8
+ +
= =
26.67 MB 13.33 MB
×2 for RGB and single-channel textures
26.67 MB

11. 11|Neural Texture Block Compression | MAM-MANER 2024 | July 2, 2024
Comparisons
Diffuse Normal Displacement
Uncompressed textures
NTBC,
26.7 MB
Ref. BC,
48 MB
38.64 dB 35.38 dB 38.87 dB
42.53 dB 38.34 dB 49.39 dB

12. 12|Neural Texture Block Compression | MAM-MANER 2024 | July 2, 2024
Comparisons
Uncompressed
textures
NTBC
26.7 MB
Naive approach
26.7 MB
Ref. BC
40 MB
27.81 dB 30.98 dB 37.44 dB
38.53 dB 44.20 dB 53.06 dB
Diffuse
Displacement

13. 13|Neural Texture Block Compression | MAM-MANER 2024 | July 2, 2024
Performance
•Evaluate training + inference performance with a single AMD Radeon
TM
RX 7900 XT GPU
•Use “MetalPlates013” material
•2 RGB textures
•4 single-channel textures
Diffuse Normal AO Displacement Roughness Metalness

14. 14|Neural Texture Block Compression | MAM-MANER 2024 | July 2, 2024
Performance - Training
•NTBC training time over PSNR averaged over all the textures in the material
•20k iterations gives saturated results in 16 minutes
•2k iterations provides reasonable quality just in 100 seconds

15. 15|Neural Texture Block Compression | MAM-MANER 2024 | July 2, 2024
Performance - Inference
…
Uncompressed
textures
Disk
VRAM
�
0�
1Indices
�
0�
1Indices
�
0�
1Indices
�
0�
1Indices
…
Block-compressed
data
Pre-process
…
…
…
Mat A
Mat B
Mat C
Neural models
……
……
…
model
training
inference
……
Overhead
BC1 & BC4: 49.84 ms
BC1 or BC4: 25.57 ms

16. 16|Neural Texture Block Compression | MAM-MANER 2024 | July 2, 2024
Limitations
•Color degradation
•Happens if the texture contains high-frequency details both in luminance and colors
•Loss of detailed content
•Block artifacts and blurred details
•NTBC uses lower-resolution grids than the original texture, which could cause the errors
Color degradation Block artifacts Blurred detail

17. 17|Neural Texture Block Compression | MAM-MANER 2024 | July 2, 2024
Future Work
•Input encoding specialized for textures
•Use frequency information to handle high-frequency details efficiently
•Extension to more complex formats such as BC6H and BC7
•BC1 and BC4 are simple but only show limited-quality results
•Mipmap support
•Evaluation in practical real-time applications