Apache Parquet

https://www.youtube.com/watch?v=1Kur7SitXRA&index=5&list=PLAV_dWz2GNAiDnEVI9ynfVr3WYt3moS3c

1. Apache Parquet https://parquet.apache.org/

2. Why Parquet Columnar storage format Can store nested Data E fficiency in file size and query performance Nested field can be read independently of other fields Many data processing understand avro format (Hive, Spark, Pig and MapReduce,etc)

3. Data Model boolean int32 int64 int96 float double binary fixed_len_byte_array UTF8 ENUM DECIMAL(precision,scale DATE LIST MAP

4. Record parquet message Nom { (required,optional,repeated) type nom ; required int32 age ; required binary nom (UTF8) } {"name": "right", "type": "string", "order": "descending"} Avro Parquet

5. Parquet File Format Header Block .. Block Footer Magic number Using parquet Magic number ,Schema,Encoding method,Block position,.. Columns Pages 128MB

6. Encoding & Compression Run-length : True , True ,True ,False (3 true , 1 false) Dictionary encoding ( indexation ) Plain encoding Compression algorithms Snappy,gzip,...

7. Configuration parquet.block.size int parquet.page.size int parquet.dictionary.page.size int parquet.enable.dictionary boolean true parquet.compression String (SNAPPY,UNCOMPRESSED,LZO...)

8. Writing and reading Parquet files

9. Parquet MapReduce

10. Parquet MapReduce

11. Parquet MapReduce