File Formats
CSV/TSV
Pros
Tabular Row storage.
Human-readable is easy to edit manually.
Simple schema.
Easy to implement and parse the file(s).
Cons
No standard way to present binary data.
No complex data types.
Large in size.
JSON
Pros
Supports hierarchical structure.
Most languages support them.
Widely used in Web
Cons
More memory usage due to repeatable column names.
Not very splittable.
Lacks indexing.
AVRO
Avro is a data serialization system often used in big data processing and storage frameworks like Apache Hadoop.
Examples of use cases for AVRO files include RPC, message queues, log files, and data pipelines.
Parquet
Parquet is a columnar storage file format optimized for use with Apache Hadoop and related big data processing frameworks. Twitter and Cloudera developed it to provide a compact and efficient way of storing large, flat datasets.
Best for WORM (Write Once Read Many).
The key features of Parquet are:
Columnar Storage: Parquet is optimized for columnar storage, unlike row-based files like CSV or TSV. This allows it to efficiently compress and encode data, which makes it a good fit for storing data frames.
Schema Evolution: Parquet supports complex nested data structures, and the schema can be modified over time. This provides much flexibility when dealing with data that may evolve.
Compression and Encoding: Parquet allows for highly efficient compression and encoding schemes. This is because columnar storage makes better compression and encoding schemes possible, which can lead to significant storage savings.
Language Agnostic: Parquet is built from the ground up for use in many languages. Official libraries are available for reading and writing Parquet files in many languages, including Java, C++, Python, and more.
Integration: Parquet is designed to integrate well with various big data frameworks. It has deep support in Apache Hadoop, Apache Spark, and Apache Hive and works well with other data processing frameworks.
In short, Parquet is a powerful tool in the big data ecosystem due to its efficiency, flexibility, and compatibility with a wide range of tools and languages.
Difference between CSV and Parquet
| Dataset | Size | Query Run Time | Data Scanned | Cost |
|---|---|---|---|---|
CSV | 1 TB | 236 Seconds | 1 TB | $5.75 |
Parquet | 130 GB | 6.78 Seconds | 2.51 GB | $0.01 |
Savings | 87% less space | 34x faster | 99% less data scanned | 99.7% savings |
Parquet Compression
Snappy (default)
Gzip
Snappy
Low CPU Util
Low Compression Rate
Splittable
Use Case: Hot Layer
Compute Intensive
GZip
High CPU Util
High Compression Rate
Splittable
Use Case: Cold Layer
Storage Intensive
Last updated