The formats the data world hands around
Parquet, Avro, ORC and Arrow files open like any other file — by double-click, from the open panel, or dropped on the window — and become ordinary tables. The grid, SQL, macros and every export work on them unchanged, and saving writes the same format back. They are read by the app itself: no Python, no Spark, nothing else to install.
Apache Parquet
- Types: booleans, 32- and 64-bit integers signed and unsigned, floats, doubles, byte arrays, fixed-length byte arrays and the old 96-bit timestamps.
- Meanings: strings, dates, times, timestamps in milli-, micro- and nanoseconds, decimals (including ones wider than 64 bits), UUIDs, JSON and enumerations. A decimal keeps its own point rather than being rounded through a double.
- Encodings: plain, dictionary, run-length and bit-packed, delta-packed integers and both delta byte-array forms; both page versions and several row groups.
- Compression: Snappy, gzip, LZ4 and uncompressed. Zstandard, Brotli and LZO are named rather than guessed at.
A repeating column — a list — has nowhere to go in a table, so it is named in the note under the window rather than
dropped in silence; the fields of a struct become ordinary columns such as who.first.
Apache Avro and ORC
Avro: records, enumerations (read back as their names), arrays, maps and unions; dates, times, timestamps,
exact decimals, UUIDs and durations; the deflate and Snappy codecs. A union of null and one type becomes
an ordinary empty cell.
ORC: the files Hive and Spark write — every run-length encoding, dictionary strings, zlib, Snappy and LZ4. Timestamps are read against the writer's own time zone, as each stripe names it, so a moment comes out as it was written down.
Arrow, Feather and BSON
Arrow IPC and Feather files open column by column — including the Feather files pandas writes. BSON, the stuff of a MongoDB dump, opens a document to a row.
Checked against Apache Arrow
What the app writes is a real Parquet or ORC file: the tests have Apache Arrow read what this app writes, and read what Apache Arrow writes.
