The formats the data world hands around
Parquet, Avro, ORC, Arrow and Feather files open like any other file — by double-click, from the open panel, or dropped on the window — and become ordinary tables. So do Delta Lake and Iceberg tables, folder and all, and the pickles pandas writes. The grid, SQL, macros and every export work on them unchanged, and saving writes the same format — to a copy, so the file you opened stays exactly as it was. They are read by the app itself: no Python, no Spark, nothing else to install.
Apache Parquet
- Types: booleans, 32- and 64-bit integers signed and unsigned, floats, doubles, byte arrays, fixed-length byte arrays and the old 96-bit timestamps.
- Meanings: strings, dates, times, timestamps in milli-, micro- and nanoseconds, decimals (including ones wider than 64 bits), UUIDs, JSON and enumerations. A decimal keeps its own point rather than being rounded through a double.
- Encodings: plain, dictionary, run-length and bit-packed, delta-packed integers and both delta byte-array forms; both page versions and several row groups.
- Compression: every one Parquet names — Snappy, gzip, LZ4, Zstandard, Brotli and LZO — and none.
A repeating column — a list — has nowhere to go in a table, so it is named in the note under the window rather than
dropped in silence; the fields of a struct become ordinary columns such as who.first.
Apache Avro and ORC
Avro: records, enumerations (read back as their names), arrays, maps and unions; dates, times, timestamps,
exact decimals, UUIDs and durations; the deflate and Snappy codecs. A union of null and one type becomes
an ordinary empty cell.
ORC: the files Hive and Spark write — every run-length encoding, dictionary strings, zlib, Snappy, LZ4 and Zstandard. Timestamps are read against the writer's own time zone, as each stripe names it, so a moment comes out as it was written down.
Delta Lake and Apache Iceberg
Both keep their rows in ordinary Parquet files and a log of which files make up the table now. A folder read by listing its Parquet files shows rows that were deleted and changed rows twice — so the log is read first, and the table opens as it stands. A table handed over on a disc has moved, so paths in the log are found where the files are. These open locked: saving one would mean writing a commit.
Arrow, Feather, pickles and MessagePack
- Arrow and Feather: IPC files and both versions of Feather, column by column — categories and timestamps to the nanosecond included.
- Python pickles: pandas DataFrames and Series, NumPy arrays and lists of records, read without running any of the pickle — a pickle that would run a command in Python is refused by name. Written back as a DataFrame.
- MessagePack and BSON: a document to a row, whole numbers to the digit.
Checked against pandas and pyarrow
What the app writes is read back by pandas, pyarrow, R and the other libraries people use — thirty-seven formats written here, every value compared — and files they write open here with every value as it was.
