Before adding compute, inspect the Parquet layout

A slow query can be doing very little useful work

A pipeline can spend much of its time discovering files, reading metadata, scheduling tasks, and decoding columns that the final result barely uses. Adding workers may shorten one stage while amplifying another. Before deciding that a dataset needs a larger cluster, I would ask what the engine actually read and how much of that work contributed to the requested answer.

This study uses a hypothetical event dataset stored as Parquet objects. It compares layouts rather than vendors or price lists. There are no performance results here: the goal is a reproducible experiment that can distinguish object overhead, scan volume, skew, and compute. Those distinctions matter because the remedies differ. Compaction, better partitioning, a narrower projection, and more memory are not interchangeable versions of the same optimization.

Separate files, row groups, and columns

A Parquet file contains row groups; each row group contains column chunks, which in turn contain pages. The Parquet concepts guide defines those levels, while the file-format description explains how metadata locates the chunks. That hierarchy gives a query engine opportunities to avoid irrelevant work. It does not guarantee that a particular workload or writer configuration makes those opportunities useful.

For example, selecting two columns from a wide table is different from reading every column and discarding most values afterward. Likewise, a selective predicate can only benefit from available statistics when those statistics distinguish the relevant portions of the data and the reader can use them. If every row group mixes the full range of the filter column, the metadata may rule out almost nothing.

I would therefore record the physical layout alongside the logical schema: file count and size distribution, row-group count, compression settings, sort or clustering choices, and the columns commonly projected. A single average file size hides a long tail of tiny objects and a few oversized ones. The distribution is often closer to the actual scheduling problem than the headline total number of bytes.

Partition for the questions people actually ask

A partition key is a decision about which objects a query can avoid opening. Date is a plausible candidate for time-bounded analysis, but the useful granularity depends on ingestion volume and query shape. A day partition that receives little data may already be small; adding several high-cardinality dimensions can produce a directory full of nearly empty fragments. A partition scheme should be justified against observed access patterns, not selected because its folder tree looks organized.

I would keep a representative query set covering narrow time windows, broad historical scans, selective entity lookups, and joins used by downstream consumers. Each layout receives the same logical records and query parameters. If one layout also changes sorting or compression, those changes become named factors in the experiment. Otherwise it is impossible to attribute the difference to partitioning with any confidence.

The collection also needs a schema-evolution policy. Compaction should not quietly coerce a decimal, change a timezone interpretation, or turn an absent field into a different business meaning. I would validate row identity and stable aggregates before comparing runtime. A smaller, faster dataset is not an improvement if it lost valid rows or changed the semantics of a column used in training.

Measure the work, then choose the intervention

A rough accounting model is useful even before precise instrumentation: total work includes object discovery, metadata reads, selected column reads, decoding, shuffles, and the final operation. These terms can overlap in wall-clock time, so their sum is not a promised latency formula. The point is to identify which observations would support a diagnosis and which would contradict it.

For a small-files hypothesis, I would look for high object or task counts relative to useful scan volume, then compare a compacted snapshot containing the same records. For a poor-pruning hypothesis, I would inspect selected row groups and bytes under the actual predicates. For a skew hypothesis, I would examine the distribution of partition and task sizes. An average duration can conceal one long-running partition that determines the whole query's completion time.

Spark's SQL performance guide documents file-partition planning settings and related execution controls. I would record engine version and configuration with each run rather than copy a setting out of context. Reader partition planning and writer file layout are related but distinct. Changing a reader setting does not rewrite the underlying objects, and a universal target file size is not established by the presence of a tuning option.

Treat compaction as a new snapshot

A compactor should write a candidate output to a separate location, validate it, and publish a new dataset version through the table or manifest mechanism the system actually supports. Readers should use one committed snapshot. Replacing files individually inside a live directory can expose a mixture of old and new objects, or cause double counting if readers discover both sets. A directory rename assumption deserves particular scrutiny when the storage API is object-based.

The illustrative plan below uses a manifest to make the intended boundary visible. It does not implement a table format or claim an atomic object-store directory operation. The publication mechanism must supply the actual concurrency guarantees. I would also retain the source manifest long enough to reproduce the comparison and recover from a bad publication. Cleanup is a later step, not evidence that the new snapshot was correct.

Validation should include a row-identity strategy appropriate to the data, schema compatibility, null patterns, and aggregates with defined precision. For append-only events, a deterministic event key can help identify missing or duplicate records. For mutable data, the current-state reconstruction rules need to be part of the comparison. Byte-for-byte equality is not generally expected after changing encoding or file boundaries; logical equivalence is the property being tested.

1{
2  "study": "layout-comparison",
3  "source_manifest": "events/snapshots/source-v1.json",
4  "candidate_manifest": "events/snapshots/compact-v2.json",
5  "same_logical_records": true,
6  "query_set": "queries/layout-study-v1",
7  "record": [
8    "engine_version", "writer_configuration", "object_count",
9    "row_group_count", "selected_bytes", "task_distribution",
10    "elapsed_time", "peak_memory", "validation_results"
11  ],
12  "publish_only_after": "logical validation and query comparison"
13}

The useful result explains when the layout helps

I would run repeated trials with a declared cache policy and separate preparation cost from query cost. A compaction job can improve many reads while being a poor trade for a dataset queried once. Similarly, a layout optimized for daily summaries can make single-entity lookups less convenient. The report should keep those cases visible rather than collapse them into one speedup number.

The decision record would include the workload distribution, retained snapshots, validation evidence, and the conditions that would trigger another layout review. Growing ingestion volume, a new access pattern, or a different join can change the right choice. That is why I prefer an inspectable experiment to a permanent rule about how many megabytes every file should contain.

If the evidence points to compute after those checks, adding resources has a clearer justification. The same applies if the simplest answer is to project fewer columns or fix an avoidable full scan. A good optimization changes the dominant work while preserving the dataset's contract. It should leave behind enough detail for another engineer to explain both the improvement and the workloads for which it does not apply.