01Data engineering
Start with the question the data must answer.
Partitioning, file formats, and orchestration shape what a team can reliably ask of its data. The lesson I carry from AWS data engineering and Foundry is to design around access patterns and provenance together. A fast query is useful; a fast query whose inputs can be explained is a better foundation for research.
In practice. Document the source, the transformation, and the expected consumer. Make reruns safe before making them fast.
AWS Data Engineer · Palantir Foundry