Data platforms rarely become expensive because of one dramatic mistake. Cost often accumulates quietly: streaming jobs commit every few seconds, upserts create delete files, snapshots remain forever, and thousands of tiny objects replace a manageable set of well-sized files. Storage grows, query engines perform more planning and file-open work, and engineers add compute to compensate. That is the small-file tax.
Apache Iceberg separates table state from individual files and provides snapshots, schema evolution, and safe concurrent operations. Those capabilities do not remove maintenance. Iceberg’s own documentation explains that more data files create more manifest metadata, while small files cause unnecessary metadata and less efficient queries because of file-open costs. The lesson is practical: an open table format improves control, but teams still need an operating routine.
Why small files multiply
Frequent micro-batches prioritize freshness over file size. Highly partitioned tables spread each write across many destinations. Low-volume partitions may never accumulate enough rows for an efficient object. Merge, update, and delete patterns can add data and delete files. Retries and failed jobs may also leave orphaned files. None of these conditions is automatically a design failure; they are predictable consequences of workload choices.
The business effect appears later. A dashboard takes longer to open, a batch window slips, or a serverless query scans many more objects than expected. Teams scale clusters or raise concurrency, increasing spend while the physical layout continues to deteriorate. Maintenance should therefore be treated as part of ingestion design, not as emergency housekeeping.
Measure the tax before changing anything
Build a table-health view with file count, median and percentile file sizes, files per partition, snapshots retained, manifest count, delete-file count, orphan candidates, query planning time, bytes scanned, and maintenance compute. Segment by table and workload. A table receiving hourly append batches has different needs from a change-data-capture table with constant merges.
Link technical measures to a unit that leaders understand: cost per pipeline run, cost per terabyte served, query cost per dashboard refresh, or engineering hours spent on failed windows. Avoid claiming savings before establishing this baseline. Compaction consumes compute and writes data again; a maintenance job run too often can cost more than it saves.
Compaction is a policy, not a button
Iceberg provides rewrite-data-files operations that combine small files into larger ones. Choose target size from the query engine, object store, workload, and partition pattern rather than copying a universal value. Select candidates using size thresholds and limit each run so maintenance cannot monopolize compute. Partial-progress approaches can reduce the blast radius of large rewrites.
Flink-based Iceberg deployments can use the TableMaintenance API for small-file compaction, snapshot expiration, and orphan cleanup. The project documentation notes that native Flink maintenance can avoid operating a separate Spark environment solely for optimization. That can simplify architecture, but it does not remove the need for scheduling, observability, resource isolation, and rollback planning.
Expire history with intent
Every write can create a new snapshot. Time travel and rollback are valuable, but indefinite retention increases metadata and may retain files that no current workload needs. Define retention around recovery objectives, audit needs, branch and tag use, and downstream readers. Iceberg protects files still required by retained snapshots, branches, or tags, so teams must understand references before expecting storage to fall.
Separate snapshot expiration from orphan-file deletion. An orphan cleanup identifies files that table metadata no longer references; an unsafe retention setting can delete files from another writer whose commit has not yet become visible. Use a conservative age threshold, verify object-store consistency assumptions, and rehearse recovery. Never turn cleanup into an unreviewed recursive delete.
Tune ingestion before paying for cleanup
Maintenance treats symptoms. Reduce recurrence by lengthening streaming commit intervals where latency permits, avoiding excessive partition cardinality, clustering for common filters, and sizing writers appropriately. Iceberg’s streaming guidance recommends tuning commit rate because rapid commits produce more files, manifests, and snapshots. A one-minute trigger may be appropriate for one system and unacceptable for another; make the trade-off explicit.
For factory or IoT data, do not create one partition per device merely because each sensor has an ID. Use time and domain boundaries that match access patterns, then preserve device identity as data. Operational alerts can remain real time in a stream while the analytical lakehouse lands data in healthier batches. Freshness and file efficiency do not have to be solved by the same layer.
Operate maintenance like production software
Run compaction, snapshot expiration, manifest rewriting, and orphan cleanup through versioned configuration. Publish start, finish, files rewritten, bytes rewritten, snapshots removed, duration, failures, and estimated benefit. Cap concurrency and schedule around peak workloads. Alert when backlog or small-file ratios cross thresholds, not only when the job crashes.
Test on representative tables, verify readers during concurrent maintenance, and confirm that rollback and audit requirements remain satisfied. Review the policy quarterly as ingestion and query patterns change. The goal is not a perfectly tidy table; it is lower total cost and predictable performance.
A practical 30-day plan
Week one: inventory the most expensive tables and establish health metrics. Week two: tune one ingestion path and design a bounded compaction policy. Week three: add snapshot retention with stakeholder approval and dry-run orphan reporting. Week four: compare query latency, compute, object counts, and maintenance cost against baseline. Expand only where net value is demonstrated.
The fastest query optimization may not be another cluster. Sometimes it is simply stopping the data platform from paying interest on millions of tiny files.
Sources
Build it with Cogniquaint experts
Cogniquaint’s data engineering experts can assess lakehouse health, quantify small-file and retention overhead, design safe Iceberg maintenance, tune streaming writes, and build cost-and-performance dashboards that prove whether each optimization creates value.
Work with Cogniquaint
Ready to elevate your operations with AI-powered insights?
Get in touch with us to build your next intelligent solution.
