AWS recently announced that Aurora PostgreSQL can query Apache Iceberg and Parquet data stored in S3 alongside live operational data, using embedded DuckDB and integrations with catalogs and S3 Tables. The phrase “no ETL” is compelling: analysts could join a current order or asset state with years of history without first copying that history into the transaction database.
The opportunity is real, but no-ETL does not mean no engineering. Teams still need consistent identifiers, table ownership, access policy, file layout, statistics, freshness expectations, and protection for the operational workload.
Choose the right questions
Good candidates combine a small, selective operational set with a larger historical set: current customers against archived activity, active equipment against long-term telemetry summaries, or open orders against seasonal demand. Avoid unconstrained joins that pull large lake tables through an interactive transaction path.
Define which source is authoritative for each field. A lake snapshot may be minutes or hours behind. Show that freshness to users rather than silently blending timestamps.
Protect the operational database
Set query timeouts, concurrency limits, resource monitoring, and separate read paths where appropriate. Benchmark predicate pushdown, partition pruning, caching, and failure behavior. A poorly filtered historical query must not delay order entry or plant operations.
Observe bytes read, remote calls, planning time, execution time, cache effectiveness, and impact on writer and reader capacity. Compare the total cost with the ETL pipeline it replaces, including support effort and duplicate storage.
Govern the shared boundary
Use least privilege across database, catalog, and object storage. Align row, column, and tenant controls. Audit who queried which data. Iceberg table maintenance remains necessary: compact small files, manage snapshots, and preserve compatible schemas.
No-ETL architectures reduce movement; they do not remove semantics. Stable business keys, tested schema evolution, data-quality contracts, and lineage remain essential.
Roll out deliberately
Start with one read-only decision workflow. Validate results against the existing pipeline, test stale catalogs and unavailable storage, and define fallback. Keep high-volume transformation and training workloads on an analytical platform. Use the federated path where it shortens time to insight without turning production into a data warehouse.
The best outcome is not eliminating every pipeline. It is eliminating the copies that add cost and delay without adding control.
From announcement to dependable operations
Start with a bounded business problem and an accountable owner. Record the current cost, delay, failure rate, and manual work before changing the architecture. Define a measurable outcome, but do not present an estimate as guaranteed savings. Include security, finance, operations, and the people who will support exceptions after launch.
Map the current workflow before selecting technology. Identify the source of truth, approval points, downstream consumers, sensitive data, recovery objectives, and every manual workaround. Decide which step the new capability improves and which steps remain unchanged. This prevents a demonstration from being mistaken for an end-to-end operating solution.
Build a representative pilot with real permissions, data volume, dependency behavior, and user roles. Test peak demand, missing data, stale configuration, denied access, duplicate input, partial failure, recovery, and rollback. Document what continues when the new capability is unavailable. Keep an approved manual path for high-consequence decisions.
Security must be part of the design. Use separate workload identities, short-lived credentials, least privilege, encrypted transport and storage, controlled secrets, and auditable administrative access. Treat retrieved documents, event payloads, model output, and third-party responses as untrusted input. Validate every action at the boundary where it affects money, data, equipment, customers, or regulated records.
Cost controls need similar depth. Tag usage to a product and owner, define budgets and anomaly thresholds, measure cost per successful business outcome, and include integration, support, data movement, licensing, and training. A lower infrastructure bill can be offset by more exceptions or human correction. Evaluate total process cost rather than one service price.
Operate the result like a product. Monitor availability, latency, unit cost, data quality, security findings, user adoption, exceptions, and support effort. Version configurations and decision rules. Log changes and approvals. Review whether the capability still solves the original problem as prices, products, and workflows evolve.
Prepare the support model before launch. Give responders dashboards, runbooks, ownership, escalation paths, and safe diagnostic access. Alert on stale data and silent degradation, not only complete failure. Practice a dependency outage and verify that queued work, retries, reconciliation, and user communication behave as designed.
Change management is also technical risk control. Explain what users should trust, what they must verify, and how they report a wrong result. Train with real scenarios and capture overrides as useful feedback. Do not measure adoption by logins alone; measure whether the process becomes faster, safer, or more accurate.
Maintain a decision record that states why the technology was selected, which alternatives were considered, which risks were accepted, and what would trigger redesign or retirement. Review it after incidents and major vendor changes. Portability does not require avoiding managed services; it requires knowing which data, contracts, identities, and operational procedures must survive a change.
Expansion should follow evidence. Compare the pilot with its baseline, include the cost of integration and governance, and retire components that no longer earn their complexity. A new feature creates value only when teams can understand it, control it, recover it, and improve it without depending on one specialist.
Finally, schedule a formal review after thirty and ninety days. Ask whether users changed behavior, whether exceptions moved elsewhere, and whether actual costs match the forecast. Publish the findings to sponsors and delivery teams. If value is weak, narrow the scope or stop. Responsible innovation includes the ability to end an experiment cleanly.
Sources
Build it with Cogniquaint experts
Cogniquaint can assess federated-query use cases, design Aurora and Iceberg governance, tune lake layouts, protect transactional performance, and quantify whether removing a pipeline truly lowers total cost.
Work with Cogniquaint
Ready to elevate your operations with AI-powered insights?
Get in touch with us to build your next intelligent solution.
