Explore everything pg-cdc can do. Select a capability to see how it delivers governed, real-time operational data for analytics and AI.
Continuous WAL Capture
pg-cdc streams inserts, updates, and deletes in near real-time straight from PostgreSQL logical replication slots — no triggers, no dual-writes, no impact on your application path.
Reads directly from logical replication slots (pgoutput).
No application changes — No triggers, dual-writes, or code modifications required.
Minimal database overhead — Reads WAL changes without impacting your application’s transaction path
Exactly ordered change stream — Preserves PostgreSQL commit order for consistent downstream processing
Governed data lake output — Writes optimized Parquet/Iceberg datasets to Amazon S3 for analytics and AI
Schema Evolution Without Downtime Handles column additions, removals, and type-compatible changes while keeping downstream consumers running.
Exactly-Once Delivery Guarantees Prevents duplicate records across retries and restarts, ensuring consistent and reliable datasets.
Open Table Formats, Zero Vendor Lock-in Your data remains in open Apache Iceberg or Parquet formats, allowing you to switch query engines without migrations.
Fault-tolerant recovery — Resumes safely from PostgreSQL replication slots after interruptions.
AI-Ready Historical Context Every Iceberg snapshot preserves a complete historical view of your operational data, enabling reproducible analytics, auditing, and AI workflows with built-in time travel.
Built for AI and analytics — Creates governed operational data that can be consumed by data platforms and AI agents without exposing production databases.
Every change lands as open-format Apache Iceberg (or Hive-style Parquet) on S3 — queryable from Athena, Redshift Spectrum, EMR, Spark, Trino, and DuckDB without proprietary lock-in.
Columnar Storage for Analytics — Optimized Parquet encoding reduces storage costs and accelerates analytical queries.
AWS Glue Catalog integration out of the box.
Every flush is an immutable Iceberg snapshot — time travel included.
Automatic Partitioning & Optimization - Organizes data into analytics-friendly layouts with configurable partitioning and optimized file sizes for fast query performance.
ACID Table Transactions — Apache Iceberg guarantees consistent reads and atomic commits, even with concurrent writers and readers.
Engine-Agnostic Data Lake — Query the same dataset from Athena, Spark, Trino, DuckDB, Flink, or Redshift Spectrum without data duplication.
Snapshot-Based Rollback — Instantly query or restore previous table versions using Iceberg snapshots.
Incremental Table Scans — Downstream pipelines can process only newly committed snapshots instead of rescanning entire datasets.
Automatic Metadata Management — Maintains Iceberg manifests and table metadata transparently as data grows.
Cloud-Native Object Storage — Writes directly to Amazon S3 with no HDFS or distributed filesystem required.
Future-Proof Open Formats — Your data remains portable and accessible without proprietary storage engines or vendor lock-in.
Compression Built In — Uses efficient columnar compression to reduce storage footprint while maintaining high query performance.
Optimized for AI & Lakehouse Workloads — Creates datasets that are immediately consumable by analytics platforms, feature engineering pipelines, and AI applications.
Source schema changes are absorbed automatically. Add a column or change a type upstream and pg-cdc evolves the target schema without manual intervention or a broken pipeline.
Zero-Downtime Schema Updates - Continue streaming while schemas evolve—no pipeline restarts or data reloads required.
Automatic Metadata Synchronization - Updates the Iceberg table definition and AWS Glue Catalog as schemas evolve.
Historical Schema Preservation - Previous Iceberg snapshots remain queryable using the schema that existed at the time they were written.
Compatible with Continuous Delivery - Supports frequent application deployments without requiring coordinated database and analytics changes.
Resilient to Incremental Database Changes - Handles common schema evolution patterns without interrupting downstream consumers.
Producer and Consumer Decoupling - Applications can evolve independently from analytics and AI workloads, reducing deployment dependencies.
Enterprise-Scale Schema Management - Designed to support long-lived operational databases where schemas evolve over months and years.
pg-cdc checkpoints its committed position continuously. After a restart or failure it resumes from the last durable offset — no data loss, no duplication.
Automatic Crash Recovery - Restarts processing automatically from the last committed checkpoint after unexpected failures.
Fast Service Restarts - Avoids full database rescans by resuming directly from the saved replication position.
Recovery Independent of Process Lifetime - Checkpoint state is durable, so container replacements and host reboots do not interrupt replication.
Replication Slot Consistency - Coordinates checkpoints with PostgreSQL logical replication slots to ensure a continuous, gap-free change stream.
Idempotent Commit Handling - Ensures partially processed batches are safely reconciled after recovery without corrupting downstream datasets.
Cloud-Native Resilience - Designed for rolling deployments, Kubernetes pod replacements, and infrastructure maintenance without requiring manual recovery.
Operationally Safe Upgrades - Upgrade pg-cdc binaries or container images while preserving replication progress and downstream consistency.
Start from a consistent point-in-time snapshot of your existing tables, then switch seamlessly to incremental streaming — so your lake has full history from day one.
Production-Safe Initial Load — Captures the initial dataset using PostgreSQL’s logical replication capabilities without requiring application downtime.
Single-Step Bootstrap — Creates a complete historical baseline and transitions automatically to live change capture.
Ready for Existing Databases — Onboard years of operational data without exporting, restoring, or rebuilding your database.
Consistent Cross-Table Capture — Maintains a transactionally consistent view across all selected tables at snapshot time.
Automatic Progress Tracking — Tracks snapshot completion and transitions to streaming without manual intervention.
Incremental from Minute One — New transactions begin flowing immediately after the snapshot completes, keeping your lake continuously current.
Scales to Large Datasets — Efficiently snapshots multi-terabyte databases while preparing for continuous change capture.
One-Time Bootstrap — Perform the initial historical load once, then remain continuously synchronized through WAL replication.
AI-Ready Historical Context — Gives analytics and AI workloads immediate access to both historical records and live operational changes in a single dataset.
Simplified Migration Path — Move from an operational PostgreSQL database to a governed lakehouse without custom ETL or backfill pipelines.
Lifecycle Policy Compatible — Works with S3 Lifecycle rules to automatically transition data to Standard-IA, Glacier Instant Retrieval, Glacier Flexible Retrieval, or Deep Archive.
Multi-Environment Ready — Easily isolate development, staging, and production datasets using separate buckets or prefixes.
No Infrastructure to Manage — No HDFS clusters, NAS appliances, or proprietary storage systems required.
Cloud-Native by Design — Integrates naturally with the broader AWS analytics and AI ecosystem while keeping your data under your control.
Cross-Region Replication Ready — Compatible with Amazon S3 Cross-Region Replication (CRR) for disaster recovery and global data distribution.
AI agents query governed, up-to-date data through the Model Context Protocol — with authentication via AWS IAM and Lake Formation only, never database credentials.
Read-Only AI Access — AI agents can retrieve business context without any ability to modify production data.
Always-Current Context — Agents consume continuously updated operational data instead of stale exports or manually refreshed datasets.
Governed by Existing AWS Security — Access policies are enforced through IAM and Lake Formation, eliminating a separate AI permission model.
Multi-Agent Ready — Serve the same governed data to ChatGPT, Claude, Gemini, internal copilots, and custom AI agents from a single source.
Structured Business Context — Exposes typed, queryable data rather than requiring AI models to parse unstructured documents.
Decouples AI from Production Systems — Production databases remain isolated from AI workloads, reducing operational and security risk.
Natural Language Meets SQL — AI agents can translate user requests into governed queries over trusted operational data.
Shared Source of Truth — Every AI application consumes the same governed datasets, ensuring consistent answers across models.
Enterprise-Scale Authorization — Security policies are enforced before data reaches the AI agent, not after.
Future-Proof Integration — Open MCP and standard data interfaces allow new AI frameworks and agent platforms to be adopted without changing your data pipeline.
Governance is not an add-on. AWS Lake Formation tags gate every read at the column level, and untagged data is invisible — zero-trust by construction.
Least-Privilege Access — Consumers only see the columns and tables explicitly authorized by policy.
Zero-Trust Data Architecture — Access is denied by default until governance policies explicitly grant permission.
Centralized Policy Management — Define permissions once in Lake Formation and enforce them consistently across all supported query engines.
Consistent Enforcement Across Services — The same governance rules apply whether data is accessed through Athena, Spark, Trino, Redshift Spectrum, or MCP.
No Security Logic in Applications — Authorization is enforced by the data platform rather than individual applications or AI agents.
Fine-Grained Data Protection — Secure sensitive business attributes without duplicating or maintaining separate datasets.
Audit-Ready by Design — Access decisions are traceable through centralized governance policies and audit records.
Policy Changes Without Data Movement — Update access permissions instantly without rewriting, copying, or repartitioning data.
Separation of Data and Permissions — Governance metadata is managed independently from the underlying datasets, simplifying operations.
Enterprise Security Alignment — Integrates with existing AWS identity, governance, and compliance controls instead of introducing a separate security model.