Soda Core Data Quality

Soda Core data testing

Soda Core runs YAML checks for missing, duplicate, invalid and fresh values, plus custom SQL, on SQL tables and DataFrames. Each dataset gets one contract file with its schema and checks.

Compare: data contracts tools · data testing tools

• SQL Table Tests: Contract checks cover missing count/percent (completeness), duplicate count (uniqueness), invalid values and regex (validity), numeric ranges (accuracy), freshness assertions, and arbitrary custom SQL checks (docs).

• DataFrame Tests: Soda Core supports in-memory sources; contracts can be verified against Pandas and Spark DataFrames in pipelines (docs).

• Data Contracts: Data contracts are the core primitive in Soda v4 — YAML-defined agreements covering schema, quality checks, and freshness, verified from CLI or CI pipelines (docs).

• Inspect Failed Rows: Soda Core CLI outputs failed row samples for failing checks during local runs (docs).

• Unstructured Data Tests: No LLM-based semantic validation of free-text warehouse columns in Soda Core (docs).

• Table Comparison Tests: The soda-reconciliation extension compares aggregates between two tables; it may require a commercial Soda Library licence rather than the pure OSS Soda Core package — needs verification (docs).

• Data Diff: Record reconciliation checks are documented under the soda-reconciliation extension; unclear whether they are available in the OSS Soda Core package without a commercial licence (docs).

Soda Core data observability

Soda Core profiles columns and checks freshness, volume and custom SQL metrics. It has no anomaly detection or UI: that's in Soda Cloud.

• Data Profiling: Soda Core profiles column-level statistics (null %, distinct count) during scans to support check authoring (docs).

• Freshness Monitors: SodaCL freshness assertions check that a timestamp column is within an expected window — run as one-off checks via CLI, not as continuous managed monitors (docs).

• Volume Monitors: SodaCL row count checks assert expected row counts — run as one-off checks via CLI, not as continuous managed monitors (docs).

• Custom SQL Monitors: SodaCL supports arbitrary custom SQL check expressions run via CLI — as one-off pass/fail checks, not as time-series metrics (docs).

• Partition-Based Monitors: Dataset filters allow scoping checks to a recent time slice (e.g. WHERE ts > NOW() - INTERVAL '1d') to reduce full-table scan cost — a documented Soda Core/Library CLI feature (docs).

• Monitors-as-Code: Contracts are YAML-defined and version-controlled in Git; Soda Core CLI verifies them in CI/CD pipelines for GitOps-style data quality (docs).

• Anomaly Detection: Anomaly detection is a Soda Cloud managed observability feature; not available in Soda Core OSS (docs).

• Schema Monitors: Schema change detection over time requires Soda Cloud managed monitoring; not available in Soda Core OSS (docs).

• Field Monitors: Continuous column-level metric time series with anomaly detection requires Soda Cloud; not available in Soda Core OSS (docs).

• Distribution Monitors: Distribution shift monitoring over time requires Soda Cloud; not available in Soda Core OSS (docs).

• Segment Monitors: The group-by (segment) feature is explicitly not supported in Soda Core OSS — requires Soda Library or Soda Cloud (docs).

• Window-Based Monitors: Configurable tumbling/sliding time windows for intra-day metric evaluation are not documented in Soda v4 (docs).

• UI-Based Monitors: Soda Core is a CLI/Python library with no graphical interface; all configuration is done in YAML or Python (docs).

• Pipeline Monitoring: No pipeline job monitoring in Soda Core OSS (docs).

Soda Core workflows

Soda Core runs from the CLI or a Python library. In CI pipelines such as GitHub Actions it exits with an error when checks fail, so bad data blocks the build.

• CI/CD Integrations: Soda Core CLI runs checks in GitHub Actions and other CI pipelines and exits non-zero on failure, blocking deployments on contract violations (docs).

• Python SDK: Soda Core is pip-installable and provides a Python API for programmatic contract verification and in-memory DataFrame scanning (docs).

• Scheduling: No managed scheduler in Soda Core; scans are triggered manually via CLI or from an external orchestrator (Airflow, CI/CD) (docs).

• Webhooks: No webhook system in Soda Core OSS; webhooks are a Soda Cloud feature (docs).

• REST / GraphQL API: No REST API in Soda Core OSS; the public API is a Soda Cloud feature (docs).

• MCP Server: Soda MCP server connects to Soda Cloud; not available as a standalone Soda Core OSS feature (docs).

• dbt Integration: Ingesting dbt test results via 'soda ingest' requires a Soda Cloud account to store and visualize results; not available in standalone Soda Core OSS (docs).