Data

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

Python Stream Processing

The best place to learn data engineering. Built and maintained by the data engineering community.

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

👾 nao is the #1 open source agentic BI. (1) Create context with nao-core cli, (2) deploy nao chat interface for everyone

Feathr – A scalable, unified data and AI engineering platform for enterprise

获取微信4.0版本以上数据库密钥和图片密钥的工具 | A tool for obtaining database keys and image keys for WeChat versions 4.0 and above

More than 2000+ Data engineer interview questions.

Fast and efficient unstructured data extraction. Written in Rust with bindings for many languages.

Data processing & ETL framework for Ruby

A curated list of awesome dbt resources

Lightweight and extensible compatibility layer between dataframe libraries!

A comprehensive list of 180+ YouTube Channels for Data Science, Data Engineering, Machine Learning, Deep learning, Computer Science, programming, software engineering, etc.

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.

The most comprehensive SQL guide from a real-world expert! Learn everything from basics to advanced queries, optimizations, and real-world SQL

Database Reporting Tool and Tasks (.Net)

ktx is an executable context layer for data and analytics agents 🐙 Allow Claude Code, Codex, or other AI agents to query analytical databases accurately and with full context of your company

看海量化回测平台 khQuant 开源版 V2.2：免费开源的 A 股量化回测桌面软件。不用开户、不用装 QMT，BaoStock / Tushare 下载数据到本地 DuckDB，Python 写策略，一键出回测报告。

A Python stream processing engine modeled after Yahoo! Pipes

:bar_chart: :clipboard: Dashboards using YAML or JSON files

Python Streaming DataFrames for Kafka

Drone Log analyzer: A high-performance universal dashboard application for organizing and analyzing DJI/Litchi flight logs privately in one place. Supports plugin for custom flight log formats. Built with Tauri v2, DuckDB, and React.

OLake - Fastest Databases, Kafka & S3 Replication to Apache Iceberg with Table optimization (Called OLake Fusion). ⚡ Efficient, quick and scalable data ingestion for real-time analytics. Supported sources : Postgres, MongoDB, MySQL, Oracle, MSSql, DB2, Kafka, S3.

Dozer is a real-time data movement tool that leverages CDC from various sources and moves data into various sinks.

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

The open document intelligence platform for builders and hackers - DMS for the agentic world

Hop Orchestration Platform

Concurrent Python made simple

Clean APIs for data cleaning. Python implementation of R package Janitor

Postgres intelligence for ai agents

A curated collection of 300+ engineering blog articles from top tech companies. Learn how the best engineering teams solve real-world problems at scale.

Your entire engineering context, deeply understood

Actively maintained successor to Alibaba DataX — a fast, versatile, open-source ETL tool for 30+ RDBMS and NoSQL data sources.

Example end to end data engineering project.

First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.

Support agile Ontology DataOps Based on Flink, DataX and Flink-CDC with Web-UI

Postgres, MySQL, or MariaDB to Elasticsearch/OpenSearch sync

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

visual data prep powered by python

An extensible framework for linking databases and interactive views.

Take your agent-built product live: hosting, database, domain, email, payments — on your own accounts. Open-source Agent Skill + zero-dependency Node CLI: detect → plan → approve → apply → verify. No GoLive account, backend or telemetry.

Quilt is a Scientific Data Management Platform on AWS that helps teams and AI find, trust, and reuse data through deeply versioned, context-rich data packages.

dbt adapter for DuckDB

This repository is a getting started guide to Singer.

TypeScript ETL toolkit for indexing Ethereum, Solana, and Substrate data, sourced from SQD Network.

a go daemon that syncs MongoDB to Elasticsearch in realtime. you know, for search.

One browser tab for PostgreSQL, MySQL, Oracle, SQL Server, MongoDB, Redis, SQLite, Couchbase, ClickHouse, Druid, DuckDB, Turso and more. An open-source SQL IDE with SSO, audit trail and AI-assisted queries MIT licensed, with nothing held back behind an enterprise wall.

High performance Rust stream processing engine seamlessly integrates AI capabilities, providing powerful real-time data processing and intelligent analysis.