Data

Scratch is a swiss army knife for big data.

Personal Data Engineering Projects

FlyFish is a data visualization coding platform. We can create a data model quickly in a simple way, and quickly generate a set of data visualization solutions by dragging.

SawitDB: The First Agricultural-Based Database with Anti-Corruption Protocols. Built for Data Sovereignty to oppose expensive, unreliable state infrastructure.

🔥 🔥 🔥Open Source & AI driven Data Onboarding Platform:Free flatfile.com alternative

An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.

Smarter data pipelines for audio.

Neum AI is a best-in-class framework to manage the creation and synchronization of vector embeddings at large scale.

Accumulated knowledge and experience in the field of Data Engineering

Polyglot workflows without leaving the comfort of your technology stack.

Data API Framework for AI Agents and Data Apps

Utility that converts an .etl file containing a Windows network packet capture into .pcapng format.

The open source, end-to-end computer vision platform. Label, build, train, tune, deploy and automate in a unified platform that runs on any cloud and on-premises.

Transform, query, and download geospatial data on the web.

VectorFlow is a high volume vector embedding pipeline that ingests raw data, transforms it into vectors and writes it to a vector DB of your choice.

📄 ⚙️ ETL processes for medical and scientific papers

Realtime sync data from MySQL/PostgreSQL/MongoDB to Meilisearch

SQL Notebook — Casual data exploration in SQL

Synmetrix – production-ready open source semantic layer on Cube

A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)

A list about Apache Kafka

SmartCode = IDataSource -> IBuildTask -> IOutput => Build Everything!!!

基于web版kettle开发的一套分布式综合调度,管理,ETL开发的用户专业版B/S架构工具

This project provides a comprehensive data pipeline solution to extract, transform, and load (ETL) Reddit data into a Redshift data warehouse. The pipeline leverages a combination of tools and services including Apache Airflow, Celery, PostgreSQL, Amazon S3, AWS Glue, Amazon Athena, and Amazon Redshift.

Extract, Transform, Load: Any SQL Database in 4 lines of Code.

A serverless cluster computing system for the Go programming language

A hub for various industry-specific schemas to be used with VLMs.

Universal MCP-Server for your Databases optimized for LLMs and AI-Agents.

A playground for running duckdb as a stateless query engine over a data lake

Code review for data in dbt

Power of appbase.io via CLI, with nifty imports from your favorite data sources

Exchangis is a lightweight,highly extensible data exchange platform that supports data transmission between structured and unstructured heterogeneous data sources

ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ

Airflow DAGs for exporting, loading, and parsing the Ethereum blockchain data. How to get any Ethereum smart contract into BigQuery https://towardsdatascience.com/how-to-get-any-ethereum-smart-contract-into-bigquery-in-8-mins-bab5db1fdeee

🪄 Inline SQL in any Python program

An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration

Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!

Modern SQL client

DuckDB extension to read and write Google Sheets using SQL

A Python micro-ORM

Serverless HTAP cloud data platform powered by Arrow × DuckDB × Iceberg

Embedded storage built for AI model management over gRPC. Smart machines don't need to read JSON, they only need protocol buffers. The world's first MODMS (Machine-Oriented Database Management System), built to support SQLite and eventually DuckDB as backends.

Use SQL to build ELT pipelines on a data lakehouse.

Observability platform for Digital Employee, providing real-time tracing, session insights, and cost analysis for multi-agent workflows

On-device property graph database. Schema-as-code. One CLI → One Folder. No Server. Think: DuckDB for graphs.

:whale: Tool to automate data quality checks on data pipelines

High performance embedded vector database

NL2SQL - Ask questions in plain English, get SQL queries and results. Powered by LangGraph.

A curated list of awesome public DBT projects

An extensible, multi-database GUI client for macOS — add any database by implementing a single Swift protocol. Built for contributors.