9.3k repos tracked
updated 2d ago
Most starred

Data

Pipelines, warehouses, orchestration and analytics engineering.

Trending New Most starred 580 repositories · topics: data-engineering, etl, data-pipeline, database, duckdb
  1. 401
    🚀 GizmoSQL — High-Performance Database Server
    C++ created 2y ago pushed 5d ago adbcapache-arrowapache-arrow-flight
    flat
    measured
    ★ 376
    22
  2. 402
    DataMate is an enterprise-level data processing platform designed for model fine-tuning and RAG retrieval.
    Python created 15mo ago pushed 5d ago data-evaluationdata-pipelinedata-synthesis
    flat
    measured
    ★ 374
    22
  3. 403
    YARD is an editor plugin for Godot to manage resources and query them at runtime, with a table view in the editor as the cherry on top.
    GDScript created 10mo ago pushed 3d ago databasegame-developmentgame-engine
    +4.0/wk
    measured
    ★ 371
    26
  4. 404
    An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.
    Python created 3y ago pushed 21mo ago apache-airflowapache-kafkaapache-spark
    +1.0/wk
    measured
    ★ 370
    9
  5. 405
    Modern SQL client
    TypeScript created 2y ago pushed 11mo ago duckdbjavascriptmotherduck
    flat
    measured
    ★ 365
    7
  6. 406
    The database IDE for VS Code, Cursor, and Windsurf. 80+ databases: Postgres, MySQL, SQL Server, MongoDB, Snowflake, and more. AI queries, ER diagrams, SQL notebooks.
    JavaScript created 2y ago pushed 6d ago cursordatabasedatabase-client
    +3.0/wk
    measured
    ★ 363
    29
  7. 407
    DuckDB extension to read and write Google Sheets using SQL
    C++ created 2y ago pushed 8mo ago duckdbduckdb-extensiongoogle-sheets
    flat
    measured
    ★ 359
    7
  8. 408
    Fast DuckDB-powered TypeScript library for tabular analysis, geospatial analysis, network graph analysis, vector similarity search, AI, Google Sheets, and data visualization.
    TypeScript created 4y ago pushed 5d ago aibuncsv
    +1.0/wk
    measured
    ★ 357
    23
  9. 409
    DuckDB extension for querying, modifying, and storing data through Apache Arrow Flight servers.
    C++ created 2y ago pushed 16d ago apache-arrowarrowarrow-flight
    +1.0/wk
    measured
    ★ 351
    22
  10. 410
    A Python micro-ORM
    Python created 11mo ago pushed 10mo ago databasemariadbmysql
    flat
    measured
    ★ 338
    7
  11. 411
    Serverless HTAP cloud data platform powered by Arrow × DuckDB × Iceberg
    created 3y ago pushed 3y ago arrowduckdbiceberg
    flat
    measured
    ★ 333
    7
  12. 412
    A data platform for investment research. Ingests financial data sources and computes point-in-time financial metrics for valuation and screening. Delivered as an MCP server.
    Python created 2mo ago pushed 5d ago alpha-researchbacktestingduckdb
    +5.0/wk
    measured
    ★ 329
    27
  13. 413
    DataKit is a browser-based data analysis platform that processes multi-gigabyte files locally. All processing happens in your browser - no data is sent to external servers.
    TypeScript created 18mo ago pushed 2mo ago data-analysisdatabaseduckdb
    flat
    measured
    ★ 327
    17
  14. 414
    Illuminate your data. Agent framework turning natural language into SQL, charts, dashboards and reports.
    Python created 6y ago pushed 2d ago ai-agentsconversational-aidata-exploration
    +3.0/wk
    measured
    ★ 326
    27
  15. 415
    AxData 是一个开源量化数据库框架,覆盖通达信、巨潮、腾讯财经、新浪财经、东方财富、财联社、开盘红等公开源接口。
    Python created 4mo ago pushed 52d ago a-sharechina-stockdata-platform
    +1.4/day
    measured
    ★ 325
    25
  16. 416
    A simple terminal UI for PostgreSQL, written in Go with Bubble Tea
    Go created 11mo ago pushed 2mo ago clidatabasego
    +1.0/day
    measured
    ★ 322
    29
  17. 417
    Formation PostgreSQL 18 | De débutant à avancé | 21 chapitres + 7 annexes | Architecture, SQL avancé, Performance, HA, Production | Français
    created 11mo ago pushed 4mo ago architectureclouddatabase
    flat
    measured
    ★ 321
    12
  18. 418
    Use LLMs to robustly extract web data
    TypeScript created 18mo ago pushed 5mo ago ai-agentsarticle-extractorcrawler
    flat
    measured
    ★ 321
    10
  19. 419
    六十七门真实网课的模块合辑全书 + 思维导图复习笔记 + 逐字稿,全部由 Video2Book 从课程音频重构:683 册模块全书、921 篇复习笔记、3,522 份逐字稿,从逐讲精读到考前速查,笔记支持 Markmap / XMind 脑图导入,可按课点播定制。
    created 56d ago pushed 5d ago 8086aiassembly
    +9.9/day
    measured
    ★ 313
    45
  20. 420
    Stream Postgres to Apache Iceberg on S3 via logical replication, queryable over the Postgres wire protocol.
    Go created 7mo ago pushed 16d ago duckdbicebergparquet
    flat
    measured
    ★ 310
    21
  21. 421
    Every database. One studio. A free, open-source database GUI for SQL, document, and key-value databases.
    TypeScript created 11mo ago pushed 3d ago clickhousecloudflare-d1database-client
    +3.3/day
    measured
    ★ 310
    37
  22. 422
    A SQL transformation engine that type-checks your whole pipeline and catches breaking changes before they run — branches, replay, column-level lineage, compile-time contracts, per-model cost. Adapters: Databricks, Snowflake, BigQuery, DuckDB. Single static Rust binary. Apache 2.0.
    Rust created 6mo ago pushed 3d ago bigquerycolumn-lineagedagster
    +1.0/wk
    measured
    ★ 309
    23
  23. 423
    AI-native SQL client for humans and agents. Query, explore, visualize, and open agent database work as editable SQL workspaces.
    TypeScript created 10mo ago pushed 4d ago aiclickhouseclickhouse-ui
    +3.0/wk
    measured
    ★ 309
    25
  24. 424
    SDK for robotics teams to verify the quality of their data used for AI model training.
    Python created 2mo ago pushed 3d ago airflowdata-engineeringdata-pipeline
    +2.0/wk
    measured
    ★ 306
    24
  25. 425
    The data transformation tool the open-source community deserves.
    Python created 14d ago pushed 2d ago data-engineeringduckdb
    +5.3/day
    measured
    ★ 305
    37
  26. 426
    Agentic SQL database manager for 40+ databases — PostgreSQL, MySQL, SQL Server, SQLite, DuckDB, ClickHouse, Oracle and more. Lightweight, privacy-first, AI-powered, cross-platform. Built with Tauri (Rust).
    Rust created 9mo ago pushed 3d ago bigqueryclickhousecockroachdb
    +1.0/wk
    measured
    ★ 304
    23
  27. 427
    Embedded storage built for AI model management over gRPC. Smart machines don't need to read JSON, they only need protocol buffers. The world's first MODMS (Machine-Oriented Database Management System), built to support SQLite and eventually DuckDB as backends.
    Rust created 2y ago pushed 15mo ago duckdbgrpcproto
    flat
    measured
    ★ 304
    7
  28. 428
    A command-line tool for querying databases
    Rust created 9mo ago pushed 15d ago adbcapache-arrowcli
    flat
    measured
    ★ 300
    21
  29. 429
    Open-source research workspace for connecting sources, evidence, notes and ideas. Built for academic research with Zotero integration. Available on MacOS, Windows and Linux.
    TypeScript created 4mo ago pushed 2d ago academic-researchacademic-toolsarchive
    +1.6/day
    measured
    ★ 299
    45
  30. 430
    DuckDB extension to read and write to SQLite databases
    C created 5y ago pushed 3d ago duckdbsqlsqlite
    +2.0/wk
    measured
    ★ 297
    31
  31. 431
    Practice Databricks coding skills with hands-on exercises. Import into Databricks Free Edition, write code, run assertions, check pass/fail. Covers Delta Lake, Spark SQL, PySpark, Auto Loader, medallion architecture, window functions, and more.
    Python created 7mo ago pushed 19d ago auto-loadercoding-practicedata-engineering
    +2.0/wk
    measured
    ★ 296
    23
  32. 432
    Get started with dbt in less than 1 minute from `git clone` to `dbt docs serve` for free!
    Shell created 4y ago pushed 29d ago datadbtduckdb
    flat
    measured
    ★ 295
    20
  33. 433
    A live database structure viewer for Laravel that renders your schema as a scrollable, zoomable ER diagram. Structure only, never data.
    PHP created 3mo ago pushed 3d ago databasedatabase-schemadeveloper-tools
    flat
    measured
    ★ 292
    22
  34. 434
    Use SQL to build ELT pipelines on a data lakehouse.
    JavaScript created 5y ago pushed 4y ago apache-icebergapache-sparkdata-engineering
    flat
    measured
    ★ 290
    7
  35. 435
    A Redis-compatable key-value store. Up to 10x faster. Native vector support.
    Rust created 7mo ago pushed 32d ago databasedragonflydbembeddings
    flat
    measured
    ★ 289
    20
  36. 436
    A lightweight schema-on-read analytics in a single binary
    Go created 8mo ago pushed 2mo ago analyticsdatabasedevops
    flat
    measured
    ★ 286
    15
  37. 437
    Making DAG construction easier
    Python created 6y ago pushed 17d ago airflowdata-etldata-pipeline
    flat
    measured
    ★ 286
    21
  38. 438
    企业数据分析、统计分析、数据可视化与管理报告工作台|Analysis Lab、可追溯证据、FastAPI、Next.js、Windows。
    Python created 3mo ago pushed 3mo ago ab-testingbusiness-analyticsbusiness-intelligence
    flat
    measured
    ★ 286
    14
  39. 439
    DuckDB extension for serving SQL queries over HTTP and returning JSON results.
    C++ created 2y ago pushed 16d ago apiapi-servercommunity-extension
    +1.0/wk
    measured
    ★ 285
    22
  40. 440
    Örnek SQL sorguları, T-SQL pratikleri ve Northwind veritabanı çalışmaları.
    TSQL created 8mo ago pushed 7mo ago codingdatabasenorthwind
    +1.6/day
    measured
    ★ 284
    24
  41. 441
    Unified local observability for AI coding assistants
    Go created 10mo ago pushed 53d ago aiclaude-codecodex-cli
    +4.0/wk
    measured
    ★ 283
    37
  42. 442
    A fast, simple, and embedded key-value store for Rust.
    Rust created 10mo ago pushed 2mo ago asyncbtreemapdatabase
    flat
    measured
    ★ 279
    16
  43. 443
    Open Lakehouse Compute Fabric: an open-source lakehouse platform with Microsoft Fabric Compute Extension.
    Python created 27d ago pushed 19d ago apache-icebergapache-sparkdata-engineering
    −1.0/wk
    measured
    ★ 274
    21
  44. 444
    Pluggable data replication with checkpointing, batching, and integrity events
    Go created 4mo ago pushed 3d ago change-data-capturedatadata-collection
    +4.0/day
    measured
    ★ 271
    43
  45. 445
    A minimal DuckDB Wasm build for browsers and serverless environments like Cloudflare Workers
    TypeScript created 10mo ago pushed 13d ago cloudflare-workersduckdbemscripten
    +1.0/wk
    measured
    ★ 270
    22
  46. 446
    A portable Datamart and Business Intelligence suite built with Docker, Dagster, dbt, DuckDB and Superset
    Python created 3y ago pushed 13d ago business-intelligencedagsterdata-engineering
    −1.0/wk
    measured
    ★ 266
    21
  47. 447
    ANTLR-based PostgreSQL query parser for Go. Extracts tables, columns, joins, CTEs, parameters, DDL actions, and full column-usage metadata from SQL into a structured IR.
    Go created 9mo ago pushed 2mo ago antlrdatabasedeveloper-tools
    +1.0/wk
    measured
    ★ 265
    18
  48. 448
    The agent-first database
    Python created 7mo ago pushed 4mo ago agentsaidatabase
    flat
    measured
    ★ 264
    12
  49. 449
    Established strong JEV based natural language to SQL performance and well-covered semantic operators for PostgreSQL with optimized cost and latency. Reusable evidence, with native execution, parallel plans and explainable embeddings in development preview.
    Python created 45d ago pushed 13d ago databaseembeddingsjev
    +1.0/wk
    measured
    ★ 260
    22
  50. 450
    :whale: Tool to automate data quality checks on data pipelines
    Vue created 9y ago pushed 4y ago big-datadata-pipelinedata-quality
    flat
    measured
    ★ 257
    7
← previous page 9 of 12 next →