9.3k repos tracked
updated 2d ago
Most starred

Data

Pipelines, warehouses, orchestration and analytics engineering.

Trending New Most starred 580 repositories · topics: data-engineering, etl, data-pipeline, database, duckdb
  1. 501
    End-to-end streaming data platform on AWS: Kafka → Iceberg lake → Kimball model → Athena, orchestrated with Airflow 3. Terraform, Glue PySpark, dbt.
    Python created 2mo ago pushed 2mo ago apache-airflowapache-icebergathena
    +1.0/wk
    measured
    ★ 155
    16
  2. 502
    Turn any folder into a Notion-style database — 7 views, 17 column types, formulas, relations, rollups, and subtasks. All stored in plain Markdown frontmatter.
    TypeScript created 8mo ago pushed 2mo ago calendarchartsdatabase
    +2.0/wk
    measured
    ★ 155
    22
  3. 503
    Modern documentation site generator for dbt Core — lineage explorer, health scoring, full-text search. Live demo: https://demo.docglow.com
    Python created 8mo ago pushed 8d ago analytics-engineeringdata-catalogdata-engineering
    +2.0/wk
    measured
    ★ 153
    36
  4. 504
    📖 Data Engineering from Raw Corpora
    TypeScript created 6mo ago pushed 4mo ago aiartificial-intelligencebenchmark
    flat
    measured
    ★ 150
    9
  5. 505
    Discover, mask, and verify sensitive data in SQL databases — an auditable scan → mask → validate workflow for safe database copies.
    Python created 5mo ago pushed 4d ago anonymizationdata-engineeringdata-governance
    +1.0/wk
    measured
    ★ 150
    22
  6. 506
    relational database stored in discord guilds
    Go created 7mo ago pushed 7mo ago databasediscordrelational-database
    flat
    measured
    ★ 149
    6
  7. 507
    MAC address manufacturer lookup database combining IEEE, Nmap, Wireshark and HDM Mac-Tracker, with downloadable formats and a browser lookup. https://ringmast4r.github.io/OUI-Master-Database/
    JavaScript created 12mo ago pushed 3d ago cross-platformcybersecuritydatabase
    +4.0/wk
    measured
    ★ 149
    25
  8. 508
    Native lakehouse experience in Postgres powered by DuckDB & DuckLake
    C++ created 9mo ago pushed 13d ago analyticsduckdbducklake
    +2.0/wk
    measured
    ★ 149
    30
  9. 509
    A modern, open-source database client for desktop and web.
    Rust created 10mo ago pushed 32d ago databasedatabase-clientdbeaver-alternative
    +3.0/wk
    measured
    ★ 147
    22
  10. 510
    Extract a slice of your production database to reproduce bugs locally. Point dbslice at a record, it follows foreign keys and gives you a complete, importable subset
    Python created 8mo ago pushed 4mo ago databasepostgresqlpython
    flat
    measured
    ★ 144
    10
  11. 511
    Production-grade SQL gateway in front of DuckDB Quack + DuckLake + Iceberg. Multi-tenant pools, pluggable auth (DB/JWT/OIDC), table-level ACLs, role-aware routing, and a live admin console
    Scala created 5mo ago pushed 3d ago apache-arrowarrow-flight-sqlduckdb
    +2.0/wk
    measured
    ★ 144
    23
  12. 512
    从数据到智能:企业级数据平台的构建、演进与 Agentic BI 实践。一个数据开发的八年数据工程手记:从 0 到 1 构建企业级医药数据平台,再到 Data + AI 转型下的 Agentic BI。
    HTML created 4mo ago pushed 3mo ago agentic-bidata-agentdata-ai
    +2.0/wk
    measured
    ★ 140
    27
  13. 513
    Interactive Emacs database client for MySQL, PostgreSQL, SQLite, MongoDB, Redis, and JDBC.
    Emacs Lisp created 8mo ago pushed 3d ago databaseemacsemacs-lisp
    +1.8/wk
    measured
    ★ 139
    23
  14. 514
    Datos públicos de Chile curados, normalizados y validados: geografía, demografía, economía, salud, educación y distritos electorales. Listos para consumir en una línea de código con Polars, DuckDB, SQLite y Excel.
    Python created 5mo ago pushed 5d ago chiledata-engineeringdatasets
    +1.0/wk
    measured
    ★ 137
    22
  15. 515
    Observability platform for Digital Employee, providing real-time tracing, session insights, and cost analysis for multi-agent workflows
    JavaScript created 7mo ago pushed 5mo ago ai-agentscost-trackingdashboard
    flat
    measured
    ★ 136
    7
  16. 516
    DuckDB extension for Microsoft SQL Server (TDS + TLS), with catalog integration and pushdown.
    C++ created 9mo ago pushed 5d ago duckdbduckdb-extensionhugr
    +1.0/wk
    measured
    ★ 136
    22
  17. 517
    The Boring query platform - Connect and query anything
    TypeScript created 12mo ago pushed 3mo ago agentic-aiaiclickhouse
    +1.0/wk
    measured
    ★ 135
    14
  18. 518
    Zero-config entity resolution feeding a durable identity layer: messy records from any source become stable golden entities, a Customer 360 with provenance, merge/split and audit. Fellegi-Sunter beats hand-tuned Splink. Arrow-native/Rust, 250M rows in 11.2 min. Python + edge TypeScript (WASM), SQL-native in Postgres & DuckDB, 97 MCP tools + REST.
    Python created 7mo ago pushed 3d ago data-cleaningdata-engineeringdata-matching
    +1.0/wk
    measured
    ★ 135
    26
  19. 519
    Skills related to Data Engineering Work for Claude Code
    Shell created 9mo ago pushed 12d ago claudeclaude-codeclaude-skills
    +3.0/wk
    measured
    ★ 133
    38
  20. 520
    A modern, lightweight database management tool. Create, manage, and run multiple database instances locally with an intuitive interface.
    TypeScript created 12mo ago pushed 3mo ago backenddatabasedatabase-management
    —
    stale
    ★ 130
    14
  21. 521
    Pivot is a high-performance real time analytics database on open data formats.
    Rust created 7mo ago pushed 2d ago aianalyticsbig-data
    +3.8/wk
    lifetime avg
    ★ 128
    25
  22. 522
    AI-native cross-platform database workbench for developers and data teams.
    TypeScript created 2mo ago pushed 3d ago aiai-assistantclickhouse
    −2.0/wk
    measured
    ★ 127
    21
  23. 523
    Type-safe SQL query builder. Zero dependencies, AST-first, hookable, tree-shakeable. Pure TypeScript.
    TypeScript created 7mo ago pushed 2mo ago astdatabaseesm
    flat
    measured
    ★ 127
    15
  24. 524
    Servidor MCP e CLI para localizar empresas brasileiras por CNAE e região, utilizando dados abertos de CNPJ da Receita Federal.
    TypeScript created 6d ago pushed 5d ago cnaecnpjduckdb
    +36/day
    lifetime avg
    ★ 127
    51
  25. 525
    Token Efficiency For AI Agents
    Python created 7mo ago pushed 6d ago ai-agentsautonomous-agentscli
    +1.0/wk
    measured
    ★ 125
    35
  26. 526
    A cognition aware database engine for AI agent memory. Purpose built in Rust with WAL, HNSW, knowledge graphs, and speculative context pre assembly. Not a wrapper, a ground up storage engine that thinks.
    Rust created 7mo ago pushed 3d ago agent-memoryaiai-agents
    +2.0/wk
    measured
    ★ 124
    27
  27. 527
    A simple Python CLI to download Garmin Connect data as local files and load them into a SQLite database for analysis
    Python created 12mo ago pushed 49d ago ai-agentsclietl
    —
    stale
    ★ 124
    17
  28. 528
    Pure-Go SQLite, no cgo. Official mirror of gitlab.com/cznic/sqlite — issues and PRs welcome here.
    Go created 11mo ago pushed 5d ago cgo-freedatabasego
    +5.0/wk
    measured
    ★ 123
    34
  29. 529
    Linter for dangerous Postgres migration patterns in Diesel and SQLx. Prevents downtime caused by unsafe schema changes.
    Rust created 11mo ago pushed 4d ago clicode-reviewdatabase
    +1.0/wk
    measured
    ★ 122
    22
  30. 530
    The open-source pulse on software across Greece
    Python created 10mo ago pushed 4d ago athensawesome-listcareer-opportunities
    +2.0/wk
    measured
    ★ 121
    23
  31. 531
    Apache Phoenix Adapters
    Java created 12mo ago pushed 2mo ago awsdatabasedynamodb
    flat
    measured
    ★ 121
    14
  32. 532
    🐬 Beginner-to-advanced MySQL course — filtering, joins, subqueries, window functions, CTEs & more. Hands-on guides with copy-paste SQL on one sample database. 🎥 Full video walkthrough on YouTube.
    created 5mo ago pushed 13d ago cruddatabasedbms
    +4.0/wk
    measured
    ★ 121
    26
  33. 533
    A robust, configuration-driven ETL and data import framework for Laravel. Handles CSV/Excel streaming, queues, validation, and relationships.
    PHP created 12mo ago pushed 4mo ago csvdata-ingestionetl
    —
    stale
    ★ 120
    10
  34. 534
    Complete structured archive of every CIA World Factbook edition from 1990-2025. 281 countries, 36 years, 1M+ parsed data fields in SQL Server. Python ETL scripts included. Website created to allow users to explore the data without downloading. www.worldfactbookarchive.org
    TSQL created 8mo ago pushed 11d ago cia-factbookcia-world-factbookcountry-data
    +3.0/wk
    measured
    ★ 119
    29
  35. 535
    A Scala 3 web framework. Direct-style. The case class is the source of truth. Deploy with one command.
    Scala created 3mo ago pushed 2d ago activerecorddatabasedirect-style
    +1.2/day
    lifetime avg
    ★ 119
    27
  36. 536
    Lightweight GPL-3.0 desktop database client (PostgreSQL / MySQL / SQLite / Redis). Built with Tauri + Rust, under 15MB.
    Rust created 12mo ago pushed 12d ago aicross-platformdatabase
    —
    stale
    ★ 117
    20
  37. 537
    Fast, distributed per-aggregate event log with strict ordering and optimistic concurrency control.
    Rust created 11mo ago pushed 40d ago databaseevent-drivenevent-sourcing
    +1.0/wk
    measured
    ★ 116
    32
  38. 538
    Microsoft Fabric Unified Data Foundation with Databricks & Purview 2026
    HTML created 4mo ago pushed 3d ago ai-azd-templatesazd-templateazd-templates
    flat
    measured
    ★ 114
    21
  39. 539
    MCP server for 200M+ patents, scientific literature, chemistry and pharma records. Search prior art and R&D intelligence powered by PatSnap's proprietary databases via Claude, Cursor and MCP-compatible clients.
    created 6mo ago pushed 2mo ago bioinformaticscheminformaticsdatabase
    —
    stale
    ★ 112
    15
  40. 540
    Backend Engineering, Low-Level Design, System Design, Networking, Security, SQL, and Distributed Systems explained through visual diagrams.
    JavaScript created 10mo ago pushed 13d ago backend-engineeringdatabasedesign-patterns
    flat
    measured
    ★ 112
    20
  41. 541
    Open-source text-to-SQL engine you tune and measure on your own database
    Python created 3mo ago pushed 4d ago data-agentsdatabasedatabricks
    +1.0/wk
    measured
    ★ 112
    22
  42. 542
    Free Open-source AI SQL client that helps developers and data teams explore databases using natural language.
    TypeScript created 9mo ago pushed 3mo ago ai-toolsdatabaseopen-source
    +4.0/wk
    measured
    ★ 111
    30
  43. 543
    OpenFGA-to-PostgreSQL authorization compiler. Generates specialized SQL functions from .fga schemas that query your existing tables. Zero sync, always consistent.
    Go created 10mo ago pushed 2d ago authorizationdatabasefga
    +4.0/wk
    measured
    ★ 111
    30
  44. 544
    Dex lets your agent build your analytics.
    Python created 4mo ago pushed 2d ago agentagent-skillsanalytics
    +1.1/day
    measured
    ★ 111
    27
  45. 545
    Scan-to-Practice: a field-tested AI skill and methodology for turning scanned learning materials into structured practice products.
    created 3mo ago pushed 2mo ago ai-skillcomputer-visiondata-pipeline
    flat
    measured
    ★ 110
    14
  46. 546
    Application-level quota management for Laravel with calendar periods, Eloquent integration, middleware, and multiple storage backends.
    PHP created 3mo ago pushed 3mo ago calendar-periodsdatabaseeloquent
    flat
    measured
    ★ 110
    12
  47. 547
    A graph database provides Cypher query, fluent builder and management UI implemented in Golang.
    Go created 9mo ago pushed 4mo ago databasegolanggraph
    flat
    measured
    ★ 110
    10
  48. 548
    Terminal database client with AI chat and an MCP server
    Go created 2mo ago pushed 14d ago auroraclicockroachdb
    +1.0/wk
    measured
    ★ 108
    21
  49. 549
    View SQLite databases
    Rust created 10mo ago pushed 15d ago databasegnomelinux
    +2.0/wk
    measured
    ★ 107
    37
  50. 550
    Data streaming runtime focused on performance, consistency, and extensibility. Write plugins in Rust or WASM and process data with data guarantees.
    Rust created 5mo ago pushed 3d ago arrowdata-sciencedatafusion
    +3.0/wk
    measured
    ★ 107
    33
← previous page 11 of 12 next →