9.3k repos tracked
updated 2d ago
Most starred

Data

Pipelines, warehouses, orchestration and analytics engineering.

Trending New Most starred 580 repositories ยท topics: data-engineering, etl, data-pipeline, database, duckdb
  1. 451
    A lightweight local first SQL analytics tool. Get your data ๐Ÿฆ† in a row
    TypeScript created 20mo ago pushed 7d ago analyticsdataduckdb
    flat
    measured
    โ˜… 256
    22
  2. 452
    A Postgres Compatible Database for Heavy, Multi-Tenant Workloads
    Zig created 9mo ago pushed 2d ago aws-s3databasedbms
    +9.4/day
    measured
    โ˜… 255
    41
  3. 453
    Bindings for H3 to DuckDB
    C created 4y ago pushed 23d ago duckdbduckdb-extensiongeospatial
    flat
    measured
    โ˜… 254
    20
  4. 454
    Independent ground-up Rust reimplementation of SQLite with page-level MVCC concurrent writers (serializable snapshot isolation). Runs today in compatibility mode over standard SQLite files; RaptorQ erasure-coded self-healing durability is in-progress research, not yet a live guarantee.
    Rust created 9mo ago pushed 3d ago databasemvccraptorq
    +1.0/wk
    measured
    โ˜… 252
    23
  5. 455
    High-assurance AI data visualization IDE & analytics workbench. Natural language to charts with local ground-truth table verification.
    Python created 40d ago pushed 5d ago ai-analyticsanalyticsbi
    +6.6/day
    lifetime avg
    โ˜… 249
    39
  6. 456
    High performance embedded vector database
    Rust created 11mo ago pushed 9mo ago aidatabasedocument-retrieval
    flat
    measured
    โ˜… 245
    7
  7. 457
    The memory & storage database engine for AI agents. Multi-model, edge-to-cloud, PostgreSQL-compatible. Supports graph, vector, array, timeseries.
    Rust created 7mo ago pushed 3d ago agent-memoryai-agentscolumnar
    +1.0/day
    measured
    โ˜… 243
    34
  8. 458
    NL2SQL - Ask questions in plain English, get SQL queries and results. Powered by LangGraph.
    Python created 10mo ago pushed 9mo ago a2aa2a-protocolai-agents
    flat
    measured
    โ˜… 238
    7
  9. 459
    end-to-end data engineering project to get insights from PyPi using python, duckdb, MotherDuck
    TypeScript created 2y ago pushed 53d ago bigquerydataengineeringduckdb
    flat
    measured
    โ˜… 238
    18
  10. 460
    Harness for building data engineering agents designed around human control, correctness, and safety.
    Java created 5y ago pushed 3d ago apidata-engineeringdata-pipeline
    +3.0/wk
    measured
    โ˜… 235
    40
  11. 461
    Next-generation SQL static analyzer written in Rust. 282+ rules. Zero false positives. Security, performance, reliability, cost, compliance, quality. 14 SQL dialects. Offline. CI/CD ready.
    Rust created 11mo ago pushed 4mo ago bigqueryci-cdclickhouse
    flat
    measured
    โ˜… 231
    11
  12. 462
    The duckdb R package
    R created 3y ago pushed 15d ago databaseduckdbolap
    +1.0/wk
    measured
    โ˜… 231
    22
  13. 463
    This project provides a comprehensive data pipeline solution to extract, transform, and load (ETL) Reddit data into a Redshift data warehouse. The pipeline leverages a combination of tools and services including Apache Airflow, Celery, PostgreSQL, Amazon S3, AWS Glue, Amazon Athena, and Amazon Redshift.
    Python created 3y ago pushed 3y ago apache-airflowawscelery
    +1.0/wk
    measured
    โ˜… 230
    8
  14. 464
    One database for user-user (follows), user-item (likes), and item-item (related items) interactions โ€” precomputed at write time, served as simple lookups
    Kotlin created 12mo ago pushed 17d ago databasegraphdbhigh-throughput
    +2.0/wk
    measured
    โ˜… 230
    38
  15. 465
    A collection of tools for extracting FHIR resources and analytics services on top of that data.
    Jupyter Notebook created 6y ago pushed 4d ago analyticsapache-beamdata-pipeline
    flat
    measured
    โ˜… 228
    22
  16. 466
    A playground for running duckdb as a stateless query engine over a data lake
    Python created 3y ago pushed 2y ago duckdblambdaserverless
    +1.0/wk
    measured
    โ˜… 225
    8
  17. 467
    A curated list of awesome public DBT projects
    created 3y ago pushed 2y ago data-pipelinedbttransformation
    flat
    measured
    โ˜… 222
    7
  18. 468
    A local-first relational database for the browser, React Native and your backend.
    Rust created 10mo ago pushed 2d ago databaseembedded-databaselocal-first
    +2.0/wk
    measured
    โ˜… 222
    24
  19. 469
    A collection of AI skills for working with Dagster
    Python created 9mo ago pushed 3d ago ai-toolsclaude-codedagster
    +2.0/wk
    measured
    โ˜… 218
    25
  20. 470
    Declarative text based tool for data analysts and engineers to extract, load, transform and orchestrate their data pipelines.
    Scala created 5y ago pushed 3d ago bigquerydata-engineeringdata-integration
    +1.0/wk
    measured
    โ˜… 217
    26
  21. 471
    In-memory key-value server in Rust that speaks RESP, so redis-cli and Redis clients connect. Locks, rate limits, job queues and a change feed are single commands; Raft-based failover. Clients for Rust, .NET, Go, Java, Python and Node.js.
    Rust created 23d ago pushed 3d ago cachedatabasedistributed-lock
    +2.6/day
    measured
    โ˜… 215
    32
  22. 472
    Pushdown compute from Snowflake to DuckDB running on your infrastructure
    Jupyter Notebook created 2y ago pushed 12mo ago databricksdbtduckdb
    โ€”
    stale
    โ˜… 208
    7
  23. 473
    ERD Builder Pro is a database design and documentation tool for developers. Build ERDs, flowcharts, notes, and drawings โ€” all in one workspace
    TypeScript created 7mo ago pushed 3d ago codingdata-engineeringdata-visualization
    flat
    measured
    โ˜… 207
    22
  24. 474
    Genomic sequence storage, indexing and alignment inside PostgreSQL: https://bioseqdb.com/
    VHDL created 41d ago pushed 2d ago alignmentbioinformaticsbiology
    โˆ’1.0/wk
    measured
    โ˜… 207
    22
  25. 475
    Your coding agent starts every session blank. Rekal is the memory your team is missing
    Go created 8mo ago pushed 59d ago agent-memoryai-codingclaude-code
    +1.0/wk
    measured
    โ˜… 206
    18
  26. 476
    AgentSeek is an application development lifecycle toolkit for AI ecosystem apps, built by OceanBase OSS Team.
    Python created 8mo ago pushed 30d ago agentic-aiagentsai
    +2.0/wk
    measured
    โ˜… 204
    22
  27. 477
    Open-source in-browser DuckDB SQL editor
    TypeScript created 2y ago pushed 4mo ago apache-arrowcomlinkduckdb
    โ€”
    stale
    โ˜… 204
    11
  28. 478
    ใ€ŠEasy Data x AIใ€‹ๆ˜ฏไธ€้—จ้ขๅ‘ๆ‰€ๆœ‰ AI ็ˆฑๅฅฝ่€…็š„ๆ•ฐๆฎไธŽ AI ๅŸบ็ก€ๅ…ฅ้—จ่ฏพ็จ‹ใ€‚้€š่ฟ‡ๅŒ่ฝจๅญฆไน ่ทฏๅพ„๏ผŒไปŽๆ•ฐๆฎ้ฉฑๅŠจ่ง†่ง’ๅญฆไน ๆž„ๅปบ AI Agentใ€‚
    Python created 8mo ago pushed 31d ago agentic-aiaiai-agents
    +4.0/wk
    measured
    โ˜… 200
    25
  29. 479
    A ่‚กใ€ๆœŸ่ดงใ€ๆœŸๆƒๆ•ฐๆฎๆœฌๅœฐๅŒ–็ฎก้“๏ผšTushare Pro ๆ‹‰ๅ– โ†’ Parquet ๅˆ†ๅŒบๅญ˜ๅ‚จ โ†’ DuckDB ๆœฌๅœฐๆŸฅ่ฏข๏ผŒๆ”ฏๆŒๅขž้‡ๅŒๆญฅไธŽๅฎšๆ—ถ่ฐƒๅบฆ
    Python created 6mo ago pushed 24d ago a-shareapschedulerchina-stock
    +1.0/wk
    measured
    โ˜… 197
    21
  30. 480
    sqlite3 viewer
    Rust created 6mo ago pushed 38d ago databasedebianrust-lang
    flat
    measured
    โ˜… 196
    19
  31. 481
    the DATABASE Editor. https://discord.gg/cEe8v2Y7PJ
    Rust created 8mo ago pushed 22d ago databasedatabase-managementeditor
    flat
    measured
    โ˜… 194
    20
  32. 482
    A lightweight embedded database engine that natively fuses vector search, graph, and document-oriented metadata within a single storage kernel โ€” ready to use the moment data is written.
    Rust created 7mo ago pushed 4d ago agentagent-memoryai-agents
    +1.0/day
    measured
    โ˜… 188
    42
  33. 483
    Focusing on building industry-leading ETL engines.
    Rust created 10mo ago pushed 11d ago data-collectordataprocessingetl
    flat
    measured
    โ˜… 187
    21
  34. 484
    A fast, minimal, cross-platform database client built with C++ and ImGui
    C++ created 8mo ago pushed 3d ago cppdatabaseimgui
    +3.0/wk
    measured
    โ˜… 186
    24
  35. 485
    Job board aggregator indexing 1,000,000+ active positions from 20,000+ companies across Greenhouse, Lever, Ashby, Workday, and other major ATS platforms. Multithreaded Python ETL pipeline, daily automated refreshes via GitHub Actions, filterable client-side search.
    Python created 10mo ago pushed 3d ago atsdata-engineeringdata-pipeline
    +5.0/wk
    measured
    โ˜… 185
    26
  36. 486
    The smallest backend framework and among the top three in the world.
    TypeScript created 12mo ago pushed 55d ago apibackendbun
    flat
    measured
    โ˜… 182
    17
  37. 487
    Dadbod Grip turns database tables into editable Vim buffers, with schema browsing, staged mutations, generated SQL, relationship navigation, and cross-database federation inside Neovim.
    Lua created 8mo ago pushed 4d ago aicross-databasedatabase
    +2.0/wk
    measured
    โ˜… 182
    31
  38. 488
    An extensible, multi-database GUI client for macOS โ€” add any database by implementing a single Swift protocol. Built for contributors.
    Swift created 7mo ago pushed 6mo ago contributions-welcomedatabaseextensible
    flat
    measured
    โ˜… 181
    7
  39. 489
    RaccoonX(DBCheck): Database Inspection Tool
    Python created 7mo ago pushed 3d ago aidatabaseinspection
    flat
    measured
    โ˜… 181
    22
  40. 490
    Cognitive memory database for AI agents โ€” consolidates duplicates, detects contradictions, fades stale memories via temporal decay. Rust, Apache-2.0, ships as library / MCP server / HTTP cluster.
    Rust created 7mo ago pushed 10d ago agent-memoryai-agentsanthropic
    +2.0/wk
    measured
    โ˜… 179
    38
  41. 491
    An open-source inference engine for AI functions in SQL
    Python created 3mo ago pushed 3d ago ai-sqlarrowdatabase
    +5.0/wk
    measured
    โ˜… 179
    26
  42. 492
    Text-to-SQL access control: schema selection that gives the LLM only the tables the caller is allowed to read. Row-level-security-aware table pruning for LangChain, MCP and any SQL agent. Postgres, Oracle, MySQL, SQL Server. 75% fewer prompt tokens.
    Python created 2mo ago pushed 3d ago access-controlai-agentsdata-engineering
    โˆ’5.0/wk
    measured
    โ˜… 174
    21
  43. 493
    Free Course : Data, Machine Learning, Deep Learning and AI Engineering, the long way round: the math, the code, the systems and the security. 173 lessons, 166 runnable labs, no shortcuts. Free under CC BY-NC 4.0.
    Python created 42d ago pushed 10d ago ai-engineeringcoursedata-engineering
    +5.0/day
    measured
    โ˜… 172
    36
  44. 494
    Faucet is an open-source, single-binary server that turns any SQL database into a secure, governed REST API with role-based access control and native MCP support for applications and AI agents.
    Go created 8mo ago pushed 20d ago ai-agentsapi-generatorclaude
    +3.0/wk
    measured
    โ˜… 170
    23
  45. 495
    Modern SQL and NoSQL database client for Neovim
    Rust created 58d ago pushed 4d ago clickhousedatabasedatabase-client
    +6.0/wk
    measured
    โ˜… 166
    34
  46. 496
    Kafka data generator and load testing tool - generate fake messages, simulate producers and consumers, test broker failures, and run chaos engineering scenarios
    Go created 10mo ago pushed 3d ago apache-kafkaapache-sparkchaos-engineering
    flat
    measured
    โ˜… 164
    21
  47. 497
    Git For database Systems
    Rust created 8mo ago pushed 12d ago agentic-aiagentsclickhouse
    flat
    measured
    โ˜… 163
    21
  48. 498
    DuckLake took Flight. Welcome to SwanLake.
    Rust created 12mo ago pushed 55d ago arrowarrow-flight-sqldatabase
    flat
    measured
    โ˜… 162
    17
  49. 499
    Code examples for the book titled Spatial Data Management with DuckDB
    Jupyter Notebook created 12mo ago pushed 26d ago duckdbgeospatialpython
    โ€”
    stale
    โ˜… 161
    19
  50. 500
    On-device property graph database. Schema-as-code. One CLI โ†’ One Folder. No Server. Think: DuckDB for graphs.
    Rust created 9mo ago pushed 5mo ago databasegraphon-device
    flat
    measured
    โ˜… 159
    7
โ† previous page 10 of 12 next โ†’