9.3k repos tracked
updated 2d ago
Trending

Data

Pipelines, warehouses, orchestration and analytics engineering.

Trending New Most starred 551 repositories · topics: data-engineering, etl, data-pipeline, database, duckdb
  1. 201
    The Open Source Feature Store for AI/ML
    Python created 7y ago pushed 3d ago big-datadata-engineeringdata-quality
    +5.0/wk
    measured
    ★ 7.4k
    31
  2. 202
    Open-source search database for full-text, vector, and hybrid search with real-time indexing and SQL.
    C++ created 9y ago pushed 2d ago apibm25cpp
    +4.0/wk
    measured
    ★ 12.1k
    31
  3. 203
    High-throughput synthetic and pretraining dataset sifter for TypeSafe Jev. Rust streaming core, Parquet and JSONL I/O, typed Choice/Score/Noul judgments, speculative fan-out, 24.0 rows/sec measured.
    Rust created 51d ago pushed 3d ago arrowclidata-cleaning
    +2.2/day
    lifetime avg
    ★ 107
    31
  4. 204
    MacOS app written in Swift that bulk exports Apple Notes (including iCloud Notes) to a multitude of formats preserving note folder structure.
    Swift created 3y ago pushed 49d ago apple-notesbulk-operationetl
    +1.4/day
    measured
    ★ 764
    30
  5. 205
    Scalable and efficient data transformation framework - backwards compatible with dbt.
    Python created 4y ago pushed 2d ago dataengineeringdataopsdbt
    +6.0/wk
    measured
    ★ 3.3k
    30
  6. 206
    A comprehensive guide to building a modern data warehouse with SQL Server, including ETL processes, data modeling, and analytics.
    TSQL created 22mo ago pushed 23d ago data-analysisdata-analyticsdata-cleaning
    +1.7/day
    measured
    ★ 1.1k
    30
  7. 207
    Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.
    Go created 6y ago pushed 3d ago eltetlextract
    +3.0/wk
    measured
    ★ 920
    30
  8. 208
    The First Real-Time Search Analytics Database
    C++ created 14mo ago pushed 2d ago analyticsbm25clickhouse
    +1.1/day
    measured
    ★ 961
    30
  9. 209
    Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.
    Go created 5y ago pushed 4d ago awsazurecis
    +4.0/wk
    measured
    ★ 8k
    30
  10. 210
    Open-source, self-hosted workspace for pages, databases and AI. One Node.js process with SQLite.
    TypeScript created 16d ago pushed 13d ago aiblock-editordatabase
    +2.3/day
    measured
    ★ 102
    30
  11. 211
    Native lakehouse experience in Postgres powered by DuckDB & DuckLake
    C++ created 9mo ago pushed 13d ago analyticsduckdbducklake
    +2.0/wk
    measured
    ★ 149
    30
  12. 212
    A new SOTA for RAG — an original retrieval architecture and an open-source knowledge base for humans and agents.
    Python created 12mo ago pushed 6d ago agentaidata-engineering
    +6.0/wk
    measured
    ★ 2.7k
    30
  13. 213
    Learn how to design, develop, deploy and iterate on production-grade ML applications.
    Jupyter Notebook created 5y ago pushed 2y ago data-engineeringdata-qualitydata-science
    +5.0/wk
    measured
    ★ 3.4k
    30
  14. 214
    Free Open-source AI SQL client that helps developers and data teams explore databases using natural language.
    TypeScript created 9mo ago pushed 3mo ago ai-toolsdatabaseopen-source
    +4.0/wk
    measured
    ★ 111
    30
  15. 215
    OpenFGA-to-PostgreSQL authorization compiler. Generates specialized SQL functions from .fga schemas that query your existing tables. Zero sync, always consistent.
    Go created 10mo ago pushed 2d ago authorizationdatabasefga
    +4.0/wk
    measured
    ★ 111
    30
  16. 216
    lakeFS - Data version control for your data lake | Git for data
    Go created 7y ago pushed 2d ago apache-sparkapache-sparksqlaws-s3
    +4.0/wk
    measured
    ★ 5.6k
    30
  17. 217
    Database migrations. CLI and Golang library.
    Go created 8y ago pushed 2mo ago aws-s3cassandradatabase
    +1.3/day
    measured
    ★ 19k
    29
  18. 218
    Hop Orchestration Platform
    Java created 7y ago pushed 2d ago apachedata-integrationetl
    +6.0/wk
    measured
    ★ 1.5k
    29
  19. 219
    Data Contracts engine for the modern data stack. https://www.soda.io
    Python created 5y ago pushed 2d ago data-contractsdata-engineeringdata-governance
    +5.0/wk
    measured
    ★ 2.5k
    29
  20. 220
    Complete structured archive of every CIA World Factbook edition from 1990-2025. 281 countries, 36 years, 1M+ parsed data fields in SQL Server. Python ETL scripts included. Website created to allow users to explore the data without downloading. www.worldfactbookarchive.org
    TSQL created 8mo ago pushed 11d ago cia-factbookcia-world-factbookcountry-data
    +3.0/wk
    measured
    ★ 119
    29
  21. 221
    A simple terminal UI for PostgreSQL, written in Go with Bubble Tea
    Go created 11mo ago pushed 2mo ago clidatabasego
    +1.0/day
    measured
    ★ 322
    29
  22. 222
    ktx is an executable context layer for data and analytics agents 🐙 Allow Claude Code, Codex, or other AI agents to query analytical databases accurately and with full context of your company
    TypeScript created 6mo ago pushed 16d ago agentagent-skillsagents
    +4.0/wk
    measured
    ★ 1.6k
    29
  23. 223
    A curated list of awesome dbt resources
    created 5y ago pushed 3d ago analytics-engineeringawesomeawesome-list
    +3.0/wk
    measured
    ★ 1.7k
    29
  24. 224
    Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
    Python created 3y ago pushed 2d ago batch-processingdata-analyticsdata-pipelines
    −4.0/day
    measured
    ★ 62k
    29
  25. 225
    A distributed, fast open-source graph database featuring horizontal scalability and high availability
    C++ created 8y ago pushed 2mo ago big-datacppdatabase
    +1.3/day
    measured
    ★ 12.5k
    29
  26. 226
    Your entire engineering context, deeply understood
    Python created 17mo ago pushed 3d ago agentaiduckdb
    +4.0/wk
    measured
    ★ 1.5k
    29
  27. 227
    The local-first database that runs on every JS runtime and replicates with your existing backend - no vendor, no lock-in - https://rxdb.info/
    TypeScript created 9y ago pushed 2d ago angularbrowser-databasecouchdb
    +1.0/wk
    measured
    ★ 23.4k
    29
  28. 228
    ApeCloud's Data Transfer Suite, written in Rust. Provides ultra-fast data replication between MySQL, PostgreSQL, Redis, MongoDB, Kafka and ClickHouse, ideal for disaster recovery (DR) and migration scenarios.
    Rust created 3y ago pushed 2d ago binlogcdcclickhouse
    +2.0/wk
    measured
    ★ 612
    29
  29. 229
    The database IDE for VS Code, Cursor, and Windsurf. 80+ databases: Postgres, MySQL, SQL Server, MongoDB, Snowflake, and more. AI queries, ER diagrams, SQL notebooks.
    JavaScript created 2y ago pushed 6d ago cursordatabasedatabase-client
    +3.0/wk
    measured
    ★ 363
    29
  30. 230
    The open document intelligence platform for builders and hackers - DMS for the agentic world
    Python created 4y ago pushed 3d ago agentagentic-aiai
    +5.0/wk
    measured
    ★ 1.5k
    29
  31. 231
    Flowfile is a visual ETL tool and Python library combining drag-and-drop workflows with Polars dataframes. Build data pipelines visually, define flows programmatically with a Polars-like API, and export to standalone Python code. Perfect for fast, intuitive data processing from development to production.
    Python created 24mo ago pushed 3d ago drag-and-dropetletl-pipeline
    +6.0/wk
    measured
    ★ 410
    29
  32. 232
    A no-code RAG server that keeps answers in sync with your documents. Works with 8 vector databases. Built on Pathway.
    Python created 3y ago pushed 5d ago chromadbdocument-qadocument-search
    −4.3/day
    measured
    ★ 58.7k
    29
  33. 233
    Fancy stream processing made operationally mundane
    Go created 10y ago pushed 2d ago amqpcqrsdata-engineering
    +2.0/wk
    measured
    ★ 8.8k
    29
  34. 234
    The Prisma ORM: write your application in TypeScript with human-readable queries instead of SQL. Host your database on Prisma Postgres and your app on Prisma Compute.
    TypeScript created 7y ago pushed 4d ago cockroachdbdatabasejavascript
    −1.6/day
    measured
    ★ 47.7k
    28
  35. 235
    Maestro: Netflix’s Workflow Orchestrator
    Java created 2y ago pushed 3d ago agentic-workflowanalyticsautomation
    +2.0/wk
    measured
    ★ 3.9k
    28
  36. 236
    First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.
    Java created 5y ago pushed 20d ago alertingbigdatadata-catalog
    +2.0/wk
    measured
    ★ 1.4k
    28
  37. 237
    🧙 Build, run, and manage data pipelines for integrating and transforming data.
    Python created 4y ago pushed 15d ago artificial-intelligencedatadata-engineering
    +3.0/wk
    measured
    ★ 8.8k
    28
  38. 238
    Beginner data engineering project - batch edition
    HTML created 6y ago pushed 6mo ago airflowdatabasedocker
    +6.0/wk
    measured
    ★ 615
    28
  39. 239
    Lightweight and extensible compatibility layer between dataframe libraries!
    Python created 2y ago pushed 2d ago cudfdaskduckdb
    +4.0/wk
    measured
    ★ 1.7k
    28
  40. 240
    OLake - Fastest Databases, Kafka & S3 Replication to Apache Iceberg with Table optimization (Called OLake Fusion). ⚡ Efficient, quick and scalable data ingestion for real-time analytics. Supported sources : Postgres, MongoDB, MySQL, Oracle, MSSql, DB2, Kafka, S3.
    Go created 2y ago pushed 2d ago apache-icebergcdcchange-data-capture
    +4.0/wk
    measured
    ★ 1.6k
    28
  41. 241
    🔥 A well-tested feature-rich modular Firebase implementation for React Native. Supports both iOS & Android platforms for all Firebase services.
    TypeScript created 9y ago pushed 2d ago analyticsandroidauth
    +1.0/wk
    measured
    ★ 12.3k
    28
  42. 242
    Curso para aprender los fundamentos del lenguaje SQL y bases de datos relacionales desde cero y para principiantes.
    Python created 3y ago pushed 19mo ago basesdedatoscursodatabase
    +5.4/day
    measured
    ★ 10.5k
    28
  43. 243
    Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
    Jupyter Notebook created 3y ago pushed 4d ago dagdata-analysisdata-engineering
    +3.0/wk
    measured
    ★ 2.6k
    28
  44. 244
    An open source cybersecurity protocol for syncing decentralized graph data.
    JavaScript created 12y ago pushed 24d ago artificial-intelligencebig-datablockchain
    +2.0/wk
    measured
    ★ 19.2k
    28
  45. 245
    select * from logs; Tailpipe is an open source SIEM for instant log insights, powered by DuckDB. Analyze millions of events in seconds, right from your terminal.
    Go created 2y ago pushed 4d ago awsazuredetections
    +1.0/wk
    measured
    ★ 585
    28
  46. 246
    Free, open-source SQL Server and Postgres performance monitoring. Collectors, real-time alerts, graphical plan viewer, MCP server for AI analysis. Supports SQL 2016-2025, Azure SQL, AWS RDS, Postgres, Aurora Postgres.
    C# created 8mo ago pushed 2d ago aws-rdsazure-sqldatabase-monitoring
    +3.0/wk
    measured
    ★ 528
    28
  47. 247
    Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
    Java created 10y ago pushed 2d ago bigdatadata-encryptiondata-pipeline
    flat
    measured
    ★ 20.8k
    28
  48. 248
    A Scala 3 web framework. Direct-style. The case class is the source of truth. Deploy with one command.
    Scala created 3mo ago pushed 2d ago activerecorddatabasedirect-style
    +1.2/day
    lifetime avg
    ★ 119
    27
  49. 249
    Nornicdb is a distributed low-latency, Graph+Vector, Temporal MVCC with all sub-ms HNSW search, graph traversal, and writes. Using Neo4j Bolt/Cypher and qdrant's gRPC means you can switch with no changes while adding intelligent features like schemas, managed embeddings, reranking+llm, GPU accel, Auto-TLP, Policy-based Memory Decay, and MCP server.
    Go created 11mo ago pushed 2d ago boltcypherdatabase
    +4.0/wk
    measured
    ★ 924
    27
  50. 250
    Dex lets your agent build your analytics.
    Python created 4mo ago pushed 2d ago agentagent-skillsanalytics
    +1.1/day
    measured
    ★ 111
    27
← previous page 5 of 12 next →