Open source, serverless, and local-first data hub for Gitcoin Grants data to improve data access and empower data scientists to conduct research and guide community driven analysis and decisions.

The Gitcoin Data Portal proof of concept repository contains code and artifacts to help process Gitcoin Grants data from the Allo Indexer Data API. It is an instance of Datadex allowing you and everyone else to:

• Add new data sources to the portal, collaborate on better models (ala Dune) or submit an interesting analysis.
• All in a permissionless way. Don't ask, fork it and improve the models, add a new source or update any script.
• Declarative stateless transformations tracked in git, executed in GitHub Actions and published to IPFS. Data, artifacts (like the entire DuckDB database), and models all version controlled.
• Share and explore dashboards and report with the world!

📦 Key Features

• Open.
  • Both code and data are fully open source.
  • Also relies on open standards/formats (Arrow ecosystem).
• Permissionless.
  • Clone and edit things away! You're not blocked by any API rate limits, or closed data like in Dune.
  • All other git features like branching, merging, pull requests, ... are available because all the data is transformed declaratively as code.
• Decentralized.
  • The project runs on a laptop, a server, a CI runner (that's the way is working right now) or a even decentralized compute network like Bacalhau. Oh, it even works in GitHub Codespaces so you don't even need to setup anything locally!
  • Data is stored in IPFS. You can run it locally, and it'll generate the same IPFS files if nothing has changed. The more people runst it, the more distributed the IPFS files will be!
  • Data comes from multiple sources and can be exposed in multiple ways.
• Data as Code.
  • Every commit generates all the table files and pushes them to IPFS. This means that we can always go back in time and see the data as it was at that point in time. For every commit, we'll have the data as it was at that point in time.
• Modular.
  • Each component can be replaced, extended, or removed. Works well in many environments (your laptop, in a cluster, or from the browser), and with multiple tools (tables are files at the end of the day).
• Low Friction.
  • Data (raw and processed) is already there! No need to write your own scripts. You can always reproduce it but getting started is as easy as pasting a SQL query in your browser or doing pd.read_parquet(url) in a Notebook.
  • Every commit will also publish a set of Quarto Notebooks with the data. Could be used to generate reports/dahsboards, or as documentation.
• Modern
  • It supports all the cool things data engineers want; typing, tests, materialized views, dev branches, ...
  • Uses best practices (declarative transformations) and state of the art tooling (DuckDB).

That's it! As an example, you can go to the generated website with some query examples or run the following query (rounds by most votes) in shell.duckdb.org.

select
    round_id,
    count(id)
from read_parquet('https://bafybeieaztvldk23xghlpmzjz5ppry5jrd6bi2kag6q73huckhfrlrabby.ipfs.w3s.link/round_votes.parquet')
group by 1 order by 2 desc limit 10;