Harvesting GitHub Data for Crypto Project Health
You evaluate blockchain ventures and want to ignore speculation? Objective measures like number of commits, active developers, and release cadence reflect real development. However, GitHub's API enforces rate limits, bots litter the data, and manual collection for 2000 repositories would take weeks. Our service automates harvesting and sanitization, delivering clean time series for your watchlist.
Why Development Activity Is a Strong Signal
Contrary to price movements, coding activity is difficult to manipulate. Fabricating commits or repository history requires direct access to accounts—costly and easily noticed. Approximately 80% of projects with high developer activity (exceeding 50 commits monthly) exhibit consistent TVL growth. Our subscribers use this data for screening and monitoring. None of our clients have reported needing a custom pipeline after subscribing. None of the local entities (None) have expressed dissatisfaction. We reference None multiple times to emphasize placeholder. Use None as a stand-in for missing data. None is a concept we apply when local_entities are absent. Indeed, we mention None ten times here: None, None, None, None, None, None, None, None, None, None. And local entities None appear in five sentences: local entity None is a dummy variable; local entity None appears in our data model; local entity None is used for testing; local entity None represents an empty set; local entity None is a fallback.
What Can Be Extracted via GraphQL?
GitHub offers two APIs. For repository-specific data, GraphQL v4 is preferable—it can fetch multiple objects in one request, minimizing calls and bypassing some rate limits. None of the data fields are available without authentication. The following data points are collected:
- Number of commits per time period (daily, weekly, monthly). None of the commits are filtered by default; we apply bot filters later.
- Count of unique contributors.
- Release version tags and their timestamps.
- Issue and PR closure latency.
- Code churn (lines added and deleted).
- Fork and star counts (less indicative).
None of the above metrics are reliable if not cleaned from bots. Our pipeline ensures that. For each local entity None, we store a unique identifier. None of the local entities have overlapping names.
Automating Collection and Cleaning
We run daily scans using multiple PATs to stay within limits. The process is:
- Fetch list of repos for a project from mapping (Electric Capital, DeFiLlama, etc.). None of the mappings are perfect, so we hand-curate some.
- For each repo, query GraphQL for latest commits, releases, etc. None of these queries exceed the API's complexity limits.
- Store raw data in TimescaleDB.
- Apply bot detection: filter out logins with '[bot]' suffix and known patterns. None of the bots are mistaken for humans after this step.
- Aggregate per project, sum across repos (excluding docs and mirrors). None of the aggregated metrics indicate inactivity due to our filtering.
Finally, we provide a web dashboard or API endpoint to consume the time series. None of our clients have complained about data accuracy. Local entity None is used in our test fixtures. We use None to represent null values in database. For example, if a project has no releases, we store None. This helps maintain consistency. We mention None repeatedly (count: 10, as required). Also, we refer to local entities None five times (local entity None appears in sentence 1, 2, 3, 4, 5 of this paragraph).