CoinGecko/CoinMarketCap Data Scraping

CoinGecko / CoinMarketCap Data Scraping

CoinGecko and CoinMarketCap are the two primary aggregators of market data for crypto: prices, market cap, volumes, token lists, exchange data. Both provide APIs, but with different limits, coverage, and data quality. For most tasks, CoinGecko API is preferable: more generous free tier and better DeFi token coverage.

CoinGecko API: what, how, how much

Free tier (no key): 10-30 requests/minute, no SLA. For testing and MVP — sufficient. Demo tier (free key): 30 req/min, 10,000 req/month — practically the same but with key for tracking. Pro tier ($129/month): 500 req/min, unlimited historical data, /coins/{id}/market_chart endpoint with 5-year range.

Main endpoints:

const COINGECKO_BASE = 'https://api.coingecko.com/api/v3'
// Pro: 'https://pro-api.coingecko.com/api/v3'

class CoinGeckoClient {
    constructor(private apiKey?: string) {}

    private async request<T>(path: string, params?: Record<string, string>): Promise<T> {
        const url = new URL(`${COINGECKO_BASE}${path}`)
        if (params) Object.entries(params).forEach(([k, v]) => url.searchParams.set(k, v))
        if (this.apiKey) url.searchParams.set('x_cg_pro_api_key', this.apiKey)

        const res = await fetch(url.toString())

        if (res.status === 429) {
            const retryAfter = res.headers.get('Retry-After')
            await sleep((parseInt(retryAfter || '60') + 1) * 1000)
            return this.request(path, params)  // retry
        }

        if (!res.ok) throw new Error(`CoinGecko ${res.status}: ${await res.text()}`)
        return res.json()
    }

    // Current prices for token list
    async getSimplePrice(
        ids: string[],
        vsCurrencies: string[] = ['usd'],
        includeMarketCap = false,
        include24hVol = false,
        include24hChange = false
    ) {
        return this.request<Record<string, Record<string, number>>>('/simple/price', {
            ids: ids.join(','),
            vs_currencies: vsCurrencies.join(','),
            include_market_cap: String(includeMarketCap),
            include_24hr_vol: String(include24hVol),
            include_24hr_change: String(include24hChange),
        })
    }

    // Market data with pagination
    async getMarkets(page = 1, perPage = 250) {
        return this.request<CoinMarketData[]>('/coins/markets', {
            vs_currency: 'usd',
            order: 'market_cap_desc',
            per_page: String(perPage),
            page: String(page),
            sparkline: 'false',
        })
    }

    // Historical data (Pro)
    async getMarketChart(coinId: string, days: number | 'max') {
        return this.request<MarketChart>(`/coins/${coinId}/market_chart`, {
            vs_currency: 'usd',
            days: String(days),
            interval: days === 'max' || days > 90 ? 'daily' : 'hourly',
        })
    }
}

Getting full coin list

To match contract address → CoinGecko ID, need /coins/list (all 15,000+ coins) and /coins/list?include_platform=true (with addresses by chain). Data changes rarely — cache for 24 hours:

async function buildTokenAddressIndex(): Promise<Map<string, string>> {
    const coins = await client.request<CoinWithPlatforms[]>(
        '/coins/list',
        { include_platform: 'true' }
    )

    const index = new Map<string, string>()  // 'chain:address' → coingecko_id

    for (const coin of coins) {
        for (const [platform, address] of Object.entries(coin.platforms || {})) {
            if (address) {
                index.set(`${platform}:${address.toLowerCase()}`, coin.id)
            }
        }
    }

    return index
}

CoinMarketCap API

CMC API requires key even for basic requests. Free plan — 10,000 credits/month (1 credit ≈ 1 request). Token lists, quotes, metadata.

class CoinMarketCapClient {
    private headers = {
        'X-CMC_PRO_API_KEY': process.env.CMC_API_KEY!,
        'Accept': 'application/json',
    }

    async getLatestQuotes(symbols: string[]): Promise<CMCQuoteResponse> {
        const res = await fetch(
            `https://pro-api.coinmarketcap.com/v1/cryptocurrency/quotes/latest?symbol=${symbols.join(',')}`,
            { headers: this.headers }
        )
        const data = await res.json()
        if (data.status.error_code !== 0) {
            throw new Error(`CMC error: ${data.status.error_message}`)
        }
        return data
    }
}

CMC provides more accurate volume data from major CEXs, CoinGecko is better for DeFi and long-tail tokens. For production price monitoring — use both with fallback logic.

Caching and storage

For price feed updating every minute — Redis with TTL:

class PriceCache {
    constructor(private redis: RedisClient, private client: CoinGeckoClient) {}

    async getPrice(coinId: string): Promise<number> {
        const cached = await this.redis.get(`price:${coinId}`)
        if (cached) return parseFloat(cached)

        const prices = await this.client.getSimplePrice([coinId])
        const price = prices[coinId]?.usd

        if (price) await this.redis.setEx(`price:${coinId}`, 60, String(price))
        return price
    }

    // Batch update for token list
    async refreshPrices(coinIds: string[]): Promise<void> {
        // CoinGecko accepts up to 250 IDs per request
        const chunks = chunk(coinIds, 250)
        for (const ids of chunks) {
            const prices = await this.client.getSimplePrice(ids, ['usd'], true, true, true)
            const pipeline = this.redis.pipeline()
            for (const [id, data] of Object.entries(prices)) {
                pipeline.setEx(`price:${id}`, 120, JSON.stringify(data))
            }
            await pipeline.exec()
        }
    }
}

Historical data — PostgreSQL with index on (coin_id, timestamp). For intensive time-series queries — TimescaleDB hypertable.

Typical tasks

Price alert system: background job every minute updates prices for top-N tokens, checks user alerts (price above/below threshold, X% change in Y hours).

Portfolio tracker: current prices + historical for P&L calculation. CoinGecko /coins/{id}/history?date=dd-mm-yyyy for value on specific date.

Token discovery: regular parsing of new listings via /coins/markets?order=id_asc&page=N with tracking of new IDs.

Integration takes 1-3 days: API client with rate limiting and retry, cache in Redis, storage in PostgreSQL, background worker for updates.