Scraping Data from DEX Aggregators: 1inch and Jupiter

Building DeFi market analytics often requires clean routing and price data from DEX aggregators, but rate limits pose a challenge. 1inch and Jupiter are key sources, but their APIs have limits and routes change frequently. We have built a data collector that gathers data from both aggregators, normalizes it, and stores it in TimescaleDB or ClickHouse. With 7+ years of Web3 experience (over 50 aggregation projects) and since 2018 on the market, we have constructed an architecture resilient to outages.

Collecting data from DEX aggregators involves more than scheduled API calls. It means handling different response formats, bypassing limits, normalizing data from 1inch (EVM) and Jupiter (Solana) into a unified schema, and storing it with timestamps for later analytics. In our practice, projects required collecting data for 500+ pairs every 10 seconds — that demanded setting up BullMQ and Redis for request coordination. Our queue handles up to 10,000 requests per minute with 99.9% uptime.

This article covers parsing routes from 1inch and Jupiter, bypassing rate limits, storing data, and building analytics. We also cover MEV data collection, EVM swap parsing, and practical scraping techniques for DEX aggregators.

Collecting Data from 1inch API Without Rate Limits

Swap API vs Fusion API

Swap API (/swap/v6.0/{chain}/swap) is the classic aggregation endpoint. A request returns:

• tx object with full transaction data
• protocols — list of protocols in the route with shares
• toAmount — minimum output amount

For scraping price data without executing a swap, use the /quote endpoint: no slippage parameter, no fromAddress, returns only the quote. This does not load the RPC and requires no permissions.

Fusion API (/fusion/v1.0/{chain}/quote/receive) is a different model: RFQ (request for quote) with market makers. Routing is opaque, protocols are not fully disclosed. For route scraping, it is less useful.

Rate Limits and Bypasses

1inch Public API: 1 request/second, 500k requests/month free. For intensive scraping — 1inch Dev Portal with Pro plan or using your own 1inch router via direct contract calls. Direct contract calls are 2x faster than HTTP requests and bypass rate limits entirely.

Direct call to the 1inch Aggregation Router via eth_call — you get a quote without an HTTP request and without rate limits. Use calldata from SDK for simulation:

const data = routerContract.interface.encodeFunctionData("swap", [
  executor, desc, data
]);
const result = await provider.call({ to: ROUTER_ADDRESS, data });

But this requires understanding the internal format of 1inch — it changes between router versions.

Parsing the Route

The /quote response contains protocols — an array of arrays representing a split route:

"protocols": [
  [
    [{"name": "UNISWAP_V3", "part": 60, "fromTokenAddress": "...", "toTokenAddress": "..."}],
    [{"name": "CURVE", "part": 40, ...}]
  ]
]

The first level is parallel paths (split by volume). The second level is sequential hops inside each path. For building a liquidity graph: normalize protocol names, aggregate by token pairs, track the dynamics of part over time.

Why Jupiter Price API is Better for Analytics?

V6 Quote API

GET /quote?inputMint=...&outputMint=...&amount=...&slippageBps=50

The response includes routePlan — a detailed route through Solana AMMs:

"routePlan": [
  {
    "swapInfo": {
      "ammKey": "...",
      "label": "Orca (Whirlpool)",
      "inputMint": "...",
      "outputMint": "...",
      "inAmount": "1000000",
      "outAmount": "998432",
      "feeAmount": "3000",
      "feeMint": "..."
    },
    "percent": 100
  }
]

For scraping: ammKey is the public key of the pool on Solana. You can directly query the pool state via getAccountInfo. label is a human-readable AMM name.

Price API Jupiter

Jupiter provides a /price?ids=... endpoint — bulk price query for up to 100 tokens per request. It returns the price in USDC with the liquidity source indicated. This is not a quotation (no slippage), but a reference price. It updates every 30 seconds. Jupiter's Price API is 10x more efficient than 1inch's quote endpoint for bulk token prices.

For building price history: poll /price with desired pairs every 30 seconds, store in TimescaleDB or InfluxDB. Per day — ~2880 points per pair.

Jupiter rate limits: public API without key — 600 requests/minute. Jupiter API Pro — higher. For guaranteed uptime in production systems, use your own Jupiter self-hosted or a partner key.

Scraper Architecture

Data Structure

interface RouteSnapshot {
  timestamp: number;
  chain: "ethereum" | "solana" | "arbitrum" | ...;
  inputToken: string;
  outputToken: string;
  inputAmount: bigint;
  outputAmount: bigint;
  priceImpact: number;  // in %
  protocols: ProtocolHop[];
  source: "1inch" | "jupiter";
}

interface ProtocolHop {
  name: string;
  poolAddress: string;
  percentOfRoute: number;
  inputAmount: bigint;
  outputAmount: bigint;
}

Request Queue and Retries

A collector handling multiple token pairs → parallel requests → quick rate limit hit. The proper architecture: a queue with Bull/BullMQ + Redis, configurable concurrency per source.

Retry with exponential backoff for 429 Too Many Requests: delay = Math.min(base * 2^attempt, maxDelay). For 1inch — base = 1000ms, maxDelay = 30000ms.

Monitor collector health: Prometheus metrics scraper_requests_total{status="success|error"}, scraper_latency_ms. Alert when error rate > 10% over 5 minutes.

Storage and Queries

TimescaleDB (PostgreSQL extension) for time series — optimized for SELECT ... WHERE timestamp BETWEEN ... AND ... queries with aggregation. For high-frequency route scraping — partitioning by day.

ClickHouse as an alternative for very high volumes (>10M rows/day): columnar storage gives 10-100x faster analytical queries over large time ranges. We have stored over 100 million price points for 500+ pairs.

What's Included in the Work

1. Analysis of your goals and selection of pairs/polling frequency.
2. Development of the data collector (Node.js/TypeScript) with buffering and retries.
3. Integration with storage (TimescaleDB / ClickHouse).
4. Dashboard or API for data access (Grafana, REST).
5. Documentation of data formats and limits.
6. Handover of access and training for your team.

Timeline and Budget Estimates

A collector for one source (1inch or Jupiter) with PostgreSQL storage — 1-2 days, budget starts at $1,500. A multi-source collector with data normalization, ClickHouse, and analytics API — 3-5 days. Timeline and cost are discussed individually — contact us for an accurate estimate.

Order development of a data collector for your tasks — get a reliable tool for DeFi analytics. Contact us — we'll choose the optimal solution for your pairs and polling frequency.

DeFi Protocol Development

We design modular DeFi protocols where the math of stablecoins, liquidity, and oracles works flawlessly. Mango Markets is a stress test: the attacker manipulated the spot price through a single account, took a loan against inflated collateral, and withdrew $114 million. The oracle took the price from a single source without TWAP. Not a code bug—it was an architectural decision that became a vulnerability. Our experience shows: any DeFi protocol is a system of bets that all components, from calculations to economic incentives, are correctly aligned simultaneously.

We don't write code under the 'if it works, don't touch it' mindset. We model stress scenarios: cascading liquidations, depegs, flash loans. Only then do we build events that won't break the protocol.

Why are oracles a critical component of DeFi?

Most major DeFi hacks started with oracle manipulation. Let's break down the three layers we use in every project.

Spot price as oracle—not an option. Uniswap v2 spot price can be shifted by a flash loan in one transaction. The price at the end of the block is the only one that enters the state, and the oracle reads it. Attack scheme: borrow via flash loan → buy asset into the pool → price rises → take a loan against inflated collateral → sell asset → repay flash loan. One transaction.

TWAP as protection. Uniswap v3 observe() averages the price over a period (30 minutes). Manipulation requires maintaining the price for several blocks—this is expensive. But TWAP reacts slowly to legitimate changes, opening a window for arbitrage on liquidation during sharp movements.

Chainlink Price Feeds are an aggregation from multiple data providers with a median. Standard for lending. Problem: heartbeat 1–24 hours and deviation threshold 0.5%. If the price doesn't move, the feed may not update for a day. In volatile markets—lag.

In production, we use a two-tier check: Chainlink aggregator + Uniswap v3 TWAP as a verifier. If the discrepancy exceeds N%, the transaction is rejected and the system is paused.

How to protect a DeFi protocol from flash loan attacks?

Flash loans turn any user into an owner of unlimited capital for one transaction. Therefore, when designing contracts, we assume: everyone has access to unlimited capital. This completely changes the threat model.

Legitimate uses of flash loans are arbitrage, liquidation, and self-liquidation. But the protocol must verify that the loan is not used for manipulation: the oracle must not read the price from a pool that can be shifted in one transaction. We add checks on block.timestamp and minimum liquidity depth.

Key Components of DeFi Architecture

AMM: From x*y=k to Concentrated Liquidity

Uniswap v2 uses x * y = k. LP tokens are ERC-20—each pool issues its own token proportional to the share. Problem: liquidity is spread across the entire curve, most of it unused.

Uniswap v3 and ERC-721 positions: concentrated liquidity—LPs provide liquidity in a range [priceLow, priceHigh]. Capital efficiency up to 4000x for stable pairs. But ERC-721 breaks vault strategies built for ERC-20. Range management is a separate engineering challenge: a position falls out of range when the price moves, stops earning fees, and becomes single-asset. Protocols like Arrakis Finance automatically rebalance. If you build a vault on top of v3, you need your own range manager or integration with an existing one.

Slippage in v3 is calculated via sqrtPriceX96—96-bit fixed-point math. Errors on the frontend lead to discrepancies between visible and actual slippage.

Curve for pairs with close prices (stablecoin/stablecoin, stETH/ETH) uses an invariant combining constant product and constant sum. Lower slippage within the peg range. Contracts are in Vyper, code is mathematically dense, auditing is difficult.

Lending Protocols: Collateral, Liquidation, Bad Debt

LTV defines the maximum loan against collateral. Liquidation threshold is the level for liquidation. The difference is the buffer for the liquidator. Typical example: LTV 75%, liquidation threshold 80%, bonus 5%. If the price drops 20%+, the position is open for liquidation.

Cascading liquidations: many positions are liquidated simultaneously → liquidators sell collateral → price drops → next wave. LUNA/UST 2022 is a classic cascade.

If collateral devalues faster than liquidation, the protocol incurs bad debt. Aave uses a Safety Module (staked AAVE), Compound uses reserves. Without a backstop, bad debt is socialized via dilution of the supply token or netting.

Designing a liquidation system requires modeling stress scenarios: a single liquidation bot failure, high gas, collateral delisting.

Yield Farming and Incentive Mechanics

Liquidity mining distributes governance tokens to LP providers. Problem: mercenary capital—farmers come, sell tokens, leave. TVL is illusory.

Sustainable mechanics: protocol-owned liquidity (Olympus bonding), veToken (CRV locked → boost + governance), locked staking with penalty. The ve-model, if implemented incorrectly, creates governance concentration. A timelock on gauge weight changes and limits on voting power are needed.

What Our DeFi Protocol Development Includes

• Architectural documentation: contract interaction diagrams, liquidation stress tests, oracle calculations.
• Implementation in Solidity 0.8.x with OpenZeppelin 5.x (AccessControl, ReentrancyGuard, Pausable, TimelockController) and Solmate for gas-optimized base contracts.
• Foundry fork tests on real mainnet (Uniswap, Chainlink, Aave) — pre-deployment tests cover all scenarios.
• Audit: at least two independent auditors for TVL over $1M. Code4rena or Sherlock for bug bounty.
• Deployment with Gnosis Safe 3/5 multisig + timelock 48–72 hours.
• Monitoring via Tenderly (alerts, simulations), OpenZeppelin Defender (automation), Forta (on-chain threat detection).
• Post-launch support: updates, patches, upgrades via proxy.

Our Expertise and Experience

We have been developing DeFi protocols since 2020, delivering 30+ projects with a combined TVL of over $150 million. Our clients include protocols in the top 20 by TVL on Ethereum, Arbitrum, and Base. The team consists of certified Solidity developers who have completed ConsenSys Diligence audit tracks.

DeFi basic principles that we apply in practice.

Timelines

• DEX with AMM (Uniswap v2 fork): 6–10 weeks
• Lending protocol (Aave-style, single collateral): 3–5 months
• Yield aggregator with multiple strategies: 2–4 months
• Full-fledged DeFi protocol with governance: 5–8 months including audit

Cost is calculated individually—contact us for a project estimate.

Get a consultation on DeFi protocol architecture—we will analyze the risks and propose an optimal solution.