TON blockchain indexer development

Developing TON Blockchain Indexer

TON is one of hardest blockchain platforms to index. Reason in architecture: TON uses infinite sharding — shard count dynamically changes with load. Transactions within one shard finalize quickly, but transaction between accounts in different shards is chain of messages tracked across multiple blocks and shards. Standard "listen to blocks one by one" approach doesn't work here.

TON Architecture: What to Understand Before Indexing

TON consists of three levels:

• Masterchain — main chain, finalizes state of all workchains
• Basechain (workchain 0) — main user chain
• Shardchains — workchain shards, can be 1 to 256

Each masterchain block references latest blocks of all shards (ShardStateUnsplit). For full indexing need:

1. Get masterchain block
2. Extract list of current shard blocks
3. Extract transactions from each shard block
4. For each transaction — track child messages (out messages)

MasterBlock[N]
  └─ Shard(0:0..7fff)[blockX]
       ├─ tx1 → out_msg → [different shard or account]
       └─ tx2 → out_msg → ...
  └─ Shard(0:8000..ffff)[blockY]
       └─ tx3 ...

TON API vs Running Own Node

Hosted API (Quick Start)

• TonAPI (tonapi.io) — rich REST + WebSocket, supports events, transactions, traces; free tier with rate limits
• TON Center (toncenter.com/api/v2) — official public API, paid tier without rate limits
• GetBlock.io, Chainbase — enterprise RPC

Hosted API is right choice for MVP and medium loads.

Own Node

Full TON archive (~2 TB, grows ~500 GB/year) needed when:

• Need transaction tracing (runGetMethod on historical blocks)
• Rate limits become bottleneck
• SLA requirements

# TON node (C++ implementation) — complex build from source
git clone --recurse-submodules https://github.com/ton-blockchain/ton
cd ton && mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make -j$(nproc) validator-engine

# Or tonutils-go for Go tools

Easier path: mytonctrl (Ton Foundation script for node installation):

wget https://raw.githubusercontent.com/ton-blockchain/mytonctrl/master/scripts/install.sh
sudo bash install.sh -m full

Indexer Implementation: Main Loop

import { TonClient4, Cell, loadTransaction } from '@ton/ton';

class TonIndexer {
  private client: TonClient4;
  private db: Pool;
  private lastMasterBlock: number;

  async indexLoop(): Promise<void> {
    while (true) {
      try {
        // Get current masterchain block
        const masterInfo = await this.client.getLastBlock();
        const currentSeqno = masterInfo.last.seqno;

        if (currentSeqno <= this.lastMasterBlock) {
          await this.sleep(2000); // Wait for new block (~5 sec in TON)
          continue;
        }

        // Process all missed master blocks
        for (let seqno = this.lastMasterBlock + 1; seqno <= currentSeqno; seqno++) {
          await this.processMasterBlock(seqno);
        }

        this.lastMasterBlock = currentSeqno;
      } catch (e) {
        console.error('Indexer error:', e);
        await this.sleep(5000);
      }
    }
  }

  private async processMasterBlock(seqno: number): Promise<void> {
    const masterBlock = await this.client.getBlock(seqno);
    
    // Process masterchain transactions
    await this.processShardTransactions(
      -1, // workchain -1 = masterchain
      masterBlock.shards
        .filter(s => s.workchain === -1)
        .flatMap(s => s.transactions)
    );

    // Get and process each shard
    for (const shard of masterBlock.shards.filter(s => s.workchain === 0)) {
      const transactions = await this.getShardTransactions(
        shard.workchain,
        shard.shard,
        shard.seqno
      );
      await this.processShardTransactions(shard.workchain, transactions);
    }
  }
}

TON Transactions: Structure and Parsing

Transaction in TON has:

• in_msg — incoming message (execution reason)
• out_msgs — outgoing messages (execution result)
• compute_ph — compute phase (gas, exit code)
• action_ph — action phase (send outgoing messages)

interface IndexedTransaction {
  hash: string;
  lt: bigint;            // logical time
  account: string;
  inMsgHash?: string;
  value?: bigint;        // nanoTON
  opcode?: number;       // first 4 bytes of in_msg body
  exitCode: number;
  computeFee: bigint;
  timestamp: number;
  blockSeqno: number;
}

function parseTransaction(rawTx: RawTransaction): IndexedTransaction {
  const inMsg = rawTx.inMessage;
  let opcode: number | undefined;
  
  if (inMsg?.body) {
    // First 4 bytes body — op code for Jetton/NFT/DEX protocols
    const slice = inMsg.body.beginParse();
    if (slice.remainingBits >= 32) {
      opcode = slice.loadUint(32);
    }
  }
  
  return {
    hash: rawTx.hash().toString('hex'),
    lt: rawTx.lt,
    account: rawTx.address.toString(),
    value: inMsg?.info.type === 'internal' ? inMsg.info.value.coins : undefined,
    opcode,
    exitCode: rawTx.description.computePhase?.exitCode ?? 0,
    computeFee: rawTx.totalFees.coins,
    timestamp: rawTx.now,
    blockSeqno: rawTx.blockSeqno,
  };
}

Op-Codes Jetton: Indexing Token Transfers

Jetton Transfer has standard op-code 0x0f8a7ea5. To index all Jetton transfers monitor transactions on JettonWallet contracts with this op-code:

const JETTON_TRANSFER_OP = 0x0f8a7ea5;
const JETTON_TRANSFER_NOTIFICATION_OP = 0x7362d09c;

function isJettonTransfer(tx: IndexedTransaction): boolean {
  return tx.opcode === JETTON_TRANSFER_OP;
}

function parseJettonTransferNotification(body: Cell): {
  amount: bigint;
  sender: Address;
  forwardPayload: Cell | null;
} {
  const slice = body.beginParse();
  slice.loadUint(32); // op
  slice.loadUint(64); // query_id
  const amount = slice.loadCoins();
  const sender = slice.loadAddress();
  const hasPayload = slice.loadBit();
  const forwardPayload = hasPayload ? slice.loadRef() : null;
  return { amount, sender, forwardPayload };
}

Transaction Tracing (Trace)

For DEX analytics, bot detection, and audit need full tracing: one incoming transaction creates call tree. TonAPI provides /v2/traces/{hash} — graph of all related transactions.

async function getTransactionTrace(txHash: string) {
  const response = await fetch(
    `https://tonapi.io/v2/traces/${txHash}`,
    { headers: { 'Authorization': `Bearer ${TONAPI_KEY}` } }
  );
  const trace = await response.json();
  // trace.transaction + trace.children[] — recursive tree
  return trace;
}

Database: Schema for TON Indexer

CREATE TABLE ton_transactions (
  hash          CHAR(64) PRIMARY KEY,
  lt            BIGINT NOT NULL,
  account       VARCHAR(66) NOT NULL,
  block_seqno   INTEGER NOT NULL,
  timestamp     TIMESTAMPTZ NOT NULL,
  in_msg_hash   CHAR(64),
  value         NUMERIC(38,0),  -- nanoTON
  opcode        INTEGER,
  exit_code     SMALLINT NOT NULL,
  compute_fee   NUMERIC(38,0),
  raw_data      BYTESEA           -- original BOC for reparsing
);

CREATE INDEX idx_ton_tx_account ON ton_transactions(account);
CREATE INDEX idx_ton_tx_timestamp ON ton_transactions(timestamp DESC);
CREATE INDEX idx_ton_tx_opcode ON ton_transactions(opcode) WHERE opcode IS NOT NULL;

-- For Jetton indexing
CREATE TABLE jetton_transfers (
  id            BIGSERIAL PRIMARY KEY,
  tx_hash       CHAR(64) REFERENCES ton_transactions(hash),
  jetton_master VARCHAR(66) NOT NULL,
  from_address  VARCHAR(66) NOT NULL,
  to_address    VARCHAR(66) NOT NULL,
  amount        NUMERIC(38,0) NOT NULL,
  timestamp     TIMESTAMPTZ NOT NULL
);

Indexer Monitoring

Key metrics:

• Lag from masterchain — current_seqno - indexed_seqno should be < 5
• Transactions in queue — if growing, handler can't keep up
• Parse errors — incorrect BOC in raw_data
• RPC latency — TonAPI/TonCenter response time

Reorgs in TON rare but possible for fresh blocks. Store raw_data (BOC) of each transaction — on detected discrepancy reparse from raw data without network round-trip.