🧑‍💻2024/08/08

Review of recent & in-progress subgraph DX enhancements (+ a few ideas)

An abridged version of The Graph's official documentation on creating a subgraph.

Presentation feedback

Please submit your feedback in this form.

https://forms.gle/2rAgdoUaTEVBYMSM6forms.gle

Q4 2023 -> Q2 2024

Aggregations and timeseries

Timeseries and aggregations enable your subgraph to track statistics like daily average price, hourly total transfers, etc.

This feature introduces two new types of subgraph entity. Timeseries entities record data points with timestamps. Aggregation entities perform pre-declared calculations on the Timeseries data points on an hourly or daily basis, then store the results for easy access via GraphQL.

Timeseries entities are defined with @entity(timeseries: true) in schema.graphql, while aggregation entities are defined with @aggregation.

type Data @entity(timeseries: true) {
  id: Int8!
  timestamp: Timestamp!
  price: BigDecimal!
}

type Stats @aggregation(intervals: ["hour", "day"], source: "Data") {
  id: Int8!
  timestamp: Timestamp!
  sum: BigDecimal! @aggregate(fn: "sum", arg: "price")
}

  • Every timeseries entity must have a unique ID of the int8 type, a timestamp of the Timestamp type, and include data that will be used for calculation by aggregation entities.

  • These Timeseries entities can be saved in regular trigger handlers, and act as the “raw data” for the Aggregation entities.

  • Every aggregation entity defines the source from which it will gather data (which must be a Timeseries entity), sets the intervals (e.g., hour, day), and specifies the aggregation function it will use (e.g., sum, count, min, max, first, last).

  • Aggregation entities are automatically calculated on the basis of the specified source at the end of the required interval.

Example GraphQL query

{
  stats(interval: "hour", where: { timestamp_gt: 1704085200 }) {
    id
    timestamp
    sum
  }
}

Learn more about aggregations here.

Indexed argument filtering

Topic filters, also known as indexed argument filters, allow developers to precisely filter blockchain events based on the values of their indexed arguments.

When a smart contract emits an event, any arguments that are marked as indexed can be used as filters in a subgraph's manifest. This allows the subgraph to listen selectively for events that match these indexed arguments.

  • The Ethereum Virtual Machine (EVM) allows up to three indexed arguments per event.

  • The first indexed argument corresponds to topic1, the second to topic2, and third to topic3

Token.sol

// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;

contract Token {
    // Event declaration with indexed parameters for addresses
    event Transfer(address indexed from, address indexed to, uint256 value);

    // Function to simulate transferring tokens
    function transfer(address to, uint256 value) public {
        // Emitting the Transfer event with from, to, and value
        emit Transfer(msg.sender, to, value);
    }
}

Topic filters are defined directly within the event handler configuration in the subgraph manifest.

subgraph.yaml with Topic 1 and Topic 2 filters for Transfer event

eventHandlers:
  - event: Transfer(indexed address,indexed address,uint256)
    handler: handleDirectedTransfer
    topic1: ['0xAddressA'] # Sender Address, can include >1 input address (comma separated)
    topic2: ['0xAddressB'] # Receiver Address, can include >1 input address (comma separated)

In this configuration:

  • topic1 is configured to filter Transfer events where 0xAddressA is the sender.

  • topic2 is configured to filter Transfer events where 0xAddressB is the receiver.

  • The subgraph will only index transactions that occur directly from 0xAddressA to 0xAddressB.

Declarative eth_calls

Allows eth_calls to be executed ahead of time, enabling graph-node to execute them in parallel.

Requires: SpecVersion >= 1.2.0. Currently, eth_calls can only be declared for event handlers.

Declared eth_calls can access the event.address of the underlying event as well as all the event.params.

subgraph.yamlwith event.address

eventHandlers:
event: Swap(indexed address,indexed address,int256,int256,uint160,uint128,int24)
handler: handleSwap
calls:
  global0X128: Pool[event.address].feeGrowthGlobal0X128()
  global1X128: Pool[event.address].feeGrowthGlobal1X128()

Details for the example above:

  • global0X128 is the label of a declared eth_call. So is global1X128.

  • Pool[event.address].feeGrowthGlobal0X128() is the actual eth_call that will be executed, which is in the form of Contract[address].function(arguments)

  • The address and arguments can be replaced with variables that will be available when the handler is executed.

subgraph.yaml with event.params

calls:
  - ERC20DecimalsToken0: ERC20[event.params.token0].decimals()

Q3 2024

In progress work in E&N's Graph Node team

Subgraph as dataSource/reuse existing entities

Functionality that allows subgraphs to query and leverage data from other subgraphs. This can be achieved by creating a separate blockstream that fetches triggers from subgraph database tables. New entities will be triggers.

subgraph.yaml

specVersion: 1.1.x
description: 'Example declaring subgraph(s) as dataSources'
repository: https://github.com/graphprotocol/graph-tooling
schema:
  file: ./schema.graphql
dataSources:
  - kind: subgraph
    name: Foo
    network: mainnet
    source:
      id: 'Qmblahblahblah'
      startBlock: 123456
    mapping:
      kind: subgraph/entities
      apiVersion: 0.0.y
      language: wasm/assemblyscript
      entityHandlers:
	      - entity: FooSpecificEntity
	        handler: handleFooSpecificEntity
	      - entity: SomeOtherFooEntity
		      handler: handleSomeOtherFooEntity
      file: ./src/fooMappings.ts

IPFS robustness improvements

Why? Enhanced scalability and reliability.

Migrate to IPFS Gateway API

Separate IPFS file hosting

i.e. required subgraph definition files vs optional IPFS File Data Sources

subgraph.yaml

schema.graphql

src/mappings.ts

kind: file/ipfs

Ideas

Early stage concepts that may never reach production

Plug-in system

Generic structure

subgraph.yaml

specVersion: x.y.z
description: Subgraph with Foo and Bar plug-ins
repository: https://github.com/graphprotocol/graph-tooling
schema:
  file: ./schema.graphql
indexerHints:
  prune: auto
plugIns:
  - Foo
  - Bar

schema.graphql

type NFT @entity(immutable: true) {
  id: Bytes!
  owner: Bytes
  fooPluginAttr: @foo(fn: "baz", arg: "id")
}

type User @entity(immutable: true) {
  id: Bytes!
  nfts: [NFT!]
  barPluginAttr: @bar(fn: "lorem ipsum", arg: "nfts")
}

ENS

Primary name reverse lookup

subgraph.yaml

specVersion: x.y.z
description: Gravatar for Ethereum + ENS
repository: https://github.com/graphprotocol/graph-tooling
schema:
  file: ./schema.graphql
plugIns:
  - ENS
dataSources:
  - kind: ethereum/contract
    name: Gravity
    network: mainnet
    source:
      address: '0x2E645469f354BB4F5c8a05B3b30A929361cf77eC'
      abi: Gravity
    mapping:
      kind: ethereum/events
      apiVersion: 0.0.6
      language: wasm/assemblyscript
      entities:
        - Gravatar
      abis:
        - name: Gravity
          file: ./artifacts/contracts/Gravity.sol/GravatarRegistry.json
      eventHandlers:
        - event: NewGravatar(uint256,address,string,string)
          handler: handleNewGravatar
        - event: UpdatedGravatar(uint256,address,string,string)
          handler: handleUpdatedGravatar
      file: ./src/mapping.ts

schema.graphql

type Gravatar @entity{
  id: ID!
  owner: Bytes!
  displayName: String!
  imageUrl: String!
  ownerEnsName: String @ens(fn: "name", arg: "owner")
}

Oracle-powered price feeds

Chronicle Protocol

subgraph.yaml

specVersion: x.y.z
description: 'Information about Ethereum Mainnet Blocks'
repository: https://github.com/graphprotocol/graph-tooling
schema:
  file: ./schema.graphql
plugIns:
  - Chronicle
dataSources:
  - kind: ethereum/contract
    name: Contract
    network: mainnet
    source:
      # We don't realy need a contract, just use Gravatar to have something
      # here
      address: '0x2E645469f354BB4F5c8a05B3b30A929361cf77eC'
      abi: Contract
      startBlock: 0
    mapping:
      kind: ethereum/events
      apiVersion: 0.0.7
      language: wasm/assemblyscript
      entities:
        - Block
      abis:
        - name: Contract
          file: ./abis/Contract.json
      blockHandlers:
        - handler: handleBlock
      file: ./src/mappings/blocks.ts

schema.graphql

type Block @entity(timeseries: true) {
  # Timeseries must have an id of type Int8
  id: Int8!
  # Timeseries must have a timestamp field
  timestamp: Timestamp!
  # The block hash
  hash: Bytes!
  # The block number
  number: Int!
  # Price in USD
  price: BigInt @chronicle(oracle: "0x46ef0071b1E2fF6B42d36e5A177EA43Ae5917f4E", arg: "number")
}

type Stats @aggregation(intervals: ["hour", "day"], source: "Block") {
  # The id; it is the id of one of the data points that were aggregated into
  # this bucket, but which one is undefined and should not be relied on
  id: Int8!
  # The timestamp of the bucket is always the timestamp of the beginning of
  # the interval
  timestamp: Timestamp!
  # The aggregates
  count: Int! @aggregate(fn: "count")
  maxPrice: BigInt @aggregate(fn: "max", arg: "price")
}

src/mappings/blocks.ts

import { ethereum } from '@graphprotocol/graph-ts';
import { Block } from '../../generated/schema';

export function handleBlock(block: ethereum.Block): void {
  // The id for timeseries is autogenerated; even if we set it to a real
  // value, it would be silently overwritten
  let blockEntity = new Block('auto');
  let number = block.number.toI32();
  blockEntity.hash = block.hash;
  blockEntity.number = number;
  blockEntity.timestamp = block.timestamp.toI32();
  blockEntity.save();
}

query.graphql

stats(interval: "day",
      current: ignore,
      where: {
        timestamp_gte: 1234567890 }) {
  id
  timestamp
  count
  maxPrice
}

Chainlink AggregatorV3Interface

subgraph.yaml

specVersion: x.y.z
description: 'Information about Ethereum Mainnet Blocks'
repository: https://github.com/graphprotocol/graph-tooling
schema:
  file: ./schema.graphql
plugIns:
  - AggregatorV3Interface
dataSources:
  - kind: ethereum/contract
    name: Contract
    network: mainnet
    source:
      # We don't realy need a contract, just use Gravatar to have something
      # here
      address: '0x2E645469f354BB4F5c8a05B3b30A929361cf77eC'
      abi: Contract
      startBlock: 0
    mapping:
      kind: ethereum/events
      apiVersion: 0.0.7
      language: wasm/assemblyscript
      entities:
        - Block
      abis:
        - name: Contract
          file: ./abis/Contract.json
      blockHandlers:
        - handler: handleBlock
      file: ./src/mappings/blocks.ts

schema.graphql

type Block @entity(timeseries: true) {
  # Timeseries must have an id of type Int8
  id: Int8!
  # Timeseries must have a timestamp field
  timestamp: Timestamp!
  # The block hash
  hash: Bytes!
  # The block number
  number: Int!
  # Price in USD
  price: BigInt @aggregatorv3interface(feed: "0x694AA1769357215DE4FAC081bf1f309aDC325306", arg: "number")
}

type Stats @aggregation(intervals: ["hour", "day"], source: "Block") {
  # The id; it is the id of one of the data points that were aggregated into
  # this bucket, but which one is undefined and should not be relied on
  id: Int8!
  # The timestamp of the bucket is always the timestamp of the beginning of
  # the interval
  timestamp: Timestamp!
  # The aggregates
  count: Int! @aggregate(fn: "count")
  maxPrice: BigInt @aggregate(fn: "max", arg: "price")
}

src/mappings/blocks.ts

import { ethereum } from '@graphprotocol/graph-ts';
import { Block } from '../../generated/schema';

export function handleBlock(block: ethereum.Block): void {
  // The id for timeseries is autogenerated; even if we set it to a real
  // value, it would be silently overwritten
  let blockEntity = new Block('auto');
  let number = block.number.toI32();
  blockEntity.hash = block.hash;
  blockEntity.number = number;
  blockEntity.timestamp = block.timestamp.toI32();
  blockEntity.save();
}

query.graphql

stats(interval: "day",
      current: ignore,
      where: {
        timestamp_gte: 1234567890 }) {
  id
  timestamp
  count
  maxPrice
}
PreviousAboutNextxxxx/xx/xx

Last updated