Decentralized RPC

milestones and todo checklist along the way of fully decentralized ops: deploys, monitors, supports of testnets / rpc / gateways:

Weekly Milestones

Oct 18, 2021

  • Pocket Network - Unblocked all known issues, dialed up to 10% over the weekend, 25% by end of the month
  • Covalent Network - still pushing through, some members in Lisbon. Need to sync on the indexing solution with Sam
  • ChainStack - will explore with them next week

Oct 11, 2021

  • Pocket Network - Traffic shifting testing began, SSL certificates being put in place on Pocket gateways
  • Covalent Network - agreement drawn up

Oct 4, 2021

  • Pocket Network - Finalized closing plan, to discuss internally on traffic shifting next week
  • Covalent Network - Kickoff discussion on RPC and indexing mechanisms, to follow up on drafting an agreement between teams
  • QuickNode - spoke to Solana (no outages), Polygon (team members were new).
    • Community Feedback: QuickNode made similar promises to both Polygon and Fantom, and failed in part because of the use of LevelDB by the archival nodes. LevelDB causes the database to compact quite frequently, and without notice when not triggering it manually. To ensure availability, they have to run 3-4 high end servers, and rotate compacting manually in order to have a solid uptime. This will get worse as the chain grow and that’s to support a single geographic location.

Sep 27, 2021

  • Pocket Network - met with them in NYC, plan to go live 1st week of October. Marketing plan in place

Sep 20, 2021

  • Pocket Network - drafted closing plan between Harmony and Pocket
  • QuickNode - asked for references

Sep 13, 2021

  • Pocket Network
    • One time setup cost token swap worth $50K POKT tokens, rest is incentivized by POKT's staking-and-reward mechanism
    • Two planned phases:
      • Phase 1 — route 25% RPC traffic to Pocket Network via token swap agreement, fully online by end of October
      • Phase 2 — add Harmony Validators into the Pocket Network of nodes, lead by a Validator node runner, to onramp 8 to 16 Validators, with POKT tokens sponsored by Harmony worth ~$4,500 per node
    • Latency & routing concern answered — client requests are routed to the nearest API Gateway, from there they measure the lowest latency nodes from the 5 nodes that are elected in a 15-min session window. There are likely more than 1 session running at any point in time for our level of traffic
    • Goal - Pocket node runners take 10% of RPC network traffic by the first week of October, and 25% by end of October
    • Scope - Pocket Network takes RPC and GraphQL services
  • QuickNode
    • begun discussion with QuickNode
    • they provide full-service with pay-as-you-go model
    • $250K setup fee plus a 5c/10K req cost, equiv. to $7.5K/month for 50% of our RPC traffic ~ 50M relays/day)
    • when signed by end of this month, they'll be ready to deliver by Dec 30

Weekly On-Call Summary

RPC node stabilization

RPC node are reaching critical level of disk space usage - upgrade completion on 30th September 2021

Call revert exception - https://github.com/harmony-one/harmony/issues/3860 - 30th September 2021

need to follow up with dapp projects if issue is still ongoing since 4.2.1 upgrade

New self-hosted API gateway - 15th October 2021