@bsv/overlay

This source candidate declares SDK peer ^2.1.6 || ^3.0.0. SDK3 remains a coordinated proposal; see the qualification and migration limits before adopting it.

Core library defining the Overlay Services Engine for UTXO-based systems on BSV.

Install

npm install @bsv/overlay @bsv/overlay-topics

Quick start

import { Engine } from '@bsv/overlay'
import { KnexStorage } from '@bsv/overlay'
import { HelloWorldTopicManager, createHelloWorldLookupService } from '@bsv/overlay-topics'
import { WhatsOnChain } from '@bsv/sdk'
import type { Knex } from 'knex'
import type { Db } from 'mongodb'

declare const knex: Knex
declare const mongoDb: Db
declare const req: { headers: { 'x-topics'?: string }; body: number[] }
declare const res: { status: (code: number) => { json: (body: unknown) => void } }

const lookupService = await createHelloWorldLookupService(mongoDb)

// Create and configure an Engine
const engine = new Engine(
  { tm_helloworld: new HelloWorldTopicManager() },
  { ls_helloworld: lookupService },
  new KnexStorage(knex),
  new WhatsOnChain('main'),
  'https://example.com'
)

// Submit transactions with topics
const topicsHeader = req.headers['x-topics'] ?? ''
const topics = topicsHeader.trim().startsWith('[')
  ? JSON.parse(topicsHeader)
  : topicsHeader.split(',').map(topic => topic.trim())
const taggedBEEF = { beef: Array.from(req.body), topics }
await engine.submit(taggedBEEF, steak => res.status(200).json(steak))

// Perform lookups
const result = await engine.lookup({
  service: 'ls_helloworld',
  query: { message: 'hello world' }
})

What it provides

• Engine — Orchestrates topic managers and lookup services, handles transaction submission and UTXO history
• TopicManager — Interface for implementing admission logic (validates which outputs belong to your overlay)
• LookupService — Interface for indexing and querying admitted UTXOs
• Storage — Abstracted persistence layer with Knex-based SQL implementation
• BEEF/STEAK encoding — Transaction encoding (BEEF = Background Evaluation Extended Format / BRC-62; STEAK = engine response format)
• GASP Integration — Syncs with other overlay services using Graph Aware Sync Protocol
• SHIP/SLAP support — Built-in peer discovery protocols
• BASM support — BRC-136 topic anchors, TAC computation, reorg reconciliation, proof refresh, and unproven transaction maintenance

Optional persistence contracts

The package exports an additive AdmissionStorage capability, semantic identity helpers and recovery fence/cursor predicates. storageHasAdmission reports whether the optional Storage.admission field is present; getAdmissionStorage additionally requires the v1 protocol and both commit and reconciliation methods. These define the local durable receipt and pending index/propagation boundary for future adapters. Current Engine.submit, its early STEAK callback and Knex storage do not use the capability. See the persistence v1 specification for the shared fixtures and explicit limits. No consumer migration is required.

Optional Mongo foundation

The package also contains an opt-in MongoDB foundation for schema bootstrap, content-addressed payload publication, reference guards, and payload collection. It is not an Engine integration, an AdmissionStorage implementation, or a default storage selection; importing @bsv/overlay does not load MongoDB.

Applications using a Mongo deep entry point install the optional peer first:

npm install @bsv/overlay mongodb@^7.5.0

The initial entry points are @bsv/overlay/storage/mongo/MongoSchema and @bsv/overlay/storage/mongo/MongoPayloadStore. They require an explicitly operated unsharded replica set; the supported deployment profile is three members. See the Mongo v1 foundation.

Common patterns

Implementing a TopicManager

import type { TopicManager } from '@bsv/overlay'
import { Transaction } from '@bsv/sdk'

class CustomTopicManager implements TopicManager {
  async identifyAdmissibleOutputs(beef, previousCoins) {
    const tx = Transaction.fromBEEF(beef)
    return { outputsToAdmit: [0], coinsToRetain: [] }
  }

  async getDocumentation() {
    return 'Custom topic documentation'
  }

  async getMetaData() {
    return {
      name: 'custom',
      shortDescription: 'A custom topic'
    }
  }
}

Implementing a LookupService

import type { LookupService } from '@bsv/overlay'

class CustomLookupService implements LookupService {
  readonly admissionMode = 'locking-script' as const
  readonly spendNotificationMode = 'none' as const

  async outputAdmittedByTopic(payload) {
    if (payload.mode === 'locking-script') {
      // Index the output
    }
  }

  async lookup(question) {
    // Return a LookupFormula: outpoints that the Engine should hydrate.
    return []
  }

  async outputEvicted(txid, outputIndex) {
    // Remove the UTXO from any service-specific index.
  }

  async getDocumentation() {
    return 'Custom lookup service documentation'
  }

  async getMetaData() {
    return {
      name: 'custom lookup',
      shortDescription: 'A custom lookup service'
    }
  }
}

Configuring storage with Knex

const storage = new KnexStorage(knex)
// Run the standard KnexStorageMigrations before first use.

Maintaining BASM and unproven transactions

BASM-capable deployments need a chain tracker that can validate Merkle roots and resolve canonical block headers. The Engine exposes maintenance operations used by Overlay Express and monitor processes:

await engine.startBASMSync()

await engine.refreshUnprovenTransactionProofs({
  thresholdBlocks: 144,
  proofProvider: async txid => await lookupProof(txid)
})

await engine.maintainUnprovenTransactions({
  thresholdBlocks: 144,
  proofProvider: async txid => await lookupProof(txid)
})

maintainUnprovenTransactions first tries to prove old unproven rows, then evicts rows that remain unproven past the configured threshold. Provider-level terminal invalidation can also call evictAppliedTransaction so double-spent or invalid transactions stop appearing in lookup results immediately.

Key concepts

• TopicManager — Validates which outputs are admissible to the overlay based on protocol rules
• LookupService — Indexes and queries admitted UTXOs; notified on admission/spend/eviction
• AdmissionMode — Whether lookup service receives locking-script details ('locking-script') or whole transaction ('whole-tx')
• SpendNotificationMode — How lookup service is notified when a UTXO is spent ('none', 'txid', 'script', or 'whole-tx')
• Storage — Abstracted persistence layer; Knex implementation handles SQL migrations automatically
• GASP — Graph Aware Sync Protocol for inter-service synchronization
• BASM — Block-anchored synchronization model for proving topic state against canonical block headers
• Unproven lifecycle — Transactions can enter as unproven, be proved later by callbacks or proof providers, or be evicted after the configured age threshold

When to use this

• Building an indexing service for a specific category of transactions
• Implementing custom business logic for transaction admission
• Creating a queryable ledger of on-chain data
• Syncing overlay state with other nodes

When NOT to use this

• For simple transaction broadcast — use @bsv/teranode-listener
• For client-side token operations — use @bsv/sdk or @bsv/overlay-topics directly
• For non-blockchain applications — use traditional databases

Spec conformance

• Implements BSV Overlay protocol for UTXO tracking
• Supports SHIP (Service Host Interconnect Protocol) and SLAP (Service Lookup Availability Protocol) for peer discovery
• Integrates Graph Aware Sync Protocol (GASP) for historical synchronization with other overlay nodes
• Supports BRC-136 BASM primitives for topic anchors and chain reorg handling

SHIP tracker responses are untrusted discovery hints. The Engine authenticates the canonical identity-linked advertisement signature and binds its one-satoshi output, BEEF/TXID, and requested topic before using the endpoint for GASP; the connection still requires the normal public-HTTPS and DNS-pinning controls.

Common pitfalls

1. Topic vs Service naming — Topic managers are prefixed tm_*, lookup services ls_* by default in discovery
2. BEEF encoding required — All transactions must be submitted in BEEF format; raw hex will fail
3. Knex migrations — Custom storage implementations must handle schema creation; KnexStorage provides standard migrations
4. GASP sync context — SHIP/SLAP topics have special handling for peer discovery configuration
5. Chain validation — If chainTracker is 'scripts only', SPV proofs are not validated; unsafe for production
6. Unproven cleanup — Eviction should normally be refresh-before-evict so delayed proofs have a chance to land before state is removed.
7. Double spends — Provider-confirmed double spends should be treated as terminal and evicted from admitted overlay state immediately.

Related packages

• @bsv/overlay-express — HTTP server wrapper for Engine
• @bsv/overlay-topics — Pre-built topic managers and lookup services
• @bsv/overlay-discovery-services — SHIP/SLAP peer discovery
• @bsv/gasp — Graph Aware Sync Protocol

Reference

• API reference (TypeDoc)
• Source on GitHub
• npm