Pub/Sub Backbone

To maximize modularity and flexibility, the backbone of TrustGraph is Apache Pulsar. Pulsar is a publish/subscribe (pub/sub) infrastructure. Modules can be added to the backbone by simply subscribing and publishing to data feeds (queues). These data feeds are described with a JSON schema.

tip

While TrustGraph has a built-in schema for the core data feeds, additional custom schemas can be added for custom modules.

Pulsar Workflows

• For knowledge processing flows, Pulsar accepts the output of a processing module and queues it in the data feed for input to the next subscribed module.
• For services such as text completion and embeddings, Pulsar provides a client/server model. A Pulsar queue is used as the input to the service. When processed, the output is then delivered to a separate queue where a client subscriber can request that output.

The entire architecture, the pub/sub backbone and set of modules, is bundled into a single Python package. A container image with the package installed can also run the entire architecture.