Skip to main content

Getting Started

TrustGraph is a fully containerized AI Engine. A full discussion of TrustGraph use cases can be found here.TrustGraph deploys in minutes locally with Docker, Podman, or Minikube. TrustGraph can also be deployed in Google Cloud with Kubernetes. There are currently 4 deployment options:

Container DepoloymentHost EnvironmentContainer Orchestration
Docker ComposeLocalDocker Engine
Podman ComposeLocalPodman Machine
MinikubeLocalKubernetes
KubernetesGoogle CloudKubernetes
tip

Before launching TrustGraph, be sure to have either the Docker Engine, Podman Machine, or Minikube installed and running on the host machine.

caution

TrustGraph has been tested on Linux and MacOS with Docker, Podman, and Minikube. Windows deployments have not been tested.

Install the TrustGraph CLI​

pip3 install trustgraph-cli==0.16.10
note

The TrustGraph CLI version must match the desired TrustGraph release version.

Full TrustGraph CLI documentation can be found here.

Configuration UI​

While TrustGraph is endlessly customizable through the YAML launch files, the Configuration UI can build a custom configuration in seconds that deploys with Docker, Podman, Minikube, or Google Cloud. There is a Configuration UI for the both the lastest and stable TrustGraph releases.

The Configuration UI has three sections:

  • Component Selection ✅: Choose from the available deployment platforms, LLMs, graph store, VectorDB, chunking algorithm, chunking parameters, and LLM parameters
  • Customization 🧰: Customize the prompts for the LLM system, Data Extraction Agents, and Agent Flow
  • Finish Deployment 🚀: Download the launch YAML files with deployment instructions

The Configuration UI will generate the YAML files in deploy.zip. Once deploy.zip has been downloaded and unzipped, launching TrustGraph is as simple as navigating to the deploy directory and running:

docker compose up -d
tip

Docker is the recommended container orchestration platform for first getting started with TrustGraph.

When finished, shutting down TrustGraph is as simple as:

docker compose down -v

TrustGraph Releases​

TrustGraph releases are available here. Download deploy.zip for the desired release version.

Release TypeRelease Version
Latest0.17.11
Stable0.16.10

TrustGraph Deployement Options​

Unzipping the deploy.zip will add the deploy directory with the following subdirectories:

  • docker-compose
  • minikube-k8s
  • gcp-k8s

Each directory contains the pre-built configuration files needed to launch TrustGraph.

TrustGraph Configuration Files​

Launching TrustGraph is a simple as running a single line of code. All the necessary configurations have been built into a set of YAML files. There is a YAML configuration file for each possible model deployment and graph store configuration.

Model DeploymentGraph StoreLaunch File
AWS Bedrock APICassandratg-bedrock-cassandra.yaml
AWS Bedrock APINeo4jtg-bedrock-neo4j.yaml
AzureAI APICassandratg-azure-cassandra.yaml
AzureAI APINeo4jtg-azure-neo4j.yaml
AzureOpenAI APICassandratg-azure-openai-cassandra.yaml
AzureOpenAI APINeo4jtg-azure-openai-neo4j.yaml
Anthropic APICassandratg-claude-cassandra.yaml
Anthropic APINeo4jtg-claude-neo4j.yaml
Cohere APICassandratg-cohere-cassandra.yaml
Cohere APINeo4jtg-cohere-neo4j.yaml
Google AI Studio APICassandratg-googleaistudio-cassandra.yaml
Google AI Studio APINeo4jtg-googleaistudio-neo4j.yaml
Llamafile APICassandratg-llamafile-cassandra.yaml
Llamafile APINeo4jtg-llamafile-neo4j.yaml
Ollama APICassandratg-ollama-cassandra.yaml
Ollama APINeo4jtg-ollama-neo4j.yaml
OpenAI APICassandratg-openai-cassandra.yaml
OpenAI APINeo4jtg-openai-neo4j.yaml
VertexAI APICassandratg-vertexai-cassandra.yaml
VertexAI APINeo4jtg-vertexai-neo4j.yaml

Model API Configuration​

caution

All tokens, paths, and authentication files must be set PRIOR to launching a YAML configuration file.

For Docker and Podman deployments, set the following parameters for your selected model deployment option prior to launch. Set parameters for only the model deployments you plan to use.

AWS Bedrock API​

export AWS_ID_KEY=<ID-KEY-HERE>
export AWS_SECRET_KEY=<TOKEN-GOES-HERE>
note

The current default model for AWS Bedrock is Mixtral8x7B in US-West-2.

AzureAI API​

export AZURE_ENDPOINT=<https://ENDPOINT.API.HOST.GOES.HERE/>
export AZURE_TOKEN=<TOKEN-GOES-HERE>

Azure OpenAI API​

note

The OpenAI service within AzureAI is similar to deploying a serverless model in Azure, but requires setting the API version and model name. Interestingly, AzureAI gives the user the ability to set the model name however they choose. Thus, the model name is set within AzureAI by the user.

export AZURE_ENDPOINT=<https://ENDPOINT.API.HOST.GOES.HERE/>
export AZURE_TOKEN=<TOKEN-GOES-HERE>
export API_VERSION=<API_VERSION-HERE>
export OPENAI_MODEL=<user-defined-model-name-here>

Anthropic API​

export CLAUDE_KEY=<TOKEN-GOES-HERE>
note

The current default model for Anthropic is Claude 3.5 Sonnet.

Cohere API​

export COHERE_KEY=<TOKEN-GOES-HERE>
note

The current default model for Cohere is Aya:8B.

Google AI Studio API​

export GOOGLE_AI_STUDIO_KEY=<TOKEN-GOES-HERE>
tip

Google is currently offering free usage of Gemini-1.5-Flash through Google AI Studio.

Llamafile API​

caution

The current Llamafile integration assumes you already have a Llamafile running on the host machine. Additional Llamafile orchestration is coming soon.

danger

Running TrustGraph and a Llamafile on a laptop can be tricky. Many laptops, especially MacBooks have only 8GB of memory. This is not enough memory to run both TrustGraph and most Llamafiles. Keep in mind laptops do not have the thermal management capabilities required for sustained heavy compute loads.

export LLAMAFILE_URL=<hostname>
note

The default host name for Llamafile is http://localhost:8080/v1. On MacOS, if running a Llamafile locally set LLAMAFILE_URL=http://host.docker.internal:8080/v1.

Pinecone API​

note

Unlike Qdrant and Milvus which are deployed locally with TrustGraph, Pinecone is accessed through an API. You will need your own Pinecone API key to use it as your VectorDB.

export PINECONE_API_KEY=<TOKEN-GOES-HERE>

Ollama API​

tip

The power of Ollama is the flexibility it provides in Language Model deployments. Being able to run LMs with Ollama enables fully secure AI TrustGraph pipelines that aren't relying on any external APIs. No data is leaving the host environment or network. More information on Ollama deployments can be found here.

note

The current default model for an Ollama deployment is Gemma2:9B.

danger

Running TrustGraph and Ollama on a laptop can be tricky. Many laptops, especially MacBooks have only 8GB of memory. This is not enough memory to run both TrustGraph and Ollama. Most SLMs, like Gemma2:9B or Llama3.1:8B require roughly 5GB of memory. Even if you do have enough memory to run the desired model with Ollama, note that laptops do not have the thermal management capabilities required for sustained heavy compute loads.

export OLLAMA_HOST=<hostname>
note

The default Ollama host name is http://localhost:11434. On MacOS, if running Ollama locally set OLLAMA_HOST=http://host.docker.internal:11434.

OpenAI API​

export OPENAI_TOKEN=<TOKEN-GOES-HERE>
note

The current default model for OpenAI is gpt-3.5-turbo.

VertexAI API​

mkdir -p vertexai
cp <json-credential-from-GCP> vertexai/private.json
note

The current default model for VertexAI is gemini-1.0-pro-001.

tip

If you're running SELinux on Linux you may need to set the permissions on the VertexAI directory so that the key file can be mounted on a Docker container using the following command:

chcon -Rt svirt_sandbox_file_t vertexai/
danger

Environment variables are currently not supported for Minikube deployments. The above values for a chosen model deployment must be set within the selected YAML file. These values can be found in the command list for the containers named text-completion and text-completion-rag. The Minikube configuration files are quite complex. It is recommened to do a search for the neccesary variable name and remember to update it for both containers.

Custom Parameters (Optional)​

The YAML configuration files are pre-built with default parameters. However, the power of TrustGraph is the ability to easily tweak these configurations to optimize performance. These parameters can be adjusted prior to launching TrustGraph within the selected YAML configuration file in the command list for the applicable container.

Chunking​

Extraction performance can vary signficantly with chunk size. The default chunk size is 2000 characters using a recursive method. Decreasing the chunk size may increase the amount of extracted graph edges at the cost of taking longer to complete the extraction process. The chunking method and sizes can be adjusted in the selected YAML file. In the selected YAML file, find the section for the chunker container. Under the command list, modify the follwing parameters:

Choose one of the two available chunking methods:

  • chunker-recursive
  • chunker-token

Adjust the chunk size:

- --chunk-size
- '<number-of-characters/tokens-per-chunk>'

Adjust the chunk overlap:

- --chunk-overlap
- '<number-of-characters/tokens-to-overlap-per-chunk>'

Model Selection and Parameters​

Any available model can be set for the selected model deployment. Most configurations allow adjusting some model parameters. For configurations with adjustable parameters, the temperature and max_output tokens can be set in the selected YAML file:

- -x
- '<max_model_output_tokens>'
- -t
- '<model_temperature>'
- -m
- '<model_name_in_deployment_option>'
tip

The default temperature in TrustGraph is set to 0.0. Even for models with long input contexts, the max output might only be 2048 (like some intances of Llama3.1). Make sure max_output is not set higher than allowed for a given model.

AWS Region (Bedrock Only)​

Model availability in AWS Bedrock is region specific. The desired AWS region can be set in the YAML configuration file with:

For US-West-2:

- -r
- 'us-west-2'

For US-East-1:

- -r
- 'us-east-1'
note

Other global AWS regions would be set the same way, but have not been tested with TrustGraph.

Mixed Model Deployments​

The pre-built YAML configuration files set the Naive Extraction model deployment the same as the GraphRAG model deployment. Prior to launching TrustGraph, the YAML configuration file can be modified to configure different models for those processes. For instance, extraction might require a more robust LLM, which is a one time process, while using a SLM for GraphRAG to enable cost and resource efficiencies. The available model modules are:

  • text-completion-azure
  • text-completion-azure-openai
  • text-completion-bedrock
  • text-completion-claude
  • text-completion-cohere
  • text-completion-googleaistudio
  • text-completion-llamafile
  • text-completion-ollama
  • text-completion-openai
  • text-completion-vertexai

The first line of the command list of the text-completion and text-completion-rag containers specifies the model deployment from the above options.

Launching TrustGraph​

Navigate to the subdirectory for your chosen deployment method. You will then select a YAML file based on your desired model deployment and graph store combination. Once you've selected your YAML configuration file, you will run:

Docker​

docker compose -f <config-file.yaml> up -d

Podman​

podman compose -f <config-file.yaml> up -d

Minikube​

kubectl apply -f <config-file.yaml>
danger

Support for Google Cloud and Kubernetes is still experimental. Please join the TrustGraph Discord Community for testing this deployment option.

Running TrustGraph​

While the trustgraph-cli downloads all the packaged needed to run TrustGraph, there are slight differences in commands depending on whether you deploy with a Docker or Kubernetes approach. Refer to the below guides for a step-by-step process for running TrustGraph once deployed.