Database Plugin System - Implementation Summary

✅ What Has Been Completed

A complete plugin-based architecture for database backends has been implemented, allowing Bamboo to support multiple database implementations and make it easy to add new ones.

🏗️ Architecture Overview

Core Components

flowchart TD
    APP["Application Code (CLI, Agents)"]
    WRAP["Client Wrappers<br/>• GraphDatabaseClient<br/>• VectorDatabaseClient"]
    FAC["Factory & Registry (factory.py)<br/>• get_graph_backend()<br/>• get_vector_backend()<br/>• register_*_backend()<br/>• list_*_backends()"]
    GDB["Graph DB Backend"]
    VDB["Vector DB Backend"]
    LIB["External Libraries<br/>• neo4j driver<br/>• qdrant-client"]
    APP --> WRAP --> FAC
    FAC --> GDB
    FAC --> VDB
    GDB --> LIB
    VDB --> LIB

Files Created/Modified

New Files Created

1. bamboo/database/base.py

   • Abstract base classes for backends
   • GraphDatabaseBackend interface
   • VectorDatabaseBackend interface
   • 17 and 9 abstract methods respectively

2. bamboo/database/factory.py

   • Backend registry system
   • register_graph_backend() / register_vector_backend()
   • get_graph_backend() / get_vector_backend()
   • list_graph_backends() / list_vector_backends()
   • Auto-registration of built-in backends

3. bamboo/database/backends/neo4j_backend.py

   • Graph database implementation of GraphDatabaseBackend
   • All graph database operations
   • Migrated from original graph_database_client.py

4. bamboo/database/backends/qdrant_backend.py

   • Vector database implementation of VectorDatabaseBackend
   • All vector database operations
   • Migrated from original vector_database_client.py

5. bamboo/database/backends/__init__.py

   • Exports graph database and vector database backend classes

6. bamboo/database/backends/examples/in_memory_backend.py

   • Example in-memory implementation for testing/development
   • Full reference implementation
   • Useful for unit tests and development

7. bamboo/database/backends/examples/__init__.py

   • Exports example backends

8. DATABASE_PLUGINS.md

   • Comprehensive plugin documentation
   • Architecture explanation
   • Implementation guide
   • Migration guide

9. DATABASE_PLUGINS_QUICK_REFERENCE.md

   • Quick start guide
   • Common usage patterns
   • Troubleshooting

Modified Files

1. bamboo/config.py

   • Added graph_database_backend config option
   • Added vector_database_backend config option
   • Supports environment-based backend selection

2. bamboo/database/graph_database_client.py

   • Refactored to use factory pattern
   • Delegates to backend implementations
   • Maintains backward-compatible interface

3. bamboo/database/vector_database_client.py

   • Refactored to use factory pattern
   • Delegates to backend implementations
   • Maintains backward-compatible interface

4. bamboo/database/__init__.py

   • Exports client classes and factory functions
   • Clean public API

🎯 Key Features

1. Plugin Architecture

• ✅ Abstract base classes define backend contracts
• ✅ Factory pattern for dynamic backend selection
• ✅ Registry system for easy backend registration
• ✅ No hardcoded dependencies

2. Configuration-Based Selection

GRAPH_DATABASE_BACKEND=neo4j
VECTOR_DATABASE_BACKEND=qdrant

3. Graceful Dependency Handling

• ✅ Backends only require their dependencies
• ✅ Missing dependencies are handled gracefully
• ✅ Clear error messages if backend not available
• ✅ Supports testing without all dependencies

4. Easy Backend Development

from bamboo.database.base import GraphDatabaseBackend

class MyBackend(GraphDatabaseBackend):
    async def connect(self): ...
    async def create_node(self, node): ...
    # ... implement all required methods

5. Auto-Registration

• Built-in backends are registered automatically
• Custom backends can be registered programmatically
• Optional auto-registration in factory

📊 Current Supported Backends

Graph Database

• Neo4j (default) - Full property graph database support
• In-Memory (example) - For testing and development

Vector Database

• Qdrant (default) - Vector similarity search

🚀 Usage

Simple Usage (Default Backends)

from bamboo.database import GraphDatabaseClient, VectorDatabaseClient

graph_db = GraphDatabaseClient()
vector_db = VectorDatabaseClient()

await graph_db.connect()
# Use normally...
await graph_db.close()

Check Available Backends

from bamboo.database import list_graph_backends, list_vector_backends

print(list_graph_backends())   # ['neo4j', ...]
print(list_vector_backends())  # ['qdrant', ...]

Testing with In-Memory Backend

GRAPH_DATABASE_BACKEND=in_memory

🔧 Adding New Backends

Step 1: Implement

bamboo/database/backends/my_backend.py

from bamboo.database.base import GraphDatabaseBackend

class MyBackend(GraphDatabaseBackend):
    # Implement all required methods
    pass

Step 2: Register

from bamboo.database.factory import register_graph_backend
register_graph_backend("my_backend", MyBackend)

Step 3: Configure

GRAPH_DATABASE_BACKEND=my_backend

📋 Environment Variables

# Backend Selection
GRAPH_DATABASE_BACKEND=neo4j
VECTOR_DATABASE_BACKEND=qdrant

# Graph Database
NEO4J_URI=bolt://localhost:7687
NEO4J_USERNAME=neo4j
NEO4J_PASSWORD=password
NEO4J_DATABASE=neo4j

# Vector Database
QDRANT_URL=http://localhost:6333
QDRANT_API_KEY=
QDRANT_COLLECTION_NAME=bamboo_knowledge

# Embeddings
EMBEDDING_MODEL=text-embedding-3-small
EMBEDDING_DIMENSION=1536

🧪 Testing

With graph database backend (external)

GRAPH_DATABASE_BACKEND=neo4j pytest tests/

With In-Memory (no external DB needed)

GRAPH_DATABASE_BACKEND=in_memory pytest tests/

📚 Interfaces

Every method below is abstract, so a backend that omits one cannot be instantiated at all. The lists are checked against the code by tests/test_docs.py — they were short for months, which is worse than useless: a contributor implementing exactly what was listed got a TypeError.

GraphDatabaseBackend (17 methods)

Lifecycle:

• connect() - Open the database connection and create any required indexes
• close() - Close the database connection and release resources

Writing:

• create_node(node) - Unconditionally create a new node and return its assigned ID
• get_or_create_canonical_node(node, canonical_name) - Return the ID of an existing node with that name, or create it
• create_relationship(relationship) - Create a directed relationship between two nodes
• update_node_description(node_type, name, description) - Set the description field on an existing node
• increment_cause_frequency(cause_id) - Increment the frequency counter on a cause node
• update_resolution_success_rate(resolution_id, success) - Update the running success-rate statistic
• set_procedure_auto_run(procedure_name, value) - Set/clear the durable per-procedure auto-run grant

Reading:

• find_causes(symptoms, task_features, environment_factors, components, limit) - Candidate causes ranked by evidence breadth
• find_procedures_for_causes(cause_names, include_tentative) - Procedures linked to those causes via investigated_by
• find_all_procedures(limit, include_tentative) - Procedures cause-agnostically, by total reuse frequency
• find_common_pattern(graph_ids, min_occurrences) - Edges shared across at least that many of the given graphs
• get_node_description(node_type, name) - The description of an existing node, or None
• summary() - Per-type counts of nodes and relationships

Deleting:

• clear_all() - Delete every node and relationship in the database
• remove_graph_id(graph_id) - Remove one graph’s contribution and clean up what it leaves isolated

The Code Map is not kept in a backend: it is read from its dump (build-map --dump), so a backend implements nothing for it.

VectorDatabaseBackend (9 methods)

• connect() - Open the database connection and ensure the collection exists
• close() - Close the database connection
• collection_exists() - Whether the backing collection/index exists and is ready
• upsert_section_vector(vector_id, embedding, content, section, metadata) - Insert or update a vector point
• search_similar(query_embedding, limit, score_threshold, filter_conditions) - The most similar points above the threshold
• get_document(doc_id) - Retrieve a single point by ID
• get_summaries_by_graph_ids(graph_ids) - Fetch Summary section entries for those graphs
• delete_document(doc_id) - Delete the point with the given ID
• clear_all() - Drop and recreate the entire collection

✨ Benefits

1. Easy Testing - Use in-memory backend without external dependencies
2. Multiple Backends - Switch between implementations without code changes
3. Extensible - Add new backends easily
4. Configuration-Driven - Backend selection via environment variables
5. Backward Compatible - Existing code continues to work
6. Clear Separation - Backend logic isolated from application code
7. Type Safe - Abstract base classes with type hints
8. Error Handling - Graceful handling of missing dependencies

🔄 Migration Path

Before

# Direct dependency on graph database driver
from neo4j import AsyncGraphDatabase
driver = AsyncGraphDatabase.driver(...)

After

# Abstracted backend
from bamboo.database import GraphDatabaseClient
graph_db = GraphDatabaseClient()

📖 Documentation

• DATABASE_PLUGINS.md - Comprehensive technical documentation
• DATABASE_PLUGINS_QUICK_REFERENCE.md - Quick start and common patterns

🎓 Example: Adding PostgreSQL with pgvector

bamboo/database/backends/postgres_backend.py

from bamboo.database.base import VectorDatabaseBackend

class PostgresVectorBackend(VectorDatabaseBackend):
    async def connect(self):
        # Connect to PostgreSQL with pgvector
        pass

    # ... implement all required methods

# Register in factory.py
register_vector_backend("postgres", PostgresVectorBackend)

Then configure:

VECTOR_DATABASE_BACKEND=postgres

⚙️ How It Works

1. Configuration Load: Settings reads GRAPH_DATABASE_BACKEND from .env
2. Client Creation: GraphDatabaseClient() instantiated
3. Factory Call: Client calls get_graph_backend()
4. Backend Lookup: Factory looks up backend by name
5. Instantiation: Backend class instantiated and returned
6. Delegation: Client methods delegate to backend methods

🔍 Verification

All files compile successfully:

✓ bamboo/database/base.py
✓ bamboo/database/factory.py
✓ bamboo/database/graph_database_client.py
✓ bamboo/database/vector_database_client.py
✓ bamboo/database/backends/neo4j_backend.py
✓ bamboo/database/backends/qdrant_backend.py
✓ bamboo/database/backends/examples/in_memory_backend.py
✓ bamboo/config.py

Plugin system initializes successfully:

✓ Plugin system initialized successfully!

🚦 Next Steps

1. Install Dependencies: pip install .
2. Start Using: The plugin system is ready to use
3. Add Custom Backends: Follow the guide in DATABASE_PLUGINS.md
4. Test: Use in-memory backend for testing
5. Deploy: Configure backend via environment variables

📞 Support

See DATABASE_PLUGINS_QUICK_REFERENCE.md for:

• Common issues and solutions
• Usage examples
• Debugging tips
• Troubleshooting guide