What is Hivemind?
Hivemind is a distributed task processing framework designed for scalability, fault tolerance, and ease of use. It enables efficient distribution of computational workloads across multiple nodes, making it ideal for data processing, machine learning, and other compute-intensive tasks.
How does it work?
Hivemind uses a coordinator-worker architecture to distribute tasks across a cluster of nodes. Tasks are broken down into smaller units of work, distributed to available workers, and results are aggregated back to provide a unified output.
Key features include:
- Dynamic scaling: Automatically scales workers based on workload demands
- Fault tolerance: Handles node failures gracefully with task redistribution
- Task prioritization: Processes critical tasks first based on configurable priorities
- Resource awareness: Optimizes task allocation based on available system resources
- Pluggable backends: Supports various storage and messaging systems
- Simple API: Easy integration with existing applications
Hivemind Overview
Hivemind uses a coordinator-worker architecture for distributed task processing:
- Client Application: Submits tasks to the system
- Coordinator: Central component that manages task distribution
- Task Queue: Stores tasks waiting to be processed
- Worker Nodes: Process tasks in parallel
- Result Store: Collects and aggregates results
The coordinator distributes tasks to available workers, monitors their progress, and handles failures by redistributing tasks when necessary. Results are collected and returned to the client application.
Where can I get it?
Download Hivemind from GitHub and get started with it now!
If you have any problems or require assistance, please open a support issue here .
High-level architecture
Task Distribution Flow
Hivemind distributes tasks through a coordinator to worker nodes:
The Hivemind task distribution process follows these steps:
- Client submits a task to the Coordinator
- Coordinator validates and prioritizes the task
- Task is broken down into subtasks and placed in the Task Queue
- Workers pull subtasks from the queue
- Workers process subtasks and store results
- Coordinator monitors progress and handles failures
- Results are aggregated and returned to the client
If a worker fails, the Coordinator detects the failure through missed heartbeats and reassigns the subtask to another worker. The system can also dynamically scale by adding more workers when the task load is high.
System Components
The Hivemind framework consists of several key components working together:
Hivemind consists of several key components:
Client Library: Provides API for task submission and result retrieval
Coordinator:
- API Server: Handles client requests
- Task Manager: Manages task lifecycle
- Worker Manager: Tracks worker status
- Scheduler: Assigns tasks to workers
- Failure Detector: Identifies worker failures
- Load Balancer: Distributes workload evenly
Storage Layer:
- Task Store: Persists task information
- Result Store: Stores task results
- Metadata Store: Stores system metadata
Worker Node:
- Worker API: Communicates with coordinator
- Task Executor: Runs task code
- Resource Monitor: Tracks resource usage
- State Manager: Maintains worker state
Plugins: Extensible system for storage, authentication, and metrics
Getting Started
Installation
# Clone the repository
git clone https://github.com/ao/hivemind.git
cd hivemind
# Install dependencies
pip install -r requirements.txt
# Install the package
pip install -e .
Basic Usage
Starting a Coordinator
from hivemind import Coordinator
# Initialize and start the coordinator
coordinator = Coordinator(host="0.0.0.0", port=5000)
coordinator.start()
Starting Workers
from hivemind import Worker
# Connect to the coordinator
worker = Worker(coordinator_url="http://coordinator-host:5000")
worker.start()
Submitting Tasks
from hivemind import Client
# Connect to the coordinator
client = Client(coordinator_url="http://coordinator-host:5000")
# Define a task
def process_data(data):
# Process the data
return transformed_data
# Submit the task
task_id = client.submit_task(
function=process_data,
args=(input_data,),
priority=1
)
# Get the result
result = client.get_result(task_id)