Distributed Graph Processing System
A full-stack system for distributed graph algorithm execution with WebAssembly-accelerated visualization.
Abstract
This system combines a Rust-based MPI backend with a WebGL + WebAssembly frontend. It accepts user-uploaded graph files, partitions and distributes algorithm execution across multiple compute nodes using MPI (Message Passing Interface), and visualizes results in a 3D/2D interactive canvas.
A key design goal was to move computation as close to the data as possible: heavy algorithmic work runs in Rust on the backend cluster, while graph parsing and rendering pre-computation run in a Rust-compiled WebAssembly module in the browser, keeping the JavaScript main thread free for UI interaction.
Why This System?
Graph algorithms are foundational to a wide range of domains — social network analysis, logistics routing, compiler dependency resolution, and distributed systems topology. As graph datasets grow in scale (millions of nodes, billions of edges), single-machine processing becomes a bottleneck.
This system bridges that gap by providing:
- A distributed MPI backend written in Rust that executes nine graph algorithms across a cluster of Docker containers, each running as an MPI rank.
- A REST API server (Rocket framework) on the master node that accepts graph files and algorithm requests via HTTP.
- A React frontend served from nginx that renders graphs in WebGL via Reagraph (Three.js), with algorithm results animated step-by-step.
- A WebAssembly computation layer compiled from Rust that handles graph parsing, adjacency map construction, node color computation, path set building, and edge styling — all inside the browser at near-native speed.
- A Web Worker that runs all WASM computation off the main thread, keeping the UI responsive even for graphs with thousands of nodes and edges.
Tech Stack at a Glance
| Layer | Technology |
|---|---|
| Backend algorithms | Rust + MPI (OpenMPI) |
| HTTP server | Rocket (Rust) |
| Frontend framework | React 19 + Vite |
| Graph renderer | Reagraph (Three.js / WebGL) |
| Browser computation | WebAssembly (Rust via wasm-bindgen) |
| CSS utility | UnoCSS |
| Containerization | Docker Compose |
| Web server | nginx |
Next Steps
- Installation — Clone the repo and get it running
- Architecture Overview — Understand the full system design
- Algorithms — Browse all nine supported algorithms
- REST API Reference — API endpoints and request/response formats