r/MachineLearning • u/Important-Home4431 • 1d ago
Project Deepity: A C++ library showing Predictive Coding Networks can match Backprop (97.73% on MNIST in 60s) [P]
I've spent the last month building a local C++ machine learning library called Deepity to test alternative credit assignment algorithms; specifically Predictive Coding Networks (PCNs). While PCNs are fascinating for biological plausibility and continual learning, naive implementations are painfully slow.
By implementing recent research (Accelerated PCNs via Direct Kolen-Pollack Feedback Alignment) and utilizing algorithmic caching to bypass redundant forward projections during the inference settling phase, I managed to close the performance gap with backpropagation on my CPU when training on MNIST (50 epochs).
- PyTorch Backprop (Feedforward): 98.27% test accuracy in ~70s.
- Deepity DKPPCN: 97.73% test accuracy in 59.5s.
Next up is porting these kernels to CUDA to scale up the architecture and testing its capabilities in continual learning scenarios where standard backprop struggles.
If you are interested in local learning, alternative credit assignment, or HPC for ML, I'd love your feedback!
- GitHub: https://github.com/ra4ster/deepity
- Project Site: https://ra4ster.github.io/Deepity
10
u/gkbrk 1d ago
Was the PyTorch version trained on a CPU? MNIST reaches these accuracies in 5 seconds usually.
9
6
u/Important-Home4431 1d ago
Yep, both were CPU-only. The reason it usually takes 5 seconds is because of offloading to the GPU or running for ~10 epochs. To get a fair hardware benchmark against my C++ engine, I forced both to run for 50.
I actually just tightened the experiment by giving PyTorch Adam as well, which brought the training time down ~12 seconds, meaning PyTorch’s ATen is still outperforming overall, but since PCNs require iterative settling for each input batch, getting a local learning rule this close to PyTorch’s speed is still pretty satisfying imo.
13
u/wizard_of_menlo_park 1d ago
MNIST is too simple of a benchmark .
5
u/RegisteredJustToSay 18h ago
Isn't it amazing how we now consider it trivial enough to even accidentally obtain good scores on, when it was a legitimately tough dataset at one point in time?
4
2
u/Disastrous_Room_927 1d ago edited 1d ago
Keep at it. I’ve been working on a non-backprop approach that exceeded backprop on MNIST and CIFAR (it hit the same reconstruction loss way faster), but the real work began when I tried to build a language model. I feel like somebody is going to crack egg sooner or later.
-2
u/Thelastreddditor 1d ago
"Deepity: A C++ library showing Predictive Coding" - practical and clear. Bookmarking the idea.
40
u/IntelArtiGen 1d ago
Competing with backprop on ILSVRC (the full version) would be a substantial next step I'd say. Many algorithms managed to compete with backprop on MNIST, but never went much further.