OKSA is the model manager and hardware calibration engine of PUU OS. Taking baby steps on the web/management side. Not very flashy but at this point what is shown here is more for "power on testing" and debugging than anything else.
Likely stuff to land soon here is mainly QSFP networking configuration for parallel inference.
Given that every build has an integer number that identifies that build *uniquely* as builds are *reproducible* I also likely find a lean way to use this information to notify the user if the nodes co-operating in parallel inference have a version mismatch (i.e., non-enforcing warning).
This is because for optimal results in parallel inference it would be feasible if all nodes run exact same version of the same operating system, and with PUU OS it is possible to achieve in bit-by-bit level.
In Puu OS bootc containers and reproducible builds are not used because they are cool but instead what they bring on the table is one of the key features for implementing stable inference across a mesh of nodes.
This is fun area to innovate because not much of it is that well documented when it comes to inference hosting. And likely lot of it is also a trade secret for AI companies. So for things like hardware calibration there was no tutorial or AI assistance. I simply have to figure out how it could be done, fail, learn and retry. And still doing it :-)
It's amazing how far Buildroot can be streched.
#buildroot
Likely stuff to land soon here is mainly QSFP networking configuration for parallel inference.
Given that every build has an integer number that identifies that build *uniquely* as builds are *reproducible* I also likely find a lean way to use this information to notify the user if the nodes co-operating in parallel inference have a version mismatch (i.e., non-enforcing warning).
This is because for optimal results in parallel inference it would be feasible if all nodes run exact same version of the same operating system, and with PUU OS it is possible to achieve in bit-by-bit level.
In Puu OS bootc containers and reproducible builds are not used because they are cool but instead what they bring on the table is one of the key features for implementing stable inference across a mesh of nodes.
This is fun area to innovate because not much of it is that well documented when it comes to inference hosting. And likely lot of it is also a trade secret for AI companies. So for things like hardware calibration there was no tutorial or AI assistance. I simply have to figure out how it could be done, fail, learn and retry. And still doing it :-)
It's amazing how far Buildroot can be streched.
#buildroot
