Power is the main reason to do custom ASICs. I’d be curious as to your opinion of Recogni given they are claiming a 10x power reduction per unit compute.
Unfortunately, I've worked at several players which promise power reductions. It doesn't matter though. People don't care about cost at this point. If you are cost-sensitive you're not doing the kind of revolutionary AI work these companies need to create a competitive moat. And once your model works on NVIDIA and is trained, how much are you going to spend on ML engineers to make it work on something else? Because that cost better be less than the marginal cost reduction on electricity. Plus, NVIDIA et al will likely get more and more efficient.
The only exception is running things on mobile. There is demand for porting models to run natively on mobile, and somewhat reasonable support for doing this.
