In my experience, trying to switch VFX companies from CPU-based rendering to GPU-based rendering 10+ years ago, a 2-5x performance improvement wasn't enough. We even provided a compatible renderer that accepted Renderman files and generated matching images. Given the rate of improvement of standard hardware (CPUs in our case, and GPU-based inference in yours), a 2-5x improvement will only last a few years, and the effort to get there is large (even larger in your case). Plus, I doubt you'll be able to get your HW everywhere (i.e. mobile) where inference is important, which means they'll need to support their existing and your new SW stack. The other issue is entirely non-technical, and may be an even bigger blocker -- switching the infrastructure of a major LLM provider to a new upstart is just plain risky. If you do a fantastic job, though, you should get aquahired, probably with a small individual bonus, not enough to pay off your investors.
We're targeting the edge market first, such as NVIDIA's Jetson line, because it's far less supported/focussed on. In our experience, whenever we did training runs on H100 clusters with x86, any pip package would be easily installable, and a wide array of software just worked. This is not the case in Jetson, where we constantly have to rebuild packages from source, and in general, NVIDIA will only release a better board every five years. As for the second part of your question, we agree. Much of our work has been trying to make switching to our software layer straightforward (a single line of code). The ideal endgame is that, given an ONNX file, we can parse the generated node tree and determine if our hardware supports all the nodes. Of course, this is assuming we have a large enough share of the market using our software, so we know what operations we need to support on the hardware side of things.
I cannot see any way of building HW profitably for the Jetson market. You are really competing with Raspberry PI, not Jetson, IMO. I mean, I'm no expert, but I would suggest doing a deep dive on your business plan if you intend to target the small hardware world rather than spending any time designing HW or SW. Then reduce your estimate by at least half since doing anything in that embedded/edge world has many more technical issues.
In general, Jetson has quite a large market. Vehicle companies use automotive-rated Jetson Orins, and defense companies also use Jetson Orins to power ML applications on the edge (Anduril). Many of the companies we currently talk to are robotics companies that are forced to use Jetsons because they are both the least of the bad options and the only edge compute provider with enough juice to run larger transformer models.
And the auto and Defense markets are so easy to enter! /s
Both of these markets have long lead times, tight HW build times, and move incredibly slowly. They are not the kind of markets that like using stuff from new companies with no history. Again, I'm no expert, but I'd say you need to be concentrating on sales and market research now.
With respect it doesn't sound like you know much about any of these businesses. This startup is extremely early, the road to silicon is long, and there is a lot of external change and learning by doing that will happen between here and there. This is them getting started and based on my related work experience I think it's pretty interesting.
We are not under the illusion these markets are easy to enter. Still, we believe providing an effortless and compatible experience for edge ML computing is a strong competitive advantage. We have not met anyone who likes using Jetsons yet, unlike A100/H100s in the server market.
Edit: I should note that if it weren't for Dusty and his docker image generating GitHub repo for Jetson, we would have spent weeks trying to get our kernels and optimized models shipped to customers.
What's your point? Is it that one shouldn't attempt to enter a market just because it's difficult? Or are you trying to educate the founders about something obvious that they likely have already spent 1000x more time thinking about than you?
I think the only way this could work is if you had the backing of one of the major LLM providers who decided that your ideas are worth doing a PoC. That way you actually have a client on board before you spend all the money. I know you guys probably like the designing of the HW and SW, and maybe the implementation of both, but really, what you need now is to do sales.
There are multiple ways to run a business like this.
1. Go deep on the tech, there are funders who will want equity stakes in risky startups because they operate in adjacent markets. It's often cheaper to invest 1MM on a startup than internal R&D activities. If it has promising results, those same investors may ramp up their spend or pivot to an acquisition strategy.
2. Get early customers, if you have 1-10 large enterprises with a committed spend - then you are likely golden. However as nice as this option sounds, there are few avenues to get this type of commitment. If you are in the fortunate position of knowing the exec/founding/investor team of a large LLM provider - it's possible. But easier said than done.
3. Build it and they will come, business strategies take time to develop - maybe that time is poorly spent. Build the best version of your product and someone might take it up. There are a few investors who will take a flyer on this type of founder mentality. Benefit to the investor is that they can get a much larger equity stake/board position in exchange for the early creative freedom. If it works out, the investor can get a lot of alpha. A card which handled LLM inference at 1/100th the cost of an H100 could produce quite a bit of value for the right buyer.
4. Do the technical work to get it a little bit beyond just an idea and then get acqui-hired by a large company who has the resources to push this.
So if I was them I would be doing thought experiments on how this technology could benefit a whole range of businesses e.g. gaming consoles, televisions etc. Not many people would've guessed LG acquiring Palm for example.
I'm currently working on a portable computer vision project using Pi/Jetson with some Luxonis camera modules and I completely see where you're headed. In the long-game I think you could capture hw accelerated robotics CV.
Why not target the enthusiast first? The buzz created around something interesting an "amateur" cooked up may be what you need. The investment involved with creating dev hardware should be minimal, correct?
I may be wrong, but from few other enthusiast niches I conclude, enthusiasts number is very little to feed hardware development.
- Need millions sells, but really most real project have made thousands sells.
And this is long known - even Raspberry born for other market, fortunately, was not just killed but conversed to target enthusiast and even now incomplete project.
Having been working in DL inference for now 7+ years (5 of which at startup) which makes me comparably ancient in the AI world at this point. The performance rat race/treadmill is never ending, and to your point a large (i.e 2x+) performance improvement is not enough of a "painkiller" for customers unless there is something that is impossible for them to achieve without your technology.
The second problem is distribution: it is already hard enough to obtain good enough distribution with software, let alone software + hardware combinations. Even large silicon companies have struggled to get their HW into products across the world. Part of this is due to the actual purchase dynamics and cycle of people who buy chips, many design products and commit to N year production cycles of products built on certain hardware SKUs, meaning you have to both land large deals, and have opportune timing to catch them when they are evening shopping for a new platform. Furthermore the people with existing distribution i.e the Apple, Google, Nvidia, Intel, AMD, Qualcomms of the world already have distribution and their own offerings in this space and will not partner/buy from you.
My framing (which has remained unchanged since 2018) is that for silicon platform to win you have to beat the incumbents (i.e Nvidia) on the 3Ps: Price (really TCO), Performance, and Programmability.
Most hardware accelerators may win on one, but even then it is often theoretical performance because it assumes their existing software can/will work on your chip, which it often doesn't (see AMD and friends).
There are many other threats that come in this form, for example if you have a fixed function accelerator and some part of the model code has to run on CPU the memory traffic/synchronization can completely negate any performance improvements you might offer.
Even many of the existing silicon startups have been struggling with this since them middle of the last decade, the only thing that saved them is the consolidation to Transformers but it is very easy for a new model architecture to come out and require everyone to rework what they have built. This need for flexibility is what has given rise to the design ethos around GPGPU as flexibility in a changing world is a requirement not just a nice to have.
Best of luck, but these things are worth thinking deeply about as when we started in this market we were already aware of many of these things but their importance and gravity in the AI market have only become more important, not less :)
We've spent a lot of time thinking about these things, in particular, the 3Ps.
Part of making the one line of code work is addressing programmability. If you're on Jetson, we should load the CUDA kernels for Jetson's. If you're using a CPU, we should load the CPU kernels. CPU with AVX512, load the appropriate kernels with AVX512 instruction, etc.
The end goal is that when we introduce our custom silicon, one line of code should make it far easier to bring customers over from Jetson/any other platform because we handle loading the correct backend for them.
We know this will be bordering impossible, but it's critical to ensure we take on that burden rather than shifting it to the ML engineer.
Why start a company to make this product? Why not go work at one of the existing chip manufacturers? You'd learn a ton, get to design and work on HW and/or SW, and not have to do the million other things required to start a company.
We were waiting for a Bitnet-based software and hardware stack, particularly from Microsoft, but it never did. We were essentially nerd-sniped into working on this problem, then we realized it was also monetizable.
On a side note, I deeply looked into every company in the space and was thoroughly unimpressed with how little they cared about the software stack to make their hardware seamlessly work. So, even if I did go to work at some other hardware company, I doubt a lot of customers would utilize the hardware.
I recommend getting a job at NVIDIA. They care deeply about SW. It is a great place to learn about HW and the supporting SW. There is much to learn. Maybe you will learn why you are unimpressed with their SW offerings. For me, the hard part was the long lead time (8+ years) from design to customers using the product. One of the things that always amazed me about NVIDIA was that so many of the senior architects, who have no financial need to keep working (true for more than a decade), are still working there because they need the company to do what they love.
I think there is a comment somewhere here where I comment on NVIDIA, but I think NVIDIA is the best hardware company for making good software. We had a very niche software issue for which NVIDIA maintained open-source repos. I don't think NVIDIA's main advantage is its hardware, though; I think it's the software and the flexibility it brings to its hardware.
Suppose that Transformers die tomorrow, and Mamba becomes all the rage. The released Mamba code already has CUDA kernels for inference and training. Any of the CSPs or other NVIDIA GPU users can switch their entire software stack to train and inference Mamba models. Meanwhile, we'll be completely dead in the water with similar companies that made the same bet, like Etched.
You said (implied?) that your reason for starting a company was that you were waiting for somebody (MS) to build your favorite tech, and you realized it was monetizable. Finding a gap is a great start. But, if money is your goal, it is far easier to make money working at a company than starting one. Existing companies are great places to learn about technology, business, and the issues that should really drive your desire to start something yourself.
I don't think I ever implied we started this for money. We started working on the technology because it was exciting and enabled us to run LLMs locally. We wouldn't have started this company if someone else came along and did it, but we waited a month or two and didn't see anyone making progress. It just so happens that hardware is capital intensive, so making hardware means you need access to a lot of capital through grants (which Dartmouth didn't have for chip hardware) or venture capital (which we're going for now). I'm not sure where you got the idea we're doing this solely for money when I explicitly said "We were essentially nerd-sniped into working on this problem"
Glad to hear money isn't your focus. Your comment "...then we realized it was also monetizable" was the reason for my interpretation. Its also a very common rational. I don't know what "nerd-sniped" means, so...
Good luck with the VCs. I hope you all stay friends through the challenging process.
When performing performance optimization on CPUs, I was impressed with Intel's suite of tools (like VTUNE). NVIDIA has some unbelievable tools, like Nsys and, of course, its container registry (NGC), which I think surpasses even Intel's software support.
Is GPU rendering used today for VFX? From a quick google it seems that yes GPU based rendering is definitely an option, even if there's various reasons to still prefer CPU. So in your case was it really what you were aiming to do was pointless or simply your particular solution failed to succeed?
You're right that as a small player it's very hard to gain traction, even if the tech is fantastic because it's risky to switch your tech stack over. Though if you do do a good job with the tech I'd say you have a decent chance of an acquisition from a bigger player who wants a ready-made (or 90% of the way there) solution they can make their own. Perhaps you can call this an aquihire but I think you're significantly underplaying the potential upside of this exit. Imagine this startup is seen as having a great ternary transformer solution and ternary transformers are the way to go you could get multiple large players eyeing up an acquisition to get ahead pushing the price up.
My feeling is custom ASICs for ternary transformers is a great area to look at. There is a genuine chance of providing a significant step up from GPUs in terms of power efficiency and potentially performance. Plenty of risk of course, ternary models might just not perform as well as the full fat equivalents and building custom silicon, especially as a start-up, comes with all kinds of issues.
Yes by small studios with the agility to change their workflow without too much friction, and whose projects are small enough to fit into the constraints of GPU renderers, but largely not by huge studios who already have in-house CPU farms and whose projects need hundreds of gigs of RAM to render anyway.
