444.6x Cheaper, 193.6x Faster: Putting a New AI Model to Work
A week in the life of Hiroka Koizumi, CEO of Gizin Inc.
| Back of the Line |
|
| Art: Sumi |
|
On September 15, out of nowhere, a new AI model called Jev appeared.
Its selling point, according to the official announcement, is that it is 444.6 times cheaper and 193.6 times faster than conventional LLMs. That alone sounds appealing, doesn't it? And on top of that, it's reasonably smart.
So how did it manage to be that cheap and that fast?
The AI we all know from chat apps and the like is what's called “generative AI”: LLMs (large language models) that generate text and other content. Jev, however, doesn't generate text.
Typical generative AI, like the kind in chat apps, generates characters to express its result even when all it's asked for is a judgment. For a short answer the load is small, but have it output several judgments or probabilities and generation keeps running for that much longer, which makes it heavy processing.
Jev returns several judgments in parallel, in formats specified in advance (choices or numbers), which cuts out the wait of generating text one piece at a time. That design, specialized for this kind of processing, is the foundation that keeps computation down and lets it be offered fast and cheap.
These days, when a model's version going up by 0.1 hardly makes a noticeable difference anymore, I was genuinely moved by the decision to specialize for one use, boldly strip away everything else, and gain performance that way.
Have Jev make the judgments, then have a lightweight LLM do the work based on the results, and you can expect things to be far faster and cheaper than having a single high-performance LLM do both the judging and the work.
This time, I'll walk you through how we tried to bring Jev into our AI organization: the use cases we put into practice at GIZIN.
The Flood of Robots That Put Our Machine in Danger
Last Monday, a disk space alert popped up. Just 14GB left. A critical situation. Memory, which we had a full 94GB of, was running short too. The cause: robots running 12 in parallel. The mass of robots had been producing a mass of worktrees and Unity junk. We had cleaned up only a few days before.
Thanks to the evolution of GIZIN FLEET, our system for orchestrating an AI organization, it now looks as if every request becomes a card and moves along kanban-style. That's where we let our guard down. It had become hard to see how much cost was going into redoing work behind the scenes.
The work itself kept moving along smoothly, so hitting the Start work button again and again was fun, and a huge number of jobs really were in progress. But behind that, when I looked at the breakdown of the work, a whopping 40% of it was redos.
Under the card system, a high-performance AI (Fable) first takes the order from a human and writes it down on a project card. The project card is divided into phases, and for each phase it sets out who (which AI) is in charge of what. That phase AI (Opus) writes the work order cards it hands to the worker AI (Terra), and this is where it got hard.
|
The rest is for paid subscribers.
What was so hard about the work order cards. Whether the questions and send-backs from worker AIs can be reduced before the job is even handed over. Where in GIZIN FLEET we've started building in Jev; how the customer-facing AI that used to line up disclaimers began stating two particular points and started passing human check on the first try; a progress report on testing local LLMs that run on our own machines; and practice with a four-panel comic the AI draws using coordinates. (About 2,300 characters in the original, 4 images.)
Subscribe to the paid edition
|
|
— Hiroka Koizumi, CEO, Gizin Inc.
| Back of the Line |
|
| Art: Sumi |
|
|