Enlarge / An example of AI-generated Cyberpunk 2077 LoRA rendered with Flux dev.
Last week, enthusiasts trying out the new Flux AI image synthesis model discovered that it was surprisingly good at rendering facsimiles of custom-trained typefaces. While much more efficient ways to display computer fonts have existed for decades, this new technique is useful for AI image enthusiasts. Because Flux can render accurate representations of text, users can now insert words rendered in custom fonts directly into their AI image generation.
Creating AI-replicated fonts isn't big news in itself, since the technology has existed since the 1980s (and in research since the 1970s) to accurately create smooth, computer-rendered fonts with custom shapes. But new techniques could mean that a particular font could appear in AI-generated images, like a photorealistic restaurant chalkboard menu or a printed business card held by a cyborg fox.
Shortly after the emergence of mainstream AI image synthesis models like Stable Diffusion in 2022, some people began to wonder, “How can I inject my own product, clothing item, character, or style into an AI-generated image?” One answer emerged in the form of LoRA (Low Rank Adaptation), a technique discovered in 2021 that allows users to extend the knowledge of AI-based models with custom-trained modular add-ons.
Cyberpunk 2077 LoRA example rendered with Flux dev.
Cyberpunk 2077 LoRA example rendered with Flux dev.
Cyberpunk 2077 LoRA example rendered with Flux dev.
Cyberpunk 2077 LoRA example rendered with Flux dev.
These modules, called LoRAs, allow image synthesis models to create new concepts that were not originally present (or well represented) in the training data of the underlying model. In practice, image synthesis enthusiasts use them to render unique styles (e.g., all of the chalk art) or subjects (e.g., a detailed image of Spider-Man). Each LoRA must be specially trained with examples provided by the user.
Until Flux, most AI image generators weren't very good at rendering accurate text in a scene. Ask Stable Diffusion 1.5 to render a sign that says “Cheese” and it will return gibberish. OpenAI's DALL-E 3, released last year, was the first mainstream model that handled text reasonably well. Flux still makes mistakes with words and characters. But it is the most capable AI model I've seen to render “in-world text” (as you might call it).
Because Flux is an open model that can be downloaded and tweaked, the past month marked the first time that training LoRA on typefaces might make sense, as AI enthusiast Vadim Fedenko (who did not respond to interview requests at press time) recently discovered. “I'm really impressed with the results. Flux is able to figure out what a character looks like in a particular style/font, allowing us to train Lora on specific fonts, typefaces, etc. We plan to do more training in the near future,” Fedenko wrote in a Reddit post.
The first Flux typeface, LoRA, a Y2K example.
Y2K LoRA example.
Y2K LoRA example.
For his first experiment, Fedenko chose a cheery, “Y2K”-style font reminiscent of fonts popular in the late 1990s and early 2000s, and published the resulting model on the Civitai platform on August 20. Two days later, a Civitai user by the name of “AggravatingScree7189” posted a second typeface, LoRA, which recreates a font similar to one featured in the Cyberpunk 2077 video game.
“The previous text was awful so I had no idea this was possible,” wrote Reddit user eggs-benedryl in response to Fedenko's post about the Y2K font. “I didn't know the Y2K journal was fake until I zoomed in on it,” another Reddit user wrote.
Is that too much?
Enlarge / Cyberpunk 2077 LoRA example rendered with Flux dev.
Admittedly, using a well-trained image synthesis neural network to render old fonts on a plain background may be overkill, and you probably wouldn't want to use it as a replacement for Adobe Illustrator when designing documents.
“This looks good, but reinventing the concept of fonts as a 300MB LoRA is a bit weird,” wrote one Reddit commenter in a thread about Cyberpunk 2077 fonts.
Generative AI is often criticized for its environmental impact, a legitimate concern in large cloud data centers. But Flux found that it could insert these fonts into an AI-generated scene in quantized (reduced size) form while running them locally on an RTX 3060 (the full development model can run on an RTX 3090), with a similar power consumption to playing a video game on the same PC. The same can be said about the creation of LoRA: the creators of the Cyberpunk 2077 font trained LoRA for three hours on a 3090 GPU.
The use of AI image generators also raises ethical questions, such as how they are trained on data collected without the consent of content owners. While the technology has been controversial among some artists, a large community of people use it every day and share their results online through social media platforms such as Reddit, which has led to new applications of the technology like this one.
At the time of writing, there are only two custom Flux typefaces, LoRA, but we are already hearing plans to create more as of writing. Although it is still early days, the technology behind creating the LoRA typefaces could become the basis if AI image synthesis becomes more widely adopted in the future. Adobe, which has its own image synthesis model, is probably taking notice.
