Flux training on Replicate

      Comments Off on Flux training on Replicate

That’s a photo of me. And at the same time it’s not, because this is an AI generated me. So is it me or it’s not? ?

Guess the answer is yes and no. However this post is not about philosophical questions, but about how to generate all kind of images based on specific training. The AI model used for this generation is Flux on Replicate, trained on a photos of me as a LоRA model. What are LоRA models? LoRA or Low ranking adaptation for LLMs are a lightweight way to create a specific training . They are relatively small models, that apply minor modifications to standard LLMs models in order to adapt them for specific tasks. LoRAs provide a quick and fast way to train a model on a given task, which can be generating photos for blog posts, presentations or well, yourself.

What I used to generate the image above is Flux by Black Forests Labs, which seems to be the most popular image generation model lately, but you can use another one like Stable Diffusion. Flux models are trained on 12 billion of parameters and are capable of generating very realistic photos. Replicate on other hand is a cloud platform, that allows you to run AI models and train these via easy and well documented way.

So to come back to the photo above, I’ve used 9 photos of myself and followed up the guide here – https://replicate.com/docs/get-started/fine-tune-with-flux . While you can do the same with the API and there is a Python and Node.js example provided, I chose to use this web option https://replicate.com/ostris/flux-dev-lora-trainer/train . In short what you need is 10ish photos to train the model on, preferable these should be from different angles, clear and simple. I didn’t do the best job selecting these, most of my photos were with sunglasses and wearing accessories, which makes the model to produce some debatable results, but still quite good. There are several more settings that you can use – training steps, auto captioning using Llava v1.5, link to upload the new LoRA to HuggingFace, etc. The most important part here is to select an unique TOC – trigger word which you will use to trigger the LoRA model in your prompts. When ready, click “Create Training” and wait a few minutes. When the training finishes successfully, you can find the new LoRA under Dashboard -> Recently pushed models. When you click on the model, you will be presented of a few options to try your new LoRA – web at the Replicate playground, Python, Node.js, HTTP or Cog(a tool for containerasing AI models). We will look again at the playground option. The prompt is up to you, but you have to be careful to include the trigger word for the LoRA model. There are a few interesting setting to play with:

  • image, to use as an image to image generation option
  • mask image
  • format
  • LoRA weight
  • Extra LoRA
  • Extra LoRA weight

To me the most interesting was to play with the lora_scale, which defines how much priority you want to add to the training and the extra_lora, which allows you to get a link to another LoRA model from HuggingFace to use in addition to yours. This is a great option to use when you want to create different style of images. For example I used a sketch LoRA for this one:

And a photo session one for this one:

While definitely not all the images are good, some have weird accessories, some have scratches where the model got confused what to do, etc, most of them are fine and generated in no time. The LoRA can be deployed on Replicate and used as API calls for your AI applications, presentations content, blog posts. Have fun generating your own images in all kind of styles and landscapes!

WordPress

WordPress is my passion and love! Started as a blog system it turns in a really powerful CMS to allow us create professional, functional and really good looking web sites. WordPress also offers us a great community to help us, to develop amazing plugins and themes and to create a new better versions with more and more features. I have worked with different frameworks based on WordPress, created and customized themes and plugins so if my experience can help - you are welcome!