Playing with Gradio - it.ivdimova.com

With the sharp growth in AI during the last few years, everyone seems to build models and apps based on LLMs. My work also has been related to playing with Python and AI in the past year. In relation to this I watch a lot of training courses and videos and try to play with some learning in my free time. My last inspiration is Gradio – Python library, often used as an easy way to create demos for ML.

Let’s go through creating a simple Gradio app and deploying to share it.

Gradio provides a quick way to create demos with input and output fields that could be media, audio, text, etc.

import gradio as gr

def greet(name, intensity):
    return "Hello, " + name + "!" * int(intensity)

demo = gr.Interface(
    fn=greet,
    inputs=["text"],
    outputs=["text"],
)

demo.launch()

This is a quick example of a text input and output in Gradio. Setting up

demo.launch(share=True)

will generate proxy/tunnel through the Gradio servers and while the app will still run on your machine, you will get a link, available to share for 72hours.

Now what’s left is to build the actual functionality in the Gradio app. First we need to pick a model for this app, which is quite overwhelming with the number of available models lately. As probably the biggest area for open source models, Hugging Face is the natural place to look at. Choose the model menu and select from the criteria in the left hand panel – my choice for this example is simple – text2text generation, open source license. Then sort the results by trending or by most downloaded and check the model cards. I needed something, that I can run locally, but still gives good results, so I choose https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0

After choosing the right model, click on the <Use in Transformers> link on the right, to see an example how to load it. Here is my very basic example for the purposes of this blog post:

def travel(country):
   messages = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who loves to help people find the perfect vacation spot. You can answer questions about popular vacation spots in different countries.",
    },
    {"role": "user", "content": f"Where can I go on vacation from {country}?"},
	]
   prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
   outputs = pipe(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)

   output_text = outputs[0]['generated_text']
   assistant_message_start = output_text.find('<|assistant|>') + len('<|assistant|>')
   assistant_message = output_text[assistant_message_start:].strip()

   return assistant_message

And now all we need to do is to load this function in the Gradio app.

import gradio as gr

demo = gr.Interface(
    fn=travel,
    inputs=["text"],
    outputs=["text"],
)

demo.launch()

And here, you have a simple app, that you can ask about travel destination ideas.

If you want to get a permanent demo for sharing with the external world, Hugging Face Spaces are a great option. Just select a free space (you have to be very mindful about the model size tho as the free option is quite limited in resources), clone the repo locally, add your app files and deploy to Spaces.

And just like that, we have our own LLM powered app. What are you building with AI these days and what’s your favourite playground? Let me know in the comments.