Next-Level Web Applications with On-Device Generative AI: A Look at Google Chrome’s Built-In Gemini Nano LLM

Web development is on the brink of a significant transformation with Google Chrome Canary’s latest experimental feature, a new tool called the window.ai API, allowing websites to harness the power of on-device generative AI. With Google’s Gemini Nano AI model built into the browser, websites can offer smarter, more personalized experiences directly on the user’s device. Let’s dive into what this means and how you can use it to supercharge your web applications.

Meet Gemini Nano

Gemini Nano is a compact yet powerful AI model from Google. It is the same model used in some Google Pixel phones for offline AI features. Its small size (1.7 GB) and impressive capabilities make it perfect for on-device applications, ensuring users get the benefits of advanced AI without needing to be online.

What is the window.ai API?

Google Chrome Canary is known for bringing new and exciting features to developers, and the window.ai API is no exception. This API lets your website’s JavaScript code communicate directly with Gemini Nano, an advanced AI model running on the user’s device. This setup means all AI tasks happen locally, using the computer’s GPU, so no data is sent over the internet. This is a game-changer for privacy and offline functionality.

How to Use the window.ai API

Getting started with the window.ai API is straightforward. Here’s how you can integrate it into your website:

1. Check for Availability: First, make sure the window.ai API is available in the user’s browser:

   if (window.ai) { console.log("The window.ai API is available!");
   } else { console.log("The window.ai API is not available on this browser.");
   }
   

2. Send a Prompt to the AI: Once you know the API is available, you can start sending prompts to Gemini Nano and get responses. Here’s an example:

   async function getAIResponse(prompt) { if (window.ai) { try { const session = await ai.createTextSession({ temperature: 0, topK: 1 }); return await session.prompt(prompt); } catch (error) { console.error("Error generating text:", error); } }
   } // Example usage
   getAIResponse("Write a poem about the ocean.");
   

3. Use the AI Response: You can then use the AI’s response in various ways, such as displaying it to the user, updating your UI, or for more complex processing.

Some Exciting Use Cases

The possibilities with on-device generative AI are endless. Here are a few exciting ways you can use it:

Privacy-Focused Apps

Use local AI for sensitive applications like health, finance, or where strict data protection policies are required, ensuring complete privacy and data security.

Offline-First PWAs

Enhance chatbots and virtual assistants with more natural and responsive interactions, even when offline.

Enhanced User Inputs

Create smart user interfaces by integrating AI into the existing form components. Imagine a select box suggesting the most relevant options if a user types in an invalid option, rather than just displaying “Not found”.

// The user tries to type in "android developer" in a searchable select box to fill Occupation form, not knowing the granularity of the options await session.prompt(`which are the 3 closest phrases to "android" from the following phrases:
1. DevOps Engineer
2. SEO specialist
3. Dentist
4. Cashier
5. Mobile Developer
6. Web Developer
7. Carpenter
8. Desktop Developer`); // => ' The three closest phrases to "Android" from the given phrases are: 1. Mobile Developer 2. Web Developer'

Assisted Writing

Enhance text areas with AI writing assistance, allowing text summarization and rephrasing without sending user data to external servers.

await session.prompt(`Rephrase this sentence: Facebook suggested your profile, I looked at your profile and I found your story inspiring.`); // => ' Facebook displayed your profile to me, and upon reviewing it, I found your story to be incredibly inspiring.'

Accessibility and Improved Website Navigation

Create user interfaces that people with disabilities can use more comfortably. Future prompt boxes and text to speech capabilities can intelligently recommends the most relevant pages, sub-menus, and settings based on users’ intent. LLMs can easily understands different words that users might use to express the same idea.

const choices = ['Like post', 'Save post', 'Share post', 'Block user', 'Follow user']; await session.prompt(`Which choice is the most relevent to "I do not want to see any posts from this user again!":
${choices.map((choice, idx) => `${idx + 1}. ${choice}\n`)}
`) // => ' The most relevant choice to the command "I do not want to see any posts from this user again!" is **"Block user"**.'

await session.prompt(`Which choice is the most relevent to "I want to see future posts from this user!":
${choices.map((choice, idx) => `${idx + 1}. ${choice}\n`)}
`) // => ' The most relevant choice is **5: Follow user.**'

Improved Text Search

// When a "Find in page" fails to find an exact match, try to search for similar words! await session.prompt(`Generate 10 words similar to "battery runtime", in the context of "smart phones"`); // => '1. Battery life 2. Battery endurance 3. Battery longevity 4. Battery capacity 5. Battery duration 6. Battery power 7. Battery energy 8. Battery performance 9. Battery efficiency 10. Battery usage'

Chat With Your Data

Although the Gemini Nano is not the best model for factual reasoning, it can handle basic questions, for example allow users to ask questions about the data in a classic table.

await session.prompt(`Who is the tallest person in the following list:
1. Alice, Weight: 52 kg, Height: 169 cm.
2. Bob, Weight: 71 kg, Height: 174 cm.
3. Eve, Weight: 66 kg, Height: 172 cm.
`); // => 'The tallest person in the list is Bob, who weighs 71 kg and is 174 cm tall.'

Tips for Prompt Engineering

To get the best results from the AI, follow these prompt engineering tips:

1. Play with Parameters: For factual questions, set the temperature to 0, and set TopK to 1 for more predictable and accurate answers.
2. Use Line Breaks: Keep related data on the same line to maintain context.
3. Use Parentheses: Signal elaboration with parentheses to add clarity.
4. Main Question First: Put the main question at the beginning of the prompt, followed by related data.
5. Response Parsing: The model usually uses Markdown to format it’s response, use it to your benefit. For example, just take the bold words (inside double asterisks) if you’re looking for a few keywords, but the response is a long text.

Why On-Device AI is Awesome

1. Privacy and Security: All data stays on the user’s device, enhancing privacy and reducing the risk of data breaches.
2. Offline Functionality: Apps work seamlessly without an internet connection, providing uninterrupted service.
3. Better Performance: Using the device’s GPU leads to faster response times and a more responsive application.
4. Cost Savings: Less reliance on cloud-based services can lower your operational costs.

Things to Keep in Mind

While the window.ai API is fantastic, there are a few things to consider:

• Device Compatibility: Right now, this feature is only in Google Chrome Canary. Make sure to check compatibility with other browsers and older devices.
• Resource Use: Running AI models can be heavy on resources, especially on lower-end hardware.
• Model Limits: On-device models might not be as powerful as their cloud-based counterparts, so balance functionality accordingly.

Conclusion

The window.ai API in Google Chrome Canary is a groundbreaking tool that brings the power of on-device generative AI to web development. It enhances privacy, performance, and user experience by using local computation. As this feature moves from experimental to mainstream, now is the perfect time to explore its potential and create smarter, more responsive, and secure web applications.