babel_fish – real time language translation

This is a submission for the Cloudflare AI Challenge.

What we built

Douglas Adams came up with the idea of the Babel fish in Hitchhiker’s Guide to the Galaxy. It’s a fish that, once you’ve stuck it in your ear, allows any language to be translated to you in real time.

babel_fish is a real time language translation app built with Cloudflare, Svelte, and multiple language models. The app allows you to share a room with your friends in different languages. All messages get translated to your chosen language in real time!

@alicelovescake and I thought this would be a perfect app for trying out the fancy new Cloudflare Workers AI!

Demo

Le code

Frontend

The Svelte application contains the frontend code as well as all of our API endpoints interacting with the language models.

Backend

Our web socket backend uses Cloudflare Workers and Durable Storage. It’s a simple logic engine that powers the chat rooms.

Journey

Tired: trying out one new technology during a hackathon
Wired: trying out tons of new technology during a hackathon

I’ve been admiring Cloudflare from afar for a while now. I love what they’ve been doing with Pages, Workers, and now AI Workers. This hackathon finally gave me a good excuse to play around with the platform!

Aside from all the fancy tooling Cloudflare provide, this was also my first time using Svelte (sorry Rich!). I actually really enjoyed the experience. I’d love to spend more time writing Svelte; I have a lot left to learn.

So… AI?

We used a ton of Cloudflare’s available AI models to build babel_fish.

• @cf/openai/whisper for voice-to-text
• @hf/thebloke/mistral-7b-instruct-v0.1-awq for language detection
• @cf/meta/m2m100-1.2b for language translation

Since a true Babel fish doesn’t produce text we also used ElevenLabs for text-to-speech.

Challenges

We only had a single day to work on our project. The tight timeline and all new tech made it an exciting 24 hours.

The biggest challenge of this project actually ended up being the backend audio processing. With the initial POC, user messages weren’t transcribed and sent over the wire until they were completely finished speaking. We were able to improve this by using AudioWorklets. Now we regularly check for gaps in a user’s message and periodically process and broadcast messages.

The rest was easy because Cloudflare and Svelte made it so. 😎

Thanks to @cloudflaredev and @thepracticaldev for putting on this hackathon. If you’re interested in this kind of thing, come work on experimental AI projects with me at Stripe!

<><