Millions of daily users, hundreds of clients, one embeddable widget. How did we build it and what could possibly have gone wrong?