McKinsey's Lilli: A Wake-Up Call for AI Startups

McKinsey's GenAI bet spells potential trouble for swaths of new B2B AI startups

Welcome to Fully Distributed, a newsletter about AI, crypto, and other cutting-edge technology. Join our growing community by subscribing here:

---

McKinsey just unveiled its new internal generative AI tool “Lilli” - a chat application trained on more than 100,000 proprietary internal documents and interview transcripts. Lilli (even in its current beta) has been a huge success internally - both in terms of impact and adoption:

• The tool has already dramatically cut down the time spent on research and planning from weeks to mere hours, and from hours to just minutes in others

• An astounding 66% of employees now use the app multiple times a week

• In the last 2 weeks alone, Lilli answered a whopping 50,000 questions!

Lilli’s success could set a precedent for other Fortune 500 giants grappling with the perennial “buy versus build” question. This should worry builders in the “B2B AI SaaS” startup world, and prompt them to seriously reconsider their product strategy and approach.

Let me tell you why.

Most AI startups today are features, not companies

Lilli can do exactly what myriads of emerging GenAI startups promise to do:

• Semantic search - The tool can zero in on relevant internal documents, files, and even pinpoint internal expert recommendations based on the user’s query.

• Retrieval Augmented Generation - It can respond to questions leveraging McKinsey’s exclusive corpus of presentations, research, and transcripts, complete with references and sources.

• Document analysis - processes and analyzes uploaded client files and documents (this feature will be deployed later)

• Gen AI Chat - Just like a chatbot using a general-purpose LLM (akin to ChatGPT), it can summarize articles, produce text, and even draft emails.

Twitter is inundated with hundreds of demos of products that individually offer one of the above use cases. The fact that Lilli can reliably do all of them confirms that these are more likely to be merely features and not standalone products (or let alone companies).

Building/Partnering makes more sense than buying

Every company working through its “AI strategy” is debating whether it should buy a third-party solution or build in-house. The main criteria are

1. Strategic priority - How pivotal is this to the core of the business?

   By now there is an overwhelming agreement that knowledge workers will benefit greatly from Gen AI and will experience significant productivity gains (at least 20-30%). This makes the technology indispensable for firms with a large knowledge worker base.

2. Urgency - how quickly do I need to have a production-ready solution?

   Given the profound impact of GenAI tools, lagging behind might render a company less competitive, making this an urgent priority. However, given the wide availability of LLM APIs and open-source LLMs, building or partnering is becoming increasingly attractive.

One other critical concern for large corporations is that of regulation, compliance, and security - from safeguarding confidential customer data to accessing employee PII, trusting an untested nascent startup might prove challenging to many.

McKinsey concluded that a combination of cost, reliability, and security warranted an in-house solution - despite the higher “tech lift” required. Indeed, McKinsey chose to build its own app on top of other LLM technologies:

• Cohere (most likely using their Embeddings and Semantic Search products); and

• OpenAI Microsoft Azure service (most likely for the “enterprise-friendly” secure GPT-4 instance).

That in itself may indicate that the tech lift of building LLM wrappers is not that high in the first place - both Cohere and OpenAI likely offered solutions architects to assist with the implementation, and the robust open-source community likely offered McKinsey developers lots of frameworks to use as inspiration.

Building in-house has another benefit - Lilli is built *exactly* the way McKinsey wants it (does not have to “settle” for an off-the-shelf all-size-fits-all solution), allowing Lilli to seamlessly “plug in” into the existing tools and workflows of the company.

There is somewhat of a self-reinforcing feedback loop here - as more companies elect to build instead of buying, the lower the perceived tech lift will be, and the more other companies will choose to build as well. This puts startups in a precarious situation, where they need to justify to enterprise customers that their tool is worth the tradeoffs involved - something that might prove to be very challenging.

LLM agnosticism

McKinsey's declaration of being "LLM agnostic" is telling. To them, the LLM is a mere means to a desired end: achieving business goals. With Lilli's in-house build, they command the flexibility to swap out underlying technologies as deemed fit.

In most cases, startups will have to offer similar flexibility in terms of the underlying technology to win over enterprises - something that will likely require a higher engineering lift and will involve continuous testing, QA, etc over multiple ever-so-evolving LLMs available on the market. This might be challenging for smaller startups with leaner technical teams.

What This Means For B2B AI Startups

The early success of McKinsey’s Lilli is promising - it means that the hype around generative AI is not unwarranted - the technology will most certainly transform the existing workflows of most knowledge workers and will supercharge their productivity. It is no surprise that so many entrepreneurs are interested in the space.

However, the fact that a professional services firm like McKinsey was able to build and deploy a working product in mere months puts into question the merits of buying/licensing third-party “core” solutions from up-and-coming AI startups.

In my view, founders should seriously reconsider their unique selling points and value proposition if they are going down the “enterprise ChatGPT”, “chat with your data”, or “AI search” route. Does their startup have the “right to win” in these areas?

To close, here are a few examples of interesting areas to explore in B2B land:

1. Focus on services around LLM applications - things like custom model training, evaluations, QA, optimizations, etc. - these things are “hard” that large organizations might need help with as they look to deploy their own applications. Humanloop has been able to onboard lots of F500 customers this way.

2. In the spirit of doing the “hard thing” - offering solutions that require deep technical expertise and proprietary technology to build might also entice businesses to buy from you. A good example here is Synthesia that allows users to create video content with human-like AI avatars in seconds.

3. Focus on "non-core” business functions that enterprises would be much more willing to “outsource”. One example is business processes that have historically been outsourced to humans in lower-cost jurisdictions (claims processing, data entry, customer support, etc.). Companies like Intercom have done well in this area. Alternatively, for lean highly leveraged companies like private equity and hedge funds, companies like Hebbia offer an off-the-shelf solution that the investment firms would never bother building on their own.

This is by no means an exhaustive list. There will be more than a few successful B2B AI-native companies built in the coming years. But the McKinsey case study shows us that aside from the powerful incumbents, startups will also have to compete with the in-house developer teams of their target customers.

As always, all thoughts / pushback welcome. Twitter DMs always open @leveredvlad

PS If you are tinkering in AI applications to the financial workflows, please reach out - I love jamming about this stuff and would love to compare notes.

---

If you enjoyed reading this, subscribe to my newsletter! I regularly write essays about AI, crypto, and other cutting-edge technology.