I recently spent some time traveling in China. I was particularly keen to understand the state of Chinese high-end technology. Trade shows like the Canton Fair are full of robots you can buy. But which of them have actually escaped the showroom and entered daily use?

I went to Shenzhen, the Silicon Valley of electronics. There, I saw several robots that we have grown accustomed to in the West in recent years, but I was also surprised by several that you don’t usually see there.

What stood out was that the successful robots were not necessarily the most advanced-looking ones. They were the ones embedded into the surrounding infrastructure: elevators, payments, hotel systems, transport networks, and cars. At the same time, I kept wondering whether this might become the deciding factor in the robot race between the West and the East.

Hotel delivery robots

Delivery robots are very common in China and have become a well-developed industry of their own. If you are staying at a good hotel, you will likely see them. They are usually docked at reception. If you order food, instead of the delivery driver or hotel staff searching for your room, they place the order inside the robot, and the robot brings it to your door. Fast and efficient.

The interesting part is that, for this to work, the robots need to be integrated with elevators and hotel room doorbell systems. This allows them to move around the building and get your attention when they arrive.

This highlights one of the main challenges of deploying robots in real-world environments. The bottleneck is often not the robot itself, but its integration with everything around it. I can imagine the friction involved in trying to deploy such a product in a legacy hotel in the West. In the places I visited in China, this seemed to be much less of a problem.

Robot receptionist

Another kind of robot you can see is the robot receptionist. You can find them in various places, including event venues and train stations. Usually, they are simple screen-on-a-Roomba-style robots. More recently, though, you can also see more advanced humanoids with robot arms and grippers.

Seeing one such robot next to a human receptionist, I asked whether she was afraid of losing her job. She didn’t seem nearly as concerned as many people in the West now seem to be about AI. Instead, she explained that she can’t be at the desk all the time because of her other duties, and the robot can help when she is not there.

If you are lucky, you can also run into more experimental applications, such as a robot sales assistant selling beverages at the Canton Fair. You order and pay at the touch screen, and the robot hands you the item.

Unlike the hotel delivery robots, the receptionist robots felt more like optional front-desk augmentation than core infrastructure. I saw them several times, but I did not see many people actually using them. That made them feel closer to experimentation or branding than a clear productivity win.

Subscribe now

Robot baristas

These are reasonably new and, for some reason, very popular. Puzzlingly enough, they also attracted the largest gatherings of Westerners at the Canton Fair.

In essence, they are a combination of one or two robotic arms operating a classic coffee machine. You can see them installed in high-end hotels or as vending machines in public places.

Robot baristas seem puzzling at first. A robotic arm operating a normal coffee machine is not obviously more efficient than a dedicated vending machine. But that may be the point. It is not only selling coffee; it is selling the experience of being served. In places where convenience and spectacle matter more than the ritual of a human-run café, this makes sense.

In London, you can find an abundance of coffee shops serving top-quality artisan coffee. From a unit-cost perspective, these are extremely inefficient. Sometimes you need to wait in line just to get served. Still, people go there for a kind of experience that a stand-alone coffee machine can never deliver.

I didn’t see many high-end coffee shops in China. Robot baristas seem to fill this niche: delivering the premium experience of being served without the need for inefficient human labour.

Self-driving taxis

If hotel robots show what happens when buildings are designed for automation, robotaxis show the same question at city scale. China also has fully self-driving robotaxis. You can see them in Shenzhen and a few other cities. You summon them using an app, and the experience is very similar to taking a Waymo.

They are not as abundant as they are in San Francisco, though. On the other hand, there seem to be more providers. Unlike Waymo, they don’t use a large spinning LiDAR on the roof. Instead, they use a more compact solid-state LiDAR and camera roof configuration.

New ADAS cars

But the more important form of autonomy may not be the robotaxi. It may be the ordinary car.

Any large shopping mall usually has a number of Chinese auto brands showcasing their latest vehicles, often EVs. These brands include Huawei, XPeng, NIO, Li Auto, and a few others. It is quite surreal to see cars and mobile phones from the same brand being sold next to each other. The cars are usually priced at around half the price of Western equivalents. They feature luxury interiors, rich infotainment systems, and screens everywhere.

You can also buy Teslas, and they are still popular. Compared to modern Chinese cars, they feel more sparse and minimalist. Interestingly, in China, you can buy Tesla models not yet available in the West, such as the six-seater Tesla Model YL.

What you notice is that all new Chinese vehicles feature a Tesla-like sensor configuration, with ADAS cameras behind the mirror and on the fenders. They are also equipped with powerful AI-accelerated computers.

The performance of the actual software is still limited to highway-style autopilot, such as lane keeping and lane changes, rather than the full Tesla city FSD experience. However, with the right sensors, connectivity, and compute, they now have all the necessary components to continuously collect data and improve their capabilities. This is how Tesla built its Autopilot, so I expect things will improve from here.

Judging the progress

All these robots are interesting, but the successful ones are not yet quadrupeds or general-purpose humanoids. They are narrow systems embedded into buildings, payments, logistics, and transport infrastructure. The robot is only one part of the product; the surrounding environment matters just as much including how open the society towards the robots is.

I think this shows both the opportunity and the challenge ahead. Perhaps the deciding factor in the robot race between the West and the East won’t be how advanced or expensive the robots are, but who builds the environments, workflows, and social habits that allow robots to become ordinary.

Subscribe now

Robots are one of the most active areas of technology right now. They combine software, hardware, and AI, and are increasingly being deployed at scale. To see what is possible today, I visited the Canton Fair in China to understand what robot hardware is available to buy in volume this year and at what prices. I also wanted to assess the depth of the supply chain, the degree of commoditisation, how advanced the software is, and what to expect in the coming years.

The Fair

The Canton Fair is a large biannual exhibition in Guangzhou showcasing everything being built in China. It is a place to see the latest manufacturing output, where distributors and manufacturers come to source components or entire white-labelled products for resale. It is held in a massive 500,000 m² exhibition complex. The event is so large that it is split into three phases. Phase 1 focuses on consumer electronics, home appliances, and industrial machinery—that’s where the robots are.

The most advanced robots are housed in “Friendship Hall.” Presumably this refers to the idea of friendship between humans and robots. The hall features large cyborg whale graphics moving around. Unlike other exhibits, it requires an additional layer of biometric security to enter.

The space is packed with robots—of all sizes and for a wide range of use cases. Delivery, patrol, inspection, firefighting, house cleaning, logistics. Robots with arms, wheels, or legs, and some with both legs and wheels. Even a massage robot.

Unlike more specialised conferences like WRC, the Canton Fair focuses on robots that can already be purchased in bulk for specific real-world use cases, rather than research prototypes.

Quadruped robots

This is the most common category at the fair. You may have seen videos of Boston Dynamics’ Spot or China’s Unitree Go. Unitree is not the only robotics company in China—there are dozens of similar companies. The supply chain for quadruped robots is well developed and becoming commoditised. You can buy both basic chassis that provide locomotion, as well as vertically integrated robots with payloads for specific use cases. These can include multi-modal sensors, robotic manipulators, net guns for catching intruders, or fire extinguishers.

Pricing is typically negotiated and depends on order size, as well as whether you are buying just the chassis or a fully integrated robot with payload. The ballpark quoted prices are:

Subscribe now

Humanoid robots

Humanoid robots are also present, but less abundant. This is still frontier robotics. You can buy pairs of robotic hands, humanoids on wheeled platforms, or fully fledged robots with legs. Prices are generally higher.

Security robots

One of the main applications for robots on display is inspection and security. About half of the robots are configured and marketed for this use case.

This starts with simple patrol robots equipped with more advanced cameras, often including infrared. These are available in both wheeled form (for flat surfaces) and quadruped form factors (able to climb stairs). Indoor wheeled patrol robots cost around $8k, larger outdoor versions about $30k, and mid-sized quadruped patrol robots about $60k.

This progresses to robots that can act, such as quadruped robots equipped with net guns for catching intruders and tear gas canisters.

If that is not enough as a deterrent, there is also a massive 200 kg rotunbot that can collide with objects at a speed and force comparable to a motorcycle. It is marketed at around $110k per unit.

Robot parts & software

If you don’t want to buy a ready-to-go wholesale robot, you can build your own. Building robots involves components that are distinct from typical electronics: robot motors, end manipulators (i.e. robot hands), robot software, and data collection devices for machine learning.

There are electric motors specifically designed for different robot joints—legs, arms, or smaller ones for fingers. These are typically high-torque, low gear ratio motors. There are also specialised design firms that can help design and manufacture motors to specific requirements.

Basic robot grippers are cheap and widely available. More delicate humanoid hands are still niche and expensive. A low-end 6 degrees of freedom (DoF) hand can cost around $3k, increasing to about $30k for an 11 DoF hand and up to $60k for a 20 DoF hand with built-in pressure and temperature sensors.

The leg locomotion software driving most quadruped robots is widely available. Most of the displayed robots can move around and climb stairs relatively smoothly. When operating autonomously rather than being remotely controlled—such as in security patrol or delivery use cases—navigation is typically handled through map-based systems using lidar sensors, SLAM (simultaneous localisation and mapping), and basic obstacle avoidance.

This changes when it comes to advanced manipulation tasks for humanoid robots. This remains frontier robotics, which is why few humanoid robots are actually performing complex tasks. Developing these capabilities involves multi-modal machine learning and requires specialised datasets of humans performing various actions. For around $8k, you can now buy kits to capture this data, which can also be used for remote teleoperation.

Judging the progress

The Chinese manufacturing industry is highly competitive and able to fast-follow trends, which is now reflected in robotics hardware. If you have an application that requires a specific hardware platform at scale, you can either buy it or design it end-to-end in China.

Software capabilities for many applications, such as security patrols or deliveries, are also already largely available. China is quick to productionise these use cases, and you can see delivery robots in hotels and self-driving taxis on the streets in Shenzhen.

Other applications, particularly those requiring precise human-like manipulation, still need advances in foundational machine learning for robotics. You won’t yet find companies capable of performing complex autonomous tasks similar to those demonstrated by Physical Intelligence. At the same time, companies like Unitree have been closing the gap in whole-body locomotion for humanoid robots quite rapidly. Once the approach to manipulation is solved, it would not be surprising to see the Chinese industrial base fast-follow in that area as well.

1. Both systems used hierarchical decomposition

Anthropic:

In the agent prompt, I tell Claude what problem to solve and ask it to approach the problem by breaking it into small pieces, tracking what it’s working on, figuring out what to work on next, and to effectively keep going until it’s perfect.

Cursor uses a more dedicated structure:

1. A root planner owns the entire scope of the user’s instructions. It’s responsible for understanding the current state and delivering specific, targeted tasks that would progress toward the goal. It does no coding itself. It’s not aware of whether its tasks are being picked up or by whom.

2. When a planner feels its scope can be subdivided, it spawns subplanners that fully own the delegated narrow slice, taking full ownership in a similar way but only for that slice. This is recursive.

3. Workers pick up tasks and are solely responsible for driving them to completion. They’re unaware of the larger system. They don’t communicate with any other planners or workers. They work on their own copy of the repo, and when done, they write up a single handoff that the system submits to the planner that requested the task.

2. Coordination and correctness is the bottleneck on parallelism.

The simple parallelism mechanism reported by Anthropic is:

1. Claude takes a “lock” on a task by writing a text file to current_tasks/ (e.g., one agent might lock current_tasks/parse_if_statement.txt, while another locks current_tasks/codegen_function_definition.txt). If two agents try to claim the same task, git’s synchronization forces the second agent to pick a different one.

2. Claude works on the task, then pulls from upstream, merges changes from other agents, pushes its changes, and removes the lock. Merge conflicts are frequent, but Claude is smart enough to figure that out.

3. The infinite agent-generation-loop spawns a new Claude Code session in a fresh container, and the cycle repeats.

Cursor tried something similar but reports the basic todo file breaks at some point:

The coordination file quickly created more problems. Agents held locks for too long, forgot to release them, tried to lock or unlock when it was illegal to, and in general didn’t understand the significance of holding a lock on the coordination file. Locking is easy to get wrong and narrowly correct, and more prompting didn’t help.

Expecting perfect commits also limits scaling:

When we required 100% correctness before every single commit, it caused major serialization and slowdowns of effective throughput. Even a single small error, like an API change or typo, would cause the whole system to grind to a halt. Workers would go outside their scope and start fixing irrelevant things. Many agents would pile on and trample each other trying to fix the same issue.

This seems it was not a problem for Anthropic at 16 parallel workers. Cursor managed to pass through this to 1000 workers by having explicit task coordination and not expect commits will be perfect. It assumed bugs will be discovered and fixed later.

3. Human input seems to still matter a lot (at least for Cursor)

While titles try to downplay this driving the hype this still matters a lot.
Cursor:

Instructions given to this multi-agent system were very important.

Initially, we didn’t make them our primary goal, but instead aimed for a stable and effective harness. But the significance of instructions became apparent quickly. We were essentially interacting with a typical coding agent, except with orders of magnitude more time and compute. This amplifies everything, including suboptimal and unclear instructions.

• Initially, the instructions focused on implementing specs and squashing bugs. Instructions like “spec implementation” were vague enough that agents would go deep into obscure, rarely used features rather than intelligently prioritizing.

• We assumed implicitly that there were performance expectations within user-friendly bounds. But it took explicit instructions and enforced timeouts to force agents to balance performance alongside other goals.

• For complex parts of the system, agents may write code that has memory leaks or causes deadlocks. Humans would notice this, but it wasn’t always obvious to agents. Explicit process-based resource management tools were required to allow the system to gracefully recover and be more defensive.

Anthropic also reports quite heavy human involvement - writing tests, instruction tuning, failure analysis etc. They describe some tasks the agents could not solve and needed workarounds, this is clearly quite involved expert activity.

Earlier this year, I wrote my predictions for AI in 2024, including the following: "What is here today and working is to stay, and there is not much new coming beyond that very soon." Many things have happened since then. Let's see where things stand in the middle of the year, what can be coming next, and what is limiting the progress of current technology.

Flattening in progress

In January, I predicted that we have mostly flattened out. While incremental improvements to model speed, accuracy, and cost of existing models will be made, this will unlikely unlock new fundamental applications. The rest of the field will also catch up with OpenAI.

It seems to me that this prediction is mostly true today. Let's look at a few stats:

• All Google Gemini 1.5 Pro, Meta LLama 3.1, and Anthropic Calude 3.5 now perform similarly to GPT4. xAI Grok is also quickly catching up.

• On the contrary, the actual raw performance of GPT4 and GPT 4o has not increased much and has remained the same over the last six months.

• Cost and speed per token, though, became cheaper and faster.

• Much of this is now open-source, thanks to Meta's efforts, and you can run it on-premises.

All this is great for developers. If there was something you could build 6 months ago, now you can make it better and cheaper than ever. Indeed, many people did exactly that. Major productivity apps, such as Gmail, Slack, and MS Office, now have small 2023-like AI features sprinkled throughout. They help you find things, summarise content, or provide suggestions.

Despite this, more is needed to unlock fundamental AI applications and drive enough revenues. Over the last few weeks, the trend has been to scrutinise the spending of significant providers on AI while questioning its impact on revenues. An excellent article, "AI’s $600B question," from Sequoia, nicely presents this point.

New physical interfaces didn't fare much better. The flop of startups like Humane or Rabbit shows limitations of HW product usability built on existing technology. While this could have been predicted, I see their case strangely similar to one of General Magic many years ago and Magic Leap more recently. As in the case of all of them, I think the issue is generally about timing, technology readiness, and commercialisation strategy rather than the idea itself. iPhone eventually happened. Apple and Meta aim to offer a more viable route to wearable AI assistants and AR glasses. Let's see whether they get it right.

Part of commoditisation is the consolidation of independent AI labs. It is happening in an entirely unexpected way. It used to be the case that if you could not stay competitive as a startup, you went bankrupt or got acquired. Now, the default exit for AI startups like Inflection.AI, Adept AI and Character AI seems to be being "hollowed out". Leadership and the team join a big company while the shell of the original company is left to continue. It's an innovation, but one to avoid regulation.

Subscribe now

Surprises

The above could suggest that we have indeed flattened out. However, that's not a complete story, and there are reasons to stay optimistic, too.

Sora surprised me. I thought the capability of generating high-fidelity videos was two years out. It is happening already today. I have yet to use it, but some examples are truly impressive.

Another hope on the horizon is new datasets. In my January predictions, I mentioned that new datasets might arise capturing humans' daily use of computers. It would unlock the ability to train LLM Agents to perform these tasks on their own. I didn't know how such datasets could be collected and raised likely privacy issues.

Microsoft, with its Recall, seems to be aiming for precisely this. It implicitly collects this dataset in the background to offer a search over your activity. The same dataset can be used to train and automate the same tasks in the future. Unsurprisingly, it immediately raised the abovementioned privacy concerns. Now, Apple, with its integrated Apple Intelligence, is aiming for the same goal but from a slightly different angle. In both cases, you need control at the OS level. These are the only two companies that can pull this off. The exception might be Google, which could embed similar functionality directly into Google Chrome.

The bull and bear case for AI

AI will eventually be more significant than anything else. The question is how long it will take. Does it take 1, 5, 10 or 100 years? What determines how quickly it will happen?

I continue to believe the key reason more progress was not made this year is the lack of further fundamental scientific improvements in the underlying technology. You just can't build many AI applications people are excited about with today's technology. You can’t build agents that work very well or robots that can accomplish medium-difficulty tasks. Current technology is also no secret anymore. All leading LLM models across big tech are built more or less the same way.

Given this, I think there are two possible ways for the future to unfold. It depends on whether further technological breakthroughs happen very soon or not.

The bear case

Due to the lack of fundamental technology breakthroughs in the coming years, LLMs will keep improving incrementally by 20% YoY (but not 10x YoY), and the field will further commoditise. Their applications will slowly increase across existing products through incremental features.

Any new AI-first product from 2024 onwards will be comparatively rare and require a unique integration of product experience that efficiently leverages existing technology instead of creating a new one.

Public company stocks will normalise to reflect this outlook. Many startups built on the premise of radically better AI becoming available very soon will go bankrupt or need to find a different path forward. By very soon, I mean 1-3 years, which is the average runway of a venture-backed startup. On the other hand, applications that have solid user traction and leverage existing AI will only work better.

The bull case

New AI technological breakthroughs will come in the next 1-3 years. They will increase the applicability of the technology 10x and unlock more value creation, propelling the industry's growth. The number of people working on AI is now higher than ever and rewards are more significant than ever. They might be due to algorithmic Improvements, increases in scale, new kinds of datasets, integrations, or new HW platforms (AR glasses, robots).

In recent years, much of the investment has focused on preparing for the bull case. You just can't miss it if you are an investor or a big company. Nobody wants to be a Nokia when the iPhone arrives. It's entirely possible, though, that the reality will resemble the bear case. I think it’s better to be prepared for both.

Subscribe now

I hope it's not too far into the new year to make my own prediction about AI. I want to believe I have something to contribute to this topic as I am thinking about this a lot and have some experience in products and AI research.

I predict while there will be some progress in 2024, we have mostly flattened out.

I also predict the real opportunity for further progress is elsewhere than where people are looking today.

All this might be wrong, and 2024 can well turn out to be a wild one. That's ok. The future of the internet in 2000 was hard to predict. Still, it's good to give it a try:

We have mostly flattened out

By this, I mean "What is here today and working is to stay" + "There is not much new coming beyond that very soon". This means no new radical AI products tomorrow—no wide disruption of tech or society. No singularity is happening.

I am not excited about that, but there are two reasons why I think this:

1. First, in the past year, entrepreneurs, big and small, tried their best to use LLMs to build new or disrupt old products. Some of this stuck, but much of it didn't. Why something possible in 2023 would suddenly take off in 2024? One year might not sound like a lot, but given the amount of attention and volume of things tried, it makes some precedent.

2. Second, I am unaware of any significant technological breakthroughs that would unlock new product categories. I like to follow the research community quite closely. It usually gives foresight at least 2-3 years ahead of tech product adoption. My read is that we have stuck with the current state of AI tech for some time.

Things will still get incrementally better, though, mainly through continuing bundling the existing stuff. Techniques you could see before in a 2023 demo or research paper will become more mainstream. This will result in higher benchmark numbers, more inputs/outputs of different kinds, and more integrations where LLM is both at the top and bottom of the application stack.

Given the slowdown, the field will likely continue to catch up. The performance of models will become more competitive between dominant players. Google’s will be comparable with OpenAI. Apple will launch LLM of its own. You will also continue to do more of that with open source. Nothing of this, though, will change the fundamental question of product use cases.

Subscribe now

How does this translate to AI having a disruptive effect on the world?

For a broad transformation, a well-defined thesis should come true. Looking back at the internet era of 2000. The thesis was "everything is much better online." And this thesis indeed came true for many things. One by one, you would better shop, make bookings and do business online in a digital way and over email than how you did it before.

Similarly, for AI to create a fundamental transformation, the thesis is "everything is much better with AI". This thesis is certainly actual in the long run. But it's not the correct thesis to think about today. AI will be limited to the current LLM technology without further fundamental breakthroughs. The thesis should, therefore, be" everything is much better with the current version of LLMs". Suddenly, things are not looking as rosy as only some things seem to be much better with them.

Let’s look more closely at some of those:

Things that work today

A handful of use cases exist where current LLMs make a better product. Those are already popular today. Things like writing SEO blog posts are significantly faster with LLMs. Similarly, copilots for coding or customer service are picking up.

The common pattern is that these tasks are centred around text/voice communication and that you usually hire groups of other people to do it for you. And the work is often reviewed word-by-word. It makes sense if LLMs make the review and suggestions for you!

Things that don't work and why

Most other use cases don't have their disruptive LLM-enhanced product yet.

1. It's hard to beat a dashboard.

One key reason is that the LLM experience needs to be significantly better than its main competitor: An existing well-designed dashboard for the task. Think about that. It's tough to beat a great dashboard! Any serious conversation leading to decisions eventually turns into figuring out and analysing data. It's not a chat interface anymore. The most suitable format to present such information is a dashboard. LLMs may be able to generate these dashboards for you as well. It will be a while until they are much better than existing product dashboards refined over time.

2. Search is still a king.

The above is true for search as well. I am not bullish that LLM chats will replace search very soon. When people search, they like to see different options, dig deeper to understand them and then make a choice. The existing search interface provides this experience. LLM chat can undoubtedly enhance it and make it better, but not replace it.

3. Automation is more difficult than you think.

It's also hard to fully automate things and, this way, create entirely new product categories. This is because building generally robust AI applications is very hard. I wrote about why this is the case here.

Subscribe now

Is there an opportunity?

So, where do we go from here? I see four avenues where progress can be successfully made in the near term. Maybe not yet in 2024, but soon.

1. LLM-features in existing products

Instead of looking for new radical LLM-first products, narrowing the focus is better. Get back to basics and think from first principles about individual features in existing products that can now exist that could not exist at all before. Experiences even the best dashboard or GUI could not previously deliver or technology to support. Why using them would be compelling?

2. Narrow and deep vertical AI

Going extremely deep on an important vertical and building dedicated AI can unlock new results impossible to achieve before. Think about AlphaFold. This likely does not lead to a consumer product you can directly experience every day. Its impact, though, can be significant. It requires a deep understanding of the problem and can also be capital-intensive. But hey, if it were easy, everyone would do it.

3. New kinds of datasets

Imagine a future where you give LLM a complex task you usually do in your day job that takes several hours to complete. Doing it might involve interfacing multiple apps, replying to emails, figuring out necessary information from your colleague through Slack, etc. The work of many office workers looks like this today. After some computing, the LLM presents your plan for review, which you approve, and goes to work. In the meantime, you get coffee and watch sports while LLM does the work that previously took you all day.

A new kind of dataset is necessary to make this possible in all its variety and using only the current technology: the one that captures how people use their computers daily to accomplish all these tasks. What they see and what they click/type as a result. An LLM trained on it should be able to construct predictions and plans. It would need to be a big one to work. Collected over millions of people, thousands of hours each.

Nothing like that kind of dataset exists today. Not sure if it ever will. It will be immensely useful, but it will have tons of data privacy issues. Humans, however, can accomplish all these tasks without training on all these data. Do we need to wait for the next core technology breakthrough for this to be possible for AI as well?

4. New physical interfaces

You might use technology differently if new physical interfaces are available to you when old ones are not. Do you remember the first iPhone in 2007? That's what I am talking about. Multi-touch became the default way you use it. This is despite the phone with multi-touch is less efficient than using a computer with a mouse and keyboard. The computer was just not with you all the time. Phone was.

A similar thing can happen if augmented reality glasses become more prevalent and with you constantly. The most convenient way of interfacing with them might be through LLM conversation. This is regardless of whether this is the best solution for any task. The success of such interfaces is very difficult to predict until you compellingly experience them. I am hopeful we will see more progress on this front soon.

That’s it! Just remember - don’t blame me if none of this turns out to be the case. What are your predictions?

Subscribe now

Everyone now seems to be doing AI. Big companies do it. Small startups do it. People of all backgrounds tinker with it in the evenings. This begs the question: who do you actually need to build ML projects today?

Often, when building AI applications, it’s enough to take any existing off-the-shelf ML component and use it. Be it an image classifier, LLM API, or diffusion image generator. You plug it together with whatever data pipeline or UI is suitable, ship it, and you are done. No ML skills are needed.

Sometimes, reusing existing things is not enough. You may be building something that has never been done before, or require performance not available in what exists, or use specific new datasets. For example, you could be trying to use ML to discover drugs, make cars drive themselves, or create the next GPT5. What should you do in such situations? Who should you hire? What should they do, and how?

I was fortunate enough to build several AI R&D teams across both startups and large corporate efforts. Starting a new ML project and hiring a team is tricky, akin to building an NBA team. There are patterns though that can help you identify the right people. Let’s first dive deeper into what ML actually does and how to measure its progress before you understand who to hire.

How Machine Learning differs from Software development

If you are a programmer building typical software, i.e. websites, games, or mobile apps, the components you use are reasonably well understood. You know how computers work, how to program them, and what to expect. You know what a database is, how backend APIs work, or how UI is rendered in the front end. Over the past few decades, all these components have been used and recombined in many ways across the industry. When starting a new project, it is possible to break it down into these pieces and plan execution. Things can still go sideways, and projects run over, but it rarely happens that it would not work at all.

This is not the case when building a brand-new ML project. If something was not done before, it’s impossible to tell with certainty how well it will work. Or whether it will work at all. What kind and how much data is necessary? What is the best neural network architecture and training procedure? Often, it’s not even clear how to measure success!

Questions like this can make seasoned engineers, managers, and product managers nervous. If success is not guaranteed, nor is the timeline or costs known, how should one manage the project? The return on investment?

The hard truth is that the only way to find answers to these questions is by trying. This requires experimentation, and the entire machine learning workflow revolves around it. Experimentation is the way an ML team achieves its results.

Subscribe now

Experimentation loop = the engine of the ML team

Training and evaluating an ML model is called an experiment. It answers a simple yes / no question called hypothesis. Usually, in the form of whether a particular training combination works better than the best one found so far. This then results in an improved system, learnings are gathered, and the process repeats. Eventually, if the performance is good enough, the product can ship to customers.

Sometimes, it’s enough to do only a few experiments to achieve good results. Often, it’s much more. Building projects of high ambiguity or requiring high performance can take hundreds of experiments. For example, building a perception and planning system for a self-driving car takes years of continuous refinement of both model, training, and evaluation sets.

There are also times when, no matter how many experiments you do, the results are just out of reach. The required performance might be unattainable with available resources or the state of technology available. That’s OK, and a result like that should also be celebrated and found out as quickly as possible to save resources.

Given the central roles of experiments, it is worth understanding what makes a good experiment vs a bad one.

First of all, a well-designed conclusive experiment contributes to progress much more profoundly than several inconclusive ones. A good experiment verifies what works or shows what should be avoided to move forward. A bad experiment does not make you any smarter and keeps you in the same place.

The speed of executing experiments also matters. The same experiment can be done in a day, a week, or a month. The faster the experiments can be executed the more progress can be made in the same period. The same is true for cost. If running experiments is cheap, many can be done in parallel. On the other hand, one can usually afford only a few expensive ones. For example, training ChatGPT from scratch takes months.

As a result, you can measure the velocity of an ML team in terms of the quality of experiments and speed of executing them:

Maximising the velocity is the main way the ML team accelerates its way to results. Each new member can be hired and evaluated based on how they contribute to the team's velocity.

Researchers vs Engineers

The velocity formula above naturally leads to two main skills you can look for when hiring people:

• Ability to design good experiments and

• Engineering required to execute them.

Sometimes, you find a unicorn - someone who is great at both. Such people exist but are exceedingly rare. It’s not the best strategy to hope you will build your entire team with them.

Most people are stronger in one area than in the other. This leads to two main roles found in ML teams:

• Researchers usually have professional training, a Ph.D. that helps them to investigate the relevant state of the art, design experiments, and scrutinise the results. On the flip side, their engineering skills are often limited, which comes at a cost in large-scale endeavours or team efforts.

• Research engineers, on the other hand, are usually much stronger in engineering and can write high-quality code. They should have experience with ML libraries and frameworks (i.e. PyTorch, TensorFlow, W&B..) and can use them to write high-performance training pipelines or prepare datasets. They might not design experiments themselves but work with research scientists to execute them.

How many of each do you need? It depends, but a good start is to have one or two of each to start the project. With them, you can get going. Once the baseline of the system exists, it opens avenues for more people to join the effort. They can carry out more experiments or accelerate their scale and execution. The velocity formula should be your guide in who you need at each point of the endeavour.

Identifying the roles you need and what they will do is only the first part of the story. You will still need to find, interview, and convince people to join your company. This is hard, especially today. Let’s cover that in another post!

Subscribe now

If you were not living under a rock, you already know ChatGPT was a key technology story of 2023. ChatGPT is appealing due to its generality and seemingly unlimited potential to automate work. Goldman Sachs estimates more than 300 million jobs could be automated around the world. Now, the question that every entrepreneur and VC is asking is: how do you actually build useful products and defensible businesses from it? What kind of products are now possible? Who will the winners and losers be and why?

When asking these questions it’s useful to look at another recent AI story: self-driving. In the past decade, many companies pursued and died on the way to build self-driving. Others, such as Waymo and Cruise were more fortunate and with billions of $ and a decade of R&D now offer fully driverless commercial deployments in large cities. Are ChatGPT applications going to follow a similar trend?

For the last 12 years, I have been working at the intersection of AI and business. First in research, doing a Ph.D. in Robotics at Oxford. Later in the industry, starting two startups in the space. Finally, in corporations, leading large self-driving efforts at Lyft Level 5 and Toyota. I have been fortunate to see both the technology and business aspects of these endeavors closely. I believe there is much to learn from self-driving as a lens to peek into what might lie ahead in AI.

Generality vs. Robustness vs. Value

The value of AI comes from the opportunity to automate today’s work or do new work. The more you automate, the more value you can generate for the end user.

Making things technically possible, however, is a whole different story. When building an AI product its good to think about two aspects:

1. Generality - how constrained or open-ended is task at hand? A system that needs to handle a wide range of situations needs to be much more capable than one that only needs to handle a narrow scope.

2. Robustness - does the system need to work perfectly (compared to humans) to generate value? A system can work fine in 99% of the cases and produce incorrect results in 1% of them. Is this case acceptable for the product to be used?

   

Why is this classification important? As it turns out, it has dramatic implications on what is technologically possible and managing user expectations. To recognise this, self-driving industry devised its own closely-matching categorisation system:

• Level 1/2 (lane-keep assistant) - is narrow and fragile delivering partial automation on highways but where the user needs to keep paying attention at all times.

• Level 2.5 (Tesla Autopilot) - is general and able to handle a wide range of city situations but is fragile requiring users to still keep paying attention at all times.

• Level 3 (BMW G70 autopilot) - is narrow and constrained to handling slow-moving traffic jams on highways but robust, not requiring users to pay attention.

• Level 4/5 (Cruise and Waymo robotaxis) - is both general and robust, able to handle a wide variety of situations without user input.

As you see, different levels of self-driving have different levels of capability and different costs to develop.

Similarly, you can put almost any AI application somewhere in this graph. Where does ChatGPT sit? As you probably guessed, ChatGPT and most applications built on top fall into the general fragile / narrow fragile quarters. They can do impressive things but their output still needs to be reviewed to be usable.

What does this mean if you are trying to build an AI company nowadays? What should you actually aim to build? Let’s look into history for insights.

Subscribe now

Learning 1: Narrow fragile applications can be built reasonably quickly.

The self-driving boom of 2014-2018 was also powered by a breakthrough in technology. In particular LIDAR sensors and supervised deep learning. LIDAR allowed machines to see in 3D. Deep learning allowed to label a lot of data and turn it into powerful perception systems. The remaining building blocks (such as simultaneous localisation and mapping - SLAM systems and trajectory planners) were available, albeit in crude form, from the research community. All it took was a small team able to put all of it together.

Many companies including Cruise, NuTonomy, Zoox, Nuro, Aurora, Voyage, PonyAI, FiveAI, Drive.AI, Oxbotica, aiMotive took this opportunity and quickly gained investor traction raising 10s-100ds millions of $ with their demos. These showcased prototypes of automated lidar-equipped vehicles operating in constrained environments, usually in parking lots or on pre-mapped roads and under human supervision. Although often magical to investors, the demos were too fragile for any real-world use because things went wrong all the time (as any robotics practitioner would be able to attest).

In parallel, the advent of ChatGPT is now being followed by a wide variety of various demos and some massive investments. On one end, you can find recommendation systems, such as Github Copilot or Jasper where the user is still in charge and can correct the system’s output. On the other end, people are playing with autonomous applications, AutoGPT and chain-of-the-thought prompting that are able to autonomously achieve seemingly any task. If you though played with any of them on your own you will agree that they often need corrections and, as such, are not yet suitable for practical applications that demand reliability.

Learning 2: To improve one needs in-domain data from a deployed system.

The key challenge in building reliable AI systems is to overcome the robustness and generality gap. A self-driving vehicle must be able to handle a wide variety of situations and handle each of them robustly 100% of the time. When lives are at stake, 99% is simply not good enough. Many possible applications of GPT also have this property. You can’t have a doctor chatbot that outputs wrong instructions 5% of the time or an automated payment accountant that sends money to the wrong account.

The way all self-driving programs approach improvement is through continuous deployment, testing, and improvement. A version of the system is deployed to a testing fleet that is supervised by a safety driver who can take control of the vehicle at any moment. This can be due to mistakes in perception, planning, or localization systems. Each such ‘disengagement’ event is then analyzed and added to the training/testing set that is used to develop subsequent iterations of the vehicle’s systems. This cycle continues, often for years, until the performance is acceptable to remove the safety drivers.

Similarly, OpenAI is using collected data to improve its chatbot to be more accurate and better at solving specific problems. This is required despite training on almost all open data on the internet. The datasets about specific ways people use ChatGPT and their responses to its output simply don’t exist. The only way to get it is from an already deployed system.

Learning 3: Startups pursuing general-robust applications usually die.

Running an in-house cycle of continuous deployment, testing, and improvement can be very expensive. It costs roughly several billions of $ to reach Level 4/5. This is something few can afford.

Many people tried to decrease reliance on real-world road testing through simulation. This is useful and more cost-effective but it doesn’t really substitute the need for extensive road-testing. It is simply too hard to discover and simulate the long tail of rare events that happen on the road, and the reaction of the vehicle and other traffic participants to it.

As a consequence, most of the startups pursuing general on-road autonomy have not survived. They have typically been acquired for talent by larger players, such as, Cruise, Zoox, NuTonomy or Voyage. Even well-funded efforts backed by Uber, Lyft, or Ford went through corporate transitions. Eventually, the market will only support a few players with deep pockets i.e. Google or GM, which can pick up the check.

A different approach is instead to focus on shipping a narrow-fragile product first and expand from there. Instead of buying the usage data, the aim is to gather it organically for free through product usage. It might be a longer way but is certainly more cost-friendly.

This is what Tesla is doing. Tesla autopilot first featured lane-keep assistance on highways only and used the data its fleet collected to gradually expand the generality and performance of autopilot over time. While not yet robust enough, the system is still significantly capable.

Implications for building ChatGPT applications

What are the learnings from all of this if you are building a ChatGPT application? I think there are several:

1. Don’t try to build something that replaces humans - Replacing humans is too difficult due to the long tail of events and robustness required. It can be achieved if you have a lot of time and $ but is not a good prerequisite for shipping a product.

2. Narrow the application domain and ship a human assistant that delivers value on day 1 - Narrow fragile applications can be built quickly and various assistants or copilots already proved to be valuable in several domains. The key is that humans are always eventually in charge of output, can review it, and take responsibility. This is in exchange for the task requiring less effort or time.

3. Collect all the data and human corrections - This allows you to gather a dataset allowing you to improve the performance of the product. With every user interaction, you will be able to make your system work a little bit better. It allows you to both broaden and deepen the model’s specialisation in your domain. As a result, this proprietary dataset will eventually create a defensibility moat separating the performance of your product from any of the new players.

4. Iteratively ship improvements to your system to increase generality and robustness - The collected dataset allows you to see how the system is used and fine-tune it for these use cases. In turn, a better model can be delivered back to users. In particular, increasing generality will expand the addressable market while increased robustness will require less interaction or perhaps, not requiring any supervision at all increasing customer ROI.

What are your thoughts and experiences with building GPT applications? Let me know in the comment below.

Credits: Thanks to Nathan Benaich, Toby Coppel and Ashesh Jain for pre-read and comments.

Subscribe now