Last week, Harvard Business School released its analysis of LLMs on 758 consultants’ performance when using AI. Three key insights emerged.

In the first experiment (inside AI’s capability frontier), consultants randomly assigned access to GPT-4 AI completed 12.2% more tasks on average & 25.1% faster. Quality improved by 40%.

Lower performing consultants benefit the most from AI augmentation, increasing performance by 43% compared to 17% for higher performers.

This rising-tide effect seems common across AI applications. The benefit to the lower quartiles is dramatic across sales, customer support, & consulting. This has broader implications, something I aim to write about later this week.

In the second experiment, consultants analyze business data to offer strategic recommendations. Here, AI reduces performance – those using AI are 19 percentage points less likely to produce correct solutions.

The “jagged frontier” conceptualizes how AI profoundly increases productivity on some tasks but provides no value or even diminishes performance on seemingly similar tasks.

As with any tool, we need to learn the best applications & techniques for hewing knowledge work effectively.

Last, the paper discusses two types of human/AI collaboration : centaurs & cyborgs. Centaurs ask for high-level help with AI. Cyborgs train the AI to act as a character. The difference remains a bit nebulous to me, but the last section of the paper in Appendix E highlights different techniques, some of which I hadn’t explored.

I haven’t yet asked an LLM to impersonate someone but given the impact to education, I could imagine LLMs becoming sales & customer support trainers for new employees in those roles.

Especially since the impact on lower quartiles (presumably many of which are ramping) is so marked.