The study that was published a few months back and conducted by Boston Consulting Group in collaboration with researchers from Harvard Business School and Massachusetts Institute of Technology. According to it, ChatGPT users outperform non-users by 25.1% in speed, 40% in output quality, and complete 12.2% more tasks. The study included 758 consultants from BCG offices around the world, or about 7% of the company’s consultant base.
Among the tasks that consultants had to solve, there was one that was specifically designed to be out of AI scope. This task showed a decline in performance when consultants relied on AI, falling from 84% to 60-70%.
Study participants were divided into two categories: above average performers and below average performers. What was interesting is that the boost in performance was truer for lower-performing consultants. They noticed a 43% performance boost, which diminished the performance gap between top and bottom consultants. Top performers only improved by 17%.
Types of Collaboration between Human and AI
For the purpose of this study, two types of collaboration between human and AI were identified:
- Cyborg behaviour – where human and AI work as a tandem, intertwining their efforts.
- Centaur behaviour – where there is a clear division of labor between human and AI. This type of collaboration meant that there’s strategic switch in labour, based on their strengths.
The Experiment
In the first experiment, participants were asked to create a concept for launching a piece of footwear and taking it to market, capturing every step involved from idea to launch. The assignment required completing 18 tasks across broad domains: (1) creativity, (2) analytical thinking, (3) writing proficiency, and (4) persuasiveness. AI proved to be highly beneficial in improving performance across all four domains. AI-assisted consultants completed 12.2% more sub-tasks and produced work of a 40% higher quality – compared to the test group.
The second experiment, however, showed that using AI in consulting too heavily can have a negative aspect. This experiment went beyond the scope of AI capabilities. In it, participants had to perform interviews with company insiders and use financial data from a spreadsheet to pinpoint which of a hypothetical company’s brands held the most potential for growth. Only one of the given brands held potential for growth, so the answers were rated as correct or incorrect. Subjects in the control group were correct about this exercise about 84.5%, while the AI-ers scored at 60-70%. At the same time, they completed the task 20% quicker. This is interesting because while being quick, they sacrificed accuracy in the process.
This phenomenon is called “falling asleep at the wheel” in the study. It claims that relying too much on AI can lead to poorer decision-making and decreased vigilance. It was said that this is because of the tendency of AI to give rather homogenous output, lacking the human factor.
To Conclude
AI in consulting is here for the long run. It has undoubtedly shown to have a transformative potential when used in a manner that emphasises its strengths. As this study shows, it requires ethical usage as well as determining the correct type of interaction for best results.
Learning more as companies and individuals will help us make more informed choices on how to integrate AI ethically and effectively in our workflow.
Check out these case studies (1, 2, 3) to learn about some of our team’s projects that have made use of generative AI.