Too Busy to Read AI-Pocalypse: The Shocking Impact on Law Firm Profitability? Let GenAI Read It For You.
By John Tredennick
Three of my favorite authors, Ryan McClead, Greg Lambert and Toby Brown of Three Geeks and a Law Blog, wrote a fascinating three-part analysis on the impact of GenAI on law firm profitability that should be required reading for everyone in the legal profession.
The series started with an in depth look at the oft-cited Goldman Sachs article suggesting that the legal profession would shrink by as much as 44% due to the productivity enhancements (or job stealing depending on your views) of GPT and its Large Language Model progeny. 44% of Investment Bankers Think They Can Make Lots of Money Off of Attorney Insecurity (AI).
Their second post took a look at different data and projected that the impact would be more like 23% (still a significant impact but half of the Goldman Sachs gloom and doom numbers). Generative AI Could Reduce Law Firm Revenue by 23.5%.
Their final post then offered insights and advice on how to ride this new wave of AI technology and end up standing on your board. AI-Pocalypse: The Shocking Impact on Law Firm Profitability.
This was serious work. At 5,600 words and 32,000 characters, there was a lot to read and digest. But the topic is important to the legal profession and the authors’ conclusions are worth serious consideration.
But I am too busy billing hours to read something this long.
Right, me too. So I took the easy way out and simply asked Claude 2 to summarize the article for me. Why Claude 2? Because GPT 4 kept rejecting my prompt because the underlying articles were too long. I even tried submitting the posts one by one. Only the third one made it through. I could have tried further trimming but I ran out of time.
For those newer to GenAI, GPT 4’s context window (the space for analysis) is 32,000 tokens (about 2,800 words). Claude 2, the leading competitor to GPT 4 in our judgment, can handle 100,000 tokens in one go (enough for an entire book).
So, if like me you feel too busy to read all three parts, Claude 2 can help. Here’s its take on the three-part blog post:
Acting as a managing partner of a big law firm, summarize the main points from this three-part blog article: [text of the three blog posts]
Here is a summary of the main points from the three-part blog article:
- The original Goldman Sachs report claiming AI could replace 44% of legal work was imprecise and based on flawed data.
- The press coverage of the report was sloppy, with different articles claiming AI would replace 44% of legal tasks, positions, or work.
- The report relied on small sample sizes from the O*NET database, with only 31 lawyers surveyed.
- The data doesn’t actually tell us much about automation’s impact on the legal industry.
- The level rankings used are not clear integers, so assumptions that AI can perform tasks “up to level 4” are questionable.
- Depending on assumptions, only 7-18% of lawyer activities are potentially automatable based on the data.
- Across the legal job family, only 9-21% of tasks are potentially exposed to automation.
- Goldman Sachs likely stands to benefit from hype around AI’s impact on legal, so they are not a disinterested party.
- The original 44% statistic was taken out of context and does not seem dire when the flawed data and methodology are considered.
In summary, the report and press coverage overstated automation’s potential impact on the legal industry based on limited data and questionable assumptions. The claims should be viewed skeptically given Goldman’s interests and the small sample size underlying the data.
Can you provide more detail about each point, explaining the author’s reasoning?
Absolutely, here is a more detailed summary:
The original Goldman Sachs report claiming AI could replace 44% of legal work was imprecise and based on flawed data:
- The report looked at automation’s potential impact across all US industries, not just legal.
- It relied on data from the ONET database of job profiles, but ONET organizes data by job titles, not industries.
- The author found it odd that legal secretaries were classified under administrative, not legal.
- The “industries” Goldman labeled were actually O*NET’s “job families.” This suggests industry categorization issues.
The press coverage of the report was sloppy, with different articles claiming AI would replace 44% of legal tasks, positions, or work:
- The author shows how coverage confused whether AI would replace legal tasks, positions, or work.
- The inconsistencies suggest journalists did not thoroughly examine the original report.
The report relied on small sample sizes from the O*NET database, with only 31 lawyers surveyed:
- O*NET surveys practitioners to gather data, but sampled just 31 lawyers and 134 legal workers.
- With such small samples, data may not represent overall legal industry accurately.
The data doesn’t actually tell us much about automation’s impact on the legal industry:
- The level rankings used to estimate automatable tasks are averages based on surveys.
- Granular job-by-job analysis shows automation’s potential impact is likely overstated.
- Small samples and inconsistent industry categorization undermine data quality.
The level rankings used are not clear integers, so assumptions that AI can perform tasks “up to level 4” are questionable:
- O*NET ranks tasks on a 1-7 scale, but decimals in data suggest averaging.
- It’s unclear whether “up to 4” means below 4.0 or 5.0. The difference significantly impacts automation estimates.
Depending on assumptions, only 7-18% of lawyer activities are potentially automatable based on the data:
- When examining lawyer data specifically, only 7-18% of tasks fall below level 4 or 5.
- This suggests automation’s impact on lawyers is overstated.
Across the legal job family, only 9-21% of tasks are potentially exposed to automation:
- Analysis of all legal roles shows only 9-21% of tasks below level 4 or 5.
- Again, this is far below the reported 44% figure.
Goldman Sachs likely stands to benefit from hype around AI’s impact on legal, so they are not a disinterested party:
- The author believes Goldman has investments poised to benefit from AI hype.
- This implies an incentive to overstate automation’s potential legal impact.
The original 44% statistic was taken out of context and does not seem dire when the flawed data and methodology are considered:
- In summary, the author shows the data issues underlying the 44% figure.
- When examined critically, it does not seem to credibly support such a high automation estimate.
- The press amplified an exaggeration based on questionable data and assumptions.
After receiving Claude’s summary, I decided to take one last try with GPT 4. I asked it to summarize the third post in the series, which was shorter than its predecessors.
In this case, I revised the prompt to ask GPT to act as a senior managing partner in a large law firm talking to his partners about these conclusions. I think you will enjoy GPT 4’s take on their report.
Summarize the key points in this article acting as a senior managing partner of a large law firm talking to his partners about these conclusions. [sending text of the third post]
Ladies and Gentlemen,
I write to you today to discuss the potential effects of Generative AI on our firm’s profitability. The primary findings of our study are that Generative AI will disproportionately impact non-partner hours, and that its effect on profitability is more complex than simple reduction of billable hours.
Our baseline model is an M&A transaction with five different timekeepers. Based on this model, our estimated annual Profit Per Equity Partner (PPEP) is around $2.6 million. However, this could change based on the billable rates, realization rate, and expenses for each timekeeper, among other factors.
Contrary to intuition, the majority of our profitability actually comes from non-partner timekeepers. This is due to the principle of leverage—the percentage of non-partner hours to partner hours. Our baseline leverage is 60%.
When Generative AI is applied, we see different scenarios. In a conservative scenario where we have a 5% reduction in partner billable hours and a 20% reduction in non-partner billable hours, we see that our total revenue is reduced by about 13%, and our PPEP drops to about $2.4 million. Our leverage also decreases to 56%.
If we start with better leverage (63%), then our PPEP can increase to about $2.7 million. However, applying the same conservative effects of Generative AI results in a decrease in PPEP to around $2.5 million. Therefore, firms with better leverage may experience greater reductions in profitability when using Generative AI.
A less favorable scenario sees a 20% reduction in partner hours and a 40% reduction in non-partner hours, resulting in a total revenue reduction of 30% and a decrease in PPEP to $2.3 million.
Now, how do we react to this AI-driven shift in our field? For the next year, we need to invest in education around Generative AI, accessing AI tools as inexpensively and safely as possible for educational and internal use only. We should not blindly invest in expensive Generative AI solutions without understanding their impact on our bottom line.
The most profitable practices could be negatively affected by Generative AI due to the disproportionate displacement of associate time. Hence, improving leverage is not only crucial but now more urgent than before. We need to build and improve our leverage model in the face of this technology.
The partner compensation models also need to be reassessed. Current models often reward partner production over other metrics. In an era where work can be done faster, more of that work should be done by associates. Partner focus should shift towards business growth. Models that continue to incentivize moving work to higher-cost resources will diminish firm profitability while raising costs for clients.
Now is also the time to consider alternative pricing methods such as flat fees and value-based pricing. Generative AI may allow for some legal tasks to be done at a fraction of the cost, leading to increased client satisfaction, improved client retention, and increasing chances of winning higher value hourly work that AI is not yet capable of.
We must remain cautious about Generative AI. It’s important to understand its potential impacts before fully integrating it into our practices. Know what you’re buying and where you’re deploying it, and understand how it will affect your bottom line.
Thank you for your attention. I believe that together, we can successfully navigate these turbulent waters and continue to provide excellent service to our clients while maintaining the profitability of our firm.
There you go. A quick read of these important posts with time for you to get back to those billable hours.
Thanks Claude 2 and GPT 4.