Can ChatGPT pass the bar, and what does it mean for future lawyers?

What is ChatGPT?

ChatGPT is an AI-powered tool developed by OpenAI. It is a new technology that operates as a chatbot, allowing users to access information quickly and accurately, create content based on prompts, or use the bot as a problem-solving partner. The technology is based on natural language processing and machine learning algorithms, making it capable of understanding complex legal concepts.

ChatGPT is making waves in many industries. In IT, it is used to write code. It is also popular among marketers and other professionals who need to quickly generate a lot of content. The solution also has a tremendous potential as a tool for lawyers, who can use it as a real-time assistant to answer questions, research case law, and draft documents.

At the same time, AI technologies like ChatGPT have their weaknesses. A major one is that the tool answers all questions confidently, even when it does not know the answers. In those cases, ChatGPT simply makes something up. It has great potential for speeding up everyday work, but it cannot be left without supervision.

The impact of AI on the legal industry

ChatGPT is an AI tool with a general focus. However, dedicated AI solutions for the law firms exist - AnyLawyer is only one example. Similar AI tools are designed to help lawyers stay on top of the legal landscape by providing them with automated support for everyday tasks.

Legal research

AI technologies can quickly analyze large amounts of data, determine its relevance, and draw conclusions from it. This allows them to effectively conduct legal research. AI applications are being used to analyze case law and generate summaries of important decisions, as well as to compare different jurisdictions to find the most favorable regulations.

Document drafting

Artificial intelligence tools can be used to create documents such as contracts and pleadings quickly and accurately. Automated document creation tools can generate entire legal documents based on a few keywords and a template, or use a detailed prompt for more tailored results.

Contract and document review

AI contract review software can quickly scan a contract for potential issues, such as conflicting clauses or missing details. This makes it possible to identify risks much faster than with manual review.

Proofreading

AI tools can be used to proofread documents. Natural language processing algorithms analyze the text and check for grammar, spelling, and punctuation errors as well as incorrect facts or logic. This saves a lot of time compared to manual reading.

E-discovery

AI-powered discovery tools can quickly process large amounts of data and identify relevant documents, as well as classify them by relevance. This is especially useful for complex cases that require a lot of research.

Litigation strategy

AI can be used to determine the best litigation strategy for a case by analyzing patterns in past rulings and other legal documents. This can help lawyers develop strategies to maximize their chances of success.

Writing emails

Legal professionals often need to write emails and other correspondence. AI tools can be used to automatically generate personalized email drafts, saving time and ensuring that all the necessary information is included.

Idea generation

AI tools can help lawyers generate ideas for their cases by suggesting arguments or solutions based on analysis of existing case law.

Testing the limits of legal AI: GPT-3

How does one test the limits of AI-powered chatbot technologies and their ability to comprehend legal statutes and codes? Two months ago, the AnyLawyer team conducted an experiment. Under our supervision, ChatGPT took an entrance exam for attorneys training in Poland, using real questions from the September 2022 edition of the exam.

Passing the exam is a requirement to begin the 3.5 years of practical training required to become an attorney in Poland. It is a single-choice test consisting of 150 questions based on 51 main legal statutes, including criminal, civil, administrative, constitutional, and European law.

ChatGPT took its time. Due to the system's instability, the exam lasted over three days. The AI achieved a result of 68 points out of 150 (45%). This means that it was 32 points shy of the minimum passing mark, which is 100 points.

A closer look reveals an interesting range of results between subject areas:

• Criminal law - 14 points achieved out of 29 possible (48%);
• Civil law - 28/51 (54%);
• Business and commercial law 9/17 (53%);
• Labor law 1/5 (20%);
• Administrative law 4/18 (22%);
• European and constitutional law - 12/29 (41%).

It is important to note that wrong answers were given to both simple and complex questions, so it does not look like ChatGPT was simply stumped by more complex problems.

We conducted this experiment to find out if machine learning systems like ChatGPT can help lawyers analyze legal cases more quickly and efficiently. The results showed that the ChatGPT of two months ago (using GPT-3) was capable of processing vast amounts of data much faster than humans, yet was not able to replicate the same level of accuracy when it comes to critical aspects of the legal profession, such as analyzing court documents or understanding complex legislation.

A more reliable AI assistant for lawyers: GPT-3.5 and GPT-4

Not long after we had ChatGPT take the entrance exam for attorneys training in Poland (and fail), the GPT-3 model was upgraded to GPT-3.5. This was a game changer. Bing (which uses GPT-3.5), passed with 111 points (out of 150), with clear improvement across all tested areas (criminal law, civil law, business and commercial law, labor law, administrative law, and European and constitutional law).

GPT-4's results were even better. It scored 121 out of 150 points, a result that puts the AI among some of the top-scoring exam takers. For context, the 2022 edition of the exam had a 59.8% passing rate.

These results show us that ChatGPT is quickly becoming a viable assistant for lawyers, who can use it to research case law or draft documents. It is still far from taking over the legal sector completely; however, intelligent tools like ChatGPT should significantly change the way law is taught and practiced.

ChatGPT passes the bar

If our own results are not convincing enough, ChatGPT recently passed the bar exam. OpenAI have been putting GPT-4 to various tests. In the US, the AI system would have a high chance to be admitted to one of the 20 most prestigious law schools, thanks to its 163 LSAT score. It would also pass the Uniform Bar Exam with a score of 298 out of 400, which is enough to be admitted to practice law in all US jurisdictions.

The future of teaching and practicing law

The results of this milestone are encouraging and demonstrate how AI systems can support legal education and the practice of law. Compared to human examinees taking the test, ChatGPT performed better on some parts of the exam while not as well on others, reflecting a nuanced understanding of legal concepts that is an important step forward in the development of AI solutions for legal professionals.

This success holds promise for the future of legal education and practice, as ChatGPT and other AI tools could be used to supplement traditional methods of teaching law and advising clients. AI-enabled technologies should help lawyers answer questions faster, more accurately, and with greater confidence.

This AI revolution in the legal sector has important implications for law schools. The way law is taught today, and the way law students are tested, is based on the cultivation of skills that will soon become obsolete. Memorizing laws which can be accessed almost instantly with the help of a solution like AnyLawyer has very little value. Instead, law schools should be teaching their students to better understand the legal system, allowing them to confidently navigate its intricacies and fluidly adapt to changes. At the same time, law school exams should reflect this change and focus on understanding law rather than merely memorizing it.

Will AI replace lawyers?

It is clear that machines are becoming highly competent at passing legal exams. Does that mean that human lawyers should fear for their jobs? Not likely. Any law firm should, however, prepare themselves for a major shift in approach to practicing law.

Soon, artificial intelligence systems could be reliably summarizing case facts, organizing evidence, drafting legal forms, acting as research partners, and more – simply put, tools like ChatGPT and AnyLawyer are rapidly learning to provide lawyers with valuable assistance in their daily tasks.

What AI means for legal professionals

AI technology is already being successfully used in various fields such as healthcare, finance and transportation. The potential for its use in the legal field is immense and AnyLawyer is at the forefront of developing these solutions, with great care taken towards accuracy, reliability and efficiency.

The legal sector is slowly beginning to recognize the potential of artificial intelligence and ChatGPT, but there is still a long way to go. In the near future, we will continue to explore how emerging AI technologies can help lawyers do their job more efficiently and accurately.

However, we are not ready to blindly trust any legal AI. Consider GPT-4, which was wrong on 29 out of 150, but answered each one with full confidence. Legal professionals using AI should carefully examine each answer they receive and rely on their own judgment. This is good news: it shows us what shape human-AI collaboration is likely to take in the coming months and years.