AI Legal Writing Tools: Capabilities and Risks

AI Legal Writing Tools: Capabilities and Risks

Introduction

AI legal writing tools have evolved into integral components of law firm operations, promising faster brief drafting, automated citation checking, and style consistency. Despite their appeal, risks are notable. A Stanford study found that GPT-4 hallucinated between 69% and 88% of the time on legal queries. Research indicates that Westlaw’s AI is accurate only 42% of the time, with a hallucination rate of 33%, while Lexis+ AI is accurate 65% of the time, with a hallucination rate of 17%. Courts have imposed sanctions on attorneys for AI-generated inaccuracies and have issued standing orders requiring AI verification. This guide explores the capabilities and limitations of AI legal tools and offers strategies to mitigate risks of sanctions or malpractice claims.

AI for legal documents offers practical functions with supervision:

  • Brief writing assistance: Structures arguments and suggests improvements.
  • Motion drafting tools: Create drafts from fact patterns and law, saving time.
  • Legal memo generation: Summarizes research for internal review.
  • Citation checking tools: Ensure formatting consistency and flag incomplete references.

These tools use language models trained on vast text, including court opinions. They predict words based on training data patterns, explaining their utility and unreliability. They recognize legal structures but often fail to identify real cases.

Style consistency is a major strength. AI maintains citation format, heading structure, and tone, crucial for paralegals handling large-scale litigation.

AI Legal Writing Tool Capabilities: Understanding AI Legal Writing Capabilities Diagram

The Stanford HAI study reports hallucination rates for AI legal writing programs. GPT-4 fabricated 49%-82% of content, and GPT-3.5 exceeded 88% on certain queries. Purpose-built legal AI, like Westlaw’s Research tool, hallucinated over 34% of queries, and Lexis+ AI showed a 17% rate.

Damien Charlotin documents 486 global cases of AI-generated fabrications as of late 2024, with two to three new cases daily, indicating acceleration of the issue despite cautions.

Landmark Hallucination Cases and Sanctions

AI Hallucination Risk Levels: Landmark Hallucination Cases and Sanctions Diagram

Mata v. Avianca: Attorney Steven Schwartz used ChatGPT to research law, filing six fabricated cases. Schwartz’s failure to verify led to sanctions and an apology order.

Noland v. Land of the Free: California’s first published appellate opinion on the misuse of generative AI in legal filings. The Court of Appeal sanctioned the plaintiff’s counsel $10,000 and directed the clerk to send the opinion to the State Bar of California (CLM Magazine).

These incidents highlight overconfidence in AI accuracy with deadline pressures.

Court-Imposed AI Standing Orders

On May 30, 2023, Judge Brantley Starr of the Northern District of Texas issued a standing order requiring attorneys to certify that generative AI drafted no part of a filing, or that a human checked any AI-drafted language. Many other judges and courts have since issued their own orders, which the RAILS AI orders tracker collects. Common types include:

  • Certification-required orders: Attorneys certify AI avoidance or verification.
  • Disclosure-only orders: Attorneys identify AI-generated portions.
  • Reminder orders: Reiterate Rule 11 obligations without new AI-specific rules.

Judicial practices vary, requiring detailed AI platform and verification disclosure. Orders impact legal staff, demanding thorough checks of compliance.

ABA Formal Opinion 512 and Ethical Obligations

ABA Formal Opinion 512 clarifies AI use in legal practice without creating new rules.

  • Rule 3.3: Verify AI-generated citations to avoid false statements.
  • Rule 1.1: Understand AI capabilities and limitations.
  • Rule 5.3: AI tools require attorney supervision.

These rules emphasize mandatory verification of AI outputs.

Practical Applications and Best Practices

Effective uses of legal writing AI include:

  • Drafting first drafts of routine motions: AI provides framework; human revision is essential.
  • Verifying citation formats: AI ensures Bluebook consistency; separate case verification is crucial.
  • Memo generation for research: Use AI for terms and doctrines; human research for completeness.
  • Maintaining style consistency: AI flags inconsistencies, valuable in large-scale litigation.

AI struggles with niche legal issues due to limited training data.

Citation Verification Protocols

Mandatory protocols are critical for AI use in legal documents:

  1. Confirm case existence through platforms like Westlaw or Lexis.
  2. Match AI quotes with actual text.
  3. Ensure cited propositions are supported.
  4. Use Shepard’s or KeyCite for case validity.
  5. Verify procedural details.

Citation Verification Workflow: Citation Verification Protocols Diagram

Document each verification step for accountability.

Managing Confidentiality and Privilege Concerns

AI raises privilege and confidentiality risks, with potential client data exposure to vendors. Understand vendor data policies, anonymize information, and consider local AI tool deployment. Negotiate vendor contracts to ensure data protection.

The Bottom Line

AI legal tools provide benefits for tasks with rigorous verification. They expedite drafts and enhance formatting, but high hallucination rates demand constant oversight. Court orders and ABA opinions clarify that AI requires comprehensive verification and responsibility.

Legal professionals must balance time savings against verification burdens for AI tools.

Share:
Markdown version
↧
Loading PDF…