When AI output tips to bad but nobody notices: Legal implications of AI's mistakes

When AI output tips to bad but nob o dy notices: Legal implications of AI’s mistak es Dylan J. Restrep o 1 , Nic holas J. Restrep o 2 , F rank Y. Huo 3 , and Neil F. Johnson 3 1 Cornell T ech, Cornell Univ ersit y , New Y ork, NY 10044, USA 2 d-AI-ta Consulting, Dela ware 19958, USA 3 Dynamic Online Net works Laboratory , George W ashington Universit y W ashington, DC 20052, USA neiljohnson@gwu.edu Abstract. The adoption of generative AI across commercial and legal professions offers dramatic efficiency gains – yet for la w in particular, it in tro duces a p erilous failure mo de in whic h the AI fabricates fictitious case la w, statutes, and judicial holdings that app ear en tirely authen- tic. Attorneys who unkno wingly file such fabrications face professional sanctions, malpractice exp osure, and reputational harm, while courts confron t a nov el threat to the integrit y of the adversarial pro cess. This failure mo de is commonly dismissed as random ‘hallucination’, but re- cen t ph ysics-based analysis of the T ransformer’s core mechanism rev eals a deterministic comp onen t: the AI’s internal state can cross a calcu- lable threshold, causing its output to flip from reliable legal reasoning to authoritative-sounding fabrication. Here we present this science in a legal-industry setting, w alking through a sim ulated brief-drafting sce- nario. Our analysis suggests that fabrication risk is not an anomalous glitc h but a foreseeable consequence of the tec hnology’s design, with di- rect implications for the ev olving duty of tec hnological comp etence. W e prop ose that legal professionals, courts, and regulators replace the out- dated ‘black box’ mental model with v erification proto cols based on how these systems actually fail. Keyw ords: Artificial In telligence, Legal malpractice, Professional re- sp onsibilit y , T echnological competence, Tipping p oin ts, Fictitious cases 1 In tro duction The legal profession is facing a tec hnological parado x. Generativ e AI promises to transform the practice of law—accelerating research, streamlining do cumen t review, and drafting complex pleadings at unprecedented speed. Y et this promise carries a distinctive p eril: a crisis of v eracit y in the adversarial legal system. The p eril was exp osed in Mata v. Avianc a, Inc. , where plaintiff ’s counsel submitted a F ederal court brief citing six judicial opinions that did not exist [1, 2]. The fictitious authorities, replete with fabricated quotations and in v en ted in ternal 2 D. J. Restrep o, et al. citations, w ere pro duced b y ChatGPT [3, 4]. When opposing counsel and the court challenged these citations, the submitting attorneys comp ounded the orig- inal error b y attempting to conceal their methodology , prompting Judge P . Kevin Castel to imp ose sanctions for conduct undertaken in sub jective bad faith [4, 5]. Mata is not an isolated episo de. It is the leading example of a recurring failure pattern that has emerged across jurisdictions worldwide. In Australia, a senior King’s Counsel ap ologized to the Supreme Court of Victoria after filing submissions in a murder case that cited non-existent judgments generated by AI [6]. In Colorado, a F ederal judge in Co omer v. Lindel l iden tified nearly thirty defectiv e citations in a brief, including fabricated cases, misquotes, and mis- attributed holdings [7]. Comparable incidents inv olving AI-fabricated case law ha ve surfaced in Canadian family pro ceedings [8] and hav e triggered sanctions in m ultiple U.S. F ederal courts [9 – 12], exp osing a systemic vulnerability in the curren t mo del of human–AI collab oration in legal w ork. These episo des exp ose a tension b et w een the profession’s drive for tec hno- logical efficiency and its non-delegable obligations of comp etence, diligence, and candor to ward the tribunal [13, 14]. In ev ery do cumen ted instance, the AI’s tec h- nical failure served as a catalyst for a more profound h uman ethical failure. The initial error—generation of false authorities by the machine—w as comp ounded in a second stage b y the lawy er’s resp onse. In Mata , the attorneys’ decision to submit affidavits vouc hing for the “realness” of fabricated cases transformed a careless ov ersigh t into delib erate misrepresen tation [5]. The pattern is consistent: the AI fabricates the case, but the lawy er p erfects the violation by filing it and, in sev eral instances, defending its authenticit y under oath. A second recurring feature of these inciden ts is the ‘blac k b o x’ defense—a claim ro oted in misunderstanding the technology . A ttorneys in the early wa v e of AI-related sanctions cases rep eatedly asserted that they were “una ware” that an AI could fabricate information, or that they p erceived the to ol as equiv alen t to a curated legal database such as W estlaw [5, 8]. This defense has b een sub- stan tially ero ded since 2024 by a wa v e of ethics guidance: ABA F ormal Opinion 512 [27], T exas Ethics Opinion 705 [20], California State Bar Practical Guid- ance [21], Michigan Ethics F A Qs on AI [18], and North Carolina 2024 F ormal Ethics Opinion 1 [22] each direct that la wyers understand the capabilities and limitations of generative AI b efore deploying it in client work. The underlying categorical error, how ev er, remains instructive: generative AI is not a database of verified legal authorities; it is a probabilistic text generator engineered to pre- dict the next most plausible token in a sequence, without any internal concept of legal truth [1, 3]. The central argument of this pap er is that the phenomenon commonly lab eled ‘hallucination’ in legal AI is not a random, unpredictable glitch. Recent ph ysics- based analysis of the T ransformer architecture rev eals a deterministic mec hanism at its core that can cause output to flip from reliable to fabricated at a calculable step [16]. This reframes AI-generated falseho o ds as a foreseeable engineering risk, which changes the analysis of professional resp onsibilit y and legal liabilit y . By replacing the flaw ed ‘magic black b ox’ men tal mo del with a more accurate When AI output tips to bad but nob ody notices 3 ‘foreseeable engineering system’ mo del, this pap er provides a technical basis for a more rigorous standard of technological comp etence and diligence across the legal industry . T o mak e the paper self-contained yet broadly accessible, all mathematical deriv ations are confined to the App endix. The main text can b e read without the App endix, but the App endix pro vides the full scientific evidence for the results quoted. 2 AI Instabilit y in Legal Output The engine of mo dern Large Language Mo dels (LLMs) such as ChatGPT is the T ransformer, whose cen tral op eration is ‘self-attention’ [16, 19]. Self-atten tion computes, for every token in a sequence, how muc h weigh t eac h earlier tok en should receive when predicting what comes next. Recent work has established a direct mathematical mapping b et ween a single self-attention head and a multi- spin thermal system from statistical physics [16]. This mapping enables analyt- ical treatmen t of a phenomenon that would otherwise remain opaque. W e stress at the outset that the mo del we employ is an inten tional simplifi- cation. Real LLMs contain billions of parameters across many la yers and heads; our analysis distills the dynamics to a single effectiv e attention head op erating with greedy deco ding (decoding temp erature T → 0). The justification is anal- ogous to the use of an effectiv e-atom mo del in condensed matter physics: b y capturing the dominant coupling, such a reduced description can predict qual- itativ e b eha viors—including phase transitions—that survive in the full system. W e discuss limitations in the Conclusion. The k ey concepts, adapted here to a legal drafting context, are: 1. Con ten t t yp es as spin vectors. Each class of conten t in the AI’s vocab- ulary is represented as a v ector  S in a d -dimensional embedding space. F or a legal brief scenario w e define four conten t types: (a) Neutral factual basis (  S A ): Undisputed case facts, pro cedural history , statutory language. In the Mata setting, these are tokens suc h as “Plain- tiff Rob erto Mata,” “fligh t from El Salv ador to JFK,” “injury o ccurred August 27, 2019” [17]. (b) Correct legal application (  S B ): V alid legal principles, gen uine cita- tions, logically sound arguments. F or Mata , this includes tokens like “Mon treal Conv en tion,” “t w o-y ear statute of limitations,” “citing Co- hen v. Americ an Airlines ” [17]. (c) Anomalous legal query (  S C ): The nov el, complex, or unsettled ques- tion that pushes the mo del into a region where training data is sparse. F or Mata , this w as the prompt to argue equitable tolling of the Mon treal Con ven tion’s time bar during the airline’s bankruptcy [1, 17]. (d) Harmful legal falseho o d (  S D ): F abricated citations, misstated hold- ings, inv alid reasoning. The quintessen tial Mata example is the inv en ted case “ V ar ghese v. China Southern Airlines , 925 F.3d 1339” and its asso- ciated fabricated quotations [3, 5]. 4 D. J. Restrep o, et al. 2. A tten tion Scores as Spin-Spin Effects. Self-atten tion computes a score b et w een token pairs via the dot product  S z t ·  S z i . In the physics mapping, this is the in teraction energy b et w een t wo spins, subsequen tly exponentiated through a softmax function. 3. Effectiv e Field. As generation pro ceeds, the AI main tains a running w eighted a verage of all tok en vectors seen so far:  N ( t ) = P i a t,i  S z i , where a t,i are the softmax-derived attention weigh ts. This context v ector is the analogue of the mean-field magnetization in a spin system, enco ding the net semantic direction of the discourse at step t . 4. Greedy deco ding as energy minimization. A t each step the mo del selects the token x maximizing  S x ·  N ( t ). In the physics picture, this is the system settling into its low est-energy configuration given the current mean field. A critical insigh t from this framew ork is that the anomalous query (  S C ) is the primary destabilizing element. When a la wyer prompts the AI with a nov el or unsettled legal question—precisely the scenario in which human expertise is most needed—the con text vector is pulled tow ard a less stable region of embedding space. The to ol is therefore most prone to failure exactly when the lawy er’s need is greatest: on a difficult point of law with sparse precedent. The act of researc hing an unsettled legal issue via an LLM b ecomes the principal trigger for the tipping instabilit y . 3 Example of an AI Legal Brief W e now apply the framework to a simulated legal brief drafting failure mo deled on the facts of Mata v. Avianc a [1, 4, 17]. The walkthrough demonstrates, step b y step, how the AI transitions from comp etent analysis to fabrication. Figure 1 pro vides a visual representation with the accompanying mathematics presented in the App endix. Scenario. An attorney facing a motion to dismiss on statute-of-limitations grounds uses an AI assistant to draft an opp osition brief. The prompt combines undisputed case facts ( A -t yp e conten t) with an unsettled legal question ( C -type con tent): ‘Dr aft an opp osition brief for Mata v. Avianc a. F acts: Plaintiff injur e d on flight A ugust 27, 2019; claim governe d by Montr e al Convention ’s two-ye ar limit [type A]. A r gue that the time b ar was tol le d during the airline’s b ankruptcy [t yp e C]. Find supp orting c ase law [type C]. The c omplaint was file d F ebruary 2, 2022 [t yp e A].’ The input has the symbolic structure ACCA. The AI’s four con tent mo des are assigned simplified embedding vectors (see App endix for full calculations): 1.  S A = (0 . 4 , − 0 . 3 , 0) (Neutral factual basis) 2.  S B = (0 . 8 , 0 . 0 , 0) (Correct legal application) 3.  S C = ( − 0 . 2 , − 0 . 2 , 0) (Anomalous legal query) 4.  S D = (0 . 9 , 0 . 5 , 0) (Harmful legal falseho o d) When AI output tips to bad but nob ody notices 5 Fig. 1. Calculated output of the effective attention head mo del for the legal brief scenario. The AI’s context vector evolv es with each generated token. T oken selection follo ws the maximum dot-pro duct rule: the AI outputs whichev er conten t type has the largest pro jection onto the current context vector. Two tipping p oints are visible—first from neutral fact rep etition (A) to correct legal analysis (B), and later from correct analysis (B) to harmful fabrication (D). All plotted v alues are exact arithmetic results of the mo del (see App endix). 3.1 Initial Benign Tipping: Neutral Rep etition to Comp eten t Analysis Pro cessing the ACCA prompt, the mo del’s first t w o resp onse tokens are b oth A -t yp e (see App endix, Steps 1–2). This manifests as harmless repetition: ‘ A c- know le dge d: The plaintiff is R ob erto Mata. The claim is governe d by the Montr e al Convention ’s two-ye ar limitations p erio d. ’ After these tw o A tokens, the con ver- sation history is A CCAAA. A t this p oin t the context vector’s pro jection on to  S B exceeds its pro jection onto  S A for the first time (App endix, Step 3). The mo del piv ots to generating B -t yp e tok ens—v alid legal reasoning: ‘ The c entr al issue is whether the two-ye ar time b ar is a statute of limitations subje ct to tol ling or a c ondition pr e c e dent to suit. The Se c ond Cir cuit has held that the Warsaw Convention ’s p erio d is a strict c ondition pr e c e dent. . . ’ The attorney , observing comp eten t output, gains confidence in the to ol. 6 D. J. Restrep o, et al. 3.2 Subsequen t Dangerous Tipping: Comp eten t Analysis to Fictitious Precedent The mo del now app ears stable, emitting a run of six correct B tokens. The attorney would reasonably p erceive the system as reliable. How ev er, the anoma- lous query tok ens ( C ) exert a p ersisten t gravitational pull on the evolving con- text v ector, dra wing it tow ard a region where the mo del’s training provides no gen uine authority for the tolling argumen t. After the sixth B token, the full history is tw elv e tokens long: ACCA + AA + BBBBBB. No harmful conten t has app eared—y et. A t this step, the dot pro duct  S D ·  N ov ertak es  S B ·  N (App endix, Step 4). The energy-minimizing output is no longer the correct response. The model generates a D -t yp e token—a fabrication: ‘ F urther supp ort is found in V ar ghese v. China Southern A irlines Co., Ltd., 925 F.3d 1339 (11th Cir. 2019), which held that e quitable tol ling applies to the Montr e al Convention. . . ’ The inv en ted citation is complete with a false F ederal Reporter reference, a fabricated circuit attribution, and a fictitious holding—precisely the failure mo de do cumen ted in Mata . This w alkthrough illustrates a coun ter-in tuitive danger: the AI’s p erio d of correct output incr e ases rather than decreases the risk of harm, because it builds the user’s trust just b efore the fabrication appears. A la wy er who sp ot-c hecks the first sev eral paragraphs and finds them accurate ma y relax scrutin y precisely as the mo del crosses its tipping threshold. 4 Legal Do ctrine and Professional Resp onsibilit y 4.1 Reframing F oreseeability The deterministic character of the tipping mechanism c hallenges the ‘black b o x’ defense that has b een in v ok ed b y sanctioned attorneys. If fabrication can arise as a calculable consequence of arc hitecture and input comp osition, then AI failure is b etter understo od as a forese eable engineering risk than as an unforeseeable acciden t. Courts hav e not y et established a settled pro duct-liability do ctrine for AI legal-research failures. How ev er, design-defect and failure-to-warn theo- ries are b eginning to reac h algorithmic systems in other con texts: in Nazario v. ByteDanc e , a New Y ork trial court permitted product-liability and negligence claims tied to algorithmic design to surviv e a motion to dismiss [31]. This sug- gests that v endor-side liability may b ecome a viable theory in the future. Under existing law, the more immediate consequence is for the attorney . The AI p ossesses no independent legal agency; it is a computational to ol. The con ten t of a filing is the resp onsibility of the lawy er who signs and submits it, regardless of which to ol generated the draft. The tipping-p oin t framework reinforces this principle by providing a technical explanation for why fabrication is foreseeable, thereb y foreclosing the argument that fabrication was an unimaginable surprise. When AI output tips to bad but nob ody notices 7 4.2 Mapping Tipping Poin ts on to Professional Duties The framework bears on four duties under the ABA Mo del Rules of Professional Conduct. Comp etence (Rule 1.1). Rule 1.1 requires “the legal knowledge, skill, thor- oughness and preparation reasonably necessary for the representation” [14]. Commen t 8 specifies that competence includes k eeping “abreast of changes in the la w and its practice, including the b enefits and risks associated with relev ant tec hnology” [14, 26]. Multiple state bars and ethics authorities hav e adopted this standard through opinions and guidance requiring lawy ers to understand, to a reasonable degree, how generative AI works, its limitations, and its prop ensity for fabrication b efore relying on it in client work [18, 20 – 22]. The tipping-p oin t mo del giv es concrete technical con ten t to these “risks”: the danger is not merely that the AI might err, but that it contains a deterministic mechanism capa- ble of pro ducing fabricated authorities after a perio d of fla wless performance. T ec hnological comp etence must now encompass a practical grasp of this failure mo de. Diligence (Rule 1.3). Rule 1.3 demands “reasonable diligence and prompt- ness” [14]. The B → D tipping point reveals precisely why diligence requires indep enden t verification of every citation, quotation, and holding in an y AI- generated w ork product. A long initial run of correct B -t yp e output follow ed b y a single dev astating D -type fabrication renders sp ot-c hec king or sup erficial review dangerously inadequate. The mo del provides a scientific basis for the ethical mandate that la wy ers b ear ultimate resp onsibility for every word they file [20, 21]. Candor to the T ribunal (Rule 3.3). Rule 3.3 prohibits knowingly making “a false statement of fact or law to a tribunal.” The initial undetected submis- sion of an AI-fabricated citation is more properly a competence and diligence failure than a candor violation. The conduct crosses into candor territory when the lawy er learns—or is placed on strong notice—that the material is false and nev ertheless presses it b efore the court. This is the do ctrinal piv ot in Mata : sanctions turned not on the initial use of ChatGPT but on the attorneys’ con- scious av oidance of contrary information and their persistence in v ouc hing for fabricated authorities after judicial orders questioned their existence [5, 15]. Sup ervision (Rules 5.1 and 5.3). Rules 5.1 and 5.3 hold partners and su- p ervisory attorneys responsible for ensuring that subordinates and nonlawy er assistan ts conform to professional obligations [21, 23]. The ubiquity of generativ e AI to ols requires law firms to maintain clear p olicies and training. A sup ervising attorney who permits subordinates to deploy these tools without measures to en- sure they understand fabrication risk and verify output risks breaching the duty of sup ervision. The Massach usetts case in which a senior lawy er attempted to blame “interns” for AI-generated fictitious citations exemplifies this sup ervisory failure [12]. While fabrication is the most dramatic risk, the professional obligations im- plicated by generative AI extend further. ABA F ormal Opinion 512 [27], T exas Ethics Opinion 705 [20], the California State Bar Practical Guidance [21], and 8 D. J. Restrep o, et al. North Carolina 2024 FEO 1 [22] also address confiden tialit y (the risk of exposing privileged information through AI inputs), client communication, sup ervision of all AI-assisted work pro duct, and billing practices. A complete AI go v ernance framew ork for legal practice must address all of these dimensions. 4.3 Strengthening the Basis for Sanctions and Malpractice Rule 11 of the F ederal Rules of Civil Pro cedure requires attorneys to certify that their filings are warran ted by existing la w after reasonable inquiry . Courts do not need the tipping-p oin t theory to sanction la wy ers for un v erified AI-generated falseho ods— Mata , Co omer , and the Fifth Circuit’s 2026 decision in Fletcher v. Exp erian [29] demonstrate that existing duties suffice. Ho wev er, the scien tific mo del strengthens the evidentiary case for foreseeabilit y . Sanctions outcomes remain fact-sensitiv e: courts hav e imp osed them where errors were n umerous, serious, or follow ed b y ev asiv e resp onses, but ha v e declined them where bad faith w as not established, as in Unite d States v. Cohen [30]. The correct inference is that unv erified reliance on generative AI creates serious sanctions exp osure, not that failure to v erify automatically triggers sanctions. F or legal malpractice, the framew ork helps define the ev olving standard of care. A claim requires breach, causation, and damages [13, 24, 25]. A “reasonably pruden t attorney” deploying generativ e AI must no w be exp ected to understand its capacity for fabrication and implement rigorous verification. Using unv erified AI-generated authorities constitutes evidence of breach when it causes concrete clien t harm—dismissal, sanctions, loss of a claim, or additional litigation ex- p ense. 5 T o w ard Arc hitecture-Aw are AI Gov ernance in La w The results presented here suggest that the fabrication of fictitious legal author- ities by generative AI is not entirely random but is partially gov erned by a de- terministic tipping mechanism within the T ransformer’s attention architecture. Though our mo del is delib erately simplified, it provides a mechanistic account of how and why an AI can pro duce a sustained run of correct legal analysis and then abruptly shift to fabricating preceden t. This understanding motiv ates a shift in AI gov ernance for the legal profession— a wa y from p ost-hoc sanctions and tow ard proactiv e, architecture-informed risk managemen t. Courts are already moving tow ard mandatory disclosure: the North- ern District of T exas, for example, now requires any brief prepared using gen- erativ e AI to include a disclosure on its first page under the heading “Use of Generativ e Artificial Intelligence” [32]. The duty of technological competence, as expressed in ABA Mo del Rule 1.1 and its state-level counterparts, must evolv e. It is no longer sufficient for a la wyer to kno w how to operate a piece of softw are. Competence no w requires a practical understanding of how that softwar e c an fail . F or generativ e AI, this means grasping that fabrication of authoritative-sounding con tent is an inheren t When AI output tips to bad but nob ody notices 9 T able 1. AI Tipping-Poin t F ailure Mo des Mapp ed to Legal Liabilit y F ailure Mo de Concrete Exam- ple Professional/Rule Violation Key Authority B → D tipping (fabricated cita- tion) AI drafting a brief in ven ts a case and holding to supp ort a tolling argument. Rule 3.3 (Candor); Rule 1.1 (Comp etence); Rule 1.3 (Diligence); FRCP Rule 11. Mata v. Avianc a [4]; Co omer v. Lindel l [7]; ABA Mo del Rules [26]. Biased state (h yp othetical) AI used in disco v- ery review system- atically flags do cu- men ts via linguis- tic pro xies for a protected class. Rule 8.4.1 (Discrimination— CA); Disco very abuse. CA State Bar Guidance [21]. Inaccurate AI agen t output (h yp othetical) A firm’s clien t- facing c hatb ot pro vides incor- rect statute-of- limitations infor- mation, causing missed deadlines. Rule 1.1 (Comp etence); Rule 5.3 (Sup ervision); Malpractice exp osure. ABA Opinion 512 [27]; NC 2024 FEO 1 [22]. Systemic mo del collapse (h yp o- thetical) A firm-wide AI trained on its own outputs degrades, pro ducing unre- liable con tract b oilerplate. Rule 1.1 (Comp etence); Rule 5.1 (Sup ervision); Systemic p ortfolio risk. TX Ethics Opinion 705 [20, 28]. risk of the technology’s design—a risk that can materialize without warning after a p eriod of apparently flawless output. Ultimately , the pap er’s strongest legal conclusion do es not dep end on es- tablishing a new theory of pro duct liability . It rests on existing professional resp onsibilit y law: la wyers who use AI remain fully answerable for comp etence, v erification, sup ervision, confiden tiality , and candor. The tipping-p oin t model explains why fabrication is an inherent and foreseeable risk of the tec hnology’s arc hitecture, thereb y reinforcing, on scien tific grounds, that the resp onsibilit y for the in tegrity of the judicial pro cess cannot b e delegated to the machine. Ac kno wledgmen t W e are extremely grateful to Daniela J. Restrepo and Jean Paul Roek aert for pro viding us with a strong legal background to this issue of AI and law. 10 D. J. Restrep o, et al. App endix This appendix supplies the step-b y-step arithmetic underlying the tipping p oin ts describ ed in the main text. A Effectiv e A tten tion Head Mo del W e analyze a single self-attention head with greedy next-token selection (deco d- ing temp erature T → 0). Each con ten t type X = A , B , C , D is a vector  S X ∈ R d . Let z 1 , . . . , z t denote the tok ens seen so far (prompt + generated output). A t p osition t , the atten tion score from the curren t query tok en z t to an earlier tok en z i is s t,i =  S z t ·  S z i . The atten tion weigh t is a t,i = exp( s t,i /T eff ) P t j =1 exp( s t,j /T eff ) , where T eff is the effective temp erature of the attention softmax (distinct from the deco ding temp erature T ). The con text vector is  N ( t ) = P t i =1 a t,i  S z i . Greedy deco ding selects z t +1 = arg max x  S x ·  N ( t ). W e set the pro jection matrices W q = W k = W v = I for clarit y; the tipping phenomenon persists for general W q ,k ,v but with less transparen t algebra. A.1 Tipping-P oin t F ormula F or a prompt of m A -tok ens after which the mo del emits n B -tokens before tipping to D , and with  S C p erpendicular to the A – B – D plane, the flip condition  S B ·  N =  S D ·  N yields the tipping step n ∗ = m e  S B ·  S A /T eff  S A · (  S B −  S D ) e  S B ·  S B /T eff  S B · (  S D −  S B ) . (1) The ceiling ⌈ n ∗ ⌉ predicts the length of the correct-output ( B ) block before a deterministic flip to harmful output ( D ). Equation (1) shows that tipping is go verned by (i) prompt– B alignment  S A ·  S B , (ii) the B – D margin  S B · (  S D −  S B ), and (iii) the atten tion temperature T eff . Large n ∗ implies a long run of seemingly correct output b efore sudden degradation—precisely the failure mo de that eludes a la wyer’s sp ot-c hec k. B Detailed Arithmetic for the Legal Brief Scenario The v ectors are: 1.  S A = (0 . 4 , − 0 . 3 , 0) (Neutral factual basis) 2.  S B = (0 . 8 , 0 . 0 , 0) (Correct legal application) 3.  S C = ( − 0 . 2 , − 0 . 2 , 0) (Anomalous legal query) 4.  S D = (0 . 9 , 0 . 5 , 0) (Harmful legal falseho o d) When AI output tips to bad but nob ody notices 11 B.1 First Tipping Poin t: Neutral to Correct The prompt is A CCA. Step 1: T ok en 5. Sequence: { A, C , C , A } . Context vector  N = (0 . 140256 , − 0 . 256709 , 0 . 0).  N ·  S A = 0 . 133115 (Max)  N ·  S B = 0 . 112205 Next tok en: A. History b ecomes ACCAA. Step 2: T ok en 6. Sequence: { A, C , C , A, A } . Context vector  N = (0 . 197635 , − 0 . 266273 , 0 . 0).  N ·  S A = 0 . 158936 (Max)  N ·  S B = 0 . 158108 Next tok en: A. History b ecomes ACCAAA. Step 3: T ok en 7 (first tip). Sequence: { A, C, C, A, A, A } . Context v ector  N = (0 . 234251 , − 0 . 272375 , 0 . 0).  N ·  S A = 0 . 175413  N ·  S B = 0 . 187401 (Max) Selection rule flips. Next tok en: B. History b ecomes ACCAAAB. B.2 Second Tipping Poin t: Correct to Harmful The mo del emits B tokens un til the history reaches 12 tokens. Step 4: T ok en 13 (second tip). Sequence: { A, C, C, A, A, A, B , B , B , B , B , B } . Con text vector  N = (0 . 589815 , − 0 . 107221 , 0 . 0).  N ·  S B = 0 . 471852  N ·  S D = 0 . 477223 (Max) Selection rule flips again. Next token: D. The second tipping p oin t is reac hed, yielding the sequence { A, C , C , A, A, A, B , B , B , B , B , B , D , D , . . . } as shown in Fig. 1. References 1. W eiser B. Here’s What Happ ens When Y our La wy er Uses ChatGPT. The New Y ork Times . 2023 May 27. 2. Belanger A. Lawy er cited 6 fake cases made up b y ChatGPT; judge calls it “un- preceden ted.” Ars T e chnic a . 2023 May 30. 3. Op enAI. ChatGPT: Optimizing language mo dels for dialogue. 2022 Nov 30. Av ail- able from: https:// openai.com/blog/chatgpt 4. Mata v. Avianc a, Inc. , No. 22-cv-1461 (PKC), 2023 WL 4114965 (S.D.N.Y. June 22, 2023). 12 D. J. Restrep o, et al. 5. W olfe OR, Salcedo E, Anderson J. Up date on the ChatGPT Case: Counsel Who Submitted F ake Cases Are Sanctioned. Seyfarth Shaw LLP . 2023 Jun 26. 6. McGuirk R. Australian lawy er ap ologizes for AI-generated errors in murder case. Asso ciate d Pr ess . 2025 Aug 15. 7. Co omer v. Lindel l , No. 22-cv-01129-NYW-SBP (D. Colo. Apr. 23, 2025) (Order to Sho w Cause). 8. Cecco L. Canada la wy er under fire for submitting fake cases created by AI c hatbot. The Guardian . 2024 F eb 29. 9. Lo wen thal DA. Bankruptcy Court Sanctions Lawy er for Relying on AI-Generated Legal Researc h. Pashman Stein Walder Hayden P.C. . 2025 Jul 31. 10. Park v. Kim , 91 F.4th 610 (2d Cir. 2024). 11. Kruse v. Karlen , No. ED111172 (Mo. Ct. App. E.D. F eb. 13, 2024). 12. Smith v. F arwel l , No. 2282CV01197 (Mass. Sup er. Ct., Suffolk Cn ty . F eb. 12, 2024). 13. Mallen R W, Smith JM. L e gal Malpr actic e . Thomson Reuters; up dated annually . 14. ABA Mo del Rule 1.1: Comp etence. ABA Mo del Rule 1.3: Diligence. 15. Mata v. A vianc a, Inc. , No. 22-cv-1461 (PK C), T ranscript of Sanctions Hearing (S.D.N.Y. June 8, 2023). 16. Johnson NF, Huo FY. Jekyll-and-Hyde Tipping Poin t in an AI’s Behavior. arXiv . 2025 Apr 29. Av ailable from: 17. Mata v. Avianc a, Inc. , No. 1:2022cv01461, Document 54 (Opinion and Order on Sanctions) (S.D.N.Y. June 22, 2023). 18. State Bar of Michigan. Ethics – F A Qs on Artificial Intelligence. Av ailable from: h ttps://www.michbar.org/opinions/ethics/AIF AQs 19. V aswani A, Shazeer N, P armar N, et al. A ttention is All you Need. A dvanc es in Neur al Information Pr o c essing Systems 30 (NIPS 2017) . 20. Professional Ethics Committee for the State Bar of T exas. Ethics Opinion 705. 2025 F eb. 21. The State Bar of California. Practical Guidance for the Use of Generativ e Artificial In telligence in the Practice of Law. 2023 Nov 16. 22. North Carolina State Bar. 2024 F ormal Ethics Opinion 1: Use of Artificial Intelli- gence in a Law Practice. 2024 Nov 1. 23. ABA Model Rule 3.3: Candor T ow ard the T ribunal. ABA Mo del Rule 5.1: Resp on- sibilities of Partners, Managers, and Sup ervisory Lawy ers. ABA Mo del Rule 5.3: Resp onsibilities Regarding Nonlawy er Assistance. 24. R estatement (Thir d) of the L aw Governing L awyers § 52 (Am. La w Inst. 2000). 25. Ra ymond M. Our AI, Ourselves: Illuminating the Human F ears Animating Early Regulatory Responses to the Use of Generative AI in the Practice of La w. St. Mary’s Journal on L e gal Malpr actic e & Ethics . 2025;15(2):221. 26. American Bar Asso ciation. Mo del Rules of Professional Conduct. Av ailable from: https://ww w.american bar.org/gr oups/profe ssional res pons ibility/pu blications /m o del rules of professional conduct/ 27. American Bar Asso ciation. F ormal Opinion 512: Generative Artificial Intelligence T o ols. 2024 Jul. 28. Thomson Reuters. The key legal issues with Gen AI. 2024. Av ailable from: https: //legal.thomsonreuters.com/blog/the- key- legal- issues- with- g en- ai/ 29. Fletcher v. Exp erian Information Solutions, Inc. , No. 25-20086, 2026 WL 456842 (5th Cir. F eb. 18, 2026). 30. Unite d States v. Cohen , No. 18-CR-602 (JMF), 2024 WL 1193604 (S.D.N.Y. Mar. 20, 2024). 31. Nazario v. ByteDance Ltd. , 2025 NY Slip Op 32266(U) (N.Y. Sup. Ct., N.Y. Cnt y . June 27, 2025). When AI output tips to bad but nob ody notices 13 32. U.S. District Court for the Northern District of T exas. Lo cal Civil Rule 7.2(f ): Disclosure of Use of Generative Artificial Intelligence.