Supreme Court Confronts GenAI's 'Hallucinations' as PIL Seeks Regulation in Judiciary

New Delhi – The Supreme Court of India is set to deliberate on the profound constitutional and ethical questions posed by Generative Artificial Intelligence (GenAI), as it took up a Public Interest Litigation (PIL) demanding a comprehensive regulatory framework for the use of such technology within the country's judicial and quasi-judicial systems. The hearing, marked by a candid admission from the Chief Justice of India, has brought the intersection of advanced technology and the administration of justice into sharp focus, signaling a pivotal moment for legal-tech governance in India.

A bench led by Chief Justice of India B.R. Gavai and comprising Justice S.V.N. Bhatti was hearing the petition filed by advocate Kartikeya Rawal, which seeks to pre-empt the potential chaos that unregulated GenAI could introduce into a legal system founded on precedent and reason. Acknowledging the pervasive nature of AI-driven misinformation, CJI Gavai made a telling observation: “Yes, yes, we have seen our morphed pictures too.” This remark, though delivered with a hint of irony, underscored the judiciary's awareness of the very real threats AI poses to institutional integrity and public trust.

While expressing an initial inclination to dismiss the plea, the bench ultimately adjourned the matter for two weeks, keeping the critical questions it raises on the judicial docket. The court's decision to further consider the case, Kartikeya Rawal vs Union of India , suggests a recognition of the gravity of the issues at stake.

---

The Core of the Challenge: GenAI's "Hallucinations" and Article 14

The PIL, filed through Advocate-on-Record Abhinav Shrivastava, draws a crucial distinction between conventional AI, which operates on pre-defined parameters, and the more advanced GenAI. The petition argues that GenAI’s capacity to create novel content—including text, images, and code—presents a unique and formidable challenge to the legal process.

At the heart of the petitioner's argument is the phenomenon of "hallucinations," where a GenAI model generates information that is factually incorrect, nonsensical, or entirely fabricated, yet presents it with authority. The plea contends that this could manifest in the form of non-existent case laws, distorted legal principles, or biased judicial observations.

"The skill of Gen AI to leverage advanced neural networks and unsupervised learning to generate new data...can lead to ‘hallucinations’, resulting in fake case laws, AI bias, and lengthy observations," the plea states. "This process of hallucinations would mean that the GenAI would not be based on precedents but on a law that might not even exist. Such arbitrariness is a clear violation of Article 14."

This potential for arbitrariness, the petition argues, directly contravenes the Right to Equality guaranteed under Article 14 of the Constitution. If judicial or quasi-judicial bodies were to rely on such opaque and unpredictable tools, the foundational principles of consistency, fairness, and reasoned decision-making would be fundamentally compromised. The "black box" nature of many GenAI systems—where even their creators cannot fully explain the logic behind a specific output—exacerbates this risk, making it impossible for citizens to understand the reasoning behind a decision, a cornerstone of natural justice.

Broader Constitutional Concerns: Articles 19 and 21

Beyond the violation of Article 14, the PIL highlights the threats to fundamental rights under Articles 19 (Freedom of Speech and Expression) and 21 (Right to Life and Personal Liberty). An opaque, AI-driven judicial process could erode the right to a fair hearing and undermine personal liberty by basing decisions on inscrutable algorithms.

The petition also raises serious concerns about algorithmic bias. Since AI models are trained on vast datasets, they can inherit and amplify existing societal biases present in that data. The plea warns that integrating such systems into judicial functions without rigorous safeguards could lead to discriminatory outcomes, particularly against marginalized communities. To counter this, the petitioner calls for complete transparency in the data used to train AI tools for judicial use and clear accountability for data ownership.

The Relief Sought: A Proactive Governance Framework

Recognizing the imminent risks, the petitioner has sought multi-pronged relief from the Apex Court. The PIL calls for directions to the Union of India to enact a comprehensive law or policy framework for the regulated and uniform adoption of GenAI in judicial and quasi-judicial bodies.

The plea also seeks the formation of an expert committee, comprising jurists, technologists, government officials, and civil society members, to study the impact of GenAI on the justice delivery system. This committee would be tasked with recommending standards for the ethical and responsible deployment of AI.

Furthermore, the petition requests a direction to the Secretary General of the Supreme Court to formulate an internal policy for regulated GenAI use, which could then serve as a model for High Courts, subordinate courts, and tribunals across the country.

The Larger Context: India's Nascent AI Regulation Landscape

The Kartikeya Rawal case does not exist in a vacuum. The Supreme Court is already seized of another PIL, filed by Aarati Sah, which seeks a broader regulatory and licensing framework for all AI tools, with a specific focus on combating "deepfakes." That plea also calls for grievance redressal mechanisms from major tech platforms like Meta and Google to deal with AI-generated impersonations.

These judicial interventions reflect a growing national and global consensus on the need for proactive AI governance. As jurisdictions like the European Union move forward with comprehensive, risk-based AI regulations, the Indian judiciary's engagement with these issues signals an intent to ensure that technological advancement does not outpace legal and ethical safeguards.

The CJI's acknowledgment of morphed judicial images serves as a potent reminder that these are not abstract, future concerns. The technology is already being used to erode trust in public institutions. For the legal profession, the implications are immediate. Lawyers who have begun using GenAI for research are already encountering fabricated citations, a risk that could lead to professional sanctions if not diligently managed.

The Supreme Court's decision to hear this matter after two weeks will be closely watched. Its approach could set a precedent for how India balances the promise of technological efficiency in the justice system with the non-negotiable principles of fundamental rights, transparency, and human oversight. The outcome may shape the future of legal practice, judicial administration, and the very architecture of justice in the digital age.