Supreme Court Examines AI's 'Black Box' Risks for Indian Judiciary

New Delhi – In a hearing that brought the abstract dangers of artificial intelligence into sharp focus, the Supreme Court of India has begun examining a writ petition calling for a comprehensive regulatory framework to govern the use of AI within the country's judicial system. The proceedings took a personal turn when Chief Justice of India BR Gavai revealed he had been a target of AI-generated misinformation, underscoring the urgency of the issues at hand.

The bench, comprising CJI BR Gavai and Justice K Vinod Chandran, was hearing the public interest litigation, RAWAL vs. UNION OF INDIA , which argues that the unchecked integration of AI tools, particularly Generative AI (GenAI), poses a fundamental threat to judicial integrity, transparency, and constitutional principles.

The gravity of the subject was highlighted when the petitioner's counsel began to elaborate on the potential "ills" of AI. The Chief Justice interjected, lending a powerful, real-world example to the debate. “We are aware of it, we have seen the morphed video of us (two),” CJI Gavai remarked, referring to a fabricated video that had circulated on social media. This candid admission from the head of the Indian judiciary signals a high-level awareness of AI's capacity for malicious use and its potential to erode public trust in institutions. The Court has scheduled the matter for a more detailed hearing in two weeks.

The Petitioner's Case: Unpacking the 'Black Box'

The petition, filed with the assistance of Advocate-on-Record Abhinav Shrivastava, meticulously deconstructs the technological and legal risks associated with embedding AI into judicial functions. It moves beyond a general apprehension of technology to pinpoint specific vulnerabilities inherent in current AI models, especially GenAI.

At the heart of the petitioner's argument lies the dual problem of "datafication" and "data opaqueness." The plea explains that GenAI systems learn not from direct programming but by identifying patterns in vast datasets through a process of machine learning. This process, known as "datafication," can inadvertently absorb and amplify existing societal prejudices, embedding "systemic biases" into the very algorithms intended to be neutral.

This leads to the critical issue of the "black box"—a term used to describe AI systems whose internal logic is so complex that it is incomprehensible, even to their own creators. The petition elaborates on this danger:

"The opacity of such algorithms, often described as 'black boxes,' means that even their creators may not fully understand the internal logic, thereby creating the risk of arbitrariness and discrimination, against which could not even be controlled or in the knowledge of the creator."

For the judiciary, where decisions must be reasoned, transparent, and reviewable, such opacity is antithetical. If a judge or legal professional relies on an AI tool whose decision-making process is inscrutable, it becomes impossible to verify the output's fairness, accuracy, or legal validity.

Constitutional and Legal Red Flags

The petitioner contends that these technological flaws translate directly into constitutional violations. The arbitrary and potentially biased outputs from a "black box" system could fundamentally undermine the Right to Equality under Article 14 of the Constitution. If AI tools used for case analysis, evidence processing, or even sentencing recommendations operate on hidden biases, they could lead to discriminatory outcomes that are procedurally unfair.

Furthermore, the plea raises the alarm over AI "hallucinations"—a phenomenon where GenAI confidently generates false or fabricated information. The petition warns this could manifest as "fake case laws and AI-modified court observations which may not be accurate." The implications for legal practice are profound. An advocate citing an AI-generated, non-existent precedent or a judge relying on an AI-summarized but distorted version of a testimony could corrupt the entire judicial process. This, the petitioner argues, would not only alter hearings and decisions arbitrarily but also violate the citizens' Right to Know under Article 19(1)(a), as the basis for judicial determinations would be obscured and potentially falsified.

The petition also highlights the heightened risk of cyberattacks, suggesting that the integration of complex, and at times poorly understood, AI systems into sensitive judicial databases could create new vectors for security breaches.

The Path Forward: A Call for Guardrails

While acknowledging that the Supreme Court itself is exploring the use of AI to enhance efficiency, the petition argues that adoption must be preceded by robust regulation. It calls for a framework ensuring that any AI integrated into the judiciary uses data that is demonstrably free from bias and that data ownership is transparent to ensure accountability.

The case of RAWAL vs. UNION OF INDIA arrives at a pivotal moment. Courts across India are increasingly adopting technology for case management, transcription services (such as the Supreme Court's own SUVAS tool), and legal research. However, the leap from administrative tools to AI systems that assist in substantive legal analysis and decision-making requires a different order of scrutiny.

This litigation compels the legal community to confront a series of challenging questions: 1. Accountability: If an AI tool contributes to a flawed judgment, who is liable? The judge, the AI developer, or the institution that deployed the technology? 2. Transparency: How can the judiciary fulfill its duty to provide reasoned decisions if the tools it uses are inherently opaque? 3. Bias Mitigation: What standards and auditing mechanisms are required to detect and eliminate systemic bias from judicial AI systems? 4. Verification: How can legal professionals and judges trust AI-generated outputs, from case summaries to legal precedents, without an infallible method of verification?

As the Supreme Court prepares to delve deeper into this matter, its conclusions could set a global precedent. The challenge is to craft a framework that harnesses AI's potential for improving access to justice and efficiency while erecting impenetrable safeguards to protect the rule of law, due process, and the fundamental rights of every citizen.