A lawyer who wants to use Claude on a client file usually starts with one question: how much should be hidden before the text reaches Claude? The instinct is to mask everything, names, dates, amounts, even the court that issued the ruling. That instinct is wrong. A court name or an article number is not personal data, and once it disappears, Claude no longer reads the same file the lawyer has in hand.
Why court names break Claude’s reasoning
Claude works from the text it receives. If a brief cites “Cour de cassation” and a pseudonymisation tool turns that into a code, Claude is reasoning about a court it cannot name, in a ruling it cannot place. The same happens when an article number or a case reference changes. Claude can still produce an answer, but the answer stops tracking the actual legal reasoning in the file, because the legal reasoning depends on which court said what, under which article, on which date. Masking a name protects a client. Masking a court or a law does not protect anyone, it only removes information Claude needs to help.
Masking everything is easy, but then Claude cannot do the work. Shield masks your clients’ data and keeps the law readable: articles, courts, dates and amounts stay in clear. That is the design choice behind RAGbase Shield, and it is also what the public benchmark results show.
What the benchmarks show about legal terms
RAGbase measured how often each tool changes a legal term, meaning a court name, a law, or an article reference, on the benchmarks of its public results file. Three of the four benchmarks hold legal terms: MAPA, built from decisions of the Cour de cassation, REF, built for employment law, and Max, built from law firm files. On these three, Shield changes 0 to 4.2 % of the legal terms. Hexagone changes between 17.8 and 58.3 % of them, and Presidio between 15.3 and 55.6 %. On MAPA specifically, Shield changes 0 % of the legal terms in both of its modes, while Hexagone changes 43.1 % in Fast mode and 47.6 % in Precis mode, and Presidio changes 18.0 %.
| Benchmark | Shield | Hexagone | Presidio |
|---|---|---|---|
| MAPA | 0 % (both modes) | 43.1 % Fast, 47.6 % Precis | 18.0 % |
| REF | 4.1 % (both modes) | 17.8 % Fast, 47.1 % Precis | 15.3 % |
| Max | 2.1 % Fast, 4.2 % Precise | 47.2 % Fast, 58.3 % Precis | 55.6 % |
When a tool changes the name of a court or the number of an article, the text that Claude receives no longer says what the file says. That is the court name problem in practice: a tool that masks aggressively to look safe ends up handing Claude a document that misquotes its own sources.
The overall score and why masking everything fails it
RAGbase also computes an overall score for each tool, the F1 computed by word, across four benchmarks in French: the two public ones, FR-PII-Bench and MAPA, and two built by RAGbase, REF and Max. The overall score drops when a tool leaves personal data readable, and it also drops when a tool masks text by mistake, including law it should have left alone. A tool that masks everything leaks nothing, but its overall score drops, because it masks the law too. This is why “mask more” is not the same test as “mask well”.
Shield has the highest overall score on the four benchmarks: FR-PII-Bench 96.9, MAPA 82.2, REF 97.1, Max 95.3. On these same four benchmarks, Hexagone scores between 37.9 and 80.7, and Presidio between 47.4 and 79.2.
| Benchmark | Shield Fast | Shield Precise | Hexagone Fast | Hexagone Precis | Presidio |
|---|---|---|---|---|---|
| FR-PII-Bench | 96.9 | 96.2 | 80.7 | 80.7 | 66.5 |
| MAPA | 80.4 | 82.2 | 43.8 | 37.9 | 63.5 |
| REF | 97.1 | 97.1 | 66.7 | 61.1 | 79.2 |
| Max | 95.3 | 91.8 | 54.0 | 48.9 | 47.4 |
The results show two Shield columns. Fast is the default, a local engine that finds names and identifiers with rules and dictionaries. Precise adds a local AI model. Both keep the legal terms readable, which is why both score close together on every benchmark above.
Getting Shield onto a lawyer’s computer
RAGbase Shield is free and open source, under the Apache-2.0 licence. It runs local, reversible pseudonymisation: names and numbers are replaced by codes before the text leaves the computer, and Claude’s answer is put back into clear names afterwards, also on the computer. Nothing goes to RAGbase. The source code, the evaluation scripts and the full results used above are public on GitHub, at https://github.com/ragbase/ragbase-shield/blob/main/evaluation/laptop/RESULTS_PUBLIC.md, so a firm’s AI lead can read the method and check the numbers directly rather than take a vendor’s word for it.
Adding Shield takes no technical skill. On the Install page, click “Add Shield to Claude on Mac” or “Add Shield to Claude on Windows”. In the Claude app, click “Sync”, “Add”, then “Continue”. Nothing else to install. You need the Claude app, with a paid Claude plan.
Then, in a new conversation, write “Protect a folder with Shield” and choose the folder of one client in the Shield window. Claude only ever sees codes for people and companies. The court names, dates, amounts and articles stay exactly as written in the file. Only a folder chosen in the Shield window is protected: do not attach a file or add a folder to the conversation.
One limit matters here, because this is about using the tool day to day: detection is never complete. Read the sensitive passages again, because a name on its own can be missed, and read the final document before you use it.
The best pseudonymisation engine on the market.
The limits of Shield are on the Limits and compliance page.
Sources
https://github.com/ragbase/ragbase-shield/blob/main/evaluation/laptop/RESULTS_PUBLIC.md; https://ragbase.ai