Results
Mask the data, keep the law.
A tool that masks everything leaks nothing, but Claude can do nothing with the document. The right tool masks your clients' data and leaves the law intact. That is what we measure.
Shield against Hexagone and Presidio
Two public benchmarks and two RAGbase benchmarks, all in French · updated on
| Measure | Data set | Shield 0.4.4 | Hexagone 0.3.9 | Presidio 2.2.364 | ||
|---|---|---|---|---|---|---|
| Fastdefault | Precisewith AI model | Fast | Precis | |||
| Overall score note 1 F1 by word, higher is better | ||||||
| Overall score note 1 F1 by word, higher is better | FR-PII-Bench public · French texts | 96.9 (highest F1 in the row)Pprecision 95.0R, recall 98.9 | 96.2Pprecision 93.6R, recall 99.1 | 80.7Pprecision 68.4R, recall 98.4 | 80.7Pprecision 67.7R, recall 99.9 | 66.5Pprecision 69.2R, recall 64.1 |
| MAPA note 2 public · Cour de cassation | 81.1Pprecision 83.1R, recall 79.1 | 82.8 (highest F1 in the row)Pprecision 78.7R, recall 87.3 | 43.8Pprecision 28.5R, recall 95.3 | 37.9Pprecision 23.8R, recall 93.9 | 63.5Pprecision 51.0R, recall 84.1 | |
| REF RAGbase · employment law | 97.3 (highest F1 in the row)Pprecision 97.9R, recall 96.7 | 97.3 (highest F1 in the row)Pprecision 97.9R, recall 96.7 | 66.7Pprecision 50.6R, recall 97.6 | 61.1Pprecision 44.0R, recall 99.9 | 79.2Pprecision 82.5R, recall 76.1 | |
| Max note 2 RAGbase · law-firm files | 94.9 (highest F1 in the row)Pprecision 95.2R, recall 94.7 | 91.5Pprecision 87.9R, recall 95.3 | 54.0Pprecision 38.1R, recall 92.4 | 48.9Pprecision 32.7R, recall 96.2 | 47.4Pprecision 38.0R, recall 63.0 | |
| Legal terms changed articles, courts, laws · lower is better | ||||||
| Legal terms changed articles, courts, laws · lower is better | MAPA public · Cour de cassation | 0.0 % (lowest value in the row) | 0.0 % (lowest value in the row) | 43.1 % | 47.6 % | 18.0 % |
| REF RAGbase · employment law | 4.1 % (lowest value in the row) | 4.1 % (lowest value in the row) | 17.8 % | 47.1 % | 15.3 % | |
| Max RAGbase · law-firm files | 2.1 % (lowest value in the row) | 4.2 % | 47.2 % | 58.3 % | 55.6 % | |
| Speed and memory note 3 no graphics card, lower is better | ||||||
| Speed and memory note 3 no graphics card, lower is better | Time per document of 10 pages | 0.63 s (lowest value in the row) | 17.7 s | 26.93 s | 9 min | 1.59 s |
| Memory peak, same document | 79 MB to 80 MB (lowest value in the row) | 3.7 GB to 3.8 GB | 1.6 GB to 1.7 GB | 3.8 GB to 5.4 GB | 0.57 GB | |
Bold: the highest overall score in each row; elsewhere, the lowest value.
- 1 Overall score = F1, computed by word. It drops when the tool leaves data readable (recall, R) and when it masks text by mistake (precision, P).
- 2 MAPA and Max: precision is a lower limit.
- 3 Test machine: 6 cores, 16 GB, no graphics card.
Source: method and detailed results, public on GitHub.
What the numbers show
The law stays readable
On MAPA, decisions of the Cour de cassation, Shield changes 0.0 % of the legal terms. Hexagone changes 43.1 % in Fast mode and 47.6 % in Precis mode.
The highest overall score
The overall score drops when a tool leaves data readable, and also when it masks text by mistake. Shield has the highest score on the four benchmarks.
An immediate answer
Shield handles a document in 0.63 s. Hexagone takes 26.93 s in Fast mode and 9 min in Precis mode.
The method, in short
- The tools: Shield 0.4.4, Hexagone 0.3.9 and Presidio 2.2.364.
- Four benchmarks in French: two public (FR-PII-Bench, MAPA) and two built by RAGbase (REF, Max).
- Each tool gets the same files. The same program scores every output.
- Test machine: 6 cores, 16 GB, no graphics card.
To go further
The method, the scripts and the detailed results are public on GitHub.
The two sets built by RAGbase, REF and Max, are public, with their answer keys and one command to run the measurement again.