Results

Mask the data, keep the law.

A tool that masks everything leaks nothing, but Claude can do nothing with the document. The right tool masks your clients' data and leaves the law intact. That is what we measure.

Shield against Hexagone and Presidio

Two public benchmarks and two RAGbase benchmarks, all in French · updated on

Comparative trial, summary: word-level F1, with the precision and the recall under each F1, then resources, for Shield 0.4.4 (Fast and Precise), Hexagone 0.3.9 (Fast and Precis) and Presidio 2.2.364. F1, precision and recall: higher is better. Resources: lower is better.
Measure Data set Shield 0.4.4 Hexagone 0.3.9 Presidio 2.2.364
Fastdefault Precisewith AI model Fast Precis
Overall score note 1 F1 by word, higher is better
Overall score note 1 F1 by word, higher is better FR-PII-Bench public · French texts 96.9 (highest F1 in the row)precision 95.0, recall 98.9 96.2precision 93.6, recall 99.1 80.7precision 68.4, recall 98.4 80.7precision 67.7, recall 99.9 66.5precision 69.2, recall 64.1
MAPA note 2 public · Cour de cassation 81.1precision 83.1, recall 79.1 82.8 (highest F1 in the row)precision 78.7, recall 87.3 43.8precision 28.5, recall 95.3 37.9precision 23.8, recall 93.9 63.5precision 51.0, recall 84.1
REF RAGbase · employment law 97.3 (highest F1 in the row)precision 97.9, recall 96.7 97.3 (highest F1 in the row)precision 97.9, recall 96.7 66.7precision 50.6, recall 97.6 61.1precision 44.0, recall 99.9 79.2precision 82.5, recall 76.1
Max note 2 RAGbase · law-firm files 94.9 (highest F1 in the row)precision 95.2, recall 94.7 91.5precision 87.9, recall 95.3 54.0precision 38.1, recall 92.4 48.9precision 32.7, recall 96.2 47.4precision 38.0, recall 63.0
Legal terms changed articles, courts, laws · lower is better
Legal terms changed articles, courts, laws · lower is better MAPA public · Cour de cassation 0.0 % (lowest value in the row) 0.0 % (lowest value in the row) 43.1 % 47.6 % 18.0 %
REF RAGbase · employment law 4.1 % (lowest value in the row) 4.1 % (lowest value in the row) 17.8 % 47.1 % 15.3 %
Max RAGbase · law-firm files 2.1 % (lowest value in the row) 4.2 % 47.2 % 58.3 % 55.6 %
Speed and memory note 3 no graphics card, lower is better
Speed and memory note 3 no graphics card, lower is better Time per document of 10 pages 0.63 s (lowest value in the row) 17.7 s 26.93 s 9 min 1.59 s
Memory peak, same document 79 MB to 80 MB (lowest value in the row) 3.7 GB to 3.8 GB 1.6 GB to 1.7 GB 3.8 GB to 5.4 GB 0.57 GB

Bold: the highest overall score in each row; elsewhere, the lowest value.

  1. 1 Overall score = F1, computed by word. It drops when the tool leaves data readable (recall, R) and when it masks text by mistake (precision, P).
  2. 2 MAPA and Max: precision is a lower limit.
  3. 3 Test machine: 6 cores, 16 GB, no graphics card.

Source: method and detailed results, public on GitHub.

What the numbers show

The law stays readable

On MAPA, decisions of the Cour de cassation, Shield changes 0.0 % of the legal terms. Hexagone changes 43.1 % in Fast mode and 47.6 % in Precis mode.

The highest overall score

The overall score drops when a tool leaves data readable, and also when it masks text by mistake. Shield has the highest score on the four benchmarks.

An immediate answer

Shield handles a document in 0.63 s. Hexagone takes 26.93 s in Fast mode and 9 min in Precis mode.

The method, in short

  • The tools: Shield 0.4.4, Hexagone 0.3.9 and Presidio 2.2.364.
  • Four benchmarks in French: two public (FR-PII-Bench, MAPA) and two built by RAGbase (REF, Max).
  • Each tool gets the same files. The same program scores every output.
  • Test machine: 6 cores, 16 GB, no graphics card.

To go further

The method, the scripts and the detailed results are public on GitHub.

The two sets built by RAGbase, REF and Max, are public, with their answer keys and one command to run the measurement again.

Try Shield on one of your documents