Two AI Teachers Take on the Problem of Learning from Mistakes

Connecting clues across a document

Document-level relation extraction asks software to identify relationships between named entities, such as people, organizations, chemicals or diseases. Unlike sentence-by-sentence analysis, it must bring together evidence scattered throughout a document. Its task is to identify relationships expressed in the text, not establish whether those relationships are true outside it.

Training such systems requires examples with the relevant connections marked. That work becomes demanding when a document contains many entities and dozens of possible relationship types. One alternative is pseudo-labelling: a model supplies provisional annotations for additional documents, creating more material to learn from.

The catch is circularity. If a model learns from its own incorrect guesses, it can reinforce them. Researchers call this confirmation bias.

Khai Phan Tran and Xue Li tackle that problem in Syn2DRE, described in an arXiv preprint posted on November 6, 2026. The supplied source is its first version; it does not establish peer-reviewed publication.

Teachers that change slowly

Syn2DRE uses two student networks, each accompanied by a teacher. In its main setup, one student uses the KD-DocRE architecture, which incorporates attention-based processing of entity relationships. The other, UGDRE, adds a graph connecting relevant document tokens. These differences are intended to provide complementary learning signals.

Each teacher mirrors its student, but its parameters change through an exponential moving average: a running blend that gives substantial weight to earlier settings. Rather than separately training another model from scratch at every stage, the framework maintains a more slowly changing version of the student.

The teachers generate provisional relationship labels. To estimate uncertainty, they process the same input repeatedly with dropout, which temporarily switches off some network components. Predictions that fluctuate less receive priority. Stability is a selection signal, however—not proof of correctness.

The method retains the least uncertain 30% of predicted positive entity pairs and 10% of negative pairs. Treating these groups differently addresses the overwhelming prevalence of pairs without a relationship.

Learning from disagreement

The second ingredient is cross-supervision. Periodically, selected pseudo-labels on which the systems disagree pass to the opposite student. The exchange uses soft labels—distributions of predicted probabilities—rather than only categorical answers.

The idea is that a different architecture may supply information a student would not readily discover through its own teacher. But two models can still share mistakes; disagreement alone does not identify which prediction is right.

The overall procedure includes pre-training on distantly supervised data, whose labels are obtained automatically, followed by a human-labelled warm-up and combined supervised and pseudo-labelled learning. The claimed simplification concerns avoiding repeated denoising-and-retraining cycles, not eliminating training stages or repeated prediction passes.

What the benchmarks show

The authors evaluate five datasets covering general-domain English documents, challenging Chinese examples and biomedical relationships. Their main metric is micro-averaged F1, which balances precision—how often predicted relationships are correct—with recall—how many annotated relationships are recovered.

On Re-DocRED, Syn2DRE reports test F1 of 79.40, compared with 78.14 for UGDRE: a gain of 1.26 points. On DocRED, the reported improvement is smaller, from 65.58 to 66.14; the latter is identified as the best single run rather than an average.

Biomedical results vary sharply. On the chemical–disease CDR benchmark, the reported score rises from the strongest listed baseline’s 76.3 to 84.77. On the gene–disease GDA benchmark, the corresponding gain is only 0.15 points. These are benchmark extraction results, not evidence of clinical usefulness.

Component-removal experiments offer a narrower view of the mechanism. In the reported best-performing Re-DocRED runs, removing cross-supervision reduces test F1 from 79.50 to 79.48. That small difference supports caution about attributing the overall improvement primarily to exchanging disagreements.

Faster training has a price

The paper reports 26 hours of training versus UGDRE’s 55 hours, while using 30,480 rather than 101,873 unlabelled documents. Because data quantities differ, this is a comparison of reported training setups, not a controlled measurement of architectural speed alone. Experiments used an NVIDIA H100 with 80GB of memory.

The authors also acknowledge nearly double UGDRE’s GPU memory usage, continued dependence on labelled examples and fixed pseudo-label selection thresholds. For three benchmarks lacking unlabelled data, validation documents were used without their labels—an evaluation detail worth retaining when comparing results.

Syn2DRE therefore offers evidence for a practical training strategy, not a guarantee of reliable machine annotation. Its promise lies in combining cautious label selection with complementary models; its limits remain visible in memory demands, uneven gains and the possibility of shared errors.