Teaching AI to Read Chemical Diagrams by Judging Whole Molecules

Turning a chemical diagram into data is a little like reading a map whose labels are written in a specialized code. A computer must identify atoms and bonds, then represent their connections in a format such as SMILES, a text notation for molecules, or a molecular graph. A misplaced bond or ring marker can change the structure—or make the result chemically invalid.

The task is especially difficult in patents and scientific papers, where diagrams may include abbreviations such as “Me,” unusual notation, nearby text and scanning artifacts. Models trained on clean, computer-generated diagrams may struggle with this variety.

A paper by Zhuoqi Lyu and Qing Ke introduces COMO, short for Closed-loop Optical Molecule Recognition, to address one source of errors: the gap between how many systems are trained and how they are used. During conventional training, a model often sees the correct preceding symbols as it learns to predict the next one. When reading a new image, however, it must build on its own earlier predictions. An early mistake can affect what comes next. The authors call this exposure bias.

There is also a mismatch in what is rewarded. Token-level training measures individual output symbols, while users and benchmarks care about the complete molecule: is it chemically valid, structurally similar to the reference, or an exact match?

Score the complete prediction

COMO combines conventional training with Minimum Risk Training (MRT). In the MRT stage, the model samples candidate molecule descriptions from its own predictions. The system checks those candidates, then scores them using a reward that can include chemical validity, similarity to the reference and exact match. Similarity can be measured in different ways, including by comparing strings, molecular fingerprints or rendered images.

The training update favors candidates with higher rewards. In effect, the model practices evaluating the kinds of outputs it will produce when used, rather than learning only from correct prefixes. The authors interleave MRT with ordinary training, which supplies token-level and bond-prediction supervision.

The team tested three versions of COMO, each using a different similarity signal, across ten benchmarks: two synthetic sets and eight real-world sets, including diagrams from patents and scientific literature. They measured exact-match accuracy and compared results with existing methods. They also ran an ablation—a test designed to isolate a component’s contribution—to examine whether gains came from MRT or simply from additional training data.

The authors report that the strongest COMO version on each benchmark achieved the best result among the methods compared. They highlight larger gains on some real-world sets than on synthetic ones. On JPO, COMO-Edit achieved 89.1% exact match, compared with 78.9% for the strongest prior result listed. On ACS, COMO-Edit reached 88.2%, compared with 83.4% for MolSight. These figures describe particular test sets, not guaranteed accuracy on every chemical document.

For the ablation, the researchers converted real-world annotations into standard SMILES and rendered new images for conventional, MLE-only training. Adding this rendered data did not reproduce the gains seen with the combined MLE–MRT approach. They also applied MRT to a different recognition model, SwinOCSR, and reported improvements on the benchmarks tested. This suggests MRT can work with more than one architecture, but does not show that it will help every system.

What remains uncertain

Some incorrect predictions were structurally close to their references. The authors’ manual inspection also found examples where the benchmark labels appeared to be wrong. Exact-match scoring can penalize a prediction that matches the image if the reference answer is faulty. The paper notes known labeling issues in some datasets. For a comparison with MolParser, the authors rely on results published by its developers because they could not reproduce them with the model; the evaluation procedure for one benchmark is not fully documented, making a direct comparison difficult.

Other limitations matter too. Each configuration was trained once, so the study does not measure how results vary across repeated runs. The visual-reward version was not clearly better than versions using string or fingerprint similarity; the authors suggest the visual encoder may be limited. The method depends on labeled examples, which can contain errors, and uses a relatively small sample of candidate outputs per example. In the reported setup, MRT also increased training time per epoch, although inference cost was unchanged.

COMO is presented in an arXiv preprint, a manuscript that may not have undergone peer review. Its benchmark results are evidence for a potentially useful training strategy, not proof that automated recognition can replace checking by a chemist. Results depend on the selected datasets, labels and evaluation rules. The central idea is narrower: training a model to score complete molecular predictions may better align learning with the structures researchers need.