GlobalPuzzle uses the whole picture to assemble eroded image fragments

Looking beyond damaged edges

An eroded image puzzle presents two linked problems: deciding which fragments belong together and deciding where those groups belong in the complete picture. Pieces may have useful internal relationships yet occupy the wrong region. A solver needs to preserve those relationships without becoming trapped by an incorrect placement.

GlobalPuzzle combines learned visual guidance with a genetic algorithm, which maintains a population of candidate arrangements and creates new candidates by recombining and modifying them. The authors’ central idea is to judge the assembled picture, rather than relying primarily on whether individual fragment boundaries appear compatible.

The task remains a grid-based image puzzle. Although its damaged boundaries simulate a challenge relevant to fragmented cultural artifacts, the reported benchmark results are not demonstrations of archaeological restoration.

Two networks guide the search

One learned component, the Policy Network, examines a partially filled canvas and a map showing occupied positions. Working with encoded fragment features, it scores which pieces could fill which locations. A one-to-one assignment procedure, the Hungarian algorithm, helps complete the arrangement without using a fragment twice.

A separate Evaluation Network scores complete candidate canvases. During training, its target combines two measures: 60% comes from the fraction of pieces in their correct absolute positions, and 40% from the fraction of original directional neighbor relationships preserved. Training also encourages correct ranking of candidates and penalizes overestimated incorrect assemblies. During solving, this network predicts quality from the candidate canvas; the correct arrangement is not supplied as input.

The genetic search favors higher-scoring candidates. Its crossover operation identifies connected groups whose directional relationships agree between two parent arrangements. Some groups retain their absolute positions; others can move together while preserving their internal layout. Mutations change placements through swaps, three-piece cycles or small block exchanges. Periodic local refinement tests two-position swaps of the best candidate.

The standard configuration uses 256 candidates and up to 30 generations. The returned solution is the highest-scoring arrangement encountered, not a guaranteed correct reconstruction.

A substantial gain, with many puzzles still unsolved

The researchers evaluated GlobalPuzzle on GAP-3 and GAP-5: eroded-image puzzles arranged in three-by-three and five-by-five grids. Each dataset has 14,000 training puzzles, 3,000 validation puzzles and 3,000 test puzzles under the official splits.

Two reported metrics answer different questions. Perfect assembly accuracy counts puzzles with every piece correctly placed. Absolute accuracy counts correctly positioned pieces across the evaluated puzzles.

GlobalPuzzle achieved 68.6% perfect assembly accuracy and 81.6% absolute accuracy on GAP-3. On GAP-5, those figures fell to 19.4% and 48.7%. PuzzleFlow, the strongest listed baseline on perfect assembly accuracy, reached 28.5% on GAP-3 and 0.3% on GAP-5.

These are large improvements within the reported comparison, but most 25-piece puzzles still were not assembled perfectly. Average reconstruction times were approximately nine seconds for GAP-3 and 44 seconds for GAP-5 on an NVIDIA RTX 4060 GPU.

What the component tests reveal

Replacing the evaluator’s outputs with random scores reduced perfect assembly accuracy to 1.9% on GAP-3 and zero on GAP-5. Randomizing policy scores left GAP-3 perfect accuracy at 65.1%, but reduced GAP-5 to zero. Under the tested search budget, useful placement guidance mattered especially for larger puzzles.

Removing crossover reduced GAP-5 perfect accuracy to 10.4%; removing swapping reduced it to 11.3%. The full system reached 19.4%, supporting the value of complementary ways to revise candidates.

Reported full-system results average three search seeds with fixed pretrained networks. That checks variability in stochastic search, not variability from independently retraining the networks.

Reconstruction is not recovery of missing evidence

GlobalPuzzle uses the pretrained LaMa inpainting model in two distinct stages. Before solving, each fragment is filled independently to provide RGB inputs, without access to the correct complete arrangement. After solving, the original fragments are assembled and LaMa fills the remaining gaps using the whole canvas. This final cosmetic reconstruction does not determine piece placement, and generated gap content should not be mistaken for recovered original imagery.

The authors also show photographic examples on rectangular grids without retraining. These are representative demonstrations rather than a quantified generalization benchmark. Comparisons with general-purpose AI agents are likewise qualitative case studies, not directly comparable benchmark scores. The preprint supports a promising combination of visual evaluation and flexible search, while leaving broader real-artifact performance unestablished.