MOAI speeds up transformer inference on encrypted data

The privacy problem—and the cost

When someone sends text to a cloud-hosted AI service, the provider ordinarily needs access to the input to process it. Fully homomorphic encryption (FHE) offers a different approach: the user encrypts the data, the server performs calculations directly on the ciphertext, and the user decrypts the result. In principle, the server need not learn either the input or the output.

That protection comes with a substantial computational burden. In the scheme studied here, some operations—especially encrypted “rotations,” which rearrange values within a ciphertext—are costly. Transformer models repeatedly perform matrix operations and functions such as Softmax and LayerNorm, so inefficiencies can accumulate across many layers.

MOAI, short for Module-Optimizing Architecture for non-interactive secure transformer Inference, targets that bottleneck. “Non-interactive” means that, after the encrypted input is sent, the server can carry out inference without repeatedly asking the user to help with intermediate computations.

Reorganizing the encrypted calculation

The authors build their system around the CKKS approximate homomorphic-encryption scheme. It packs multiple numerical values into ciphertexts so that one encrypted operation can work on many values at once. MOAI combines two ways of arranging those values—column packing and diagonal packing—and designs the sequence of computations so that data can pass from one transformer operation to the next without expensive format conversions.

The system also introduces rotation-free procedures for Softmax and LayerNorm. Rather than rotating values within a ciphertext to assemble sums, the approach places values that need to be summed in matching positions across different ciphertexts. For BERT-base, the authors say this removes 2,448 rotations from those two operations. For matrix multiplications, column packing and a layout called interleaved batching further reduce rotation requirements.

The team implemented MOAI and evaluated encrypted inference on BERT-base, including a 128-token input setup, using models fine-tuned for SST-2, QNLI and RTE classification tasks. They measured performance on CPUs and GPUs, and compared GPU timings with earlier systems under the hardware settings described in the paper. They also applied the approach to LLaMA-3-8B, a much larger decoder-only model, though the reported LLaMA results use an eight-token input setup and aggregate the required components.

Faster in these tests, but still measured in minutes

For BERT-base, the paper reports at least 1.7 times fewer homomorphic-encryption rotations than the compared systems across matrix multiplications. In its same-hardware comparison with THOR, MOAI reduced total evaluation time by 52.8%. The authors also applied MOAI’s packing and evaluation methods to Powerformer, a system that replaces Softmax and LayerNorm with alternative functions; they report a 55.7% reduction in evaluation time in that comparison.

Those percentages describe particular implementations and test conditions, not a general guarantee. The paper also reports an amortized time of 2.36 minutes per BERT input on a single GPU in its end-to-end evaluation. “Amortized” means the cost was averaged over a batch of inputs; it is not a claim that every individual request will take exactly that long. The result illustrates both the progress and the remaining gap between faster encrypted inference and ordinary, unencrypted service.

The authors checked whether the approximate arithmetic used by CKKS changed model results. On the three reported BERT classification benchmarks, scores for encrypted and unencrypted inputs were close, with some mismatches. That supports the claim that the added encryption-related approximation had little effect in these tests. It does not establish unchanged accuracy for all models, datasets or input distributions.

What the findings do—and do not—show

The central contribution is an engineering one: coordinate the way encrypted data is packed across layers and avoid rotations in selected operations. The reported speedups show that these choices can substantially reduce evaluation time relative to the cited systems in the authors’ experiments. They do not show that encrypted transformer inference is already fast enough for every real-world use, or that encryption eliminates every privacy or deployment risk.

The study evaluates particular models, sequence lengths, tasks, encryption settings and hardware. Its LLaMA-3-8B demonstration shows that the techniques can be applied beyond BERT, but the supplied results do not establish broad performance across longer conversations or varied production workloads. Comparisons also depend on implementation details and reported baselines; the paper notes that some earlier work did not provide fully comparable end-to-end measurements.

The source identifies the work as published at ICLR 2026, and says the implementation is publicly available. As with any research result, reproducing the measurements independently and testing additional workloads would help establish how well the approach transfers. MOAI offers evidence that careful data layout can make encrypted transformer inference more efficient—not that its computational cost or practical constraints have disappeared.