AI Project

Cursive Transformer Handwriting: Polar Tokens + Cross-Attention GPT

2025-09-155 min816 wordsupdated 2026-08-20

// series: handwriting synthesis — part 2 of 2[ part 1]

A 6-layer cross-attention GPT decoder with a polar-coordinate stroke tokenizer produces cursive handwriting at 96% character accuracy (up from 82% with the LSTM in Part 1), and the baseline-drift problem disappears entirely. This is Part 2 of the handwriting generator write-up. The model powers hand-magic.com. It's also the architecture that recently showed up in The Cursive Transformer (arXiv:2504.00051), two projects converging on the same polar-GPT recipe independently.

TL;DR

6-layer GPT decoder, d_model=384, 6 heads, ~102MB of weights.

Polar tokenizer: each pen offset → (angle_bin, radius_bin, end_of_stroke), giving 128 × 64 × 2 = 16,384 tokens.

Cross-attention over the character sequence replaces the LSTM's Gaussian windows.

Style embedding with contrastive training beats hidden-state priming.

Result: 96% character accuracy, zero baseline drift, 150ms/line on GPU.

What was wrong with the LSTM version

The LSTM handwriting generator worked, but it had two problems I couldn't shake:

Baseline drift. Vertical error compounded over long words. The last character of a paragraph would float half a line above or below the first.
Fragile style transfer. Priming the LSTM's hidden state with a sample of target handwriting worked, until it didn't. Prime it wrong and you'd get recognizable gibberish halfway through.

Both felt like symptoms of recurrence without strong long-range memory. So I did what any reasonable 2025 developer does: I tried a transformer.

The polar tokenizer

The original model predicted (dx, dy) as a mixture of Gaussians. Cute, but 20 Gaussian components mean 6 × 20 = 120 output parameters per step just to parameterize the distribution, and 2D Gaussian mixtures get messy to tune.

I switched to a discrete tokenizer working in polar coordinates. Each pen offset is quantized into an (angle, radius, end_of_stroke) triple:

~ python

# pseudo: per-step pen offset → three discrete tokens
angle_bin  = quantize_angle (atan2(dy, dx), n_bins=128)   # 128 bins, 2π/128 per bin
radius_bin = quantize_radius(sqrt(dx*dx + dy*dy), n_bins=64, scale="log")
eos_bin    = 0 or 1
vocab_size = 128 * 64 * 2  # = 16,384

Training is now plain cross-entropy over this vocab. No more MDN log-likelihood.

Why polar?

Angles carry more information than raw dy for handwriting. Strokes have consistent directional patterns (upstrokes, crossbars, loops) that map cleanly onto angle bins.
Log-scaled radius bins match the empirical distribution of stroke lengths. Short strokes get fine bins; long strokes get coarse ones.

The Cursive Transformer (arXiv:2504.00051), published March 2025, arrives at essentially the same tokenization independently, with the same reasoning behind it.

Cross-attention conditioning

Each decoder layer has a cross-attention block that attends to the character sequence. Unlike the LSTM's shifting Gaussian windows, any output token can attend anywhere in the prompt. In practice, the lower layers learn "what character am I currently writing" and the upper layers learn "what stroke within that character am I on."

The baseline drift from Part 1 just... disappeared. The model maintains perfectly horizontal alignment over multi-line text.

Style embedding with contrastive training

Instead of priming the hidden state (the LSTM trick), the transformer has a small style-embedding table with up to 10 style slots per deployment. A learned projection injects the style embedding into every decoder layer.

Training uses a contrastive objective:

Pair type	Objective
Two samples from the same author	Pull embeddings together
Two samples from different authors	Push embeddings apart

At inference you either pick an existing style slot or provide a few lines of reference handwriting, and the model projects the reference to the closest existing embedding.

LSTM vs. Transformer, side by side

Metric	LSTM (Part 1)	Transformer (Part 2)
Weights	14 MB	102 MB
Character accuracy (OCR round-trip)	~82%	~96%
Baseline drift	Visible after long words	None
Multi-line stability	Poor	Excellent
Training time	~12h on one GPU	~40h on one A100
Inference (GPU)	~90ms/line	~150ms/line
Inference (CPU, laptop)	~0.8s/line	~1.2s/line

Where it's deployed

hand-magic.com. Type text, pick a style, get back a PNG. Actually try it, and tell me where it breaks. Some of the weirdest failure modes only show up on specific uncommon character combinations, and I genuinely can't predict which.

Key takeaways

Discrete polar tokens beat continuous MDN outputs for handwriting: standard cross-entropy, cleaner training, better generation.
Cross-attention kills baseline drift because every output token has direct access to the prompt.
Contrastive style embeddings scale better than hidden-state priming and are more interpretable.

What's next

Longer context (currently truncated at ~500 strokes).
A separate head for stroke thickness so the model can emit pressure-style handwriting.
Combining the style embedding with reference-image conditioning for offline (photographed) handwriting.

If you have thoughts, reach out. And check out my other projects.

References

The Cursive Transformer (arXiv:2504.00051)
Part 1: LSTM Handwriting Generator
hand-magic.com
github.com/OmarMusayev/ai-handwriting-generator
Alex Graves, 2013: Generating Sequences With Recurrent Neural Networks (prior art)

//FAQ

What is a polar tokenizer for handwriting?

Instead of predicting raw (dx, dy) pen offsets, a polar tokenizer quantizes each offset into (angle, radius, end_of_stroke). Handwriting has strong directional structure, and strokes aren't uniformly distributed in Cartesian space, so angles carry more information than raw dy. Radius bins can be non-linearly scaled so short strokes get fine-grained bins and long strokes get coarse ones.

Why does cross-attention fix the baseline-drift problem?

The LSTM's attention is recurrent and has finite memory, so vertical-drift errors compound over hundreds of tokens. Cross-attention lets every token attend directly to any character in the prompt, so there's no hidden-state bottleneck to carry baseline information through. In practice, multi-line output stayed perfectly horizontal after the switch.

How does the style embedding work?

There's a small style-embedding table (up to 10 entries per deployment). A contrastive loss pulls same-author samples together and pushes different-author samples apart. At inference, the user picks a slot or provides a few lines of reference handwriting, and the model projects them to the closest existing embedding.

How big is the transformer model?

6 transformer decoder layers, d_model=384, 6 attention heads, about 102MB of weights. It trains in roughly 40 hours on a single A100, and ships at 1.2 seconds per line on laptop CPU or 150ms per line on GPU.

What's the accuracy?

Character accuracy, measured by OCR'ing the generated handwriting back to text and comparing, is about 96%, up from 82% on the LSTM version. That gain comes mostly from eliminating drift and giving the model direct attention to every character in the prompt.

How does this relate to The Cursive Transformer paper?

The Cursive Transformer (arXiv:2504.00051) independently lands on the same core insight: tokenize pen offsets in polar coordinates and train a plain GPT decoder with cross-attention over ASCII text. Both projects ended up there because it works.

share:[x][linkedin][email]

#deep-learning #pytorch #transformers #handwriting

//FAQ

//related