Figured outnthe problem. The index adjustment screw was torn loose, so the indexing was off. Also, one of the pulleys was worn to nubs.
I swapped it for a Deore SGS M5591 I had in the parts pile for now. Same device, but a longer cage and a weaker spring. Should be fine, but I will probably get something with a clutch later.
It's nice that mountain 9-speed RDs are interchangeable with road 10-speed.
Eventually, I may re-tap the threads on the rd-5700sg and replace the pulleys.