Recirc solenoid bumps/oscillates in recirc mode

I have an 05 Prerunner Dbl Cab TRD Sport. If I start it with the AC in recirc mode, the solenoid that moves the dampers to the recirc mode bumps in the recirc position. I removed the glove box and looked above and to the left of the air filter and can see the white plastic linkage that moves the outside air damper and it is moving slightly against the firewall. If I put slight pressure on the linkage, the solenoid stops the bumping. When I cycle the button, the solenoid strokes smoothly. I put it in Recirc mode and removed the plug from the solenoid because it is anoying when it does this, and I like to have it in recirc most of the time due to the hot weather and the better cooling achieved in the recirc mode.

Does anyone know how to fix this? Is it a bad solenoid or could it be misaligned slightly?