You may need to run a thicker wire for that kind of load. The current wire is probably adequate for either charging or the fridge, but suffers when doing both.
Things to note;
1) DC voltage drops with distance.
2) Voltage drops with high current load.
3) Combination of above two points causes drastic voltage drop.
4) If your battery is discharged to begin with, it will cause your reading to be very low until it gets close to optimal.
First thing I would do is measure the battery voltage with it disconnected. If it is reading something like 10-11 volts, it is low and probably shot. Hook it to an outside charger, when fully charged, it should read about 12.6 volts (with the charger disconnected). If it won't come up close to that, the battery isn't charging properly and needs to be replaced. If it does, let it sit for an hour and measure the voltage again, if it drops significantly, the battery may be bad and need to be replaced.
With the battery still out of the camper, start up the fridge and measure the voltage. If the voltage is less than 12.6, then your circuit is supplying less current than drawn by the fridge, and you definitely need to upgrade the wiring. I would want it to read at least 13.5 volts with the fridge ON and the battery OUT. With inadequate power, the fridge can actually burn itself out, and will definitely drain the onboard battery.
To upgrade the circuit, what you will want to do is run a new separate THICK power wire from the panel under the hood to the connector. The thicker the better. I'm thinking 3 or 4 ga. Make sure its fine stranded NOT SOLID. You might also want to run that heavy gauge through your camper, at least up to the distribution panel. You can run a SHORT piece of thinner gauge wire to actually get it into the plugs (since certainly a 3 ga wire won't fit in), just make sure to keep it short so you don't lose too much voltage.