Hard to start 02 Taco 2.7 after 3 weeks without using it.

I had the same thing happened about 2 years ago. I went on Vacation for about 3 weeks. I tried starting it but nothing, I replaced the battery (under warranty) and still nothing. Tested the EFI and starter relays and they were all good the starter relay clicked. After switching the relays from one spot the other, I tried again and bam, it started (2 years ago)

The pump seems quiet almost no noise even today. I got aggravated and kept cranking and after about 7 seconds it seemed to struggle to start but finally it did. I saw my engine shake before it started idling fine.

This only happens when it sits for weeks at the time and the battery had enough charge to start. It's kinda odd. I even cleaned the battery terminals.

Anyone ever experienced this?

I replaced the fuel filter about 3 years ago. It's my daily truck for work.