I have the exact same problem. I can lift weights all day long. I'm a high volume guy who has a home gym, so my workouts can go for a while when I want. And my total reps are 700+ for any given day. My back will be fine. But... if I spend more than 5 minutes doing dishes, more than 30 minutes of weed whacking, or even a few minutes of digging a hole, my back is on fire! But mines always been like that. I think its because when we're lifting we're usually super focused on posture. But while digging a hole or washing dishes it's odd angles. Those tiny muscles in your back are supporting all your weight.
As someone who had a very bad back for years, just pay attention to it. Try stretching a few times a week. If you need to take a break, do it. Don't do what I did and power through the pain! That shit cost me almost 8 years of my life. Ignoring my back pain is a huge reason why me at 6ft got down to 137lbs. Because I thought I was to tough or to cool to take a break.
I wish I had a solution for ya, all I can say is stay hydrated and if it starts to hurt, stop! Dealing with a bad lower back is no joke