You can be more gentle (or cowardly) and mute them.
Or directly report them.
And no one should require an explanation from you.
I routinely unfollow people to avoid getting my Home timeline flooded. Have they done something wrong? Not at all. Sometimes it is they write a lot. Sometimes it is they write about something I'm not currently so interested. I keep them in lists to make them coming back when the tide changes.
Nobody asked why.
@jgg I guess it depends. We all are allowed to conduct ourselves in our socials however we see fit, and we owe no explanations to any stranger