Think of midi-chlorians as parasites that infest those with Force sensitivity, with the stronger Force users having more, and it all makes sense. They're not a cause of feeling the Force, but rather an effect.

https://screenrant.com/star-wars-midi-chlorians-george-lucas-abandoned/