Are there any downsides to being in a double nat?

So I have my main router in a room far away and pretty much no devices connect to it but i sometimes use it so I cant turn it off then I grab a cable from it and connect it to wan port of the router inside the house which everybody uses. I used to have it set to Access point mode but some features like qos werent avaliable so I switched it over to router mode. Any downsides to doing this? Im already behind a cgnat so im not gonna be port forwarding and no ipv6 if I wanted to anyways.