Mac and cheese Roux help

I need help to make a smooth cheese sauce for a Mac and cheese. Typically when I make it the end product comes out a bit gritty and I don’t love that. Trying to get away from the gritty texture.

My usual recipe is for the cheese sauce is 1/4 cup flour and 1/4 cup fat (usually butter but sometimes bacon grease if I have it). I will also throw some pepper in the hot oil before I add the flour to let it bloom. I cook the roux at a low temp because I read that can help with the texture. Cook that stirring continuously for about 5-10 mins. Then I add 2 cups of milk (usually skim) and stir that until it is smooth. Once that looks satisfactory I will add other spices and the cheese. When the dish is don’t I find the cheese sauce isn’t as smooth as I would like it and it seems like I added a bunch of small bread crumbs to it. Not ideal

A few notes. I find when I add milk that the roux turns into a big clump, then when I add more milk it smooths back out, not sure if that’s normal. I also don’t get a ton of colour on the roux before I add milk, usually just a cent darker than the milk.

If you have any solutions or suggestions to get the cheese sauce to a smooth consistency that would be appreciated.