Looking into the inlet again, I sprinkled a little bit of Febreeze Extra Strength in there, walked around and hopped inside. Smell pretty nice. That's when I got an idea.
So I took everything apart again. The brown stuff on the fan is what was giving off the bad smell.
I cleaned the fan and the fan's housing.
The smell is almost completely gone now. I'm sure there is more to clean up inside the duct but I'm not sure how to take it all apart yet for a good cleaning. It's much better than before and almost none existent.
I really think my blower fan acted as a blender to something today.