I'm taller than you(over 5'6"), skinnier than you(106lbs), and have a larger chest than you(probably a 30FF or a 28G but I'm wearing a 32F as I can't find bras with the right band and cup size) and I don't have a difficult time finding shirts that fit so I can't imagine you would either.

As for bras, when I was a 32D I had a difficult time finding bras, but now I've noticed that more and more bras are being made in that size and I see them all the time now. One brand that I know makes 32D is Calvin Klein, has the majority of my 32D bras were Calvin Klein, Cabarnet is another brand that makes that size. I know a lot more brands carry it in their range now, I just don't know which ones since I know longer wear that size. I do know that Frederick's of Hollywood, which has many stores as well as an online store, carries bras in that size.

Also, Freya is amazing and has a wonderful range of bra sizes, though it is difficult to find Freya bras in the US. I suggest you check out this website, you're bound to find a bra that fits. They also have swimsuits figleaves.com - Bras, Panties, Swimsuits, and Full Figure Lingerie

Speaking of swimsuits, why don't you just buy a swim suit that's a two piece with tops and bottoms sold separately? That way you can make sure that each piece fits the appropriate area.

Anyways, 32D isn't really even that big.