Then trust him. =]
You two should talk to one another extremely openly about this and try to come up with solutions amongst yourselves, that usually works the best.

Communication is crucial, you need to stop assuming the worst and ask questions, this requires you being completely open with him, holding back will cause jealousy. Being open is crucial as well, if your partner truly cares about you, they will help you in the process, but let him know you sincerely trust him, you know he wouldn't hurt you, but you still have the thoughts.

She was his first love, but they are over for a reason. It is your turn now, have fun and enjoy it, don't be worried, realize those thoughts will eventually be replaced with you. =]