I think you should just let it go... he's probably just been curious and your panties were the perfect opportunity to fulfill the curiosity.
Unless he's like... in love with you, and worshiping them.
Or performing voodoo on them..
Or like, sniffing them..

Yeah I'm hoping it's not any of those options.. Just let it go :P