Something similar-ish happened to me. For some reason people who I knew started recieving spam e-mails from an address I used. Obviously I wasn't sending them. I even recieved one from myself
. I think I maybe had a virus or something.
Maybe that's what happened with you. That guy's been getting spam e-mails from your address and thinks you're responsible for it.