My bf is doing drugs and im uncomfortable with it, what do I do?

Ok, so my bf is a regular pot smoker, which I'm fine with. He gets his work done and prioritizes. Plus, I don't pot is that big a deal anyway. HOWEVER, now hes doing shrooms to, which I do think are a big deal. We have discussed before and I told him I didn't care if he tried them or even did them once or twice a year but I didn't think it should be an all the time thing. HE AGREED. Every time we talked. But now its been 3 weeks since he did them and hes going to do them again. I got angry (which may not have been the best approach admittedly) and told him I didn't feel comfy w/ it. What happened to our agreement? He said they weren't as big a deal as we thought and he wasn't gonna do them weekly, just more than once or twice a year. He got angry with me when I said he should care that I didn't like it and said if I cared I wouldn't put limitations on him.SHould I accept it? He is in college, time to experiment. I don't want to lose him over it but I still don't like it.