Is my Varicocele affecting my testosterone?

Iv had varicocele since I was about 14 but only recently have I been worried about it. I'm 23 years old and I still can't grow a full beard! I had a testosterone test and it come back around 507, the doc said it was normal but it seems through what I've researched it's should be more around 700 for my age. The doc told me it was a level 1 varicocele. I just don't want to be in my 30's with testosterone problems. Either it's my genes that cause low/normal levels or this varicocele that I'm becoming OCD over. Anyone got any advice? And is there any chance that I can regenerate the veins with out surgery?