Gynaecomastia is rare, and there is often no obvious cause.
But US specialists report in the New England Journal of Medicine that three boys developed the condition after using the oils.
The National Institute of Environmental Health Sciences (NIEHS) team say doctors who see boys should ask about their use of such products.
Oils 'mimicked hormones'
Clifford Bloch, a child hormone specialist at the University of Colorado at Denver and Health Science Center's School of Medicine started to suspect the link after seeing the three boys.
The first, aged four, had been experiencing symptoms for two to three weeks.
His mother said she had recently begun applying a "healing balm" containing lavender oil to his skin.
The second boy, who was 10, had been developing enlarged breast tissue over the previous five months.
When questioned, it emerged that he was using a shampoo and hair gel containing lavender oil and tea tree oil every morning.
The third boy, aged seven, had a one-month history of gynaecomastia.
He had been using lavender-scented soaps and skin lotions. His twin used the soap, but not the lotions, and had not developed the condition.
Each of the boys stopped using the relevant products, and several months later the tissue growth was found to have subsided.
The team at the NIEHS carried out tests on human cells and found pure lavender and tea tree oils could mimic the actions of female hormones and inhibit the effects of male hormones, and therefore disrupt the endocrine system in the body.
'Reconsider' product use
Dr Bloch said: "Since there was no identifiable cause for prepubertal gynaecomastia in the three patients we reported, we speculated that environmental factors might be contributing to their condition.
"Together, the case histories and NIEHS studies provide support for our hypothesis that topical exposure to lavender and tea tree oils likely caused gynaecomastia in the three patients."
It is not known whether the oils could have similar effects on the endocrine systems of young girls, teenagers or adults.
But Dr Derek Hanley, who led the NIEHS research said the oils did not appear to alter the levels of the usual forms of hormones circulating in the boys' bodies.
And he said: "We do not anticipate any long term effects on hormonal levels."
Dr Ken Korach, who also worked on the study, said: "We want to encourage doctors who may be seeing patients with gynaecomastia to ask their patients about the products they are using.
"Patients with prepubertal gynaecomastia may want to consider reducing the use of products that contain these oils."
Professor Ieuan Hughes, a child hormone specialist at the University of Cambridge, said: "The scientists have shown a convincing effect, albeit on cell lines; that these oils can mimic oestrogens and act as anti-androgens [male hormones]."
He said: "We don't really understand gynaecomastia, but the consensus is that it's related to a hormone imbalance."
Professor Hughes said hormone levels in prepubescent boys were particularly sensitive to changes.
He added: "The next step is to see how widespread this problem is.
"As the researchers say, it is important to say people should be a little bit careful about using these products until we know more."