Researchers studied the viewing habits of more than 2700 children at age two and again at five.
Five-year-olds who watched TV for two hours or more a day were more likely to develop behavioural problems or have trouble with social skills. Having a TV in their bedroom also increased the risk and often led to sleep difficulties.
But children could be weaned off TV without lasting damage. Those who were heavy viewers at age two but reduced their exposure by the age of five showed no increased risk of problems.
"It is vital for clinicians to emphasise the importance of reducing television viewing in early childhood among those children with early use," said the study's senior author, Dr Cynthia Minkovitz of the Johns Hopkins Bloomberg School of Public Health in Baltimore.
The study, published in the Pediatrics journal, has reignited fierce debate around the influence of TV on the developing child brain. Recent studies have blamed TV for causing ADHD and autism, a claim vigorously contested by some childhood development experts.
Australian specialists told The Age that studies of viewing habits without considering other factors were often flawed.
"Children who are at risk or vulnerable who watch a lot of violent television may be inclined to be more aggressive but that really only applies to a select group of kids," said Professor Margot Prior from Melbourne University, one of Australia's leading experts on child development and behavioural adjustment. "If parents manage television and monitor what their children see and are open to explaining and discussing what's on television then you can identify very few, if any, harmful effects."'
Sixteen per cent of two-year-olds and 15 per cent of five-year-olds watched two hours or more of TV a day. Forty-one per cent of all children had a TV in their bedroom.
Professor Alasdair Vance, head of academic child psychiatry at the Royal Children's Hospital, said the younger the child the less time they should spend watching TV.
But he said many programs such as Play School or Sesame Street had educational benefit and TV could not solely be blamed for bad behaviour. "A given child watching two or more hours of television a day may have little contact with an empathic, nurturing, sensitive and responsive parent who can help them develop those emotion management, anxiety management and aggression management skills," he said.
Australian Childhood Foundation chief executive Joe Tucci said TV was often used as a "babysitter" and parents should act as a filter for preschoolers.
"Kids are like a sponge at that age," he said. "They're taking in what's going on around them and trying to develop rules for how the world works, and if those rules are based on activities that have some violence in it then they're the rules that they'll try to emulate."
Proportion of children watching two or more hours of TV a day:
* Age 2: 16 per cent
* Age 5: 15 per cent
* Proportion of all children with a television in their bedroom: 41 per cent.