I had an E-PM1 for a while, and although my subjects were pets, had the same feeling. Plus noise and autofocus in low (indoor) light seemed poor to me. From reading a bunch of forums, the consensus was to get a Panasonic OIS lens (like the 14-42) if you wanted to shoot decent video on an Olympus.
I was considering switching to a Panasonic m4/3 body and wanted one with the new 16MP sensor, but was simply blown away by a NEX-F3. Insanely good in low light for video, and probably the best continuous AF I've seen in any interchangeable lens camera.
TBH though, if you are making lots of videos of your kids, no camera will be able to beat a decent camcorder. The camcorders have much smaller sensors (that's why everything is always in focus on them and AF is easy), and the lens can also therefore be smaller and easier to optically stabilize.
You obviously haven't used a Panasonic M43 to take video.
Even if you put an OIS lens on a Olympus camera, AF is still useless for video.