I've been using
and updating HandBrake from www.handbrake.fr.com for years. Different Macs and Windows computers. You are wise to be cautious. If it makes any difference, I have over 5,000 forum posts - mostly in this camcorder forum, dating back a few years. I used to be an IT manager - my group took care of over 500 Mac and 500 Windows machines on the same network across 13 offices in California. That should not allow you to let your guard down about HandBrake - but I have always found the application to be malware-free.
CNET likes it, too
I am afraid there may be something else going on with those camcorder media files if you could not get VLC Player to play them. Please also consider locating a nearby Macintosh User Group
or making an appointment for assistance at your nearest Apple Store...