Most people don't; and it's as safe as the Windows OS
These days, cross-platform apps are not as common as they used to be. That was the primary purpose for Java, and (with graphics) Flash. Chances are you don't need it: for example, when did you see anything that uses Silverlight, aka Microsoft's Java look-alike? I would say uninstall it simply to make space, if something complains or starts crying, re-install it.
Safety? Back in pre-Cambrian days, Mac users would proudly boast (like *nix users) "we don't have malware!" I need to make it clear that the first malware ever seen was a primitive worm on a Unix system, back in the, aaahh, very late sixties if my memory serves... Maybe in the very early seventies. The point is, malware makers aim at the most popular OSes. The bigger the target, the more chance of getting a hit. Java was not seen to be a worthwhile target as it was one of the early sandbox users IRRC. However, like Macs and *nix machines, increasing complexity opens more loopholes, and thus you get a perception of decreased security.
If you decide to keep it on your machine--"just in case"--you don't need to worry overmuch: any security problems in Java don't happen until you run it, rather like the worm in the email attachment, which does nothing until you double-click it, or the substandard email client (we won't mention names like vantage-point) which automatically opens attachments...
Like JBinOZ I need Java to access the Western Australian Landgate mapping system, and that keeps whingeing that my latest version is the wrong version... I have yet to see an app that requires Silverlight.