Sure it'd be worth having the 35mm and 50mm if you like prime lenses
At one point I was considering adding a 28mm prime to my bag, but in the end went with an ultrawide (but that's beside the point).
One thing to consider between the 35mm and 50mm is how much room you have to move. Since you have the 18-55mm lens, try walking around with it only at 50mm (especially indoors) and then repeat with it at 35mm. If you're constantly against a wall trying to take pictures of the kids at 50mm then you'll know that you'll get more use out of the 35mm.
That being said I'm both a big fan of primes and a big fan of the 50mm
