You do not really "lose" any points- It costs 15 points to add a new skill which starts @ 10 points- It APPEARS you lose 5 points, but the initial 5 points are "buying"the first point in the new skill (assuming this makes sense)- when you respec and place that skill back in the Left colum (skills which are availible but YOU dont have) you get all your initial skill points back.
Personally I feel you SHOULD lose points as I (and others I am sure) respec on the fly for absoulutly no penalty. If I want to afk grind I spec myself for those skills needed and an hour and a half later if I want to "play" I respec on the fly to add Summoning and Spells. While Creafting I respec depending on what I am doing and also to get easier work orders- All on the fly with no penalty.Once your crafter gains a mere 200 skill poiints in ANYTHING he can be a Master in EVERYTHING based on how you spec him at any given moment- Chopping Trees? Respec the 200 points in Woodcutting for a couple hours and then switch back to Mining. No Penalty.
I understand the need for respec (so folks do not unintintionally Gimp their toon) but without actual penalties of any sort its pure exploitable. My Master Miner or Swordsmen shouldnt be able to "forget" his craft and become a Master of a new Craft without Major Roadblocks (i.e. Huge amount of Gold, Permanentlky lost Skill Points/XP, or maybe 1 time a week)
An example- I was running the sewers last weekend and instead of dealing with poision damage, I quickly gave myself max Poision resistance and dropped Swordfighting (I already had max Bow) and went about my buisness. An hour and a half later I instantly switched back-
EDIT: Note that I am fairly certain some sort of penalty will be added eventually for skill respecing, right now this honestly help to test out the skills as they are added.
Fact is, there is NO LOSS and no reason not to switch your stats on a dime for whatever you happen to be doing.