As per Employment Act, it is legal, and your boss is not wrong. Reference can be found under my Page “Sources”:
Annual leave in Special Situations
However, some company do have policies that allow employees to take half a day. Thus, do also be clear on your company policies.