Electrons in carbon nanotubes (CNTs) possess spin and orbital degrees of freedom. The latter is inherited from the bipartite graphene lattice with two inequivalent Dirac points. The electronic spectra obtained in several transport experiments on CNT quantum dots in parallel magnetic field often show an anticrossing of spectral lines assigned to the opposite Dirac valleys. So far this valley mixing has been attributed to the disorder, with impurity induced scattering. We show that this effect can arise also in ultraclean CNTs of the armchair class and it can be caused solely by the presence of the boundaries. In contrast, in CNTs of the zigzag class it does not occur. These two fundamentally different classes of spectra arise because of different symmetries of the low energy eigenstates of the two types of CNTs. The magnitude of the level splitting depends in a nonmonotonous way on the distance of the involved energy levels from the charge neutrality point.