Biomechanics of the extracellular matrix in arteries determines their macroscopic mechanical behavior. In particular, the distribution of collagen fibers and bundles plays a significant role. Experimental data showed that, in most arterial walls, there are preferred fiber directions. However, the realignment of collagen fibers during tissue deformation is still controversial: whilst authors claim that fibers should undergo affine deformations, others showed the contrary. In order to have an insight about this important question of affine deformations at the microscopic scale, we measured the realignment of collagen fibers in the adventitia layer of carotid arteries using multiphoton microscopy combined with an unprecedented Fourier based method. We compared the realignment for two types of macroscopic loading applied on arterial segments: axial tension under constant pressure (scenario 1) and inflation under constant axial length (scenario 2). Results showed that, although the tissue underwent macroscopic stretches beyond 1.5 in the circumferential direction, fiber directions remained unchanged during scenario 2 loading. Conversely, fibers strongly realigned along the axis direction for scenario 1 loading. In both cases, the motion of collagen fibers did not satisfy affine deformations, with a significant difference between both cases: affine predictions strongly under-estimated fiber reorientations in uniaxial tension and over-estimated fiber reorientations during inflation at constant length. Finally, we explained this specific kinematics of collagen fibers by the complex tension-compression interactions between very stiff collagen fibers and compliant surrounding proteins. A tensegrity representation of the extracellular matrix in the adventitia taking into account these interactions was proposed to model the motion of collagen fibers during tissue deformation.