Precise knowledge of the optical constants of perovskite lead halide quantum dots (QDs) is required to both understand their interaction with light and to rationally design and optimize the devices based on them. However, their determination from colloidal nanocrystal suspensions, or films made out of them, remains elusive, as a result of the difficulty in disentangling the optical constants of the organic capping ligands and those of the semiconductor itself. In this work, we extract the refractive index and extinction coefficient of ligand-free methylammonium lead iodide (MAPbI3) and bromide (MAPbBr3) nanocrystals. In order to prevent the use of organic ligands in the preparation, we follow a scaffold assisted synthetic procedure, which yields a composite film of high optical quality that can be independently and precisely characterized and modelled. In this way, the contribution of the guest nanocrystals can be successfully discriminated from that of the host matrix. Using a Kramers-Kronig consistent dispersion model along with an effective medium approximation, it is possible to derive the optical constants of the QDs by fitting the spectral dependence of light transmitted and reflected at different angles and polarizations. Our results indicate a strong dependence of the optical constants on the QD size. Small nanocrystals show remarkably large values of the extinction coefficient compared to their bulk counterparts. This analysis opens the door to the rigorous modelling of solar cells and light-emitting diodes with active layers based on perovskite QDs.