Different methods are available for measuring medication adherence. In this paper, we conducted a scoping review to identify and summarize evidence of all studies comparing the Medication Event Monitoring System (MEMS) with alternative methods for measuring medication adherence. A literature search was performed using the open database www.iAdherence.org that includes all original studies reporting findings from the MEMS. Papers comparing methods for measuring adherence to solid oral formulations were included. Data was extracted using a standardized extraction table. A total of 117 articles fulfilled the inclusion criteria, including 251 comparisons. Most frequent comparisons were against self-report (n = 119) and pill count (n = 59). Similar outcome measures were used in 210 comparisons (84%), among which 78 used dichotomous variables (adherent or not) and 132 used continuous measures (adherence expressed as percentage). Furthermore, 32% of all comparisons did not estimate adherence over the same coverage period and 44% of all comparisons did not use a statistical method or used a suboptimal one. Only eighty-seven (35%) comparisons had similar coverage periods, similar outcome measures and optimal statistical methods. Compared to MEMS, median adherence was grossly overestimated by 17% using self-report, by 8% using pill count and by 6% using rating. In conclusion, among all comparisons of MEMS versus alternative methods for measuring adherence, only a few used adequate comparisons in terms of outcome measures, coverage periods and statistical method. Researchers should therefore use stronger methodological frameworks when comparing measurement methods and be aware that non-electronic measures could lead to overestimation of medication adherence.