The metabolism of mephenytoin to its two major metabolites, 4-OH-mephenytoin (4-OH-M) and 5-phenyl-5-ethylhydantoin (nirvanol) was studied in human liver microsomes by a reversed phase HPLC assay. Because of preferential hydroxylation of S-mephenytoin in vivo, microsomes (5-300 micrograms protein) were incubated separately with S- and R-mephenytoin. After addition of phenobarbital as internal standard, the incubation mixture was extracted with dichloromethane. The residue remaining after evaporation was dissolved in water and injected on a 60 X 4.6-mm reversed-phase column (5 mu-C-18). Elution with acetonitrile/methanol/sodium perchlorate (20 mM, pH 2.5) led to almost baseline separation of mephenytoin, metabolites, and phenobarbital. Quantitation was performed by uv-absorption at 204 nm by the internal standard method. Propylene glycol was found to be the best solvent for mephenytoin, but inhibited the reaction noncompetitively. 4-OH-M and nirvanol could be detected at concentrations in the incubation mixture as low as 40 and 80 nM, respectively. The rates of metabolite formation were linear with time and protein concentration. The reaction was found to be substrate stereoselective. At substrate concentrations below 0.5 mM S-mephenytoin was preferentially hydroxylated to 4-OH-M, while R-mephenytoin was preferentially demethylated to nirvanol at all substrate concentrations tested (25-1600 microM). These data provide a mechanistic explanation for the stereospecific pharmacokinetics in vivo. The dependence of both metabolic relations on NADPH and the inhibition by CO suggest that they are mediated by cytochrome P-450-type monooxygenases.(ABSTRACT TRUNCATED AT 250 WORDS)