Fever may serve as the primary indicator of underlying infection in children admitted to the pediatric emergency department (PED), especially in high-risk young infants. This study aimed to identify early clinical factors that could help predict bacteremia in young febrile infants. The study included infants under 90 days of age who were admitted to the PED due to fever. Patients were divided into two groups based on the presence or absence of bacteremia and further divided into three age groups: (1) less than 30 days, (2) 30 to 59 days, and (3) 60 to 90 days. Several clinical and laboratory variables were analyzed, and logistic regression and receiver operating characteristic (ROC) analyses were used to identify potential risk factors associated with bacteremia in young febrile infants. A total of 498 febrile infants were included, of whom 6.4% were diagnosed with bacteremia. The bacteremia group had a higher body temperature (BT) at triage, especially in neonates, higher pulse rates at triage, longer fever subsidence time, longer hospital stays, higher neutrophil counts, and higher C-reactive protein (CRP) levels than those of the non-bacteremia group. ROC analysis showed that the best cut-off values for predicting bacteremia in infants with pyrexia were a BT of 38.7 °C, neutrophil count of 57.9%, and CRP concentration of 53.8 mg/L. A higher BT at triage, increased total neutrophil count, and elevated CRP levels may be useful for identifying bacteremia in young febrile infants admitted to the PED.