In phase I trials, the main goal is to identify a maximum tolerated dose under an assumption of monotonicity in dose-response relationships. On the other hand, such monotonicity is no longer applied to biologic agents because a different mode of action from that of cytotoxic agents potentially draws unimodal or flat dose-efficacy curves. Therefore, biologic agents require an optimal dose that provides a sufficient efficacy rate under an acceptable toxicity rate instead of a maximum tolerated dose. Many trials incorporate both toxicity and efficacy data, and drugs with a variety of modes of actions are increasingly being developed; thus, optimal dose estimation designs have been receiving increased attention. Although numerous authors have introduced parametric model-based designs, it is not always appropriate to apply strong assumptions in dose-response relationships. We propose a new design based on a Bayesian optimization framework for identifying optimal doses for biologic agents in phase I/II trials. Our proposed design models dose-response relationships via nonparametric models utilizing a Gaussian process prior, and the uncertainty of estimates is considered in the dose selection process. We compared the operating characteristics of our proposed design against those of three other designs through simulation studies. These include an expansion of Bayesian optimal interval design, the parametric model-based EffTox design, and the isotonic design. In simulations, our proposed design performed well and provided results that were more stable than those from the other designs, in terms of the accuracy of optimal dose estimations and the percentage of correct recommendations.