A unifying mathematical theory of nonequilibrium thermodynamics of stochastic systems in terms of master equations is presented. As generalizations of isothermal entropy and free energy, two functions of state play central roles: the Gibbs entropy S and the relative entropy F , which are related via the stationary distribution of the stochastic dynamics. S satisfies the fundamental entropy balance equation dS/dt = e p - h d/T with entropy production rate e p ≥ 0 and heat dissipation rate h d, while dF/dt = -f d ≤ 0. For closed systems that satisfy detailed balance: Te p(t)=f d(t). For open systems, one has Te p(t) = f d(t)+Q hk(t), where the housekeeping heat, Q hk ≥ 0, was first introduced in the phenomenological nonequilibrium steady-state thermodynamics put forward by Oono and Paniconi. Q hk represents the irreversible work done by the surrounding to the system that is kept away from reaching equilibrium. Hence, entropy production e p consists of free energy dissipation associated with spontaneous relaxation (i.e., self-organization), f d, and active energy pumping that sustains the open system Q hk. The amount of excess heat involved in the relaxation Q ex = h d - Q hk = f d -T(dS/dt). Two kinds of irreversibility, and the meaning of the arrow of time, emerge. Quasistationary processes, adiabaticity, and maximum principle for entropy are also generalized to nonequilibrium settings.