Protein tyrosine phosphatase 1B (PTP1B) is considered a potential target for the treatment of type II diabetes and obesity due to its critical negative role in the insulin signaling pathway. However, improving the selectivity of PTP1B inhibitors over the most closely related T-cell protein tyrosine phosphatase (TCPTP) remains a major challenge for inhibitor development. Lys120 at the active site and Ser27 at the second pTyr binding site are distinct in PTP1B and TCPTP, which may bring differences in binding affinity. To explore the determinant of selective binding of inhibitor, molecular dynamics simulations with binding free energy calculations were performed on K120A and A27S mutated PTP1B, and the internal changes induced by mutations were investigated. Results reveal that the presence of Lys120 induces a conformational change in the WPD-loop and YRD-motif and has a certain effect on the selective binding at the active site. Ser27 weakens the stability of the inhibitor at the second pTyr binding site by altering the orientation of the Arg24 and Arg254 side chains via hydrogen bonds. Further comparison of alanine scanning demonstrates that the reduction in the energy contribution of Arg254 caused by A27S mutation leads to a different inhibitory activity. These observations provide novel insights into the selective binding mechanism of PTP1B inhibitors to TCPTP.