Heparanase is a promising anticancer target because of its involvement in cancer invasion and metastasis. Heparanase cleaves heparan sulfate (HS), a sulfated polysaccharide, and activates a series of HS-mediated cell proliferation and angiogenesis processes. Understanding the substrate specificity of heparanase will aid the discovery of heparanase inhibitors. Here, we sought to determine the specificity of heparanase using synthetic polysaccharide substrates. The substrates were prepared using purified HS biosynthetic enzymes. Using these substrates, we were able to dissect the structural moieties required for heparanase. Our data suggest that heparanase cleaves the linkage between a GlcA unit and an N-sulfo glucosamine unit carrying either a 3-O-sulfo or a 6-O-sulfo group. In addition, heparanase cleaves the linkage of a GlcA unit and N-sulfo glucosamine unit with a 2-O-sulfated GlcA residue, not a 2-O-sulfated IdoA residue, in proximity. We also discovered that the polysaccharide with repeating disaccharide units of IdoA2S-GlcNS inhibits the activity of heparanase. Our findings advance the understanding of the substrate specificity of heparanase and identify a lead compound for developing polysaccharide-based heparanase inhibitors.