Heparanase activity correlates with metastatic potentials of lymphoma, melanoma and mammary adenocarcinoma cell lines. We investigated the ability of various modified species of heparin and size homogeneous oligosaccharides derived from depolymerized heparin to inhibit: a) heparanase-mediated degradation of heparan sulfate (HS) in the extracellular matrix (ECM) deposited by cultured endothelial cells, and b) lung colonization of B16-BL6 melanoma cells in C57BL mice. For this purpose, melanoma cells or conditioned medium were incubated with metabolically sulfate-labeled subendothelial ECM in the absence and presence of heparin, heparin fragment or nonanticoagulant species of heparin. Labeled HS degradation fragments released into the incubation medium were analyzed by gel filtration over Sepharose 6B. The B16-BL6 melanoma cells were also tested for lung colonization following their intravenous administration to C57BL mice, in the absence and presence of the various species of heparin. Inhibition of both heparanase and melanoma lung colonization depended on the size and degree of sulfation of the heparin molecule, the position of sulfate groups, and the occupancy of the N position of the hexosamines. Inhibition of heparanase was best achieved by heparin species containing 16 sugar units or more and having sulfate groups at both the N and O positions. Low sulfate oligosaccharides were less effective heparanase inhibitors than medium and high sulfate fractions of the same size saccharide. While O-desulfation abolished the heparanase inhibiting effect of heparin. O-sulfated, N-substituted (e.g., N-acetyl or N-hexanoyl) species of heparin retained a high inhibitory activity, provided that the N-substituted molecules had a molecular size of about > or = 4,000 daltons. Potent inhibitors of heparanase activity were also efficient inhibitors of tumor invasion and lung colonization. The antimetastatic and anticoagulant activities of heparin were unrelated, as indicated by using heparin fractions with high and low affinity for antithrombin III. These heparins differ about 200-fold in their anticoagulant activity, but expressed similar high antiheparanase and antimetastatic activities. It appears that heparanase-inhibiting species of heparin interfere with the passing of tumor cells across the capillary wall, as they significantly inhibited metastasis even when injected up to 3 h after lodgment. Structural requirements for inhibition of heparanase activity and lung colonization of melanoma cells by species of heparin were different from those identified for a) release of ECM-bound basic fibroblast growth factor (bFGF), and b) stimulation of bFGF receptor binding and mitogenic activity.(ABSTRACT TRUNCATED AT 400 WORDS)