A 57-year-old man consulted our hospital with a history of the gradual onset of dyspnea and a productive cough. Chest computed tomographic (CT) scans showed a nodular shadow in a cavity lesion, and reticulonodular, cystic, and ground-grass opacities in the bilateral lung fields with honeycombing. He was diagnosed as having pulmonary aspergilloma and idiopathic pulmonary fibrosis (IPF). As an outpatient, he suffered from dyspnea upon physical exertion with exacerbation of the high-resolution CT (HRCT) opacities. An inhalation provocation test for Aspergillosis fumigatus was positive and chronic hypersensitivity pneumonitis (CHP) caused by Aspergillus was finally diagnosed. Insidious CHP is sometimes misdiagnosed as IPF. The diagnosis of insidious CHP should be made on the basis of a detailed history, specific HRCT findings, and lymphocyte-dominant bronchoalveolar lavage fluid cell findings.