A method was developed for registering three-dimensional knee implant models to single plane X-ray fluoroscopy images. We use a direct image-to-image similarity measure, taking advantage of the speed of modern computer graphics workstations to quickly render simulated (predicted) images. As a result, the method does not require an accurate segmentation of the implant silhouette in the image (which can be prone to errors). A robust optimization algorithm (simulated annealing) is used that can escape local minima and find the global minimum (true solution). Although we focus on the analysis of total knee arthroplasty (TKA) in this paper, the method can be (and has been) applied to other implanted joints, including, but not limited to, hips, ankles, and temporomandibular joints. Convergence tests on an in vivo image show that the registration method can reliably find poses that are very close to the optimal (i.e., within 0.4 degrees and 0.1 mm), even from starting poses with large initial errors. However, the precision of translation measurement in the Z (out-of-plane) direction is not as good. We also show that the method is robust with respect to image noise and occlusions. However, a small amount of user supervision and intervention is necessary to detect cases when the optimization algorithm falls into a local minimum. Intervention is required less than 5% of the time when the initial starting pose is reasonably close to the correct answer, but up to 50% of the time when the initial starting pose is far away. Finally, extensive evaluations were performed on cadaver images to determine accuracy of relative pose measurement. Comparing against data derived from an optical sensor as a "gold standard," the overall root-mean-square error of the registration method was approximately 1.5 degrees and 0.65 mm (although Z translation error was higher). However, uncertainty in the optical sensor data may account for a large part of the observed error.