Thermal sensors are robust to conditions that are constraints for visual sensors, such as illumination changes. Based on this effectiveness, studies on hand tracking have enabled the demonstration of higher performance with low computation times. In this paper, we propose a novel hand detection method and tracking framework based on information extracted from thermal images. An adaptive hand detection (AHD) method was designed with five models that use temperature analysis to obtain the region of interest of the hand. To improve performance, we introduced an AHD-based automatic tracking-by-detection algorithm using the Kernelized correlation filters tracker. Finally, we combined the hand detection and tracking algorithms into a single framework, a guidance framework for tracking by detection (GFTD). To verify the performance, we evaluated the accuracy of the hand detection using success rate (Intersection over Union) and the trajectory using object tracking error (OTE). The proposed GFTD improves the performance in terms of success rate and OTE by 15% and 16.3%, respectively, compared with the conventional methods.