Abstract: Agricultural heritage is a living heritage with important social, economic, and ecological value, as well as distinctive cultural landscape features. Identifying the cultural landscape genes of agricultural heritage sites has positive significance. In existing researches related to landscape genes, most traditional settlements and historical architectural neighborhoods have been studied from the perspective of landscape genes. There have been very few analyses and research results on the landscape genetic characteristics of agricultural heritage sites. This study can fill the gaps in landscape genetic research on agricultural heritage sites in the domestic academic field, which is a complementary and helpful area of research on landscape genes and dynamic conservation of agricultural heritage. Based on landscape gene theory and local theory, the Chaozhoudancong Tea cultural system is considered an example to identify landscape genes, and farmers are used as research subjects to explore the intrinsic link between tea cultural landscape genes and farmers’ perceptions and to observe the effect of differences in the perceptions of different livelihood types of farmers on the evolution of tea cultural landscape genes. The three dimensions of ecological, production, and living genes constitute the landscape genes of the Chaozhoudancong Tea culture system. Combined with questionnaire surveys and in-depth interviews with local farmers, the data were analyzed using the related software SPSSAU (SPSS Online), and the local farmers’ perceptions of the changes in the landscape genes of the tea culture were obtained. The results indicate the following: 1) Farmers observed significant changes in the ecosystem, living environment, farming tools and techniques, and lifestyle of the tea plantations and villages in the last decade. 2) Farmers in high mountain tea-growing areas perceived environmental changes in the mountains, tea plantations, and villages more strongly, and farmers in low mountain tea-growing areas perceived changes in cultivation techniques and knowledge more strongly. 3) Farmers with different livelihood strategies showed significant differences in their perceptions of the landscape genes of tea culture, with tea farmers of a single livelihood perceiving indicators, such as “changes in the area of hills and terraces”, more strongly. 4) There were differences in the perception of landscape genes of the tea culture among farmers with different total tea revenues. Farmers in high-altitude tea areas, with single livelihood strategy and higher tea incomes, had a stronger perception of landscape genes changes in tea culture and a higher awareness of ecological conservation but a low degree of livelihood diversification, and they were exposed to higher shocks from market risks, natural disasters, and other external risks. Farmers in low mountain tea areas, with diverse livelihood strategies and lower tea incomes, had a poorer awareness of ecological conservation and weaker perception of landscape genes changes. However, ecosystems and land resources, such as natural capital, are important components of farmers’ livelihood capital, and ecological vulnerability can lead to the vulnerability of farmers’ livelihoods. Methods to enhance the resilience of the livelihoods of different types of farmers can be proposed for raising their awareness of ecological protection and heritage inheritance and providing ideas for the sustainable development of agricultural heritage sites.