Abstract: Improving the ecological efficiency of rice production is key to achieving the dual goals of guaranteeing the stable growth of grain production and reducing the environmental loss of cultivated land use. Cultivated land is the basic source of agricultural production. Therefore, exploring the impact of cultivated land operating scale on the ecological efficiency of rice production by farmers is of great importance in promoting moderate-scale agricultural operations and the sustainable utilization of cultivated land. Unlike other studies that have mostly focused on the medium and macro dimensions of cities and provinces, this study was based on the micro dimension of grain production—farmers’ perspectives. Moreover, the conclusions obtained from this study can compensate for the lack in previous studies in terms of targeted exploration on the impact mechanism of cultivated land operating scale on farmers’ rice production (ecological) efficiency and the existence of a nonlinear relationship between them. Using data from 416 questionnaire surveys from farmers in Changde City, a random frontier analysis method was used to construct an efficiency calculation model and an impact model to empirically test the relationship between cultivated land operating scale and the farmers’ ecological efficiency of rice production. The results were: 1) Farmer scale levels and the rice production ecological efficiency need to be improved. Among the sample farmers, the number of smaller operating scale farmers was the highest, accounting for 94.95% of the total sample, indicating that small-scale farmers remain the main forces of agricultural production. The average farmers’ ecological efficiency of rice production was 0.830, with a room for improvement of 0.170. 2) Cultivated land operating scale had a significant impact on the farmers’ ecological efficiency of rice production. The ecological efficiency of rice production by farmers tended to increase and then decrease with the expansion of the cultivated land operating scale. Their relationship was non-simple linear but exhibited an “inverted U” with an inflection point located in the range of 1.2−1.4 hm². It is not necessarily true that the larger the cultivated land operating scale, the higher the ecological efficiency of rice production by farmers. The blind expansion of the cultivated land operating scale could lead to a decrease in the farmers’ ecological efficiency of rice production. Therefore, it is necessary to promote moderate-scale cultivation of cultivated land. 3) The farmers’ ecological efficiency of rice production was also significantly positively affected by the education level of the household head, dependency ratio, and the proportion of agricultural income. It was significantly negatively affected by the age of the household head. The aforementioned information indicates that we must recognize that small-scale farmers will continue to exist for a long time. Based on this, we should further increase plot sizes to promote moderate-scale cultivation of cultivated land and promote cultivated land concentration to reduce the degree of cultivated land fragmentation. Furthermore, the government should not only continuously promote fertilizer reduction and efficiency improvement to accelerate the development of low-carbon agriculture, but also develop new farmers and increase their support to promote the ecological transformation of rice production.