Thanks to John Deere, this stoop-dwelling, turnstile-hopping New York City kid got to ride in her first tractor! Fortunately for everyone, I didn’t have to drive it. That’s because, with just a swipe on a smartphone app, the tractor takes care of that part itself. Julian Sanchez, the director of emerging technology at Deere, met with me at the company’s test farm just outside of Des Moines, Iowa to walk me through the details of how its new kit, which is designed for John Deere model 2020 tractors and was first revealed at CES 2022, delivers full autonomy.
We began inside a garage-like structure, where the tractor — a tilling attachment protruding from the back of it — loomed silently, dwarfing the pair of us. I was used to seeing these things glide through cornfields from a distance as I drove by in my car. This was an entirely different experience.
“For the last two decades, we have been making significant investments in getting all the vehicles ready for autonomy,” Sanchez told me. “The last piece is perception, to be able to detect obstacles.” This perception, he continued, comes in the form of six pairs of stereo cameras — three in the front, two on each side and one in the back.
In addition to those cameras, the tractor has GPS guidance, gathers data about the health of crops in the field and other important business metrics, and with the companion smartphone app, farmers have access to live video, images and data, the farmer can adjust the tractor’s orders (i.e. change speed or direction), monitor its job progress and receive alerts of anomalies the software doesn’t know how to handle.
As Sanchez put it, farmers are known for having a very “prove it” mentality, and so, it was crucial that Deere make it easy for them to check in on their equipment and its status. “When we started developing autonomy, we put in place a ride-along mode while it’s in autonomy,” he said, explaining that this mode is an entire experience designed specifically to allow a farmer to sit in the cab of the tractor while it runs autonomously and receive information about what the tractor is doing and intending to do. This builds trust, Sanchez said.
The smartphone application is also part of building that trust as it alerts the farmer anytime there is an issue like an obstacle in the way. From the app, the farmer can see exactly what each other those six cameras is seeing, enabling him to make a decision. “They can look at what the cameras are seeing and say, ‘Oh, that’s just an old tire; go over it,” Sanchez said.
Can you just “go over” a tire? Apparently, yes, you can sometimes. “The point is,” he reminded me, “the farmer can make the call.”
Training the algorithm
However, teaching these cameras to understand exactly what they were perceiving, and therefore, know when and if to alert the farmer at all, was its own challenge. “The hard thing for these systems is that you need a lot of training data to teach the [AI] algorithm so it can figure out what’s an obstacle and what’s not an obstacle,” Sanchez said.
With traditional autonomous driving — more in line with what we’re seeing from Tesla, for instance — training can happen out on city streets where the AI gets all sorts of real-world input and data points: Pedestrians, stop signs, streetlights, bicycles, you name it.
“But on a farm,” Sanchez pointed out, “you can drive for hours and you’re not getting anything valuable to teach it.” He said that Deere gathered “gazillions” of images but wasn’t able to teach the algorithms anything of value because, well, there wasn’t anything in the images.
“First, we thought about putting things artificially out there to train the algorithm, but that was expensive and time consuming,” he continued, adding that this method also felt contrived — after all, the whole purpose of the AI is to detect real-world situations.
“Then, we had an ‘ah ha’ moment,” he recalled, directing me over to the garage window, beyond which the test farm stretched in all directions. “Look out there.” He pointed to the right. “You see dirt, you see trees and you see skyline. Then you look over there.” He swiveled to the left. “What do you see? Dirt, trees, sky. Same thing over there,” he said, gesturing directly in front of us.
After looking at hundreds of images of fields, all of which were nearly identical to what was laid out before me, the engineers had an idea. “What if, instead of training the algorithm to detect objects, you trained it detect dirt, trees and sky,” Sanchez suggested. So anytime the algorithm says something isn’t one of those three things, it alerts that there is an obstacle. Because it doesn’t matter what the obstacle is, does it? There is no reason for anything at all to be in front of a tractor. Even if it can handle driving over a tire.
“We play[ed] to our strengths,” said Sanchez. “That was a big unlock for us.”
And this is what makes it fully autonomous. In fact, according to Sanchez, John Deere’s autonomy kit can make any of its tractors a Level 5 autonomous vehicle. “Before those cameras and before the algorithm, we would say our equipment is self-driving. You had to be there. This is like level 5 in the field.” While the farmer must still manually drive the tractor to the edge of the field to begin work, once it’s there, it’s simply a matter of swiping the on switch.
Why autonomy?
To answer this question, maybe it’s best to begin with another question: What is it that farmers really do? Of course, there is the short answer, which is produce crops; but then, there is the longer, more detailed answer that involves several time-sensitive and time-consuming steps.
“You take several trips to the field and go over the same land to complete the cycle of a crop. In many ways, we think of the tractor and the tillage implement as the first step,” Sanchez informed me, adding that generally, farming can be broken down into four steps, which are:
- Tillage: Taking place in the fall, this step prepares the land for crops and requires a tilling machine to be attached to the back of tractor.
- Planting: In the Spring, farmers attach a planter to the same tractor to plant seeds into the tilled soil.
- Spraying: Occurring during the summer, spraying begins once the plants start to grow, and may involve spray nutrients or herbicides to keep the crops healthy.
- Harvesting: Finally, the farmer will go through the fields with a harvester, gathering up all the crops.
While only two of these steps — tillage and planting — require a tractor, all of them require skilled labor, something that is in alarmingly short supply, which is why farmers have been asking John Deere for more autonomy for nearly two decades.
“Over the last 30 years, fewer people live in rural areas and so it’s hard to find any labor, but especially skilled labor,” Sanchez said. “If you take the technology away, to become proficient at tilling, you’d need about 40-50 hours of practice; but for harvesting with a combine, you probably need about 1,000 hours of practice to be considered proficient.”
Because the tractors are highly connected, its objectives can be pre-configured on the smartphone app, so as long as there is connectivity, a farmer can instruct his operator to simply go into the tractor’s cab, align the vehicle with the beginning of the field and hit accept. From there, the tractor will complete the pre-established job, all by itself. In this way, autonomy and connectivity can reduce — though likely not eliminate — the dependance on skilled labor.
Though a tractor driving itself might not move through a field any quicker than one with a driver onboard, it does enable the farmer to spend his time elsewhere, whether that’s tending to other business or relaxing. And let me tell you, they deserve the break. According to a 2020 Farm Journal Pulse report, 50% of the 1600 farmers and ranchers polled said that they typically worked between 10 and 14 hours daily. Nearly 20% said that they worked about 15 hours a day.
Additionally, farmers have long expressed that the most expensive aspect of their job are what Sanchez referred to as “inputs” — fertilizer, herbicides, seeds, etc. “Farmers want to use less stuff, which is good for their pocket and good for the environment. Any technology we can put in place that helps with precision and helps them do more with less, is a technology that we know they will adopt.”
Here, he is referring to the company’s efforts around using sensors and GPS technology to enable its tractors to control the placement of seeds and chemicals with sub-inch accuracy, reducing wasted product. In a previous conversation, John Deere’s Director of Intelligent Solutions Nancy Post told Enterprise IoT Insights that the company estimates that a farmer could save up to 90% on their input with precision technology.
A recent attention-grabbing headline further illustrates both the level of sophistication and the benefits of this sophistication present in the modern-day John Deere vehicle lineup. In early May, CNN reported that Russian troops in the occupied city of Melitopol stole 27 pieces of farm machinery from an equipment dealership and shipped them to Chechnya. Ukrainian officials know that’s where the equipment was sent because it was all equipped with GPS. However, after the nearly 700-mile journey, it was discovered that all the equipment had been locked remotely by the dealership, making it inoperable.
What’s next?
John Deere can comfortably claim that is has achieved autonomy in the tillage phase of farming, but according to Sanchez, that’s only the beginning. “Our goal would be a fully autonomous production cycle by 2030,” he said. “We solved the labor pressure for the tillage, which is an important one because tillage happens while harvesting, so it’s like two things are happening at once. We solved that, but we want farmers to be able to experience autonomy at every production stage and rely on it.”
The company is also paying attention to the impact of climate change and changing weather patterns on farming. “Climate change makes the windows of production much less predictable,” stated Sanchez. “And we think that technology can provide a buffer of predictability.”
My tour ended out in a muddy field, where I climbed into a roaring, vibrating tractor for the very first time. Sanchez sat beside me as another engineer stood several yards away with a smartphone. We gave him a signal and he swiped it on. The tractor lurched forward, and we were off. It was a bumpy, but sort of pleasant, ride. It was easy to imagine sitting up there with a cup of coffee as the tractor took two to three hours to till or plant a field. It sounded nice, honestly.
But then I thought about what else can happen in three hours, whether it’s tackling a work task that’s been weighing on you, training a new hire so that you can run your business more efficiently, taking care of an important errand, having breakfast with your kids. A lot can happen in a few hours, and I was moved by John Deere’s commitment to giving farmers back some of this precious time, especially as they face ongoing labor and climate challenges, all while shouldering the burden of feeding a growing population.
