Introduction to AI for Agriculture

Hannah Kerner
Assistant Professor, Arizona State University
AI/ML Lead, NASA Harvest
Center Faculty, ASU Center for Global Discovery and Conservation Science
1
Agriculture, food security, and climate change
is one of the most at-risk industries to

Climate change
Agriculture 🌽
🌡
is one of the biggest contributors to

Effects of climate change on agriculture: Drivers of climate change from agriculture:

🌡 Changing temperatures & precipitation 🐄 GHGs from methane emissions of
patterns can reduce crop productivity livestock and rice production, emissions
🌪 Increased frequency of extreme weather from fertilizers, etc.
events (droughts, floods) cause 🌳 Land use changes reduce carbon sinks
increased crop loss (e.g., forests, grasslands) and biodiversity
🚰 Reduced water & increased competition 🦠 Degraded soil health reduces carbon
for resources sequestration and ecosystem health
Who needs information about agriculture / food security?
Who are the end-users / decision makers?
Who needs information about agriculture / food security?
Farmers

Policymakers

Aid organizations

Consumers

Economists

Agronomists

…
What do farmers want to know?
What do you think?
What do farmers want to know?
● When to plant?
● Crop performance
● Potential threats to production (e.g.,
climate change)
● Actual threats to production (e.g.,
nearby pest/disease outbreak or
weather forecasts)
● Soil moisture, rainfall, temperature, etc.
● Productivity potential (yield gap)
What do farmers want to know?
● When to plant?
● Crop performance
● Potential threats to production (e.g.,
climate change)
● Actual threats to production (e.g.,
nearby pest/disease outbreak or
weather forecasts)
● Soil moisture, rainfall, temperature, etc.
● Productivity potential (yield gap)
What do farmers want to know?
● When to plant?
● Crop performance
● Potential threats to production (e.g.,
climate change)
● Actual threats to production (e.g.,
nearby pest/disease outbreak or
weather forecasts)
● Soil moisture, rainfall, temperature, etc.
● Productivity potential (yield gap)
● Suitability of crops (would a different
crop or variety grow better?)
What do policymakers want to know?
● Crop performance
● Potential threats to production
● Actual threats to production
● When to intervene
● How to intervene
● Productivity potential
● Suitability of crops
● How suitability will change
● Measure impacts of policies

Source: IPCC Sixth Assessment Report

NASA Harvest enables adoption of
satellite Earth observations by public and
private organizations to benefit food
security, agriculture, and human and
environmental resilience worldwide.
🚀
Harvest AI + Africa teams with Rwanda 2017
Space Agency leadership in Kigali

🚀
NASA2023
Acres focuses on applying satellite
Earth observation (EO) information to the
most pressing agricultural and food
security challenges facing U.S. farmers,
Acres AI team with local partners at ranchers, and agrifood system
Hawaii Taro Farm in Maui stakeholders.
NASA Harvest enables adoption of
FEWS NET provides evidence-based
satellite Earth observations by public and
analysis to help government
private organizations to benefit food
decision-makers and relief agencies plan
security, agriculture, and human and
for and respond to humanitarian crises.
environmental resilience worldwide.

Request for
analysis
Event X in region Y
What is the impact of
corresponded to a
extreme event X on
change of N (%, ha) in
cultivation in region Y?
crop cultivation.

12
Request for
analysis
What is the impact of Cultivated area in Sudan
the 2023 war on ROIs is 20-30% lower in
cultivation in Sudan? 2023 compared to 2022.

September 2023 November 2023

13
Goal: create baseline geospatial datasets for
measuring and monitoring agricultural production
in Maui County to support policy & efforts to improve
food security

● Machine learning models predict where crops

growing based on satellite data
● Integrate crop maps with other relevant
datasets (e.g., socioeconomic and price data)
in a public Food Security Dashboard
● End users: Maui United Way, farmers, Dept of
Ag, county council, community organizations Crop map
People usually think of precision ag on the ground…
But we also have data from the sky

Since the 1970’s

Source: NASA
ML needed to extract Essential Agricultural
Variables (EAVs) from satellite observations!
Survey of key topics
Proximal / In Situ Sensing

Crop mapping → Binary classification

Artificial Intelligence Crop type mapping → Multi-class classification

(including ML, CV)
Agriculture and
Food Security Field boundary delineation → Segmentation
⭐ this section
Yield estimation → Regression

Pest and disease detection → OOD detection

Remote Sensing

Domain adaptation, distribution shift, multi-fidelity

data fusion, learning from limited labeled data, etc.
19
Crop mapping
binary classification of pixels as crop or non-crop

22
Crop mapping
binary classification of pixels as crop or non-crop
Classifying cropland in Togo (Kerner & Tseng et al., 2020)

12 month time series of Sentinel-2

23
and other satellite observations
Crop type mapping
multi-class classification of pixels into N crop types
USDA Cropland Data Layer

Corn Soybean Sweet corn Alfalfa …

24
Crop type mapping
multi-class classification of pixels into N crop types
Time series vision transformer (TS-ViT) for crop type classification (Tarasiou et al., 2023)

Tarasiou, M., Chavez, E., & Zafeiriou, S. (2023). Vits for sits: Vision
transformers for satellite image time series. CVPR. 25
Crop type mapping
binary classification of one vs. rest crop types

Coming soon: deep dive

Predicted map of taro in Maui county using fine-tuned

Presto (pre-trained remote sensing transformer) 26
Field boundary delineation
segmentation of individual field/parcel boundaries
Radiant Earth South Africa Field Boundaries

27
Field boundary delineation
segmentation of individual field/parcel boundaries
Field boundary delineation using deep
transfer learning and weak supervision
(Wang et al., 2022)

● Trained model using label-rich dataset

from France and fine-tuned using
sparse dataset from India
● Weak supervision: loss masked to
ignore pixels without labels (India)
● FracTAL-ResUNet
○ self-attention layer: FracTAL unit
○ skip-connections (ResNet)
○ encoder-decoder architecture
(U-Net)
28
Yield estimation
estimation of crop harvested per unit area, e.g., kg/ha
Maize yields in US 2018 (Deines et al., 2020)

29
4 tonnes/ha 17 tonnes/ha
Yield estimation
estimation of crop harvested per unit area, e.g., kg/ha
TIML: Task-Informed Meta-Learning
(Tseng et al., 2022)

● Task: predict county-scale yield in

US using satellite time series
● All pixels in entire county too large to
use as model input
→ 3D histograms (band, time, bin)
● TIML algorithm uses task metadata
(lat/lon, state) to move starting
weights closer to optimum for
transfer learning
Tseng, G., Kerner, H., Rolnick, D. (2022). TIML: Task-Informed Meta-Learning for crop type
mapping. Association for the Advancement of Artificial Intelligence (AAAI) Workshops, AI for 30
Agriculture and Food Systems (AIAFS).
Pest, disease, and hotspot detection
detection of in-field anomalies that represent unfavorable
growing conditions

low high
Downy mildew disease anomaly score
31
Pest, disease, and hotspot detection
detection of in-field anomalies that represent unfavorable
growing conditions
Despite great promise, few recent studies using AI for detection in remote sensing images
Domain-agnostic Outlier Ranking Algorithms (DORA)

low high Kerner et al., 2022

32
anomaly score
Survey of key topics
Proximal / In Situ Sensing

Precision agriculture (resource optimization)

⭐ this section
Artificial Intelligence Robotic farming
(including ML, CV)
Agriculture and
Food Security Yield estimation and optimization

Pest and disease identification

Livestock and rangeland management

Remote Sensing

Domain adaptation, distribution shift, multi-fidelity

data fusion, learning from limited labeled data, etc.
33
Precision agriculture
data-driven management of on-farm resources (water, nutrients, equipment, etc.)
Precision agriculture
data-driven management of on-farm resources (water, nutrients, equipment, etc.)

Optimizing Nitrogen Management with Deep

Reinforcement Learning and Crop Simulations
(Wu et al., 2022, AAAI Workshops)

● Goal: learn policy for N application that

minimizes input and leaching without
jeopardizing yield
● Train management policies with deep
Q-network and soft actor-critic
algorithms
● Gym-DSSAT interface models daily
interactions between the simulated crop
environment and RL agents
● RL policies achieve higher or similar yield
while using less fertilizer for maize in Iowa
and Florida experiments
Robotic farming
automating farming operations such as seeding, harvesting, sorting, or spraying

Semi-Supervised Object Detection for

Agriculture
(Tseng et al., 2023, AAAI Workshops)
● Goal: detect identify locations of
weeds vs. crops (wheat) in field
robot images for precision spraying
● Train student-teacher models for
semi-supervised object detection for
two classes: wheat and weeds
● Autonomous robots can then spray
individual weed plants while avoiding
crops
Yield estimation and optimization
data-driven management of on-farm resources (water, nutrients, equipment, etc.)

Counting Apples and Oranges

with Deep Learning: a Data
Driven Approach
(Chen et al., 2016, IEEE Robotics
and Automation Letters)

● Goal: estimate fruit yield in

orchards using AI + drones
● Train model to detect fruit
instances and regress
counts using real-time
drone images
Pest, disease, and hotspot detection
detection and identification of crop diseases
Example: Scoring root necrosis in
cassava using semantic segmentation
Tusubira et. al, 2020, CVPR AgVision
Workshop

● Goal: calculate area of root

necrosis caused by Cassava
Brown Streak Disease (CBSD)
● Necrosis score: percentage of
area predicted as necrotized
● Labels by specialists at National
Crop Resources Research
Institute (NaCRRI)

38
PlantVillage Nuru app (Silva et al., 2021)
Livestock and rangeland management
monitoring & optimizing animal behavior, health conditions, and feeding patterns
Example: Automated Body Condition Scoring
of Dairy Cows using 3-Dimensional Feature
Extraction from Multiple Body Regions
Song et. al, 2019, Journal of Dairy Science

● Goal: automatically assess body

condition of dairy cows for livestock
● Manually computed body condition
score (BCS) for dairy cows in real
conditions
● Extract vision-based features related to
body condition from camera images
● Compute BCS of new images using
1-nearest neighbor in training set
Deep dive: how Presto works and how it is used for
crop type mapping (and other agriculture use cases)

Predicted map of taro in Maui county using fine-tuned

Presto (pre-trained remote sensing transformer)
Self-supervised “foundation”
models for remote sensing

Supervised fine-tuning
using small[er] labeled datasets
Self-supervised pre-training for specific regions and/or
leveraging massive archive of application tasks
unlabeled remote sensing data
Presto’s design is inspired by agriculture use
cases and the unique character of satellite data

Presto
(Pretrained remote
sensing transformer)
Presto’s design is inspired by agriculture use
cases and the unique character of satellite data
1. Use a wide diversity of
input sensors at
pre-training time
Presto’s design is inspired by agriculture use
cases and the unique character of satellite data
1. Use a wide diversity of
input sensors at
pre-training time

2. Apply structured masking

strategies so that the
model can handle missing
channels or timesteps

Masked autoencoder
Refresher: what are Transformers?

https://jalammar.github.io/illustrated-transformer/
Masked autoencoders Je suis étudiant

�� https://jalammar.github.io/illustrated-transformer/
Masked autoencoders NDVIt1 , NDVIt2 , …

https://jalammar.github.io/illustrated-transformer/
NDVIt1 , <mask> , …
Presto’s masked autoencoder
Presto’s design is inspired by agriculture use
cases and the unique character of satellite data
1. Use a wide diversity of
input sensors at
pre-training time

2. Apply structured masking

strategies so that the
model can handle missing
channels or timesteps

3. Use time-series inputs,

which are much smaller
than images. Smaller
inputs → smaller model
Fine-tuning Presto for crop type classification
Crop type label

Mask
missing
channels /
timesteps
(if needed)

Latitude Longitude Crop

type
Encoder
21.014 -156.607 Taro
Presto
features
fine-tune or train
Taro label point in Sentinel-2
Small classifier (MLP, random forest, etc.)
Presto’s design is inspired by agriculture use
cases and the unique character of satellite data
1. Use a wide diversity of input
sensors at pre-training time

2. Apply structured masking

strategies so that the model
can handle missing channels
or timesteps

3. Use time-series inputs, which

are much smaller than
images. Smaller inputs →
smaller model

4. Global representation and

diversity in pre-training data
Fine-tuning Presto for downstream tasks is
computationally efficient

Presto embeddings + logistic regression 🌽 CropHarvest

benchmark
Presto is lightweight and accessible

1. Easy loading from a

python package

2. Single-file version for

integration into other
applications

3. Model weights stored on

git (3.17Mb)

4. Finetuning possible in
minutes on a 2017
Macbook Pro
What are the challenges?

What do you think?

What are the challenges?

Learning about the real world Getting initial promising results Trying to make your models work
impact we can have with AI on clean data for real data and deployment
Challenges for AI + agriculture
• Labeled/ground-truth data sparse
and difficult/expensive to acquire

Rolf, E., Klemmer, K., Robinson, C., and Kerner, H. (2024). Mission
Critical–Satellite Data is a Distinct Modality in Machine Learning. ICML.
Challenges for AI + agriculture
• Labeled/ground-truth data sparse
and difficult/expensive to acquire
• Benchmark datasets over-idealized
and not representative of real world

Tarasiou, M., Chavez, E., & Zafeiriou, S. (2023). ViTS

• High intra-class variance for SITS: Vision transformers for satellite image time
series. CVPR.
• Low inter-class variance
• Multi-label (e.g., intercropping)
• Labels change inter-annually
Maize intercropped with beans in Kenya • Noisy data and labels
Muoni et al., 2019 • Label acquisition difficult
Challenges for AI + agriculture
• Labeled/ground-truth data sparse
and difficult/expensive to acquire
• Benchmark datasets over-idealized
and not representative of real world
• Spatial and temporal generalization
difficult

28 July 2018 30 July 2019

Challenges for AI + agriculture
• Labeled/ground-truth data sparse
and difficult/expensive to acquire
• Benchmark datasets over-idealized
and not representative of real world
• Spatial and temporal generalization
difficult
• Lack of open data and code for
reproducibility, re-use, and
benchmarking

Tseng, G., Zvonkov, I., Nakalembe, C., Kerner, H. (2021). CropHarvest: a global
satellite dataset for crop type classification. To appear in Neural Information
Processing Systems (NeurIPS) Datasets and Benchmarks.
Challenges for AI + agriculture
• Labeled/ground-truth data sparse
and difficult/expensive to acquire
• Benchmark datasets over-idealized
and not representative of real world
• Spatial and temporal generalization
difficult
• Lack of open data and code for
reproducibility, re-use, and
benchmarking
• End-user uptake, communication,
deployment, and sustainability takes
significant time, effort, and $$
Challenges for AI + agriculture
• Labeled/ground-truth data sparse
and difficult/expensive to acquire
• Benchmark datasets over-idealized
and not representative of real world
• Spatial and temporal generalization
difficult
• Lack of open data and code for
reproducibility, re-use, and
benchmarking
• End-user uptake, communication,
deployment, and sustainability takes
significant time, effort, and $$
• Lower performance and investment
in data-sparse regions (e.g., Kerner et al. (2024). How accurate are existing land cover maps for agriculture in
Sub-Saharan Africa? Nature Scientific Data.
sub-saharan africa)
Ready to get started?

🌽 CropHarvest dataset
https://github.com/nasaharvest/cropharvest

🧠 CVPR 2022 tutorial

https://nasaharvest.github.io/cvpr2022.html

🛰 CVPR 2022 tutorial

https://appliedsciences.nasa.gov/what-we-d
o/capacity-building/arset/arset-agriculture-tr
ainings