Why Data is Foundational to the Future Agricultural Economy

And how it's time for regulation and innovation to align

We as a collective human species are an innovative group. Surviving by our brains and networks propelled us from the trees to the moon, and shortly Mars. 

We face a seemingly insurmountable list of problems from contagion to climate, economics to freedom. For the agricultural sector, these seem to affect the ability to feed the world.

There is a hopeful light at the end of the proverbial tunnel. Yes, we are an innovative lot, and these challenges are an opportunity to progress, to grow, to do what we have always done.

Here, I lay out some challenges faced by the agricultural industry and the innovation working to mitigate the worst. Data is the key component that will support the next wave of solutions to the climate crisis.

The 21st Century challenges to the agricultural economy

The 21st century will be hard to predict and full of risk. According to the World Economic Forum, extreme weather and the lack of action on climate change are leading concerns for global business. 

In the water-scarce regions of the world, over 90 percent of all water consumed goes to farms:

“That means that in most parts of the world, we cannot alleviate our water shortages if we can’t find ways to use less water on farms,” Richter says. “But, on the flip side, if we can find ways to save just a small percentage of the water used on farms, it will free up a great volume of water that can be used for other purposes, including restoring depleted rivers and lakes.” Brian, The Nature Conservancy. 

According to the Environmental Defense Fund:

“Nearly half of all agricultural loans are held by lenders with at least one-quarter of their portfolio concentrated in farm operating or real estate loans, and many of those lenders also have correlated risks because of concentrations of loans in particular geographies or related agricultural businesses. This contributes to lending sector vulnerability to climate-related disruptions.”

With so much climate pressure on the flow of capital to agriculture, there is a concern that a single domino collapsing, say the California Central Valley’s Powerhouse agricultural producers, will send the rest of the global agricutural market into uncertainty and collapse. 

Disruption of the “normal” cycles

The normal cycles that the agricultural sector is accustomed to are no longer reliable. The U.S. west is seeing the impacts of disruption firsthand. 

The U.S. Bureau of Reclamation released an in-depth report on drought in the west that used Paleohydrology, use of tree rings to determine drought conditions, for the first time. Some highlights from the report were:

• There was an increase in variability of climate in the future.

• There will be less water available in the U.S. west in the future.

• Droughts will be more persistent with severe droughts increasing in intensity. 

• Water will come earlier in the year than what crops are accustomed to, meaning changes in types of crops and yields are likely.

The above applies to surface and groundwater supply, the twin fuels of the agricultural economy. Conflict over water is constant, but it too will increase in frequency and variability. 

Conflict is increasing

Drought is increasing across the U.S. States are continuing or starting new fights over scarce water resources. For example, Texas is suing New Mexico as it deals with 75 percent of the state in drought conditions. Also, the Colorado river system is experiencing record water stress, teeing up future water conflict amongst the compact states vying for their share of a dwindling pie.

Conflict is an economic stressor on its own, and the battles over life-sustaining resources are sure to increase the negative impacts of climate change on the agricultural economies of the world. 

Lack of investment in infrastructure

The state of infrastructure for water and power in the U.S. west needs an upgrade. Declining infrastructure in California’s Central Valley is one example of the lifeline of an agricultural system that is decades overdue for upgrades. Politics had had little to offer for help in this arena.

One of the missing ingredients for water infrastructure is Capital. This is due in part from, “The relative uncertainty and duration of returns on investment (namely, water supply benefits) associated with private investments by water users in water conservation and efficiency infrastructure can further suppress capital flow.” See Groundwater Trading as a Tool for Implementing California’s Sustainable Groundwater Management Act, Environmental Defense Fund.

Due to the lack of information, analysis, and true market systems, the lack of investment will continue unless something creative and bold is done about it. 

Tons of data, difficulty in making it actionable

Having lots of data that is unstructured and disorganized makes taking action a challenge. Additionally, the data you get may not be granular enough to make decisions for your locality. Sometimes small data is king.Getting that data is a challenge and something that needs massive investment from the finance and tech communities. There is hope that this is a solvable problem. It is the political side of the solution that lags.

Regulatory friction

Innovation is ready to build resiliency into the modern agricultural economy, but regulatory friction is still present:

"Although California has a water market, it is bogged down by patchwork regulations that discourage transfers and routinely benefit only well-capitalized users. As a result, water users with fewer resources, such as small farmers, poor communities, and the environment, have suffered disproportionately during the drought. And even well-capitalized users have been hindered by the system’s complexity." - Christina Babbitt, Environmental Defense Fund.

The barrier to introducing innovation to water management will plague the agricultural economy. I wrote about some of the challenges and benefits of innovations like groundwater markets previously. 

The 21st Century solutions for the agricultural economy

Despite the numerous hurdles, there is hope and momentum stemming from the magnitude of the climate change calamity. The movement towards transparency in data, advancement in machine learning and artificial intelligence is increasing compute power and lowering costs, and the recurring impacts of climate change has invigorated some elements of government to reach out to NGOs and the private sector to secure a future for agriculture. Is this the tipping point towards a solution?

Open data

California is leading in the effort to take massive amounts of public data and provide that in one spot for the public to use. The portal is far from perfect for actionable data, but it’s a healthy start. For water data, the California Water Data Consortium seeks to unify government agencies and their data. The Environmental Defense Fund is a key player in this effort and has set out goals for water data moving forward that include better information, funding for sensors, and the push for implementation of Open ET.

OpenET provides the possibility of a standardized key dataset for understanding water demand in agricultural operations available to anyone. This is a glimpse into the future value of open data projects and how they assist policy makers, tech companies, agricultural businesses, and the communities interconnected with the agricultural economy. 

Software in the ag space

There are many companies vying for solutions to agriculture economy. However, focusing on the real, core problems that agricultural stakeholder are facing will differentiate successful and struggling companies in the agricultural software space.

Some of the challenges for companies building digital solutions in agriculture is the long sales cycles with agricultural finance community, building trust with the farmers, and keeping costs reasonable for the farmers so that they can feel good about their cost to benefit analysis. 

Without financial help in the form of incentives to mitigate climate risk from deep pockets, it will continue to be a challenge to scale innovation in the space. 

Working together

An ecosystem of tech, government, finance, and NGOs is beginning to form in California. The typical lack of capital flowing into the climate change problem space is beginning to change as pressure from investors, constituents, and markets are pressing politicians to do something about it. 

Capital from the government has traditionally come slowly and with more strings than Pinocchio would care to deal with. However, NGOs and some government agencies are expert at handling that. Where these string ninjas come up short is rapid experimentation, innovation, and growth. Enter the scrappy startups, possibly backed by venture capitalists or arms of big tech. 

The pairing of string ninjas and scrappy startups is one of the most interesting innovations of the 21st century and sets the scene for some real change. 

Why now is the time to innovate faster

With a climate crisis as our backdrop, the string ninjas, and scrappy startups the players, the question is why is now the time to innovate faster? Cost reductions, compute power increases, and political momentum.

Satellite data is getting cheaper

One of the most exciting developments in technology for land and water nerds like myself has been satellites. Since the push for cheaper and faster iterations for space thanks to Elon Musk, the ability to utilize satellite data for scrappy startups is getting easier. For example, there are now companies like Albedo that provide a per task payment model similar to calling an API or having your own satellite to gather the data your need! 

As companies make gathering satellite data cheaper, the cleaning and translation of that data will also become a lucrative business. There is room for innovation in turning satellite data into actionable data for use by water managers and agricultural operations to determine groundwater levels, soil, and plant moisture, and turn that into crop yield predictions, water efficiency studies, and risk analytics. Models for a state-wide or country-wide scale will need help from the string ninjas and governments to properly scale and imbed in decision making. But how will companies that translate data for use by industries and individuals manage the amount and complexity of that data?

Machine learning is cheap, in the cloud, and easily deployed

The ability to launch a machine learning first company is no longer a cost prohibitive endeavor. In his book The AI First Company, Ash Fontana lays out the steps:

“The steps to building a data learning effect with intelligent machines are (1) capturing a critical mass of data, (2) developing capabilities to process that data into information, and (3) feeding that information into a computer that runs calculations over data to learn something new.”

Those steps used to be solely in the realm of IBM and the Pentagon. More recently big tech, like Amazon and Google, moved the machine learning and artificial intelligence tools to the masses via cloud infrastructure. Now the scrappy startups can run thousands of experiments, feeding cleaned data into the cloud computer infrastructure, iteration, and a lightning pace. 

With all this in the cloud, companies can pull talent from across the world, collaborate with government and academic institutions anywhere, and do so for a fraction of the cost. The pace is fast, but it still may be stalled by the laggards in power. 

The political atmosphere is ripe for change

Critical to continued innovation in the climate space is compatible regulation. I talked about the need for public and private partnerships in a previous article as it relates to groundwater markets in California. More broadly, reliable, accessible, and granular data can be provided as a Public good. 

There should be a public database that publishes climate data that is viewable using a GIS based map. Additionally, there should be open and free access to APIs that allow software companies to use the data in experimental, iterative ways such as machine learning and artificial intelligence applications. 

Unfortunately, we are bogged down by solid data, licensing issues, and a lack of a unified way to view and analyze data that leads to actionable steps for a variety of industries, including agricultural business. 

An open data portal that respects individual privacy but makes the bulk of data available and cheaply consumable will align with the innovation path set by the scrappy startups and big tech. The string ninjas and local agricultural communities can use government sponsored forums to provide valuable input into the innovation flywheel. We are all the users in this ecosystem challenged by scarcity and pollution, and we need channels and political representatives that get technology and science enough to support innovation.

This will take education from all stakeholders to get government to understand the “How” in tackling climate change. We get that there is a climate crisis, it’s time to clear the hurdles to solving it.