Ear counting using image processing systems

For ear density, we have developed ear counting systems for ground and aerial platforms using RGB and thermal imagery under low- and high-resolution imaging conditions. These systems were designed for field data acquisition experiences and specific image characteristics. The validation was performed primarily using image-based data.

Ear counting using RGB imagery

We proposed the use RGB images acquired using a simple method holding the camera by hand at around 1 m (ground platform, in Chapter 1 and Chapter 2), and also using a more complex method mounting the camera on a drone at around 25 m a.g.l. (aerial platform, in Chapter 3). In both cases, we could observe similar limitations and requirements due to the visual wavelength characteristics such as shadows and bright surfaces and overlapping ears. Sunlight reflections on leaves could be one of the most important limitation for the automatic ear counting systems proposed. On the one hand, with the ground platform the sunlight reflections are similar to ears into the image, and additionally, the image processing system used local peaks (based mainly on the bright color

confused with each other. On the other hand, from the aerial platform the sunlight reflects off of bending leaves under direct sunlight conditions, and also the low spatial resolution did not allow for correct visual differentiation between ears and leaves using orthomosaic images. Despite these limitations, both the automatic ear counting systems have achieved high accuracy (e.g. R2 _{= 0.75 and R}2 _{= 0.89;}

using ground and aerial platforms, best results respectively). In both platforms we further improved the system; in the first one, we added the training and classifying step (Chapter 3) in order to increase the robustness of the image processing system; and in the second one, we employed a higher resolution RGB camera (and lower flight altitude) in order to increase the number of matching features found from the Structure from Motion (SfM) process used to build the orthomosaic with higher spatial resolution (Aasen et al., 2018).

Ear counting using thermal imagery

We purposed the use of thermal images acquired holding the thermal camera by hand at approximately 1 m (ground platform, in Chapter 4) above the canopy surface. We also observed some limitations using this data such as low spatial resolution, no temperature differences between canopy and ears (depending on the acquisition time) and overlapping ears. Low spatial resolution was identified the most important limitation for the proposed automatic ear counting system. Thermal imagery can avoid shadows and bright surfaces (the main RGB limitation); and moreover, thermal images filter high frequency details intrinsically due to the manner in which this technology detects much longer

leaves which in turn contributes to simplify the image processing tasks. However, its low spatial resolution is still a major issue and this technology sensor cannot yet be used from an aerial platform for this reason. On the other hand; for the development of the thermal ear counting algorithm, visual interpretation of RGB images (RGB images acquired at the same time as the thermal images) was crucial in correctly locating the presence of ears in thermal images. Despite these limitations, the thermal image based automatic ear counting systems have achieved high accuracy (R2 _{= 0.80) (Fernandez-Gallego et al., 2019a). This}

system may furthermore be improved using a thermal and RGB fusion sensor to be able to increase the spatial resolution in order to cover more footprint area at higher distances.

1.1. Phenological stage and data acquisition time for ear density

The ear density trait can be estimated from anthesis to late grain filling using RGB and thermal imagery (Chapter 1, Chapter 2, Chapter 3 and Chapter 4). At late grain filling in the rainfed trial (near to maturity), the ground level algorithm did not perform as well in ear identification (R2 _{= 0.17, GS 91) when automatic}

and manual image-based counting were compared. The late growth stage did not permit consistent ear identification due to the lack of contrast between de leaves and ears. However, from anthesis (support irrigation and rainfed) to late grain filling (support irrigation) the approach had good accuracy. Even under low resolution sceneries the determination coefficient was higher at late grain filing under support irrigation (R2 _{= 0.75, GS 81) than anthesis under support irrigation}

at late grain filling using the aerial level algorithm could not be done, this was mainly due to sunlight/sunglint and resolution issues (Ortega-Terol et al., 2017). However, the determination coefficient was also higher at a later stage in the aerial data in the same way as the ground data - the correlation at grain filing (R2

= 0.89, GS 75) was higher than at the anthesis (R2 _{= 0.83, GS 61) growth stage.}

Using thermal imagery, the determination coefficients performed almost the same, although we compared growth stages between experimental stations instead of use the same site. The determination coefficient was higher at grain filling in Seville (R2 _{= 0.76, GS 69; unpublished results) than anthesis in Aranjuez}

(R2 _{= 0.65, GS 61-65; unpublished results); but the correlation decreased at late}

grain filling in Valladolid (R2 _{= 0.70, GS 77; unpublished results) compared with}

grain filling in Seville (shown above) when automatic and manual image-based counting were compared.

1.2. Ear density trait and grain yield

The ear counting systems in field conditions (including also manual in-situ counting) has shown low correlation with GY when all data plots were used in cross-validated linear regression (LR) models. Using ground scale images (Chapter 1 and Chapter 2), the automatic ear counting system achieved relatively low correlation with GY (R2 _{= 0.30), yet it was higher than the manual in-situ}

counting (R2 _{= 0.24) when compared to GY. In the same way when using aerial}

images (Chapter 3), the automatic ear counting system also achieved low correlation (R2 _{= 0.28) with grain yield, and also for this platform, it was higher}

between the ear density and number of kernels per ear (Slafer et al., 2014; Slafer and Savin, 2007); and (b) hidden ears, due to zenithal images only consider the upper ears which frequently correspond to the main and primary tillers (Ishag and Taha, 1974). On the other hand, when genotyping (G), nitrogen (N) fertilization, and G + N effects were included in cross-validated multiple linear regression (MLR) models; the relationship between ear counting (automatic and manual counting) and GY increased. Best predictions (R2 _{= 0.41-0.46) were achieved}

when G + N effects were included, followed by N (R2 _{= 0.34-0.36) and G (R}2 ₌

0.06-0.20) effects; which suggest that the relationship between ear density and GY is more supported by the N treatment factors than genotypic differences. Additionally, grouping by N treatment and using a LR model, the best relationship (R2 _{= 0.46) was achieved for the lower N treatment, it may observations that, at}

lower N levels, the contribution of secondary and tertiary tillers to GY is usually minor if not negative. In summary, the automatic ear density can explain around 30% (under different nitrogen treatment) (Fernandez-Gallego et al., 2019b, 2018a) and around 50% (under low N conditions) of the variability in yield (Fernandez-Gallego et al., paper under review). This system could be improved using 3-dimensional data to assess the ear size/volume.