Classification of hazelnut cultivars: comparison of DL4J and ensemble learning algorithms

Classification of hazelnuts is one of the values adding processes that increase the marketability and profitability of its production. While traditional classification methods are used commonly, machine learning and deep learning can be implemented to enhance the hazelnut classification processes. This paper presents the results of a comparative study of machine learning frameworks to classify hazelnut (Corylus avellana L.) cultivars (‘Sivri’, ‘Kara’, ‘Tombul’) using DL4J and ensemble learning algorithms. For each cultivar, 50 samples were used for evaluations. Maximum length, width, compression strength, and weight of hazelnuts were measured using a caliper and a force transducer. Gradient boosting machine (Boosting), random forest (Bagging), and DL4J feedforward (Deep Learning) algorithms were applied in traditional machine learning algorithms. The data set was partitioned into a 10-fold-cross validation method. The classifier performance criteria of accuracy (%), error percentage (%), F-Measure, Cohen’s Kappa, recall, precision, true positive (TP), false positive (FP), true negative (TN), false negative (FN) values are provided in the results section. The results showed classification accuracies of 94% for Gradient Boosting, 100% for Random Forest, and 94% for DL4J Feedforward algorithms.


Introduction Introduction Introduction Introduction
Hazelnut (Corylus avellana L.) is an important agricultural crop produced by a small number of countries in the world. Shelled hazelnut production in the world in 2017 was 1,006,178 tonnes. Approximately 67% (675,000 tonnes) of this production was from Turkey, and 14% (132,281 tonnes) was from Italy. The remaining 18% share was from Azerbaijan, USA, China, Georgia, Chile, Iran, France, and Spain (Food and Agriculture Organization of the United Nations, 2019). Turkey, Georgia, Italy, and Azerbaijan are the leading countries in hazelnut trade (Ercisli et al., 2011). Hazelnut is mostly produced in Ordu, Giresun, Samsun, Sakarya, Düzce, Trabzon, and Zonguldak provinces of Turkey (Bostan and Güler, 2016).

AcademicPres Notulae Botanicae Horti Cluj-Napoca Agrobotanici
Knowing the physical properties of agricultural products is important for the design and development of machinery, equipment, and systems for various harvesting and post-harvest processes (Sayıncı et al., 2015).
Shape, geometric dimensions, mass, density, hardness, static and sliding coefficient, coefficient of friction, compression strength are the major technical properties of biological materials (Mohsenin, 1980). Among these properties, the shape is a crucial factor for the classification, grading, and packaging of agricultural products (Sayıncı et al., 2015). Agricultural products are cleaned from foreign substances and classified based on their types, size, and quality. Biological materials are classified based on their physical, chemical, and biological properties. Mechanical, thermal, optical, and electrical properties of materials are used in the separation process (Mohsenin, 1980). The emergence of machine learning, big data technologies, and high-performance computing have enabled new opportunities in multidisciplinary agricultural studies (Liakos et al., 2018). Artificial intelligence and computer vision applications have been increased in recent years. As in every field, applications of these technologies in agriculture have also increased. Non-time-consuming, fast, non-destructive classification models are used for the post-harvest classification of agricultural products. Machine learning is widely used in solving classification problems. For example, Kavdir and Guyen (2004) used fuzzy logic for apple grading. Classification results obtained by the fuzzy logic expert system was 89%. Pronprasit and Natwichai (2013) predicted of mango fruit quality by using NIR spectroscopy and ensemble classification. with 250 images. They used principal components analysis (PCA) in addition to support vector machines (SVMs) and linear discriminant analysis (LDA) algorithms for feature extraction and classification. Their results showed 85% accuracy using SVMs algorithm with a linear kernel function and 84% accuracy using LDA algorithm for ripeness classification. Results were found to be 84% successful for LDA and 77% successful for ANN. Brahimi et al. (2017) tried to predict the diseases with convolutional neural network (CNN) by using 14828 tomato leaf images, which caught nine different diseases. According to the results, the accuracy of the model was 99%. Manickavasagan et al. (2018) classified palm varieties grown in various regions of Saudi Arabia based on their hardness. Instead of the traditional expensive and time-consuming methods used to determine the quality of dried fruits, they developed an artificial neural network and linear discriminant analysis methods by subtracting histogram and tissue properties from monochrome images of 1800 samples to classify dried fruits as soft, semi-hard, and hard. Ishikawa et al. (2018) classified strawberries in their studies by taking advantage of the shape information obtained from digital images. Using SHAPE software (The University of Tokyo 1-1-1 Yayoi, Bunkyo, Tokyo 113-8657, JAPAN), they obtained fruit length, width, projection area, and fruit boundary from a total of 2969 images. The machine learning method was successful in diagnosing strawberry fruits with nine different shapes. Vaishnav and Rao (2018) classified 49 (including apple, mango, cherry, orange, and pineapple) fruit images that are obtained from Google. Logistic regression, neural network, Knearest neighbour, decision tree, random forest, Naive Bayes were used for the classification process, and the data set was run ten-fold cross-validation method. Logistic regression predicted 85% for the first set of experiments and 91% accuracy for the second set of experiments. Saranya et al. (2019) used machine learning and deep learning algorithms for apple, banana, orange, and pomegranate fruit to classify. R, G, B colour, size, height, and width of fruit were used as physical properties in the study. The comparative results of study were 49 for K-nearest neighbour, 61 for support vector machine.
On the other hand, classification system accuracy was 97% for convolutional neural network. Shin et al. (2019) used three image processing techniques and two supervised machine learning techniques for early detection of powdery mildew disease. ANN determined the highest accuracy, with a 90% success rate. Datta et al. (2019) used image processing and Faster R-CNN to detect cracked eggs. The performance of the damage detection in the test data was evaluated using the mean sensitivity (MAP) value. The network yielded a MAP value higher than 75% and was able to detect most of the damaged side. Caladcad et al. (2020) classified Philippine coconuts according to maturity levels using artificial neural network, random forest, and support vector machine. A total of 132,300 data were partitioned 10-fold-cross validation method to estimate maturity levels (pre-mature, mature, and over-mature). The study results that the total accuracy of system was 80%. Also, random forest model predicted 91% and 84% accuracies for training and testing, respectively. Miraei Ashtiani et al. (2020) estimated mass of almond kernels using from its shell features such as length, maximum width, and maximum thickness. For this purpose, a multi-linear regression model, multilayer perceptron neural network, radial basis function neural network, adaptive neuro-fuzzy inference system, and support vector machine algorithms were used. The high overall accuracy was 96% for the mass of almond kernels using the radial basis function neural network classifier. A similar study was conducted by Vidyarthi et al. (2020). In the study, size and mass of 100 pistachio kernels were predicted using random forest algorithm. The measured length of 18.002 mm for all pistachio kernels used in the study was approximated as 18.608 mm using the random forest classifier. The objective of this research was to implement a comparative study of machine learning frameworks to classify native hazelnut (Corylus avellana L.) varieties using DL4J and ensemble learning algorithms.

Materials and Methods Materials and Methods Materials and Methods Materials and Methods
Load measurement set-up An inductive force transducer and NI 6009 (National Instruments, USA) data acquisition board were used for load measurements. A testing unit designed for this study was used to compress the hazelnut to measure the forces required to break hazelnuts shells. A 10 mm diameter hydraulic cylinder was mounted at the center of the testing unit for the ease of connection of the inductive force measuring device with the testing unit ( Figure 1). The piston rod moved at a constant speed of 1.5 m.s -1 to compress the samples while breaking the hazelnut shells. The breaking force of each sample was determined from the force-displacement curve recorded on the computer during compression.

Data acquisition system
A graphical program developed in LABVIEW 8.2 (National Instruments Corporation, 11500 N Mopac Expwy Austin, TX 78759-3504, USA) was used for the data acquisition. The developed program facilitates the collection and recording of the inductive load data with the NI 6009 data acquisition board to a personal computer via USB cable. Force was applied until the nutshells were cracked. The force applied to the shell was recorded on the computer in a text format every 5 ms. The developed graphical program also displayed the measurements on the computer screen during the experiments (Figure 2). For sensor calibration, constant weights ranging from 1 kg to 10 kg were used. The analog force value read from the sensor was transmitted to the NI 6009 data acquisition board. The analog data was then converted to digital data with the calibration equation, which was obtained in Excel in Figure 3.        Deep learning is a subset of machine learning that uses multi-layered artificial neural networks to provide state-of-the-art accuracy in tasks such as artificial intelligence, object identification, speech recognition, and language translation. Deep learning is used in areas such as automated driving, aerospace and defense, medical research, industrial automation, electronics, and agriculture. Deep learning frameworks include Theano with Keras warper, Torch, Caffe, Tensorflow, and Deeplearning4J (DL4J) (Kovalev et al., 2016). In this study, DL4J Feedforward algorithms were selected for data classification as deep learning frameworks. The main purpose of DL4J feedforward is mostly used for data classification. The algorithm was chosen for high-performance instead of traditional machine learning algorithms.
The first step in building network architecture is to create a network. Generally, neural networks are structured in layers, so the type and number of layers must determine to create a particular network architecture. In the KNIME program, each different layer is represented as its own node. Layer nodes that are intended to be used sequentially to form a network are linked together. The first node in each architecture will always work with a DL4J model initializer that will create an empty deep learning model. This model is used for storing the network and its configuration. Each layer node can have different parameters that can be configured in a particular layer node dialog box. Since this layer will be initiated by a learning node, it is not necessary to add an output layer to an architecture. When using KNIMA, the DL4J Model Initializer is selected and then dense layer is added and the necessary settings were made. Because of this reason, DL4J feedforward architecture, activation function as ''ReLU'' and weight initialization strategy as ''XAVIER'' were selected.

Ensemble learning
Ensemble methods aim to improve the performance of a particular statistical learning or model technique (Al-Barazanchi et al., 2017). The general principle of community methods is to create a combination of some methods rather than using a single fit of the method (Bühlmann, 2012). A community (ensemble) is a collection of estimators (average of all estimates) that come together to give a final estimate. The reason we use the ensemble is that many different determinants used to predict the same target variable will do a better job than any single predictor alone. Ensemble techniques are classified as Boosting, Bagging (Random Forest), AdaBoost, and Stacking (blending, MAVL) (Seker and Erdogan, 2018).

Bagging
Bagging is a method that is generating multiple versions of a predictor and using these to get an aggregated predictor (Aoki et al., 2015). Also, Bagging is a simple collection technique in which we build many independent determinants/models/learners and combine them using some average model techniques. Bagging that often considers homogeneous weak learners learns them independently from each other in parallel and combines them following deterministic averaging process (Shi et al., 2018). Each observation is selected by modifying it to be used as input for each of the models. Thus, each model will have different observations based on the bootstrap process. This technique reduces error by reducing variance, as it takes many unrelated data to generate a final model. An example of a bagging community is a random forest model. In this study, the random forest algorithm was selected as the bagging method. The random forest algorithm developed by Leo Bieman generates multiple trees to solve a problem and creates different decision trees. Random forest algorithm can be used in both classification and regression problems. Random forest is an advanced version of the classification and regression trees (CART) algorithm, in which many trees are created based on subsets of data. At each node, branches are formed according to the criteria of the CART algorithm (GINI index). The GINI index measures class homogeneity and can be expressed by the following equation (Eq. 1, 2, and 3) (Salzberg, 1994).
, T) shows the probability that the selected sample belongs to the ''Ci Ci Ci Ci class'' (Akar and Güngör, 2012).

Gradient boosting (Boosting method)
Boosting is a very successful technique for solving the two-class classification problems. Boosting is a community in which predictions are made sequentially, not independently. This technique uses the logic that subsequent estimators learn from the mistakes of previous estimators. Therefore, the likelihood of observations in subsequent models is uneven, and those with the highest errors appear most. The estimators can be selected from a range of models such as decision trees, regressors, and classifiers as the new estimators learn from the mistakes of the previous estimators. Gradient boost is an example of the acceleration algorithm. In this study, the gradient boosting algorithm was selected as a boosting method. The gradient boosting can be expressed by the following equation (Eq. 4, 5, and 6) (Natekin and Knoll, 2013). Initialize f /0 = 0, k = 1,2, … K For m = 1 to M:

Data analysis
Before starting the data analysis, the ''statistic'' node was connected to the Knime workflow to find the missing values, and then the data set was partitioned 10-fold-cross validation method to prevent overfitting. The values in the dataset were labeled in Excel file as follows (Table 1). DL4J Feedforward, random forest, and gradient boosting machine were connected to workflow, respectively, in Figure 6.  Figure 6. KNIME workflow At the end of analysis, accuracy, Cohen's Kappa, TP, FP, TN, FN, P, R, and F-means values were given in the relevant part of the study. The "scatter plot'' node was connected to provide a more visual representation of the results.
One of the classifier performances, accuracy, shows how accurate the system can predict. Recall or sensitivity or true positive rate (TPR) is the value that the system is predicting what rate is the passing assessment of the overall passing assessments. Precision or predicted position value (PPV) is the value that indicates how true or correct the system can predict. Also, F-measure is the overall efficiency assessment derived from the means of precision and recall (Takran et al., 2017).

Results and Discussions and Discussions and Discussions and Discussions
The classifier performances, which are accuracy, error, F-measure, Cohen's Kappa, recall, precision, TP, FP, TN, FN values were given in Tables 1-2. As a result of the comparison, it was found that the Random Forest (Bagging method), was better than the other algorithm. When the shell compression strength was considered only, boosting method was found to be successful with 72%, Bagging method (Random forest) with 83%, and DL4J (Deep learning algorithm) with 71% accuracy. In cases where the compression strength value is higher than 431.212 in the Z-axis, the 'Sivri' hazelnut was classified as 100% also, in cases where the compression strength value is less than 103.22 'Tombul' hazelnut was classified as 100%. When hazelnut weights were added in addition to shell compression strength values, GBM method was found to be successful with 85%, Random Forest method with 87%, and DL4J with 73% accuracy. It was found that there is a parameter that increases the success rate in weight classification. On the other hand, in addition to the shell compression strength values and weight of hazelnut shell thickness, a Gradient Boosting method was found to be successful with 95%, Random Forest method with 100%, and DL4J with 95% accuracy. Comparison of deep learning and ensemble learning algorithm values are given in Figure 7.  Menesatti et al. (2008) evaluated the potential use and effectiveness of shaped techniques to distinguish four traditional Italian cultivars. Approximately 400 hazelnut color RGB images were analyzed by a morphological method based on the elliptic Fourier approach to closed contours in a two-dimensional plane. The coefficients of the harmonic equations were analyzed by partial least square discrimination analysis (PLS-DA). The results showed that the percentage of correct classification for a lateral view is high (77% to 99% in the independent test). The hazelnut samples selected in the study have success rates varying between 94% and 100% according to the classification parameters considered. A higher success rate was obtained than the classification made based on the RGB image processing methods reported in the literature. At the same time, good positioning results (95 and 97 respectively) were found for random positioning appearance, especially for in-shell kernel between two round varieties and between two long varieties. Guvenc et al. (2015) studied on the classification of partly shell removed hazelnut kernel, shell removed, and rotten hazelnuts kernels by using a computer vision approach. In the proposed method, the processed hazelnut kernels were classified with 94% accuracy. This ratio is parallel to the classification results in the study. Giraudo et al. (2018) developed an automated identification of defective hazelnut kernels based on multivariate analysis of RGB images through the principal component analysis and subsequently used to build classification models. Their system success was approximately found as 97% of the test samples. The 97% success rate obtained in this study was a little more successful than the gradient boosting machine (94%) and DL4J Feedforward (9%) algorithms presented in the literature. However, it was lower than the random forest (100%) algorithm. Solak and Altınışık (2018) worked on the classification of hazelnut cultivars by using image processing and mean-based, k-means clustering technique. Hazelnut fruits were classified with 100% accuracy using image processing techniques. A comparison of mean-based and K-means clustering methods was found that 90% to 100% accuracy. The classification results in this study are also close to the results of Solak and Altinisik (2018). Jalali and Mahmoudi (2013) worked on pistachio nut cultivars sorting by data mining and fuzzy logic classifier. A correct classification rate of 93% for training and 97% for the testing set was observed. Confusion matrix was selected as the model performance criteria of algorithms for compare to success rate. The performance evaluations and accuracy criteria of the algorithm were shown in Tables 1 and 2.

Conclusions Conclusions Conclusions Conclusions
There are many studies on the post-harvest classification of various agricultural products. Researchers have focused on techniques and methods for developing prototypes capable of non-destructive classification. Among these methods, artificial intelligence and machine learning stand out as current issues. These methods are the primary issues used in developing solutions for the classification and automation of agricultural products. In this study, three different hazelnut cultivars ('Sivri', 'Kara', 'Tombul') were classified based on five attributes of hazelnuts using the most common classification and prediction algorithms. The best prediction was found by the Bagging method. The results showed 100 % accuracy for Random Forest, 95% accuracy for gradient boosting machine, and 95% accuracy for the DL4J Feedforward algorithm, respectively. With the integration of classification machines, machine learning has a new future for automated decision-making or support to provide practical tools for knowledge-based agricultural practices. The results of this research can be used in future studies by the researchers for classification issues.
Vidyarthi SK, Tiwari R, Singh SK, Xiao H (2020). Prediction of size and mass of pistachio kernels using random Forest machine learning. Journal of Food Process Engineering 43(9):e13473. https://doi.org/10.1111/jfpe.13473 The journal offers free, immediate, and unrestricted access to peer-reviewed research and scholarly work. Users are allowed to read, download, copy, distribute, print, search, or link to the full texts of the articles, or use them for any other lawful purpose, without asking prior permission from the publisher or the author.