Keywords

1 Introduction

With the recent developments of AI (Artificial Intelligence) technology have caused various changes in various industrial fields [13, 17]. Also, the design industry is also being developed and expanded through various changes [10], and the role of design in the industrial society is increasingly recognized as an important means. Since the design industry is an industry on which human resources are based, one’s personal competence is more important than other industries, and the need for competent designers is increasing. Therefore, the completeness of the portfolio, which serves as an indicator for evaluating the personal competence of the designer and as an individual career certificate, is becoming more important. As such, we are aware of the importance of the portfolio, but the evaluation criteria depend on the subjective opinion of the evaluator. Therefore, not only the designer but also the evaluator is confused about producing and evaluating the portfolio.

The purpose of this study is to find out the objective index for evaluating the personal competence of the designer as a preliminary study for the AI - based service development that objectively and quantitatively evaluates the portfolio, identifies priorities of components that affect portfolio completeness assessment and provide evaluation guidelines. In order to accomplish the purpose of this study, we have identified the needs of designers and portfolio evaluators who produce portfolios through in-depth interviews and in-depth interviews. Based on this, Python was used to extract and analyze the 820 component data of the portfolio, and the priorities of components to be considered in the portfolio identification were derived. This study is meaningful in that a new direction using data was devised and presented for objectification and quantification of the portfolio evaluation method. As a result of this study, it is expected that designers will provide a guideline to express their competence more effectively in the portfolio and more objective and standardized evaluation criteria to the evaluator. In addition, if subsequent research based on this research is continued, it is expected that AI services for objective evaluation can be constructed in the evaluation of the completeness of the portfolio.

2 Literature Review

2.1 Portfolio and Evaluation Methods

Design is a specialized area for presenting solutions to problems and acting on those tasks. Therefore, compared to other direct groups, designers are each valued for their individual competencies. because the portfolio is the most important material for assessing the competency of individual designers [6]. A portfolio is a communication tool that shows the designer’s personal competence and communicates with others as a mirror showing the designer’s thoughts and problem-solving methods, rather than simply a completed collection of works. (Kim 2008) was carefully analyzed according to the characteristics of the structural elements of the interior design portfolio and arranged the elements of the portfolio to focus on the way the presentation of the portfolio. (Han [6]) was presented in six categories of guidelines for the objective competency evaluation of designers through portfolios through expert in-depth interviews, looking at portfolios from the perspective of designer recruiters. This study is significant that this is a study that objectively presents the evaluation criteria of the designer through the portfolio. However, the limitations of the study mentioned by the authors of this paper are as follows. 1. Participants in the in-depth interview for the elicitation of needs were limited. All the interviewee were evaluators. 2. Since the evaluation method of the portfolio has already been evaluated as subjective and abstract language, it is difficult to objectify the evaluation method. The guidelines are shown in Table 1.

Table 1. Basic elements of portfolio evaluation guidelines (Han [6])
Full size table

2.2 Changes in the Industrial Sector Due to the Emergence of AI

In addition to recent advances in AI technology, many changes have been taking place throughout the industrial society of Automobiles, robotics, education, etc. In the design area, you can find many examples of using AI. Companies such as Adobe, Microsoft, and Autodesk that produce 3D Tools and 2D Tools that are used primarily by designers provide artificial intelligence-based ‘intelligent services’ to provide designers with the convenience of using tools. Services such as Google’s Quick Draw is to convert complete rough sketches to detailed graphics [12]. And such as ‘tailorbrands’, ‘hatchful’ and ‘Logoshi’ are automatically to create logos. The application of artificial intelligence in the design area has been applied to a variety of areas, from design tools utilized by professional designers to services that make it easier for non-professionals to access the design.

3 Preliminary Survey

A questionnaire survey was conducted on 117 designers. The purpose of this questionnaire is as follow. (1) to understand how designers perceive portfolios. (2) To find out the needs for an objective portfolio evaluation method (Table 2).

Table 2. Items in questionnaire
Full size table

A total of 117 people participated in the survey, and 2018.09.26–2019.09.30. It took 5 days in total. Data collected from questionnaires were analyzed using the SPSS program, and frequency analysis was conducted. The general characteristics of the questionnaire are shown in Table 3.

Table 3. General characteristics of questionnaire
Full size table

Design skills and design expression (65.8%), design thinking (60.7%), and planning ability (57.3%) were the most frequent factors in the frequency analysis. Design Tool skill & design expression was remarkably high. The creativity of designers was ranked 5th with 31.6% (Table 4).

Table 4. Designer’s essential competency (including duplicate votes)
Full size table

Portfolios, academic backgrounds, and self-introduction were the first, second, and third most important factors in evaluating designer competencies. Among the factors that assess the competence of designers, the portfolio was significantly higher at 98.3%. As a result, it can be seen that designers are aware of ‘portfolio’ as a tool for expressing their competence. Table 4 also shows that the ‘portfolio’ is effective in evaluating ‘Design Tool & Design Expression’, which is an essential capability for designers.

Table 5. Important factors in evaluating a designer’s competency (including duplicate votes)
Full size table

In order to achieve the purpose of this first study, we conducted post interviews on the services needed to develop the competency of the designer. This is because interviewing methods are more effective in identifying the needs of designers. The interview results were coded and the needs were typed as shown in Table 6. The frequency analysis showed that ‘Education service (40%)’, ‘Portfolio service (24%)’ and ‘Community service (20%)’ were ranked first, second and third respectively. Designers want to grow. Thus the need for education services so high. The need for portfolio services was also high. Portfolio service needs were detailed in the order of ‘Production Guide’, ‘Attachment’, and ‘Objective Evaluation’.

Table 6. Services needed to develop a designer competency
Full size table

This study confirms that the importance of portfolio and objectification of the evaluation method are needed in the industrial structure where a designer’s personal competence becomes increasingly important through previous research and pilot questionnaire. Confirms the priority of components affecting the evaluation of portfolio completeness, suggest evaluation guidelines.

4 Method

4.1 RandomForest

Random forest used in this study is to generate several decision trees from data and then to predict the most selected class among the predicted values of each individual tree [1, 16]. In other words, it is an ensemble technique that improves learning performance by applying several simultaneously. The ensemble means harmony or unification in French, and it learns several models in machine learning and predicts better values than one model using the predictions of the models. This method is called an ensemble learning or an ensemble method [14, 16]. Random Forest works by constructing more than 30 datasets from the same data, extracting them from the same data, applying decision trees to each, and collecting the learning results [5]. However, each tree only learns a few of the variables, which is an attempt to increase diversity by making the individual trees view the data differently [14]. Random forest is a way to eliminate the overfitting problem of the training data of existing decision trees. To realize this, it is necessary to make many decision trees [15, 16]. Each tree should be well predicted and distinguished from other trees [14, 16]. Random forests are generated randomly when a tree is created so that the trees are different from each other. There are two ways of randomly choosing the data to be used when creating the tree, and randomly selecting the characteristics in the partitioning test [14, 16].

4.2 Research Procedure

Design portfolio is a critical factor in evaluating the capability of a designer. However, there is no standard way to evaluate a design portfolio. Additionally, there has been no academic study to give a guideline for evaluating a design portfolio. We had an in-depth interview with designers in the field to collect possible criteria for design portfolio evaluation. According to the criteria, we organized the elements that good design portfolios have and made a data-driven classifier for identifying good design portfolios. Eventually, we came up with a guideline for evaluating a design portfolio based on our study. This guideline suggests the priority of the elements that organize good design portfolios.

Step1

Aleading study of portfolio components for portfolio improvement and for the development of business designers’ needs (Kim 2008) and the prior art of the evaluation of the Portfolio Assessment guidelines (Han 2011) was, based on this, the required capabilities of the designer were typed into five. Based on the structure of the Korean Design Industry (Korea Industrial Design Statistics Report, Kidp 2017), Survey and In-depth interviews have conducted with 15 designers who were in the office of start-up, Design Agency, and in-house. As a result, it was confirmed that the main evaluation criteria for evaluating the portfolio are different for each group (Table 7). In this study, tried to show between the Designers competency defined by ‘basic skills’ and portfolio elements are related. Table 8 shows the types of designer competencies evaluated through the portfolio through the interview coding process (Table 8).

Table 7. Evaluation criteria of a portfolio by design industry group
Full size table
Table 8. Basic elements of portfolio
Full size table

Step2. Portfolio Gathering

Behance (behance.net) is an online website that can display and search works created by designers from around the world with Adobe Creative Cloud [2]. Pinterest (pinterest.com) is a social network service that users post and share images with other users. In this study, Using Python, the 900 Portfolios, which is open online, such as ‘Behance’ (behance.net) and ‘ Pinterest’ (pinterest.com), has been Crawled [2, 11]. Among them, the 820 Portfolios were used in the analysis for convenience in the analysis, the 80 Portfolios which are non-portfolio images and dynamic images (such as a GIF) were excepted. The components of the portfolio are largely divided into images and texts, and each itemized component of the data extracted from the 820 portfolios is shown in Table 9.

Table 9. The data element of portfolio
Full size table

Step3. Data Preprocessing

The 820 portfolios are to be used for analysis, 40 were divided as Good and 780 were divided as Normal. Standard of between good and normal has been divided as of whether or not having a ‘Behance recommendation label’ for the convenience of analysis, the extracted data is encoded with matched with numbers and then, the learning model could process them. The N-gif and N-tool were not encoded because their values are already represented by integers and the values range from 0 to 5. On the other hand, main-color and key-color are encoded the extracted RGB values based on the color tone. A total of five color tones were used as the standard, monotone as 1, pastel tone as 2, vivid tone as 3, deep tone as 4, and natural tone as 5. The value of N-Text is encoded by 200 units, with 0–199 as 1, 200–399 as 2, 400–599 as 3, 600–799 as 4, and 800 or more as 5 (Table 11). We select the portfolio completeness evaluation as the ‘target variable’ and the six attributes selected in Step 3 as the ‘predictor variables’. The variables used for the prediction of the portfolio completeness among the total variables were selected through the preliminary study and the in-depth interview. A total of six variables were selected, one for design skill, one for comprehension and analysis, and one for originality. Selected variables were extracted from 820 portfolios and created a dataset (Table 10).

Table 10. Dataset
Full size table

Step4. Data Partition

In order to make an ideal model using data mining, it is desirable to create multiple prediction models from one data set and compare and analyze them [7]. Therefore, we divide the whole data into analytical data and verification data, create a model with analytical data, and apply the verification data to the model to compare the performance of the model. In this study, was divided by 7:3. Since random forests arbitrarily select variables to construct a tree, the ratio of analysis data to verification data does not affect the analysis [16].

Step5. Data Training and Evaluation

Random forests can be used to rank the importance of each variable in a classification or regression problem [1, 16]. Feature_importances_ was used to calculate the importance value of predictive variables. The data used in the analysis are unbalanced in the target variable. In the case of unbalanced data in RandomForest, there is a way to compensate overfitting by assigning a weight to a small number of variables [4]. However, in this case, there is a drawback that it is more susceptible to noise, and analysis by a weighted method does not affect the performance of the prediction model [4]. To build a RandomForest model, you must define the number of trees to be created. You can set the number of trees by the n_estimator value of RandomForestClassifier. In this analysis, n_estimator = 500, random_state = 42 was set, and a training model was created using scikit-learn. In order to reduce data noise and reduce the probability of overfitting, we normalized the normalized score to 0 with a mean and a standard deviation of 1. Principal Component Analysis (PCA) was used to build the model.

5 Result

5.1 Feature Importance

RandomForest was used to derive important feature importance for class discrimination and feature priority. As a result, N-Tool, N-gif, and N-Text were ranked first, second and third. This is a significant result because it reflects the latest trend in which the portfolio using dynamic images such as gif and video has appeared. Also, according to a result of in interviews has conducted with 15 designers, there were many opinions that text is important in portfolio evaluation. In other words, the portfolio is recognized as a means of communicating between the designer and the evaluator. so Text is considered more important than Image in the portfolio because the text implies that the value of the designer and the process of solving the idea. Although this study identifies only the number of simple texts, there are limitations, but it is meaningful in that it is in line with the main elements derived through in-depth interviews. The feature priorities for class discrimination are summarized in Table 11 below.

Table 11. Importance of features.
Full size table

5.2 Accuracy Score

Accuracy Score to create an ideal model, data is divided the data set into two parts: ‘training data’ and ‘test data’ at a ratio of 3:7 and analysis using RandomForest. As a result, as shown in Tables 12 and 13, the training score is 0.9913 and the test score is 0.959933 (Tables 12 and 13).

Table 12. Training score
Full size table
Table 13. Test score
Full size table

6 Conclusion and Suggestion

Portfolios are a fundamental document for assessing the competence of designers, and the importance of the portfolio is rapidly increasing. However, related research is insufficient. In addition, existing research is only a conceptual guideline based on the opinion of the evaluator. Therefore, this study analyzed the literature related to the portfolio evaluation, analyzed and typified the portfolio components through in-depth interview with the working designer, and presented a comprehensive direction considering the specificity of the Korean design industry structure. Based on the results of this study, it is expected that designers will use the dynamic image to create a portfolio that reflects the latest trends in the portfolio and will help improve the expression and evaluation of individuals when considering the weighting of the text in portfolio formulas.

However, this study has the following limitations. the number of portfolio samples used in this study is 820, which is not enough. In the process of creating a dataset as a property of the portfolio, class classification criterion is ambiguous, it is classified as Good & Normal. When evaluated the portfolio through in-depth interviews, confirmed that the main competence such as designer ‘s understanding and analytical ability was evaluated through the context of the text. but, in this study, attribute describing the text context in the portfolio could not be found.