Archives of Design Research

Norman et al. (1995) introduced the term of user experience to the field of human-computer interaction (HCI) in the 1990s. They used the word "User Experience" to describe the relationship between the human interface and the usability of the application in the Apple Computer. User experience design explores a more effective innovation by observing a person's emotion using a product, service or system, besides the usability, which benefits the HCI community. Nevertheless, there is no consensus on user experience definition or an empirical standard in implementation, so the wisdom of user experience needs to be debated and reviewed before used in the competency modeling.

The term "user experience" has a wide range of meanings, from usability to beauty or function to emotion in experiencing the use of technology (Hassenzahl & Tractinsky, 2006). The International Standard ISO 9241-210 (2010) defined user experience as "a person's perceptions and responses resulting from the use or anticipated use of a product, system, or service." It also provided details of user experience, including user physical and psychological responses, resulting from design concerning usability, brand image, presentation, and function toward users' personal goals. From this standard definition, a user experience designer needs to be qualified with a large body of knowledge and skills to achieve the design objective.

The goal of user experience is to create a meaningful product, service, or system for users by enhancing their satisfaction and perception of the value of the product, service, or system. To achieve this goal, user experience designers need to understand multiple disciplines, for example, cognitive, psychological, affective aspects and traditional design practice, in designing a user experience, that results in user satisfaction (Zhou et al., 2012; Zarour & Alharbi, 2017).

Characterizing the nature of user experience design into a rigid process is challenging due to its complexity and subjective nature. However, the user experience design always starts with understanding user cognitive goals, expectations, frustrations, emotions and pain points (Pucillo & Cascini, 2014), through cognitive analysis methods, for example interviews, observations, process tracing methods, critical decision methods and conceptual methods (Zhou, Xu, & Jiao, 2011). This can be described as opportunity creation (Moon & Han, 2016), before deploying the classic design steps, for example, idea generation, design development, design refinement, design specification, prototyping and usability testing. User experience design integrates many design and engineering disciplines to create ‘touchpoints’, where users can interact with a business, for users. Some examples of those disciplines are visual design, information design, information architecture, interaction design and usability testing (Halvorson, 2010; Garrett, 2011).

The outputs of user experience design are not only a product, service, or system, but include product management, user training and change management, for implementing the design for the business and bridging the enterprise to the users (Finstad et al., 2009). From these perspectives, user experience design cannot focus only on the end-user, but also needs to gather the design requirements from other user roles, for example business owners, stakeholders, partners and workers, to set design parameters (Goodwin, 2009).

Lately, digital technology has developed rapidly and affected most people, in any context, for example behavior, business, product and service development, marketing, social and culture. Digital technology also introduces many new challenges in design, as the behavior of users keeps changing, due to new demands for use of digital technology. In the era of the digital economy, digital technology remarkably affects user experience design and has brought it into a spotlight. The relationship between digital technology and design has three aspects: digital design and touchpoints, mixed-reality between physical and digital space and data-driven design: these are summarized in Table 1.

Our competency model development started with a context analysis. Context analysis can define requirements and criteria from the current situation for crafting a competency model, constructively aligned with business plans, goals, and needs (Draganidis & Mentzas, 2006). Also, the context includes concerns for ethics and standards in an industry, which are essential to every competency model.

Since user experience design boundaries are still being debated, we surveyed academic articles to optimize our studies, because they allowed us to gather various perspectives broadly and rapidly. In the literature from 1983 to 2019, we found 32 articles, that mentioned the roles and functions of user experience designers, 23 articles on user experience evaluation, and 14 articles on user experience design case studies, that we used as our materials for context analysis to identify the elements of user experience design competency. Also, we added data from ISO 9241-210 (ISO, 2010) and AIGA Designer 2025 (American institute of graphic arts [AIGA], 2018), since these two reports covered ethics and standards in user experience design.

Our context analysis used ‘PEST analysis’, a tool for examining trends in a business context with four aspects of political, economic, social, and technology factors, within a business environment (Ward & Peppard, 2002). We altered the label of each factor, so that it was relevant to user experience design, e.g., business needs, user needs, design movement, and technology movement, as shown in Figure 2.

Figure 2 Modified PEST Analysis in the UXD Context

The body of knowledge is an essential starting point of any competency model development. To better understand the body of knowledge in design, we examined the different concepts in developing the design profession, both in education and industry, and summarized the interesting concepts.

In the 21^st century, the design profession was defined differently from the art and science of creating artifacts. Buchanan (1992) suggested that the field of design had grown as a discipline to resolve the wicked problem, and Cross (1993) depicted the design as a scientific process and discussed the development of design science. Findeli (2001) classified design into three main domains: art, science and technology. He discussed how the design profession would develop to meet the new world view in the 21^st century that the art domain would change to perception, and science and technology would combine into action. The perception concerned with visual intelligence, and action referred to the technological act reflecting the design logic. These ideas depicted design as a discipline of balancing between art and science knowledge that involved technology feasibility.

The design discipline became more apparent and vital with the emerging of the human-centric design movement. Moreover, the body of knowledge of design had extended beyond the area of art and science, as several experts suggested that it involves research activities, technology and business (Frayling, 1993; Swann, 2002; Faiola, 2007). Therefore, the design had already created its discipline, as a balancing activity between business stakeholders, through the design itself and design research (Oygur & Blossom, 2010; Åman et al., 2017). These ideas formed the rigid discipline of design and applied the design process into the business ecology.

We used thematic analysis by mapping the relationships between different ideas from many design experts, as shown in Table 3. The mapping and analysis helped to recognized different themes for the body of knowledge of design. We concluded that there are four domains for the body of knowledge in design: design, art, science and business. These four domains served as a framework in molding the UXD competency model.

A brief description of each domain is:

Design Domain – As design had grown as its discipline, the design domain illustrated the design principle, process and research activities, reflecting the interdisciplinary nature of design for gathering insights from various perspectives.

Art Domain – The meaning of art always stands for a human perspective. Art provides an understanding of the human cognitive process and emotion, then creating the aesthetic artifacts for humankind.

Science Domain – Science served as logical thinking, resulting in the creation of artifacts, that fulfilled human needs in terms of usability, function and features of the product, service or system.

Business Domain – Design has a significant role in business. Designers need to understand the business environment, design for value, and manipulate a design project for business success.

From our literature review of user experience design, 226 memos of competency for user experience designers were extracted covering all four domains. These memos were clustered into distinct elements and units for molding the competency model in the next phase.

Competency model development identifies components in a competency model, which are units and elements of competency. The units of competency are groups or clusters of abilities and proficiency levels in different tasks. Besides, the elements of competency refer to the details of knowledge, skills, behaviors and characteristics in the competency model. The components of a competency model are shown in the paths shown in Figure 3.

Figure 3 Path model of components in a competency model

We applied the text analysis and thematic analysis, with the grounded theory technique, a qualitative and inductive approach (Charmaz, 2006). This technique allowed us to classify elements of competency into different units of competency by recognizing themes and patterns from the text (Bernard, 2018). The procedure of thematic analysis is shown in Figure 4. We extracted 226 memos of key ideas from the literature and classified them by card sorting and mapping. We recognized 63 elements of competency for user experience design and categorized them into different themes to form distinct units of competency. An example of card sorting procedure is shown in Figure 5. Thematic analysis hypothesized that there are nine units of competency: design principle, design process, design research, aesthetic value, information art, usability value, information technology, business acumen and project management. Table 4 summarizes key factors in user experience design competencies from previous studies. Figure 6 shows a path model of the hypothesis of user experience design (UXD) competency model.

Figure 4 Thematic analysis procedure

Figure 5 Example of card sorting procedure

Figure 6 Hypothesis formed from nine units of competency in user experience design (UXD) competency model

Since competency modeling is a qualitative approach, previous studies used various qualitative and quantitative methods to evaluate their models. Qualitative methods included expert interviews, in-depth interviews, panels of experts, validation workshops or critical incident technique (CIT) interviews. Quantitative methods were based on basic statistics, e.g., analysis of variance, t-test or x². Many pieces of research on competency model or framework development suggested that subject matter expert judgment was an adequate method for evaluating the model to confirm the concept (Marrelli et al., 2005; Calhoun et al., 2008; Donohoe & Needham, 2009; Horng et al., 2011).

To evaluate the competency model, 25 experts were sampled, through the method of snowball sampling, from both academic and industrial fields. There were ten academic and 15 industrial experts. Our experts had an average 9.6 years of experience in user experience design, including interaction design, human-computer interaction, digital design, digital media design and digital platform development. Since the target population of this research was homogeneous, 25 expert judgments were adequate for assessing the validity and reliability of the model. Macmillan (1971) stated that if a number of experts ≥17 → an allowable error < 0.02, thus our sample was adequate for validating expert judgments.

A content validity index (CVI) was used to assess model validity, and a modified kappa statistic was used to measure inter-rater reliability and confirm the agreement. CVI is a good indicator for validating content, within a model or a research instrument through expert judgment on a 5 point scale (1=not relevant, 2=somewhat relevant, 3=quite relevant, 4=high relevant, 5=very high relevant). Item-level CVI (I-CVI) is a validity index for each item in a model by counting expert ratings of 4 or 5 and dividing by the number of experts. Lynn (1986) suggested that the cut-off for an excellent item was >0.80 when there were more than ten experts.

The modified kappa statistic (K*) indicates the level of agreement of experts by calculating the probability of chance agreement (P_c) for each item, then calculating kappa following these formulas (Polit et al., 2007):

P_c = [N! / A! (N-A)!]*0.5^N

where N = number of experts, and A = number agreeing on good relevance (rating above 4 or 5)

K* = (I-CVI – P_c) / (1-P_c)

If K* > 0.74, it can be considered excellent, between 0.60 and 0.74 is good, between 0.40 and 0.59 is fair, and below 0.40 is poor. After calculating the modified kappa statistic, each item, with I-CVI ≥ 0.78, can be considered excellent reliability regardless of the number of experts. Also, CVI can evaluate each unit of competency through scale-level CVI (S-CVI) by calculating the average of I-CVI within a unit (S-CVI/Ave). If S-CVI/Ave ≥ 0.90, the content within a unit is valid (Polit et al., 2007).

The user experience design (UXD) competency model had four domains, nine units and 63 elements of competency, as shown in Figure 7.

Figure 7 UXD competency model

First, the design domain for user experience competence consists of principle, process and research practice in design. For user experience design, the principle of design focuses on awareness of designer roles and responsibilities in creating opportunities and solutions based on design thinking, process and research for complex problems. UX designers must have well-rounded knowledge and skills for working with a multidisciplinary team. They need to control and facilitate the process and research for design-oriented projects.

Art Domain is always a significant competence for every design field, since art is the design fundamental. However, the art domain in the digital era differs slightly now, especially for the career of a user experience designer. In the UXD competency model, the art domain has two units – aesthetic value and information art. Aesthetic value is the perception of the emotional hemisphere of a human to a design output, where designers implement a look and feel to create a desirable design. User experience designers must focus more on the information and content, as the digital era allows users to create, edit, store, transfer, manipulate or erase information and content easily. User experience designers need to demonstrate an ability to direct and construct information and content into an information architecture, content strategy and story. Also, UX designers need to assist users in using, searching and extracting meaningful information.

Scientific practice and thinking provides logical and realistic factors in design activities, because design is a scientific process of experimenting from a new abstract idea to a credible output. For user experience design, satisfaction occurs when users can use a product, service or system that works for them. Therefore, user experience designers must consider the usability value in creating a usable, practical and safe design output for users. Design in the digital era is influenced by information technology, so user experience designers need to apply this technology and the big data to deliver a product ecosystem or a platform with a meaningful experience.

Finally, understanding the business context is a required competence for every career. However, there are some specific areas concerning the business context, that user experience designers need to understand. A promising task for user experience designers is to diffuse the user experience design concept from the team level to the organizational level. User experience designers can assist business in branding, generating new business models and foreseeing trends and driving forces, related to business research and development. To diffuse the user experience design concept to a business, user experience designers must use necessary skills in project management to prepare and control user experience design projects, and transfer the knowledge throughout a company in its specific terminology.

Figure 8 shows I-CVI and K* values obtained from an expert discussion. CVI indicates that the content within this competency model was validated both at item level (elements of competency) and scale level (unit of competency). K* demonstrates the reliability of the model assessed by experts from snowball sampling. Cronbach's α was used to measure the reliability coefficient for each unit, which indicated that the expert' judgments were reliable (>0.60-0.80) to very reliable (>0.80-1.00) (Ahdika, 2017).

Figure 8 I-CVI and K* for each element of the UXD Competency Model

The average value of each unit was compared with the model mean (the grand mean) using t-tests to classify different types of competencies. T-test results are in Table 5. The unit of competency that has a high significant score can be considered a core competency. On the other hand, it can be classified as a cross-functional competency, if it has a low significant score. A functional competency is a unit that is not significantly different from the model mean. The classifications are shown in Figure 9.

Figure 9 Core, functional, and cross-functional competencies of user experience designers

The usability value and design research units are core competencies for user experience design, because they have highly significant scores. Experts had a consensus that user experience designers were responsible for conducting design research and finding user insight for a successful digital product, service, or system design project. Usability value was a mandatory competence in designing user experience, since the basis of a successful design is a usable product, service or system. Moreover, UX designers needed to be aware of user risks and errors, and provide help and support for users to complete their tasks using the design outputs.

Design principle, design process, aesthetic value, information art, business acumen and project management were not significantly different from the model average and can be considered functional competencies. The functional competencies were required knowledge, skills and attitudes in designing a user experience. However, they were adjusted to the conditions and criteria of different design projects. The design principle and design process units allowed designers to work on complex design projects. They needed to generate multiple ideas and control the process of making tangible design outputs. Aesthetic value and information art identified that designers need to express and enhance the user experience in understanding and using products, services or ‘beautiful’ systems. To possess a successful design role in any organization, designers must understand the business value, work in a business environment, and diffuse the concept of user experience to teams and organizations.

Although many experts considered information technology an important competency for user experience designers, they recommended that user experience designers were not technicians, who work with technical issues in the product, service or system development. However, it is a ‘nice-to-have’ competence, that helps designers in working with multidisciplinary teams. Accordingly, information technology was classed as a cross-functional competency for user experience designers.