Archives of Design Research

On the wide spectrum of UX design, present study focused on website UX design as the World Wide Web has become an indispensable part of people’s everyday lives (Gao et al., 2022). Considering its prevalent use, in recent years, much focus and extensive research and debate has been placed on the UX design of websites. Website user experience can be defined as a user’s perceptions and responses that result from the actual and or the anticipated use of the website (Bevan et al., 2015). A number of research have suggested that usability and user satisfaction in terms of system use and acceptance are the two important aspects of user preference that predicts web success (For example, see Palmer, 2002; Muylle et al., 2004; Zviran et al., 2006; Lee and Koubek, 2010). It was found that once users feel frustrated or are not satisfied with the web page, they tend to abandon it and immediately visit other web pages (Parush, Shwarts, Shtub, & Chandra, 2005). It is, therefore, important to research and understand the attempts to reduce user’s frustration and increase user satisfaction with the goal of creating a successful website.

A law of UX, named the Peak end rule states that people judge an experience based on the emotional peak of the event or the end, instead of having an average of the duration of the overall experience (Kahneman, 2000, 2011). This observation is of close interest as it suggests that we should pay close attention to the critical highs and lows to ensure users evaluate an overall experience positively(Yablonski, 2020).

Research on Peak-end effects were mainly explored in general hedonic experiences, such as receiving gifts (Do, Rupert & Wolford, 2008), remembering the enjoyable experience of holidays (Kemp et al., 2008), and tourism or visitor experiences (Dixon, Victorino, & Kwortnik, 2007; Kim & Kim, 2019). Despite the wide range of psychological studies that show the influence of Peak end rule, only a limited number of studies have examined its effect on UX. Recently, some studies have demonstrated the Peak- end effect in the field of UX, such as in form-filling interfaces (Cockburn et al., 2015), game player’s experience (Gutwin et al., 2016) and while viewing a short virtual reality (VR) movie (Strijbosch et al., 2019). Peak-end effect was also found to be applicable on the emotional evaluations done during product operating tasks (Ho & Chen, 2019). Through inspecting such studies, it can be stated that if Peak-end effects are validated and understood in UX design, it may provide important implications. For example, Cockburn, Quinn, andl Gutwin (2017) claim that designs that fail to consider peak-end effects may cause adverse user reactions. Therefore, additional research may be needed to develop the idea of exploring Peak-end effects and add on to the discussion of UX design.

This study specifically aims on the manipulation of negative peaks created by error pages (404 pages) for the following two reasons. First, as mentioned previously, error pages cannot be avoided completely and the negative peaks created through it may be critical. Second, referring to a popular point made by Don Norman, UX designers should design for errors: meaning assume errors would happen and plan for its recovery.

Users may encounter 404 error pages for various reasons, such as through mistyped or broken links or when they reach moved or deleted content. Currently, many companies and brands offer custom- designed error page instead of blank landing pages provided by the server to help the dead-end situation encountered by users. By default, error pages serve to deliver a message informing of the error. Communication with web users is usually performed through visual or textual messages within the website (Jankowski et al., 2018). To communicate the error message, there are various approaches taken by the companies and brands' websites. For example, Google even has its own guidelines for creating 404 error pages.

Fig. 1 shows Google’s present error page, which represents a custom- designed error page. Including Google, we qualitatively reviewed the custom- designed error pages used by various companies and brands' websites'. The cases were collected for a total of 30 days, from 2022 March 29th to April 27th. As error pages can be encountered during any web experience, we aimed to collect diverse cases, not limiting the domain to a particular area using the Google search engine. Keywords used for searching included but were not limited to page not found design, error website design, and 404 error pages. Some error pages were also obtained while navigating large companies’ homepages.

Figure 1 A screenshot of Google’s present error page

Consequently, 100 cases were collected and analyzed. Thematic analysis was conducted on the contents of the error pages, and they were finally cross-checked by the two researchers. Because visual presentation and contextual characteristics were shown as the most prominent features, two factors were analyzed as the main criteria. Various forms of graphics were presented, from still photos to illustrations (see Figure 2). There were instances where only typographic items such as visual quotes were the main graphical factor on the design. Graphical content was not only limited to still images, but also included gifs and short animations. The context of error pages varied as well; some texts plainly informed the users of the 404 error, yet had differences in tone. For instance, as observed in Figure 1, the context of the message only informed the user of the error. In contrast, Figure 2 (b) informs of the error without explicitly mentioning it, yet sending a message that something has gone. The error context message were also commonly used in a fun, light manner, at times joined with a joke (see Figures 2 (a) and (c)). We categorized the error page context according to Tzeng (2004), who categorized the contextual approach in HCI into simple technical and emotional. Simple and technical context showed plain and direct message, while emotional approach presented creative and witty messages that entertain the user. Furthermore, we noticed that interactive approaches that attempt to entertain or give solutions to recover from the error page were an important factor. For instance, along with the error message, navigation links were provided, where users could explore other sections of interest. These both serve as solutions that help the users to deal with the error page by looking at alternative options, or simply to go back to the main page and begin their experience again. Another take to entertain and interact with users was providing an interactive game on the error page, as shown in Figure 2 (c). Level of interaction is important as content and search capabilities of commercial websites are strong predictors of user satisfaction (Zviran et al., 2006). Moreover, it was found that users value interactive services on the basis of how well they satisfy their needs in a particular situation, meaning website success is much more than the website’s actual characteristics (Hassenzahl, 2003). Thus, we identified three categories of error pages, which are (1) Visual presentation (2) Contextual message and (3) Level of Interaction.

Figure 2 Example of categorized error pages: (a) Marvel, (Source: www.marvel.com) (b) KonMari, (Source: konmari.com) (c ) Kualo, (Source: www.kualo.com) (d) Flickr, (Source: www.flickr.com)

The categorized contents of the error pages are summarized as follows:

Visual presentation: Graphical information that appears on the page expressed in the form of an image. It mainly takes the form of photos, icons, typography and logos. Some websites display short animated images or videos.

Contextual Message: The error message contains information about the occured error. The context of the message was either plain and straight forward or had emotional context within.

Here, the tone of the message differs within the context.

Level of Interaction: The level of interaction depends on whether the solution is given to the user or not. This is mainly provided by buttons or a list of instructions for the user to try in order to self-serve and resolve the issue. On instances, interactive games may be provided to entertain the user.

We identified three categories of error pages, which are (1) Graphical presentation (2) Contextual message and (3) Level of Interaction. From the categorization, graphical presentations were controlled in all conditions as there are too many confounding variables, mainly on the concern of the dominating effect of color. For instance, color has been found to be the most noticeable design characteristic at first sight (Reinecke et al., 2013). Thus, we used a 2 (Contextual message; neutral, witty) x 2 (Level of Interaction; none, high) between-participants design. The neutral contextual message and absent levels of interactions were to serve as control conditions, as they were the most frequently found desings in 404 error pages. Specifically, Tzeng (2004) mentioned that the most common form of the error messages in computer interfaces were short and technical. The stimuli were presented among one of the four conditions, and we tried to explore the user's reaction for each of the four types of error page presentations.

We chose online shopping sites as the context of the website stimuli. Instead of selecting experimental stimuli from the 100 websites that we collected, we decided to create our own stimuli to control for confounding variables. For instance, a user's previous experience may alter their view of shopping website brands. A virtual shopping site ‘Myshop’ was developed by the researchers. One main page and four error page design prototypes for each condition were created and was saved and uploaded into the Qualtrics software. To ensure the presentation of stimuli remained equivalent, participants were requested to complete the survey on a PC condition. See Fig. 3 for error page stimuli.

Figure 3 Error page stimuli, conditions from left to right: NH (Neutral Message x High Interaction), NL (Neutral Message x Low Interaction), WH (Wit Message x High Interaction), WL (Wit Message x Low Interaction)

Participants were recruited via the opportunity sampling method, specifically through online advertising. Final sample (N = 141) had an average age of 24.79 years (SD = 3.60; range 19-39 years), with a gender breakdown of 66 males, 75 females. The participants were randomly assigned to the experimental groups (NH N = 37; NL N = 35; WH N = 33; WL N = 36). For the semi-structured interview, we recruited participants from the survey who noted that they are willing to participate in further interviews. We messaged all 43 participants, and those who were available at the particular time and willing to take part comprised the final participants for the interview. A total of 4 people, one from each experimental group was gathered. Demographic information is shown in Table 1.

To measure mood after viewing an error page, Positive and Negative affect Schedule (PANAS; Watson, Clark, & Tellegen, 1988; Lee, Kim, & Lee, 2003) was used. The scale consists of 20 adjectives; 10 Positive Affect (PA) and 10 Negative Affect (NA). Although PA and NA may seem to be opposite mood factors, they have emerged as two distinctive dimensions (Watson et al.,1988). PA is conceptualized as the extent to which a person feels feelings of enthusiasm, activeness, and alertness, while NA is a general dimension of subjective distress and unpleasurable engagement that subsumes various aversive mood states such as anger, nervousness or fear (Watson et al., 1988). Using a 5 point likert scale, participants indicated the degree of each of the adjectives that were representative of their current mood. Then, an overall scores of both PA and NA of the participants were calculated to examine the current mood of the participants. Continued intention was measured with three questions, using a 7 point Likert scale (Bhattacherjee, 2001, Lee et al., 2015, Liou et al., 2015, Shin, 2011). Questions of continued intention asked if participants would intend to use or would frequently use the system in the future. Reliability analyses for all measures show good internal consistency with Cronbach’s α of 0.85 and 0.96. See Table 2 for Cronbach’s α values.

The semi- structured interview was divided into two phases; (1) questions to build rapport and (2) main questions asking about the error page experience in detail. To ask main questions, we first took an analysis of the data that individual participants have reported in the online survey.

Participants could start the study over an anonymous link distributed by Qualtrics. The first screen contained a short introduction to the study. On the next screen, participants were given a short shopping scenario to read and imagine about. Then the participants viewed a screenshot of a main page of a virtual shopping page, where they could navigate to click on the desired area of interest. Once the participants clicked an area, an error page was presented. They were randomly assigned to one of the error-page experimental conditions, meaning participants viewed only one error page after clicking one area of interest on the main shopping page. After the interaction, participants undertook a survey of current mood, continued interaction of use. Finally, demographical data was collected.

The semi-structured interview was conducted after about 1-2 weeks after they had participated in the online survey. Interviews were held individually by meeting at an online appointment via Zoom, a popular video conferencing platform, which lasted approximately 20 minutes and were audio- and video-recorded for later transcription.

The analysis showed that there were no main effects of interactivity in both PA and NA. Differences between high interactivity(M = 18.49, SD = 7.60) and low interactivity(M = 17.28, SD = 7.04) was not significant on PA, F(1, 137) = 1.54, p = .217, η_p² = .011. Likewise, high interactivity(M = 20.39, SD = 6.55) and low interactivity(M = 20.51, SD = 7.65) differences in NA were not found, F(1, 137) = 0.00 p = .994, η_p² = .000. These two results suggest that interactivity alone does not impact the emotion of the users. Findings from the qualitative interview supported this statement, as most participants mentioned that the provision of interactivity was not meaningful to them. The participants that were in high interactivity conditions(P1 and P3) both mentioned that they did not clearly remember the provided interaction points, meaning that the high interactivity may not be an important factor in error page designs. Specifically, P3 stated that, “I remember the overall visual style of the error page very clearly. It was the first thing that caught my attention. However, I cannot remember the details of the provided solutions(interactivity),” which may be explained in that the overall visual presentation style would be more important than providing higher levels of interaction to the users. For interview participants in low interactivity conditions, we asked if they would feel better if the pages gave more interactivity (i.e., solutions). However, P4 mentioned that “even though any interactivity would be given to me, it would be meaningless as it does not solve the main problem that occurs from encountering 404 error pages,” which puts an emphasis on the lack of efficacy on the provision of solutions to smoothen people’s negative emotions caused by 404 error pages. Perhaps the location of the interactions may have been the issue in our stimuli, as Padilla (2005) claimed that error messages located on the top can help users to clearly distinguish the error message from the rest of the applications and capture the user’s attention. Therefore, the participants may have not found the error suggestions and may have gone to an easier solution that they know of: to simply abandon the page(Parush et al., 2005) or click the go back or refresh button provided by the browser.

A significant main effect was found for context of the message: witty message(M = 20.30, SD = 8.15) had a significantly higher PA compared to neutral message(M = 15.56, SD = 5.55), F(1, 137) = 17.48, p < .001, η_p² =.113. Namely, witty messages may generate positive emotions to users compared to neutral messages. On the other hand, contextual message did not have significant effects on NA, as the difference between neutral message (M = 15.56, SD = 5.55) and witty message(M = 20.30, SD = 8.15) was not significant, F(1, 137) = 2.94 p = .089, η_p²= .021. Findings from the qualitative interview may explain this result as P3, who was in witty condition, mentioned that he gave positive affect ratings because he, “thought that the witty message made me get away from the rigid feeling from text-oriented messages that I usually see in 404 error pages.” Perhaps the slight relief given through witty messages may explain the reasons for the higher positive affect ratings. Moreover, P2, who saw a neutral error page, suggested “kind and witty designs would be good for improvement in error page designs,” which hints that witty messages may become a good error recovery medium.

Considering interaction effects of the two variables on PA and NA, significant interaction effects of level of interactivity x contextual message was found on PA, F(1, 137) = 4.99, p = .027, η_p² = .035. Further simple main effect analysis identified that in neutral message condition, the differences between high interactivity (M = 15.00, SD = 4.88) and low interactivity (M = 16.14, SD = 6.20) was not significant, t(70) = -1.72, p = .386. There were also no significant differences between witty message(M = 18.39, SD = 7.70) and neutral message (M = 16.14, SD = 6.20) in low interactivity conditions, t(69) = .38, p = .181. However, in witty message conditions, high interactivity (M = 22.39, SD = 8.24) was significantly higher in PA compared to low interactivity (M = 18.39, SD = 7.70), t(67) = 2.09, p = .041. Likewise, in high interactivity conditions, witty message(M = 22.39, SD = 8.24) had a significantly higher PA compared to neutral message(M = 15.00, SD = 4.88), t(68) = -4.63, p < .001.This aligns with the findings of simple main effects: witty messages create more PA than neutral messages. With the analysis of interaction effects, it can be stated that when a witty message is given with high interactivity conditions, positive emotions may be felt by users.

Significant interaction effects for level of interaction x contextual message was also found on NA, F(1, 137) = 5.30, p = .023, η_p² = .037. Further simple main effect analysis identified that in neutral message condition, the differences between high interactivity (M = 18.16, SD = 6.46) and low interactivity (M = 20.86, SD = 6.84) was not significant, t(70) = -.87, p = .090. Similarly, in witty message condition, the differences between high interactivity (M =22.88, SD = 5.78)and low interactivity (M =20.17, SD = 8.44) was not significant, t(67) = 1.54 p = .128. There were also no significant differences between witty message(M = 18.39, SD = 7.70) and neutral message(M = 16.14, SD = 6.20) in low interactivity conditions, t(69) = .38, p = .707. However, in high interactivity condition, witty message(M = 22.39, SD = 8.24) had a significantly higher NA than neutral message(M = 15.00, SD = 4.88), t(68) = -3.20, p = .002. This result is interesting as it indicates that witty messages may also give rise to more NA compared to neutral messages, when combined with high interactivity conditions.

Interaction effects and simple main effects together demonstrated both high PA and NA in witty messages combined with high interactivity conditions. To further understand this mixed finding, each item from the PANAS scale was observed in detail. From the high interactivity x witty message condition, items that had the highest rating in PA were ‘interested’ (M = 2.52, SD =1.39), ‘excited’ (M = 2.39, SD =1.25), and ‘attentive’ (M = 2.36, SD =1.14). Participants may have felt such emotions as the witty context in the error page was different from what they usually saw, as P3 described this type of error page as ‘unusual’. Thus, such positive emotions may have been caused by viewing an error page that is new and refreshing. Items that had the highest rating in NA were ‘nervous’ (M = 3.15, SD =1.20), ‘irritable’ (M = 2.97, SD =1.10), and ‘upset’ (M = 2.82, SD =1.07). Participants may have felt such emotions due to the lack of apologetic messages in witty designs. Findings from Tzeng (2004) claim that error messages that are apologetic aids in more desirable psychological experience in users. As witty messages are characterized with fun, entertaining messages, the lack of apologetic voice may have caused more NA in users. Findings from the qualitative interview align with this statement, as P1 mentioned that the “error message should be polite and kind to improve negative feelings that come from encountering error pages.”

It is important to acknowledge individual differences in explaining mixed emotions felt in high interactivity x witty error page designs. Different aspects of the error page give rise to different emotions. Referring to the interview results, even four of the participants gave different responses on their preference of contextual messages; as they suggested or preferred either ‘polite(P1, P4)’, ‘kind’ (P1, P2), ‘friendly’ (P2), ‘witty’ (P3) messages. Therefore, depending on the user, witty designs can be perceived differently in the dimensions of PA and NA.

In addition, the mixed results of PA and NA in witty designs could be due to differences in the current mood of the users. As the user’s tolerance for error may be different, the current mood may have played an important factor. For instance, angry people, given their baseline differences in state affect, may have felt more negative emotions (i.e. irritation) when viewing error pages. This suggests that current mood may have been likely to bias emotional ratings. P3 also commented that “depending on the current mood of the user, the witty message error page may feel too playful.” Thus, further studies should measure current mood as covariate. Moreover, as this study used a scenario approach in the main survey, it may have not fully created or represented the emotion that would have been felt in real life (Schoefer and Diamantopoulos, 2008).

High interactivity(M = 3.47, SD = 1.34) and low interactivity(M = 3.12, SD = 1.46) difference on continued intent of use was not found, F(1, 137) = 2.37 p = .126, η_p² = .011. This seems to be clear as P1 stated that, “although there are alternative links given, I will stop using the website because I think there is a problem with seeing the 404 not found error page to begin with.” This also aligns with the finding that once users feel frustrated or are not satisfied with the web page, they tend to abandon it and immediately visit other web pages (Parush et al., 2005). Again, the downfalls of encountering error pages itself are clear, thus it is important to reduce the instances of error pages to not lose customers. No significant interaction effects for level of interaction x contextual message was found on continued intent of use as well, F(1, 137) = 0.05, p = .822, η_p² = .000. Contextual messages, however, had significant main effects. Continued intention of use was significantly higher in witty message(M =3.54, SD = 1.42) compared to neutral message(M = 3.06, SD = 1.37), F(1, 137) = 4.34, p = .039, η_p² = .031. Namely, when witty messages are used, users are more likely to continuously use the website despite viewing error pages.

This finding may be explained by the aspect of humor that is included in witty messages. Witty context is characterized by a sense of humor, and humor is related to several consumer benefits. For example, when humor is used in advertisements, they attract attention, enhance PA, and are more memorable (Eisend, 2009). In typical computer mediated communications such as chatbots, humor gives benefits such as a more enjoyable interaction, a higher system rating and is suggested as a good error recovery medium (Niculescu and Banchs 2015; Morkes, Kernal, and Nass 1999; Niculescu, van Dijk, Nijholt, Li, & See, 2013). Thus, when websites tackle errors with a sense of humor, users may intend to come back. Yet, humor in both business and customer service interactions requires a more nuanced approach (Malone, 1980; van Dolen, de Ruyter, Streukens, 2008). P3 also mentioned that “when appropriate witty designs are used according to the target, users would intend to come back to the website despite the error.” Meaning, only an appropriate use of witty designs that suits the taste of the users may increase the intention to reuse. In respect to the mixed emotions felt by the users in high interaction x witty message context, it can be concluded that witty designs should be used carefully, considering both nuance and the characteristics of the users.