Archives of Design Research

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/), which permits unrestricted educational and non-commercial use, provided the original work is properly cited.

We are inundated with a myriad of display surfaces. With global access to 6 billion mobile phones alone, (Wang 2013) not to mention other devices such as tablets and laptops, the small screen circulates in our midst as an ubiquitous “black mirror”, reflecting ourselves as well as also being a portal to an instant-on state of existence in a world of apps, games, communication and images. From metro stations to airports, it is virtually impossible not to walk through a building without passing an LED screen illuminated with the latest advertisement. For some displays the screen is eschewed altogether, such as the holographic announcers of “Holly” and “Graham” at London Luton airport, which are used to “communicate important security messages as passengers pass from check-in to the departure lounge”. (BBC 2011; Fig. 1)

Figure 1 Hologram at London Luton Airport

Of course, display surfaces for images are not limited to screens. Paleolithic humans used holes in their shelters to form camera obscuras (Gatton 2010), with the ensuing image reflected on cave walls. Light, including 3D displays, is also projected on surfaces such as sheets of water and building surfaces. One example, out of many, is Nike’s show for their new Jordan Melo M8 shoe, displaying a light projection on the Hudson River of NBA star Carmelo “Melo” Anthony dribbling a ball on the water. Screens are also no longer watched in cinephiliac isolation (Sontag 1996). In view of what Susan Lord and Janine Massechault call “fluid screens” (2009), viewers move restlessly between screens, transferring their gaze from the cinema screen to the mobile one and back again. Francesco Casetti (2011) writes on precisely this decentralised gaze in Atom Egoyan’s short film, Artaud Double Bill (Chacun son cinema). The film’s two central characters are friends watching films in separate spaces: Nicole is watching Atom Egoyan’s The Adjuster, while Anna is at another cinema viewing an excerpt of Jeanne d’Arc from Jean-Luc Godard’s Vivre sa vie. Anna sends to Nicole a clip from Vivre sa vie on her mobile phone, so that Nicole ends up watching The Adjuster as well as the clips from Anna on her phone, hence participating in her friend’s filmic experience as well. Casetti reads this interplay of screens in three traits: as facilitating a network of social discourses; as a decentralised gaze; and as “a kind of surfing” sliding over the film, isolating only elements of interest.

These studies and constructions of screens underscore not only the significance of images, but how they are framed and contained. As oftquoted from Anne Friedberg (2009), “...how the world is framed may be as important as what is contained within that frame.” (1) This article traces an evolution of display spaces from its cultural construct as a framed and demarcated space to one that is more porous and fluid, interweaving images with physical realities. Using case studies of displays taken from cinema, digital video and video mapping, I argue that our cultural conception of the screen as a display surface evolves with advancing digital technologies as screen realities give rise to ever more complex relationships between image and referent. What happens when new technologies disrupt our cultural constructs of screens and display spaces? How may we re-define the spaces of image and those of the world? How then do we re-locate ourselves within those spaces? In the process, these questions prompt us as well to re-think our engagement with the spaces of image worlds and our understanding of the physicality of our realities in terms of surface, mass and dimensionality.

The flat dimensions of a display surface are familiar. Typically a screen, or perhaps even a sheet of white cloth, such display surfaces are a specific, angular (typically rectangular in varying ratios, with two longer ends as its length and two shorter as its width) surface on which the image appears, enclosed by its edges. This geometrical delineation of space for the image persists in various forms as a manifestation of display surface – even virtual worlds, such as Second Life, host virtual screens in such two-dimensional displays of images, even though there are no physical barriers, constraints on room size, limitations of light or sight for keeping to this composition.

Such a specific space in which the image appears thus frames it in two ways. Firstly, it marks the borders of the image, or at least the references to the borders of the image, in terms of where the image is (designated as onscreen), and where it is not (offscreen). In considering the space of a film image, both onscreen and offscreen materials are considered significant (Sikov 2009, 30), with the definitions of that display space made in reference to what the viewer can and cannot see and, correspondingly, where the image exists and where it does not.

Secondly, the screen delineates the boundaries which contain the diegesis. What happens in the story world of the display is enclosed, even ensconced, in and behind the screen. In this sense, the screen is a threshold bordering two worlds, concealing yet revelatory: a virtual window, as theorised by Anne Friedberg (2009), simultaneously taking into account framing, mediating, lighting, perspective and architecture. In her book, The Virtual Window, Friedberg traces the application of the window, so long a cultural construct for pictorial framing, to the screen, where the perspectival framing of the window (and thereby the screen) enables the spectator not only to look and apprehend the represented reality of the image, but is separated (and protected) from the filmic space. Hence, as Friedberg argues, the screen as window is the paradoxical aperture through which the psychic space of the film (as described in apparatus theory) enfolds onto the physical space of the screen, and through which the spectator acquires her mobilized gaze. To Friedberg, the construction of the screen as window is based precisely on the existence of its frame, and thereby on the (virtual and physical) solidity of that structure: “a virtual window is reliant not on its transparency but on its opacity”. (344)

Recognition is thus made, implicitly or otherwise, of the screen’s boundary; its norm is to be a barrier which demarcates filmic and physical spaces, and any detraction from that is an exception. The direct address of a character in the film looking into the camera (ie at the audience), for instance, is considered to be such a transgression of that enclosed story world, a violation implicit in its name – to break the fourth wall. In direct address, the film character’s acknowledgement of the audience destroys the illusion of the screen’s boundary, with the general ensuing effect of ambivalence (Brown 2012) lying precisely in that merging of the viewer’s and the film’s worlds. Interactive cinema, too, plays precisely on blurring those limitations. For instance, Chris Hales’s interactive film show, Cause and Effect, includes works such as Natural History (2003), which requires the audience to hum together in order to refocus an image (Hales 2005, 61). Hales thus uses cause and effect to underscore the breaching of the threshold presented by the screen. The Japanese horror film, Ringu (1998), also mines the supposed inviolability of the screen for its climactic effect. A horror thriller about a cursed videotape that kills a week later anyone who watches it, Ringu follows the efforts of an investigative reporter to save herself, having inadvertently watched the tape and fallen under its curse. The played videotape, disseminating the curse, is confined to the world behind the screen. It is thus entirely appropriate that the curse itself, manifest in Sadako’s lurching form, should stretch out of the screen to reach its terrified victim, for the horror of Ringu lies not simply in the terrifying semblance of Sadako but in the violation of the screen’s boundaries, which had erstwhile kept the horror of the curse (as the videotape) at bay, so to speak, and enclosed in the separate world behind the screen. That the effect of the horror is doubled for the cinema audience who had been similarly watching the film (and the cursed videotape) on a screen separating them from the story world of Sadako’s curse is a further aspect of the film’s potent horror, for which the breach of the screen in the film is its core: the screen no longer a marker of a confined image-space into which we peer and from which we are protected.

To that extent of the screen’s protective ambivalence, Serge Daney’s (2002) characterisation of the screen is intriguing. He relates the screen particularly in reference to French film critic Andre Bazin’s (1969) aesthetic of “pure cinema” – which Daney terms a “fantasy”. In counterpoint, Daney emphasises the reality of the screen:

What overdetermines the Bazinian fantasy [of perfect illusion] and, in its wake, a whole swath of idealist discourse on the cinema is a comical vision of the screen as the surface of a Teflon saucepan (in glass), capable of ‘sealing’ [in the culinary sense] (saisir) the signifier. (34)

The word “sealing” (saisir) is indicative, in that the screen is not only a covering, but also encloses in a complete, even smothering, way. Of this, Daney is clearly dismissive, as evident in his casting of such a construction as “a comical vision”. For him, the image exceeds the screen, bursts free of it: “The screen, the skin, the celluloid, the surface of the pan, exposed to the fire of the real and on which is going to be inscribed—metaphorically and figuratively—everything that could burst them.” (34-35) More telling is Daney’s own metaphor for the screen in terms of the bandage, or, more precisely, the plastering of mummification:

The transparent continuum that clings to the real takes its form, the bandages that preserve for us the mummy of reality, its still living corpse, its eternal presentness: that which allows us to see and protects us from what is seen: the screen. (ibid, italics in original)

The preserving of “the mummy of reality” refers to Andre Bazin’s (1967) charging of the recorded image’s (and indeed of the visual arts in general) origins to the “mummy complex” (3), as is cinema itself, in view of its recording of time and duration, as “a mummification of change”. (9) Here Daney equates the screen to bandages – the external materials which envelop, blanket and enshroud, yet which also achieve the image’s preservation of reality. In that sense, the screen is not simply a protective covering, but also a facilitator: if the mummy represents a certain preservation of some form of life, its bandages both conceal and accomplish that preservation. In that sense, too, there is more than a hint of a fetish in thinking about the screen, namely in terms of the eroticism of necrophilia, with the screen enabling access to that “still living corpse”. Daney does not explicitly state as much, although he refers to neurosis: “The attachment to representation, the taste for simulacra, a certain love for the cinema (cinephilia), all derive less from ontology than from obsessional neurosis.” (35) The transgressions in Daney’s implied necrophilia and in Sadako’s propagation of death across the screen reinforce the demarcation of its boundaries: to cross the threshold of the screen is to enter the illicit, to actualise the curse.

The reality of the screen is bound to another purpose, acknowledged by none other than André Bazin in contradiction to his championed aesthetic of realism. Bazin champions “a perfect aesthetic illusion of reality”– a cinema so real not so much in how realistic it looks but in its inherent and organic integrity with event – a cinema that is no longer cinema, with “no more actors, no more story, no more sets”. (60) By this logic, the screen itself should cease to exist in the complete raising of this “reality coefficient”. (30) Yet, in this thinking, what, then, would be the point of cinema? The aesthetic of realism contains an inherent fantasy, which is that art has to be separated from life, and therefore necessitates a certain amount of artifice. A “fundamental contradiction” between reality and cinema is necessary “because art can only exist when such a choice [between what is worth preserving and what should be discarded] is made. Without it, supposing total cinema was here and now technically possible, we would go back purely to reality.” (26) In other words, the virtual window – the delimitations of the screen – is more than a frame for another reality; it defines it, it is the dialectic upon which our realities – physical and virtual, experience and representation, life and art – exist. The boundaries of the screen are thus concrete in that sense – they are the basis of demarcating a space for art and for another world.

The advent of digital video heralds a nascent diffusion of these screen boundaries. The portability of digital video on cameras and mobile phones not only brings an explosion of images, but also a proliferation of screens. This is already part of a long trend of precedents established by technologies such as videotape, smaller film stock (Super 8, 16mm), and early lightweight cameras (the latter’s aesthetic impact being arguably most visibly seen in the Nouvelle Vague films of the 1950s and ‘60s). As these technologies lowered costs and increased access to film production and exhibition (if not yet circulation), moving images were made on a broadening scale and exhibited in a wider range of forums, such as classrooms and home theatres.

With digital technologies, the mobile screen comes to the fore (Verhoeff 2009). As images are produced in every conceivable situation from holidays to revolutions to at-arm’s-length self-portrait sessions (the last also known as the “selfie”, a visual phenomenon so ubiquitous the word itself is one of Time magazine’s “top 10 buzzwords” of 2012, alongside recent variations such as the “shelfie” (at-arm’s-length photographs produced of one’s book collection)), events are regularly, almost obsessively, recorded, viewed, annotated, stored, uploaded or otherwise shared as image representations on screens in real-time. In this profusion of images both produced and received, digital video arguably brings to fruition Heidegger’s assertion that “[t]he fundamental event of the modern age is the conquest of the world as picture.” (134) In turn, this intensified engagement with the onscreen photographs or videos (as opposed to its referent) produces a certain “thickened” reality for the event, whereby it is experienced physically as well as through the abundance of images recorded and shared usually almost instantaneously due to the ubiquity of cameras, connectivity and social media. Visuality via the mobile screen thus not only becomes a representation of the event, but part of the event itself. Images taken of the Arab Spring revolutions, for instance, were constituent of its very reality. As Lina Khatib (2012) writes:

Photographic images, whether through cameras or video, brought the Arab Spring closer to audiences within and outside the immediate locales of the protests, acting as ‘a means of making “real” (or “more real”) matters that the privileged and the mere safe might prefer to ignore’. In this, the media were not mere reflectors of change, they were also themselves mediators and part-creators of social change. (10; emphasis added.)

With the abundance of digital images generated from portable digital video as well as their proximity to the event in time and place, an intense engagement arises with respect to screen representations. Images are not only produced and shared, but add a “thickening” to the event itself – a screen reality – which heralds additional engagement with the event’s visuality.

In this profusion of digitally produced, exhibited and circulated images, the screen reveals its porosity. The image as part of the “thickened” event fuses the screen to the spectator’s own reality of space and time. The screen is no longer a demarcation of a separate story-world and reality; it is its reality. In this case, the spectatorship of images is not an external viewing through the construct of a window or a layering, as discussed above; it is a partaking of the image’s story-world. As such, the screen does not so much enclose or separate as it blends into or fuses the realities of the image and the spectator.

Cinema, of course, has always been conscious of its screen boundaries; the implicit transgression of the fourth look, as discussed above, reveals this awareness. However, there is a conceptual leap from the implicit acknowledgement of screen boundaries to its specific affirmation which marks the porosity of the screen with its attendant fluidities of realities. I suggest that such mainstream affirmation was first made in The Blair Witch Project (1999), and echoed in subsequent films such as Cloverfield (2008) and Redacted (2007). In The Blair Witch Project, characters in the film hold portable digital cameras and record themselves as well as events around them. Supposedly unmediated, unedited and unprocessed, the resulting footage of video and sound is presented to the audience as a “film”. The term “film” here is ambiguous on a number of levels, not least of all due to Project’s fictional basis. In that sense, the boundaries of the frame persist, for in the “film” there remains an element of artifice, if anything simply in terms of how the action is framed: what has been chosen to be included (in the frame), and what has been chosen to be discarded. To that extent, the screen persists as a virtual window separating reality and diegesis.

However, on another level the screen is not a demarcation of any boundaries because it is the reality of the image-world; it is a self-conscious part of the diegesis. The characters address the audience directly rather than in transgression: in an early scene in Blair Witch, Heather faces the camera while introducing her room, opening her arms wide and explicitly addressing the audience. In Cloverfield, the character holding the camera climbs onto the roof of a building, addressing the camera directly as he does so: “all right, if this is the last thing you see, that means I died”. The direct addresses in voice and gesture are not breaches of a demarcated film world, but an explicit and deliberate reaching out into the spectator’s world and a drawing of the spectator into the film world. These words are meant for them. The spectator is not looking out of the screen as a structure framing an other film world; rather, the screen is simultaneously part of both worlds and a conduit through which both worlds fluently communicate. This is also the difference with interactive films such as those made by Chris Hales, which allow actions from the spectator to affect the film. In that sense, the frame of the screen remains a bordered demarcation to be breached. In contrast, the screen frame in films such as Blair Witch is porous in that it allows the realities of the film (Heather, Josh and Mike in the Black Hills) and of the spectator (who are watching Heather, Josh and Mike in the Black Hills), to seep through and into each other’s worlds. In the process, the audience is co-opted into the narrative, partaking, so to speak, in the film’s events not simply as a spectator but as an actual participant in its “thickened” reality borne by its profuse visuality.

One of the clearest and boldest indications of this “thickened” reality of the film’s “event” is the scene in Blair Witch where Heather, having inadvertently touched a bloody shirt, frantically washes her hands in the river and tries to reassure herself that her hands are clean. She holds them up to the camera and says: “See? See? They are clean.” Her appeal to the camera is significant in two ways. The first is that it is an acknowledgement of the camera’s agency in terms of a new visual relationship between Heather and the camera: by forcing the camera to register her cleansed hand, she is calling for it to provide for her own eyes its manner of seeing — an independent eye without engagement, participation or assumption — and to implicate the agency of the camera’s cold eye to affirm her washed hand. The second (related) significance of Heather’s appeal is how she uses the camera (and thereby the produced images on the screen) as a supporting reality. The images affirm or deny her reality; they are a constituent of her experienced ordeal, the reality through which she sees her clean hands.

If digital video diffuses the boundaries of the screen, in comparison video mapping displaces the screen altogether. The technology of video mapping, also known as 3D projection mapping, turns irregularly shaped three-dimensional objects into two-dimensional display surfaces. It achieves this by first spatially mapping a virtual replica of the object, creating a virtual model of the projection surface in the computer. By using opacity templates to “mask” the exact shapes and positions of the different elements of the object, the projected light is thus able to form an integral part of the display space and in this way turn any object in a room into a surface for light projection. This technique has been used to project light onto three-dimensional objects such as a vase on a table, but more spectacular usage thus far has been to map large building surfaces for use in light night displays such as the Fete des Lumieres in Lyon, publicity events such as the launch of the Nokia Lumia 800 with Windows phone, or in concerts such as Roger Water’s “The Wall” tour in 2011 (McElroy, 2011). Video mapping display was used to great fanfare for the New Year countdown to 2012 in Shanghai with a light display projected on two historical buildings (Fig. 3) – the Customs House (江海关大楼) and the SPD Bank Building (浦发银行) – along the Bund, and a different light display being put up for each new year countdown since then. Video mapping is not only projected outdoors; Microsoft is developing a proof of concept system which they have called the IllumiRoom (Siggraph 2013) that “augments the area surrounding a television with projected visualizations to enhance traditional gaming experiences.” (Fig. 2) A “self-calibrating” system, IllumiRoom “does not use flat white projection screens, but instead adapts the projection to the existing viewing environment” by using video mapping to project onto the surfaces of the user’s living room. The result is to “change the appearance of the room, induce apparent motion, extend the field of view, and enable entirely new physical gaming experiences.” (np)

Figure 2 Illumiroom by Microsoft

Video mapping effectively displaces the screen. No longer demarcated, the image is now co-opted as an element of our physical realities – it is part of our armchair, part of the building in front of us. The image is thus not contained within a frame; it has burst through and past it. If the screen displaying digital video fuses the image world with the real, video mapping coalesces the screen itself into the physical world. Image is no longer demarcated but is instead part of the physicality of our world. This is neither window nor frame, but a three-dimensional structure via which images simultaneously connote both flat surface and physical mass. With the image no longer contained within a frame, might this be the fruition of Bazin’s fantasy of no more cinema?

I suggest two answers. The first is that it is not so much a case of no more cinema, but of simply more cinema. Bazin’s fantasy for no more cinema stems from an ontological desire: to see a reality so clarified through the window of the screen surrounding cinema that we should be able to get at some kernel of truth of ourselves and of our world from it. In this case, with the image of the building physically fused to its referent, the abandonment of the frame for the image is precisely to project images with more reality than ever: how much more realistic can the image be if not to be mapped specifically to every contour, nook and cranny of its surface? If we have done away with the screen, it is not for more reality but for more cinema and for more spectacle.

The second answer, then, is to invert that relationship: video mapping is not about the object being the basis of the image for more truth, but about the image being the basis of the object for less. The manipulation of the digital image to date has tended to follow Bazin’s aspiration for an aesthetic of realism: to create a graphic so real that it is indistinguishable from reality. Yet, in projection mapping, I suggest that it is the converse: instead, the three-dimensional object is manipulated precisely to create a graphic so unreal that it is completely distinguishable from reality. This is the appeal of collapsing structures and flying objects characteristic of video mapping, where the spectacle lies in moving the immovable, in collapsing a rigid structure or in folding up erect columns. On one level, of course, these building surfaces still operate as a screen. Images are projected onto them as they would be on any surface and the outlines of the building frame the image similar to that of a screen. The Nokia Lumia light projection (Fig. 5), for example, would serve as such an example, presenting itself explicitly as a screen in terms of how the regular rectangular surface of the Millbank Tower echoes a mobile phone display, as is most likely the logic behind Nike’s choice of the building. A similar sentiment can be seen in the New York Times article (2011) on projection mapping, self-explanatorily headlined “When Skyscrapers Are Your Screen”.

Yet, on another level these video-mapped buildings transcend being screens in how their very physical features are co-opted into the manipulability and lability of the image, to be warped, flexed, contorted and stretched in explicit defiance of the object’s rigidity. Its tangible three-dimensional bulk becomes a manipulable mass-less image; solid buildings or objects are transformed into animated graphics. To that extent, video mapping does not so much reveal truths of our physical realities as it brings to light (literally) latent realities in the physical monumentality of the set structures before us as walls pulsate, bricks fly off their foundations and columns fold down like giant origami. In the process, the image co-opts its object not just as a display space but to actually achieve a combinative reality which fuses the bending, warping, manipulable and spectacular of the digital with the physical, monumental and concrete groundedness of its three-dimensional solidity. As Serge Daney writes, “Whoever passes through the screen and meets reality on the other side has gone beyond jouissance.” (34) In this, Daney possibly alludes to Bazin (2003) again in terms of the pleasure of what the latter deems unrepresentable (namely, death and love (in the sense of the orgasm, le petite mort)); by that reference Daney thus refers to a pleasure beyond the unrepresentable – one has gone beyond the image, beyond the screen, beyond representation, and beyond the level of reality which had thus far been within the confines of visuality as expressed through the frame of the window. As the spectator passes through the screen, she encounters the image not as such but as a digital reality – one simultaneously liberated in the manipulability of a digital object, yet still grounded in the monumentality of the building, for the video mapping only achieves its effect because it plays precisely off the solidity of the building. This, then, is a cinema which is digitally manipulable yet foundational to a physical, laborious, effortful being in the world. In this respect, I note how Vivian Sobchack (2000) writes of the electronic “instant” creating electronic space as “abstract, ungrounded, and flat – a site for play and display”, where “its flatness” is for “spectator interest at the surface”. (151) To Sobchack, electronic space “disembodies”, because of its orientation of the body with “a purely spectacular, kinetically exciting, often dizzying sense of bodily freedom.” Video mapping, however, represents a new engagement with the body: it is not about an ungrounded, abstract space, for the display space itself contains depth. Like the Lescaux drawings, these images take their perspectives not from the frame but from rock relief and irregularities, just as a cinema with video mapping take their perspectives from the etchings, carvings and sculpting of its display space conducted through material and labour. The smoothness of digital projections belie the discontinuities of its surface; the unreality of the image takes on, instead, the gravity of daily collective effort. In this sense, video mapped space re-embodies as the images of our digital reality inhabit the corporeality of built structures.

In this article, I have traced a trajectory of the engagement between image and referent through different technologies, and in turn attempting an interweaving of cinema and object, of displays and digitalities, of surface and mass. The cultural construction of the screen in cinema demarcates it as a separate space; on the other hand, the proliferation of small screens on today’s mobile devices gives rise to a “thickened” reality of event, lived and experienced through its images, so that the screen is no longer a separate space but one of slippage between event and image. Video mapping takes this development to fruition by its complete co-opting of physical realities with imagery to give rise to a digital reality which takes into account both the manipulability of the digital and the groundedness of the monumentality and of fixture. In all these ways, the screen evolves as an interface between image and spectator, and thus also instrumentally changes how images are accessed and how cinema itself is conceived. As Lev Manovich (2008) writes, “Far from being a transparent window into the data inside a computer, the interface brings with it strong messages of its own”. (184) I suggest that a re-thinking of the screen or the framing of the world brings with it messages of how realities are engaged in relation to image, and in turn how we are embodied in that engagement. Maxim Gorky (1960) writes of cinema as “the kingdom of shadows”, where cinema is “not life but its shadow, it is not motion but its soundless spectre.” (407) The digital media projections of cinema today, for all its spectacular images, are realities hosted on constructions of labour which can only take place over gruelling and laborious time. This is not the kingdom of shadows – this is the kingdom of endeavour and effort, and namely, of life itself.