Game tech - so much more than play
This report outlines some of the demonstrators from Create Converge that mix creative technologies to provoke novel play experiences. We also offer a perspective on the role of games and games technologies in other creative industries sectors, where games can enhance existing experiences, and sectors further afield where games can contribute to real-time decision making and understanding complex problems. We consider the opportunities afforded by combining games technology with developments in next-generation 5G networking. In our concluding “Next Steps” we present some takeaway messages from our experiences and offers a Create Converge pathway for future collaboration.
Computer games offer visually rich, interactive and immersive play experiences. Game players can feel transported into another world that they are able to explore and drive change in a range of realistic and fantastic environments. Games draw together the creative disciplines of art, audio and design with technical disciplines of programming, networking and mathematics to deliver products that are incredibly diverse in nature and scale.
That same offer of highly engaging worlds that provoke exploration and interaction of virtual worlds without consequence can be used to explore real-world domains in new ways and generate insight into existing problem spaces. Game engines can be used to build models or abstractions of real-world systems where multiple virtual agents interact in a spatially structured environment over time. Decisions that agents make can have consequences for other agents and their environment, just as in real-life. Games engines can also be used to blend real and virtual environments through technologies such as Augmented Reality, where the real world can be overlaid with game engine sourced data streams in real time.
Create Converge has developed expertise in computer game engines, novel game controllers, and evaluation of the play experience, and showcased highly interactive technology demonstrators, including hands-on workshops, to audiences from industry, government, and the public. These demonstrators range from highly technical augmented reality experiences to accessible self-build game controllers and sample games. Working with partners, we have delivered industry case studies that span the sectors of virtual production, audience engagement and insight, public installations, oil and gas, tourism and food production.
Intrigued? We love a chat so get in touch so we can immediately hook you up with the experts. Call Julie Craik on +44 (0)1382 432483 or email julie.craik@createconverge.eu
Mixing Creative Technologies
Some of our work has sought to provoke disruptive thinking around what constitutes a game and indeed a game controller. The examples below serve to illustrate that a game can simply be a simple audio-visual experience, and much simpler than contemporary devices are capable of, and game controllers can be anything capable of generating a readable digital signal.
For example, Virtua Walker ’87, developed as a collaboration between Create Converge and colleagues in the School of Design & Informatics at Abertay University, UK, explores current nostalgia for early game technologies and in particular “dissonant aesthetics in games, contrasting rich tactile experience (the act of walking barefoot on natural materials) with the disenchantment that results from antiquated sound and graphics and a monotonous ludic interface” (see Figure 1 and Sloan et al. 2017, https://rke.abertay.ac.uk/ws/portalfiles/portal/31904268/Virtua_Walker_Supporting_Document.pdf).
Virtua Walker ’87 also challenges the notion of what is meant by a game controller: movement is controlled by a player walking on the spot, barefoot in a sandpit containing two sensors that measure pressure (Figure 2). Virtua Walker has been demonstrated internationally including at VR READY – AR YOU? at Filmby Aarhus (Create Converge event, 2017). Virtua Walker source code and other documentation including a demonstration video is available here (https://rke.abertay.ac.uk/en/publications/virtua-walker-87).
Fig 1: Virtua Walker ’87 from https://rke.abertay.ac.uk/en/publications/virtua-walker-87
Fig 2: Bespoke controller https://rke.abertay.ac.uk/en/publications/virtua-walker-87
At a more accessible scale, and partly in an effort to excite the next generation of creative industry professionals we have developed 4 bespoke-created custom controllers and provided free-to-download computer games that showcase the controllers in action. The controllers can be built by non-experts with cheap components and our materials show both code listings and wiring diagrams. The controllers afford users with novel input mechanisms (pressure, tilt orientation, distance from a surface, material flex) and seek to provoke new in-game mechanics based on a mix of hardware, software and physics. Custom controller instructions exist for:
- A distance sensing controller with helicopter game in Unity
- A pressure controller with rocket ship game in Unity
- An orientation controller with balance-ball game in Unity
- A flex controller is included with a partial description (wiring diagram and Unity code) to encourage exploration
All software including demonstration games is available at https://github.com/DrPaulRobertson/custom-controller-demos.
By way of example, Figure 3 shows a pressure sensor controller for a rocket ship, shown in Figure 4 where the player has to land in front of the Create Converge logo (that is just coming into view in Figure 4).
Fig 3: Completed circuit board for a homemade controller, with instructions from https://github.com/DrPaulRobertson/custom-controller-demos
Fig-4:Demonstration game screenshot with logo from https://github.com/DrPaulRobertson/custom-controller-demos
As an enabling technology, these custom controllers seek to stimulate future thinking on what constitutes a game controller. As discussed below, game technologies can add value in multiple domains, and that value can be enhanced by embedding games in situ. Custom controllers are one means of effecting that embedding, where for example an Internet of Things (IoT) device in a manufacturing context becomes an input controller for a games engine.
We have also explored technologies and developed new pipelines at the interface between film and game engine capabilities for application in Augmented Reality. In order to speed up asset creation for interactive humanoid avatars in Augmented Reality we combined video game know-how and modern filming techniques to allow creation of new assets in a single step. Using a custom designed camera rig and off the shelf stereo video camera we have been able to capture actor performance on green-screen and export this into an AR application for a convincingly realistic effect (Figure 5).
Figure 5: Schematic of stereo camera capture to AR application
This development has the benefit of creating convincingly 3D avatars which are immediately relatable to the user and able to be created and iterated on quickly by the creator. These artifacts are compatible in common game engine systems and have been tested in the Microsoft HoloLens. We also carried out a proof-of-concept project with a company to a real-time holographic interface that would work on the factory floor of via the HoloLens, using QR tags linked to a database with real-time information on factory floor component status.
Create Converge, co-supported by a Story Futures funding award, Create Converge developing new training materials for virtual production aimed at undergraduate film students. By working with existing game-engine derived virtual production techniques and existing cost-effective hardware to create educational material in the form of workshop sessions and practical exercises with supporting example projects. We designed the material for integration into existing taught programs but also make these available to the wider sector through industry facing training events. Our introductory material Virtual Production is available on the Create Converge website and please see our Summary and Next Steps to engage with us.
Beyond Play
In Create Converge we have worked with other creative and cultural sectors and with sectors outwith those sectors. In sectors such as tourism and museums (for film see above) game technologies offer increases in audience interactivity and connectedness. The wider context of any exhibit or location can be brought to life with extended realities for rich audio-visual experiences. Interactivity can come from e.g., custom controllers that require physical interaction to manipulate the games technology experience. Moreover, interactivity need not be limited to a single person: multi-person cooperative or competitive experiences can provide a platform for audience connectivity across generations, and with the advent of high bandwidth, low latency networking provided by 5G (and see Convergent Technologies below) such multi-person activities could be very rich in visual detail. Further, the many successful audience engagement strategies of computer games, such as reward systems and progress through levels, are readily transposable – termed gamification – into other sectors in the creative industries.
Looking into sectors more distant from computer games entertainment, games technologies have a clear role to play in simulation and in decision making. Game engines can provide real-time data overlays on top of the real world using augmented reality technology to provide support for system management and decision making, and our case studies have explored this space.
Game engines also afford a combination of digital twinning and augmented or virtual reality an instrumented, system-scale view of a particular domain, e.g. manufacturing.
Game engines are one of several enabling technologies supporting the rapidly expanding market of digital twins, where a real-world component and/ or system of components is twinned with a virtual model of that component and/ or system, and see for example IBM’s summary (https://www.ibm.com/topics/what-is-a-digital-twin). Digital twins thus provide a mirror of their real-world counterparts. Digital twins can be constructed using games engines, with the components such as IoT devices serving as inputs to the engine (as per our custom controllers). Game engines can then be used for real-time visualisation and exploration of the digital twin which has value in monitoring. Using a mix of data acquisition and physics modelling, simulations of systems can be built such that the digital twin becomes “playable” – explorable in terms of predicting future scenarios to inform decision making. This capacity then turns the digital twin into a tool for unravelling complex system dynamics and Abertay has with colleagues previously illustrated the use of playable simulations for predictive modelling in cancer system biology (https://www.oncotarget.com/article/8747/text/).
5G Next-Generation Networking
Create Converge partners, we have investigated the potential for 5G networking combined with creative industries technology to shape sectors within creative industries (film, games, esports) and beyond (retail, port management and agriculture). Create Converge has helped shape the future use of a new 5G R&D Testbed. The Testbed benefits from both a controlled laboratory environment for development and pilot testing together with public facing spaces for solution deployment.
5G will make a significant impact in the Creative Industries and other sectors. The World Economic Forum predicts that “fast, intelligent internet connectivity enabled by 5G technology is expected to create approximately $3.6 trillion in economic output and 22.3 million jobs by 2035 in the global 5G value chain” (http://www3.weforum.org/docs/WEF_The_Impact_of_5G_Report.pdf). The Scotland 5G Centre commissioned a report by Vodafone that considered the potential for new products and services incorporating technologies such as Artificial Intelligence (AI), robotics, automation, advanced communications and IoT. The report notes: “The impact of this is significant. The latest estimates show the digital sector contributing approximately £150 billion to the economy each year –7.7% of total UK output – and accounting for
1.5 million jobs. This revolution is also a strong draw for investors, with the UK tech sector attracting £6.3 billion in venture capital investment in 2018.”
(http://www3.weforum.org/docs/WEF_The_Impact_of_5G_Report.pdf).
5G provides a very low latency network that can support a large number of connected devices without degradation in network performance. The 5G landscaping undertaken by Create Converge in partnership with Dundee City Council (lead), Scottish Futures Trust and Abertay University has Identified key sectors within the creative sector and beyond. Within the creative and cultural sectors, there are opportunities in eSports, virtual production, tourism and play, where the games-film crossover together with high connection speeds will support multi-person visually rich experiences over mobile connection.
Beyond the creative and cultural sectors, 5G’s capacity to connect many, many IoT devices that can be monitored in real-time is key. This capacity can be combined with games engines to implement high-speed, high-resolution digital twins in manufacturing, smart ports and agritech. The role of game engines, much like in computer games, can be to integrate components into a connected whole that can be explored and interacted with visually to provide real-time information in an accessible form, to inform decision making and when connected to predictive models to predict future scenarios.
Next Steps
Games technologies are potentially powerful enablers of real-time interactive experiences. The successful deployment of these technologies depends on co-design, iterative development and effective evaluation.
Co-design
Co-design is crucial to working with games technologies. In our experience it is crucial that both games experts and the problem owners work together to devise a games-based solution. One of the key challenges is communication. Partly this is down to the gaps among the disciplines involved – game development itself requires a team with expertise in art, audio, design, programming and production and this team must then interact with the discipline(s) of the problem space. Co-design puts communication at its core and centres on the unpacking of the problem and the formulation of the solution.
Iterative development
Often, the co-design process generates more questions than answers, at least initially. It can take time for the real problem, and so the real solution, to surface. Iterative development is a valuable process, where the problem and solution are steered through a progressively detailed and focused series of prototypes. Iterative development can reveal misunderstandings of the problem space and possible solutions. Regarding the problem space iterative development serves as an ideal platform for co-design. Iterative design can also uncover and then resolve misunderstandings around game technology capabilities, where certain functioning can be far more complex (or indeed much simpler) than first appears.
Effective evaluation
Finally, as with any development activity it is important to establish what success looks like. Initial communication challenges and the typically uncertain nature of the solution can make it difficult to define success from the outset. Best practice is to revisit, but not necessarily relax, success criteria through that co-design process. In this way, success criteria then reflect the understanding gained as the design and its implementation evolve.
Contact Create Converge
If you would like to explore the use of games technologies in your industry then please get in touch through the Create Converge website and we can discuss pathways forward. These pathways include working with Create Converge partners directly or we can support access to industry partners.