Background research in AR object detection

Smart Garbage Visual Detection, Monitoring and Analytics – The MANGUSTA

Garbage and generic waste management is a challenging task in modern cities. Every area has its peculiar waste production pattern in terms of kind and volume of produced waste, and optimizing collection is key to reduce costs and ensure at the same time that city decor is always maintained.For some cities, this task is made even more difficult due to the impossibility of installing underground containers. This is the case of Amsterdam, where in most part of the city center, garbage collection relies on citizens and tourists to drop trash bags at given collection spots, at given hours (twice a week). In this case, it is of course vital to optimize the collection process and to minimize the amount of trash bags accumulating at any of these spots.Many projects that aim to solve this problem involve some form of sensors to be scattered through the city, which would be responsible to collect data about garbage distribution (IoT-style). We find this approach expensive, both for installation and maintenance, not at all scalable and not environmental-friendly. The solution to environmental problems cannot be to produce and scatter even more disposable electronics all over a city. –> More info on: https://becominghuman.ai/smart-garbage-visual-detection-monitoring-and-analytics-a0061fff2b76

YOLO – AI object detection

You only look once (YOLO) is a state-of-the-art, real-time object detection system. On a Pascal Titan X it processes images at 30 FPS and has a mAP of 57.9% on COCO test-dev.

How It Works

Prior detection systems repurpose classifiers or localizers to perform detection. They apply the model to an image at multiple locations and scales. High scoring regions of the image are considered detections.

We use a totally different approach. We apply a single neural network to the full image. This network divides the image into regions and predicts bounding boxes and probabilities for each region. These bounding boxes are weighted by the predicted probabilities.

https://pjreddie.com/darknet/yolo/

Other object detection AI with AR

1) ARkit: Record spatial features of real-world objects, then use the results to find those objects in the user’s environment and trigger AR content

In iOS 12, you can create such AR experiences by enabling object detection in ARKit: Your app provides reference objects, which encode three-dimensional spatial features of known real-world objects, and ARKit tells your app when and where it detects the corresponding real-world objects during an AR session.

https://developer.apple.com/documentation/arkit/scanning_and_detecting_3d_objects

2)Spark AR: uses target market like vuforia 

https://sparkar.facebook.com/ar-studio/learn/articles/world-effects/target-markers#Adding-the-target-marker

3) Vuforia: object recognition

Object Recognition allows you to detect and track intricate 3D objects, in particular toys (such as action figures and vehicles) and other smaller consumer products. Use the Object Scanner and the accompanying object target scanning image to easily scan your detailed toys, models, and educational tools.  

Object Targets should be viewed indoors under moderately bright and diffuse lighting. To the extent possible, the surfaces of the object should be evenly lit and not contain shadows caused by other objects or people. This should also be accounted for when scanning the object.

For Object Recognition to work well, the physical object should be:

  • Opaque, rigid and contain none or only very few moving parts. 
  • The surface of the object should have a large number of contrast-based features and rich texture. 

https://library.vuforia.com/features/objects/object-reco.html

Conclusion:

Ar and object detection requires AI and an algorithm to be trained by millions of trash images (machine learning + python). Therefore it would be best to stick to 3D design of different types of trash and adding them to the experience like we did last semester

Week 3 – Finalized our project adaptation – xxxHolic opening theme

In a team with Inga, Adrika from MA VR and Rosie Spencer from Sound Arts

Started of initial brainstorming of what we are going to adapt. 

We are on a dilemma on whether we want to adapt a movie poster or a movie opening theme

We have finalized our top choice to A VR adaptation of an opening sequence from xxxHolic animation series. The adaptation is supposed to act as a advertisement for the series.

Inspo scenes from the actual opening theme of xxxHolic:

xxxHolic opening theme: https://www.youtube.com/watch?v=qXDoJxAdwjQ

Bibliography:

xxxHolic (2006) Directed by T. Mizushima. [animation series] Japan: Production I.G.

Week 4 Collaborative Unit updates:

  • Added Rosie Spencer (sound arts student) in our group chat and briefed her on our project.
  • Named our team xxxWorkaHolic
  • Created a variety of boards on Trello
    1. IDEAS board
      1. Initial Ideas
      2. xxxHolic Ideas
      3. Storyboard ideas
      4. Any other ideas that may occur are added to this board
    2. To Do list
      1. All general todo tasks
      2. All big things done
    3. Classic Kanban for technical stuff
      1. Each of us has their own: 
        1. To Do
        2. Doing
        3. Done
    4. Timeline
      1. Weekly timeline general milestones
  • Held a zoomcall where each of us presented their storyboard
  • Created a Google doc where we added all of our Storyboard ideas and the goal is to figure out the final scene order based on each of our storyboard scenes by next Monday

Photos of zoom call:

Gallery of photos of our storyboards and our further inspo:

Future To do:

Combine our storyboards / Finalize scene order

Thoroughly brief Rosie Spencer and figure out how a sound arts students can enhance our project and vice versa – assign roles

Initial personal Storyboard for xxxHolic

This is a Gallery of certain parts of my initial storyboard

We then met as a group for the purpose of combining each of our individual storyboards and finalising the scene order

All the scenes in order in paper + inspo
Screenshot of scene 1 of storyboard made in tiltbrush
Screenshot of scene 1 of storyboard made in tiltbrush
Screenshot of scene 4 of storyboard made in Tiltbrush
Screenshot of scene 4 of storyboard made in Tiltbrush

Questionnaire #1 responses

Additional comments from curators when asked about tips on how to highlight works we are exhibiting in a potential VR gallery:

  1. Virtual galleries can be more accessible to the public and therefore allow an audience not usually found within traditional gallery curation, so using multi-media platforms can assist in highlighting works! Also I feel it completely depends on whether you plan on making a profit from the exhibition, because that will change how you advertise and curate the works as you’ll attract different audiences. This idea is so interesting, and I feel it would be very easy to highlight art using recycled products in the current environment, but I think it would do better if the whole project was sustainable, so by being virtual you can try creat no footprint and by making it free and easily accessible it will be more of a statement that agrees with the pieces themselves, but I don’t know how economically viable that would be for you as a curator!
  2. I had a hard time answering the question related to the intention of art curating these days, because I feel like the industry (and therefore the practice) is very polarized. Although mega galleries are on the rise, smaller, more independent collectives, galleries, museums, projects are also emerging. They seem to be more conscious morally, socially, and environmentally. They are also in touch with AR/VR better. Transfer Gallery would be a great example. One thing I found interesting this year is how the Art Review completely changed up it’s “Most Powerful” list and replaced gallerists with academics, celebrity artists with politically minded collectives. So a central question would be what can curating and tech can offer to promote this shift? I think VR is a good answer bc it allows you to experience “the other.” There are thousands of articles written about VR in the context of environmental studies, gender studies, disability studies. They could be a useful guides.

Questionnaire #1 on Digital Art Curation and Galleries Today

This week I set up and distributed the first questionnaire targeted to art curators or people who have or have had stake in that sector. The purpose of this questionnaire is to get an initial feeling of experts’ opinions on art galleries nowadays and on possibly future digital art galleries

Oliver Grau – From Illusion To Immersion / inspiration & research

[A brief history of 360 illusion of space ]

“We are experiencing the rise of the computer-generated, virtual spatial image to image per se, to images that appear capable of autonomous change and formulating a lifelike, all embracing visual and sensory sphere.”

“Media Artists represent a new type of artist who not only sounds out the aesthetic potential of advanced methods of creating images and formulates new options of perception and artistic positions in this media revolution, but also specifically researches innovative forms of interaction and interface design, thus contributing to the development of the medium in key areas, both as artists and as scientists.”

Art and science together.

How virtual art fits into the art history of illusion and immersion

Virtual image spaces -> their historical genesis throughout stages of Western history:

Fresco rooms

Illusion Spaces. E.g. Panorama 1789, paul Sandby landscape room 1793

Circular cinema

Computer art. E.g. vR, CAVE room

Illusion Spaces – The Panorama:

European Tradition of image spaces of illusion.

Panorama, Robert barker 1789
Shows how in this epoch extraordinary efforts were made to produce maximum illusion with the technical means at hand
https://www.arromanches360.com/visit/360deg-cinema
Located on the Arromanches clifftops, the Arromanches 360 Circular cinema dominates the remnants of one of the two artificial Mulberry harbours set up by the Allies.
A Cave Automatic Virtual Environment (better known by the recursive acronym CAVE) is an immersive virtual reality environment where
projectors are directed to between three and six of the walls of a room-sized cube.

Moodboard (s) – Initial research on the theme of trash in art

https://www.youtube.com/watch?v=wthTmQHmuZ0&feature=share 50’ to save the world video
https://nanocrit.com/issues/issue7/trash-trash-art
Trash has had a place in avant-garde art since the early twentieth century, when Marcel Duchamp introduced the idea of the readymade: any slightly modified, often discarded, manufactured object selected and displayed as an art work. From 1959, Gustav Metzger began writing manifestos for what he called “auto-destructive art,” art that resulted from its own destruction, an anti-capitalist, anti-consumerist public art form committed to social and political justice (Wilson 143). For many art historians and cultural critics, the subject of trash and art is associated with discrete art objects or installations made from waste material sourced from junk shops or the street that are transformed into works of art through their alteration and presentation in a gallery, where anything can become a commodity.
One perspective on art and trash that considers these questions can be found in the work of the social anthropologist Michael Thompson. Writing in the late 1970s, he addressed the status of art and rubbish in relation to production and consumption. His categories of “durables” (valued objects held in museums), “rubbish” (discarded objects), and “transients” (objects in circulation that are not yet classified as either durables or rubbish) identified different states of value granted to objects, states that were changeable and dynamic and could be created and destroyed.
Artists present trash as trash in very different ways. Dieter Roth archived his daily waste in his installation Flat Waste (1975-76/1992), a work which operates as both diary and self-portrait, and which offers the viewer the opportunity to delve into a library of his personal trash. Over the course of a year, Roth saved all of his waste material on a daily basis, flattening or folding it in order to place it all in transparent sleeves in ring binders chronologically filed on shelves. (Roth first made Flat Waste over the period of a year in 1975-6, but some of the volumes lost from the original year were replaced with binders of rubbish collected on the same day in later years.) Between every specially designed shelving unit, a lectern enables viewers to look at a selection of the ring binders–no waste escaped his collection: used toilet paper, toothpaste, train tickets and cigarette butts are all there. Roth’s preservation of decaying waste conveys to the viewer a sense of the scale of one individual’s everyday waste over a single year, and its ultimate fate as excess.
https://nanocrit.com/issues/issue7/trash-trash-art
https://nanocrit.com/issues/issue7/trash-trash-art
 “to think about what they don’t want to think about” because disposal breaks the connection between object and consumer, vanquishing waste from the consumer’s mind by making it appear to disappear. 
Another more recent work that brings together a unique chronicling of waste, the process of decay, and more recent forms of environmental art can be found in Joshua Sofaer’s The Rubbish Collection at the Science Museum, London (16 June–14 September 2014), part of the museum’s Climate Changing program.
https://nanocrit.com/issues/issue7/trash-trash-art
One iconic historic reference to this idea is the Daily Mirror’s 1976 headline, “WHAT A LOAD OF RUBBISH,” an attempt to capture the public outrage at The Tate Gallery’s (now Tate Britain’s) purchase of Carl Andre’s Equivalent VIII (1966), a purchase funded by British taxpayers during an economic slump. 
 The headline labeling the work as rubbish played with the idea that it was both trash and trashy, even though it was made out of new materials—firebricks—and nothing else. Recently turned into a painting by the collective Claire Fontaine, this iconic headline set the stage for establishing Equivalent VIII, which came to be known as “The Bricks,” as the epitome of “rubbish,” low quality contemporary art.

Assets I got for DIE experience – Referencing

Unit 2 Title and artist name of models I used for my story board for my short story for unit 2
Unit 2 Title and artist name of models I used for my story board for my short story for unit 2
Unit 2 Title and artist name of models I used for my story board for my short story for unit 2
An additional source of a model for the DIE experience / Unit 2 / Morty and the Magic Bananas / Model from Google poly
DIE extra model used in experience – sourcing / UNIT 2

StoryBoard Process

Initial StoryBoard Making in Tiltbrush & moviemaker app

unit 2
Unit 2 – Storyboarding in Celtx
(in the new storyboard the scenes are 7)

My headset broke so I could not edit the story board I had made in Tiltbrush so I made a new storyboard in unity

New Story Board making (post dead headset)

DIE
DIE – Environment Layout – Flowchart
A diagram that represents the user journey of the experience – Based on the Flow Diagram
DIE / The making of scene 1&2 in act 1
Morty and magic bananas unit 2/ act 2 Scene 3,4,5 – Morty’s Dream

FINAL VERSION OF STORY BOARD IN UNITY:

ACT 1

night scene
scene1,2

ACT 2

DREAM scene
Scene 3,4,5

ACT 3

Day Scene scene scenes 6,7