Research_03

After finalizing the subject as MASK, the research needs to be updated too.

Will be updated periodically…

How is a mask to be used traditionally?

How is a mask to be used untraditionally?

– Frank Sidebottom; movie to watch Frank 

How to fool the perception, to self and to others?

• Qs to answer
  • “What consciousness is?”
    • American philosopher Thomas Nagel argued that, even if you were somehow able to transport yourself into the mind of a bat, you still wouldn’t really know what it’s like to be a bat; you would only have the experience of being a person inside the mind of one. Nagel’s arguments rest on the fact that we don’t yet fully understand what consciousness is, and perhaps we never will.

Techhhh

• Code stuff
  • Computer vision, HTML5, Three.js, WebRTC
• Physical stuff:
  • Fabrication: cardboard, paper, paper mache
  • Arduino
• 2/25, got advice from Alex Kauffmann, who works on the Cardboard in Google!
  • The distance between lenses and phone screen is fixed (in the current version, somewhere in the vicinity of 44mm). Provided they respect the fixed focal distance, you should be fine. Velcro is your friend. So is hot glue.
  • For interaction, I recommend not having straps—make people hold the mask as they wear it, it avoids all sorts of latency problems (your torso rotates much slower than your neck, and when you’re holding the mask, you can only turn your head as fast as your torso). Also, don’t do motion in more than one direction (if there is motion at all) and don’t accelerate and decelerate (it’s a recipe for vomit). In virtual reality, there is no friction, so no need to simulate it.
  • In terms of software, I would steer you either towards WebVR (here’s a good overview with relevant links) or the Unity plugin for the Cardboard SDK. They take care of most of the difficult optical things so you can focus on just building your 3D environments.

Experiment_04

Computer vision

• Talked with Kyle McDonald
  • My project reminded him of anonymous hugging
  • Suggested testing different ways to max the performance, setting benchmark and see which part costs the most
• Face detection
  • Use small canvas for executing face detection, display the image with big size of canvas.
  • From below test pics you can see, it’s much better to analyze with smaller canvas, but there’s no big difference between displaying big and small canvas, so for better resolution, it seems ok to display bigger canvas.

bigCanvasAnalyze

bigCanvasDisplay

smallCanvasDisplay

Greyscale + Blur

multiple faces detection, yet not stable due to changing lightness

• Collect personal moment as taking photos –> accompany with you all the time to comfort you to confront unknown
• Affecting virtual world

Wireless Wooo

Thanks to Andy‘s advice on hooking up localhost of my laptop through itpsandbox wifi, now I can run codes on mobile phone using laptop as server! (ps. It also works at home, just in NYU because of the security issue, using ITPSANDBOX is needed.)

*Note* It’s not advised to run server elsewhere (e.g. heroku, digitalOcean) because it takes more time to transfer the data back and forth. Localhost with laptop is the best option for proof of concept!

Paper Mache

–> Decide to do it after finalizing the virtual content.

MASK_01 – Control Room

First prototype (or maybe not just a prototype…) of MASK collection – Control Room.

Story

It’s the kind of fantasy castle that a child builds up and role-play inside. The interior is rough and the space is tiny, but with the protection and imaginary friends, you feel less timid to the outside world.

Features

1. It’s a townhouse cardboard mask
2. User is virtually in a cardboard room, and sees the real world through different windows
3. The distance between self and the world is created, thus decreased the threatening feeling
4. When you stay still, imaginary friends start to encourage you

Things I learned

1. For improving performance of phone, reducing face amount
2. Detailed model –> NO, simple yet interesting shape –> YES
3. Texture’s effect is more powerful than model
4. …

Content test:

Imagination friend test

Fabrication process:

Failure

• Normal, discarded cardboard is too thick… have to buy a good thin quality cardboard.

• Scale up for eyeglasses. Too big unfortunately. Width between them will be perfect. One good thing is although the mat board is a little bit heavier, it’s not that big difference.

Experiment_03

Control Room

• The first mask I’m going to make is Control Room. The user is virtually in a small room and sees the world through different windows. It creates a feeling of controlling a robot, which is the physical form of self. The distance between self and the world is created, thus decreased the threatening feeling from outside world.
• It’s also a space for your imaginative friends.
• Possible purposes
  • Although hiding in control room, the user still hope to interact with others. Inside control room is just a way to deal with the social awkwardness
• Appearance: Townhouse
• Verb
  • User
    • Lives daily life, sees through different windows (passive v.)
    • Interacts with imaginative friends (active v.)
      • What kinds of interaction?
  • Others
    • Have few interactions with the user in general
    • Windows and doors invited to be knocked to start conversations –> not like small talk but have certain level of efforts
• Features to have: multi-camera input, prosthetic limbs,

And here’s a video explaining what’s going on.

Touch Event

• Copper tape extension
• For other to trigger sounds, as part of the “An Afternoon Under A Mask” experiment

https://vimeo.com/120315575

But then I found out it’s super finicky if I use multiple ones and extend them T_T

https://vimeo.com/120407665

An Afternoon Under A Mask

– How does it feel to live daily life under a mask exactly? What’s the reaction of others? What do I hope to happen? I’m going to try to answer those questions on Monday(2/23).

Couldn’t do it because the copper tapes are too finicky..

WTF Moment_01

Last night I found out that the idea that “Wearing mask = isolating yourself” is totally wrong. If I really want to isolate myself, I can just escape and hide myself in an empty room. On the moment that I put on the mask, I choose to stay in the environment and deal with it. Wearing mask is not isolating myself. It’s a way to interact with others relaxedly.

WTF.

I spent so much time on thinking about examples based on the “mask = isolate” rule, which is wrong.

WTF.

Oh well. At least I figured it out. Yay.

MASK fantasy

The construction of this MASK fantasy is

1. Purpose of the mask.
2. What does it do?
3. What’s the effect of it to user?
4. What’s the effect of it to the other?

And the symbol ▲ stands for under the category of SHELTER, and ▼ for IDENTITY SHIFT, and ◎ for THE SELECTED (probably).

Temporary result: ▲*5, ▼*6,  ◎*3 (▲*2, ▼*1 )

Here we go!

Block annoying person (ref. tv series Black Mirror) ▲
– Blur face, muffle sound / music volume goes along with the volume of environment noise
– Ignore the person without being noticed, decrease interference
– Able to finish the unaware annoying conversation, maybe irritated because the user is wearing a mask

“I wonder what he/she is doing now?” –> { Duet } ▼
– Geolocation as blinking sphere in virtual world
– Sense of secure, feel the existence of someone under Spatial awareness
– Under the agreement, the participant might feel being watched and surveillance
&
– Change perspective of the other when you want to, shorten the distance
– Less paranoid, less nostalgic, closer relationship
– People will know the user is missing someone

Not be afraid to talk to people/be social –> { Control Room } ▲ ◎
– Virtually in a tiny room, and see world through a window on the wall, like controlling a robot (= your physical body)
– To improve social ability, the wall next to the window having top 5 news of the day for conversation start point
– Can have multiple windows on the wall, and each windows show different views of the world
– The distance between self and the world is created, thus decreased the threatening feeling.
– This is a funny guy. He/she seems know what to say. ???

Be an awesome listener –> { Awesome Listener } ▲ ◎
– Mask speaks for the user with single note yet different tune sound to represent the emotions e.g. “Oh” expresses “Yes”, “No”, “What?”, “Alright”, “Really?”, “Hah”. And the mask is made of mirror
– The user can keep on the conversation nonverbally
– The other has to guess what the user means by listening carefully to the single note sound, and analyzes to comprehend. Might be annoyed because of limited feedback
– Because of the reflection in mirror, and the universal sound as feedback, the other feels like talking to his/her subconscious. The other might have narcissistic moment during the conversations, thus will be attracted by the user

Stop comparing yourself to other people ▼
– Turn everyone “you”, replacing the head image
– Feel like talk to yourself, surprised when “you” say/do something unexpected, exploring the possibilities that you might have those hidden characters as “you”
– The other might has a “murmuring” conversation because the user is talking to “him/herself”

Provide protection from ▲
– Propaganda (of gov policy, commercial)
– Mainstream (fashion e.g. everyone wears black in NYC, brand, music, slang)
– Peer pressure
– For user, easier to maintain who he/she is
– For other, the user seems very unique and weird

Avoid judgement ▲ ◎
– Flash texts or audios to compliment the user, and the mask is made of mirror
– User’s confidence will be boosted up
– The other feels weird to talk to him/herself, might have narcissistic moment during the conversations, thus the other will be attracted by the user

Crave to be liked(as “Like” button of Facebook, opposite to block ppl) –> { Like Me } ▼ ◎
– Buttons on the mask inviting others to press, the user’s vision gets clearer once buttons being pressed, or it’ll gradually blur
– The user has to create conversations and interact with people in order to get clear vision
– The other has incentive to enter the conversation (because of lovely buttons!) and has fun

Story sharing anonymously –> { Anonymous Story  } ▼
– With virtual recording button, user upload the recorded vision and audio, just as writing blog posts. It could be a diary, an instruction, or a fiction of daily life.
– The user creates movies about the user self, effortlessly. And since it’s a POV, point of view shot, the user expresses self anonymously
– It’s an unconditional way to comprehend other by literally taking non-self perspective.

Be animalized to avoid having human emotions ▼
– Provide powers of different animals, flying vision with air fan, hear only small sound, see yourself with mirror rig
– The user express him/herself as the animal, freely. No moral burden as a human
– The other will be surprised by the unusual ability of the user. Might be scared. Might ask to use the user’s superpower

Escape from reality –> { Evasive Wander } ▲
– In virtual world, real world is just a standing screen in a forest, and there are other masked creatures wandering around
– User can isolate self completely in a second, and communicate with other masked creatures. Identify self as a group
– The other might feels the existence of two world, and one of which belongs to the masked people. Difference might be widen, yet the other might want to join the virtual world as well

Train to be used to possible future distorted perception ▼
– Providing experience of Alice in Wonderland Syndrome
– Different kind of distortions let user to externalize the perception. Under the assumption that what he/she sees is not real, the user can repress instinct reaction and makes rational judgement.
– The unusual calmness of the user might terrify the other. Or the other might consult the user’s…

Experiment_02

Mask Iteration

– doodle of mask ideas

– 3D models built in Maya

– Pattern unfold by Pepakura

– Prototyping with paper

– Prototyping with fabric

– Mask iteration display –> Mask Gang

From left to right, dirty design based on the different functions:

• Spikes – being inhuman, scary and aggressive
• ChickenCow – ???
• Box – attached to body, so the mask is hard to take off
• Ears – the sound is magnified by funnels sticking out near the ears,  thus user becomes more aware of the environment around

Virtual World Construction

– doodle of world: 1) Closed, 2) Open

– Coding coding coding….

Production Schedule

Research_02

For deeper thesis research, I first looked more into Nonsense, then on the format – Helmet, and discovered some traces and had some thoughts:

• In this loud, information overloaded, rapid speedy world, people might use nonsense as a way to stand out and be noticeable, thus created this nonsense everywhere phenomenon.
• With a contrary character of hiding the face and being conspicuous, helmet might represent the contradiction/dynamic equilibrium between introversion and extroversion.
• Helmets could be utilized to bring out the good old value of nonsense in current society.

Deeper into Nonsense  

—Different kinds of nonsense in history. Why did they arise? What are they about? What’s the nonsense now? Nonsense used to be able to shock people, yet now it is everywhere. What’s the impact of it currently compared to in the past? Does nonsense still have value?

{ Past }

Cubism

• 1907, Picasso
• instead of depicting objects from one viewpoint, the artist depicts the subject from a multitude of viewpoints
• unhistorical deductions

Dada

• started around 1910
• thinks the logic and reason of bourgeois caused the WWI, so express the rejection by embracing chaos and illogic
• anti-war, anti-art
• a systematic work of destruction and demoralization

Surrealism

• 1920’s
• resolve the previously contradictory conditions of dream and reality
• pure psychic automatism, expressing the subconscious
• Freud’s work with free association, dream analysis, and the unconscious
• aimed to revolutionize human experience

Fluxus

• 1960’s
• blending different artistic media and disciplines
• anti-commercialism and an anti-art sensibility

{ Now }

Frankly, I can’t define what’s the Nonsense is nowaday. But below are what I found related:

Arty Narcissism

“…pretentious, meaningless, tongue twisting nonsense spoken in some artists’ statements. The culprits are mainly the conceptual artists who need to bamboozle you with an attempt at philosophical genius in order to camouflage the insignificance of the poorly executed end product.“

“The contemporary market has been subject to extreme speculation, but there is a difference between speculation and fraud. The economic downturn will moderate the speculative bubble that has seen some less deserving artists command extreme prices.”

Justwalkingism

“A game that you just walk in. It’s not about winning, it’s not about overcoming the challenges, it’s not even about challenges anymore… Justwalkingism is giving you something to go through and what is important is that you actually go through it. You.”

“Justwalkingism is an art movement, meaning that its artists express themselves on many platforms. It’s like colouring books: the outlines are there but the finished piece is still yours to make.”

Helmet/Mask – Interconnection between Nonsense and Introversion

Introvert vs Extrovert (by Carl Jung)

• Introversion: an attitude-type characterised by orientation in life through subjective psychic contents (focus on one’s inner psychic activity)
• Extraversion: an attitude type characterised by concentration of interest on the external object (the outside world)

Mask

• The existence is the clashing of introversion and extroversion
• Identify/Mindset changing
• A shelter / lense / armor to approach the physical world

Some notes/thoughts after reading The Power of Introverts in a World That Can’t Stop Talking

• We have the ability to be whomever we want –> power of free will.
• Because I consider myself as an introvert, be careful DON’T beautify introversion.
• Solitude can be a catalyst to innovation? What about work in a group?
• If introvert and extrovert do complete each others, a person can be a “complete” one, by shifting between introvert and extrovert?

Experiment_01

Camera Feed

For my first experiment, in order to create strong nonsense moment, I want to connect virtual world with the reality, so as to play with perception and make conflict, and one way to do this is importing the camera feed from the mobile phone. This way, the user can not only be able to see the real world, but also experience illogical, contradictory events in the virtual world, which is triggered by the real world.

Possible scenario

–> Contradict to reality –> Think everyone is monkey –> Encourage to say Hi–> Focus enhancement

Real + Virtual Mashup

After talking with professor Shawn Van Every, I decided to use browser instead of App as platform first, and test the limitation of browser and Javascript. With flexible HTML5 and Chrome browser, I can get camera feed from mobile just like from webcam of laptop, and getUserMedia & MediaStreamTrack function of WebRTC API allow users to choose camera and set up constraints as they want. Below are the gists of it:

• Get the user/rear camera into video
• Put into canvas; use the canvas as texture for plane geometry
• Translate the position of geometry but keep the center of the plane mesh (geometry + material = mesh) on center
• Rotate the mesh as rotating the camera (== user’s head rotation)

Computer Vision on Phone

Nonsense is built on sense, so in order to make nonsense in virtual world, the V World needs to know what’s going on in the real world, and my first attempt is using computer vision to analyze the image captured from camera. Below are the computer vision JS libraries I found:

• https://github.com/inspirit/jsfeat
• http://trackingjs.com/
• https://github.com/auduno/clmtrackr (face)
• https://github.com/auduno/headtrackr (good for face)
• https://github.com/sightmachine/simplecv-js
• https://github.com/peterbraden/node-opencv
• https://cloudcv.io/

Issue #1 – Currently I used jsfeat to grayscale the footage first, and then found the bright area pixels by pixels. It’s obviously slow. Next step will be trying the combination of OpenCV and Node.js (Thanks to Pedro), see if “perform CPU-intense image processing routines in the cloud, let Node.js server handle client requests and call C++ back-end” will optimize the performance or not.

Issue #2 – Have to figure out how to translate the pixel location from canvas to 3D world, since let eyeScreen rotating with camera (head) making everything complicated.