{"id":2262,"date":"2015-02-10T23:50:20","date_gmt":"2015-02-11T04:50:20","guid":{"rendered":"http:\/\/www.jhclaura.com\/?p=2262"},"modified":"2015-11-20T17:10:16","modified_gmt":"2015-11-20T22:10:16","slug":"experiment_01","status":"publish","type":"post","link":"http:\/\/www.jhclaura.com\/experiment_01\/","title":{"rendered":"Experiment_01"},"content":{"rendered":"<h1>Camera Feed<\/h1>\n<p>For my first experiment, in order to create strong <em>nonsense moment<\/em>, I want to connect virtual world with\u00a0the reality, so as\u00a0to play with perception and make\u00a0conflict, and one way to do this is importing\u00a0the camera feed from the mobile phone. This way, the user can not only be able to <strong>see the real world<\/strong>, but also <strong>experience illogical, contradictory events in\u00a0the virtual world<\/strong>, which is triggered by the real world.<\/p>\n<h3>Possible scenario<\/h3>\n<p><a href=\"http:\/\/itp.io\/thesis\/jhc551\/wp-content\/uploads\/sites\/617\/2015\/02\/scenario_01a.png\"><img loading=\"lazy\" class=\"alignnone size-full wp-image-127\" src=\"http:\/\/itp.io\/thesis\/jhc551\/wp-content\/uploads\/sites\/617\/2015\/02\/scenario_01a.png\" alt=\"scenario_01a\" width=\"1000\" height=\"271\" \/><\/a>&#8211;&gt; Contradict to reality<a href=\"http:\/\/itp.io\/thesis\/jhc551\/wp-content\/uploads\/sites\/617\/2015\/02\/scenario_01b.png\"><img loading=\"lazy\" class=\"alignnone size-full wp-image-128\" src=\"http:\/\/itp.io\/thesis\/jhc551\/wp-content\/uploads\/sites\/617\/2015\/02\/scenario_01b.png\" alt=\"scenario_01b\" width=\"995\" height=\"276\" \/><\/a>\u00a0&#8211;&gt; Think everyone is monkey<a href=\"http:\/\/itp.io\/thesis\/jhc551\/wp-content\/uploads\/sites\/617\/2015\/02\/scenario_01c.png\"><img loading=\"lazy\" class=\"alignnone size-full wp-image-129\" src=\"http:\/\/itp.io\/thesis\/jhc551\/wp-content\/uploads\/sites\/617\/2015\/02\/scenario_01c.png\" alt=\"scenario_01c\" width=\"1000\" height=\"276\" \/><\/a>\u00a0&#8211;&gt; Encourage to say Hi<a href=\"http:\/\/itp.io\/thesis\/jhc551\/wp-content\/uploads\/sites\/617\/2015\/02\/scenario_01d.png\"><img loading=\"lazy\" class=\"alignnone size-full wp-image-130\" src=\"http:\/\/itp.io\/thesis\/jhc551\/wp-content\/uploads\/sites\/617\/2015\/02\/scenario_01d.png\" alt=\"scenario_01d\" width=\"1000\" height=\"254\" \/><\/a>&#8211;&gt; Focus enhancement<\/p>\n<p>&nbsp;<\/p>\n<h3>Real + Virtual Mashup<\/h3>\n<p><a href=\"http:\/\/itp.io\/thesis\/jhc551\/wp-content\/uploads\/sites\/617\/2015\/02\/camFeed.png\"><img loading=\"lazy\" class=\"alignnone size-full wp-image-86\" src=\"http:\/\/itp.io\/thesis\/jhc551\/wp-content\/uploads\/sites\/617\/2015\/02\/camFeed.png\" alt=\"camFeed\" width=\"1000\" height=\"563\" \/><\/a><\/p>\n<p>After talking with professor Shawn Van Every, I decided to use browser instead of App as platform first, and test the limitation of browser and Javascript.\u00a0With flexible\u00a0HTML5 and Chrome browser, I can get camera feed from mobile just like from webcam of laptop, and\u00a0<strong>getUserMedia<\/strong> &amp;\u00a0<strong>MediaStreamTrack<\/strong> function of <strong>WebRTC<\/strong> API allow users to choose camera and set up constraints as they want. Below are the gists of it:<\/p>\n<ul>\n<li>Get the user\/rear camera into video<\/li>\n<li>Put into canvas; use the canvas as texture for plane geometry<\/li>\n<li>Translate the position of geometry but keep the center of the plane mesh (geometry + material = mesh) on center<\/li>\n<li>Rotate the mesh as rotating the camera (== user&#8217;s head rotation)<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<h3>Computer Vision on Phone<\/h3>\n<p><a href=\"http:\/\/itp.io\/thesis\/jhc551\/wp-content\/uploads\/sites\/617\/2015\/02\/camFeed_cv.png\"><img loading=\"lazy\" class=\"alignnone size-full wp-image-133\" src=\"http:\/\/itp.io\/thesis\/jhc551\/wp-content\/uploads\/sites\/617\/2015\/02\/camFeed_cv.png\" alt=\"camFeed_cv\" width=\"1000\" height=\"580\" \/><\/a><\/p>\n<p>Nonsense is built on sense, so in order to make nonsense in virtual world, the V\u00a0World\u00a0needs to know what&#8217;s going on in the real world, and my first attempt is using computer vision to analyze the image captured from camera. Below are the computer vision JS libraries I found:<\/p>\n<ul>\n<li>https:\/\/github.com\/inspirit\/jsfeat<\/li>\n<li>http:\/\/trackingjs.com\/<\/li>\n<li>https:\/\/github.com\/auduno\/clmtrackr (face)<\/li>\n<li>https:\/\/github.com\/auduno\/headtrackr (good for face)<\/li>\n<li>https:\/\/github.com\/sightmachine\/simplecv-js<\/li>\n<li>https:\/\/github.com\/peterbraden\/node-opencv<\/li>\n<li>https:\/\/cloudcv.io\/<\/li>\n<\/ul>\n<p><strong>Issue #1<\/strong> &#8211;\u00a0Currently I used jsfeat to grayscale the footage first, and then found the bright area pixels by pixels. It&#8217;s obviously slow. Next step will be trying the combination of OpenCV and Node.js (Thanks to Pedro), see if &#8220;perform CPU-intense image processing routines in the cloud, let Node.js server handle client requests and call C++ back-end&#8221; will optimize the performance or not.<\/p>\n<p><strong>Issue #2<\/strong> &#8211; Have to figure out how to translate the pixel location from canvas to 3D world, since let eyeScreen rotating with camera (head) making everything complicated.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Camera Feed For my first experiment, in order to create strong nonsense moment, I want to connect virtual world with\u00a0the reality, so as\u00a0to play with perception and make\u00a0conflict, and one&#8230; <a class=\"read-more\" href=\"http:\/\/www.jhclaura.com\/experiment_01\/\">Read The Rest &rarr;<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[244],"tags":[249,250],"_links":{"self":[{"href":"http:\/\/www.jhclaura.com\/wp-json\/wp\/v2\/posts\/2262"}],"collection":[{"href":"http:\/\/www.jhclaura.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.jhclaura.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.jhclaura.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.jhclaura.com\/wp-json\/wp\/v2\/comments?post=2262"}],"version-history":[{"count":1,"href":"http:\/\/www.jhclaura.com\/wp-json\/wp\/v2\/posts\/2262\/revisions"}],"predecessor-version":[{"id":2263,"href":"http:\/\/www.jhclaura.com\/wp-json\/wp\/v2\/posts\/2262\/revisions\/2263"}],"wp:attachment":[{"href":"http:\/\/www.jhclaura.com\/wp-json\/wp\/v2\/media?parent=2262"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.jhclaura.com\/wp-json\/wp\/v2\/categories?post=2262"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.jhclaura.com\/wp-json\/wp\/v2\/tags?post=2262"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}