Running JavaScript in Grasshopper3D for Pose Estimation

‘Catwalk’ was a 1 day project that I started as a software replacement for the Kinect sensor in the Grasshopper3D environment.

After self-learning JavaScript and p5.js, I stumbled upon Dan Schiffman’s pose estimation tutorial. This idea reminded me of the typical kinect-firefly-grasshopper experiments, where the pose estimation would drive some kind of façade visualization in real-time. I was not in possession of a Kinect sensor or alternate hardware that would allow such experiments at the time and so I set out to determine whether p5.js/ml5.js could run INSIDE the Grasshopper3D environment — just for fun!

After some digging, I found a way to essentially run an instance of the Google Chrome browser (Chromium, to be precise) within Grasshopper by simply referencing an open source library — CefSharp. Once I had Chrome running in Grasshopper, I simply had to create a webpage that would exchange information with Grasshopper. This webpage ran a JS script, similar to Dan Schiffman’s pose estimation p5 sketch, but instead of displaying the pose in a p5.js canvas, it simply exposed the raw pose data through a JS function (CefSharp allows calling functions through it’s C# API).

I could then call the function from a Grasshopper component and it would return all the data in JSON format which could then be de-serialized to an object. Exporting the properties of this object in the component’s output parameter gave me a simple way to basically run JavaScript functions inside Grasshopper, in real-time.

With a simple script creating geometry using the output data, I arrived at a result that looked like this:

That’s my sister dancing. She insisted that I make the webcam feed unrecognizable, which is why it looks that way, but no camera filters were required to achieve this. The orange figures are screen captures of the Rhino viewport and the timing of both feeds is matched, so you can notice the latency (which seems surprisingly comparable to the latency of running a similar webpage on Chrome normally).

I packaged the plugin and put it up on Food4Rhino, but it seems to have a ton of issues when running on various system configurations. Feel free to try it out. If you have a specific idea you would like to try using this setup, we’re always looking for interesting projects to collaborate on at ARPM Design and Research.

I might expand on this idea of running JavaScript inside of Grasshopper or even Unity3D. I haven’t come across any other examples of this being done anywhere, so it could lead to some interesting results.