Rick Barraza

Thursday September 10, 2009

Rendering Vector Fields in Silverlight

Silverlight Advanced Rendering Techniques: Part 3

[ Download VectorFieldProject.zip ]

Now that we have explored how to render particles directly into a WriteableBitmap (see: Advanced Render Techniques with WriteableBitmaps), we can introduce the concept of a vector field and see how these two techniques work together.

Static versus Dynamic Vector Fields

A Vector Field is basically a map that stores information about how much change in the x and y direction is represented by any given point in a map. With a static vector field, you may calculate these values once (as we will do using a regular photo as our source) and simply use them as the velocity control of other moving elements. In a dynamic vector field, you may also choose to dissipate the actual Vector values in the grid over time using some algorithm (for example, using Jos Stam’s technique for “Real Time Fluid Dyamics”) to get a more liquid effect as each vector bleeds into the vectors around it.

However, for this introduction we will only be exploring static vector fields. Once you understand how a vector field works in general and how you can use one to animate moving particles, you may want to explore the more advanced references in the Fluid Dynamics post or follow the link above.

Step 1: Setting up a Photo Source

If you look at the code for the demo above, you will see that I’m defining a WriteableBitmap called bmpSource that will render whatever image is currently being shown in the SourceContainer canvas. This lets me switch out the photos I will use as my inputs easily by simply selecting which child photo I make visible. One other gotcha I ran into is that you need to wait until at least one of the source images is fully Opened (not just having the Image object Loaded). Once the ImageOpened() event has been fired, you can grab a reference to the photo in XAML, render it (or in this case, its visual parent) into a WriteableBitmap, and start accessing its pixel data directly.

Step 2: Converting the Photo to Greyscale

In the setupGreyscale() function, we loop through each pixel in bmpSource and calculate its greyscale value. We then draw this greyscale pixel into the WriteableBitmap called bmpGreyscale so we can display the result on screen.

Since Pixel color data is stored as a 32-bit integer in WriteableBitmaps, we usually need to extract the Alpha, Red, Green and Blue values represented in this one integer and break them into their individual byte values. This makes it easier to manipulate them or average them out to calculate their greyscale value as we're doing below:



You should also notice that since I already have the pixel value broken down into its color components, I go ahead and create a C# Color object and store it in the array sourceColors[] for later. This will come in handy in our render loop, and I’d rather calculate and store the values once then continuously perform the byte operations every time I need to color values in different formats.

Step 3: Edge Detection and Calculating Vectors

Now here comes the tricky part. The reason we converted the source photo into greyscale was to run an edge detection algorithm against the pixels. Here is the code:



There is a fantastic openFrameworks tutorial that describes the hows and whys of this technique and you can find it here http://wiki.openframeworks.cc/index.php?title=Making_VectorFields_from_images and it definitely served as both a guide and inspiration for this project in Siverlight. In summary, though, what we’re doing is this: For every non-edge pixel (starting at the second row and column), we are looking at the 3x3 grid of pixels that surround it to calculate a Point that represents the directional change at that point. In other words, we look at the 3 pixel row above and below our current pixel to assign a value to the amount of vertical change (the y value) influencing that pixel. We then look at the 3 pixel column to the left and right of our current pixel to assign a value to the amount of horizontal change ( the x value) influencing that pixel. In our code, we put these two directional change values into a Point and then store that Point into an array.

This array of points that analyzes the greyscale changes in X and Y for each pixel and converts it into a Point that represents directional data is our Vector Field.

While the Vector Field now stores directional change values for every point of the original photo, it would be too expensive (and conceptually challenging) to represent this directly. So for the static vector field display in our demo, we are taking a simplified approach which we are also using for our animated particle cloud. We create a simplified grid of particles (in our case, spaced 3 pixels apart) and query the Vector Field for the direction vector for only that particle's position. If the vector values for either x or y are greater than our threshold, we will drop that particle down and color it based on the original source photo. Using a variation of the line() function we’ve been using in the previous post, we then draw a line into bmpVectorField from our particle’s starting point to where it would be after applying its vector (see code below).

Step 4: Use Vector Field for Particle Control

Up until now, we have used a photo as our source data, duplicated it into a greyscale image to calculate its edges, and converted this greyscale variance for each pixel into a vector field that represents directional change for any point we query. On top of this, we have created a simplified grid of particles for those vectors whose values are higher than our given threshold values and then we've drawn these velocities into a static image.



So to create the particle cloud, we simply animate this collection of particles along their vectors over time in an animation loop (instead of all at once as we did in bmpVectorField) and add any extra special effects we can think of.

We’re using the same fade and blur techniques we introduced in the previous post to give us a greater sense of motion and color. Every particle knows its original starting point, and then moves along its vector path for the length of its life. Each particle has a random life limit, so they’re not all resetting at the same time. We are also using the horizontal position of the mouse to add additional directional noise to each particle, for some visual variety. The vertical position of the mouse affects how quickly the particle trails fade to black. I wanted to add a little more eye candy, so you can toggle glow on or off and also put in a couple lines of code to push any particles away from the mouse. All of this code is found in the main updateVectorField() function that gets called from CompositeTarget.Rendering().

In Conclusion

We’ve covered a fair amount of ground and presented a scattering of various design solutions during these past three tutorials. We started off demonstrating a finished Fluid Dynamics simulation in Silverlight and then slowly built upon the new WriteableBitmap to learn some advanced rendering techniques. Finally, we spent some time today looking at Vector Fields and how they can be used with our new found particle rendering code.

Along the way, I’ve tried to incorporate references, links and techniques from various languages, including Flash, Processing and OpenFrameworks. I’m a big believer in experience designers being multi- lingual and studying the deeper patterns that exist across all of these languages. Most of the byte manipulations for color, the procedural easing alogrithms, the line functions, and even the trigonometry for pushing particles arounds are all techniques that are common across multiple Interactive Design languages and once you understand the principles of them, you can usually translate them easily from one language to the other. This is a far more important skill to have as an experience designer than simply memorizing a solution approach specific to a single design language. Being multilingual and seeing the patterns behind interactive programming solutions is at least one way to help ensure you against the future.

With the release of Silverlight 3 and its powerful new features (including the WriteableBitmap we've been exploring in this series), I feel it has finally demonstrated a maturity and richness that deserves to be taken seriously by the interactive design community. I'm happy I've had a chance to hopefully show off some of the great experiences that are now possible in Sivlerlight 3, and I can't wait to see what the future holds for Silveright 4.

Rick Barraza
twitter.com/rickbarraza
September 10, 2009