Friday, July 20, 2007

8: Mouse Handling

This entry is part 8 of a 12-part series on WPF 3D.

3D and the Mouse

Many 3D applications want to offer interactive capabilities using the mouse:

• Click on an object in the 3D scene to "select" so that further actions can be applied that specific item.
• Click and drag to rotate, zoom or pan.
• Show coordinates or other information when hovering over specific objects.

The mouse lives in a 2D world.  The objects in your scene are in a 3D world.  You need a few tricks to bridge the gap.

The Need for an Overlay

WPF makes it fairly simply to get mouse notifications for any Visual.  It is therefore tempting to just add mouse event handlers to the Viewport3D object, somewhat like this:

<Viewport3D
    MouseUp="OnViewportMouseUp"
    MouseDown="OnViewportMouseDown"
    MouseMove="OnViewportMouseMove" >

The problem with this approach is that the Viewport3D will only receive mouse events when the mouse is actually hovering over one of the rendered triangles in the scene.  When the mouse is over the background, no events are sent. 

For some situations (such as picking a 3D object, or a 3D scene which fills the entire screen), this is fine.  For others (such as interactive rotation of a model centered on the screen), this may not be not so good.

The usual fix for this problem is to overlay another element directly on top of the Viewport3D.  The overlay must be transparent to allow the Viewport3D to be completely visible.  The mouse event handlers should be placed on the overlay instead, as the Viewport3D will receive no mouse events at all.  Because the overlay and the Viewport3D have the same 2D coordinate system, all the math works out just fine.  In XAML, this approach might look something like this:

<Grid >
  <Viewport3D Grid.Row="0" Grid.Column="0" >
  (model stuff goes here)
  </Viewport3D>
  <Canvas Grid.Row="0" Grid.Column="0"
    Background="Transparent"
    MouseUp="OnViewportMouseUp"
    MouseDown="OnViewportMouseDown"
    MouseMove="OnViewportMouseMove"  />
</Grid>

From 2D to 3D

Inside the mouse handlers, you want to take the 2D coordinates of the mouse click and find the 3D object where that click occurred.  WPF 3D makes part of this work simple with VisualTreeHelper.HitTest().  After that, the handling will depend greatly on your application.

For example, in Sawdust, every 3D scene is generated on the fly, constructed from a data structure which was built from the solid modeling code.  When the user clicks on the 3D model, a specific piece of wood is selected so that further operations can be applied directly to it.  My code for OnViewportMouseDown() looks something like this:

/// <summary>
/// On mouse click, select the specific board
/// where the click happened.
/// </summary>
/// <param name="sender"></param>
/// <param name="args"></param>
public void OnViewportMouseDown(
    object sender,
    System.Windows.Input.MouseEventArgs args)
{
    if (vstuff.models == null)
    {
        return;
    }

    if (
        Keyboard.IsKeyDown(Key.LeftCtrl)
        || Keyboard.IsKeyDown(Key.RightCtrl)
        )
    {
        // extending the selection. 
        // don't unselect all first.
    }
    else
    {
        UnselectAll();
    }

    RayMeshGeometry3DHitTestResult rayMeshResult =
        (RayMeshGeometry3DHitTestResult)
          VisualTreeHelper.HitTest(myVP, args.GetPosition(myVP));
   
    if (rayMeshResult != null)
    {
        PartialModel found = null;
        foreach (PartialModel pm in vstuff.models)
        {
            if (pm.mesh == rayMeshResult.MeshHit)
            {
                found = pm;
                break;
            }
        }

        if (found != null)
        {
            if (IsSelected(found.bag.solid))
            {
                Unselect(found.bag.solid);
            }
            else
            {
                Select(found.bag.solid);
            }
        }
    }
}

I included all the code, even though most of this routine is specific to my application.  In particular, vstuff, PartialModel, IsSelect(), Select(), Unselect(), and UnselectAll() are all elements of Sawdust.  Your app will have different things, depending on the manner and data structures used to generate your 3D scenes.

The key is the RayMeshGeometry3DHitTestresult returned by VisualTreeHelper.HitTest().  The members of this object contain plenty of helpful information.  In my particular case, I use the MeshHit member to search my data structures so I can find the original information used to construct that piece of the 3D model.  My life would be a bit easier if MeshGeometry3D had something like a Tag member, so I could just tuck a reference to my PartialModel object right there for easy retrieval.