WebGL: Mouse Picking

Related Topics: OpenGL FBO

• Selection Buffer
• Convert: Index to Color
• GLSL Shader for Selection
• Example

To interact with objects in a scene, you need to allow user pick an object using the mouse pointer. This page explains how to select an object with mouse cursor using selection buffer.

Selection Buffer

One way to select an object with the mouse pointer in WebGL is using a special framebuffer object. You render all objects in the scene by specifying a unique color for each object to a WebGL-managed framebuffer, then read the color value where the mouse pointer is positioned on the screen. If the color matches to an object, the object is selected. Here is the steps for drawing for selection;

Drawing to Selection Buffer

1. Create a framebuffer with the same resolution as the rendering screen
2. Assign a unique index to each object
3. Convert the index values to 32-bit colors (r, g, b, a)
4. Draw the objects with the monotone colors to the framebuffer

Reading from Selection Buffer

1. Get the mouse position (x,y) from the screen
2. Read the 1-pixel color value at the mouse position using glReadPixels()
3. Convert back the color value to the object index
4. If the index value is non-zero, an object is selected (hit)
5. If the index value is zero, no object is selected (not hit)

Here is the implementation of this special framebuffer integrating reading a single pixel at the mouse position and converting an index value to RGBA color. You can view the full source code of SelectBuffer class from GitHub.

A typical usage of this SelectBuffer class is;

// global object
gl = {};
...

//create selection buffer
gl.selectBuffer = new SelectBuffer(gl);
gl.pickIndex = 0;       // 0 mean no selection

// draw for selection buffer
function drawForSelection()
{
    gl.selectBuffer.bind();
    gl.clearColor(0,0,0,0);
    gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT);
    gl.program = gl.shaderPrograms["select"];
    gl.useProgram(gl.program);

    // set shader uniform for selection color of an object
    let index = ...;                             // set index value
    let color = gl.selectBuffer.getColor(index); // convert index to color
    gl.uniform4fv(gl.program.uniform.pickColor, color);

    // draw an object
    ...

    // unbind selection buffer
    gl.selectBuffer.unbind();
}

// hit test whenever mouse move
function handleMouseMove(e)
{
    //get mouse position
    let mouseX = e.pageX;
    let mouseY = e.pageY;

    // hit test
    // if it is non-zero, hit
    gl.pickIndex = gl.selectBuffer.pick(mouseX, gl.canvas.height - mouseY);
    ...
}

Note that the Y-axis in WebGL is bottom-to-top orientation, but the Y value of the screen in Javascript is top-to-bottom. So, make sure you invert the Y value of the mouse position before calling pick().

Conversion: Index to Color

To convert an integer value to a color calue or vice versa, you need bitwise operators.

Index to Color

To convert an index to a color, we break down the integer to 3 of 8-bit sections using the right shift operator and AND operator. Each 8-bit section represents each color component; red, green and blue. With this conversion, the maximum index value is 16,777,215 (0xFFFFFF).

// index to color
let index = ...;
let r = index & 0xFF;
let g = index >> 8 & 0xFF;
let b = index >> 16 & 0xFF;

// normalized color value (r,g,b,a)
let color = [r/255, g/255, b/255, 1];

Color to Index

To convert a color (r, g, b) to the corresponding index value, we simply shift each color component using the left shift operator, then add them together.

// color (r, g, b) to index
let color = ...;    // array of (r, g, b) components
let index = color[0] + (color[1] << 8) + (color[1] << 16);

GLSL Shader for Selection

The GLSL shader is simple. We only need to pass the vertex position to the vertex shader along with the matrix uniforms, and the object color converted from the index value as a uniform.

// vertex shader for mouse picking

// vertex attributes
attribute vec3 vertexPosition;          // vertex pos in object space

// uniforms
uniform mat4 matrixModelViewProjection; // model-view-projection matrix

void main(void)
{
    // transform vertex position to clip space
    gl_Position = matrixModelViewProjection * vec4(vertexPosition, 1.0);
}

// Fragment Shader for Mouse Picking

// uniforms
uniform vec4 pickColor;

void main(void)
{
    gl_FragColor = pickColor;
}

Example: Mouse Picking

This example draws multiple cubes and highlight the focused cube with the mouse pointer. It draws the scene twice; render to the selection buffer first, then render to the default frame buffer.

Fullscreen Demo: test_pick.html
GitHub Repo: test_pick.html

© 2025 Song Ho Ahn (안성호)