Ambassador

Disclaimer: I am not a Bullet Physics expert, I'm barely a beginner. I will try to sound like a newbie throughout the post.

I have a compact 3D engine which is developed for graph visualization. It's an interesting topic in computer science and I was curious about the concept.

Rendering in OpenGL is usually simple, I think (well, depends on what you are rendering, of course). Rendering efficiently and really fast is another challenge, but that hasn't been my real focus yet. Of course, I don't use fixed pipeline, and my code runs on OpenGL ES 2.0 as well, and runs with OpenGL 2.1 on Mac OS X. I have spent least effort for performance; texture packing, and sharing some array buffers were some.

Physics, however, has become a great challenge to me. Long story short; right before my final render call, object transform was multiplying with the wrong matrix when rendering, which leads physics engine to think the object is somewhere else.

My mistake
I have my subclassed btMotionState, but I didn't read the btDefaultMotionState class source code - big mistake. I also didn't notice "setWorldTransform" method's name explicitly say "world transform". When I have realized this, I felt really stupid!

I was basically putting my vertices to given coordinates on the screen with its own transform, somewhat detached from the Bullet motion state. I have fixed this problem before anything else. I've started placing meshes around the origin (0, 0, 0). Therefore, for instance, a unit cube (width=1, height=1, depth=1) sitting at (3, 4, 5) coordinates, my vertices used to start at 3, 4, 5 and I was adding extents. This was wrong, but I don't know why I've taken this approach at the time. They now start at (-half_width, -half_height, -half_depth) and have extents half of each respective dimension's extent.

I already designed everything assuming physics owns spatial information, but...
I think I've understood the issue; when a physics engine is involved, it takes over all spatial information. All meshes' center should be on origin and all world transform is going to be handled by/done via the physics engine.

After putting object's center at the origin, the next stage was to remove any other transform and bind all transform directly to physics engine. Bullet has a btRigidBody type, which is, basically, the collision object itself. This rigid body, among other things, has a motion state, which holds world transform (translation and rotation). Bullet processes all active collision objects and interpolate their world transform. I have basically updated the motion state of the rigid body. I have basically changed my btMotionState subclass, and made setWorldTransform multiply center of mass transform (comes from bullet) with the world transform (comes from camera). There is one important point here (which I missed for ~3 hours); world transform does not include projection matrix, it's purely model-view matrix (look at matrix, in other words). So, my motion state basically multiplied camera's look at matrix with the center of mass transform (argument passed by Bullet). In getWorldTransform, this multiplication result will be returned.

// pseudo code

// bullet calls this method; this is a virtual method.
void my_motion_state::setWorldTransform(const btTransform& centerOfMassWorldTrans)
{
    set_world_transform_full(centerOfMassWorldTrans, true);
}

// actual implementation, can be called for drawing AABB for debugging as well, which passes false to transform_to_world
void my_motion_state::set_world_transform_full(const btTransform& centerOfMassWorldTrans, bool transform_to_world)
{
    if (transform_to_world) {
        vmMatrix4 transform_matrix = current_camera->get_modelview() * util::convert<vmMatrix4>(centerOfMassWorldTrans);        
        world_transform.setFromOpenGLMatrix(util::convert<float[16]>(transform_matrix);
    }
    else
        world_transform = centerOfMassWorldTrans;
    
    if (rigid_body_)
        rigid_body_->set_world_transform(final_trans);
}

void my_motion_state::getWorldTransform(btTransform &centerOfMassWorldTrans) const {
    centerOfMassWorldTrans = world_transform_;
}

Rendering changes
When engine is about to draw, it checks whether shader uniform has changed since last render, if so, then model-view-projection matrix gets multiplied with object's transform, and engine draws the VBO range. After converting motion state to take "world transform", I have removed this multiplication. That is, every object's motion state already contains world transformed coordinates, therefore no multiplication is required... not! But that's next problem!

I've then remembered I've removed projection matrix from world transform multiplication! Render pass should multiply motion state's world transform with the projection matrix then draw VBO range.

Un-projecting - creating the ray
The final issue, which is somewhat easy, is to convert the screen coordinates to world coordinates and create a ray from that point to far clip plane. There are a multiple examples you can find to do it, if you don't know how to do it (e.g. myself). I couldn't figure out one myself, so I have used gluUnProject source code (I think I've grabbed the one in MESA). Note that fixed function pipeline (glMatrixMode, glBegin, etc.) is dead; I don't understand why Bullet examples still use that. May be it's quicker to get an example up and running but that shouldn't be the goal. Nonetheless, gluUnProject requires model-view and projection matrices, point on the screen. Ray requires two vectors; origin and direction (and magnitude, if I recall correctly from math classes, but that's included in two vectors). gluUnProject requires a z value. To get origin, passing 0.0f should give the near clip plane. To get the destination (direction * magnitude?), passing 1.0f works. This creates a ray from the point on near clip plane to the point in far clip plane.

Filtering ghost objects
Bullet kinematic objects have a ghost object to track "contacts" (if I understood correctly). Some of my rigid bodies have ghost objects. Again, if I understood correctly, ghost objects are not collision objects. Therefore btCollisionWorld::ClosestRayResultCallback would hit an object, but since it's not always a rigid body, it would give a NULL btRigidBody. Instead, I've used btCollisionWorld::AllHitsRayResultCallback, and filter out ghosts. I'm sure there is better way to filter them out, but I haven't checked yet (probably subclassing RayResultCallback will be enough).

This has taken more than enough, and my motivation was near zero when I began today.