Prevent out-of-bounds read when sorting segment sides

GCC 6 `std::sort` sometimes compares an element to itself.  For a normal
implementation of comparison, this is useless, but not harmful.  The
render comparison predicate relies on accessing A[B[a][b]] when
comparing `a` and `b`.  Array `B` has `-1` in positions where `a == b`,
which causes an access to `A[-1]`, which is undefined behavior.  This
crashes when using _GLIBCXX_DEBUG:

    Error: attempt to subscript container with out-of-bounds index -1, but
    container only holds 8 elements.

    Objects involved in the operation:
	sequence "this" @ 0x0x335adf0 {
	  type = std::__debug::array<int, 8ul>::_Array_check_subscript<8ul>;
	}

Since this is undefined behavior, non-debug builds might also misbehave.
Current data layouts make it likely that the failure would not have
externally observable consequences.

Prevent the invalid access by short-circuiting the result if `a == b`.
This commit is contained in:
Kp 2017-02-11 21:42:43 +00:00
parent dfe827e793
commit 0fd80b12df

View file

@ -958,6 +958,8 @@ static bool compare_children(const vcsegptridx_t seg, sidenum_fast_t s0, sidenum
{
Assert(s0 != side_none && s1 != side_none);
if (s0 == s1)
return false;
if (Side_opposite[s0] == s1)
return false;
//find normals of adjoining sides