CMake and the executable stack size in Visual Studio

Today I have been struggling with my first mystical error at my new job. The C++ project was able to compile and execute. At least, the screen was showing the 3D initial menu. However, when I tried to open any of the sub-applications where big 3D models where used, strange exceptions were thrown, such as out of memory when loading a texture, vertex buffer can not be allocated, and similar ones depending on the sub-application executed.

I have been porting the project from a manually maintained .sln file to CMake. So, the first step was to make sure that the original executable was working, and it was. Then, I compared the compilation and linking flags in the both project files, and I removed the new flags, which were not present in the original solution, one by one. Finally, I discovered (after 5 hours) that CMake (version 2.8.10) adds to the linker flags /STACK:10000000, which means that the heap size is initialized to 10MB, instead of the default 1MB. Adding the following lines to the main CMake file:
STRING(REGEX REPLACE “/STACK:[0-9]+[ ]” “” CMAKE_EXE_LINKER_FLAGS “${CMAKE_EXE_LINKER_FLAGS}”)
SET(CMAKE_EXE_LINKER_FLAGS “${CMAKE_EXE_LINKER_FLAGS}”
CACHE STRING “Flags used by the linker.”
FORCE
)

solved the problem. Note that a similar problem may occur if there are multiple threads in the application, since the total amount of memory used will be num_threads by stack_size.

Note that this flag must be also removed from all your project dependencies that use CMake for generating project files. Then, the modified dependencies must be rebuild and finally your project, too.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s