Building GCC with Stack-Smashing Protector on Solaris

John Cartwright | Published: 24th May 2003 | Originally published at grok.org.uk

How to build GCC 2.95.3 with Stack-Smashing Protector (ProPolice) support for Solaris on SPARC and x86, allowing third-party applications to be compiled with stack overflow protection.

Introduction

This document outlines how to build a version of GCC for the SPARC and x86 architectures that produces 32-bit executables with IBM’s Stack-Smashing Protector (SSP), otherwise known as ProPolice.

This technology has been implemented in a number of open-source operating systems, such as OpenBSD, but requires modification to work in a Solaris environment.

Whilst we cannot rebuild the core OS itself without source code, we can still make use of this tool to compile popular third-party applications to raise the bar against stack-smashing exploits directed at Solaris machines.

Update: These instructions apply to GCC 2.95.3 only. Please see http://grok.org.uk/tools/ssp/ for other supported versions.

Prerequisites

The following files are required for the compiler build:

GCC 2.95.3: [ tar.gz | md5 ]

Hiroaki Etoh’s ProPolice patch: [ tar.gz | md5 ]

Solaris compatibility patch: [ patch | md5 | asc ]

Patch details

The original SSP patch functions correctly on Solaris in terms of detecting and halting stack-smashing attacks, reporting errors to stderr. However, it assumes a BSD-style /dev/log socket, which Solaris does not provide.

This patch replaces that mechanism with a call to syslog(3C) and removes several irrelevant headers and definitions. It also patches the Solaris-specific configuration files (i386/t-sol2 and sparc/t-sol2) rather than Linux-specific ones.

The patch has been verified under Solaris 8 on Ultra5, Ultra60, and dual-PIII x86 platforms.

Building the compiler

Follow this procedure:

# gzip -cd gcc-2.95.3.tar.gz | tar xf -
# cd gcc-2.95.3/gcc
# gzip -cd ../../protector-2.95.3-20.tar.gz | tar xf -
# gpatch < ../../propolice-2.95.3-20-solaris.patch
# gpatch -p1 < protector.dif
# cd ..
# ./configure --prefix=/opt/local/gcc --enable-languages=c,c++
# gmake bootstrap
# gmake check
# gmake install

Use GNU patch and make for best results. Testing requires additional tools such as DejaGNU.

Testing

Construct a test program:

/* test-propolice.c */
#define OVERFLOW "This is longer than 10 bytes"

int main(int argc, char *argv[]) {
    char buffer[10];
    strcpy(buffer, OVERFLOW);
    return 0;
}

Compile and run:

# gcc -fstack-protector -o test-propolice test-propolice.c
# ./test-propolice
stack smashing attack in function main

You should also see a syslog entry:

test-propolice[19961]: [ID 702911 auth.crit] stack smashing attack in function main

Notes

SSP should be used alongside Solaris protections such as non-executable stacks:

set noexec_user_stack = 1
set noexec_user_stack_log = 1

Applications should also be linked against the Solaris non-executable stack mapfile:

gcc -fstack-protector -Wl,-M,/usr/lib/ld/map.noexstk \
  -o test-propolice test-propolice.c

Conclusion

With a few simple changes, SSP can be successfully deployed on Solaris. Applications should be built without protection first, then recompiled for comparison. SSP requires /dev/urandom, so Solaris patches 112438-xx (SPARC) or 112439-xx (x86) must be applied, particularly in chroot environments.

John Cartwright <johnc@grok.org.uk>