Compiling¶

This section provides information about the compilers at HPC2N. There are instructions about their usage, and some of the most popular flags.

For how to use the provided libraries and how to link to them, see the Libraries section.

There is also a section with an introduction to makefiles, a few examples, and links to more information.

Installed compilers¶

There are compilers available for Fortran 77, Fortran 90, Fortran 95, C, and C++. The compilers can produce both general-purpose code and architecture-specific optimized code to improve performance (loop-level optimizations, inter-procedural analysis and cache optimizations).

Compilers and Compiler Toolchains¶

The compilers are arranged in ‘Compiler toolchains’, which are bundles of software making up a complete environment for compiling or using a specific prebuilt software. A toolchain usually includes a compiler suite, an MPI version, BLAS, LAPACK, ScaLapack, and FFTW. It may also be just a compiler, like GCC alone, or something in between.

If you do

ml av

or

module avail

when you have logged in, you will get a list of (mainly) the compiler toolchains and their respective versions. You should always specify the full version when loading a compiler or software package.

Currently available toolchains:

• Clang: C, C++, Objective-C compiler, based on LLVM.
• GCC: GCC only
• foss: GCC, OpenMPI, OpenBLAS/LAPACK, FFTW, ScaLAPACK
• fosscuda: gompic, OpenBLAS/LAPACK, FFTW, ScaLAPACK
• gcccuda: GCC and CUDA
• gomkl: GCC, OpenMPI, IntelMKL
• gompi: GCC, OpenMPI
• gompic: GCC, OpenMPI, CUDA
• iccifort: icc, ifort
• iccifortcuda: icc, ifort, CUDA
• iimpi: icc, ifort, IntelMPI
• imkl: icc, ifort, IntelMKL
• intel-compilers: just the Intel compilers
• intel: icc, ifort, IntelMPI, IntelMKL
• intelcuda: intel, CUDA
• iompi: icc, ifort, OpenMPI

Use

ml av <compiler toolchain>

to see which versions are available.

In order to be able to use compiler toolchain, you need to load it with

ml <compiler toolchain>/<version>

For more information about compiler toolchains, see our page about the module system.

Build environment¶

Using a compiler toolchain by itself is possible but requires a fair bit of manual work, figuring out which paths to add to -I or -L ifor including files and libraries, and similar.

To make life as a software builder easier there is a special module available, buildenv, that can be loaded on top of any toolchain. If it is missing for some toolchain, send a mail to support@hpc2n.umu.se and let us know.

This module defines a large number of environment variables with the relevant settings for the used toolchain. Among other things it sets CC, CXX, F90, FC, MPICC, MPICXX, MPIF90, CFLAGS, FFLAGS, and much more.

To see all of them, after loading a toolchain do:

ml show buildenv

Depending on the software one can use these environment variables to set related makefile variables or cmake defines, or just use them for guidelines on what to use in makefiles etc.

Exactly how to use them depends on the softwares build system.

ml foss/2021b
ml buildenv
ml show buildenv

Compiler usage¶

Compilers are preferrably loaded as part of compiler toolchains. Read more in the section Installed compilers and on our page about the module system for more about that.

OpenMP: All compilers has this included, so it is enough to load the module for a specific compiler toolchain and then add the appropriate flag (see Flags).

Note that instead of using the compiler names for the specific compiler (Intel, GCC) you can instead load the buildenv module after loading the compiler toolchain to get the environment variables that can be used in place of the compiler name.

GNU Compiler Collection (GCC)¶

The compiler toolchain modules that are using the GCC compilers:

• GCC: GCC only
• foss: GCC, OpenMPI, OpenBLAS/LAPACK, FFTW, ScaLAPACK
• fosscuda: gompic, OpenBLAS/LAPACK, FFTW, ScaLAPACK
• gcccuda: gcccuda: GCC and CUDA
• gomkl: GCC, OpenMPI, IntelMKL
• gompi: GCC, OpenMPI
• gompic: GCC, OpenMPI, CUDA

Remember to load a specific version (see available ones with ml av TOOLCHAIN-MODULE).

Compiling with GCC

There are more information to find on the man pages which can be found by typing:

man <compiler name>

Here is a link to a place with more information: GCC online docs.

Intel Compilers¶

The Intel compilers are installed on Kebnekaise. The compiler toolchain modules that are using the Intel compilers:

• iccifort: icc, ifort
• iccifortcuda: icc, ifort, CUDA
• iimpi: icc, ifort, IntelMPI
• imkl: icc, ifort, IntelMKL
• intel: icc, ifort, IntelMPI, IntelMKL
• intelcuda: intel, CUDA
• iompi: icc, ifort, OpenMPI

Remember to load a specific version (see available ones with ml av TOOLCHAIN-MODULE).

Compiling with Intel

There are more information to find on the man pages which can be found by typing:

man <compiler name>

Here are some links to places with more information:

• Intel Fortran Compiler Classic and Intel Fortran Compiler Developer Guide and Reference
• Intel® C++ Compiler Classic Developer Guide and Reference
• Resource & Documentation Center
• Intel® oneAPI DPC++/C++ Compiler
• Developer Zone

Clang¶

C, C++, Objective-C compiler, based on LLVM. Does not include C++ standard library – use libstdc++ from GCC.

External info: Clang: a C language family frontend for LLVM

Compiler flags¶

This page contains information about some of the more important/popular flags for the compilers available at HPC2N. The flags below can all be taken to be valid for the Fortran and C/C++ compilers alike, as well as for compiling with the MPI libraries included (remember to load the proper modules - see the page about the module system and the section about installed compilers for more information about that.)

Intel Compilers¶

The Intel compilers are installed on HPC2N’s systems.

List of commonly used flags:

• -fast This option maximizes speed across the entire program.
• -g Produce symbolic debug information in an object file. The compiler does not support the generation of debugging information in assemblable files. If you specify the -g option, the resulting object file will contain debugging information, but the assemblable file will not. The -g option changes the default optimization from -O2 to -O0. It is often a good idea to add -traceback also, so the compiler generates extra information in the object file to provide source file traceback information.
• -debug all Enables generation of enhanced debugging information. You need to also specify -g
• -O0 Disable optimizations. Use if you want to be certain of getting correct code. Otherwise use -O2 for speed.
• -O Same as -O2
• -O1 Optimize to favor code size and code locality. Disables loop unrolling.
• -O1 may improve performance for applications with very large code size, many branches, and execution time not dominated by code within loops. In most cases, -O2 is recommended over -O1.
• -O2 (default) Optimize for code speed. This is the generally recommended optimization level.
• -O3 Enable -O2 optimizations and in addition, enable more aggressive optimizations such as loop and memory access transformation, and prefetching. The -O3 option optimizes for maximum speed, but may not improve performance for some programs. The -O3 optimizations may slow down code in some cases compared to -O2 optimizations. Recommended for applications that have loops with heavy use of floating point calculations and process large data sets.
• -Os Enable speed optimizations, but disable some optimizations that increase code size for small speed benefit.
• -fpe{0,1,3} Allows some control over floating-point exception (divide by zero, overflow, invalid operation, underflow, denormalized number, positive infinity, negative infinity or a NaN) handling for the main program at runtime. Fortran only. Default is -fpe3 meaning all floating-point exceptions are disabled and floating-point underflow is gradual, unless you explicitly specify a compiler option that enables flush-to-zero. The default value may slow runtime performance.
• -qopenmp Enable the parallelizer to generate multi-threaded code based on the OpenMP directives. The code can be executed in parallel on both uniprocessor and multiprocessor systems.
• -parallel Enable the auto-parallelizer to generate multi-threaded code for loops that can be safely executed in parallel. The -parallel option enables the auto-parallelizer if either the -O2 or -O3 optimization option is also on (the default is -O2). You might need to set the KMP_STACKSIZE environment variable to an appropriately large size, like 16m, to use this option.

To read about other flags, and for further information, look in the man files. They can be accessed like this:

$ module load intel
$ man ifort
$ man icc 

Here are some links to places with more information:

• Intel Fortran Compiler Documentation
• Intel C++ Compiler Options
• Recommended Intel Compiler Debugging Options

GNU Compiler Collection (GCC)¶

List of commonly used flags:

• -o file Place output in file ‘file’.
• -c Compile or assemble the source files, but do not link.
• -fopenmp Enable handling of the OpenMP directives.
• -g Produce debugging information in the operating systems native format.
• -O or -O1 Optimize. The compiler tried to reduce code size and execution time.
• -O2 Optimize even more. GCC performs nearly all supported optimizations that do not involve a space-speed tradeoff.
• -O3 Optimize even more. The compiler will also do loop unrolling and function inlining. RECOMMENDED
• -O0 Do not optimize. This is the default.
• -Os Optimize for size.
• -Ofast Disregard strict standards compliance. -Ofast enables all -O3 optimizations. It also enables optimizations that are not valid for all standard-compliant programs. It turns on -ffast-math and the Fortran-specific -fno-protect-parens and -fstack-arrays.
• -ffast-math Sets the options -fno-math-errno, -funsafe-math-optimizations, -ffinite-math-only, -fno-rounding-math, -fno-signaling-nans and -fcx-limited-range.
• -l library Search the library named ‘library’ when linking.

To read about other flags, and for further information, look in the man files. They can be accessed by first loading the module (see section Installed Compilers and then doing either:

$ man gfortran
$ man gcc
$ man g++

Here are links to places with more information:

• GCC online docs
• GCC 11.3 Command options

makefiles¶

The purpose of makefiles is to help you with your coding projects. One will often have a program which have, say, several header files and a number of C source files. If you then make a change to one of the files, you will have to recompile all the files. This is handled by the program make, following the commands in your makefile - it will only recompile the files which contains changes, and also makes it easier and faster to give the order to compile everything. You no longer have to write long and complex compiler commands every time.

Usually, the file that contains all the commands to make is called makefile or Makefile. You can then just invoke make from the command line and it will follow these instructions. You can, however, give the file any name. If you call it something else you will have to use make -f NAME_OF_MAKEFILE to call it.

Structure¶

A makefile consists of three sections: target, dependencies, and rules. The target is normally either an executable or object file name. The dependencies are source code or other things needed to make the target. The rules are the commands needed to make the target.

A simple makefile might be structured like this:

# Comments are preceded by the hash symbol
target: dependencies
        command 1
        command 2
        ...
        command n

A makefile can also contain definitions of variables and inclusion of other makefiles. The variables in makefiles may be overridden in the command-line arguments that are passed to the make utility. An example is that the variable “CC” is often used in makefiles to refer to a specific C compiler, and the user may wish to provide an alternate compiler to use.

NOTE: A very important thing to remember about makefiles are that every rule line begins with a tab, not spaces. Otherwise make might think it is some sort of dependency.

Comments¶

Any line that begins with a # is a comment and will be ignored.

Explicit rules¶

These are rules that tell make which files depend on the compilation of other files, as well as the commands required to compile a particular file. They have this form:

targetfile : sourcefiles
# NOTE: there is a TAB character before each command.
    commands           

This rule means that to create the targetfile, make must perform the listed commands on the sourcefiles.

Example

main: main.c test.h List.h
    gcc -o main test.h List.h

The above example shows a rule that says that in order to create the targetfile main, the sourcefiles main.c, test.h and List.h have to exist, and make should use the command:

gcc -o main test.h List.h

to create it.

Implicit rules¶

Implicit rules are in many ways like explicit rules, they are just listed without commands. This means make will use the suffixes on the files to determine which commands to perform.

Example

test.o: test.c io.h

This will cause the following command to be executed:

$(CC) $(CFLAGS) -c test.c io.h

Variables¶

It is possible to define variables in a makefile. To do this use the following command:

VAR_NAME=value

It is a convention to write variable names in uppercase. For instance:

OBJECTS=main.o hello.o test.o

In order to get a variable’s value, put the symbol $ before the varable’s name, like this:

$(VAR_NAME)

Automatic Variables¶

(Much of this section is taken from https://www.gnu.org/software/make/manual/html_node/Automatic-Variables.html#Automatic-Variables).

Suppose you are writing a pattern rule to compile a “.c” file into an “.o” file: how do you write the “cc” command so that it operates on the right source file name? You cannot write the name in the recipe, because the name is different each time the implicit rule is applied.

What you do is use another feature of make, namely automatic variables, which are computed afresh for each rule that is executed, based on the target and prerequisites of the rule. For example, you would use “$@” for the object file name and “$<” for the source file name.

NOTE that automatic variables:

• only have values within the recipe - you cannot use them anywhere within the target list of a rule (they have no value there and will expand to the empty string).
• they cannot be accessed directly within the prerequisite list of a rule. A common mistake is attempting to use $@ within the prerequisites list; this will not work.

Example:

all: file.c main.c

Here:

• $@ evaluates to all
• $< evaluates to file.c
• $^ evaluates to file.c main.c

Examples¶

Hello: Hello.o
    cc -o $@ $<

Hello.o: Hello.c
    cc -c -o $@ $<

.PHONY: clean

clean:
    rm -f Hello Hello.o

This small makefile is started with either make or make Hello. It does one more thing than just compile the program - there is also the target “clean” which is used to remove all the files generated by the compilation with one command: make clean.

The first line in this makefile is the dependendencies (Hello: Hello.o), while the second line contains the rule to create the target Hello. The third line the dependencies for making Hello.o are listed, while line 4 contains the rules to make the target Hello.o in line 3.

Note that the PHONY tag is a technicality that tells make that a particular target name does not produce an actual file. It is not strictly necessary to include the line with the PHONY tag, unless a file named clean exists in the current directory. The “$@” and “$\<” are two of the so-called internal macros and stand for the target name and “implicit” source, respectively. There are a number of other internal macros, which can be seen here.

main.exe : main.o foo.o
    gcc main.o foo.o -o main.exe

main.o : main.c
    gcc -c main.c

foo.o : foo.c
    gcc -c foo.c

This simple makefile is run by giving the command make. The program always starts with the first rule it sees, in this case for main.exe. It then tries to build that first and finds dependencies. It will then check to see if any of those needs to be updated. This means, that the first time you call make for this makefile, it will run the following three commands:

gcc -c main.c
gcc -c foo.c
gcc main.o foo.o -o main.exe

Let us assume you then edit the file foo.c and run make again. It will then only run these commands:

gcc -c foo.c
gcc main.o foo.o -o main.exe

Because there was no reason to recompile main.c, it will not do that.

CC = gcc
CFLAGS = -g -O2
OBJECTS = main.o foo.o

main.exe : $(OBJECTS)
    $(CC) $(CFLAGS) $(OBJECTS) -o main.exe

main.o : main.c
    $(CC) $(CFLAGS) -c main.c

foo.o : foo.c
    $(CC) $(CFLAGS) -c foo.c

There are many similarities between this makefile and the last. The difference is that many of the commands have been replaced with variables. Then, when make runs, it replaces the variables in the target, dependency, and command sections of the rules. This is often an advantage, since it lets you specify the things that changes often in one place. One example is that it is very easy to change the compiler, since we are using the variable CC everywhere - and it is defined at the very top.

CC = gcc
CFLAGS = -g -O2
OBJECTS = main.o foo.o

main.exe : $(OBJECTS)
    $(CC) $(CFLAGS) $(OBJECTS) -o main.exe

%.o : %.c
    $(CC) $(CFLAGS) -c $<

This is almost the same makefile as the one above, however, if you look in the last line you will see a pattern has been defined. This can be used every time make needs to compile any source. This is much simpler if we have many sources - otherwise we would have to write a rule for including the source for each of them. In the pattern, the “%” character is used to denote the part of the target and dependency that matches whatever the pattern is used for. The “$<” is a special variable meaning “whatever the dependencies are”. There is another useful variable which we will often have use for. It is “$@“, which means “the target”.

After this addition we can now add more source files in a simple manner, by just updating the line that defines the OBJECTS variable. The variable names CC and CFLAGS are part of make’s built-in rules. To see a list of all of these, look at man make or run make -p. There is also a reference manual for make, which contains many examples and suggestions for building makefiles.

Links¶

Here is a list of links to more information about makefiles.

• GNU ‘make’
• Short Makefile Tutorial
• A short guide to Makefiles
• Learn Makefiles with examples