x86 calling conventions

Previous to microcomputers, the machine manufacturer generally provided an operating system and compilers for several programming languages. The calling conventions adopted for the platform were those defined by the manufacturer's software implementation.

Early microcomputers before Apple II Computers generally came "bare" of an OS or compilers, as did the IBM PC. The only hardware standard for IBM PC compatible machines was defined by the Intel processors (8086, 80386) and the literal hardware IBM shipped. Hardware extensions and all software standards (save for a BIOS calling convention) were thrown open to market competition.

A multitude of independent software firms offered operating systems, compilers for many programming languages, and applications. Many different calling schemes were implemented by the firms, often mutually exclusive, based on different requirements, historical practices, and programmer creativity.

After the IBM compatible market shakeout, Microsoft operating systems and programming tools (with differing conventions) predominated, while second tier firms like Borland and Novell, and open source projects like GCC, still maintained their own standards. Provisions for inter-operability between vendors and products were eventually adopted, simplifying the problem of choosing a viable convention.^[1]

Caller clean-up

In these conventions the caller cleans the arguments from the stack, which allows for variable argument lists; e.g., printf().

cdecl

The cdecl calling convention is used by many C systems for the x86 architecture.^[1] In cdecl, function parameters are pushed on the stack in a right-to-left order. Function return values are returned in the EAX register (except for floating point values, which are returned in the x87 register ST0). The values in registers EAX, ECX, and EDX do not need to be preserved, whereas the others do.^{[citation needed]}

On Linux, GCC sets the de facto standard for calling conventions. Since GCC 4.5, the stack must be aligned to a 16-byte boundary when calling a function (previous versions only required a 4-byte alignment).

For instance, the following C code function prototype and function call:

int function_name(int, int, int);
int a, b, c, x;
…
x = function_name(a, b, c);

will produce the following x86 Assembly code (written in MASM (intel) syntax, with destination first):

push c             ; arg 3
push b             ; arg 2
push a             ; arg 1
call function_name ; jump to function_name's code
add esp, 12        ; pop function args (a, b, c) off the stack
mov x, eax         ; fetch function return value

The calling function cleans the stack after the function call returns.

There are some variations in the interpretation of cdecl, particularly in how to return values. As a result, x86 programs compiled for different operating system platforms and/or by different compilers can be incompatible, even if they both use the "cdecl" convention and do not call out to the underlying environment. Some compilers return simple data structures with the length of 2 registers or less in EAX:EDX, and larger structures and class objects requiring special treatment by the exception handler (e.g., a defined constructor, destructor, or assignment) are returned in memory. To pass "in memory", the caller allocates memory and passes a pointer to it as a hidden first parameter; the callee populates the memory and returns the pointer, popping the hidden pointer when returning.

In Linux/GCC, double/floating point values should be pushed on the stack via the x87 pseudo-stack. Like so:

sub esp, 8    ; make room for the double
fld [ebp + x] ; load our double onto the floating point stack
fstp [esp]    ;  push our double onto the stack
call func     ;
add esp, 8    ;

Using this method ensures it is pushed on the stack in the correct format.

The cdecl calling convention is usually the default calling convention for x86 C compilers, although many compilers provide options to automatically change the calling conventions used. To manually define a function to be cdecl, some support the following syntax:

void _cdecl function_name(params);

The _cdecl modifier must be included in the function prototype, and in the function declaration to override any other settings that might be in place.

syscall

This is similar to cdecl in that arguments are pushed right to left. EAX, ECX, and EDX are not preserved. The size of the parameter list in doublewords is passed in AL.

Syscall is the standard calling convention for 32 bit OS/2 API.

optlink

Arguments are pushed right to left. The three lexically first (leftmost) arguments are passed in EAX, EDX, and ECX and up to four floating-point arguments are passed in ST(0) through ST(3), although space for them is reserved in the argument list on the stack. Results are returned in EAX or ST(0). Registers EBP, EBX, ESI, and EDI are preserved.

Optlink is used by the IBM VisualAge compilers.

Callee clean-up

When the callee cleans the arguments from the stack it needs to be known at compile time how many bytes the stack needs to be adjusted. Therefore, these calling conventions are not compatible with variable argument lists, eg. printf(). They may be, however, more space efficient, as the code needed to unwind the stack does not need to be generated for each call.

Functions which utilize these conventions are easy to recognize in ASM code because they will unwind the stack prior to returning. The x86 ret instruction allows an optional 16-bit parameter that specifies the number of stack bytes to unwind before returning to the caller. Such code looks like this:

 ret 12

pascal

Based on the Pascal programming language's calling convention, the parameters are pushed on the stack in left-to-right order (opposite of cdecl), and the callee is responsible for balancing the stack before return.

This calling convention was common in the following 16-bit APIs: OS/2 1.x , Microsoft Windows 3.x, and Borland Delphi version 1.x.

register

An alias for Borland fastcall.

stdcall

The stdcall^[2] calling convention is a variation on the pascal calling convention in which the callee is responsible for cleaning up the stack, but the parameters are pushed onto the stack in right-to-left order, as in the _cdecl calling convention. Registers EAX, ECX, and EDX are designated for use within the function. Return values are stored in the EAX register.

stdcall is the standard calling convention for the Microsoft Win32 API and for Open Watcom C++.

fastcall

Conventions entitled fastcall have not been standardized, and have been implemented differently, depending on the compiler vendor.^[1] Typically fastcall calling conventions pass one or more arguments in registers which reduces the number of memory accesses required for the call.

Microsoft fastcall

Microsoft or GCC^[3] __fastcall^[4] convention (aka __msfastcall) passes the first two arguments (evaluated left to right) that fit into ECX and EDX. Remaining arguments are pushed onto the stack from right to left.

Borland fastcall

Evaluating arguments from left to right, it passes three arguments via EAX, EDX, ECX. Remaining arguments are pushed onto the stack, also left to right.^[5]

It is the default calling convention of the 32 bit compiler of Embarcadero Delphi, where it is known as register.

Watcom register based calling convention

Watcom does not support the __fastcall keyword except to alias it to null. The register calling convention may be selected by command line switch. (However, IDA uses __fastcall anyway for uniformity)

Up to 4 registers are assigned to arguments in the order eax, edx, ebx, ecx. Arguments are assigned to registers from left to right. If any argument cannot be assigned to a register (say it is too large) it, and all subsequent arguments, are assigned to the stack. Arguments assigned to the stack are pushed from right to left. Names are mangled by adding a suffixed underscore.

Variadic functions fall back to the Watcom stack based calling convention.

The Watcom C/C++ compiler also uses the #pragma aux^[6] directive that allows the user to specify his own calling convention. As its manual states, "Very few users are likely to need this method, but if it is needed, it can be a lifesaver".

TopSpeed / Clarion / JPI

The first four integer parameters are passed in registers eax, ebx, ecx and edx. Floating point parameters are passed on the floating point stack – registers st0, st1, st2, st3, st4, st5 and st6. Structure parameters are always passed on the stack. Additional parameters are passed on the stack after registers are exhausted. Integer values are returned in eax, pointers in edx and floating point types in st0.

safecall

In Embarcadero Delphi and Free Pascal on Microsoft Windows, the safecall calling convention encapsulates COM (Component Object Model) error handling, thus exceptions aren't leaked out to the caller, but are reported in the HRESULT return value, as required by COM/OLE. When calling a safecall function from Delphi code, Delphi also automatically checks the returned HRESULT and raises an exception if necessary.

The safecall calling convention is the same as the stdcall calling convention, except that exceptions are passed back to the caller in EAX as a HResult (instead of in FS:[0]), while the function result is passed by reference on the stack as though it were a final "out" parameter. When calling a Delphi function from Delphi this calling convention will appear just like any other calling convention, because although exceptions are passed back in EAX, they are automatically converted back to proper exceptions by the caller. When using COM objects created in other languages, the HResults will be automatically raised as exceptions, and the result for Get functions is in the result rather than a parameter. When creating COM objects in Delphi with safecall, there is no need to worry about HResults, as exceptions can be raised as normal but will be seen as HResults in other languages.

function function_name(a: DWORD): DWORD; safecall;

Returns a result and raises exceptions like a normal Delphi function, but it passes values and exceptions as though it was:

function function_name(a: DWORD; out Result: DWORD): HResult; stdcall;

This section requires expansion.

Either caller or callee clean-up

thiscall

This calling convention is used for calling C++ non-static member functions. There are two primary versions of thiscall used depending on the compiler and whether or not the function uses variable arguments.

For the GCC compiler, thiscall is almost identical to cdecl: the calling function cleans the stack, and the parameters are passed in right-to-left order. The difference is the addition of the this pointer, which is pushed onto the stack last, as if it were the first parameter in the function prototype.

On the Microsoft Visual C++ compiler, the this pointer is passed in ECX and it is the callee that cleans the stack, mirroring the stdcall convention used in C for this compiler and in Windows API functions. When functions use a variable number of arguments, it is the caller that cleans the stack (cf. cdecl).

The thiscall calling convention can only be explicitly specified on Microsoft Visual C++ 2005 and later. On any other compiler thiscall is not a keyword. (Disassemblers like IDA, however, have to specify it anyway. So IDA uses keyword __thiscall for this.)

Intel ABI

According to the Intel ABI, the EAX, EDX, and ECX are to be free for use within a procedure or function, and need not be preserved.

x86-64 calling conventions

Microsoft x64 calling convention

The Microsoft x64 calling convention^[7] (for long mode on x86-64) takes advantage of additional register space in the AMD64/Intel 64 platform. The registers RCX, RDX, R8, R9 are used for integer and pointer arguments (in that order left to right), and XMM0, XMM1, XMM2, XMM3 are used for floating point arguments. Additional arguments are pushed onto the stack (right to left). Integer return values (similar to x86) are returned in RAX if 64 bits or less. Floating point return values are returned in XMM0. Parameters less than 64 bits long are not zero extended; the high bits contain garbage.

When compiling for the x64 architecture in a Windows context (whether using Microsoft or non-Microsoft tools), there is only one calling convention — the one described here, so that stdcall, thiscall, cdecl, fastcall, etc., are now all one and the same.

In the Microsoft x64 calling convention, it's the caller's responsibility to allocate 32 bytes of "shadow space" on the stack right before calling the function (regardless of the actual number of parameters used), and to pop the stack after the call. The shadow space is used to spill RCX, RDX, R8, and R9.^[8]

For example, a function taking 5 integer arguments will take the first to fourth in registers, and the fifth will be pushed on the top of the shadow space. So the stack will be composed (in ascendant order) by the shadow space (32 bytes) followed by the fifth parameter.

In x86-64, Visual Studio 2008 stores floating point numbers in XMM6 and XMM7 (as well as XMM8 through XMM15); consequently, for x86-64, user-written assembly language routines must preserve XMM6 and XMM7 (as compared to x86 wherein user-written assembly language routines did not need to preserve XMM6 and XMM7). In other words, user-written assembly language routines must be updated to save/restore XMM6 and XMM7 before/after the function when being ported from x86 to x86-64.

System V AMD64 ABI convention

The calling convention of the System V AMD64 application binary interface^[9] is followed on Linux and other non-Microsoft operating systems. The registers RDI, RSI, RDX, RCX, R8 and R9 are used for integer and pointer arguments while XMM0, XMM1, XMM2, XMM3, XMM4, XMM5, XMM6 and XMM7 are used for floating point arguments. For system calls, R10 is used instead of RCX.^[9] As in the Microsoft x64 calling convention, additional arguments are pushed onto the stack and the return value is stored in RAX.

Thunks and calling conventions

A "thunk" means a small block of code, which can be generated dynamically at runtime, in order to be executed before calling the "actual" function. This is sometimes very useful. One common scenario is to convert a C++ functor object to a normal pointer to a function. In general, doing so requires one to insert an additional pointer which presents the "this" pointer to the functor object.

To write such a thunk, understanding of the calling convention the compiler is using is fundamental. One can easily imagine how to provide a general way to adjust the stack for stdcall to thiscall, because stdcall does not use registers and the additional parameter is to be passed through ecx, which is free to use. But how about for example fastcall or x64 calling convention? Those calling conventions uses registers to pass arguments, usually harder to predict the actual size and/or correct register for a specific parameter. However, it does not mean it cannot be done.

For example, if the compiler is using Microsoft x64 calling convention, what we need to do is to look at the convention. It says that 4 integer register and 4 floating point registers were selected as 4 parameter slot, and each parameter must occupy exactly one slot. If you have more than 4 parameters, additional parameters is passed by stack, with 8 bytes for each parameter.

Thus if the number of parameters is from 0 to 3, although we do not know if they occupy the integer registers or the floating point registers, we can shift them both, and result in a general solution.

For function objects that have at least 4 parameters, we need to detect the fourth parameter type. if it is a floating point then we push XMMS3, otherwise push R9. In this case, we can use templates and type traits feature to detect the exact parameter type.

For all other conventions we can do the same. The solution and implementation difficulty may be different, but the concept remains the same.

List of x86 calling conventions

This is a list of x86 calling conventions.^[1] These are conventions primarily intended for C/C++ compilers (especially the 64-bit part below), and thus largely special cases. Other languages may use other formats and conventions in their implementations.

Architecture	Calling convention name	Operating system, Compiler	Parameters in registers	Parameter order on stack	Stack cleanup by	Notes
16-bit	cdecl			RTL (C)	Caller
	pascal			LTR (Pascal)	Function
	fastcall	Microsoft (non-member)	ax, dx, bx	LTR (Pascal)	Function	Return pointer in bx.
	fastcall	Microsoft (member function)	ax, dx	LTR (Pascal)	Function	"this" on stack low address. Return pointer in ax.
	fastcall	Borland compiler^[10]	ax, dx, bx	LTR (Pascal)	Function	"this" on stack low address. Return pointer on stack high address.
		Watcom compiler	ax, dx, bx, cx	RTL (C)	Function	Return pointer in si.
32-bit	cdecl			RTL (C)	Caller
	stdcall			RTL (C)	Function
		GCC		RTL (C)	Hybrid	Stack possibly on 16 bytes aligned.
	fastcall	Microsoft	ecx, edx	RTL (C)	Function	Return pointer on stack if not member function.
	fastcall	GCC	ecx, edx	RTL (C)	Function
	fastcall	Borland/Embarcadero compiler	eax, edx, ecx	LTR (Pascal)	Function
	thiscall	Microsoft	ecx	RTL (C)	Function	Default for member functions.
		Watcom compiler	eax, edx, ebx, ecx	RTL (C)	Function	Return pointer in esi.
64-bit	Microsoft x64 calling convention^[7]	Windows (Microsoft compiler, Intel compiler, Embarcadero compiler)	rcx/xmm0, rdx/xmm1, r8/xmm2, r9/xmm3	RTL (C)	Caller	Stack aligned on 16 bytes. 32 bytes shadow space on stack. The specified 8 registers can only be used for parameter number 1, 2, 3, and 4.
64-bit	System V AMD64 ABI convention^[9]	Linux, BSD, Mac (GCC, Intel compiler)	rdi, rsi, rdx, rcx, r8, r9, xmm0–7	RTL (C)	Caller	Stack aligned on 16 bytes. Red zone below stack.