c-stack-cdecl assumes arguments are passed on stack

[nikomatsakis]

In the x86_64 architecture (and others as well), the first few integer arguments are passed in registers rather than being pushed on the stack. Currently, c-stack-cdecl works by allocating some space on the C stack and then writing the arguments into that space. A small shim (upcall_call_c_stack()) then switches the stack pointer over and invokes the function, which will then find the arguments on the stack as expected. Naturally this fails when the function doesn't expect its arguments on the stack. For now, I have worked around the issue by having upcall_call_c_stack() on x86_64 load the registers from the stack, but this will only work for very simple cases. In an ideal world, LLVM should handle the details of the calling convention for us, just as it does for a typical "invoke" instruction. However, this is a bit tricky since the default invoke instruction does not switch stacks!

One possible solution would be to generate a shim function S for each function F that we would like to call on the c-stack. If F takes arguments of type X and Y and returns a type Z, the shim function would work like:

void S(struct F_Args { X x; Y y; Z *z; } *args) {
    *args->z = F(args->x, args->y);
}

We can then have some hand-coded assembly that swaps the stack pointer and invokes S on the C-stack. S has a known signature so this should be easy. LLVM ought to then generate the right code to unpack the arguments and put them where they belong. The downside of this trick is that this results in copying the arguments more than is needed.

Another option might be to add a new calling convention to LLVM. The problem is we would need to generate a new calling convention for each target architecture. On the other hand, that may not be such a big hurdle. I don't know how one defines calling conventions in LLVM or how flexible that system is.

[nikomatsakis]

(I tagged this as blocker as it blocks the x64 port)

[brson]

Can the LLVM split stacks stuff handle this for us? Can we use that shim to tell LLVM we need more stack and then hand it the C stack?

[nikomatsakis]

An interesting idea… I don't know how the split stacks stuff works well enough but it sounds plausible. The shim would probably work somewhat differently, more like:

Z S(X x, Y y) {
    <magic prologue that switches stack>
    return F(x, y);
}

which is of course just fine...

On Nov 9, 2011, at 1:41 PM, Brian Anderson wrote:

Can the LLVM split stacks stuff handle this for us? Can we use that shim to tell LLVM we need more stack and then hand it the C stack?

---

Reply to this email directly or view it on GitHub:
https://github.com/graydon/rust/issues/1163#issuecomment-2687849

[nikomatsakis]

I implemented the stub function solution I proposed. It works. Not yet pushed to master however.