I guess that's not OOM killer.
Just because to my knowledge there's no OOM killer in #Plan9.
You either faced a bug, such as an integer overflow somewhere in the allocation stack (which should be pretty thin, basically libc and _brk syscall) or git9 actually tried to allocate more memory than the physical memory available.
And note: #9front kernel overcommit memory, assuming you won't really load all the pages you ask for. AFAIK, if your process try to access a page and it's not available, then it will be killed (freeing its pages for other processes).
(ALSO: it's has been a while since I debugged this sort of stuffs on 9front... feel free to correct me if something changed or if I misremember things)