Arkanis Development

Great benchmark thanks, interesting results!

Thanks for sharing your strange little experiment. It was indeed an interesting read.

Thanks for posting your results, very interesting indeed.

This is fantastic, never finish learning. Thanks.

Great experiment and very lucky to read!

Thank you for your research.

Not that it invalidates your results but getpid() is not in the vDSO but instead cached by glibc (not that it would lead to any difference in performance.)

However for anybody looking at reproducing your results they should be aware that getpid() is no longer cached by glibc starting with 2.25.

Some explanation can be found in the RH bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1469670

The list of functions in the vDSO are architecture dependent but can be found in the man page.

Thanks for the heads up and the link Rob! I liberated the idea to use getpid() as a test from the Exokernel paper and blindly assumed Linux would do the same. Kind of stupid from me. I should have read the getpid() man page… they even state it there.

Well, that calls for some revising and a new round of benchmarking. :) I think I'll use time() or gettimeofday() next as they're listed in the vDSO man page for x86-64 systems. I don't expect it to change much but that remains to be seen. With a bit of luck I can do a new benchmark in a few months time. Maybe even with some new CPUs…

thank you for those information. i was really in need to read this and get some idea about what happen with hardware test

Good experiment! I'm trying to learn about operating systems, and my most recent topic was system call latency. It was great to see some real numbers about it!

Do you have any `mmap` related benchmark? Or `malloc` related ones? Or `pthread_mutex_lock` related ones?

A big thanks for your benchmark. I am coding [RefPerSys]( https://gitlab.com/bstarynk/refpersys) and that is why I needed them.

Basile Starynkevitch http://starynkevitch.net/Basile/ Bourg La Reine, France <basile@starynkevitch.net>

Hi Basile,

Please note that the current benchmark numbers are not what they appear to be (see comment #6 by Rob Bradford). Unfortunately I haven't had the time since then to write up a new article.

Regarding mmap() or malloc(): Quite difficult. What would a meaningful benchmark of those system calls look like? The performance of malloc() depends heavily on the implementation (does it use linked lists, memory pools, etc.). If you can you should do a micro-benchmark for your target system and situation. With mmap() I guess it depends on how many memory is available right now, how many memory maps there are, and so on. I don't know the implementation though.

I've read the refpersys project page. From that I guess you're primarily interested in garbage collection. In that case you might be interested in the madvise() system call. Specifically the MADV_DONTNEED flag. With that you can allocate one large memory area and invalidate parts of it after garbage collection. MADV_SEQUENTIAL and MADV_NORMAL might also be of interest. Before scanning a GC space you can switch to MADV_SEQUENTIAL (in case you do sequential scanning) and when done you can switch back to MADV_NORMAL. But I haven't uses those two flags myself yet. I've only ever written a Baker GC for a small Lisp interpreter so I don't know how well this applies to other GCs.

pthread_mutex_lock() is another beast entirely. I suppose the performance will depend heavily on the contention of the mutex. As far as I know it's implemented with futexes so it should be fast if only one thread uses it. But it will get slow if many threads try to lock it. There should be some good futex benchmarks out there (I hope).

I've also got some questions regarding refpersys: If every mutable object has its own lock how do you avoid deadlocks? Primarily with more complex objects that consists of many atomic objects. From the "persistent values" section I also got the impression that the language operates on a global memory model. Meaning every variable can be accessed from every tasklet. This will be quite the garbage collection challenge. Not the mention the chaotic memory access patterns and fragmentation (entailing false sharing of cache lines, etc.). Have you considered keeping most variables local to tasklets and only explicitly exchange variables with other tasklets (by transfering ownership, via message passing, etc.)? That way each agenda could run its own thread-local garbage collector and that's way simpler than a multi-threaded one. Also you could do pretty cool stuff like running a tasklet, collect its survivor variables when it's done and store those in the tasklets own private memory area. This would compact the memory for each tasklet and lower the overhread of not running tasklets. But well, I'm dreaming right now…

This got a bit longer than intended. Sorry about that. It's not often that I can talk about those things. :D

Happy programming Stephan

Thanks for sharing your knowledge.

Thanks for your work! To the old write() versus fwrite() discussion. You tell that fwrite() uses a 4kByte buffer. This increases throughput, but increases latency, too. Therefore write() is the low latency solution and fwrite() is the high throughput solution. On the next level of detail you can use fflush() together with fwrite() to have the best of both worlds. On the last level of detail you have to remember that next to the userland buffers of glibc there are kernel buffers. To flush these buffers you use fsync() if it is disk io or ioctl() for network connections. Still, your work is a nice introduction in the topic.

That's a nice way to look at it. Thanks for the compact description Paul. :)

Интересно было бы глянуть те же тесты на современных amd/intel/arm64.

The read/write results say that the vdso doesn't really help anything (except for one case, which is worth an investigation) for those cases. So it's reallly only useful as a unified break-out for the few syscalls that can do without the context switch part*.This stands to reason because on x86_64 vdso will call syscall anyway.

Not so on 32 bit x86: There the interface is to use int 0x80, except where you can break out, or where you can use sysenter or syscall, depending on manufacturer. Which then ought to speed up all syscalls. Except that you can't really because the interfaces are muddled and yucky, after figuring out whether you're on intel or amd. That's not a job for a program, so linux does it through a vdso. Though that too is rather complicated and poorly documented if libc doesn't do it for you. And that means a non-static binary and linking your assembly with libc again. *Sigh.*

So it would be more interesting to benchmark on 32bit linux: Does it pay to use vdso from assembly rather than just use int 0x80?

• Which is one or two, a small handful at most. The one use case that comes to mind is from a solaris(!) report where financial software was found to call time() a few millions of times per second, so speeding that up sped up the software considerably. I consider it a defect of design: Should've just taken one time() value, run a batch of transactions, and take another time() value to confirm the batch was completed within one second, then record that value for all transactions. Rather than several time() values for each transaction, which being single-second granular, will be the same value for the bulk of the transactions anyway.