YottaDB is a mature, hierarchical key-value, free / open source NoSQL database whose is used in enterprise-scale mission-critical applications in banking and healthcare, and also scales down to fit on IoT devices like the Raspberry Pi Zero, as well as applications in-between (like the University of Antwerp library catalog system). When a customer funded us to develop a Go API to YottaDB, we thought it would be a straightforward project. But it was a very painful exercise for us. The presentation discusses the problems we faced, and how we overcame them.
Go is a popular language for writing highly concurrent software, and works well when used in isolation. Using Go alongside software written in other languages (such as C) can be done, but there are some hidden dragons to keep an eye out for. In addition to obvious problems, such as calling C variadic functions, other more subtle problems are hidden deep in the depths of Go documentation.
Consider callback functions; how does one pass a function pointer from Go to C to provide a callback? There are strict limitations enforced by Go on what pointers may be passed to C routines, intended to prevent faults resulting from Go structures being garbage collected without knowledge of the C code. Without being able to pass pointers, even function pointers, how do we “pass” a callback function to the C code to callback into? Furthermore, how do we pass data to the callback function, since we can’t pass Go structures?
Given that we can’t pass Go structures to C code, at some point we will need to allocate C structures to store data for the C code to operate on. Go promises one thing about memory allocated in C land; it will not keep track of it for you. The garbage collector will gladly clean up any Go structures no longer needed, but will not clean up the associated C memory. How can one write code which isn’t likely to result in memory leaks, using this model?
Perhaps the most difficult challenge to overcome is that Go makes no promises about what thread is running code. Go does its best to hide the identity of Go routines from the user, so they won’t rely on this metadata for handling code execution. This presents a problem for many C applications, which often use POSIX mutexes to control access, and the owner of a mutex needs to belong to a specific thread. How can one write Go applications that allow the concurrency Go users expect, without trashing the libraries they are calling?
Of course, none of the solutions we talk about here are any good unless you can compile your program. The hidden dragons of Go also lurk behind the “go build“ command; fitting in the required C flags requires knowledge of, among other things, pkg-config, a systems tool used behind the scenes by programs like CMake and autotools.
We had to tackle all these problems, and many more, during the development of the YottaDB Go wrapper (https://gitlab.com/YottaDB/Lang/YDBGo and https://yottadb.com). This presentation hopes to pass some of our hard-learned lessons to other programmers who will use Go to interface with non-Go libraries and utilities.
