Shared Libraries: Understanding Dynamic Loading
In this post, I will attempt to explain the inner workings of how dynamic loading of shared libraries works in Linux systems. This post is long - for a TL;DR, please read the debugging cheat sheet.
This post is not a how-to guide, although it does show how to compile and debug shared libraries and executables. It’s optimized for understanding of the inner workings of how dynamic loading works. It was written to eliminate my knowledge debt on the subject, in order to become a better programmer. I hope that it will help you become better, too.
- What Are Shared Libraries?
- Example Setup
- Compiling a Shared Library
- Compiling and Linking a Dynamic Executable
- ELF - Executable and Linkable Format
- Direct Dependencies
- Runtime Search Path
- Fixing our Executable
- rpath and runpath
- Runtime Search Path: Security
- Debugging Cheat Sheet
What Are Shared Libraries?
A library is a file that contains compiled code and data. Libraries in general are useful because they allow for fast compilation times (you don’t have to compile all sources of your dependencies when compiling your application) and modular development process. Static Libraries are linked into a compiled executable (or another library). After the compilation, the new artifact contains the static library’s content. Shared Libraries are loaded by the executable (or other shared library) at runtime. That makes them a little more complicated in that there’s a whole new field of possible hurdles which we will discuss in this post.
To explore the world of shared libraries, we’ll use one example throughout this post. We’ll start with three source files:
main.cpp will be the main file for our executable. It won’t do much - just call a function from a
random library which we’ll compile:
random library will define a single function in its header file,
It will provide a simple implementation in its source file,
Note: I’m running all of my examples on Ubuntu 14.04.
Compiling a Shared Library
Before compiling the actual library, we’ll create an object file from
In general, build tools don’t print to the standard output when everything is okay. Here are all the parameters explained:
-o random.o: Define the output file name to be
-c: Don’t attempt any linking (only compile).
random.cpp: Select the input file.
Next, we’ll compile the object file into a shared library:
The new flag is
-shared which specifies that a shared library should be built. Notice that we called the shared library
librandom.so. This is not arbitrary - shared libraries should be called
lib<name>.so for them to link properly later on (as we’ll see in the linking section below).
Compiling and Linking a Dynamic Executable
First, we’ll create a shared object for
This is exactly the same as before with
random.o. Now, we’ll try to create an executable:
Okay, so we need to tell
clang that we want to use
librandom.so. Let’s do that1:
Hmmmmph. We told our compiler we want to use a
librandom file. Since it’s loaded dynamically, why do we need it in compile time? Well, the reason is that we need to make sure that the libraries we depend on contain all the symbols needed for our executable. Also note that we specified
random as the name of the library, and not
librandom.so. Remember there’s a convention regarding library file naming? This is where it’s used.
So, we need to let
clang know where to search for shared libraries. We do this with the
-L flag. Note that paths specified by
-L only affect the search path when linking - not during runtime. We’ll specify the current directory:
Great. Now let’s run it!
This is the error we get when a dependency can’t be located. It will happen before our application even runs one line of code, since shared libraries are loaded before symbols in our executable.
This raises several questions:
- How does
mainknow it depends on
- Where does
- How can we tell
mainto look for
librandom.soin this directory?
To answer these question, we’ll have to go a little deeper into the structure of these files.
ELF - Executable and Linkable Format
The shared library and executable file format is called ELF (Executable and Linkable Format). If you check out the Wikipedia article you’ll see that it’s a hot mess, so we won’t go over all of it. In summary, an ELF file contains:
- ELF Header
- File Data, which may contain:
- Program header table (a list of segment headers)
- Section header table (a list of section headers)
- Data pointed to by the above two headers
The ELF header specifies the size and number of segments in the program header table and the size and number of sections in the section header table. Each such table consists of fixed size entries (I use entry to describe either a segment header or a section header in the appropriate table). Entries are headers and they contain a “pointer” (an offset in the file) to the location of the actual body of the segment or section. That body exists in the data part of the file. To make matters more complicated - each section is a part of a segment, and a segment can contain many sections.
In effect, the same data is referenced as either part of a segment or a section depending on the current context. sections are used when linking and segments are used when executing.
readelf to… well, read the ELF. Let’s start by looking at the ELF header of
We can see that this is an ELF file (64-bit) on Unix. Its type is
EXEC, which is an executable file - as expected. It has 9 program headers (meaning it has 9 segments) and 30 section headers (i.e., sections).
Next up - program headers:
Again, we see that we have 9 program headers. Their types are
LOAD (two of those),
NOTE, etc. We can also see the section ownership of each segment.
Finally - section headers:
I trimmed this one for the sake of brevity. We see our 30 sections listed with various names (e.g.,
.note.ABI-tag) and types (e.g.,
You might be confused by now. Don’t worry - it won’t be on the test. I’m explaining this because we’re interested in a specific part of this file: In their Program Header Table, ELF files can have (and shared libraries in particular must have) a segment header that describes a segment of type
PT_DYNAMIC. This segment owns a section called
.dynamic which contains useful information to understand dynamic dependencies.
We can use the
readelf utility to further explore the
.dynamic section of our executable2. In particular, this section contains all of the dynamic dependencies of our ELF file. We only specified
librandom.so as a dependency, so we would expect there to be exactly one dependency listed:
We can see
librandom.so, which we specified, but we also get four extra dependencies we didn’t expect. These dependencies seem to appear in all compiled shared libraries. What are they?
libstdc++: The standard C++ library.
libm: A library that contains basic math functions.
libgcc_s: The GCC (GNU Compiler Collection) runtime library.
libc: The C library: the library which defines the ‘system calls’ and other basic facilities such as
Okay - so we know that
main knows it depends on
librandom.so. So why can’t
librandom.so in runtime?
Runtime Search Path
ldd is a tool that allows us to see recursive shared library dependencies. That means we can see the complete list of all shared libraries an artifact needs at runtime. It also allows us to see where these dependencies are located. Let’s run it on
main and see what happens:
Right off the bat, we see that
librandom.so is listed - but not found. We can also see that we have two additional libraries (
ld-linux-x86-64). They are indirect dependencies. More importantly, we see that
ldd reports the location of the libraries. Consider
ldd reports its location to be
/usr/lib/x86_64-linux-gnu/libstdc++.so.6. How does it know?
Each shared library in our dependencies is searched in the following locations3, in order:
- Directories listed in the executable’s
- Directories in the
LD_LIBRARY_PATHenvironment variable, which contains colon-separated list of directories (e.g.,
- Directories listed in the executable’s
- The list of directories in the file
/etc/ld.so.conf. This file can include other files, but it is basically a list of directories - one per line.
- Default system libraries - usually
/usr/lib(skipped if compiled with
Fixing our Executable
Alright. We validated that
librandom.so is a listed dependency, but it can’t be found. We know where dependencies are searched for. We’ll make sure that our directory is not actually on the search path by using
I trimmed the output since it’s very… chatty. It’s no wonder our shared library is not found - the directory where
librandom.so is located is not in the search path! The most ad-hoc way to solve this is to use
It works, but it’s not very portable. We don’t want to specify our lib directory every time we run our program. A better way is to put our dependencies inside the file.
rpath and runpath
runpath are the most complex items in our runtime search path “checklist”. The
runpath of an executable or shared library are optional entries in the
.dynamic section we reviewed earlier4. They are both a list of directories to search for.
The only difference between
runpath is the order they are searched in. Specifically, their relation to
rpath is searched in before
runpath is searched in after. The meaning of this is that
rpath cannot be changed dynamically with environment variables while
rpath into our executable and see if we can get it to work:
-Wl flag passes the following, comma-separated, flags to the linker. In this case, we pass
-rpath .. To set
runpath instead, we would also have to pass
--enable-new-dtags5. Let’s examine the result:
The executable runs, but this added
. to the
rpath, which is the current working directory. This means it won’t work from a different directory:
We have several ways to solve this. The easiest way is to copy
librandom to a directory that is in our search path (such as
/lib). A more complicated way, which, obviously, is what we’re going to do - is to specify
rpath relative to the executable.
runpath can be absolute (e.g.,
/path/to/my/libs/), relative to the current working directory (e.g.,
.), but they can also be relative to the executable. This is achieved by using the
$ORIGIN variable6 in the
Notice that we need to escape the dollar sign (or use single quotes), so that our shell won’t try to expand it. The result is that
main works from every directory and finds
Let’s use our toolkit to make sure:
Runtime Search Path: Security
If you ever changed your Linux user password from the command line, you may have used the
The password hash is stored in
/etc/shadow, which is root protected. How then, you might ask, your non-root user can change that file?
The answer is that the
passwd program has the
setuid bit set, which you can see with
s (the fourth character of the line). All programs that have this permission bit set run as the owner of that program. In this example, the user is root (third word of the line).
“What does that have to do with shared libraries?”, you ask. We’ll see with an example.
We’ll now have
librandom in a
libs directory next to
main and we’ll bake
$ORIGIN/libs7 in our
If we run
main, it works as expected. Let’s turn on the
setuid bit for our
main executable and make it run as
rpath doesn’t work. Let’s try setting
What’s going on here?
For security reasons, when running an executable with elevated privileges (such as
setgid, special capabilities, etc.), the search path list is different than normal:
LD_LIBRARY_PATH is ignored, as well as any path in
runpath that contains
The reason is that using these search path allows to exploit the elevated privileges executable to run as
root. Details about this exploit can be found here. Basically, it allows you to make the elevated privileges executable load your own library, which will run as root (or a different user). Running your own code as root pretty much gives you absolute control over the machine you’re using.
If your executable needs to have elevated privileges, you’ll need to specify your dependencies in absolute paths, or place them in the default locations (e.g.,
An important behavior to note here is that, for these kind of applications,
ldd lies to our face:
ldd doesn’t care about
setuid and it expands
$ORIGIN when it is searching for our dependencies. This can be quite a pitfall when debugging dependencies on
Debugging Cheat Sheet
If you ever get this error when running an executable:
You can try doing the following:
- Find out what dependencies are missing with
- If you don’t identify them, you can check if they are direct dependencies by running
readelf -d <executable> | grep NEEDED.
- Make sure the dependencies actually exist. Maybe you forgot to compile them or move them to a
- Find out where dependencies are searched by using
LD_DEBUG=libs ldd <executable>.
- If you need to add a directory to the search:
- Ad-hoc: add the directory to the
- Baked in the file: add the directory to the executable or shared library’s
$ORIGINfor paths relative to the executable.
- Ad-hoc: add the directory to the
lddshows that no dependencies are missing, see if your application has elevated privileges. If so,
lddmight lie. See security concerns above.
If you still can’t figure it out - you’ll need to read the whole thing again :)
- “ELF (Execuable and Linkable Format)”/ Wikipedia
- “Linker and Libraries Guide” / Oracle
- The GNU C Library (glibc)
- “Shared Libraries” / The Linux Documentation Project
- “Where do executables look for shared objects at runtime” / Unix & Linux SE
- “Application Binary Interface”
- “The ELF format - how programs look from the inside” / Christian Aichinger
- “Rpath” / Wikipedia
- “GNU Dynamic Loader Search Directories” / TechBlog
- “ld.so: Dynamic Link library support for the Linux OS”
- “How does the ‘passwd’ command gain root user permissions?” / Unix & Linux SE
- “ELF Object File Format” / nairobi-embedded
Note that we chose to dynamically link
main. It’s possible to do this statically - and load all of the symbols in the
randomlibrary directly into the
objdumpexecutable can provide similar results. In this case for example:
objdump -p librandom.so | grep NEEDEDwill print very similar output. ↩︎
The search path is different for
setguidapplications. See below for more details. ↩︎
rpath’s type is
runpath’s type is
$ORIGINis not an environment variable. If you
export ORIGIN=/pathit will have no effect. It is always the directory in which the executable is placed. ↩︎
For some strange reason, this example didn’t behave as I expected when
librandom.sowas in the same directory as
rpath. If you know why this is, please answer my question about it in Stack Overlow. ↩︎
Follow me on Twitter, Facebook or Google+
Thanks to Hannan Aharonov, Yonatan Nakar and Shachar Ohana for reading drafts of this.