At its core, Phusion Passenger is an HTTP proxy and process manager. It spawns Ruby on Rails/Rack/WSGI worker processes (which may also be referred to as backend processes), and forwards incoming HTTP request to one of the worker processes.
While this may sound simple, there’s not just one way to spawn worker processes. Let’s go over the different spawning methods. For simplicity’s sake, let’s assume that we’re only talking about Ruby on Rails applications.
Phusion Passenger could create a new Ruby process, which will then load the Rails application along with the entire Rails framework. This process will then enter an request handling main loop.
This is the most straightforward way to spawn worker processes. If you’re familiar with the Mongrel application server, then this approach is exactly what mongrel_cluster performs: it creates N worker processes, each which loads a full copy of the Rails application and the Rails framework in memory. The Thin application server employs pretty much the same approach. Note that Phusion Passenger’s version of conservative spawning differs slightly from mongrel_cluster. Mongrel_cluster creates entirely new Ruby processes. In programmers jargon, mongrel_cluster creates new Ruby processes by forking the current process and exec()-ing a new Ruby interpreter. Phusion Passenger on the other hand creates processes that reuse the already loaded Ruby interpreter. In programmers jargon, Phusion Passenger calls fork(), but not exec().
Smart spawning is supported for all Ruby applications but not for WSGI applications.
While conservative spawning works well, it’s not as efficient as it could be because each worker process has its own private copy of the Rails application as well as the Rails framework. This wastes memory as well as startup time.
Figure: Worker processes and conservative spawning. Each worker process has its own private copy of the application code and Rails framework code.
It is possible to make the different worker processes share the memory occupied by application and Rails framework code, by utilizing so-called copy-on-write semantics of the virtual memory system on modern operating systems. As a side effect, the startup time is also reduced. This is technique is exploited by Phusion Passenger’s smart and smart-lv2 spawn methods.
12.2.1. How it works
When the smart-lv2 spawn method is being used, Phusion Passenger will first create a so-called ApplicationSpawner server process. This process loads the entire Rails application along with the Rails framework, by loading environment.rb. Then, whenever Phusion Passenger needs a new worker process, it will instruct the ApplicationSpawner server to do so. The ApplicationSpawner server will create a worker new process that reuses the already loaded Rails application/framework. Creating a worker process through an already running ApplicationSpawner server is very fast, about 10 times faster than loading the Rails application/framework from scratch. If the Ruby interpreter is copy-on-write friendly (that is, if you’re running Ruby Enterprise Edition) then all created worker processes will share as much common memory as possible. That is, they will all share the same application and Rails framework code.
Figure: Worker processes and the smart-lv2 spawn method. All worker processes, as well as the ApplicationSpawner, share the same application code and Rails framework code.
The smart spawn method goes even further, by caching the Rails framework in another process called the FrameworkSpawner server. This process only loads the Rails framework, not the application. When a FrameworkSpawner server is instructed to create a new worker process, it will create a new ApplicationSpawner to which the instruction will be delegated. All those ApplicationSpawner servers, as well as all worker processes created by those ApplicationSpawner servers, will share the same Rails framework code. The smart-lv2 method allows different worker processes that belong to the same application to share memory. The smart method allows different worker processes - that happen to use the same Rails version - to share memory, even if they don’t belong to the same application.
Notes:
Vendored Rails frameworks cannot be shared by different applications, even if both vendored Rails frameworks are the same version. So for efficiency reasons we don’t recommend vendoring Rails.
ApplicationSpawner and FrameworkSpawner servers have an idle timeout just like worker processes. If an
ApplicationSpawner/FrameworkSpawner server hasn’t been instructed to do anything for a while, it will be shutdown in order to conserve memory. This idle timeout is configurable.
12.2.2. Summary of benefits
Suppose that Phusion Passenger needs a new worker process for an application that uses Rails 2.2.1.
If the smart-lv2 spawning method is used, and an ApplicationSpawner server for this application is already running, then worker process creation time is about 10 times faster than conservative spawning. This worker process will also share application and Rails framework code memory with the ApplicationSpawner server and the worker processes that had been spawned by this ApplicationSpawner server.
12.3. Smart spawning gotcha #1: unintential file descriptor sharing
ApplicationSpawner server for this application is running, then worker process creation time is about 2 times faster than conservative spawning. If there is an ApplicationSpawner server for this application running, then worker process creation time is about 10 times faster. This worker process will also share application and Rails framework code memory with the
ApplicationSpawner and FrameworkSpawner servers.
You could compare ApplicationSpawner and FrameworkSpawner servers with stem cells, that have the ability to quickly change into more specific cells (worker process).
In practice, the smart spawning methods could mean a memory saving of about 33%, assuming that your Ruby interpreter is copy-on-write friendly.
Of course, smart spawning is not without gotchas. But if you understand the gotchas you can easily reap the benefits of smart spawning.
Because worker processes are created by forking from an ApplicationSpawner server, it will share all file descriptors that are opened by the ApplicationSpawner server. (This is part of the semantics of the Unix fork() system call. You might want to Google it if you’re not familiar with it.) A file descriptor is a handle which can be an opened file, an opened socket connection, a pipe, etc. If different worker processes write to such a file descriptor at the same time, then their write calls will be interleaved, which may potentially cause problems.
The problem commonly involves socket connections that are unintentially being shared. You can fix it by closing and reestablishing the connection when Phusion Passenger is creating a new worker process. Phusion Passenger provides the API call
PhusionPassenger.on_event(:starting_worker_process) to do so. So you could insert the following code in your
environment.rb:
if defined?(PhusionPassenger)
PhusionPassenger.on_event(:starting_worker_process) do |forked| if forked
# We're in smart spawning mode.
... code to reestablish socket connections here ...
else # We're in conservative spawning mode. We don't need to do anything. end end end Note that Phusion Passenger automatically reestablishes the connection to the database upon creating a new worker process, which is why you normally do not encounter any database issues when using smart spawning mode. 12.3.1. Example 1: Memcached connection sharing (harmful) Suppose we have a Rails application that connects to a Memcached server in environment.rb. This causes the ApplicationSpawner to have a socket connection (file descriptor) to the Memcached server, as shown in the following figure: +---+
| ApplicationSpawner |---[Memcached server] +---+
Phusion Passenger then proceeds with creating a new Rails worker process, which is to process incoming HTTP requests. The result will look like this: +---+
| ApplicationSpawner |---+----[Memcached server] +---+ |
|
+---+ |
| Worker process 1 |---/
+---+
Since a fork() makes a (virtual) complete copy of a process, all its file descriptors will be copied as well. What we see here is that ApplicationSpawner and Worker process 1 both share the same connection to Memcached. Now supposed that your site gets Slashdotted and Phusion Passenger needs to spawn another worker process. It does so by forking ApplicationSpawner. The result is now as follows: +---+
| ApplicationSpawner |---+----[Memcached server] +---+ | | +---+ | | Worker process 1 |---/| +---+ | | +---+ | | Worker process 2 |---/ +---+
As you can see, Worker process 1 and Worker process 2 have the same Memcache connection.
12.4. Smart spawning gotcha #2: the need to revive threads
12.5. Smart spawning gotcha #3: code load order
is handled by Worker process 2. Both worker processes want to fetch something from Memcached. Suppose that in order to do that, both handlers need to send a "FETCH" command to Memcached.
But suppose that, after worker process 1 having only sent "FE", a context switch occurs, and worker process 2 starts sending a "FETCH" command to Memcached as well. If worker process 2 succeeds in sending only one bye, F, then Memcached will receive a command which begins with "FEF", a command that it does not recognize. In other words: the data from both handlers get
interleaved. And thus Memcached is forced to handle this as an error.
This problem can be solved by reestablishing the connection to Memcached after forking:
+---+
| ApplicationSpawner |---+----[Memcached server] +---+ | | | | +---+ | | | Worker process 1 |---/| | +---+ | | <--- created this X | new | connection X <-- closed this | +---+ | old |
| Worker process 2 |---/ connection |
+---+ |
| |
+---+
Worker process 2 now has its own, separate communication channel with Memcached. The code in environment.rb looks like this: if defined?(PhusionPassenger) PhusionPassenger.on_event(:starting_worker_process) do |forked| if forked # We're in smart spawning mode. reestablish_connection_to_memcached else # We're in conservative spawning mode. We don't need to do anything. end end end 12.3.2. Example 2: Log file sharing (not harmful) There are also cases in which unintential file descriptor sharing is not harmful. One such case is log file file descriptor sharing. Even if two processes write to the log file at the same time, the worst thing that can happen is that the data in the log file is interleaved. To guarantee that the data written to the log file is never interleaved, you must synchronize write access via an inter-process synchronization mechanism, such as file locks. Reopening the log file, like you would have done in the Memcached example, doesn’t help. Another part of the fork() system call’s semantics is the fact that threads disappear after a fork call. So if you’ve created any threads in environment.rb, then those threads will no longer be running in newly created worker process. You need to revive them when a new worker process is created. Use the :starting_worker_process event that Phusion Passenger provides, like this: if defined?(PhusionPassenger) PhusionPassenger.on_event(:starting_worker_process) do |forked| if forked # We're in smart spawning mode. ... code to revive threads here ... else
# We're in conservative spawning mode. We don't need to do anything. end
end end
This gotcha is only applicable to the smart spawn method, not the smart-lv2 spawn method.
If your application expects the Rails framework to be not loaded during the beginning of environment.rb, then it can cause problems when an ApplicationSpawner is created from a FrameworkSpawner, which already has the Rails framework loaded. The most common case is when applications try to patch Rails by dropping a modified file that has the same name as Rails’s own file, in a path that comes earlier in the Ruby search path.
For example, suppose that we have an application which has a patched version of active_record/base.rb located in RAILS_ROOT/lib /patches, and RAILS_ROOT/lib/patches comes first in the Ruby load path. When conservative spawning is used, the patched version of base.rb is properly loaded. When smart (not smart-lv2) spawning is used, the original base.rb is used because it was already loaded, so a subsequent require "active_record/base" has no effect.