Bouncer Project Road map
History
Mozilla
Bouncer's development has had an interesting path over the last few years. Essentially two versions of the project split up out of the needs of what the Mozilla Project needed, and what the OSL had developed. Once Bouncer development at the OSL slowed, more progress happened on the version that Mozilla had been working on. They tried to implement the 2.0 version that the OSL had developed, but it didn't meet their needs for two reasons:
- Load issues
- Lack of the ability to automate
From Mozilla's point of view, bouncer2 was over-engineered and so they opted to use a branch from the 1.0 tag and create the features that Mozilla wanted and needed. At Mozilla, they are using the 1.0 branch and the feature list they currently have is (as of October 2007):
- Mirror uptake -- knowing what % of available bandwidth has a particular file (code complete and in production)
- Memcache support (code complete and in prod)
- Locale support (code complete)
- Hash checking using byte ranges (code complete)
- Geo-ip location support (in progress)
- Peer network support (not started)
OpenOffice.org
OpenOffice.org has been using bouncer2 for over the last few years which has worked well but has shown some issues with regard to scalability. Some of the issues for OpenOffice.org has been the following:
- Full check requires a full download of all the sources (very very slow)
- Having multiple versions/locales of their packages in bouncer2 created scalability issues
- Sometimes bouncer wouldn't enable a new product in a timely manner
Virtual Filesystem
First off is the idea of bouncer as virtual filesystem. Bouncer can easily get a list files available. The Mozilla fork needs a local copy of the mirror in order to do hash checks (since they are partial file, it cannot be stored in the DB like v2 does). Even if a local copy is unavailable (perhaps a certain install may only want per-file touch checks), the file list could still be built via a one-argument rsync call to the master mirror. So, how would this change operation? Rather than adding new product/version/os/lang tuples, tied to a mirror location that would then be checked, the new files can be detected automatically and added. Sentry could then automatically start polling remote mirrors to see which files match.
For simple mirrors, like Mozilla, where the structure is well standardized, this would be all they would need. The change would just be made in the original download chooser on the site to point to a file path on the bouncer server instead of passing args:
http://download.server/path/to/file (redirects to) http://some.mirror/base/path/to/file
The idea of having a p/v/os/l tuple that ties to a filename could still be implemented, and would be in a sense an extra redirect on the bouncer side, /query?args -> /path/to/file. The advantage with moving the template layer `higher' would be that:
- Clients like Mozilla that don't need it don't have to use it
- The templates don't have to be used in sentry runs, just for download bounces -- sentry just needs to do filesystem-to-filesystem comparisons (think simple)
Multiple Projects within one Bouncer
Another feature (transparent if you don't want it) is for multiple projects within a single bouncer install. This basically means that for different subdirectories in your filesystem, you can have different mirror sets and p/v/os/l links. Think using bouncer on a general-purpose mirror that hosts lots of projects. If your bandwidth started getting taxed, you could pass off requests to other mirrors.
Tiered Mirroring
The ability to manage multiple sub-projects within a single instance of bouncer. This feature is also optional.
The tier system has two purposes:
- Failover to non-primary mirrors (like other people's mirrors) only if you run out of bandwidth
- A failover for the GeoIP setup so that at some capacity you fail over into a different zone. So, if South America only has two mirrors, it will start passing requests to North America only once the `capacity' is used up.
