[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Proposal for binup design document
Hello
I really think binup could be useful and fill a gap for FreeBSD.
Therefor to try to get things moving and to get some kind of structure
on binup I have begun make a kind of design document containing some
thought / ideas about binup. I think it is important to discuss these
issues (especially the HTTP proposal) so we know what to make binup do
and how it should be done.
I have a attached a text only version of the document, but it has been
written in HTML and can be found at
http://simon.nitro.dk/binup/binup-2003.html .
I plan to start looking at the server database backend today and
adding/changing the binup design document along the way.
--
Simon L. Nielsen
Binup 2003 proposal
Contents
* [1]Introduction
* [2]General
* [3]HTTP
* [4]Main use
* [5]Package abstraction
* [6]Package the base system
* [7]Client/server communication
* [8]Tasks
Updated : $Date: 2002/12/27 21:17:46 $
binup 2003 introduction
This is a proposal by Simon L. Nielsen <[9]simon@xxxxxxxx> for a
design document for the FreeBSD Binary Updater (binup). The
document contains some ideas for changes to the current binup
code/design and the beginning to a general design document about
binup.
The document is very much work in progress... and a bit of a mess
at the moment.
I have called this document binup 2003 to differentiate it from
the current "old" binup code. The [10]"old" binup document should
be read before this document since the main design and goals
haven't changed.
Please note that this document is meant as an early draft and most
likely contains flaws, bad ideas and things that have not been
thought fully through. There is also most likely a lot of spelling
and grammatical errors - please ignore them for now. It contains
comments/ideas on the subjects I think needs to be addressed
before real coding on binup 2003 can begin.
I would very much like comments on the document and the ideas. I
would prefer that comments are posted to the [11]binup
mailing-list for discussion but private comments directly to me
are also very welcome.
I plan to keep an updated version of this document on
[12]http://simon.nitro.dk/binup/binup-2003.html.
General
I think it would be a good idea to change the binup system to use
HTTP instead of doing a new TCP level protocol. Mainly because I
think this could save a lot of coding. See the [13]HTTP section
for more.
As suggested on the binup mailing-list some time ago I think binup
should be made independent of the packages actually transfered by
binup. This would make it possible to use distribution tar-balls,
.tgz's, libh, binary patches or any other packaging format without
any major change to binup. See the [14]package abstraction section
for more.
There have been talk about making the binup project do a package
API (/var/db/pkg API) and do the breaking up the base system into
packages. While I think these are good ideas that would certainly
benefit binup I don't think we should wait for this to be finished
since that may take a long time and it should be possible to do a
good binary updated without these tasks completed.
HTTP
The current binup code uses a special protocol on top of TCP. This
makes the protocol more flexible but also require a lot of code to
get working. Instead binup could be implemented on top of HTTP.
This would make some things a lot simpler since a lot of existing
programs/code can be used both server and client side. The use of
HTTP will simplify implementation since a lot of functionality
will come "for free" e.g. file transfer, SSL and connection
management. Standard server/clients also gives the advantage that
less low level C code has to be written and therefore minimizing
the risk for buffer overflows and other common C bugs.
Using HTTP it would also be quite simple for the binup server to
point a client at a normal FreeBSD mirror for the actual
downloading of updated packages (or perhaps the client can pick
it's preferred mirror site). A checksum/crypto signature could
still be send from the binup server and therefore increasing
security a bit for trojan-ed files. This might not be a big
increase - but it would help a bit.
The main drawback of using HTTP is the lack of state between
requests but I don't see this as a major drawback since most the
needed state code can be kept client side instead.
Advantages of using HTTP :
* Less reinventing the wheel
* Will be using standard Internet protocol (big advantage if
client is behind a firewall)
* Simpler prototyping (a lot of the code can be initially
implemented in a scripting language)
* Less low level C code
Disadvantages of using HTTP :
* No state
* Will mean scraping most of the current binup code
Main binup uses
This section contains some thoughts on where/how binup could be
useful. I think this is important to keep in mind to determine
exactly whan binup is supposed to do.
The main uses for binup as I see it is (in more or less
prioritized order) :
1. Security updates (e.g. 4.7p1 -> 4.7p2)
2. Port updates
3. FreeBSD upgrades (e.g. 4.6 -> 4.7)
4. Fresh FreeBSD installs with profiles
I see two main types of operation for a binup server - private and
public servers.
A private server can for example be used for a company to have a
simple way to deploy updates to a server park. In this scenario
updated will most likely have to be tested before being deployed
widespread in the administrators must therefor have ability to add
a package to be updated on a per package basis (package in a
general sense both OS upgrade and port upgrade).
The public server could be used as a kind of replacement for
buildworld/installworld for people who just want to do a network
upgrade. This could also be very useful for people who want to
keep up to date with security fixes without having to recompile
everything for a "simple" small patch.
When clients do an update the binup client should allow this to be
done either automatically or in an interactive mode.
Package abstraction
I belive it would be useful for binup only to have a minimal
"knowledge" of the packages being transfered. That way binup is
not being bound to a specific packing format which might change
once libh is fully implemented or the base system is packaged in
some special format. This would also alow update packages to be
available in multiple packaging formats which might be useful in
some cases.
It would of course make the tripwire functionality harder to
implement, but it could still be done.
This could mean that binup could be used to transfer all kind of
odd package formats even .rpm, .deb packages if that should be
needed.
binup mainly need three pieces of information for each package as
far as I can see :
1. Package name - must be unique
2. Package format - so the client knows how to install the
package
3. Package version - to find new versions
The real tricky one is package version since third party programs
use all kinds of odd version schemes.
Client/server communication
I think that when a client wants to upgrade the client should get
a list of the most recent packages for a profile the client
specifies and then the client decides which packages to download
and install. This way partial upgrades should be handles
automatically and the privacy of the client is preserved since the
server does not know which packages the client does actually have
installed.
The client -> server can be done by a simple HTTP GET or POST with
the needed variables.
For the server -> client data there must be some kind of
formatting/encoding so the client can parse it in a simple way.
The main formatting/encoding styles I see are comma separated or
XML.
XML is very general and easy to extend later but it has a few
major drawbacks. The main drawback is the requirement of an XML
parser client side and I think the binup client should depend on
as few external components as possible to minimize problems when
blowing things away under the client and when doing from scratch
installs. Another drawback for XML is the extra overhead both from
larger network transfers and from the parsing.
Comma separated would require special parsing code to be written
and would most likely break if new fields are added to the lines.
The advantage is that it requires no external library to parse and
it is rather fast to parse.
I lean mostly to comma separated format since it does not require
a XML library on the client...
binup tasks
The tasks to get a complete binup system as I see it (not all may
be required to get first version of binup working)
* Find out what binup 2003 should really do
* Specify client/server protocol
* binup server
+ Create database back-end
+ Implement server in some scripting language
(perl/PHP/???)
+ Implement server in C (perhaps apache module ?)
* binup client
* package up base system
+ Package lib API (API for /var/db/pkg)
+ Find out how to do versioning (Problem with source
installs)
Implementation notes
Database back-end should be implemented in server so it can be
replaced later e.g. with dbm files if needed.
$Id: binup-2003.html,v 1.6 2002/12/27 21:17:46 simon Exp $
References
1. #intro
2. #general
3. #http
4. #main-use
5. #package-abstraction
6. #package-base
7. #client-server-comm
8. #tasks
9. mailto:simon@xxxxxxxx
10. http://www.freebsd.org/projects/updater.html
11. mailto:freebsd-binup@xxxxxxxxxxx
12. http://simon.nitro.dk/binup/binup-2003.html
13. #http
14. #package-abstraction
Attachment:
pgpAmsZtWdKuE.pgp
Description: PGP signature
