Releasing your egg

Releasing your egg

Let's say you've been reading the eggs tutorial and you are almost done writing your first egg. You've tested that it works locally, and now you probably want to make it available to others. If you would like people to be able to install your egg using chicken-install YOUR-EGG, there are a few steps you need to take.

Note for Subversion users

Before going on details on the process for Subversion, please note that hosting new eggs in the CHICKEN central Subversion repository is deprecated. We recommend using your own code server or one of the hosts available on the Internet (e.g., github, bitbucket etc.).

If you host your egg on code.call-cc.org or you host your own, you don't have to do any of the steps below. Instead, you can use the release generating script at call-cc.org. It does all the steps below automatically.

The URI for an automatically generated release-info for egg YOUR-EGG is http://code.call-cc.org/release-info?egg=YOUR-EGG

This URI can be put in the egg-locations file as described under "Publishing your egg".

If you want more control over the process, please read on.

Creating a release-info file

First, you must create a so-called "release-info" file and make it available on a well-known location via HTTP (discussed below). This is a file which describes where the released versions of your egg can be found.

After everything's set up, releasing a new version of your egg is simply a matter of tagging your release with your VCS of choice and adding a line to this file. See below for some tools to automate even this small step.

It looks something like this:

(repo git "git://example.com/{egg-name}.git")   ; optional

(uri targz "http://example.com/{egg-name}/releases/{chicken-release}/{egg-name}-{egg-release}.tar.gz")
(release "0.1")
(release "0.2")
(release "1.0")

This example describes where to find the canonical repository and an URI pattern which describes how to find release tarballs. It then goes on to declare three official releases; 0.1, 0.2 and 1.0. Please note that releases are not supposed to be removed from the .release-info file, as it would cause consistency issues in case we want to rebuild caches.

The patterns in the URI enclosed in curly braces are seen as substitution patterns to be replaced by values. egg-name expands to the current egg's name (which is already known when the release-info file is being fetched), egg-release is replaced by the string in each release declaration. chicken-release is replaced by the major CHICKEN version for which the egg is being fetched. This allows you to supply different packages for each major CHICKEN release.

The supported types for uri are currently targz, tarbz2, zip, meta-file and files-list. The first three are all archives expected to extract to a directory with the egg sources in it (zip files are allowed to expand directly into the files, but this is not recommended). The latter two are special and deserves some more explanation; see the next section for that.

Currently repo is not used by automated tools, but is intended to make it easy for users to find the repository. In preparation for when it will be used in the future, it's a good idea to use consistent names, so please use the short name of the tool. Generally this is how it is invoked on the commandline: git, hg, svn, fossil, bzr, mtn, darcs etc.

Meta-file distribution

Sometimes it is impractical to create archives containing your egg's file contents. The egg releasing system has been designed to require as few manual steps as possible, so it is easy for people to release early and often.

A core ideal of the release system is to make it possible to release simply by tagging your code in a VCS. Some code hosting sites automatically make archives available for each tagged version, thereby making the release available immediately when the code is pushed to the server. For those that don't, the meta-file distribution file type is a way to release by tagging without having to manually make a tarball, as long as there is a way to directly obtain the raw sources of a specific release version via HTTP.

The idea is that each released egg always must have a meta-file to describe its dependencies, its author and license and so on. This meta-file can be used to provide a listing of all the files that are part of an egg. The system that manages egg releases automatically will download all these files and put them in a directory. When someone requests your egg with chicken-install, these files are all served up. Just add a files entry to your meta-file, which lists the files (these are resolved relatively to the meta-file itself).

In CHICKEN 5

uri-common.egg:

((files "uri-common.release-info" "uri-common.egg"
        "uri-common.scm" "tests/run.scm")
 (license "BSD")
 (category web)
 (components (extension uri-common))
 (dependencies uri-generic defstruct matchable srfi-1 srfi-13 srfi-14)
 (test-dependencies test)
 (author "Peter Bex")
 (synopsis "Parser for common URI schemes"))

uri-common.release-info:

(uri meta-file "http://anonymous@code.call-cc.org/svn/chicken-eggs/release/{chicken-release}/{egg-name}/tags/{egg-release}/{egg-name}.egg")

In CHICKEN 4

uri-common.meta:

((files "uri-common.setup" "uri-common.release-info" "uri-common.meta"
        "uri-common.scm" "tests/run.scm")
 (license "BSD")
 (category web)
 (depends (uri-generic 2.3) defstruct matchable)
 (test-depends test)
 (author "Peter Bex")
 (synopsis "Parser for common URI schemes"))

uri-common.release-info:

(uri meta-file "http://anonymous@code.call-cc.org/svn/chicken-eggs/release/{chicken-release}/{egg-name}/tags/{egg-release}/{egg-name}.meta")

The above is a real-world example of the uri-common egg. It lists all the files which it consists of, and these will be downloaded by chicken-install. The disadvantage of this approach is that if you forget a file, your egg is uninstallable, so if you can please use the archive distribution types instead. Another disadvantage is that every time you add, remove or rename a file you need to remember to change the meta-file.

Because this egg uses subversion and each release has a corresponding tag, it can simply point to the meta-file in the right subdirectory under tags and it will simply work. For other version systems this may require some more messing around.

files-list

This is an extremely stupid format containing a manifest of files in an egg. It's mostly useful when automatically generating lists of files. It starts with a base URI and then lists all the files in the egg. Here's the files-list for version 1.0 of the uri-generic egg for CHICKEN 4:

 http://anonymous:@code.call-cc.org/svn/chicken-eggs/release/4/uri-generic/tags/1.0
 tests/run.scm
 uri-generic.meta
 uri-generic.scm
 uri-generic.setup

The trailing slash after the URI is required.

It's not recommended you use this manually, as this format is subject to change and only really intended to be used as a communication system between pseudo-meta-egg-info and henrietta-cache.

However, if you really need to use this, you can replace the meta-file uri type in the example above by files-list, like the auto-generated release-info file for uri-generic, which starts like this:

 (uri files-list "http://code.call-cc.org/files-list?egg={egg-name};egg-release={egg-release};chicken-release={chicken-release}")
 (release "1.0")
 (release "1.1")

Publishing your egg

After creating the release-info file, you need to make it known to the chicken-install infrastructure that an egg with the given name has a release info file at the location where you published it. This step finally makes it possible for people to say chicken-install YOUR-EGG and have it install your egg, or use (depends YOUR-EGG) in meta-files of their eggs.

Currently this is done by adding a line to the egg-locations file in Subversion, which can be found at https://code.call-cc.org/svn/chicken-eggs/release/5/egg-locations. For security reasons, editing the egg-locations file requires special permissions. So, to have an entry for your egg in that file, send a message to the chicken-users mailing list announcing your egg and requesting the inclusion of its location to egg-locations.

To keep maintenance to a minimum, it's best to add a release-info location which will never change; only the release-info file itself should be changed when you make a new release. This can most easily be accomplished by pointing to the "raw file" view of your trunk/tip/head/master branch in your canonical repository's web interface. That way, this file is kept under version control alongside all the other files in your egg.

If you do have an svn account and don't want to checkout the whole egg repository to be able to just edit this file, you can work around svn's limitations like this:

$ cd /tmp
$ svn co https://code.call-cc.org/svn/chicken-eggs/release/5 --depth empty
$ cd 5
$ svn update egg-locations

If you prefer, you can just publish the file separately via HTTP somewhere and keep overwriting it. If your code hoster doesn't provide an easy way to point to a raw view on a moving "latest version" pointer via HTTP, you could instead opt to store just the release-info files of your eggs in the centralised CHICKEN eggs subversion repository. Just contact one of the project maintainers or write to the chicken-users mailinglist.

Instructions for popular code hosting methods and VCSes

Now we've explained the basic idea, here's an overview of how to figure out the correct URIs and how to automate some steps away.

Bitbucket (mercurial, git)

Location of release-info file

You can use the code browser to figure out the path to your release-info file. First, copy the link that says "raw". This URI is almost correct but will contain a revision ID hash, so it is pinned to whatever revision is currently the tip. However, you can replace it with the string "tip" and it will still work, and when you visit it again after making changes it will have picked up those changes. Example:

Clicking "mini-kanren.release-info" and then copying the "raw" link on https://bitbucket.org/ThatGeoGuy/chicken-minikanren/src gives us https://bitbucket.org/ThatGeoGuy/chicken-minikanren/raw/8af0b51f5efac79c8bc0cfe856185ad06d7aa9d0/mini-kanren.release-info

Just change it to https://bitbucket.org/ThatGeoGuy/chicken-minikanren/raw/master/mini-kanren.release-info and you have the latest master version. If you're using Mercurial instead of git, use tip instead of master.

This link should be registered in the egg-locations list for your egg and it will automatically be able to fetch any new releases as they are tagged and added to the release-info file.

Making releases

The bitbucket code browser also offers a "get source" link that allows you to fetch the files in that revision as a tarball. The same trick as with the raw files works here; just replace the link's revision ID hash with a symbolic name. You can use tags and bookmarks as symbolic names.

So to make a new release, just tag your release with a well-defined name. If you tag your eggs with the version as tag or bookmark name, you can use the following release-info file. Just don't forget to substitute your bitbucket username!

(repo hg "https://bitbucket.org/YOUR-BITBUCKET-USERNAME/{egg-name}")
(uri targz "https://bitbucket.org/YOUR-BITBUCKET-USERNAME/{egg-name}/get/{egg-release}.tar.gz")
(release "0.1")

GitLab (git)

Location of release-info file

Use the code browser ("Files" in the menu) to browse to your master branch's release-info file and copy the "Raw" link. For example, for the SDL2 egg it looks like this:

https://gitlab.com/chicken-sdl2/chicken-sdl2/raw/master/sdl2.release-info

Making releases

GitLab makes tags available for download as tarballs. To find the location, go to the code browser and switch to a tag at the top left. On the right side there's a drop-down menu that offers a download for various archive formats.

So to make a new release, just tag your release with a well-defined name. If you're using gitlab.com's hosted service and tag your eggs with the version as tag name, you can use the following release-info file. Just don't forget to substitute your GitLab project name!

(repo git "https://gitlab.com/PROJECT-NAME/{egg-name}.git")
(uri targz "https://gitlab.com/PROJECT-NAME/{egg-name}/repository/archive.tar.gz?ref={egg-release}")
(release "0.1")

GitHub (git)

Location of release-info file

Use the code browser ("source" tab) to browse to your release-info file and copy the "raw" link. This link should work as-is, as long as you do this while looking at the latest revision of the file in the master branch. For example, for the pastiche egg it would look like:

https://raw.githubusercontent.com/ckeen/pastiche/master/pastiche.release-info-5

Making releases

GitHub makes tags available for download as tarballs. To find the location, click the big "Downloads" button/link in the code browser. It will pop up a selection dialog where you can choose between tarball and "zipball". It will also offer downloads for each tag, but those are zipballs only. However, you can copy the link and replace "zipball" in the URL by "tarball", or you can first click "switch tags" and then open the "Downloads" dialog and it will offer you download links for both tar and zip for that specific tag.

So to make a new release, just tag your release with a well-defined name. If you tag your eggs with the version as tag name, you can use the following release-info file. Just don't forget to substitute your GitHub username!

(repo git "git://github.com/YOUR-GITHUB-USERNAME/{egg-name}.git")
(uri targz "https://codeload.github.com/YOUR-GITHUB-USERNAME/{egg-name}/tar.gz/{egg-release}")
(release "0.1")

Gitweb (git)

Location of release-info file

Navigate to the "tree" view for the master branch of your project and copy the "raw" link for your release-info file. The link will look something like this, for an egg "foo":

http://gitweb/gitweb.cgi?p=foo.git;a=blob_plain;f=foo.release-info

Making releases

The download link for a tarball of a specific Git tag is accessible through Gitweb with a query string like the following:

http://gitweb/gitweb.cgi?p={egg-name}.git;a=snapshot;h=refs/tags/{egg-release};sf=tgz

So, if you tag each release and use a release-info file like the following, every new tag will be made available as a tarball at that URL (make sure to substitute your Git URL and Gitweb host into repo and uri strings):

(repo git "git://gitweb/{egg-name}.git")
(uri targz "http://gitweb/gitweb.cgi?p={egg-name}.git;a=snapshot;h=refs/tags/{egg-release};sf=tgz")
(release "0.1")

Sourcehut (mercurial, git)

Location of release-info file

For a Git repository, navigate to the "tree" view for the master branch of your project, select your release-info file and copy the "View raw" link from that page. The URL will look something like this, for user "user" and egg "example":

https://git.sr.ht/~user/example/blob/master/release-info

For a Mercurial repository, go to your project's "browse" tab, select your release-info file and copy the "View raw" link. The URL should look like this:

https://hg.sr.ht/~user/example/raw/release-info

Making releases

Sourcehut provides download URLs for tags in both Mercurial and Git repositories. So, in either case you can make a new release by simply tagging your egg.

If you use version numbers as tag names, you can use the following release-info files, replacing "USERNAME" with your own. Note the leading tilde, which must be included!

For Git:

(repo git "https://git.sr.ht/~USERNAME/{egg-name}")
(uri targz "https://git.sr.ht/~USERNAME/{egg-name}/archive/{egg-release}.tar.gz")
(release "0.1")

For Mercurial:

(repo hg "https://hg.sr.ht/~USERNAME/{egg-name}")
(uri targz "https://hg.sr.ht/~USERNAME/{egg-name}/archive/{egg-release}.tar.gz")
(release "0.1")

Cgit (git)

Location of release-info file

Navigate to the "tree" view for the master branch of your project and copy the "plain" link for your release-info file (it's all the way to the right). The link will look something like this, for an egg "foo":

http://cgit.example.com/foo/plain/foo.release-info

For example, the release info for the Hardwood egg looks like this:

http://www.upyum.com/cgit.cgi/hardwood/plain/hardwood.release-info

The cgit.cgi part is optional, depending on how cgit was configured. For example, the release-info for the scsh-process egg looks like this:

http://code.more-magic.net/scsh-process/plain/scsh-process.release-info

Making releases

The download link for a tarball of a specific Git tag can be found by visiting "refs", then the tag, following it to the commit and if snapshots are shown there should be a download link, but it links directly to the commit hash. You can simply replace the hash with the symbolic tag name.

If snapshots are not shown, it's still possible to construct the link manually, and it will work:

http://cgit.example.com/cgit.cgi/{egg-name}/snapshot/{egg-name}-{egg-release}.tar.gz

So, if you tag each release and use a release-info file like the following, every new tag will be made available as a tarball at that URL (make sure to substitute your Git URL and Cgit host into repo and uri strings):

(repo git "http://cgit.example.com/cgit.cgi/{egg-name}")
(uri targz "http://cgit.example.com/{egg-name}/snapshot/{egg-name}-{egg-release}.tar.gz")
(release "0.1")

hgweb (mercurial) - aka "hg serve"

The built-in web interface for mercurial can be used to serve up eggs, usually invoked from hgweb and hgwebdir on a "real" server.

Location of release-info file

Use the browser to navigate to the latest revision of the release-info file and click the "raw" link on the left. Then replace the commit ID hash with "tip". Example:

http://example.com/my-egg/raw-file/tip/my-egg.release-info

Making releases

Since this interface doesn't have a way to serve up tarballs, you must either create your own tarballs and publish them elsewhere or use the manual "meta-file" way.

See the meta-file section at the start of this wiki page to figure out how to set up a files section in your meta-file. The URI to your meta file is similar to the release-info file, except this time you can replace "tip" with the release's tagname. If you use tagnames that are identical to the release version, you can base the release-info file on the following example:

(repo hg "http://example.com/{egg-name}")
(uri meta-file "http://example.com/{egg-name}/raw-file/{egg-release}/{egg-name}.egg")
(release "0.1")

This assumes hgwebdir is running on the domain's root URI and the egg is available in a repo which has the same name as your egg.

Fossil's default web UI

This is the web UI you get when running "fossil serve" or when running fossil as a CGI script.

There is one tricky part: by default, this interface disallows almost all outside access; you must log in, even if you just want to browse anonymously. To allow everybody to download release zipfiles or tarballs without login, add the "z" permission for the user "nobody" in the Admin / Users section of the web UI or with the "fossil user capabilities" command.

Location of release-info file

Fossil's web UI allows serving of documents directly from the repository under URLs of the form "http://example.com/my-egg/doc/{version}/{file}". If the files have a ".txt" or ".wiki" extension, they are rendered into HTML, but anything else is served as is.

For example you can store the release-info file in the repository and always access the latest version of the default branch at a URL like "http://example.com/my-egg/doc/trunk/my-egg.release-info".

Making releases

Fossil can serve tarballs or zipfiles of repository snapshots. A release-info file using zipfile packaging may look like this:

(repo fossil "http://example.com/{egg-name}")
(uri zip "http://example.com/{egg-name}/zip/{egg-name}.zip?uuid=v{egg-release}")
(release "1.0.0")

For tarball packaging, the equivalent uri entry should look like this:

(uri targz "http://example.com/{egg-name}/tarball/{egg-name}.tar.gz?uuid=v{egg-release}")

The releases are referenced using symbolic tags of the form "v{egg-release}" here. You can add these tags to the changesets corresponding to released versions of the egg using the "fossil tag add" command.

Launchpad (bazaar)

Unfortunately, Launchpad doesn't currently seem very suitable for developing eggs in a streamlined way. Here's a description of one possible way to integrate with their way of doing release management though.

Location of release-info file

You can browse to the release-info file using Loggerhead (the web interface to bazaar), but there doesn't seem to be a reliable way to construct a raw download URI for "the latest version of" a file.

A workaround is to create a release series just for the release-info, and make an (unused) milestone for that. Then you can upload a release-info file. The URI that's listed directly on the downloads page is stable though (something like http://launchpad.net/my-project/main/release-info/+download/my-egg.release-info if you named the milestone "release-info" in the "main" series) This link redirects to an unstable URI, so don't paste that one!

When you make a new release you can delete the file and upload a new one manually to the same pseudo-"release".

Making releases

If you do it through their release management, you just create release tarballs which you manually upload into the release milestone.

The only automation here is that you can use bzr export --format=tgz to create it, and the advantage that you're using someone else's site to host your code.

You can use this release-info file template if you are consistent in your milestone naming. It assumes you used the name "main" for the series from which releases for chicken-install are made. You could alternatively add several uri entries, one per series, but it's still required that your egg's release versions are unique for your egg. Series are probably more suitable for distinguishing between major CHICKEN release versions.

(repo bzr "lp:MY-PROJECT/main")

(uri targz "http://launchpad.net/MY-PROJECT/main/{egg-release}/+download/{egg-name}.tar.gz")
(release "0.1")
(release "0.2")

Helper scripts

To make life easier, there are some helper scripts, hooks etcetera that could be used. Please add your own links here.

Subversion

There's an svn-egg-author egg for CHICKEN 4 that helps you to update the release-info and meta files and tag a release of your egg, all with one command.

If you run your own subversion server with Spiffy, you can consider using pseudo-meta-egg-info instead. If you do, you do not need to use svn-egg-author. If you're using the chicken-eggs repository hosted on call-cc.org, you don't need it either because call-cc is already running pseudo-meta-egg-info for all its eggs.

Mercurial

There's an egg-author extension for Mercurial that allows you to say hg eggtag 0.1 and have it automatically update the release-info file with the new tag name.

Git

There's a git-egg-author egg for CHICKEN 4 that helps you keep your release-info file up-to-date and tag a release with one command.

Bazaar

There's also a bzr-egg-author extension for Bazaar that allows you to say bzr eggtag 0.1 and have it automatically update the release-info file with the new tag name.

Moving an egg

When you decide to migrate to a different code hosting site, and don't want to move all releases, you can define multiple uris:

;; Main ("default") repo
(repo hg "https://bitbucket.org/YOUR-BITBUCKET-USERNAME/{egg-name}")
(uri targz "https://bitbucket.org/YOUR-BITBUCKET-USERNAME/{egg-name}/get/{egg-release}.tar.gz")
(release "2.1")
(release "2.0")

;; Old repo, using "old-svn-repo" alias
(repo svn "http://anonymous:@code.call-cc.org/svn/chicken-eggs/release/{chicken-release}/{egg-name}")
(uri files-list "http://code.call-cc.org/files-list?egg={egg-name};egg-release={egg-release};chicken-release={chicken-release}" old-svn-repo)
(release "1.3" old-svn-repo)
(release "1.2" old-svn-repo)
(release "1.1" old-svn-repo)
(release "1.0" old-svn-repo)
(release "0.2" old-svn-repo)
(release "0.1" old-svn-repo)

Testing new eggs before publishing them

The initial setup of a new egg is relatively tricky. Some details have to be properly configured before the official release. For example:

The .release-info file must be publicly available, so that henrietta-cache can access it.
The syntax of the .release-info file must be correct, and the releases and tarballs it points to must be consistent.
The egg code must be compilable and installable. Usually you test it from the local source code repository, which can be in a different state from the remote source repository -- that leads to problems when making the egg available.
The egg metadata must be properly specified (e.g., dependencies) -- that's another thing that can cause problems if the egg is tested with the CHICKEN installation that is used for development (on which environment the dependencies are already installed, thus missing specification of dependencies go unnoticed).

Manually testing all these aspects can be boring and prone to mistakes. To make the life of egg authors easier, we have a tool to test initial versions of eggs: test-new-egg. Using it is very easy: just install it (via chicken-install) and provide the URI for the .release-info file as argument to it:

 $ test-new-egg <egg name> <.release-info file's URI>