Packaging Software

From Nekochan
Revision as of 00:40, 29 December 2010 by Hamei (Talk | contribs) (Directory Structure)

Jump to: navigation, search


This article provides a basic introduction to packaging software for distribution to other IRIX systems through the Nekoware software archive. The guidelines presented here are exactly that: guidelines. Please ask in the forums at Nekochan if you encounter any problems following this guide or need help building software from source.

Packages

IRIX packages are called tardists and, as the name implies, are essentially tar(1)'ed sets of distribution files. These files describe the location and permissions of all files related to a particular piece of software. These tardists can then be distributed and installed on similarly-configured SGI machines.

Tools Required

Compilers

GNU/Linux, BSD and some other UNIX variants use the GNU C Compiler suite for building their software. In IRIX, gcc(1) does not work well for building widely distributed software packages. Nekoware tardists nearly always are built using SGI's compiler suite: MIPSPro.

SGI built some rather exotic machines and, as such, requires a more highly optimized compiler for code built to run on these machines. gcc(1) is designed to be a general-purpose C compiler suite that will build on nearly every machine made: past, present and future. This portability comes at a cost, however. gcc(1) tends to produce less well-optimized code for "specialty" hardware, such as those SGI produced. For this reason, MIPSPro should be used to build IRIX software wherever possible.

Platform

In order to build software for Nekoware, a version of IRIX greater than 6.5.21 and a R5000 MIPS processor, or better, is required. The primary branch of Nekoware is optimized for the MIPS IV architecture and compatible only with the environment provided by IRIX versions greater than 6.5.22.

As of this writing, there is also a MIPS III branch of Nekoware, however IRIX 6.5.22 remains a requirement.

Other Tools

Other tools may be required to build a particular piece of software that are not present in the IRIX environment. Such tools as GNU make and autoconf are commonly used to build Open Source software, but are not distributed with IRIX. If your software requires any of these tools, be sure to install the latest version from Nekoware and document the dependency on these tools during the packaging process.

Build Environment

Be sure to set some global variables for development on IRIX. The following is a set of sensible defaults:

export CC=c99
export CFLAGS='-O3 -mips4 -I/usr/nekoware/include -TARG:platform=IP27:proc=r10000'
export CXXFLAGS=$CFLAGS
export CPPFLAGS='-I/usr/nekoware/include -I/usr/include'
export CXX=CC
export F77=f77
export LDFLAGS='-L/usr/nekoware/lib -Wl,-rpath -Wl,/usr/nekoware/lib'
export PKG_CONFIG=/usr/nekoware/bin/pkg-config
export PKG_CONFIG_PATH='/usr/nekoware/lib/pkgconfig'
export PKG_CONFIG_LIBDIR='/usr/nekoware/lib'
export LD_LIBRARY_PATH='/usr/nekoware/lib'
export LD_LIBRARYN32_PATH='/usr/nekoware/lib'
export LD_LIBRARY64_PATH='/usr/nekoware/lib64'
export GNOME2_DIR='/usr/nekoware'
export GNUMAKE='/usr/nekoware/bin/make'

All Nekoware is installed relative to /usr/nekoware and should be built and linked against the libraries there. Also, most Open Source software prefers GNU make to the standard IRIX make.

Furthermore, there are some MIPSPro options that should be set:

Defining CC and CXX specifies the MIPSPro compiler suite. The CFLAGS option indicates building for MIPS IV, that being any MIPS processor R5000 or newer (see above).

Also present are additional optimizations being done for the R10000 platform (more specifically the Origin200 IP27), which is a common processor among the O2, Octane, and Origin platforms. This build will work on R5000, but will see additional optimization on R10000 or better. If the software is optimization-friendly, e.g. mplayer or blender - it is possible to provide different builds, optimized for specific processors.

Alternatively, the -TARG option could be disgarded entirely to build without any platform-specific optimizations whatsoever. Many Nekoware packages use only -03 -mips4.

Creating a Sandbox

Typically, a location under your home directory would be used for holding source tarballs, build directories and other working files. Under this directory, creating two copies of the build directory will aid the patch creation process later on. Suppose we were building fooware, version 1.2.3, within this working directory:

(prompt) gunzip -c fooware-1.2.3.tar.gz | tar xf -
(prompt) cp -r fooware-1.2.3 fooware-1.2.3-PATCHED

We've now created two identical versions of the source directory. By maintaining the original source and working within the "-PATCHED" directory, creating a patch for the package will be much more simple.

Building the Software

Typically, after a sandbox has been created, the software should be configured from within the "-PATCHED" directory (following the directions above). Remember, all Nekoware packages are installed relative to /usr/nekoware, so the --prefix option should be passed to configure, like so:

(prompt) ./configure --prefix=/usr/nekoware

Be sure to specify any other configure arguments necessary to build the software. It may be necessary to write down any dependancies configure checks to create accurate rules later.

After the software has been successfully configured, compile the source as usual. As an example, something like this might be used to compile the software on a dual-processor machine:

(prompt) gmake -j3

If the software successfully compiled, it may be possible to simply run the software from within the working directory, without installing anything. If so, try that now and verify that all desired functionality is present. If not, the software should now be installed using something similar to the following:

(prompt) su
(prompt) gmake install
(prompt) exit

After installation, test the software as above to verify the successful build.

Creating a Patch

Often, another developer or packager will want to re-create a package, perhaps from a newer release of the software. If drastic changes have been made to the source code, this could lead to duplication of work and frustration for the future packager. To help them along, the original source code and a patch to generate the buildable source code should be included in the package. To generate this patch, change directories to the parent of the sandbox and the clean source code and issue the following (using the example above:

(prompt) diff -urnp fooware-1.2.3 fooware-1.2.3-PATCHED | \
grep -v "^Only" > neko_fooware-1.2.3_irix.patch

Having the clean source code makes this process much easier. However, the above command ignores any new files that may have been created in the "-PATCHED" directory. To include these files in the patch, review the manual page for diff(1).

Installing the Software

To easily find what files and directories are created when the software is installed, it is best to create a temporary directory, typically under /tmp, to install to. This directory will contain only the new file structure created by the installation routine. An IRIX package requires a list of all files created by the software and this temporary directory will easily provide that. Assuming write access to /tmp, issue the following from the working directory ("-PATCHED" above):

(prompt) mkdir /tmp/build
(prompt) gmake DESTDIR=/tmp/build install

The full application tree is now created under /tmp/build, populated only by the files created by the software's installation routine.

Package Preparation

Directory Structure

Nekoware packages use a consistent directory structure for patches, sources, release notes and distribution files. Create this structure under the /tmp/build directory (created above), now:

(prompt) mkdir /tmp/build/usr/nekoware/patches
(prompt) mkdir /tmp/build/usr/nekoware/src
(prompt) mkdir /tmp/build/usr/nekoware/relnotes
(prompt) mkdir /tmp/build/usr/nekoware/dist

Copy the original source distribution and the patch file to their respective directory:

(prompt) cp neko_fooware-1.2.3_irix.patch /tmp/build/usr/nekoware/patches
(prompt) cp fooware-1.2.3.tar.gz /tmp/build/usr/nekoware/src

Release Notes

Using another Nekoware package as a template, copy a Release Notes file and edit it to suit the software. Common information to include is:

  • The packager's name and contact address
  • The software's name, version and description.
  • Information about the build machine
  • The version of the MIPSPro suite and the IRIX build environment
  • The environment variables you set - see before
  • Extra build notes: if building the software required more than the patch, what else needs to be done?
  • Remarks about known bugs or missing features
  • Known dependencies

Once this file is finished, copy it to the appropriate directory:

(prompt) cp neko_fooware.txt /tmp/build/usr/nekoware/relnotes 

Creating a Package

Now that the temporary directory contains the binary software, the source code, the patch and a release notes file, create a distribution directory and run Software Packager:

(prompt) mkdir /tmp/dist
(prompt) swpkg

The Software Packager interface is designed to be intuitive, however we'll walk through the interface and provide more explanation where necessary. For more information about Software Packager, read the manual provided in the External Links section of this article.

Creating a Product Hierarchy

On the first worksheet, a tree is displayed showing a default arrangement of subsystems within a package. This may, or may not, suit the software's layout, but it is provided as a starting point for creating a new package.

The first step is to give the package a name on the root node. Something like neko_fooware is recommended. Add a description to this node on the right using other packages in the Software Manager as a guide. Be sure to assign any changes or they will be lost.

Create the other nodes according to the typical Nekoware hierarchy, as follows:

neko_fooware.sw.eoe (binaries and files for execution only)
neko_fooware.sw.hdr (headers)
neko_fooware.sw.lib (shared libraries)
neko_fooware.man.manpages (manual pages)
neko_fooware.opt.src (the original source tarball)
neko_fooware.opt.patches (patches against the original source)
neko_fooware.opt.relnotes (release notes for this package)
neko_fooware.opt.dist (distribution files, generated by swpkg)

Not all of the tree may be required by the software. Use common sense and other Nekoware package as a guide.

Versioning

Every package will have two version numbers that should never be confused: the package version and the software version.

The package version is assigned to all children of the root node on the first worksheet of the Software Packager. This number is incremented for each build of this package independantly of the software version contained within the package.

For example, the software version of this application is 1.2.3, however the initial package version is 1. If, in the future, another package is built for this software, the package version will be incremented to 2. In this future package, the software version may be 2.0, 1.2.4 or remain at 1.2.3.

Remember: a package's version is independant of the version of the software it contains.

Also, all package version numbers must be equal within a package. If, for example, the execution-only subsystem has changed in a new package, all other subsystem's must have the version numbers incremented, as well. This holds true even if no other subsystems have changed between package versions.

Dependencies

This is, perhaps, the most important portion of the packaging process and is the reason for recording the software's dependencies during the build stage above. Find the package version of each dependency using the versions(1) command. For example:

(prompt) versions -n neko_foo
(prompt) versions -n neko_bar
(prompt) versions -n neko_baz 

Each versions(1) command will output several lines, one for each subsystem in the package. Write down the version numbers for each subsystem the software depends on. Be sure to be specific: which subsystem does the software depend on? Just the shared libraries, or the execution environment? Both?

Now, in Software Packager, specify the dependencies for each subsystem in the Rules section. The rules will look somewhat like the following:

replaces self
prereq (
  neko_fooware.sw.lib 1 maxint
  neko_foo.sw.lib 2 maxint
  neko_bar.sw.eoe 7 maxint
  neko_baz.sw.hdr 1 maxint
)

Final Steps

There is also an "install by default" checkbox that should be checked for any subsystems that require it. Typically, the sw.eoe, sw.lib, sw.hdr and man.manpages will have this option checked.

Before continuing, be sure to save the .spec and .idb files that Software Packager creates to /tmp/build/usr/nekoware/dist. This will make them available to the next step.

Tagging

Every file that the software creates and installs must be tagged with a particular subsystem within the package. Above, we created a clean application tree in /tmp/build for this very purpose. Had we not, each file that the software installed would need to be found in /usr/nekoware - a very tedious process!

Import the entire /tmp/build directory and tag each file. For example, files installed in lib/ would be tagged with sw.lib, those in bin/ and share/ should be in sw.eoe and include/ is tagged with sw.hdr.

All directory entries should be removed from the distribution entirely. Leaving them is considered sloppy and clutters the package.

Set all files as owned by the root user and sys group, save the .spec and .idb files once more and exit Software Packager.

Fixing Paths

The current package now contains all files installed by the software with the correct permissions and tagged with the correct subsystem. However, due to using the sandbox, /tmp/build, all of the paths are incorrect. Thankfully, correcting them is an easy proposition:

(prompt) cat /tmp/build/usr/nekoware/dist/neko_fooware.idb | \
sed 's/\/tmp\/build//g' > tmp.idb
(prompt) mv tmp.idb /tmp/build/usr/nekoware/dist/neko_fooware.idb

Alternatively, the .idb file can be opened using a text editor and the necessary replacements done using the editor's find-and-replace functionality.

Re-launch the Software Packager and open the .spec and .idb files. Verify that the paths are correct.

Quickstarting

During most package installations using inst(1M) or swmgr(1M), certain files need to be "Re-quickstarted" using rqsall(1). When creating a new package, certain files must be marked as not needing this process. In the software's fileset, any files that are not ELF binaries should be marked with the norqs option.

Icons

You can use iconsmith(1) to create a polygon-based icon that is used to represent the packaged program or its documents on the desktop, in the filemanager or in the Icon Catalog. You will also have to create a file-type rules file to tell fm how to identify the files that are to use your icon(s), and what actions are available. Additionally the rules database has to be rebuilt via an exitop in the specfile. Install the icon(s) in /usr/lib/filetype/local/iconlib and the .ftr in /usr/lib/filetype/local. If you want to add the program's icon to the icon catalg, run iconcatalogedit -add as an exitop and remove it by calling iconcatalogedit -remove in a removeop in your specfile. To identify the executable, you can use use tag() attibute in the specfile.

f 0644 root sys usr/lib/filetype/local/iconlib/fooDocument.fti usr/src/foo/fti/fooDocument.fti neko_foo.sw.eoe
f 0644 root sys usr/lib/filetype/local/iconlib/foo.fti usr/src/foo/fti/foo.fti neko_foo.sw.eoe exitop('if [ -x $rbase/usr/sbin/iconcatalogedit ]; then chroot $rbase /usr/sbin/iconcatalogedit -add "Category:File Name:/usr/nekoware/bin/foo" -syspage Applications; fi') removeop('if [ -x $rbase/usr/sbin/iconcatalogedit ]; then chroot $rbase /usr/sbin/iconcatalogedit -remove "Category:File Name:/usr/nekoware/bin/foo" -syspage Applications; fi')
f 0644 root sys usr/lib/filetype/local/foo.ftr usr/src/foo/fti/foo.ftr neko_foo.sw.eoe exitop('if test -r $rbase/usr/lib/filetype/Makefile ; then chroot $rbase /sbin/sh -c "cd /usr/lib/filetype ; make -u > /dev/null" ; fi')

Testing and Building

Save the .spec and .idb files once more and move to the next worksheet. This worksheet provides the final steps in Software Packager. Click the Test button to test the package build process. It is likely that the process will complain about files missing from /usr/nekoware. Move to a shell and copy these files from /tmp/build into their respective directories in /usr/nekoware and run the test again.

Once the test routine passes without complaint, click the Build button and exit Software Packager. The distribution files for the package should now be in /tmp/dist. Change to that directory and archive these files:

(prompt) cd /tmp/dist
(prompt) tar -cf neko_fooware-1.2.3.tardist *

The next, and final, step is crucial. Change to the working directory, "-PATCHED" above, and uninstall the software:

(prompt) su
(prompt) gmake uninstall
(prompt) rm /usr/nekoware/dist/neko_fooware.*
(prompt) rm /usr/nekoware/src/fooware-1.2.3.tar.gz
(prompt) rm /usr/nekoware/relnotes/neko_fooware.txt
(prompt) rm /usr/nekoware/patches/neko_fooware-1.2.3_irix.patch

Now, install the package as though it were a normal Nekoware tardist:

(prompt) inst -f /tmp/dist/neko_fooware-1.2.3.tardist

Verify that the software installs correctly and runs as before, with no errors. If all goes well, the package may now be uploaded to the Nekochan FTP site for testing by other users and, hopefully, inclusion in the archive. Congratulations!

Credits

This content originally appeared here and has been reproduced with the permission of the original author.

Further Reading

SGI Software Packager User's Guide