Calibre2OpdsBuild

Home Page	User Guide	Calibre2opds Developers Guide				Issues
		Overview	Building	Localization	Customization

Building Calibre2opds

This section is aimed at those who want to build calibre2opds from the source code.

There can be many reasons why a user might want to do this. The process has been made as easy as possible so that users can do this without much in the way of specialist knowledge.

This section of dcoumentation is divided into the following topic areas:

Development Tools Pre-Requisites

You need to have downloaded and installed the following pre-requisites before trying to work on Calibre2opds.

• Java SDK: The Calibre2opds program is written in the Java programming language. If you want to be able to change and compile Calibre2Opds then you will need the (free) Java Software Development Kit (SDK).

• Subversion This tool is a cross-platform Version Control System. It is often simply called svn which is the command line version of the software. It is used to handle getting Calibre2opds source into/out of the GitHub system.
  Notes:
  • There are many implementations of Subversion, but the one we recommend if you are using Windows is the TortoiseSVN release. If installing the TortoiseSVN version make sure that when installing you select v1.8 (there are issues with v1.9) and you select the option to install the command line tools as well as these are required by the calibre2opds build system. You can then also use the GUI interface provided by Tortoise SVN to carry out most svn actions when you want to do something specific.
    
  • Most Linux distributions will include subversion in their standard software repositories, so can be installed using the distributions package manager.
  • If you want more information on the command line options available with svn, then there is a free manual available at http://svnbook.red-bean.com/. This is available either as an online manual, or as a PDF download

• GitHub client: This tool is used to get the source from the GitHub repository that is used to store Calibre2opds source code if you are not going to do it via subversion.

• Apache Maven: This tool is used to automate the build process. Install according to the instructions on the download page. Select the instructions appropriate to the OS you intend to use.

• lzPack: The Calibre2opds installer is generated using the izPack program that can generate cross-platform installation packages. If you are not interested in producing the Calibre2opds installer then you will not need this package. There are incompatibilities between the v4 and v5 versions of izpack so you need the correct version for the version of calibre2opds you intend to build.
  • Calibre2opds v3.5 and earlier uses v4.3.5
  • Calibre2opds v3.6 and later uses v5.0.9 (or later)

• Zip: (Windows only). If you want to be able to build the install package then you need a command line version of the zip program. If you are not interested in producing the Calibre2opds installer package then you will not need this program.

• Python: This is required to build the install package for Windows (.exe) and Mac (.app). It is not required if you only want to produce the ZIP and Linux (.jar) versions of the installer. It does not matter for this purpose whether Python 2 or Python 3 is used.

The other tool that is commonly used is what is known as an Integrated Development Environment (IDE). This provides a shell that can help with the development tasks and has specialist tools for editing, source control, debugging etc. However whether you use such a tool is your choice although you would almost certainly find it much easier to look at amend the source with the help of an IDE.

There is a wide choice available of IDEs for use with Java. If you already use an IDE that can be used with Java then it is probably best to stick with the one you know. If not, then IDEs for Java that have been found to be good are:

• " IntelliJ IDEA: This is a powerful Java IDE that is widely used by professional Java developers. This IDE comes in both free and commercial versions. The Free version should be more than sufficient for any calibre2opds development and is the personal favourite of the main calibre2opds developers.

• Eclipse: Eclipse is an IDE that is widely used and is strongly favored by many Java developers. If you do decide to use Eclipse you will need at the least the Eclipse IDE for Java Developers version although you may go for one of the other versions if you plan to to use Eclipse for more than Java development.

• NetBeans: The NetBeans IDE is used by many Java developers. The GUI forms in Calibre2Opds are actually set up using the NetBeans form designer. You should therefore have this installed if you are interested in amending the GUI elements of calibre2Opds, but it is not required for simply amending the main Java code.

Setting up the Development environment

Each of the development tools can require a small amount of tool-specific setup and customizaton of the settings.

Setting up Java

Normally Java is set up by simply running the installer included with the download of the java SDK. However there are some things to check

Windows

Make sure that the following Environment variables are set up:

CLASSPATH    .;D:\Program Files\Java\jre6\lib\ext\QTJava.zip

JAVA_HOME    C:\Program Files\Java\jdk1.6.0_19

If you installed to a different location, or were using a different SDK version then the above will need amending accordingly.

The PATH environment variable should contain a reference to %JAVA_HOME%\bin.

Linux

Most Linux distributions come with Java Runtime Environment (JRE) pre-installed. However to do Java development you need a Java Development Kit (JDK) installed (which includes the JRE). If your Linux system was built as a development environment then the JDK is probably already installed. You may be able to download the JDK using your distributions package manager but if that is not the case then simpy download the Oracle Java Development Kit and install it according to the instructions given on the Java home site.

Mac

TBD

Setting up Subversion

Normally Subversion is set up by simply running the installer included with the download. However there are some things to check

Windows

The calibre2opds build system needs to know the location of the 'svnversion' command line tool. This is set in the setenv.cmd file that is used by the Calibre2opds build system. It will typically contain a line of the form:

SVN_HOME=C:\Program Files\TortoiseSVN 

although the details can vary according to where you installed Subversion, and which particular package you decided to use.

Linux

The calibre2opds build system needs to know the location of the 'svnversion' command line tool. This is specified by the SVN_HOME environement variable. At runtime the value /bin/svnversion is added to the setting of this variable, so if svnversion is located in /usr/bin/svnversion then you would need a setting like

SVN_HOME=/usr

This is the default setting in the make.sh and makenotests.sh files, but if this is not appropriate for your system you can either set this explicitly before running these scripts or edit the provided scripts appropriately.

Mac

TBD (It is probably similar to Linux but this needs to be verified)

TBD

Setting up Zip (Windows only)

If you want to be able to create the calibre2opds installer package then you need the GNU Zip package installed on your system. The downloaded ZIP installer does not automatically add the relevant entry to the PATH variable to allow the ZIP program to be found from batch files, so you will probably need to do this manually. If you have used the default install location this will involve adding the C:\Program Files\GnuWin32\bin entry to the PATH environment variable.

An alternative is to edit the MakeInstall.cmd file supplied with the calibre2opds source to give a full path to the ZIP program, but this is nowhere near as convenient as adding the directory to the system path.

---

Setting up Maven

The calibre2Opds project is set up as a Apache Maven project, so you will need to have this installed to be able to build the Calibre2Opds program.

Windows

The following Environment variables should be set up

M2_HOME     C:\Program files\Maven
MAVEN_OPTS  -Xms256m -Xmx512m
M2          %M2_HOME%\bin

Note that the M2_HOME variable must reflect where you actually installed Maven - the example above assumes the normal default location.

The PATH environment variable should contain a reference to %M2%.

Linux

You will need to add the Maven location to your system search path. Linux provides many ways to do this, but probably the easiest and the most generic is to add an entry to do this to the /etc/profile script. If you insstalled maven to /opt/maven then this can be done using entries of the form:

PATH="$PATH:/opt/maven/bin".
M2_HOME="/opt/maven"
M2="$M2_HOME/bin"
export M2_HOME M2

Mac

Apache Maven is installed by default in snow leopard but can be downloaded manually. Instructions can be found here

Setting up IntelliJ IDEA

When you use IntelliJ IDEA for the first time you may want to customize some of the settings to suit your particular preferences. Access to the global IntelliJ IDEA settings is via Files->Settings. Typical things that one wants to customize at this stage are things like the theme to be used and perhaps the fonts that are used in various places.

Also the first time you try and run the calibre2opds project you will need to do the following:

• When you are prompted about Maven settings then accept the auto-import option
• Under Files->Project Structure go to the SDK's section and set the JDK home path to point to the location where you have installed the Java JDK that you are using.

Obtaining Source Code

Calibre2opds is an open-source product. This means that the source is freely available for any user to download and use.

GitHub is used to host the Calibre2opds source code. The URL for accessing this source code is: https://github.com/calibre2opds/calibre2opds/trunk

Releases of Calibre2opds prior to v3.5 were instead hosted at Google Code (this is being shut down later in 2015 which is why we have switched to gitHub). If you want the source code from Google Code instead the URL is: http://calibre2opds.googlecode.com/svn/trunk/

To get the source code from Google Code you can use either git or Subversion (svn) source control system clients. For historical reasons we are currently using the Subversion client but that is not mandated.

Windows

The following instructions are based on using the TortoiseSVN package, although they should be easily adaptable to any of the other Subversion implementations that are available:

• Create the folder that is going to be used to hold the calibre2opds source.

• Right-Click on the folder and select the SVN Checkout option

• Use the appropriate URL (for either gitHub or Google Code as mentioned above. The latest source will be on gitHub for which the URL is http://calibre2opds.googlecode.com/svn/trunk/.

• Once you have the source, you can always right-click the source folder and select the SVN Update option.

• It is also possible to use the SVN client from a Command window in which case the equivalent command to the above would be:

 svn co https://github.com/calibre2opds/calibre2opds target_folder

where target_folder is where you want the checked-out source to be placed.

TBD: Extra steps required if one want to be able to check in source code changes

Linux

• Run the command

svn co URL

where URL is the appropriate URL for the gitHub or Google Code repositories. http://calibre2opds.googlecode.com/svn/trunk/ to get the source checked out into the folder calibre2opds. You can optionally add a folder name as an additional parameter. As an example using

 svn co https://github.com/calibre2opds/calibre2opds calibre2opds

would get the source from gitHub and place it in the calibre2opds subfolder of the current folder.

• Run the commands

cd target+folder
chmod +x *.sh
chmod +x scripts/*.sh

(where target+folder is the folder holding the checked-out source to make the provided shell scripts executable.

You could alternatively use these scripts by running them preceeding script name by preceding them with the sh command. e.g.

sh make.sh

Further information on the svn command is available in the svn manual mentioned in the pre-requisites section.

Mac

It is expected that the instructions would be similar to the above, but we would welcome it if a Mac user could actually get the exact commands and update this section of the documentation appropriately.

Building Calibre2opds

Using the Command Line

To simplify the process of compiling calibre2opds from source, some scripts are included to help with automating the calibre2opds build process. These are make.cmd (for Windows) and make.sh (for Linux/Mac).

If running on other systems then check the files to check what assumptions they have made about the location for the software components. Edit the file (if necessary) to match the setting for your system. They are standard text files so you can both view and edit them with whatever text file editor you have installed on your system. They expect a number of environment variables to have been set up to say where required software components are located. Typical settings might be:

JAVA_HOME=C:\Program Files\Java\jdk1.6.0_29
M2_HOME=C:\Program Files\Maven
SVN_HOME=C:\Program Files\TortoiseSVN

although these may be different on your particular system if you have used different install locations. On Windows these are actually set up in the file setenv.cmd and a sample file called setenv.cmd.sample is provided to make it easy to create a setenv.cmd file tailored to your environment. If the environment variables are not set up correctly, then when you run the make.cmd/make.sh file error messages are issued that give appropriate guidance on what to do.

Running these from a Command/Terminal window while located in the oalibre2opds project folder will run the Calibre2Opds build process. The first time you run them they will download and build a lot of modules associated with Maven and then build Calibre2Opds. Subsequent invocations will only build Calibre2Opds.

There is also the files run.cmd (for Windows) and run.sh (for Linux/Mac) in the oalibre2opds root folder that can be used to run the version of calibre2opds that you have just built without the need to build and deploy the install package. However in practice most people do not use these as they find it easier to run from within the IntelliJ IDEA system.

Using IntelliJ IDEA

The first time you attempt to use IntelliJ IDEA with calibre2opd there are some settings that you need to set up. These are remembered for subsequent runs so this only needs doing the first time:

1. Open the calibre2opds project
2. Specify the Java details
3. Import the Maven settings

Once you have done this then you can use the IDEA environment to edit, compile and test calibre2opds without the need to use the command line approach described earlier. Note, however, that if you want to build a deployable calibre2opds package then you still need to follow the command line approach to produce the package.

Running the built Calibre2opds

If you have just built calibre2Opds from source then it is extremely likely that you will want to immediately run the version you have just built. One way is to build the Install package mentioned below and go through the normal install process. How this is inconvenient if you are in the middle of testing and do not want to overwrite the currently installed version of Calibre2Opds.

To help with this batch files are provided that you can run from a command/terminal window while located in the Calibre2Opds source folder. These simply load up the version you have just built:

& Windows: Type in the command run< to invoke the run.cmd file that is supplied. This will start calibre2opds using the java command rather than the javaw command used when not running in test mode. This will leave a Command window open so that one can see all the messages that are being written to the log file. Possibly more important it will also show any messages that are written to the 'system console' that might not make it to the log file.

• Linux/Mac: Type in the command sh run.sh or ./run.sh to invoke the run.sh file that is supplied.

if you have any special settings that you want to use then these batch files provide a good starting point for producing your own customized launch files.

Building Installer

If you only want the ZIP version of the package then this can be built without installing the IxPack package. In many cases this may be all that is required (particularly while developing) as Calibre2opds can always be installed from this package by following the documented process for a manual install from the ZIP release.

The Calibre2Opds installer packages which provide a GUI based install process are built using the lzPack package. Online documentation is available for lzPack if you want to understand more about how lzPack works, or wish to enhance the Calibre2Opds install package. Additional steps are required for a Mac OSX dmg package as detailed later in this section

Some scripts are included to help with this:

• Windows: There is the command file MakeInstall.cmd.

• Linux/Mac: There is the script file makeInstall.sh.

You should check these scripts to see what assumptions they have made about the location for the the lzPack install (if being used). Edit them to match the setting for your system.

Simply running these provided files while located in the Calibre2Opds project folder will run the Calibre2opds package build process. The following install files are generated:

• calibre2opds-XXX-YYY.zip
  The calibre2opds program files suitable for a manual install.
  This version of the package is built even if the IxPack package is not installed.

• calibre2opds-XXX-YYY.jar
  An installer that can be used on systems that support Java such as Linux.
  This package will only be built if the IzPack package is installed and can be found.

• calibre2opds-XXX-YYY.exe
  An installer for use on Windows.
  This package will only be built if the IzPack package is installed and can be found.

where XXX is the release number and YY is the build number.

All the files that are likely to need amending when producing a new Calibre2Opds release are held at the base of the Calibre2Opds project folder. These files are:

• release.txt: The current release notes. At the very least these should be changed to summarise the amendments made. These are also displayed as the last step in the installation process so anything else that the users need to know for this release can be included here.

• install.xml: This is the control file used by lzPack and it defines the files that are going to be included in the Calibre2Opds install packages. as some of the generated files include the Calibre2Opds version number they will need amending appropriately.

• shortcutSpec.xml: This defines the shortcuts for use on Windows. It will need updating to include the correct binary for the current release.

• Unix_shortcutSpec.xml: This defines the shortcuts for use on Linux. It will need updating to include the correct binary for the current release.

Building the Mac OSX .dmg package

The above steps build the install packages for Windows and Linux, and the platform independent install package. Most Mac OSX users expect applications to be packaged as what is know as a dmg' file. This section covers the additional steps that are required to build the dmg package.

Under the covers the dmg file is a archive in a specific format that contains a Mac HFS filing system. It is possible to build the dmg on a none-Mac system using a tool that can handle the dmg file format and HFS filing system. Examples of such applications are:

• TransMac (Windows)

• MagicISO (Windows)

• HFSPlus (Linux)</li>

NOTE: Not all the above Windows packages are free, so any feedback on equivalents that are free for non-commerical use would be welcomed.

The following is the steps required using the TransMac tool on Windows. Similar steps should work with any other application capable of handling the dmg format.

• Get the skelton dmg file that is included with the calibre2opds source and is available at the location install/calibre2opds.dmg relative to the base of the calibre2opds source tree.

• Mount the calibre2opds.dmg file in TransMac and then open up the dmg file

• Open the calibre2opds.dmg file

• Replace the following files at the root of the dmg file with those from the root of the calibre2opds source tree:

• • license.txt
  • readme.txx
  • release.txt

Open up the Calibre2Opds.app/Contents/Resources/Java</ folder in the dmg image. Copy to that folder all the files that are located in the calibre2opds source tree at Install\target\dependency. These will be all the .jar files that are part of calibre2opds. The exact list of files and names may change from those shown in the screenshot to reflect a particular calibre2opds release but you want all of the files present.

• Go back to the Contents folder and open up the info.plist file in a text editor that can handle Macintosh format text files. If you do not have a text editor capable of handling Mac format text files, then the " Plist Editor for Windows can be used.

• Change this file so that the details highlighted below are correct. In particular the list of files in the ClassPath array must match those that are actually part of this release of calibre2opds.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist SYSTEM "file://localhost/System/Library/DTDs/PropertyList.dtd">
<plist version="0.9">
<dict>
<key>CFBundleName</key>
<string>Calibre2Opds</string>
<key>CFBundleIdentifier</key>
<string>Gui</string>
<key>CFBundleVersion</key>
<string>3.0</string>
<key>CFBundleAllowMixedLocalizations</key>
<string>true</string>
<key>CFBundleExecutable</key>
<string>JavaApplicationStub</string>
<key>CFBundleDevelopmentRegion</key>
<string>English</string>
<key>CFBundlePackageType</key>
<string>APPL</string>
<key>CFBundleSignature</key>
<string>????</string>
<key>CFBundleGetInfoString</key>
<string>Calibre2Opds v3.0</string>
<key>CFBundleInfoDictionaryVersion</key>
<string>6.0</string>
<key>CFBundleIconFile</key>
<string>icon.icns</string>
<key>Java</key>
<dict>
<key>MainClass</key>
<string>Gui</string>
<key>JVMVersion</key>
<string>1.5+</string>
<key>ClassPath</key>
<array>
<string>$JAVAROOT/OpdsOutput-3.0-SNAPSHOT.jar</string>
<string>$JAVAROOT/CalibreQueryLanguage-3.0-SNAPSHOT.jar</string>
<string>$JAVAROOT/DataModel-3.0-SNAPSHOT.jar</string>
<string>$JAVAROOT/Tools-3.0-SNAPSHOT.jar</string>
<string>$JAVAROOT/antlr-2.7.7.jar</string>
<string>$JAVAROOT/antlr-runtime-3.1.3.jar</string>
<string>$JAVAROOT/jdom-1.1.jar</string>
<string>$JAVAROOT/jtidy-r938.jar</string>
<string>$JAVAROOT/junit-4.7.jar</string>
<string>$JAVAROOT/log4j-1.2.12.jar</string>
<string>$JAVAROOT/sqlite-jdbc-3.6.17.1.jar</string>
<string>$JAVAROOT/stringtemplate-3.2.jar</string>
</array>
<key>Properties</key>
<dict>
<key>apple.laf.useScreenMenuBar</key>
<string>true</string>
<key>VMOptions</key>
<string>-Xms128m -Xmx512m</string>
</dict>
</dict>
</dict>
</plist>

You should now have a dmg file that is ready for use on a Mac running OSX.

NOTES

1. Some users have that .dmg file is reported as damaged by OSX. This can easily be fixed using the Disk Utility found in /Applications/Utilities and selecting the option to repair the disk.
2. Because the Calibre2OPDS application is not signed by a "known" developer, on newer version of OSX it cannot be opened in the usual fashion (double-clicking). Instead, one must Control-click the (repaired) .dmg file. This will invoke a dialog where you will be able to explicitly give permission to opening this suspect app without changing security settings. Read more about Apple's Gatekeeper

Debugging

This section covers hints on using java source-code level debugging in different environments.

We would welcome additional material in this section from users who use environments other than the default IntelliJ IDEA that is the one used by the main Calibre2opds developers.

Using the Intellij IDEA Java debugger

The IntelliJ IDEA system allows you to do interactive debugging of your Java application. This will let you run the program in debug mode doing all the things one would expect of a modern interactive debugging environment. These include:

• Step through the source code
• Set break points
• Examine variables

You run your application in Debug mode by going to the Run menu and selecting the Debug option to select the configuration to run.

If you are going to use this feature then you will want to define configurations for running Valibre2opds in both Gui and Batch modes. To do this go to the Run menu and select the Edit Configurations option. This will bring up a dialog of the form:

You can add a configuration by pressing the + button and then selecting the Application option.

Debug Running in GUI mode

Most users will use calibre2opds in its GUI mode of operation. To enable debugging in this mode the IntelliJ IDEA settings are shown below although the name you us for this configuration is up to you

Debug Running in Batch mode

This is fundamentally the same as debugging in GUI mode except for the fact that you tell the IntelliJ debugger to use a different entry point as shown below:

Enabling Assertions

It is a good idea to enable assertions when debugging. The Calibre2opds source has made extensive use of the Java assert statement to both highlight any assumptions that are being made by the programmer, and also to check that the program logic is functioning as expected. Enabling assertion invokes the runtime checks that any assumptions in the program execution are not being broken. To do so you need to add either -ea or -enableassertions to the VM options when running under the IntelliJ debugger.

The screenshot above for the Gui mode settings shows how you do this when setting up an IntelliJ configuration. In practice you are probably going to want assertions enabled for debugging both Gui and Batch modes of operation.

If you use the run.cmd or run.sh commands located at the root of the calibre2opds source folder then these will pass the relevant options to the Java runtime system.