Calibre2Opds Publish

Home Page	Calibre2opds User Guide						Developer Guide
	Overview	Installing	Generating	Publishing	Using	Troubleshooting

Publishing Calibre2opds Catalogs

This section of the User Guide provides information on publishing your Calibre2opds catalog.

It is split into the following main topic areas:

Overview

To be able to use your generated Calibre2Opds catalogs you need to make them visible on the network.

NOTE: If you are using calibre2opds in Nook mode then you do not need to publish your catalogs as they will have been copied (together with their associated eBooks) to the Nook and held there as local files

The basic requirements are that:

• The method chosen provides a network path to the top level file in the generated Calibre2Opds catalog.

• The method allows all catalog files and book files to be accessed as a path relative to that of the top level catalog file.

This will tend to break down into 2 distinct families of servers:

• Web server.
  This can be your own private server, or one run by someone else on which you have space provided for you own web content. This could be an ISP provided web space - but if this is the case please read the caution about controlling access.

  Another (less obvious) solution that falls into this category is to use a WiFi enabled storage device that also supports a web server for accessing the files it is storing (just as if they were hosted on a private or ISP provided web server). One device that has proved a good match with Calibre2opds is the AirStash as it provides a web server tha can be accessed easily from any WiFi enabled device such as an iPhone, iPad, Android tablet/phone and allows the Calibre2opds catalog to be used from either a browser (HTML catalog) or a suitable OPDS aware reading app (OPDS catalog).

• "Cloud" file storage.
  This is the generic term used to describe facilities where you can have files that are located on your own PC or Mac replicated to servers on the Internet so that they can be accessed from anywhere. If the Cloude service can expose the files that are stored on it just like a web server would then they can also host a Calibre2opds catalog. Dropbox and Google Drive which are described later in this document are examples of such services, although it is likely that others are also suitable.

  NOTE: Cloud services that only allow downloading of files via a browser (i.e. they cannot act as HTML web pages) will unfortunately not work. Examples of such services that are currently known to fall into this category are iCloud Drive; Copy.com; SugarSync and OneDrive (was SkyDrive)

Security

Security will not be an issue if you have your library on your home system, and set up so that it can only be accessed from devices that are part of your home network.

If you make your Calibre library visible on the public Internet then you need to make sure your books are not easily accessible by unauthorised users. Depending on where you live, failure to do so may mean that you are infringing copyright

This would typically involve providing a username/password challenge or making sure the URL used is not easily found (which is what most "cloud" services use). Calibre2opds provides an option to 'encrypt' filenames used within the catalog to make the filenames less predictable.

Web Server

Many users run web servers on their own home network, or alternatively make use of space on web servers that are provided by their ISP. The details of these can vary, but the following should provide some general guidance.

• Copy your entire Calibre library folder to a location that is visible to the web server that you are using. This will include the Calibre2Opds generated catalogs as a sub-folder of the location holding the author folders from the Calibre library.

• If you are using a public web server then they will normally tell you what URL should be used to access the files you have just uploaded.

• If you are using a NAS (Network Addressable Storage) device then you can set up calibre to directly work with the files on the NAS thus avoiding the copying requirement.

• If you are using a web server on you owwn local WiFi network and only want to access the Calibre2Opds catalogs while at home that is probably all you have to do.

If you are running on your own private web server and want to make the Calibre Library and Calibre2Opds catalogs visible on the public Internet then some additional steps will be required:

• You will need an externally visible IP address and port that can be used for accessing your private web server.

• If you are a Cable user then although strictly speaking you are probably using DHCP to get your external IP address assigned to you it tends to change very rarely so you can probably just query your modem/router to find out what address you are currently assigned and use that address directly. You still need to be aware that it can change.

• If you are an ADSL user connecting via a telephone line then typically the address only persists for a limited period so you need a way of getting a more dynamic mapping of the external IP address to something you can use. there are service that provide such a capability such as XXX. Note that there is nothing that stops cable customers using such a service as well.

• Set up some NAT (Network Address Translation) entries in your router to allow an external IP port on the router to be mapped to a particular IP-Address/Port combination on your local network. This will be the IP_address:Port that is being used by the web server hosting your Calibre library and Calibre2Opds catalogs.

• Make sure you have some sort of security mechanism in place to stop unauthorised access.

TIP:

Many web servers have the concept of a default page that is loaded if the user does not specify an explicit page in the URL they use to access the web site. Typically this is a file called index.htm or index.html located in the web servers document root. If you include a text file of this sort that contains content allong the lines of the following

<html>
<head>
<meta http-equiv="refresh" content="0;url=_catalog/index.html">
</head>
<body>
</body>
</html>

then this will automatically redirect the user to the first catalog page without the user having to know the full URL.

Using lighthttpd

Server optimization

lighthttpd is a lightweight web server that can work on a NAS with low memory. If the web server crashes while you're browsing the library, add the following line in the server configuration file (lighttpd.conf):

 server.event-handler = "poll"

and then restart the web server.

Using Windows Home Server (WHS)

There is a very good blog post that covers using the web server built into Windows Home Server (WHS) to host the catalog produced by calibre2opds. Although it has not been validated the steps probably apply to anyone using Internet Information Server (IIS) on a Windows system to host the catalog.

Note in particular the requirement to set up the Mime types for the ebook types you want to be able to download. This is required because IIS ignores the Mime types set by calibre2opds in the HTML and sets it according to the file extension. If you do not know the Mime type for a particular ebook file type then you can load the HTML produced by calibre2opds; find the line for downloading a particular ebook file; and then look at the value for the 'type' parameter on the download link.

Cloud Services

Introduction

Many users want to use a internet based service to store their libraries and access them remotelyMany people store their libraries on an internet. As long as he product in use allows the files to be accessed via an URL, then this is no different to hosting the files on a web server. You can use Cloud Storage for hosting both the HTML and XML versions of the Calibre2opds catalogs.

The big advantage of this approach is that these product nearly always include an automatic sync facility to copy files between local PC/Mac systems and the cloud storage. this means that the user does not need to worry much about the management of the files as it is automatic.

The requirements for such products to work successfully with calibre2opds are:

• There must be a base URL that can be used to access the top level calibre2opds catalog

• It must be possible to navigate to the other catalog files using relative URLs

• It must be possible to access the ebook files that are to be downloaded using relative URLs.

These tend to be standard functionality in most such products so they work without the user having to do anything special.

The one thing that you do need to be careful about is security. If the product in use does not provide a security mechanism that can be used with the calibre2opds catalogs then you may be breaching copyright regulations depening on your country of residence.

If you are using Cloud storage then you will normally want to take steps to minimise the space used to store the calibre2opds generated catalog. There are a number of calibre2opds settings that are likely to be relevant in such a scenario:

• Generate either the OPDS (xml) version of the catalog or the HTML one. The default in calibre2opds is to generate both. Which variant you need will depend on what reading software or device(s) you are going to use. Generating only one variant of the catalog will halve the storage space used for the catalog. The options to control this are on the Catalog Generation Options tab.

• Use the option to Minimize the number of changed files on the Catalog Generation Options tab. This does not reduce space but it does try and minimise the amount of data that needs to be synced to the cloud after a calibre2opds generation run. This option is normally set by default, but you may want to check what you have it set to on your system.

• Use the options Do not Generated resized covers and Do not generate Thumbnails on the Advanced Customization tab. This avoids calibre2opds generating new image files optimised for this purpose and the catalog instead uses the existing Calibre cover.jpg files for this purpose. This will save significant storage space although it does mean that when you use the catalog more data has to be transmitted as you will be using image files for thumbnails and covers that are larger than is required. This is normally a desirable trade-off if you are using cloud based storage.

• Use the option to Use external files for icons on the Catalog Generation Option tab. By default the little icons at the front of lines in the catalog are embedded in the XML/HTML pages that are generated. Using this option will instead reference files stored on the server. This reduces slightly the size of the files generated for each page in the catalog at the expense of slightly slower times to load the page when using the catalog. The space savings from this are minimal so many users do not bother.

Updog + Cloud Services

The vast majority of the Cloud services no longer provide the capability to treat them at a web server which is a requirement for successfully hosting a web site. Dropbox and Google Drive for instance, used to offer this capability but have now removed that capability. If you want to test if a particular service is suitable then the way to do it is to upload an HTML file and then using the cloud services web interface try clicking on it to see what happens. If the HTML page is displayed correctly then the service is suitable. If instead the HTML is displayed or you are given a tile download prompt then the Cloud service does not support acting as a web server.

The site updog.co has recognized the need that some users have for such a capability and has developed a workaround for that allows a calibre+calibre2opds to work on such services. The idea is that updog provides the web server capability and et then internally uses an API provided by the Cloud service it is fronting to access the files that are to be treated as a web site. At the current time the updog service is available for fronting Dropbox and/or Google Drive, but it is intended that the list of supported services will grow over time. There is a questionnaire on the updog site where you can indicate which cloud services you are interested in updog supporting in the future.

The updog service is free for personal use if you want a single web site to be supported using DropBox as the Cloud service. The free DropBox support is expected to be sufficient for most Calibre2opdr users. The chargeable 'Pro' option adds in support for Google Drive and also gives additional capability such as support tor multiple web sites, password protection on your web rites and more freedom as to where on your hard disk the web rite files are stored.

The following is generic in nature and will apply regardless of the Cloud service that is to be used.

• Set up an Updog account: Create an account on the updog.co site.

• Decide on a web site name: When using updog you will find that your library will be under a URL that starts with website-name.updog.com where website-name is the name you chose for your web site.

• Library folder location: The simplest solution is to move the Calibre library to the updog folder and use Calibre2opds in its Default mode to generate the catalog as a sub-folder (default name of '_catalog') of the Calibre library folder. The exact name the folder will vary according to the Cloud service being used and whether you have the standard or pro type of updog account. In this scenario it is recommended that you turn of Dropbox sync while running Calibre to update your library, although in practise you will probably get away without doing this if you never run Calibre on another machine at the same time.

  The alternative is to have the Calibre library folder somewhere else and run Calibre2opds in its Publish mode to create a copy of the Calibre library and Calibre2opds catalog in the updog library folder. This would mean that you end up with a second copy of your Calibre library - one in the current location and the other in the updog folder under the Cloud service being use. This scenario avoids any potential sync issues as your master Calibre library is no longer under the Cloud service's control.

  Both the above scenarios have been used while testing Calibre2opds with updog.

• Generate catalog: Run Calibre2opds to generate the catalog for your library.

• Check the local copy of the calibre2opds generated catalog
  If you have generated the HTML catalog then you can use a browser to do a basic confidence check on your library by clicking on the index.html folder in the Calibre2opds generated catalog. This will be working on the local copy of the files and is not dependent on the files being uploaded to the Cloud service servers.

• Sync files: Wait for the files to be synced to the servers of the Cloud service being used. this may take some time the first time you use the Cloud service. Later sync's should be mush faster.

• Check access to the catalog via the web:
  The next step is to check the same link accessing it via the web, and in this case the URL will normally be of the form
  

  https://<WebSite>/<calibre-library-folder>/<catalog-folder>/yyyyyyyyyindex.html

  where:
  <WebSite> is the name you gave to your web site under updog.co
    <calibre-library-folder> will be the name of the Calibre library folder holding the books
    <catalog-folder> will be the name of the atalog folder (defaults to _catalog)
    yyyyyyyy will only be present if you have enabled the option to encrypt filenames and will be specific to your library.

DropBox + Updog

As of 1st October 2016 DropBox has discontinued its ability to have a Public folder that can act as a web site. As a result DropBox in isolation is no longer suitable as a host for a calibre2opds generated library.

There is a workaround that now provides the same level of functionality by allowing https://updog.co/ to provide the web server functionality.

---

The Calibre2Opds development all started with a thread on MobileRead, where David Pierron explained how he used Dropbox (a free synching and sharing service) to publish his calibre library on the internet, and access it from his iPhone (using Stanza). Jane Lite has written a nice tutorial on her website :

The steps are:

• Install Dropbox
  this creates, somewhere in your file system, a Dropbox directory (usually called "My Dropbox" or "Dropbox"). As it is explained in the Dropbox FAQ, every file you put in this directory, on in a subfolder, will automatically be published to the Dropbox "cloud", and synchronized on other computers that are linked to your Dropbox account.

• Link Dropbox account to updog.co
  This step sets up https://updog.co to front-end the Calibre library and Calibre2opds catalog stored in Dropbox to enable them to function as though they were a traditional web site:
  • Go to https://updog.co and click on the Log In With DropBox button. This will result in a folder called apps/updog being created within your Dropbox folder. If you check your DropBox settings you will see that updog has been given access to just this folder and cannot access Dropbox files that are outside this folder.
  • Create a 'web site' under this folder. Updog.co will allow you to have a single web site of this type for free, but if you want it to handle multiple web sites then you need to pay for the Pro capability. Select a name for the site. This will result in a folder with the name of the web site being created under the Dropbox apps/updog folder (e.g. something like Dropbox/apps/updog/<website>). Updog will make this visible via a URL of the form https://<website>.updog.com where <website> corresponds to the site name you chose for your web site.
  • All files that are placed within this folder will now be treated as if they were a traditional web site with a URL of the form https:<sitename.updog.com. By default updog will look for a file called index.html within this web site, but you can over-ride this by giving the exact filename to be used.
  • Consider security implications
    Everything you put in there will be public, which means that if you give the pdog URL that is the internet address of this folder to someone else, he will be able to access your files without your Dropbox password. You may therefore want to select a non-obvious name for your web site, and also consider using the calibre2opds option to encrypt filenames within the catalog.

• Move Calibre library files to Dropbox
  Configure Calibre to set up its library in Dropbox (or in a subfolder there). If using the free Updog service and having the Calibre catalog at the base of your web site then the new location for the Calibre library will be of the form dropbox/apps/updog/<website> .
  You can either move the library from within Calibre by using the Switch/Create library option to move your library the the new folder location.
  This can take a while for large libraries so you may prefer to use a much faster method of moving the Calibre library folder (and all its contents) using your system File Manager and the use the Switch/Create Library option within Calibre to point to the new location.
  
  If you prefer to keep the master Calibre library out of Dropbox then you can use Calibre2Opds in its Publish mode and point it to the location in DropBox that will be used for your catalog. This will create a copy of all the book files required to support the catalog as well as the catalog files at that location.

• Sync Calibre Library files to DropBox
  Wait until Dropbox has uploaded all your books (it may take a while, depending on your upload capacity). Dropbox is free for the first two gigabytes only ; for some of us, who have a lot of books, this may not be enough... Time to shell out some cash ;). DropBox also has a referral program as a way of getting additional space on free accounts.

• Download and install the calibre2opds tool, and use it to generate the catalogs as a sub-folder of the Calibre library folder you have just set up under Dropbox.

• Sync Calibre2opds catalog files to Dropbox
  wait until Dropbox has uploaded all the files that make up the Calibre2opds generated catalog.

• Configure your ebook reader
  If you are using an OPDS compatible eReader application (e.g. MegaReader/QuickReader, i2Reader, Marvin, Aldiko, another OPDS reader) to access your published catalog. The exact format of the URL will depend on the values you selected for the web site name under updog.co, your Calibre library folder and the catalog folder name and will be similar in format to the example shown above. If using the OPDS version of the catalog then the URL will finish with .xml rather than .html.

You're done !

To summarize the above information if using the free Updog service and the Calibre library is the only thing being accessed via Updog and you are using Calibre2opds with default settings.

• Calibre library will be located at Dropbox/apps/updog/<website> where <website. is your chosen web site name.
• Calibre catalog will be generated to Dropbox/apps/updog/<website>/_catalog
• URLs for accessing the library will be of the form:
    HTML: http://<website>.updog.co/_catalog/index.html
    OPDS: http://<website>.updog.co/_catalog/index.xml
  If you use the option in Calibre2opds to encrypt filenames (recommended if your site is not password protected) then the index.hmtl/xml files in the catalog will have a prefix added and the URLs above need amending appropriately

TIP
For HTML catalogs where you have not chosen the option withion Calibre2opds to encrypt filenames then you can simplify the URL required to access your catalog by putting a index.html file in the base of your Calibre library (i.e. at the same level as the metadata.db file). If you are generating the catalog in the _catalog subfolder then suggested contents are:

<html>
<head>
<meta http-equiv="Refresh" content="0; URL=_catalog/index.html">
</head>
<body>
<body>
</html>

which simply redirects to the index.html file inside the calibre2opds generated catalog. If you use a different folder for your catalog then change the file appropriately. When you use this technique then the URL to access the HTML catalog can be simplified to
  http://<website.updog.co

If you need more info, and the description above is not clear enough, please ask a question in the forum or in the Calibre2opds forum

Google Drive

Google Drive is the cloud storage system provided by Google.

WARNING

You should be VERY CAREFUL about using Google Drive with Calibre libraries. This is because Google Drive has been reported as sometimes silently removing the values in brackets in book folder names which are used by Calibre to track books. This invalidates the Calibre library as it breaks the links between the Calibre metadata.db file and the underlying file system. Having said the above, it appears that as long as though you are only using Google Drive for storing a Cloud based copy of your library for accessing via a Calibre2opds generated catalog (and not for syncing it between different computers) then the approach described here can be used with safety. It is possible that Google Drive has changed this behavior since this warning was written, but it is up to you to make sure you are not being affected by this issue.

Google Drive has discontinued its ability to have a folder that can act as a web site. As a result Google Drive in isolation is no longer suitable as a host for a calibre2opds generated library. A Google Drive based Calibre library can be fronted by a Calibre2opds generated catalog hosted on a DropBox+updog combination, but you need to be paying for the 'Pro' version of the Updog service to have Google Drive support.

The steps described below used to work with Google Drive, but it is thought they may no longer be valid.

The process described here for putting a Calibre2opds catalog into Google Drive 'appears' to work - but one should be aware that if the renaming issue mentioned above happens then all cover images and download links will not work.

The steps to follow are:

• Create a folder in Google drive and copy your Calibre library to this folder.

• Google requires a index.html file to be present in this folder to act as your home page for this web site. If you are generating the catalog in the _catalog subfolder then suggested contents are:

<html>
<head>
<meta http-equiv="Refresh" content="0; URL=_catalog/index.html">
</head>
<body>
<body>
</html>

which simply redirects to the index.html file inside the calibre2opds generated catalog. If you use a different folder for your catalog then change the file appropriately. lso, if you use the option to encrypt filenames (recommended) then the index.hmtl file in the catalog will have a prefix added and this file needs amending appropriately.

• Right-Click the folder in Windows Explorer, go to the Google Drive entry, and from there select the option to share the folder. Select the option for public sharing. The results should look something like following:
  300px
  Make a note of the value in the link to share box, particularly the value following the id= bit up to (but not including the &usp= bit.
  The URL for accessing the HTML version of your library is now going to be of the form
  https://googledrive.com/host/value_you_noted/
  where value_you_noted is the value from the share link.

• Generate the Calibre2opds catalog using the default mode, specifying the folder you have just shared as the Library folder. This will generate the catalog as a sub-folder of this folder.

• On the Advanced tab of Calibre2opds make sure that the option to Minimize changes is set. This is of particular use in Cloud services where you want to avoid changing file unnecessarily as otherwise syncing can take a long time.

• When Calibre2opds has finished, wait for Google Drive to finish syncing the catalog. Note that this can take some time, especially the first time that you generate a catalog.

OneDrive

OneDrive is the cloud storage service provided by Microsoft. Integration with OneDrive is a standard feature of Windows 10.

As far as we know this is not a suitable cloud service for supporting a Calibre2opds hosted catalog.

https://www-drv.com might be able to help here

iCloud Drive

iClould Drive is the cloud storage system provided by Apple. Support for iCloud drive is integrated into MacOS and iOS devices (iPhone/iPad/iPod).

As far as we know this is not a suitable cloud service for supporting a Calibre2opds hosted catalog.

Other Cloud Services

There are many other cloud storage services available. Feedback on whether users have successfully used them to host a Calibre Library and Calibre2opds generated catalog would be welcomed.