Calibre2Opds Publish

From MobileRead
Jump to: navigation, search
Home
Page
Calibre2opds User Guide Developer Guide
Overview Installing Generating Publishing Using Troubleshooting


Publishing Calibre2opds Catalogs


This section of the User Guide provides information on publishing your Calibre2opds catalog.

It is split into the following main topic areas:

Contents

[edit] Overview

To be able to use your generated Calibre2Opds catalogs you need to make them visible on the network.

NOTE: If you are using calibre2opds in Nook mode then you do not need to publish your catalogs as they will have been copied (together with their associated eBooks) to the Nook and held there as local files

The basic requirements are that:

  • The method chosen provides a network path to the top level file in the generated Calibre2Opds catalog.
  • The method allows all catalog files and book files to be accessed as a path relative to that of the top level catalog file.

This will tend to break down into 2 distinct families of servers:

  • Web server.
    This can be your own private server, or one run by someone else on which you have space provided for you own web content. This could be an ISP provided web space - but if this is the case please read the caution about controlling access.

    Another (less obvious) solution that falls into this category is to use a WiFi enabled storage device that also supports a web server for accessing the files it is storing (just as if they were hosted on a private or ISP provided web server). One device that has proved a good match with Calibre2opds is the AirStash as it provides a web server tha can be accessed easily from any WiFi enabled device such as an iPhone, iPad, Android tablet/phone and allows the Calibre2opds catalog to be used from either a browser (HTML catalog) or a suitable OPDS aware reading app (OPDS catalog).

  • "Cloud" file storage.
    This is the generic term used to describe facilities where you can have files that are located on your own PC or Mac replicated to servers on the Internet so that they can be accessed from anywhere. If the Cloude service can expose the files that are stored on it just like a web server would then they can also host a Calibre2opds catalog. Dropbox and Google Drive which are described later in this document are examples of such services, although it is likely that others are also suitable.

    NOTE: Cloud services that only allow downloading of files via a browser (i.e. they cannot act as HTML web pages) will unfortunately not work. Examples of such services that are currently known to fall into this category are iCloud Drive; Copy.com; SugarSync and OneDrive (was SkyDrive)


[edit] Security

Security will not be an issue if you have your library on your home system, and set up so that it can only be accessed from devices that are part of your home network.

If you make your Calibre library visible on the public Internet then you need to make sure your books are not easily accessible by unauthorised users. Depending on where you live, failure to do so may mean that you are infringing copyright

This would typically involve providing a username/password challenge or making sure the URL used is not easily found (which is what most "cloud" services use). Calibre2opds provides an option to 'encrypt' filenames used within the catalog to make the filenames less predictable.


[edit] Web Server

Many users run web servers on their own home network, or alternatively make use of space on web servers that are provided by their ISP. The details of these can vary, but the following should provide some general guidance.


  • Copy your entire Calibre library folder to a location that is visible to the web server that you are using. This will include the Calibre2Opds generated catalogs as a sub-folder of the location holding the author folders from the Calibre library.
  • If you are using a public web server then they will normally tell you what URL should be used to access the files you have just uploaded.
  • If you are using a NAS (Network Addressable Storage) device then you can set up calibre to directly work with the files on the NAS thus avoiding the copying requirement.
  • If you are using a web server on you owwn local WiFi network and only want to access the Calibre2Opds catalogs while at home that is probably all you have to do.

If you are running on your own private web server and want to make the Calibre Library and Calibre2Opds catalogs visible on the public Internet then some additional steps will be required:

  • You will need an externally visible IP address and port that can be used for accessing your private web server.
  • If you are a Cable user then although strictly speaking you are probably using DHCP to get your external IP address assigned to you it tends to change very rearely so you can probably just query your modem/router to find out whqat address you are currently assigned and use that address directly. You still need to be aware that it can change.
  • If you are an ADSL user connecting via a telephone line then typically the address only persists for a limited period so you need a way of gettign a more dynamic mapping of the external IP address to something you can use. there are service that provide such a capability such as XXX. Note that there is nothing that stops cable customers using such a service as well.
  • Set up some NAT (Network Address Translation) entries in your router to allow an external IP port on the router to be mapped to a particular IP-Address/Port combination on your local network. This will be the IP_address:Port that is being used by the web server hosting your Calibre library and Calibre2Opds catalogs.
  • Make sure you have some sort of security mechanism in place to stop unauthorised access.
TIP:

Many web servers have the concept of a default page that is loaded if the user does not specify an explicit page in the URL they use to access the web site. Typically this is a file called index.htm or index.html located in the web servers document root. If you include a text file of this sort that contains content allong the lines of the following

<html>
<head>
<meta http-equiv="refresh" content="0;url=_catalog/index.html">
</head>
<body>
</body>
</html>

then this will automatically redirect the user to the first catalog page without the user having to know the full URL.

[edit] Using lighthttpd

[edit] Server optimization

lighthttpd is a lightweight web server that can work on a NAS with low memory. If the web server crashes while you're browsing the library, add the following line in the server configuration file (lighttpd.conf):

 server.event-handler = "poll"

and then restart the web server.

[edit] Using Windows Home Server (WHS)

There is a very good blog post that covers using the web server built into Windows Home Server (WHS) to host the catalog produced by calibre2opds. Although it has not been validated the steps probably apply to anyone using Internet Information Server (IIS) on a Windows system to host the catalog.

Note in particular the requirement to set up the Mime types for the ebook types you want to be able to download. This is required because IIS ignores the Mime types set by calibre2opds in the HTML and sets it according to the file extension. If you do not know the Mime type for a particular ebook file type then you can load the HTML produced by calibre2opds; find the line for downloading a particular ebook file; and then look at the value for the 'type' parameter on the download link.


[edit] Cloud Services

Many users want to use a internet based service to store their libraries and access them remotelyMany people store their libraries on an internet. As long as he product in use allows the files to be accessed via an URL, then this is no different to hosting the files on a web server. You can use Cloud Storage for hosting both the HTML and XML versions of the Calibre2opds catalogs.

The big advantage of this approach is that these product nearly always include an automatic sync facility to copy files between local PC/Mac systems and the cloud storage. this means that the user does not need to worry much about the management of the files as it is automatic.

The requirements for such products to work successfully with calibre2opds are:

  • There must be a base URL that can be used to access the top level calibre2opds catalog
  • It must be possible to navigate to the other catalog files using relative URLs
  • It must be possible to access the ebook files that are to be downloaded using relative URLs.

These tend to be standard functionality in most such products so they work without the user having to do anything special.

The one thing that you do need to be careful about is security. If the product in use does not provide a security mechanism that can be used with the calibre2opds catalogs then you may be breaching copyright regulations depening on your country of residence.

If you are using Cloud storage then you will normally want to take steps to minimise the space used to store the calibre2opds generated catalog. There are a number of calibre2opds settings that are likely to be relevant in such a scenario:

  • Generate either the OPDS (xml) version of the catalog or the HTML one. The default in calibre2opds is to generate both. Which variant you need will depend on what reading software or device(s) you are going to use. Generating only one variant of the catalog will halve the storage space used for the catalog. The options to control this are on the Catalog Generation Options tab.
  • Use the option to Minimize the number of changed files on the Catalog Generation Options tab. This does not reduce space but it does try and minimise the amount of data that needs to be synced to the cloud after a calibre2opds generation run. This option is normally set by default, but you may want to check what you have it set to on your system.
  • Use the options Do not Generated resized covers and Do not generate Thumbnails on the Advanced Customization tab. This avoids calibre2opds generating new image files optimised for this purpose and the catalog instead uses the existing Calibre cover.jpg files for this purpose. This will save significant storage space although it does mean that when you use the catalog more data has to be transmitted as you will be using image files for thumbnails and covers that are larger than is required. This is normally a desirable trade-off if you are using cloud based storage.
  • Use the option to Use external files for icons on the Catalog Generation Option tab. By default the little icons at the front of lines in the catalog are embedded in the XML/HTML pages that are generated. Using this option will instead reference files stored on the server. This reduces slightly the size of the files generated for each page in the catalog at the expense of slightly slower times to load the page when using the catalog. The space savings from this are minimal so many users do not bother.

[edit] DropBox + Updog

As of 1st October 2016 DropBox has discontinued its ability to have a Public folder that can act as a web site. As a result DropBox in isolation is no longer suitable as a host for a calibre2opds generated library.

There is a workaround that now provides the same level of functionality by allowing https://updog.co/ to provide the web server functionality.


The Calibre2Opds development all started with a thread on MobileRead, where David Pierron explained how he used Dropbox (a free synching and sharing service) to publish his calibre library on the internet, and access it from his iPhone (using Stanza). Jane Lite has written a nice tutorial on her website :

The steps are:

  • Install Dropbox
    this creates, somewhere in your file system, a Dropbox directory (usually called "My Dropbox" or "Dropbox"). As it is explained in the Dropbox FAQ, every file you put in this directory, on in a subfolder, will automatically be published to the Dropbox "cloud", and synchronized on other computers that are linked to your Dropbox account.
  • Link Dropbox account to updog.co
    This step sets up https://updog.co to front-end the Calibre library and Calibre2opds catalog stored in Dropbox to enable them to function as though they were a traditional web site:
    • Go to [https://updog.co] and click on the Log In With DropBox button. This will result in a folder called Apps/updog being created within your DropBox folder. If you check your DropBox settings you will see that updog has been given access to just this folder and cannot access DropBox files that are outside this folder.
    • Create a 'web site' under this folder. Updog.co will allow you to have a single web site of this type for free, but if you want it to handle multiple web sites then you need to make a one-off $5 payment.switch to a subscription. Select a name for the site. This will result in a folder with the name of the web site being created under the DropBox Apps/updog folder (e.g. something like DropBox/Apps/updog/<website. Updog will make this visible via a URL of the form https://website.updog.com where <website> corresponds to the site name you chose for your web ssite.
    • All files that are placed within this folder will now be treated as if they were a traditional web site with a URL of the form https:<sitename.updog.com. By default updog will look for a file called index.html within this web site, but you can over-ride this by giving the exact filename to be used.
    • Consider security implications
      Everything you put in there will be public, which means that if you give the pdog URL that is the internet address of this folder to someone else, he will be able to access your files without your Dropbox password. You may therefore want to select a non-obvious name for your web site, and also consider using the calibre2opds option to encrypt filenames within the catalog.
  • Move Calibre library to Dropbox
    Configure Calibre to set up its library in this (or in a subfolder there) : you can do this by opening up preferences and choosing a new folder location. This will take a while.
  • Sync Calibre Library files to DropBox
    Wait until Dropbox has uploaded all your books (it may take a while, depending on your upload capacity). Dropbox is free for the first two gigabytes only ; for some of us, who have a lot of books, this may not be enough... Time to shell out some cash ;). DropBox also has a referral program as a way of getting additional space on free accounts.
  • Download and install the calibre2opds tool, and use it to generate the catalogs as a sub-folder of the Calibre library folder you have just set up under Dropbox.
  • Check the local copy of the calibre2opds generated catalog
    If you have generated the HTML catalog then you can use a browser to do a basic confidence check on your library by clicking on the index.html folder in the Calibre2opds generated catalog. This will be working on the local copy of the files and is not dependent on the files being uploaded to DropBox.
  • Sync Calibre2opds catalog files to Dropbox
    wait until Dropbox has uploaded all the files that make up the Calibre2opds generated catalog.
  • Check access to the catalog via the web:
    The next step is to check the same link accessing it via the web, and in this case the URL will normally be of the form
    https://<WebSite>/<calibre-library-folder>/<catalog-folder>/yyyyyyyyyindex.html
    where:
    <WebSite> is the name you gave to your web site under updog.co
      <calibre-library-folder> will be the name of the Calibre library folder holding the books
      <catalog-folder> will be the name of the atalog folder (defaults to _catalog)
      yyyyyyyy will only be present if you have enabled the option to encrypt filenames and will be specific to your library.
  • Configure your ebook reader
    If you are using an OPDS compatible eReader application (e.g. MeagaReader/QuickReader, i2Reader, Marvin, Aldiko, another OPDS reader) to access your published catalog. The exact format of the URL will depend on the values you selected for the web site name under updog.co, your Calibre library folder and the catalog folder name and will be similar in format to the example shown above. If using the OPDS version of the catalog then the URL will finish with .xml rather than .html.

You're done !

If you need more info, if this FAQ is not clear enough, please ask a question in the forum or in the Calibre2opds forum

[edit] Google Drive

Google Drive is the cloud storage system provided by Google.

WARNING

You should be VERY CAREFUL about using Google Drive with Calibre libraries.

This is because Google Drive has been reported as sometimes silently removing the values in brackets in book folder names which are used by Calibre to track books. This invalidates the Calibre library as it breaks the links between the Calibre metadata.db file and the underlying file system.

Having said the above, it appears that as long as though you are only using Google Drive for storing a Cloud based copy of your library for accessing via a Calibre2opds generated catalog (and not for syncing it between different computers) then the approach described here can be used with safety.

It is possible that Google Drive has changed this behavior since this warning was written, but it is up to you to make sure you are not being affected by this issue.

It appears that Google had has discontinued its ability to have a folder that can act as a web site. As a result Google Drive in isolation is no longer suitable as a host for a calibre2opds generated library.

An investigation is under way to see if a Google Drive based Calibre library can be fronted by a Calibre2opds generated catalog hosted on a DropBox+updog combination. When this investigation has concluded the remainder of this section will be updated appropriately with the findings.


The steps described below used to work with Google Drive, but it is thought they may no longer be valid.


The process described here for putting a Calibre2opds catalog into Google Drive 'appears' to work - but one should be aware that if the renaming issue mentioned above happens then all cover images and download links will not work.

The steps to follow are:

  • Create a folder in Google drive and copy your Calibre library to this folder.
  • Google requires a index.html file to be present in this folder to act as your home page for this web site. If you are generating the catalog in the _catalog subfolder then suggested contents are:
<html>
<head>
<meta http-equiv="Refresh" content="0; URL=_catalog/index.html">
</head>
<body>
<body>
</html>

which simply redirects to the index.html file inside the calibre2opds generated catalog. If you use a different folder for your catalog then change the file appropriately. lso, if you use the option to encrypt filenames (recommended) then the index.hmtl file in the catalog will have a prefix added and this file needs amending appropriately.

  • Right-Click the folder in Windows Explorer, go to the Google Drive entry, and from there select the option to share the folder. Select the option for public sharing. The results should look something like following:
    300px
    Make a note of the value in the link to share box, particularly the value following the id= bit up to (but not including the &usp= bit.
    The URL for accessing the HTML version of your library is now going to be of the form
    https://googledrive.com/host/value_you_noted/
    where value_you_noted is the value from the share link.
  • Generate the Calibre2opds catalog using the default mode, specifying the folder you have just shared as the Library folder. This will generate the catalog as a sub-folder of this folder.
  • On the Advanced tab of Calibre2opds make sure that the option to Minimize changes is set. This is of particular use in Cloud services where you want to avoid changing file unnecessarily as otherwise syncing can take a long time.
  • When Calibre2opds has finished, wait for Google Drive to finish syncing the catalog. Note that this can take some time, especially the first time that you generate a catalog.

[edit] OneDrive

OneDrive is the cloud storage service provided by Microsoft. Integration with OneDrive is a standard feature of Windows 10.

As far as we know this is not a suitable cloud service for supporting a Calibre2opds hosted catalog.

[edit] iCloud Drive

iClould Drive is the cloud storage system provided by Apple. Support for iCloud drive is integrated into MacOS and iOS devices (iPhone/iPad/iPod).

As far as we know this is not a suitable cloud service for supporting a Calibre2opds hosted catalog.

[edit] Other Cloud Services

There are many other cloud storage services available. Feedback on whether users have successfully used them to host a Calibre Library and Calibre2opds generated catalog would be welcomed.

Personal tools
Namespaces

Variants
Actions
Navigation
MobileRead Networks
Toolbox