curl: add support for the IPFS protocols:

- ipfs://<cid> - ipns://<cid> This allows you tu use ipfs in curl like: curl ipfs://<cid> and curl ipns://<cid> For more information consult the readme at: https://curl.se/docs/ipfs.html Closes #8805
2022-03-27 01:31:58 +01:00 · 2022-03-27 01:31:58 +01:00 · 65b563a96a
--- a/.github/scripts/spellcheck.words
+++ b/.github/scripts/spellcheck.words
@ -157,6 +157,8 @@ deepcode
 DELE
 DER
 deselectable
+Deserialized
+deserialization
 destructor
 detections
 dev
@ -344,6 +346,10 @@ interoperable
 interoperates
 IoT
 ipadOS
+IPFS
+IPNS
+ipld
+trustless
 IPCXN
 IPv
 IPv4
@ -854,6 +860,7 @@ Viktor
 VM
 VMS
 VMware
+vnd
 VRF
 VRFY
 VSE
--- a/docs/IPFS.md
+++ b/docs/IPFS.md
@ -0,0 +1,82 @@
+# IPFS
+For an overview about IPFS, visit the [IPFS project site](https://ipfs.tech/).
+
+In IPFS there are two protocols. IPFS and IPNS (their workings are explained in detail [here](https://docs.ipfs.tech/concepts/)). The ideal way to access data on the IPFS network is trough those protocols. For example to access the Big Buck Bunny video the ideal way to access it is like: `ipfs://bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzi`
+
+## IPFS Gateways
+
+IPFS Gateway acts as a bridge between traditional HTTP clients and IPFS.
+IPFS Gateway specifications of HTTP semantics can be found [here](https://specs.ipfs.tech/http-gateways/).
+
+### Deserialized responses
+
+By default, a gateway acts as a bridge between traditional HTTP clients and IPFS and performs necessary hash verification and deserialization. Through such gateway, users can download files, directories, and other content-addressed data stored with IPFS or IPNS as if they were stored in a traditional web server.
+
+### Verifiable responses
+
+By explicitly requesting [application/vnd.ipld.raw](https://www.iana.org/assignments/media-types/application/vnd.ipld.raw) or [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car) responses, by means defined in [Trustless Gateway Specification](https://specs.ipfs.tech/http-gateways/trustless-gateway/), the user is able to fetch raw content-addressed data and [perform hash verification themselves](https://docs.ipfs.tech/reference/http/gateway/#trustless-verifiable-retrieval).
+
+This enables users to use untrusted, public gateways without worrying they might return invalid/malicious bytes.
+
+## IPFS and IPNS protocol handling
+There are various ways to access data from the IPFS network. One such way is through the concept of public "[gateways](https://docs.ipfs.tech/concepts/ipfs-gateway/#overview)". The short version is that entities can offer gateway services. An example here that is hosted by Protocol Labs (who also makes IPFS) is `dweb.link` and `ipfs.io`. Both sites expose gateway functionality. Getting a file through `ipfs.io` looks like this: `https://ipfs.io/ipfs/bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzi`
+
+If you were to be [running your own IPFS node](https://docs.ipfs.tech/how-to/command-line-quick-start/) then you, by default, also have a [local gateway](https://specs.ipfs.tech/http-gateways/) running. In it's default configuration the earlier example would then also work in this link: `http://127.0.0.1:8080/ipfs/bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzi`
+
+## cURL handling of the IPFS protocols
+The IPFS integration in cURL hides this gateway logic for you. So instead of providing a full URL to a file on IPFS like this:
+```
+curl http://127.0.0.1:8080/ipfs/bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzi
+```
+
+You can provide it with the IPFS protocol instead:
+```
+curl ipfs://bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzi
+```
+
+With the IPFS protocol way of asking a file, cURL still needs to know the gateway. curl essentially just rewrites the IPFS based URL to a gateway URL.
+
+### IPFS_GATEWAY environment variable
+If the `IPFS_GATEWAY` environment variable is found, it's value is used as gateway.
+
+### Automatic gateway detection
+When you provide no additional details to cURL then cURL will:
+
+1. First look for the `IPFS_GATEWAY` environment variable and use that if it's set.
+2. Look for the file: `~/.ipfs/gateway`. If it can find that file then it means that you have a local gateway running and that file contains the URL to your local gateway.
+
+If cURL fails you'll be presented with an error message and a link to this page to the option most applicable to solving the issue.
+
+### `--ipfs-gateway` argument
+You can also provide a `--ipfs-gateway` argument to cURL. This overrules any other gateway setting. curl won't fallback to the other options if the provided gateway didn't work.
+
+## Gateway redirects
+A gateway could redirect to another place. For example, `dweb.link` redirects [path based](https://docs.ipfs.tech/how-to/address-ipfs-on-web/#path-gateway) requests to [subdomain based](https://docs.ipfs.tech/how-to/address-ipfs-on-web/#subdomain-gateway) ones. So a request to:
+```
+curl ipfs://bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzi --ipfs-gateway https://dweb.link
+```
+Which would be translated to:
+```
+https://dweb.link/ipfs/bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzi
+```
+Will redirect to:
+```
+https://bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzi.ipfs.dweb.link
+```
+If you trust this behavior from your gateway of choice then passing the `-L` option will follow the redirect.
+
+## Error messages and hints
+Depending on the arguments, cURL could present the user with an error.
+
+### Gateway file and environment variable
+cURL tried to look for the file: `~/.ipfs/gateway` but couldn't find it. It also tried to look for the `IPFS_GATEWAY` environment variable but couldn't find that either. This happens when no extra arguments are passed to cURL and letting it try to figure it out [automatically](#Automatic-gateway-detection).
+
+Any IPFS implementation that has gateway support should expose it's URL in `~/.ipfs/gateway`. If you are already running a gateway, make sure it exposes the file where cURL expects to find it.
+
+Alternatively you could set the `IPFS_GATEWAY` environment variable or pass the `--ipfs-gateway` flag to the cURL command.
+
+### Malformed gateway URL
+The command executed evaluates in an invalid URL. This could be anywhere in the URL, but a likely point is a wrong gateway URL.
+
+Inspect your URL.
+Alternatively opt to go for the [automatic](#Automatic-gateway-detection) gateway detection.
--- a/docs/cmdline-opts/Makefile.inc
+++ b/docs/cmdline-opts/Makefile.inc
@ -114,6 +114,7 @@ DPAGES = \
  include.d \
  insecure.d \
  interface.d \
+  ipfs-gateway.d \
  ipv4.d \
  ipv6.d \
  json.d \
--- a/docs/cmdline-opts/ipfs-gateway.d
+++ b/docs/cmdline-opts/ipfs-gateway.d
@ -0,0 +1,44 @@
+c: Copyright (C) 2023, Mark Gaiser, <markg85@gmail.com>
+SPDX-License-Identifier: curl
+Long: ipfs-gateway
+Arg: <URL>
+Help: Defines the gateway to handle IPFS protocol requests.
+Added: 8.4.0
+See-also: help manual
+Category: ipfs
+Example: --ipfs-gateway $URL ipfs://
+Multi: single
+---
+Specifies which gateway to use for IPFS and IPNS URLs.
+Not specifying this argument will let cURL try to automatically
+check if IPFS_GATEWAY environment variable is set,
+or if ~/.ipfs/gateway plain text file exists.
+
+If you run a local IPFS node, this gateway is by default
+available under http://localhost:8080. A full example URL would
+look like:
+
+ curl --ipfs-gateway http://localhost:8080 ipfs://bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzi
+
+
+You can also specify publicly available gateways. One such
+gateway is https://ipfs.io. A full example url would look like:
+
+ curl --ipfs-gateway https://ipfs.io ipfs://bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzi
+
+
+There are many public IPFS gateways. As a starting point to find
+one that works for your case, consult this page:
+
+ https://ipfs.github.io/public-gateway-checker/
+
+
+A word of caution! When you opt to go for a remote gateway you should
+be aware that you completely trust the gateway. This is fine in local gateways
+as you host it yourself. With remote gateways there could potentially be
+a malicious actor returning you data that does not match the request you made,
+inspect or even interfere with the request. You won't notice this when using cURL.
+A mitigation could be to go for a "trustless" gateway. This means you
+locally verify that the data. Consult the docs page on trusted vs trustless:
+https://docs.ipfs.tech/reference/http/gateway/#trusted-vs-trustless
+
--- a/docs/options-in-versions
+++ b/docs/options-in-versions
@ -97,6 +97,7 @@
 --http3                              7.66.0
 --http3-only                         7.88.0
 --ignore-content-length              7.14.1
+--ipfs-gateway                       8.4.0
 --include (-i)                       4.8
 --insecure (-k)                      7.10
 --interface                          7.3
--- a/src/tool_cfgable.c
+++ b/src/tool_cfgable.c
@ -109,6 +109,7 @@ static void free_config_fields(struct OperationConfig *config)
  config->url_get = NULL;
  config->url_out = NULL;

+  Curl_safefree(config->ipfs_gateway);
  Curl_safefree(config->doh_url);
  Curl_safefree(config->cipher_list);
  Curl_safefree(config->proxy_cipher_list);
--- a/src/tool_cfgable.h
+++ b/src/tool_cfgable.h
@ -128,6 +128,7 @@ struct OperationConfig {
  struct getout *url_get;   /* point to the node to fill in URL */
  struct getout *url_out;   /* point to the node to fill in outfile */
  struct getout *url_ul;    /* point to the node to fill in upload */
+  char *ipfs_gateway;
  char *doh_url;
  char *cipher_list;
  char *proxy_cipher_list;
--- a/src/tool_getparam.c
+++ b/src/tool_getparam.c
@ -116,6 +116,7 @@ static const struct LongShort aliases[]= {
  {"*r", "create-dirs",              ARG_BOOL},
  {"*R", "create-file-mode",         ARG_STRING},
  {"*s", "max-redirs",               ARG_STRING},
+  {"*S", "ipfs-gateway",             ARG_STRING},
  {"*t", "proxy-ntlm",               ARG_BOOL},
  {"*u", "crlf",                     ARG_BOOL},
  {"*v", "stderr",                   ARG_FILENAME},
@ -1137,6 +1138,10 @@ ParameterError getparameter(const char *flag, /* f or -long-flag */
          err = PARAM_BAD_NUMERIC;
        break;

+      case 'S': /* ipfs gateway url */
+        GetStr(&config->ipfs_gateway, nextarg);
+        break;
+
      case 't': /* --proxy-ntlm */
        if(!feature_ntlm) {
          err = PARAM_LIBCURL_DOESNT_SUPPORT;
--- a/src/tool_help.h
+++ b/src/tool_help.h
@ -54,20 +54,21 @@ struct helptxt {
 #define CURLHELP_HTTP 1u << 7u
 #define CURLHELP_IMAP 1u << 8u
 #define CURLHELP_IMPORTANT 1u << 9u
-#define CURLHELP_MISC 1u << 10u
-#define CURLHELP_OUTPUT 1u << 11u
-#define CURLHELP_POP3 1u << 12u
-#define CURLHELP_POST 1u << 13u
-#define CURLHELP_PROXY 1u << 14u
-#define CURLHELP_SCP 1u << 15u
-#define CURLHELP_SFTP 1u << 16u
-#define CURLHELP_SMTP 1u << 17u
-#define CURLHELP_SSH 1u << 18u
-#define CURLHELP_TELNET 1u << 19u
-#define CURLHELP_TFTP 1u << 20u
-#define CURLHELP_TLS 1u << 21u
-#define CURLHELP_UPLOAD 1u << 22u
-#define CURLHELP_VERBOSE 1u << 23u
+#define CURLHELP_IPFS 1u << 10u
+#define CURLHELP_MISC 1u << 11u
+#define CURLHELP_OUTPUT 1u << 12u
+#define CURLHELP_POP3 1u << 13u
+#define CURLHELP_POST 1u << 14u
+#define CURLHELP_PROXY 1u << 15u
+#define CURLHELP_SCP 1u << 16u
+#define CURLHELP_SFTP 1u << 17u
+#define CURLHELP_SMTP 1u << 18u
+#define CURLHELP_SSH 1u << 19u
+#define CURLHELP_TELNET 1u << 20u
+#define CURLHELP_TFTP 1u << 21u
+#define CURLHELP_TLS 1u << 22u
+#define CURLHELP_UPLOAD 1u << 23u
+#define CURLHELP_VERBOSE 1u << 24u

 extern const struct helptxt helptext[];

--- a/src/tool_libinfo.c
+++ b/src/tool_libinfo.c
@ -51,6 +51,8 @@ const char *proto_rtsp = NULL;
 const char *proto_scp = NULL;
 const char *proto_sftp = NULL;
 const char *proto_tftp = NULL;
+const char *proto_ipfs = "ipfs";
+const char *proto_ipns = "ipns";

 static struct proto_name_tokenp {
  const char   *proto_name;
--- a/src/tool_libinfo.h
+++ b/src/tool_libinfo.h
@ -44,6 +44,8 @@ extern const char *proto_rtsp;
 extern const char *proto_scp;
 extern const char *proto_sftp;
 extern const char *proto_tftp;
+extern const char *proto_ipfs;
+extern const char *proto_ipns;

 extern bool feature_altsvc;
 extern bool feature_brotli;
--- a/src/tool_listhelp.c
+++ b/src/tool_listhelp.c
@ -308,6 +308,9 @@ const struct helptxt helptext[] = {
   CURLHELP_CONNECTION | CURLHELP_DNS},
  {"-6, --ipv6",
   "Resolve names to IPv6 addresses",
+   CURLHELP_IPFS},
+  {"--ipfs-gateway",
+   "Defines the gateway to handle IPFS protocol requests",
   CURLHELP_CONNECTION | CURLHELP_DNS},
  {"    --json <data>",
   "HTTP POST JSON",
--- a/src/tool_operate.c
+++ b/src/tool_operate.c
@ -45,8 +45,6 @@
 #  include <proto/dos.h>
 #endif

-#include "strcase.h"
-
 #define ENABLE_CURLX_PRINTF
 /* use our own printf() functions */
 #include "curlx.h"
@ -699,30 +697,245 @@ noretry:
  return result;
 }

+static char *ipfs_gateway(void)
+{
+  char *gateway = NULL;
+  char *ipfs_path = NULL;
+  char *gateway_composed_file_path = NULL;
+  FILE *gateway_file = NULL;
+
+  gateway = curlx_getenv("IPFS_GATEWAY");
+
+  /* Gateway is found from environment variable. */
+  if(gateway && strlen(gateway)) {
+    char *composed_gateway = NULL;
+    bool add_slash = (gateway[strlen(gateway) - 1] == '/') ? FALSE : TRUE;
+    composed_gateway = aprintf("%s%s", gateway, (add_slash) ? "/" : "");
+    Curl_safefree(gateway);
+    gateway = aprintf("%s", composed_gateway);
+    Curl_safefree(composed_gateway);
+    return gateway;
+  }
+
+  /* Try to find the gateway in the IPFS data folder. */
+  ipfs_path = curlx_getenv("IPFS_PATH");
+
+  if(!ipfs_path) {
+    char *home = NULL;
+    home = curlx_getenv("HOME");
+    /* Empty path, fallback to "~/.ipfs", as that's the default location. */
+    ipfs_path = aprintf("%s/.ipfs/", home);
+    Curl_safefree(home);
+  }
+
+  if(!ipfs_path) {
+    Curl_safefree(gateway);
+    Curl_safefree(ipfs_path);
+    return NULL;
+  }
+
+  gateway_composed_file_path = aprintf("%sgateway", ipfs_path);
+
+  if(!gateway_composed_file_path) {
+    Curl_safefree(gateway);
+    Curl_safefree(ipfs_path);
+    return NULL;
+  }
+
+  gateway_file = fopen(gateway_composed_file_path, FOPEN_READTEXT);
+  Curl_safefree(gateway_composed_file_path);
+
+  if(gateway_file) {
+    char *gateway_buffer = NULL;
+
+    if((PARAM_OK == file2string(&gateway_buffer, gateway_file)) &&
+       gateway_buffer) {
+      bool add_slash = (gateway_buffer[strlen(gateway_buffer) - 1] == '/')
+        ? FALSE
+        : TRUE;
+
+      gateway = aprintf("%s%s", gateway_buffer, (add_slash) ? "/" : "");
+      Curl_safefree(gateway_buffer);
+    }
+
+    if(gateway_file)
+      fclose(gateway_file);
+
+    if(!gateway) {
+      Curl_safefree(gateway);
+      Curl_safefree(ipfs_path);
+      return NULL;
+    }
+
+    Curl_safefree(ipfs_path);
+    return gateway;
+  }
+
+  Curl_safefree(gateway);
+  Curl_safefree(ipfs_path);
+  return NULL;
+}
+
+/*
+ * Rewrite ipfs://<cid> and ipns://<cid> to a HTTP(S)
+ * URL that can be handled by an IPFS gateway.
+ */
+static CURLcode ipfs_url_rewrite(CURLU *uh, const char *protocol, char **url,
+                                 struct OperationConfig *config)
+{
+  CURLcode result = CURLE_URL_MALFORMAT;
+  CURLUcode urlGetResult;
+  char *gateway = NULL;
+  char *cid = NULL;
+  char *pathbuffer = NULL;
+  CURLU *ipfsurl = curl_url();
+
+  if(!ipfsurl) {
+    result = CURLE_FAILED_INIT;
+    goto clean;
+  }
+
+  urlGetResult = curl_url_get(uh, CURLUPART_HOST, &cid, CURLU_URLDECODE);
+
+  if(urlGetResult) {
+    goto clean;
+  }
+
+  if(!cid) {
+    goto clean;
+  }
+
+  /* We might have a --ipfs-gateway argument. Check it first and use it. Error
+   * if we do have something but if it's an invalid url.
+   */
+  if(config->ipfs_gateway) {
+    if(curl_url_set(ipfsurl, CURLUPART_URL, config->ipfs_gateway,
+                    CURLU_GUESS_SCHEME)
+                    == CURLUE_OK) {
+      gateway = strdup(config->ipfs_gateway);
+      if(!gateway) {
+        result = CURLE_URL_MALFORMAT;
+        goto clean;
+      }
+
+    }
+    else {
+      result = CURLE_BAD_FUNCTION_ARGUMENT;
+      goto clean;
+    }
+  }
+  else {
+    gateway = ipfs_gateway();
+    if(!gateway) {
+      result = CURLE_FILE_COULDNT_READ_FILE;
+      goto clean;
+    }
+
+    if(curl_url_set(ipfsurl, CURLUPART_URL, gateway, CURLU_GUESS_SCHEME
+                    | CURLU_NON_SUPPORT_SCHEME) != CURLUE_OK) {
+      goto clean;
+    }
+  }
+
+  pathbuffer = aprintf("%s/%s", protocol, cid);
+  if(!pathbuffer) {
+    goto clean;
+  }
+
+  if(curl_url_set(ipfsurl, CURLUPART_PATH, pathbuffer, CURLU_URLENCODE)
+                  != CURLUE_OK) {
+    goto clean;
+  }
+
+  /* Free whatever it has now, rewriting is next */
+  Curl_safefree(*url);
+
+  if(curl_url_get(ipfsurl, CURLUPART_URL, url, CURLU_URLENCODE)
+                  != CURLUE_OK) {
+    goto clean;
+  }
+
+  result = CURLE_OK;
+
+clean:
+  curl_free(gateway);
+  curl_free(cid);
+  curl_free(pathbuffer);
+
+  if(ipfsurl) {
+    curl_url_cleanup(ipfsurl);
+  }
+
+  switch(result) {
+  case CURLE_URL_MALFORMAT:
+    helpf(stderr, "malformed URL. Visit https://curl.se/"
+          "docs/ipfs.html#Gateway-file-and-"
+          "environment-variable for more "
+          "information for more information");
+    break;
+  case CURLE_FILE_COULDNT_READ_FILE:
+    helpf(stderr, "IPFS automatic gateway detection "
+          "failure. Visit https://curl.se/docs/"
+          "ipfs.html#Malformed-gateway-URL for "
+          "more information for more "
+          "information");
+    break;
+  case CURLE_BAD_FUNCTION_ARGUMENT:
+    helpf(stderr, "--ipfs-gateway argument results in "
+          "malformed URL. Visit https://curl.se/"
+          "docs/ipfs.html#Malformed-gateway-URL "
+          "for more information for more "
+          "information");
+    break;
+  default:
+    break;
+  }
+
+  return result;
+}
+
 /*
 * Return the protocol token for the scheme used in the given URL
 */
-static const char *url_proto(char *url)
+static CURLcode url_proto(char **url,
+                          struct OperationConfig *config,
+                          char **scheme)
 {
+  CURLcode result = CURLE_OK;
  CURLU *uh = curl_url();
  const char *proto = NULL;
+  *scheme = NULL;

  if(uh) {
-    if(url) {
-      if(!curl_url_set(uh, CURLUPART_URL, url,
-                       CURLU_GUESS_SCHEME | CURLU_NON_SUPPORT_SCHEME)) {
-        char *schemep = NULL;
-        if(!curl_url_get(uh, CURLUPART_SCHEME, &schemep,
-                         CURLU_DEFAULT_SCHEME) &&
-           schemep) {
-          proto = proto_token(schemep);
-          curl_free(schemep);
+    if(*url) {
+      char *schemep = NULL;
+
+      if(!curl_url_set(uh, CURLUPART_URL, *url,
+                       CURLU_GUESS_SCHEME | CURLU_NON_SUPPORT_SCHEME) &&
+         !curl_url_get(uh, CURLUPART_SCHEME, &schemep,
+                       CURLU_DEFAULT_SCHEME)) {
+        if(curl_strequal(schemep, proto_ipfs) ||
+           curl_strequal(schemep, proto_ipns)) {
+          result = ipfs_url_rewrite(uh, schemep, url, config);
+
+          /* short-circuit proto_token, we know it's ipfs or ipns */
+          if(curl_strequal(schemep, proto_ipfs))
+            proto = proto_ipfs;
+          else if(curl_strequal(schemep, proto_ipns))
+            proto = proto_ipns;
+
        }
+        else
+          proto = proto_token(schemep);
+
+        curl_free(schemep);
      }
    }
    curl_url_cleanup(uh);
  }
-  return proto? proto: "???";   /* Never match if not found. */
+
+  *scheme = (char *) (proto? proto: "???");   /* Never match if not found. */
+  return result;
 }

 /* create the next (singular) transfer */
@ -738,6 +951,7 @@ static CURLcode single_transfer(struct GlobalConfig *global,
  bool orig_isatty = global->isatty;
  struct State *state = &config->state;
  char *httpgetfields = state->httpgetfields;
+
  *added = FALSE; /* not yet */

  if(config->postfields) {
@ -866,7 +1080,7 @@ static CURLcode single_transfer(struct GlobalConfig *global,
        struct OutStruct *etag_save;
        struct HdrCbData *hdrcbdata = NULL;
        struct OutStruct etag_first;
-        const char *use_proto;
+        char *use_proto;
        CURL *curl;

        /* --etag-save */
@ -1265,7 +1479,10 @@ static CURLcode single_transfer(struct GlobalConfig *global,
        if(result)
          break;

-        use_proto = url_proto(per->this_url);
+        /* result is only used when for ipfs and ipns, ignored otherwise */
+        result = url_proto(&per->this_url, config, &use_proto);
+        if(result && (use_proto == proto_ipfs || use_proto == proto_ipns))
+          break;

        /* On most modern OSes, exiting works thoroughly,
           we'll clean everything up via exit(), so don't bother with
@ -1522,7 +1739,6 @@ static CURLcode single_transfer(struct GlobalConfig *global,
        my_setopt_str(curl, CURLOPT_PROXY_KEYPASSWD, config->proxy_key_passwd);

        if(use_proto == proto_scp || use_proto == proto_sftp) {
-
          /* SSH and SSL private key uses same command-line option */
          /* new in libcurl 7.16.1 */
          my_setopt_str(curl, CURLOPT_SSH_PRIVATE_KEYFILE, config->key);
--- a/tests/data/DISABLED
+++ b/tests/data/DISABLED
@ -70,6 +70,9 @@
 266
 579
 587
+722
+724
+727
 # 1021 re-added here due to flakiness
 1021
 1117
--- a/tests/data/Makefile.inc
+++ b/tests/data/Makefile.inc
@ -101,7 +101,8 @@ test681 test682 test683 test684 test685 test686 test687 test688 \
 \
 test700 test701 test702 test703 test704 test705 test706 test707 test708 \
 test709 test710 test711 test712 test713 test714 test715 test716 test717 \
-test718 test719 test720 test721 \
+test718 test719 test720 test721 test722 test723 test724 test725 test726 \
+test727 \
 \
 test799 test800 test801 test802 test803 test804 test805 test806 test807 \
 test808 test809 test810 test811 test812 test813 test814 test815 test816 \
--- a/tests/data/test722
+++ b/tests/data/test722
@ -0,0 +1,52 @@
+<testcase>
+<info>
+<keywords>
+IPFS
+</keywords>
+</info>
+
+#
+# Server-side
+<reply>
+<data>
+HTTP/1.1 200 OK
+Date: Tue, 09 Nov 2010 14:49:00 GMT
+Server: test-server/fake
+Last-Modified: Tue, 13 Jun 2000 12:10:00 GMT
+ETag: "21025-dc7-39462498"
+Accept-Ranges: bytes
+Content-Length: 21
+Connection: close
+Content-Type: text/plain
+Funny-head: yesyes
+
+Hello curl from IPFS
+</data>
+</reply>
+
+#
+# Client-side
+<client>
+<server>
+http
+</server>
+ <name>
+IPFS
+ </name>
+ <command>
+--ipfs-gateway http://%HOSTIP:%HTTPPORT/%TESTNUMBER ipfs://QmV5JejrpgUxnkZeFZYMxVCqAbKy3KdPXWXyuEDiMNZwUx
+</command>
+</client>
+
+#
+# Verify data after the test has been "shot"
+<verify>
+<protocol crlf="yes">
+GET /ipfs/QmV5JejrpgUxnkZeFZYMxVCqAbKy3KdPXWXyuEDiMNZwUx HTTP/1.1
+Host: %HOSTIP:%HTTPPORT
+User-Agent: curl/%VERSION
+Accept: */*
+
+</protocol>
+</verify>
+</testcase>
--- a/tests/data/test723
+++ b/tests/data/test723
@ -0,0 +1,35 @@
+<testcase>
+<info>
+<keywords>
+IPFS
+</keywords>
+</info>
+
+#
+# Server-side
+<reply>
+</reply>
+
+#
+# Client-side
+<client>
+<server>
+http
+</server>
+ <name>
+IPFS with malformed gateway URL (bad function argument error)
+ </name>
+ <command>
+--ipfs-gateway http://nonexisting,local:8080/%TESTNUMBER ipfs://QmV5JejrpgUxnkZeFZYMxVCqAbKy3KdPXWXyuEDiMNZwUx
+</command>
+</client>
+
+#
+# Verify data after the test has been "shot"
+<verify>
+# malformed gateway URL (bad function argument error)
+<errorcode>
+43
+</errorcode>
+</verify>
+</testcase>
--- a/tests/data/test724
+++ b/tests/data/test724
@ -0,0 +1,58 @@
+<testcase>
+<info>
+<keywords>
+IPFS
+</keywords>
+</info>
+
+#
+# Server-side
+<reply>
+<data>
+HTTP/1.1 200 OK
+Date: Tue, 09 Nov 2010 14:49:00 GMT
+Server: test-server/fake
+Last-Modified: Tue, 13 Jun 2000 12:10:00 GMT
+ETag: "21025-dc7-39462498"
+Accept-Ranges: bytes
+Content-Length: 21
+Connection: close
+Content-Type: text/plain
+Funny-head: yesyes
+
+Hello curl from IPFS
+</data>
+</reply>
+
+#
+# Client-side
+<client>
+<server>
+http
+</server>
+<setenv>
+HOME=%PWD/log
+</setenv>
+ <name>
+IPFS with gateway URL from gateway file
+ </name>
+ <command>
+ipfs://QmV5JejrpgUxnkZeFZYMxVCqAbKy3KdPXWXyuEDiMNZwUx
+</command>
+<file name="log/.ipfs/gateway" >
+http://%HOSTIP:%HTTPPORT/%TESTNUMBER
+</file>
+</client>
+
+#
+# Verify data after the test has been "shot"
+<verify>
+<protocol crlf="yes">
+GET /ipfs/QmV5JejrpgUxnkZeFZYMxVCqAbKy3KdPXWXyuEDiMNZwUx HTTP/1.1
+Host: %HOSTIP:%HTTPPORT
+User-Agent: curl/%VERSION
+Accept: */*
+
+</protocol>
+</verify>
+</testcase>
--- a/tests/data/test725
+++ b/tests/data/test725
@ -0,0 +1,40 @@
+<testcase>
+<info>
+<keywords>
+IPFS
+</keywords>
+</info>
+
+#
+# Server-side
+<reply>
+</reply>
+
+#
+# Client-side
+<client>
+<server>
+http
+</server>
+<setenv>
+HOME=%PWD/log
+</setenv>
+ <name>
+IPFS with malformed gateway URL from gateway file
+ </name>
+ <command>
+ipfs://QmV5JejrpgUxnkZeFZYMxVCqAbKy3KdPXWXyuEDiMNZwUx
+</command>
+<file name="log/.ipfs/gateway" >
+http://nonexisting,local:8080/%TESTNUMBER
+</file>
+</client>
+
+#
+# Verify data after the test has been "shot"
+<verify>
+<errorcode>
+3
+</errorcode>
+</verify>
+</testcase>
--- a/tests/data/test726
+++ b/tests/data/test726
@ -0,0 +1,40 @@
+<testcase>
+<info>
+<keywords>
+IPFS
+</keywords>
+</info>
+
+#
+# Server-side
+<reply>
+</reply>
+
+#
+# Client-side
+<client>
+<server>
+http
+</server>
+<setenv>
+#
+# Set a home that doesn't have a ".ipfs" folder. %PWD should be good.
+# This is to prevent the automatic gateway detection from finding a gateway file in your home folder.
+HOME=%PWD
+</setenv>
+ <name>
+IPFS with no gateway URL (no environment or home file either)
+ </name>
+ <command>
+ipfs://QmV5JejrpgUxnkZeFZYMxVCqAbKy3KdPXWXyuEDiMNZwUx
+</command>
+</client>
+
+#
+# Verify with no gateway url and no auto detection
+<verify>
+<errorcode>
+37
+</errorcode>
+</verify>
+</testcase>
--- a/tests/data/test727
+++ b/tests/data/test727
@ -0,0 +1,52 @@
+<testcase>
+<info>
+<keywords>
+IPNS
+</keywords>
+</info>
+
+#
+# Server-side
+<reply>
+<data>
+HTTP/1.1 200 OK
+Date: Tue, 09 Nov 2010 14:49:00 GMT
+Server: test-server/fake
+Last-Modified: Tue, 13 Jun 2000 12:10:00 GMT
+ETag: "21025-dc7-39462498"
+Accept-Ranges: bytes
+Content-Length: 21
+Connection: close
+Content-Type: text/plain
+Funny-head: yesyes
+
+Hello curl from IPNS
+</data>
+</reply>
+
+#
+# Client-side
+<client>
+<server>
+http
+</server>
+ <name>
+IPNS
+ </name>
+ <command>
+--ipfs-gateway http://%HOSTIP:%HTTPPORT/%TESTNUMBER ipns://QmV5JejrpgUxnkZeFZYMxVCqAbKy3KdPXWXyuEDiMNZwUx
+</command>
+</client>
+
+#
+# Verify data after the test has been "shot"
+<verify>
+<protocol crlf="yes">
+GET /ipns/QmV5JejrpgUxnkZeFZYMxVCqAbKy3KdPXWXyuEDiMNZwUx HTTP/1.1
+Host: %HOSTIP:%HTTPPORT
+User-Agent: curl/%VERSION
+Accept: */*
+
+</protocol>
+</verify>
+</testcase>