1. Introduction to NFFinder's API

NFFinder web service relies on a REST API. This means (among other things) that all interactions with NFFinder are performed through HTTP requests. For instance, if you want to create a new job to mine NFFinder's database you will have to make a POST request at some specific URL and NFFinder will answer some HTTP status code (in our example, if a new job is created a 201 CREATED is to be expected, or a 200 OK if the job had been already created), some HTTP headers and the content of the response (for instance, a JSON with the job requested).

Having a REST API also implies that the only entry point in our application is the initial URL and the server, depending on the client's actions, will provide more options for the client application using hypermedia. For instance, if we send a GET request to the API entry point the server will answer with a 200 OK status code and a list of hyperlinks we can access from there. A following OPTIONS request on those URLs will give us more information about what we can do next. In previous example, an OPTIONS request to the query end-point will return the HTTP header "Allow: POST, OPTIONS", which is telling us that we can make a POST request to that URL.

NFFinder uses Django REST framework to provide a web browsable API so if you're already familiar with REST APIs you can begin to explore NFFinder's API in your web browser. If you find all this a bit confusing don't worry, in the following sections we will explain in detail how to interact with the API.

2. NFFinder's API entry point

NFFinder's API entry point can be found at: http://nffinder.cnb.csic.es/api/

A GET request to the entry point URL will return the following JSON:

{
    "databases": "http://nffinder.cnb.csic.es/api/databases/",
    "rnaquery": "http://nffinder.cnb.csic.es/api/rnaquery/",
    "mirnaquery": "http://nffinder.cnb.csic.es/api/mirnaquery/",
    "platforms": "http://nffinder.cnb.csic.es/api/platforms/",
    "experiments": "http://nffinder.cnb.csic.es/api/experiments/",
    "papers": "http://nffinder.cnb.csic.es/api/papers/",
    "authors": "http://nffinder.cnb.csic.es/api/authors/"
}

If we make an OPTIONS request to each one of these API end-points we find out that "databases", "platforms", "experiments", "authors" and "papers" admit the GET, HEAD and OPTIONS requests. These are API end-points to NFFinder's database and we won't be discussing them here. On the other hand, "rnaquery" and "mirnaquery" admit a POST request. We will be using these two entry points to create jobs that will mine our database in search for experiments with expression profiles similar to our profile of interest.

3. Creating jobs from RNA expression data

The end-point for querying NFFinder using two lists of genes (up-regulated and down-regulated) is returned by the API's entry point in the field "rnaquery". If we look at the OPTIONS request's response we will see the following JSON:

{
    "name": "Rna Query List", 
    "description": "Run new job from RNA data", 
    "renders": [
        "application/json", 
        "text/html"
    ], 
    "parses": [
        "application/json", 
        "application/x-www-form-urlencoded", 
        "multipart/form-data"
    ], 
    "actions": {
        "POST": {
            "jobid": {
                "type": "string", 
                "required": true, 
                "read_only": true, 
                "label": "jobid", 
                "max_length": 40
            }, 
            "url": {
                "type": "field", 
                "required": false, 
                "read_only": true
            }, 
            "up": {
                "type": "string", 
                "required": false, 
                "read_only": false, 
                "label": "up"
            }, 
            "down": {
                "type": "string", 
                "required": false, 
                "read_only": false, 
                "label": "down"
            }, 
            "inverse": {
                "type": "boolean", 
                "required": false, 
                "read_only": false, 
                "label": "inverse"
            }, 
            "pvalue": {
                "type": "float", 
                "required": true, 
                "read_only": false, 
                "label": "pvalue"
            }, 
            "databases": {
                "type": "field", 
                "required": true, 
                "read_only": false, 
                "label": "databases"
            }, 
            "email": {
                "type": "string", 
                "required": false, 
                "read_only": false
            }, 
            "title": {
                "type": "string", 
                "required": false, 
                "read_only": false
            }
        }
    }
}
			

The field "actions" describes the actions that can be requested and the parameters they take. For our POST request we are only interested in the parameters described by the field "POST" inside "actions". As we can see above, there are several fields with different properties in our POST parameters description. Some of them have the field "read_only" set to true, which means we won't have to set them in our POST request; instead, those fields will be returned by NFFinder when we make our request.

Among the remaining parameters there are five mandatory ("required") and two optional ones. First of all, the mandatory parameters:

  • "up": a space-separated list of up-regulated genes. For instance: "CDCA5 HIST1H1E HSPD1 IGFBP3 POSTN SPP1 TOP2A UCHL1".
  • "down": a space-separated list of down-regulated genes. For instance: "APOD CAPS DF GSN HNT IGSF4 NDRG2 PTGDS SEMA3B TNA".
  • "inverse": a boolean indicating if we want to search for profiles opposed to the input. For instance: true.
  • "pvalue": a real number indicating the minimum confidence level (p-value) required to consider a comparison valid. For instance: 0.005.
  • "databases": a list of databases to mine. Currently, there are three valid possibilities: "GEO", "CMap" and "DrugMatrix". In our example we will be using: ["CMap", "DrugMatrix"].

The two optional parameters (not "required") are:

  • "email": an e-mail address. If provided, NFFinder will send an e-mail to this address once the search is completed and the results are available.
  • "title": a short description to identify your job. Take into account that this parameter is aesthetic; it will be used when (and if) an e-mail is sent and can be used to set a title in the web interface that shows the results, but the job itself doesn't keep track of this title. If the client doesn't keep track of it you won't be able to recover it from your job.

As we mentioned in the introduction, in order to create a new job to process our search we need to make a POST request to the corresponding URL. To do so, we can use whichever method we prefer, any standard HTTP library in you favorite programming language will do. Even a command line tool like cURL can be used to write an interface for NFFinder in any shell scripting language. To illustrate our example we will be actually using cURL.

The first thing we must do is encoding our query in a JSON object. NFFinder also accepts other formats, like HTML from data format (see "parses" parameter in our OPTIONS request), but for the sake of convenience we will be using JSON. For our example request the JSON data will be the following:

{
    "up": "CDCA5 HIST1H1E HSPD1 IGFBP3 POSTN SPP1 TOP2A UCHL1",
    "down": "APOD CAPS DF GSN HNT IGSF4 NDRG2 PTGDS SEMA3B TNA",
    "inverse": true,
    "pvalue": 0.005,
    "databases": ["CMap", "DrugMatrix"]
}
			

Once we have got our JSON we can make the HTTP POST request (if you want to understand the details of this command, see cURL's man page):

			
user@host:~$ curl -X POST -H "Content-Type: application/json" -D - -d '{"up":"CDCA5 HIST1H1E HSPD1 IGFBP3 POSTN SPP1 TOP2A UCHL1","down":"APOD CAPS DF GSN HNT IGSF4 NDRG2 PTGDS SEMA3B TNA","inverse":true,"pvalue":0.005,"databases":["CMap", "DrugMatrix"]}' http://nffinder.cnb.csic.es/api/rnaquery/
			
			

This request will return the following headers:

HTTP/1.1 201 Created
Date: Wed, 17 Dec 2014 17:08:33 GMT
Server: Apache/2.2.22 (Debian)
Vary: Accept,Cookie
X-Frame-Options: SAMEORIGIN
Location: http://nffinder.cnb.csic.es/api/jobs/1964849594bf53a977daebf5a5d213a2f6045d9f/
Allow: POST, OPTIONS
Content-Type: application/json
Transfer-Encoding: chunked

			

The only things here that are important to us are the status code (HTTP/1.1 201 Created) which indicates that the job has been created correctly; and the Location header (Location:) which tells us where we can find our newly created job. The location of our job is essential to be capable of tracking its progress and retrieving the results. In our example we can find our job at:

         http://nffinder.cnb.csic.es/api/jobs/1964849594bf53a977daebf5a5d213a2f6045d9f/

We will be using this URL later on, while explaining how to track our job's progress.

We can also take a look at the JSON returned by our request:

{
    "jobid": "1964849594bf53a977daebf5a5d213a2f6045d9f", 
    "url": "http://nffinder.cnb.csic.es/api/jobs/1964849594bf53a977daebf5a5d213a2f6045d9f/", 
    "up": "CDCA5 HIST1H1E HSPD1 IGFBP3 POSTN SPP1 TOP2A UCHL1", 
    "down": "APOD CAPS DF GSN HNT IGSF4 NDRG2 PTGDS SEMA3B TNA", 
    "inverse": true, 
    "pvalue": 0.005, 
    "databases": [
        "DrugMatrix", 
        "CMap"
    ]
}
			

Now, the two "read_only" parameters, "jobid" and "url", are filled. The first one we can ignore but notice that the second one, "url", contains the location of our job. This parameters is included for convenience, you can use this instead of recording the Location header of the HTTP answer.

4. Creating jobs from microRNAs

The end-point for querying NFFinder using a list of microRNAs is returned by the API's entry point in the field "mirnaquery". If we look at the OPTIONS request's response we will see the following JSON:

{
    "name": "Mi Rna Query List", 
    "description": "Run new job from miRNA data", 
    "renders": [
        "application/json", 
        "text/html"
    ], 
    "parses": [
        "application/json", 
        "application/x-www-form-urlencoded", 
        "multipart/form-data"
    ], 
    "actions": {
        "POST": {
            "jobid": {
                "type": "string", 
                "required": true, 
                "read_only": true, 
                "label": "jobid", 
                "max_length": 40
            }, 
            "url": {
                "type": "field", 
                "required": false, 
                "read_only": true
            }, 
            "up": {
                "type": "string", 
                "required": false, 
                "read_only": true, 
                "label": "up"
            }, 
            "down": {
                "type": "string", 
                "required": false, 
                "read_only": true, 
                "label": "down"
            }, 
            "mirnas": {
                "type": "string", 
                "required": true, 
                "read_only": false
            }, 
            "inverse": {
                "type": "boolean", 
                "required": false, 
                "read_only": false, 
                "label": "inverse"
            }, 
            "pvalue": {
                "type": "float", 
                "required": true, 
                "read_only": false, 
                "label": "pvalue"
            }, 
            "databases": {
                "type": "field", 
                "required": true, 
                "read_only": false, 
                "label": "databases"
            }, 
            "email": {
                "type": "string", 
                "required": false, 
                "read_only": false
            }, 
            "title": {
                "type": "string", 
                "required": false, 
                "read_only": false
            }
        }
    }
}

Again, we can see an "actions" field describing the types of requests we can perform on this URL. Similarly to what happens with RNA-type searches, we are interested in the fields described by the "POST" action. Most fields are common to those of the RNA-type searches but there's an additional field, "mirnas", where we can send a space-separated list of microRNAs. Also notice that the lists of up-regulated and down-regulated genes have the "read_only" property set to true. This is because after translating the list of microRNAs to a set of up-regulated or down-regulated genes to perform the search, NFFinder will return the result of the translation in those two fields.

Everything else works as described in Section 3.

5. Keeping track of your job's progress

Once our job's been created and NFFinder is performing the search, we can use the URL returned by our POST query to track the progress and access the results once they're available. In order to do that, a simple GET request to the job's URL will return the following JSON:

{
    "date": "2014-12-17T17:15:00.926Z", 
    "up": "CDCA5 HIST1H1E HSPD1 IGFBP3 POSTN SPP1 TOP2A UCHL1", 
    "down": "APOD CAPS DF GSN HNT IGSF4 NDRG2 PTGDS SEMA3B TNA", 
    "inverse": true, 
    "pvalue": 0.005, 
    "databases": [
        "http://nffinder.cnb.csic.es/api/databases/DrugMatrix/", 
        "http://nffinder.cnb.csic.es/api/databases/CMap/"
    ], 
    "status": "http://nffinder.cnb.csic.es/api/jobs/1964849594bf53a977daebf5a5d213a2f6045d9f/status/", 
    "results": "http://nffinder.cnb.csic.es/api/jobs/1964849594bf53a977daebf5a5d213a2f6045d9f/results/", 
    "experts": "http://nffinder.cnb.csic.es/api/jobs/1964849594bf53a977daebf5a5d213a2f6045d9f/experts/", 
    "drugs": "http://nffinder.cnb.csic.es/api/jobs/1964849594bf53a977daebf5a5d213a2f6045d9f/drugs/", 
    "diseases": "http://nffinder.cnb.csic.es/api/jobs/1964849594bf53a977daebf5a5d213a2f6045d9f/diseases/", 
    "experimentscsv": "http://nffinder.cnb.csic.es/api/jobs/1964849594bf53a977daebf5a5d213a2f6045d9f/experimentscsv/", 
    "drugscsv": "http://nffinder.cnb.csic.es/api/jobs/1964849594bf53a977daebf5a5d213a2f6045d9f/drugscsv/", 
    "diseasescsv": "http://nffinder.cnb.csic.es/api/jobs/1964849594bf53a977daebf5a5d213a2f6045d9f/diseasescsv/", 
    "experimentupgenescsv": "http://nffinder.cnb.csic.es/api/jobs/1964849594bf53a977daebf5a5d213a2f6045d9f/experimentupgenescsv/", 
    "experimentdowngenescsv": "http://nffinder.cnb.csic.es/api/jobs/1964849594bf53a977daebf5a5d213a2f6045d9f/experimentdowngenescsv/", 
    "expertscsv": "http://nffinder.cnb.csic.es/api/jobs/1964849594bf53a977daebf5a5d213a2f6045d9f/expertscsv/", 
    "paperscsv": "http://nffinder.cnb.csic.es/api/jobs/1964849594bf53a977daebf5a5d213a2f6045d9f/paperscsv/", 
    "expertpaperscsv": "http://nffinder.cnb.csic.es/api/jobs/1964849594bf53a977daebf5a5d213a2f6045d9f/expertpaperscsv/", 
    "paperauthorscsv": "http://nffinder.cnb.csic.es/api/jobs/1964849594bf53a977daebf5a5d213a2f6045d9f/paperauthorscsv/"
}
			

This JSON contains all the information of our job (up- and down-regulated gene lists, minimum p-value, etc) as well as fields pointing to the URL we will use to retrieve our results. Among all these fields there's one specific field, "status", that points us to the URL where we can track the progress of our job.

If we perform a GET request on the URL of a recently created job, we will get a JSON similar to the following one:

{
    "progress": 0.0, 
    "running": false, 
    "done": false, 
    "success": false, 
    "status": "Queued..."
}
			

The field "progress" tells us the percentage of the job that's been completed, and "status" gives us a short description of what's currently doing our job. Since we just created this job, our "progress" is at 0.0% and we can see that the job's status is "Queued..." (i.e.: it's waiting for an open slot in our computing cluster).

The other three fields, "running", "done" and "success", give us information about the execution status of our job. "running" tells us whether our job is currently running in the cluster or not, "done" indicates if the job execution began and ended at some point and "success" marks a job as successfully finished or finished with errors.

The following table indicates all the possible values of these three flags and their meaning:

"running" "done" "success" What it means
false false false The job is either stalled or waiting to run.
true false false The job is currently running in a node.
false true false The job finished running but an error prevented it from completing the work.
false true true The job finished successfully.

Once our job has finished, we can proceed to retrieve our results.

6. Retrieving your results

If we take a look at the JSON with the job information we can find four fields with URLs to the results:

  • "results": a list of experiments in our database that match our query, including their scores and p-values.
  • "experts": a list of experts related to the experiments found with their corresponding scores and the list of their related papers.
  • "drugs": the same list of experiments contained in "results", but grouped by drugs. This means that if a given experiment is related to more than one drug it will appear more than once in this list; once for each related drug.
  • "diseases": the same list of experiments contained in "results", but grouped by diseases and conditions. This means that if a given experiment is related to more than one disease or condition it will appear more than once in this list; once for each related disease or condition.

NFFinder returns these results paginated. This means that if we make a GET request on the URL, the JSON returned by the API will contain, beside a list with a few results, a "count" field with the total number of results and two fields, "next" and "prev", with URLs to the next and the previous results page.

Here is the results JSON returned by our job once it's finished:

{
    "count": 215, 
    "next": "http://nffinder.cnb.csic.es/api/jobs/1964849594bf53a977daebf5a5d213a2f6045d9f/results/?page=2", 
    "previous": null, 
    "results": [
        {
            "score": 87.0090847000595, 
            "pvalue": 0.000999000999000999, 
            "experiment": {
                "dbid": "Instance ID: 4730", 
                "name": "4730", 
                "title": "MCF7 cell line: levonorgestrel at 1.28e-05 M", 
                "description": "MCF7 cell line treated with levonorgestrel (Norgestrel-(-)-D [797-63-7]) at 1.28e-05 M concentration in DMSO for 6 hours.", 
                "condition": "MCF7", 
                "test": "levonorgestrel", 
                "control": "control", 
                "url": "https://www.broadinstitute.org/cmap/", 
                "drug": "Levonorgestrel", 
                "disease": ""
            }
        }, 
        {
            "score": 86.6004854961692, 
            "pvalue": 0.000999000999000999, 
            "experiment": {
                "dbid": "Instance ID: 6264", 
                "name": "6264", 
                "title": "MCF7 cell line: tropine at 2.84e-05 M", 
                "description": "MCF7 cell line treated with tropine (Tropine [120-29-6]) at 2.84e-05 M concentration in DMSO for 6 hours.", 
                "condition": "MCF7", 
                "test": "tropine", 
                "control": "control", 
                "url": "https://www.broadinstitute.org/cmap/", 
                "drug": "Tropine", 
                "disease": ""
            }
        }, 
        {
            "score": 85.8652997167356, 
            "pvalue": 0.000999000999000999, 
            "experiment": {
                "dbid": "Instance ID: 3219", 
                "name": "3219", 
                "title": "MCF7 cell line: clorgiline at 1.3e-05 M", 
                "description": "MCF7 cell line treated with clorgiline (Clorgyline hydrochloride [17780-75-5]) at 1.3e-05 M concentration in DMSO for 6 hours.", 
                "condition": "MCF7", 
                "test": "clorgiline", 
                "control": "control", 
                "url": "https://www.broadinstitute.org/cmap/", 
                "drug": "Clorgiline", 
                "disease": ""
            }
        }, 
        {
            "score": 85.5980804861718, 
            "pvalue": 0.000999000999000999, 
            "experiment": {
                "dbid": "Instance ID: 7471", 
                "name": "7471", 
                "title": "MCF7 cell line: noretynodrel at 1.34e-05 M", 
                "description": "MCF7 cell line treated with noretynodrel (Norethynodrel [68-23-5]) at 1.34e-05 M concentration in DMSO for 6 hours.", 
                "condition": "MCF7", 
                "test": "noretynodrel", 
                "control": "control", 
                "url": "https://www.broadinstitute.org/cmap/", 
                "drug": "Noretynodrel", 
                "disease": ""
            }
        }, 
        {
            "score": 85.0534210419622, 
            "pvalue": 0.000999000999000999, 
            "experiment": {
                "dbid": "Instance ID: 7546", 
                "name": "7546", 
                "title": "PC3 cell line: 16-phenyltetranorprostaglandin E2 at 1e-05 M", 
                "description": "PC3 cell line treated with 16-phenyltetranorprostaglandin E2 (16-phenyl tetranor Prostaglandin E2) at 1e-05 M concentration in DMSO for 6 hours.", 
                "condition": "PC3", 
                "test": "16-phenyltetranorprostaglandin E2", 
                "control": "control", 
                "url": "https://www.broadinstitute.org/cmap/", 
                "drug": "16-phenyltetranorprostaglandin e2", 
                "disease": ""
            }
        }, 
        {
            "score": 84.1748451109737, 
            "pvalue": 0.000999000999000999, 
            "experiment": {
                "dbid": "Instance ID: 5702", 
                "name": "5702", 
                "title": "MCF7 cell line: vanoxerine at 7.6e-06 M", 
                "description": "MCF7 cell line treated with vanoxerine (GBR 12909 dihydrochloride [67469-78-7]) at 7.6e-06 M concentration in DMSO for 6 hours.", 
                "condition": "MCF7", 
                "test": "vanoxerine", 
                "control": "control", 
                "url": "https://www.broadinstitute.org/cmap/", 
                "drug": "Vanoxerine", 
                "disease": ""
            }
        }, 
        {
            "score": 83.9415488058177, 
            "pvalue": 0.000999000999000999, 
            "experiment": {
                "dbid": "Instance ID: 595", 
                "name": "595", 
                "title": "MCF7 cell line: resveratrol at 5e-05 M", 
                "description": "MCF7 cell line treated with resveratrol (resveratrol) at 5e-05 M concentration in DMSO for 6 hours.", 
                "condition": "MCF7", 
                "test": "resveratrol", 
                "control": "control", 
                "url": "https://www.broadinstitute.org/cmap/", 
                "drug": "Resveratrol", 
                "disease": ""
            }
        }, 
        {
            "score": 83.5551219943012, 
            "pvalue": 0.000999000999000999, 
            "experiment": {
                "dbid": "Instance ID: 2566", 
                "name": "2566", 
                "title": "HL60 cell line: trichostatin A at 1e-07 M", 
                "description": "HL60 cell line treated with trichostatin A (\"Trichostatin A, from Streptomyces sp.\") at 1e-07 M concentration in DMSO for 6 hours.", 
                "condition": "HL60", 
                "test": "trichostatin A", 
                "control": "control", 
                "url": "https://www.broadinstitute.org/cmap/", 
                "drug": "Trichostatin a", 
                "disease": ""
            }
        }, 
        {
            "score": 83.4992951463128, 
            "pvalue": 0.000999000999000999, 
            "experiment": {
                "dbid": "Instance ID: 3377", 
                "name": "3377", 
                "title": "MCF7 cell line: equilin at 1.5e-05 M", 
                "description": "MCF7 cell line treated with equilin (Equilin [474-86-2]) at 1.5e-05 M concentration in DMSO for 6 hours.", 
                "condition": "MCF7", 
                "test": "equilin", 
                "control": "control", 
                "url": "https://www.broadinstitute.org/cmap/", 
                "drug": "Equilin", 
                "disease": ""
            }
        }, 
        {
            "score": 83.204771814777, 
            "pvalue": 0.000999000999000999, 
            "experiment": {
                "dbid": "Instance ID: 5703", 
                "name": "5703", 
                "title": "MCF7 cell line: guanabenz at 1.38e-05 M", 
                "description": "MCF7 cell line treated with guanabenz (Guanabenz acetate [23256-50-0]) at 1.38e-05 M concentration in DMSO for 6 hours.", 
                "condition": "MCF7", 
                "test": "guanabenz", 
                "control": "control", 
                "url": "https://www.broadinstitute.org/cmap/", 
                "drug": "Guanabenz", 
                "disease": ""
            }
        }
    ]
}

Beside the pagination fields, there's a "results" field that consists of a list of scored experiments. The "drug" and "disease" fields inside an experiment are semicolon-separated lists of drugs and diseases/conditions related to the experiment. The "dbid" is the experiment identifier inside the database from which it originates (for instance, a GDS if it is a GEO experiment). The remaining fields are self-explanatory.

These four fields are not the only fields containing the query results. There are nine special fields that link to CSV downloadable files with the results and their relationships:

  • "experimentscsv": A table with all the experiments found.
  • "drugscsv": A relational table linking drugs to the different experiments found.
  • "diseasescsv": A relational table linking diseases and conditions to the different experiments found.
  • "experimentupgenescsv": A relational table linking every experiment found with the up-regulated query genes and their differential expression values for that experiment.
  • "experimentdowngenescsv": A relational table linking every experiment found with the down-regulated query genes and their differential expression values for that experiment.
  • "expertscsv": A table containing the experts found and their scores.
  • "paperscsv": A table containing the papers related to the experiments found.
  • "expertpaperscsv": A relational table linking experts with the papers they appear on.
  • "paperauthorscsv": A relational table linking every paper with all its authors.

NFFinder uses these tables, for convenience, in order to build the dashboard. If you find them useful for your purposes, please feel free to use them.