Tutorial 1 - basic use of endpoints

** This notebook covers the basics of working with voeventdb.remote, demonstrating simple endpoint calls and the data-structures they return. **

(It also includes a few tips on using IPython’s autocomplete functionality, for newcomers to IPython / Jupyter Notebooks.)

First, let’s set up logging - by switching on ‘DEBUG’ level logging, we get to see the HTTP requests that are being made behind the scenes every time you run a query using voeventdb.remote. (NB we’ll be using the Python-3 style print operator.)

from __future__ import print_function
import logging
logging.basicConfig(level=logging.DEBUG)

Next, library imports: Most of the user-facing code lives in voeventdb.remote.apiv1, but we’ll also alias voeventdb.remote to the shorthand vr:

import voeventdb.remote as vr
import voeventdb.remote.apiv1 as apiv1

### Aside: Autocomple tion tricks For those new to IPython / notebooks: note you can use tab-comple tion to help you explore / save on typing (as long as you’ve already imported the relevant packages). For example, type:

apiv1.

in a notebook code-cell or IPython terminal and then hit Tab to see your available options. Likewise, in notebooks you can position your cursor on a function and hit Shift+Ta b to see a summary of function arguments, etc.

There are a couple of variables living in the top-level voeventdb.remote module; for example the default_host. You can set this to avoid specifying a host for every query. By default it’s set to point at the 4PiSky server:

# For testing against a local-dev server:
# vr.default_host = 'http://localhost:5000'
print(vr.default_host)

OK, let’s query the database. We use different HTTP addresses, called endpoints, to give us different summaries or subsets of the database. We’ll start off by using apiv1.count to find out how many packets are stored in this database - if it’s empty, any other queries won’t do us much good:

apiv1.count()

Getting on for a million packets, that’s quite lot! So there’s plenty here for us to work with.

IVORN’s and single-packet details

Next let’s try apiv1.list_ivorn, which returns a list of IVORN’s, unique idenitifiers which represent a VOEvent packet, analogous to how a URL represents a web-page.

ivorn_list = apiv1.list_ivorn()
len(ivorn_list)

#Take a look at the first 10
ivorn_list[:10]

We got back a list of IVORN’s as expected - but only 5000 of them, not a million. By default, voeventdb.remote applies a limit to the maximum number of entries returned by a list query, as defined in default_list_n_max:

print(vr.default_list_n_max)

You can override this, or set it to 0 to simply fetch all matching results (though this can take a while).

## How to override the default:
# vr.default_list_n_max = 2500
## Set n_max for a single query:
short_list = apiv1.list_ivorn(n_max=50)

If we know the IVORN of a packet, we can look up some details using the apiv1.packet_synopsis endpoint:

ivorn = ivorn_list[0]
apiv1.packet_synopsis(ivorn)

And we can even retrieve the raw XML, (i.e. the original packet), so we can pull out any data specific to a given observatory, etc:

xml = apiv1.packet_xml(ivorn)

A convenient way to inspect the xml-data is to use the voevent-parse library (which even has its very own tutorial).

## A brief example, see the voevent-parse tutorial for more
# import voeventparse as vp
# voe_tree = vp.loads(xml)
# vp.pull_params(voe_tree)

Summary endpoints

So we can view individual packet IVORNS, and look up packet details, but that’s very fine-grained information - we don’t want to wade through all of them to figure out what’s in the database. We can use ‘summary’ endpoints to give us a higher-level overview. For example, we can organise VOEvents by ‘streams’ - basically the bit of the IVORN between the ivo:// prefix and the # symbol. The apiv1.map_stream_count endpoint tells us how many VOEvent packets belong to each stream:

apiv1.map_stream_count()

Alternatively, we can slice up the packets by the authoring timestamp, binning by calendar month:

apiv1.map_authored_month_count()

Or, we can divide them up by assigned role, which comes in three flavours:

apiv1.map_role_count()

Similarly, apiv1.map_stream_role_count gives a breakdown of how many packets belong to each role, listed by stream.

Coming next …

So now that we have an idea of what’s in the database, how do we choose a category and drill-down to select specific VOEvents? For that we need filters, which are covered in tutorial 2.