Articles from Demo

  • Say hello to RegTAP

    image: WIRR in the browser

    GAVO's WIRR registry interface in action to find resources with radio parallaxes.

    RegTAP is one of those standards that a scientist will normally not see – it works in the background and makes, for instance, TOPCAT display the Cone Search services matching some key words. And it's behind the services like WIRR, our Web Interface to the Relational Registry (“Relational Registry” being the official name for RegTAP) that lets you do some interesting data discovery beyond what current clients support. In the screenshot above, for instance (try it yourself), I'm looking for cone search services having parallaxes presumably from radio observations. You could now transmit the services you've found to, say, TOPCAT or your own pyvo-based program to start querying them.

    The key point this query is the use of UCDs – these let services declare fairly unambiguously what kind of physics (if you take that word with a grain of salt) they are talking about. In the example, pos.parallax means, well, a parallax, and the percent character is a wildcard (coming not from UCDs, but from ADQL). That wildcard is a good idea here because without it we might miss things like pos.parallax;obs and pos.parallax;stat.fit that people might have used to distinguish “raw” and ”processed” estimates.

    UCDs are great for data discovery. Really.

    Sometimes, however, clicking around in menus just isn't good enough. That's when you want the full power of RegTAP and write your very own queries. The good news: If you know ADQL (and you should!), you're halfway there already.

    Here's one example of direct RegTAP use I came up with the other day. The use case was discovering data collections that give the effective temperatures of components of binary star systems.

    If you check the UCD list, that “physics” translates into data that has columns with UCDs of phys.temperature and meta.code.multip at the same time. To translate that into a RegTAP query, have a look at the tables that make up a RegTAP service: its ”schema”. Section 8 of the standard lists all the tables there are, and there's an ADASS poster that has an image of the schema with the more common columns illustrated. Oh, and if you're new to RegTAP, you're probably better off briefly studying the examples first to get a feeling for how RegTAP is supposed to work.

    You will find that a pair of ivoid – the VO's global resource identifier – and a per-resource table index uniquely identify a table within the entire registry. So, an ADQL query to pick out all tables containing temperatures and component identifiers would look like this:

    SELECT DISTINCT ivoid, table_index
    FROM
    rr.table_column AS t1
    JOIN rr.table_column AS t2
    USING (ivoid, table_index)
    WHERE t1.ucd='phys.temperature'
    AND t2.ucd='meta.code.multip'
    

    – the DISTINCT makes it so even tables that have lots of temperatures or codes only turn up once in our result set, and the somewhat odd self-join of the rr.table_column table with itself lets us say “make sure the two columns are actually in the same table”. Note that you could catch multi-table resources that define the components in one table and the temperatures in another by just joining on ivoid rather than ivoid and table_index.

    You can run this query on any RegTAP endpoint: GAVO operates a small network of mirrors behind http://reg.g-vo.org/tap, there's the ESAC one at http://registry.euro-vo.org/regtap/tap, and STScI runs one at http://vao.stsci.edu/RegTAP/TapService.aspx. Just use your usual TAP client.

    But granted, the result isn't terribly user-friendly: just identifiers and number. We'd at least like to see the names and descriptions of the tables so we know if the data is somehow relevant.

    RegTAP is designed so you can locate the columns you would like to retrieve or constrain and then just NATURAL JOIN everything together. The table_description and table_name columns are in rr.res_table, so all it takes to see them is to take the query above and join its result like this:

    SELECT table_name, table_description
    FROM rr.res_table
    NATURAL JOIN (
      SELECT DISTINCT ivoid, table_index
      FROM
      rr.table_column AS t1
      JOIN rr.table_column AS t2
      USING (ivoid, table_index)
      WHERE t1.ucd='phys.temperature'
      AND t2.ucd='meta.code.multip') as q
    

    If you try this, you'll see that we'd like to get the descriptions of the resources embedding the tables, too in order to get an idea what we can expect from a given data collection. And if we later want to find services exposing the tables (WIRR is nice for that – try the ivoid constraint –, but for this example all resources currently come from VizieR, so you can directly use VizieR's TAP service to interact with the tables), you want the ivoids. Easy: Just join rr.resource and pick columns from there:

    SELECT table_name, table_description, res_description, ivoid
    FROM rr.res_table
    NATURAL JOIN rr.resource
    NATURAL JOIN (
      SELECT DISTINCT ivoid, table_index
      FROM
      rr.table_column AS t1
      JOIN rr.table_column AS t2
      USING (ivoid, table_index)
      WHERE t1.ucd='phys.temperature'
      AND t2.ucd='meta.code.multip') as q
    

    If you've made it this far and know a bit of ADQL, you probably have all it really takes to solve really challenging data discovery problems – as far as Registry metadata reaches, that is, which currently does not include space-time coverage. But stay tuned, more on this soon.

    In case you're looking for a more systematic introduction into the world of the Registry and RegTAP, there are two... ouch. Can I really link to Elsevier papers? Well, here goes: 2014A&C.....7..101D (a.k.a. arXiv:1502.01186 on the Registry as such and 2015A%26C....11...91D (a.k.a. arXiv:1407.3083) mainly on RegTAP.

  • ADQL tricks at MPIA

    Aerial image of Heidelberg and Königstuhl

    The 2017-06-29 ADQL talk (red circle) from 30000 ft

    Today I was up on Heidelberg's signature mountain, Königstuhl, at the Max-Planck-Institute for Astronomy for a little talk on what I'd provisionally call “intermediate ADQL” – discussing some aspects of ADQL and some TAP techniques that may not be immediately obvious but still generally and straightforwardly applicable to everyday problems. Since I suspect the lecture notes for that talk may be of interest to some readers of this blog, I thought I should share them here.

    What this also contains is a very quick piece of pyVO-based python (which needs both this helper and a recent pyVO) for a use case that comes up fairly often: “Give me all proper motions (radio fluxes, distances, radial velocities, whatever) for object in this region.”

    This uses a discovery case I've been after for quite a while now: Find services by the UCDs of tables within them. And while that's been possible for quite a while on GAVO's Registry UI WIRR, there's still too many services that don't declare their tables to the Registry, and when talking about TAP, the situation is still a bit worse (as has been mentioned in my account of the last interop). So – enjoy the code, but very frankly, you'll still see wires sticking out for a several months yet.

    And if you run a TAP service yourself, please have a look at how to enable table discovery over on the IVOA wiki so we can finally get those pesky wires out of our users' eyes.

  • Updated Proper Motion Tutorial

    At the risk of turning this into a blog on nice TAP tricks (which it's not supposed to be): Our classic short tutorial on adding proper motions to almost arbitrary object lists has just gotten a facelift today.

    And there's new content, too – I now show what to do when you don't even have positions but just object names. In order to keep this sufficiently geeky, here's the query as a spoiler:

    SELECT col1, ra, dec
    FROM TAP_UPLOAD.t1
    LEFT OUTER JOIN ident
    ON (id=normId(col1))
    LEFT OUTER JOIN basic
    ON (oidref=oid)
    

    But to close on a non-TAP topic: Registry! There's an experimental facility to have this kind of thing in the Registry; the PM tutorial is in, for instance, with the ivoid ivo://edu.gavo.org/hd/gavo_addpms). One thing you can do with this is generate a list of registred documents that essentially updates itself from the registry.

    Another is figure out where the source code of the document is (if the authors choose to share it, which is of course a very smart thing to do); in our example it's in Volute, the IVOA's semi-official version control system. So, if you find a bug (defined as “superset of typo”) in the linked document, you're most welcome to supply patches as diffs or just directly fix things if you have commit privileges in Volute.

  • Automating TAP queries

    TOPCAT is a great tool – in particular, for prototyping and ad-hoc analyses, it's hard to beat. But say you've come up with this great TAP query, and you want to preserve it, perhaps generate plots, then maybe fix them when you (or referees) have spotted problems.

    Then it's time to get acquainted with TOPCAT's command line sister STILTS. This tool lets you do most of what TOPCAT can, but without user intervention. Since the command lines usually take a bit of twiddling I usually wrap stilts calls with GNU make, so I just need to run something like make fig3.pdf or make fig3.png and make figures out what to do, potentially starting with a query. Call it workflow preservation if you want.

    How does it work? Well, of course with a makefile. In that, you'll first want to define some variables that allow more concise rules later. Here's how I usually start one:

    STILTS?=stilts
    
    # VOTables are the results of remote queries, so don't wantonly throw
    # them away
    .PRECIOUS: %.vot
    
    # in this particular application, it helps to have this global
    HEALPIX_ORDER=6
    
    # A macro that contains common stuff for stilts TAP query -- essentially,
    # just add adql=
    TAPQUERY=$(STILTS) tapquery \
      tapurl='http://dc.g-vo.org/tap' \
      maxrec=200000000 \
      omode=out \
      out=$@ \
      ofmt=vot \
      executionduration=14400
    
    # A sample plot macro.  Here, we do a healpix plot of some order. Also
    # add value_1=<column to plot>
    HEALPIXPLOT=$(STILTS) plot2sky \
      auxmap=inferno \
      auxlabel='$*'\
      auxvisible=true \
      legend=false \
      omode=out \
      out=$@ \
      projection=aitoff \
      sex=false \
      layer_1=healpix \
        datalevel_1=$(HEALPIX_ORDER) \
        datasys_1=equatorial \
        viewsys_1=equatorial \
        degrade_1=0 \
        transparency_1=0 \
        healpix_1=hpx \
        in_1=$< \
        ifmt_1=votable \
        istream_1=true \
    

    For the somewhat scary STILS command lines, see the STILTS documentation or just use your intution (which mostly should give you the right idea what something is for).

    The next step is to define some pattern rules; these are a (in the syntax here, GNU) make feature that let you say „to make a file matching the destination pattern when you have one with the source pattern, use the following commands”. You can use a number of variables in the rules, in particular $@ (the current target) and $< (the first prerequisite). These are already used in the definitions of TAPQUERY and HEALPIXPLOT above, so they don't necessarily turn up here:

    # healpix plots from VOTables; these will plot obs
    %.png: %.vot
          $(HEALPIXPLOT) \
                  value_1=obs \
                  ofmt=png \
                  xpix=600 ypix=380
    
     %.pdf: %.vot
           $(HEALPIXPLOT) \
                   value_1=obs \
                   ofmt=pdf
    
     # turn SQL statements into VOTables using TAP
     %.vot: %.sql
           $(TAPQUERY) \
                  adql="`cat $<`"
    

    Careful with cut and paste: The leading whitespace here must be a Tab in rules, not just some blanks (this is probably the single most annoying feature of make. You'll get used to it.)

    What can you do with it? Well, for instance you can write an ADQL query into a file density.sql; say:

    SELECT
      count(*) as obs,
      -- "6" in the next line must match HEALPIX_ORDER in the Makefile
      ivo_healpix_index (6, alphaFloat, deltaFloat) AS hpx
    FROM ppmx.data
    GROUP BY hpx
    

    And with this, you can say:

    make density.pdf
    

    and get a nice PDF with the plot resulting from that query. Had you just said make density.vot, make would just have executed the query and left the VOTable, e.g., for investigation with TOPCAT, and if you were to type make density.png, you'd get a nice PNG without querying the service again. Like this:

    A density plot

    Unless of course you changed the SQL in the meantime, in which case make would figure out it had to go back to the service.

    In particular for the plots you'll often have to override the defaults. Make is smart enough to figure this out. For instance, you could have two files like this:

    $ cat pm_histogram.sql
    SELECT
      round(pmtot/10)*10 as bin, count(*) as n
    FROM (
      SELECT sqrt(pmra*pmra+pmde*pmde)*3.6e6 as pmtot
      FROM hsoy.main) AS q
    group by bin
    $ cat pm_histogram_cleaned.vot
    SELECT
      round(pmtot/10)*10 as bin,
      count(*) as n
      FROM (
        SELECT sqrt(pmra*pmra+pmde*pmde)*3.6e6 as pmtot
        FROM hsoy.main
        WHERE no_sc IS NULL) AS q
      group by bin
    

    (these were used to analyse the overall proper motions distributions in HSOY properties; note that each of these will run about 30 minutes or so, so better adapt them to what's actually interesting to you before trying this).

    No special handling in terms of queries is necessary for these, but the plot needs to be hand-crafted:

    pm_histograms.png: pm_histogram.vot pm_histogram_cleaned.vot
         $(STILTS) plot2plane legend=false \
                 omode=out ofmt=png out=$@ \
                  title="All-sky" \
                  xpix=800 ypix=600 \
                  ylog=True xlog=True\
                  xlabel="PM bin [mas/yr, bin size=10]" \
                  xmax=4000 \
                  layer1=mark \
                         color1=blue \
                         in1=pm_histogram.vot \
                         x1=bin \
                         y1=n \
                 layer2=mark \
                         in2=pm_histogram_cleaned.vot \
                         x2=bin \
                         y2=n
    

    – that way, even if you go back to the stuff six months later, you can still figure out what you queried (the files are still there) and what you did then.

    A makefile to play with (and safe from cut-and-paste problems) is available from Makefile_tapsample (rename to Makefile to reproduce the examples).

  • DaCHS, SODA, and Datalink

    DaCHS, the Data Center Helper Suite, is a comprehensive suite for publishing astronomical data to the Virtual Observatory, supporting most major protocols out there. On Dec 12, GAVO released a new version, 0.9.8. The most notable change is that now SODA is supported as specified in the last IVOA Proposed Recommendation.

    This is fairly big news, as SODA is the VO's answer to providing cutout services and the like, which obviously is important part with datasets in the Multi-Gigabyte range and the VO's wider programme of trying to enable users to only download what they need. But even for spectra, which aren't typically terribly large, we have been using SODA; for instance, when you just want to see the development of a single line over time, say,, it's nice to not have to bother with the the full spectrum. The spectral client SPLAT has been offering such functionality for a couple of years now -- watch out for the scissors icon in discovery results. These indicate SODA support on the respective services.

    Another client that will support SODA and its basis Datalink is Aladin – we've seen a promising demo of that during the last Interop in Trieste. Until the clients are there, DaCHS contains a (largely re-usable) stylesheet that generates simple UIs for Datalink documents and SODA services. Some examples:

    Note again that all of these are not actually web pages, they're machine-readable metadata collections; if you don't believe it, pull the URLs with curl. To learn more about the combo of Datalink and SODA, check out this ADASS 2015 poster (preferably before even looking at the not terribly readable standards texts).

    If you're running DaCHS yourself and can't wait to run Datalink and SODA -- here's how to do that.

« Page 2 / 2