Posts with the Tag DaCHS:

• DaCHS 2.1: Say hello to Python 3

  2020-05-29 Markus Demleitner

  

  Today, I have released DaCHS 2.1, the first stable DaCHS running on Python 3. I have tried hard to make the major version move painless and easy, and indeed “pure DaCHS” RDs should just continue to work. But wherever there's Python in your RDs or near them, things may break, since Python 3 is different from Python 2 in some rather fundamental ways.

  Hence, the Debian package even has a new name: gavodachs2-server. Unless you install that, things will keep running as they do. I will keep fixing serious DaCHS 1 bugs for a while, so there's no immediate urgency to migrate. But unless you migrate, you will not see any new features, so one of these days you will have to migrate anyway. Why not do it today?

  Migrating to DaCHS 2

  In principle, just say apt install gavodachs2-server and hope for the best. If you have a development machine and regression tests defined, this is actually what we recommend, and we'd be very grateful to learn of any problems you may encounter.

  If you'd rather be a little more careful, Carlos Henrique Brandt has kindly updated his Docker files in order to let you spot problems before you mess up your production server. See Test Migration for a quick intro on how to do that. If you spot any problems that are not related to the Python 3 pitfalls mentioned in the howto linked below or nevow exodus, please tell me or (preferably) the dachs-support mailing list.

  A longer, more or less permanent piece elaborating possible migration pains, is in our how-to documentation: How do I go from DaCHS1 to DaCHS2?

  What's new in DaCHS2?

  I've used the opportunity of the major version change to remove a few (mis-) features that I'm rather sure nobody uses; and there are a few new features, too. Here's a rundown of the more notable changes:

  • DaCHS now produces VOTable 1.4 by default. This is particularly notable when you provide TIMESYS metadata (on which I'll report some other time).
  • When doing spatial indices, prefer the new //scs#pgs-pos-index to //scs#q3cindex. While q3c is still faster and more compact than pgsphere when just indexing points, on the longer run I'd like to shed the extra dependency (note, however, that the pgsphere index limits the cone search to a maximum radius of 90 degrees at this point).
  • Talking about Cone Search: For custom parameters, DaCHS has so far used SSA-like syntax, so you could say, for instance, vmag=12/13 (for “give me rows where vmag is between 12 and 13”). Since I don't think this was widely used, I've taken the liberty to migrate to DALI-compliant syntax, where intervals are written as they would be in VOTable PARAM values: vmag=12 13.
  • In certain situations, DaCHS tries to enable parallel queries (previously on this blog).
  • Some new ADQL user defined functions: gavo_random_normal, gavo_mocintersect, and gavo_mocunion. See the TAP capabilities for details, and note that the moc functions will fail until we put out a new pgsphere package that has support for the MOC-MOC operations.
  • dachs info (highly recommended after an import) now takes a --sample-percent option that helps when doing statistics on large tables.
  • For SSA services serving something other than spectra (in all likelihood, timeseries), you can now set a productType meta as per the upcoming SimpleDALRegExt 1.2.
  • If you have large, obscore-published SIAP tables, re-index them (dachs imp -I q) so queries over s_ra and s_dec get index support, too.
  • Since we now maintain RD state in the database, you can remove the files /var/gavo/state/updated* after upgrading.
  • When writing datalink metaMakers returning links, you can (and should, for new RDs) define the semantics in an attribute to the element rather in the LinkDef constructor.
  • Starting with this version, it's a good idea to run dachs limits after an import. This, right now, will mainly set an estimate for the number of rows in a table, but that's already relevant because the ADQL translator uses it to help the postgres query planner. It will later also update various kinds of column metadata that, or so I hope, will become relevant in VODataService 1.3.
  • forceUnique on table elements is now a no-op (and should be removed); just define a dupePolicy as before.
  • If you write bad obscore mappings, it could so far be hard to figure out the reason of the failure and, between lots of confusing error messages, to fix it. Instead, you can now run ``dachs imp //obscore recover`` in such a situation. It will re-create the obscore table and throw out all stanzas that fail; after that, you can fix the obscore declarations that were thrown out one by one.
  • If you run DaCHS behind a reverse proxy that terminates https, you can now set [web]adaptProtocol in /etc/gavo.rc to False. This will make that setup work for form-based services, too.
  • If you have custom OAI set name (i.e., anything but local and ivo_managed in the sets attribute of publish elements), you now have to declare them in [ivoa]validOAISets.
  • Removed things: the docform renderer (use form instead), the soap renderer (well, it's not actually removed, it's just that the code it depends on doesn't exist on python3 any more), sortKey on services (use the defaultSortKey property), //scs#q3cpositions (port the table to have ra and dec and one of the SCS index mixins), the (m)img.jpeg renderers (if you were devious enough to use these, let me know), and quite a few even more exotic things.

  Some Breaking Changes

  Python 3 was released in 2008, not long after DaCHS' inception, but since quite a few of the libraries it uses to do its job haven't been available for Python 3, we have been reluctant to make the jump over the past then years (and actually, the stability of the python2 platform was a very welcome thing).

  Indeed, the most critical of our dependencies, twisted, only became properly usable with python3 in, roughly, 2017. Indeed, large parts of DaCHS weren't even using twisted directly, but rather a nice add-on to it called nevow. Significant parts of nevow bled through to DaCHS operators; for instance, the render functions or the entire HTML templating.

  Nevow, unfortunately, fell out of fashion, and so nobody stepped forward to port it. And when I started porting it myself I realised that I'm mainly using the relatively harmless parts of nevow, and hence after a while I figured that I could replace the entire dependency by something like a 1000 lines in DaCHS, which, given significant aches when porting the whole of nevow, seemed like a good deal.

  The net effect is that if you built code on top of nevow – most likely in the form of a custom renderer – that will break now, and porting will probably be rather involved (having ported ~5 custom renderers, I think I can tell). If this concerns you, have a look at the README in gavo.formal (and then complain because it's mainly notes to myself at this point). I feel a bit bad about having to break things that are not totally unreasonable in this drastic way and thus offer any help I can give to port legacy DaCHS code.

  Outside of these custom renderers, there should just be a single visible change: If you have used n:data="some_key" in nevow templates to pull data from dictionaries, that won't work any longer. Use n:data="key some_key" n:render="str" instead. And it turns out that this very construct was used in the default root template, which you may have derived from. So – see if you have /var/gavo/web/templates/root.html and if so, whether there is <ul n:data="chunk" in there. If you have that, change it to <ul n:data="key chunk".

  Update (2020-11-19): Two only loosely related problems have surfaced during updates. In particular if you are updating on rather old installations, you may want to look at the points on Invalid script type preIndex and function spoint_in already exists in our list of common problems.

• Parallel Queries

  2020-02-14 Markus Demleitner

  

  An experiment with parallel querying of PPMX, going from single-threaded execution to using seven workers.

  Let me start this post with a TL;DR for

  scientists:

  Large analysis queries (like those that contain a GROUP BY clause) profit a lot from parallel execution, and you needn't do a thing for that.

  DaCHS operators:

  When you have large tables, Postgres 11 together with the next DaCHS release may speed up your responses quite dramatically in some cases.

  So, here's the story –

  I've finally overcome my stretch trauma and upgraded the Heidelberg data center's database server to Debian buster. With that, I got Postgres 11, and I finally bothered to look into what it takes to enable parallel execution of database queries.

  Turns out: My Postgres started to do parallel execution right away, but just in case, I went for the following lines in postgresql.conf:

  max_parallel_workers_per_gather = 4
  max_worker_processes = 10
  max_parallel_workers = 10
  

  Don't quote me on this – I frankly admit I haven't really developed a feeling for the consequences of max_parallel_workers_per_gather and instead just did some experiments while the box was loaded otherwise, determining where raising that number has a diminishing return (see below for more on this).

  The max_worker_processes thing, on the other hand, is an educated guess: on my data center, there's essentially never more than one person at a time who's running “interesting”, long-running queries (i.e., async), and that person should get the majority of the execution units (the box has 8 physical CPUs that look like 16 cores due to hyperthreading) because all other operations are just peanuts in comparison. I'll gladly accept advice to the effect that that guess isn't that educated after all.

  Of course, that wasn't nearly enough. You see, since TAP queries can return rather large result sets – on the GAVO data center, the match limit is 16 million rows, which for a moderate row size of 2 kB already translates to 32 GB of memory use if pulled in at once, half the physical memory of that box –, DaCHS uses cursors (if you're a psycopg2 person: named cursors) to stream results and write them out to disk as they come in.

  Sadly, postgres won't do parallel plans if it thinks people will discard a large part of the result anyway, and it thinks that if you're coming through a cursor. So, in SVN revision 7370 of DaCHS (and I'm not sure if I'll release that in this form), I'm introducing a horrible hack that, right now, just checks if there's a literal “group” in the query and doesn't use a cursor if so. The logic is, roughly: With GROUP, the result set probably isn't all that large, so streaming isn't that important. At the same time, this type of query is probably going to profit from parallel execution much more than your boring sequential scan.

  This gives rather impressive speed gains. Consider this example (of course, it's selected to be extreme):

  import contextlib
  import pyvo
  import time
  
  @contextlib.contextmanager
  def timeit(activity):
    start_time = time.time()
    yield
    end_time = time.time()
    print("Time spent on {}: {} s".format(activity, end_time-start_time))
  
  
  svc = pyvo.tap.TAPService("http://dc.g-vo.org/tap")
  with timeit("Cold (?) run"):
    svc.run_sync("select round(Rmag) as bin, count(*) as n"
      " from ppmx.data group by bin")
  with timeit("Warm run"):
    svc.run_sync("select round(Rmag) as bin, count(*) as n"
      " from ppmx.data group by bin")
  

  (if you run it yourself and you get warnings about VOTable versions from astropy, ignore them; I'm right and astropy is wrong).

  Before enabling parallel execution, this was 14.5 seconds on a warm run, after, it was 2.5 seconds. That's an almost than a 6-fold speedup. Nice!

  Indeed, that holds beyond toy examples. The showcase Gaia density plot:

  SELECT
          count(*) AS obs,
          source_id/140737488355328 AS hpx
  FROM gaia.dr2light
  GROUP BY hpx
  

  (the long odd number is 2³⁵4^16-6, which turns source_ids into level 6-HEALPixes as per Gaia footnote id; please note that Postgres right now isn't smart enough to parallelise ivo_healpix), which traditionally ran for about an hour is now done in less than 10 minutes.

  In case you'd like to try things out on your postgres, here's what I've done to establish the max_parallel_workers_per_gather value above.

  1. Find a table with a few 1e7 rows. Think of a query that will return a small result set in order to not confuse the measurements by excessive client I/O. In my case, that's a magnitude histogram, and the query would be:

     select round(Rmag) as bin, count(*) as n from ppmx.data group by bin;

     Run this query once so the data is in the disk cache (the query is “warm”).

  2. Establish a non-parallel baseline. That's easy to do:

     set max_parallel_workers_per_gather=0;
     

  3. Then run:

     explain analyze select round(Rmag) as bin, count(*) as n from ppmx.data group by bin;
     

     You should see a simple query plan with the runtime for the non-parallel execution – in my case, a bit more than 12 seconds.

  4. Then raise the number of max_parallel_workers_per_gatherer successively. Make sure the query plan has lines of the form “Workers Planned” or so. You should see that the execution time falls with the number of workers you give it, up to the value of max_worker_processes – or until postgres decides your table is too small to warrant further parallelisation, which for my settings happened at 7.

  Note, though, that in realistic, more complex queries, there will probably be multiple operations that will profit from parallelisation in a single query. So, if in this trivial example you can go to 15 gatherers and still see an improvement, this could actually make things slower for complex queries. But as I said above: I have no instinct yet for how things will actually work out. If you have experiences to share: I'm sure I'm not the only person on dachs-users who't be interested.

  Update 2022-05-17: In Postgres 13, I found that the planner disfavours parallel plans a lot stronger than I think it has in Postgres 11. To make up for that, I've amended my postgres configuration (in /etc/postgresql/13/main/postgresql.conf) with the slightly bizarre:

  parallel_tuple_cost = 0.001
  parallel_setup_cost = 3
  

  This is certainly not ideal for every workload, but given the queries I see in the VO I want to give Postgres no excuse not to parallelise when there is at least the shard of a chance it'll help; given I'll never execute more than very few queries per second, the extra overhead for parallelising queries that would be faster sequentially will never really bite me.

• DaCHS 1.4 is out

  2019-10-15 Markus Demleitner

  

  Since the Groningen Interop is over, it's time for a DaCHS release, and so, roughly half a year after the release of DaCHS 1.3, today I've pushed DaCHS 1.4 into our Debian repository.

  As usual, you should upgrade as soon as you find time to do so, because upgrades become more difficult if they span large version gaps; and one of these days you will need some new feature or run into one of the odd bugs. Upgrading is a good opportunity to also get your DaCHS ready for buster by adding the repos mentioned there.

  The list of new features is rather short this time around. Here are some noteworthy ones:

  • There's now an XML grammar that can be used when you have to parse smallish snippets of XML as, for instance, in VOEvent.
  • You can now use TABLESAMPLE(1) after a table specification in DaCHS' ADQL to tell the database engine to just use 1% of a table for a query. While this isn't a precise way to sample tables, it's great when developing queries.
  • Also among new features I'd like to see in ADQL and have therefore put into DaCHS is GENERATE_SERIES(a,b), which is what is known as table-generating function in SQL . If you know SDSS CasJobs, you'll have seen lots of those already. GENERATE_SERIES, however, is really plain: it just spits out a table with a column with integers between a and b. For an example of why one might what to have that, check out the poster I'm linking to in my ADASS report.
  • If you have an updating data descriptor (usually, because you keep feeding data into a data collection), DaCHS will no longer automatically re-make its dependencies (like, say, views). That's because that's not necessary in general, and it's a pain if every update on an obscore-published table tears down and rebuilds the obscore view. For the rare cases when you do need to rebuild dependencies, there's now a remakeOnDataChange attribute on data.
  • At the interop, I've mentioned a few use cases for knowing which server software you're talking to, and I've said that people should set their server headers to informative values. DaCHS does that now.

  To conclude on a low note: This is probably going to be the last release of DaCHS for python 2. Even though we will have to shed a dependency or two that simply will not be ported to python 3, and even though I'm rather unhappy with a few properties of the python 3 port of twisted, there's probably no way to escape this, given that Debian is purging out python 2 packages quickly already.

  So, when we meet again for the next release, you'll probably be looking at DaCHS 2.0, and where you have custom code in your RDs, it's rather likely that you'll see a minor amount of breakage. I promise I'll do everything I can to make the migration easy for deployers, but I can't do higher magic, so: If there's ever been a time to add regression tests to your RDs, it's now.

• DaCHS is Bustered

  2019-08-01 Markus Demleitner

  DaCHS is developed on Debian, and Debian is the recommended deployment platform. Hence, a new major release of Debian (where major means for them: We may break stuff) is always a big thing for me. And so it was with the release that came in July, codenamed “buster”. Both on the “big thing” and on the “break” counts. This posting gives DaCHS deployers some background for their buster upgrades. Astronomers not running Debian themselves won't risk missing anything if they skip this post.

  So, after I upgraded the first thing I noticed is that DaCHS would no longer even start because astropy (which it needs, in particular, because that's where pyfits sits these days) was gone. Simple explanation: Upstream astropy doesn't support python2 any more, and so Debian buster only has python3-astropy.

  Moving DaCHS to python3, unfortunately, isn't that easy; a major dependency, nevow (essentially, a web framework), isn't ported yet, and porting it is a major thing. Believe me, I've tried. The nasty thing, in particular, is that twisted, which lies below nevow still, hands up lots of byte strings. And in python3, b"a"!="a". You wouldn't believe how many interesting bugs that simple truth introduces when you got a library that handed out “just strings” in python2 and now byte strings in python3. Yikes.

  Update (2019-08-28): After quite a bit of experimentation, I finally gave up on providing a python2 version of astropy through release, because for a complicated set of reasons (including numpy declaring a conflict with existing astropys in buster) it is impossible to provide a package that works in buster and doesn't break stretch. So, for buster only you'll have to have a second (or, if running beta, third) gavo line in your sources.list (or equivalent):

  deb http://vo.ari.uni-heidelberg.de/debian buster-foreports main
  

  The instructions at our APT repository have been updated, so you won't have to bookmark this particular page.

  But that wasn't the end of it. Buster comes with Postgres 11, which I look forward to in particular because it supports parallel query execution. That could help us quite a bit, given out large catalogs that quite often we want to run sequential scans on. But of course this means upgrading postgres. And attempting to do that on my development machine immediately hit a wall. What's nice is that the q3c and pgsphere extensions that we've had to push out ourselves so far are now part of Debian main. What's rather fatal is that our pgsphere extensions dealing with HEALPixes and MOCs aren't part of the buster pgsphere package (the reasons for that are tedious and arcane and have to do with OpenSSL and the GPL).

  Also, the pgsphere package coming with buster is called postgres-pgsphere, which is rather unfortunate as it's missing the version indication. So: If you find it on your system, remove it right away. It will conflict with the one true pgsphere package (postgresql-11-pgsphere). That one you'll get from us, and it has the HEALPix stuff built in. TL;DR: run apt install postgresql-q3c postgresql-11-pgsphere before following the postgres update recipe linked above.

  There's a bit more to upgrading the database this time. Because of fairly low-level cleanup in Postgres itself. you're risking index corruption on string indices. Realistically, for almost anything you'll have, it's unlikely that you're affected (it's essentially about non-ASCII in strings), but then it's better to be safe than sorry, and hence you should say:

  reindex database gavo
  

  first thing after you've upgraded to Postgres 11 (which you should really do once the box is on buster). Only if you have very large tables it might be worth it to restrict the index regeneration to indices that could actually need it; see the postgres link above for how to do that.

  One last thing on Postgres upgrades: I've not quite tried to work out why, but probably depending on your /etc/hosts DaCHS on buster is much more likely to connect to your database using IPv6 than it was before. Many older Postgres configurations won't let you in then. If that happens to you, just edit /etc/postgresql/11/main/pg_hba.conf and add a line:

  host    all         all         ::1/32          md5
  

  (or something less permissive if you prefer).

  The next buster-related shock was when TOPCAT's TAP uploads stopped working while my regression tests didn't find anything wrong. After a bit of cursing I eventually figured out that that's not actually buster's fault but twisted's, which in a commit from May 2018 broke chunked uploads (essentially, that's when you're not saying up front how large your upload will be). I've filed a bug report on twisted, but we can't really wait until any sort of fix will be ready and have a broken TOPCAT-DaCHS relationship until then, so for now we're also shipping a fixed twisted package. If you're running DaCHS without our repository enabled, you will have to patch your the twisted code itself. The bug report tells what to do (no warranties, though, because I'm not entriely sure why they changed it in the first place; it's a very small change, though).

  [Update (2019-08-14) scratch the part with the fixed twisted packages. They're too much trouble on stretch systems. You can keep using them on buster boxes if you want, though. The most recent stable release monkeypatches the problem out of presumably broken twisteds, and so will the next beta.]

  I hope you're not totally discouraged now, because upgrade you should (though perhaps not right before going on vacation) – distribution upgrades are unavoidable if you want to run services for decades, and that's definitely a goal within the VO. See the Debian release note for Debian's take on dist upgrades, which arguably is a bit more alarmist than it would need to; a lean, server-only system typically is really simple to upgrade.

  Given the relatively large number of Debian packages we override in buster, I'll be particularly grateful if you complain early about breakage you observe (ideally use the dachs-support mailing list, but see Support for alternatives), and as usual you are encouraged to try the upgrade first on a development system if you have one. Which you should.

• DaCHS 1.3 is out

  2019-05-28 Markus Demleitner

  

  Almost a year has passed since release 1.2 of DaCHS – I've let the normal autumn release slip last year because there weren't so many release-worthy new features in DaCHS at the traditional release time (i.e., after the College Park interop), and also because running betas when you do need a new feature is a fairly stable thing by now.

  But here it finally is: Release 1.3 (tarball for the die-hard self-builders; everyone else just switches back the release branch as necessary and then runs an update/upgrade cycle).

  Here's the commented changelog:

  • New //ssap#view mixin that should be used for future SSAP services, and that existing SSAP services should migrate to at some point. See A new view on SSAP in DaCHS on this blog for details.
  • Columns can now be hidden from TAP/ADQL (and other interfaces) by setting hidden="True".
  • There is now a setting [web]maxSyncUploadSize=500000 (meaning: about 500 kByte) as the default upload limit on sync queries. In compensation, clients uploading too much now receive a more useful error message (except it doesn't reach TOPCAT users most of the time because it does chunked uploads). To get back the behaviour of 1.2 (which is probably ok if you can live with the occasional resource hog), add maxSyncUploadSize=20000000 to your /etc/gavo.rc.
  • Adding support for https (certificate reading, certificate updating with letsencrypt, registering alternate endpoints, no WebSAMP with https). See HTTPS in DaCHS on this blog for details.
  • New source_table and preview columns in obscore. If you're using the various obscore mixins, this should be automatic. If you have defined views manually, you will have to amend these (and have a broken obscore until a dachs upgrade ran without error).
  • No longer producing arraysize="1" in VOTables for scalars (except char, for compatibility with a legacy TOPCAT workaround; see VOTable 1.3 Erratum 3 for background information).
  • Support for draft TIMESYS in VOTable (with STC 2 annotation; ask about details if you're interested. This is for draft VOTable 1.4 and probably only relevant to you if you're publishing time series).
  • You can now add targetType and targetTitle properties to URL-valued columns to help Aladin figure out what to do with URLs (see Datalinks as product URLs in the reference documentation).
  • New gavo_transform, gavo_ipix, and gavo_urlescape ufuncs for ADQL, fixed gavo_urlescape to have acceptable performance.
  • New generating CatalogResource records with auxiliary capabilities in accordance with Oct 2018 VODataService WD.
  • //soda#sdm_genDesc now matches accref rather than pubDID by default. If you use Datalink with SSA and have a custom pubDID schema (or no index on accref), add a useAccref="False" to your descriptorGenerator statement.
  • There is now a --foreground option for dachs serve start. This is mainly to play nice with systemd, and indeed, the Debian package now comes with a systemd unit file. I'm not terribly familiar with systemd, so please have an eye on DaCHS controlled by systemd and let me know if you see something that's not as it should be.
  • Fixes for various bugs (most notably: escaped quotes in ADQL, WCS in SIAP cutout products) and many minor improvements. Check out the source tree (still via subversion) and read the changelog if you want to know the whole truth.

  On systems running from the Debian package, the update should be automatic with the next system upgrade. However, you'll be saving yourself quite a bit of headache if you check the health of your installation before the upgrade; see Upgrading DaCHS in the operator's guide on how to upgrade professionally.

« Page 4 / 6 »