Posts with the Tag Licences:

  • DaCHS 2.3 on the way to Debian main

    2021-01-12 Markus Demleitner
    DaCHS, Debian, and 2.3

    DaCHS 2.3 will be the first DaCHS officially in Debian.

    DaCHS releases usually come around the Interops in (roughly) May and November. Not this one, though, for one pleasant, one unpleasant, and several other reasons.

    The unpleasant reason first: The 2.2 release has a fairly severe memory leak in it (resulting, in roundabout ways, from python 3 preserving tracebacks of nested exceptions), which of course really became virulent on my server right over the holidays. If you run a site with just a few gigs of RAM that might be hit by second-rate async clients, this will bite you and you ought to upgrade now (well, you ought to upgrade anyway).

    The pleasant reason is that DaCHS has made it into Debian main and thus, unless something disastrous happens, it will be part of the Debian version 11 (“bullseye”). This means that people who do not need to be on the bleeding edge, will not need to monkey around with our repository (and its signing key) any more starting some time in 2021 (or just about now, if they're running testing). I can't tell you how gratifying that feels to me. And well, I wanted relatively recent code corresponding to a something on our release branch in bullseye.

    One of the other reasons is that stilts' author Mark Taylor is trying to stomp out TAP services failing his taplint's validation, and many DaCHS 2.2 services (those that don't define TAP examples, which of course is a shame anyway) fail with only the (really minor) error E-EXDH-1 (see below).

    DaCHS 2.3 has some other noteworthy changes; as usual in minor version steps, my expectation is that none of this will break existing services. Still, you may want to glance over the following list, as there are some behavioural changes nevertheless. In approximate order of the wizardry involved:

    • I've long had a bad consciousness because DaCHS has stored cleartext passwords so far. That's probably not a problem for DaCHS itself (as it does not protect great riches), but people tend to re-use passwords, and I'd have hated to leak passwords that might work elsewhere. Well, no longer: the dc.users table now contains hashed passwords, and the upgrade will hash them. This, in particular, means that you cannot recover them once you have updated (which, of course, is as it should be).

    • The javascript delivered with DaCHS was no longer quite up to date with Debian's jquery. I have updated it in several ways, and I have restored the functionality of the WebSAMP button in the default response. If you have custom HTML templates containing javascript, you may need to update them to newer jquery, too, specifically,

      • change .unload( to .on("unload", (this happens in the SAMP code in defaultresponse.html, for instance).
      • also in the SAMP code in overridden defaultresponses, change the icon URL to completeURL("/logo_tiny.png") (or whatever) to avoid trouble with https installations.
      • if you compare jquery element names: these are now returned in lower case.

      And yes, WebSAMP now mostly works with HTTPS (which is unrelated to this update, except that DaCHS until 2.2 suppresses the WebSAMP button when it thinks it is delivering through HTTPS).

    • DaCHS now honours upgrade-insecure-requests headers that common web browsers issue and will then redirect them to https when appropriate. So, please don't forcibly do these redirects any more from reverse proxies – they break, among other things, TAP, and they're generally just a bad idea.

    • DaCHS now instructs the database to return all bits of floating point numbers. This may break your regression tests, but it's the right thing to do (blog post on this).

    • Another thing that may break regression tests: TAP results now have column names in the case given in the RD (where previously they were lowercased unless quoted). Let me cite rule 1 of SQL table design: Don't use mixed-case column names.

    • Wildcards in the directory parts of sources patterns are now expanded, which means that you can write things like <sources pattern="data/202?/*.fits"/>, which previously wouldn't have done what you might reasonably expect; however, this might in rare cases match additional sources when you re-import data.

    • The examples endpoint now returns a 404 if no examples are defined on a service; this fixes the stilts taplint E-EXA-EXDH-1 error I mentioned above.

    • DaCHS will now refuse to use x-unregistred as an authority when publishing resources or creating publisher DIDs. This is to protect to people who do a lot of imports before settling on their authority; sometimes DaCHS' fallback null authority got into their databases, which then caused quite a bit of cleanup effort.

    • Because of licensing problems, the Debian package no longer contains the CC logos for the time being. If you want them back, drop appropriate files cc0.png, ccby.png, and ccybysa.png into /var/gavo/web/nv_static/img

    • You can now list modules you want in a procedure application in its setup/@imports attribute. I've done this after I had to add code to a proc's setup just to run an import once too often.

    • simbadinterface's Sesame now uses the dc.metastore table to cache results rather than files as before. Previous saveNew, id, and debug parameters are no longer supported (the base.caches.getSesame interface is unchanged, so it's unlikely you'd notice this).

    • table.query() or querier.query() are now seriously deprecated (you may have used them in code embedded in RDs). See Database Queries in the reference documentation for what the recommended query patterns are (and have been for a while). Just one word of warning: table.query would macro-expand its argument, which the connection method obviously cannot. If you depend on that, call table.expand(query) manually first.

    With this: Merry upgrading and a happy new year!

  • DaCHS 1.0 released

    2017-07-11 Markus Demleitner

    Today, I have released DaCHS 1.0 – after long years in the 0.9 range, it was finally time to do so. The jump in the major version number was an opportunity to remove some cruft that had accumulated over the years; this, on the other hand, means that if you're running DaCHS, you should watch the upgrade and see if anything broke later (this might be the perfect time to add regression tests to your RDs).

    The changelog is below, but before that a bold-faced warning:

    Install python-astropy before upgrading

    This is because DaCHS now depends on astropy rather than pyfits and pywcs. The latter is no longer part of Debian stretch, and so we made the jump to astropy (that would have been due during Debian stretch's lifetime anyway) even before 1.0.

    Now, Debian holds back packages with new dependencies, and due to the way DaCHS' modules are distributed, DaCHS will break when some of its packages are held back. The symptom is error messages like "pkg_resources.DistributionNotFound: gavodachs==0.9.8". If you already see those, a apt-get dist-upgrade should get you in business again.

    With this out of the way, here is an annotated log of the major changes:

    • DaCHS' main entry point is now actually called dachs (i.e., call dachs imp q and such in the future). gavo will work as an alias for quite a while to come, though, and it's still used a lot in the documentation (you're welcome to fix this: the docs are maintained on github).
    • Hopefully more useful manpage (of course, also available with man dachs) – have a peek!
    • UWS support is now at version 1.1 (i.e., there's creationDate in jobs, filters in the joblist, and slow polling).
    • Added “declarative” licenses. Please read the Licensing chapter in the tutorial and slap licenses on your data.
    • Now using astropy.wcs instead of pywcs, and astropy.io.fits instead of pyfits. The respective APIs have, unfortunately, changed quite a bit. If you're using them (e.g., in processors), you'll have to change your code; it's unlikely services are impacted at runtime. (see also How do I update my code?).
    • Removed the //epntap#table-2_0mixin. Use
      //epntap2#table-2_0 instead (sorry).
    • Removed sdmCore (use Datalink/SODA instead); the SODA procs in //datalink are also gone, use the ones from //soda instead (sorry, SODA development has been difficult on the IVOA level).
    • Removed imp -u flag and the corresponding updateMode parse option. If you used that or the uploadCore, just mark the DDs involved with updating="True" instead.
    • Massive sanitation of input parameter processing. If you've been using inputTable, inputDD, or have been doing creative things with inputKeys, please check the respective services carefully after upgrading. See also DaCHS' Service Interface in the reference documentation. The most user-visible change in this department is if you've been using repeated parameters to fill array-valued inputs. That's no longer allowed; if you actually must have this kind of thing, you'll need a custom core and must fill the arrays by hand.
    • In DaCHS' SQL interface, tuples now are matched to records and lists to arrays (it was the other way round before). If while importing you manually created tuples to fill to array-like columns, you'll have to make lists from these now.
    • rsc.makeData or rsc.TableForDef no longer automatically make connections when used on database tables. You must give them explicit connection arguments now (with base.getTableConn() as conn:).
    • logo_tiny.png and logo_big.png are now ignored by DaCHS, all logos spit out by it are now based on logo_medium.png, including, if not overridden, the favicon (that you will now get if you have not set it before).
    • Removed (probably largely unused) features editCore, SDM2 support, pkg_resource overrides, simpleView, computedCore.
    • Removed the argparse module shipped with DaCHS. This breaks compatibility with python 2.6 (although you can still run DaCHS with a manually installed argparse.py in 2.6).

    Even though that's quite a mouthful, I expect few people will actually experience breaking services. If you do, by all means let us know on the DaCHS-support mailing list.

    As usual, the general upgrading instructions are available in the operator's guide; if you plan on upgrading to stretch soon, also have a look at hints on postgres upgrades. Stretch comes with postgres 9.6 (jessie: 9.4), and you should migrate sooner or later anyway.

    Users not using Debian's package management can, as usual, grab tarballs from http://soft.g-vo.org/dachs.

Page 1 / 1