if there's a registry (file name registry.sqlite3) you'll want that.
if there's a _policy file you'll want that too, but if the dataset was used at Bremerton then it won't be there; we didn't support in-repo policy back then.
otherwise, you just won't be able to get things from the datasets that you don't copy down. This may be a problem; I don't know what will be needed from the calibrations (CALIB datasets). And the registry might have a record of what's in the missing files (evidently the registry knows what "might" be there, not what "is" there - this is something I don't entirely grok).
But grab that stuff named above, and give it a try. Let me know how it goes...
n
________________________________________
From: Becla, Jacek
Sent: Thursday, January 21, 2016 3:06 PM
To: Pease, Nathan; qserv-l
Subject: Re: [QSERV-L] webserv and butler issue
Ah, that is what it is. So, on lsst-dev (where things were working)
_parent has lots of files. Per hipchat discussion with K-T, I only
brought _mapper, and here is how I did that:
ls -l STRIPE82L/v2/
total 0
drwxrwsr-x. 6 centos centos 55 Aug 18 21:06 deepCoadd-results
drwxrwxr-x. 2 centos centos 20 Jan 19 23:05 _parent
ls -l STRIPE82L/v2/_parent/
total 4
-rw-rw-r--. 1 centos centos 23 Aug 5 18:07 _mapper
I can't just blindly copy the whole thing because _parent has
a symlink inside to CALIB directory which is over 300 GB,
and I am trying to come up with a nice small data set here
that will fit into a small VM. So how do I know what really
needs to be copied?
Jacek
On 01/21/2016 02:48 PM, Pease, Nathan wrote:
> a mapper instance can be passed into Butler at init time, but that's not being done here:
> "/home/centos/stack/repos/dax_imgserv/python/lsst/dax/imgserv/locateImage.py",
> line 214, in _getImageButler
> butler = lsst.daf.persistence.Butler(self._dataRoot)
>
> (mapper would be a 2nd arg).
>
> it's possible there was a parent repo specified in the repo dir you copied over. you could look for a symlink inside the dir called _parent. If you find it, it would point to another repo on the originating filesystem. Look there for a file called _mapper. If you find that, copy it to the repo you have on your system, or copy the whole parent repo (and recreate the _parent symlink to be accurate if necessary). You may have to follow several _parent links. (Even if you find the _mapper right away, you may need to keep following _parent to make sure you have all the data needed to run the script)
>
> ping me if you need help or clarification.
>
> n
>
> ________________________________________
> From: [log in to unmask] <[log in to unmask]> on behalf of Jacek Becla <[log in to unmask]>
> Sent: Thursday, January 21, 2016 2:26 PM
> To: qserv-l
> Subject: [QSERV-L] webserv and butler issue
>
> I need some here, probably Nate, Serge or John will be able to help.
>
> So I am trying to setup webserv for ipac... I copied this directory over:
>
> /raid/lauren/rerun/LSST/STRIPE82L/v2/deepCoadd-results
>
> from lsst-dev, and I ran Serge's ingest script, ending up
> with some tables in the database becla_bremerton on our
> lsst10 server.
>
> I pointed webserv to lsst10 (like we did at Bremerton),
> and tried to run some queries we used to run at Bremerton, eg
>
>
> curl
> 'http://localhost:8661/image/v0/raw?ra=359.195&dec=-0.1055&filter=r' > 1.
>
> (see full list at:
>
> https://confluence.lsstcorp.org/display/DM/Webserv+for+SUI+Tests#WebservforSUITests-ExampleImageQueries
> )
>
>
> the imgserv is failing with
>
>
> 2819111 [0x7f7b4179c740] INFO root null - raw ra=359.195 dec=-0.1055 filt=r
> 2819125 [0x7f7b4179c740] INFO root null - SET time_zone = '+0:00'
> 2819126 [0x7f7b4179c740] INFO root null - SELECT
> ra,decl,run,camcol,field,filterName,(power((ra - 359.195),2) +
> power((decl - -0.1055),2)) as distance FROM Science_Ccd_Exposure WHERE
> scisql_s2PtInBox(ra, decl, 359.194305556, -0.106194444444,
> 359.195694444, -0.104805555556) = 1 order by distance LIMIT 1
> 01/21/2016 10:18:05 __main__ ERROR: Exception on /image/v0/raw [GET]
> Traceback (most recent call last):
> File
> "/home/centos/stack/Linux64/flask/0.10.1+11/lib/python/Flask-0.10.1-py2.7.egg/flask/app.py",
> line 1817, in wsgi_app
> response = self.full_dispatch_request()
> File
> "/home/centos/stack/Linux64/flask/0.10.1+11/lib/python/Flask-0.10.1-py2.7.egg/flask/app.py",
> line 1477, in full_dispatch_request
> rv = self.handle_user_exception(e)
> File
> "/home/centos/stack/Linux64/flask/0.10.1+11/lib/python/Flask-0.10.1-py2.7.egg/flask/app.py",
> line 1381, in handle_user_exception
> reraise(exc_type, exc_value, tb)
> File
> "/home/centos/stack/Linux64/flask/0.10.1+11/lib/python/Flask-0.10.1-py2.7.egg/flask/app.py",
> line 1475, in full_dispatch_request
> rv = self.dispatch_request()
> File
> "/home/centos/stack/Linux64/flask/0.10.1+11/lib/python/Flask-0.10.1-py2.7.egg/flask/app.py",
> line 1461, in dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File
> "/home/centos/stack/repos/dax_imgserv/python/lsst/dax/imgserv/imageREST_v0.py",
> line 94, in getRaw
> return _getIFull(request, W13RawDb)
> File
> "/home/centos/stack/repos/dax_imgserv/python/lsst/dax/imgserv/imageREST_v0.py",
> line 149, in _getIFull
> imgFull = w13db.getImageFull(ra, dec)
> File
> "/home/centos/stack/repos/dax_imgserv/python/lsst/dax/imgserv/locateImage.py",
> line 74, in getImageFull
> img, metadata = self.getImageFullWithMetadata(ra, dec)
> File
> "/home/centos/stack/repos/dax_imgserv/python/lsst/dax/imgserv/locateImage.py",
> line 85, in getImageFullWithMetadata
> img, butler = self._getImageButler(res)
> File
> "/home/centos/stack/repos/dax_imgserv/python/lsst/dax/imgserv/locateImage.py",
> line 214, in _getImageButler
> butler = lsst.daf.persistence.Butler(self._dataRoot)
> File
> "/home/centos/stack/Linux64/daf_persistence/2015_10.0-5-g4063539+6/python/lsst/daf/persistence/butler.py",
> line 137, in __init__
> cls = Butler.getMapperClass(root)
> File
> "/home/centos/stack/Linux64/daf_persistence/2015_10.0-5-g4063539+6/python/lsst/daf/persistence/butler.py",
> line 103, in getMapperClass
> (mapperFile,))
> RuntimeError: No mapper provided and no _mapper available
> 01/21/2016 10:18:05 werkzeug INFO: 127.0.0.1 - - [21/Jan/2016 22:18:05]
> "GET /image/v0/raw?ra=359.195&dec=-0.1055&filter=r HTTP/1.1" 500 -
>
> ---
>
> So why am I missing _mapper?
>
> BTW, I suspect we need to redo the sample image queries (I'll need
> help with that :).
>
> ---
>
> Oh, and I just noticed that imgserv has this hardcoded:
>
> database="DC_W13_Stripe82",
> table="Science_Ccd_Exposure",
>
> that is pretty ugly. We need to sort it all out, but right now I'd
> like to just get to where we were during Bremerton meeting.
>
> J.
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the QSERV-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
|
