 |
 |
if there's a registry (file name registry.sqlite3) you'll want that. if there's a _policy file you'll want that too, but if the dataset was used at Bremerton then it won't be there; we didn't support in-repo policy back then. otherwise, you just won't be able to get things from the datasets that you don't copy down. This may be a problem; I don't know what will be needed from the calibrations (CALIB datasets). And the registry might have a record of what's in the missing files (evidently the registry knows what "might" be there, not what "is" there - this is something I don't entirely grok). But grab that stuff named above, and give it a try. Let me know how it goes... n ________________________________________ From: Becla, Jacek Sent: Thursday, January 21, 2016 3:06 PM To: Pease, Nathan; qserv-l Subject: Re: [QSERV-L] webserv and butler issue Ah, that is what it is. So, on lsst-dev (where things were working) _parent has lots of files. Per hipchat discussion with K-T, I only brought _mapper, and here is how I did that: ls -l STRIPE82L/v2/ total 0 drwxrwsr-x. 6 centos centos 55 Aug 18 21:06 deepCoadd-results drwxrwxr-x. 2 centos centos 20 Jan 19 23:05 _parent ls -l STRIPE82L/v2/_parent/ total 4 -rw-rw-r--. 1 centos centos 23 Aug 5 18:07 _mapper I can't just blindly copy the whole thing because _parent has a symlink inside to CALIB directory which is over 300 GB, and I am trying to come up with a nice small data set here that will fit into a small VM. So how do I know what really needs to be copied? Jacek On 01/21/2016 02:48 PM, Pease, Nathan wrote: > a mapper instance can be passed into Butler at init time, but that's not being done here: > "/home/centos/stack/repos/dax_imgserv/python/lsst/dax/imgserv/locateImage.py", > line 214, in _getImageButler > butler = lsst.daf.persistence.Butler(self._dataRoot) > > (mapper would be a 2nd arg). > > it's possible there was a parent repo specified in the repo dir you copied over. you could look for a symlink inside the dir called _parent. If you find it, it would point to another repo on the originating filesystem. Look there for a file called _mapper. If you find that, copy it to the repo you have on your system, or copy the whole parent repo (and recreate the _parent symlink to be accurate if necessary). You may have to follow several _parent links. (Even if you find the _mapper right away, you may need to keep following _parent to make sure you have all the data needed to run the script) > > ping me if you need help or clarification. > > n > > ________________________________________ > From: [log in to unmask] <[log in to unmask]> on behalf of Jacek Becla <[log in to unmask]> > Sent: Thursday, January 21, 2016 2:26 PM > To: qserv-l > Subject: [QSERV-L] webserv and butler issue > > I need some here, probably Nate, Serge or John will be able to help. > > So I am trying to setup webserv for ipac... I copied this directory over: > > /raid/lauren/rerun/LSST/STRIPE82L/v2/deepCoadd-results > > from lsst-dev, and I ran Serge's ingest script, ending up > with some tables in the database becla_bremerton on our > lsst10 server. > > I pointed webserv to lsst10 (like we did at Bremerton), > and tried to run some queries we used to run at Bremerton, eg > > > curl > 'http://localhost:8661/image/v0/raw?ra=359.195&dec=-0.1055&filter=r' > 1. > > (see full list at: > > https://confluence.lsstcorp.org/display/DM/Webserv+for+SUI+Tests#WebservforSUITests-ExampleImageQueries > ) > > > the imgserv is failing with > > > 2819111 [0x7f7b4179c740] INFO root null - raw ra=359.195 dec=-0.1055 filt=r > 2819125 [0x7f7b4179c740] INFO root null - SET time_zone = '+0:00' > 2819126 [0x7f7b4179c740] INFO root null - SELECT > ra,decl,run,camcol,field,filterName,(power((ra - 359.195),2) + > power((decl - -0.1055),2)) as distance FROM Science_Ccd_Exposure WHERE > scisql_s2PtInBox(ra, decl, 359.194305556, -0.106194444444, > 359.195694444, -0.104805555556) = 1 order by distance LIMIT 1 > 01/21/2016 10:18:05 __main__ ERROR: Exception on /image/v0/raw [GET] > Traceback (most recent call last): > File > "/home/centos/stack/Linux64/flask/0.10.1+11/lib/python/Flask-0.10.1-py2.7.egg/flask/app.py", > line 1817, in wsgi_app > response = self.full_dispatch_request() > File > "/home/centos/stack/Linux64/flask/0.10.1+11/lib/python/Flask-0.10.1-py2.7.egg/flask/app.py", > line 1477, in full_dispatch_request > rv = self.handle_user_exception(e) > File > "/home/centos/stack/Linux64/flask/0.10.1+11/lib/python/Flask-0.10.1-py2.7.egg/flask/app.py", > line 1381, in handle_user_exception > reraise(exc_type, exc_value, tb) > File > "/home/centos/stack/Linux64/flask/0.10.1+11/lib/python/Flask-0.10.1-py2.7.egg/flask/app.py", > line 1475, in full_dispatch_request > rv = self.dispatch_request() > File > "/home/centos/stack/Linux64/flask/0.10.1+11/lib/python/Flask-0.10.1-py2.7.egg/flask/app.py", > line 1461, in dispatch_request > return self.view_functions[rule.endpoint](**req.view_args) > File > "/home/centos/stack/repos/dax_imgserv/python/lsst/dax/imgserv/imageREST_v0.py", > line 94, in getRaw > return _getIFull(request, W13RawDb) > File > "/home/centos/stack/repos/dax_imgserv/python/lsst/dax/imgserv/imageREST_v0.py", > line 149, in _getIFull > imgFull = w13db.getImageFull(ra, dec) > File > "/home/centos/stack/repos/dax_imgserv/python/lsst/dax/imgserv/locateImage.py", > line 74, in getImageFull > img, metadata = self.getImageFullWithMetadata(ra, dec) > File > "/home/centos/stack/repos/dax_imgserv/python/lsst/dax/imgserv/locateImage.py", > line 85, in getImageFullWithMetadata > img, butler = self._getImageButler(res) > File > "/home/centos/stack/repos/dax_imgserv/python/lsst/dax/imgserv/locateImage.py", > line 214, in _getImageButler > butler = lsst.daf.persistence.Butler(self._dataRoot) > File > "/home/centos/stack/Linux64/daf_persistence/2015_10.0-5-g4063539+6/python/lsst/daf/persistence/butler.py", > line 137, in __init__ > cls = Butler.getMapperClass(root) > File > "/home/centos/stack/Linux64/daf_persistence/2015_10.0-5-g4063539+6/python/lsst/daf/persistence/butler.py", > line 103, in getMapperClass > (mapperFile,)) > RuntimeError: No mapper provided and no _mapper available > 01/21/2016 10:18:05 werkzeug INFO: 127.0.0.1 - - [21/Jan/2016 22:18:05] > "GET /image/v0/raw?ra=359.195&dec=-0.1055&filter=r HTTP/1.1" 500 - > > --- > > So why am I missing _mapper? > > BTW, I suspect we need to redo the sample image queries (I'll need > help with that :). > > --- > > Oh, and I just noticed that imgserv has this hardcoded: > > database="DC_W13_Stripe82", > table="Science_Ccd_Exposure", > > that is pretty ugly. We need to sort it all out, but right now I'd > like to just get to where we were during Bremerton meeting. > > J. > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the QSERV-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1