I noticed one more thing... imgserv is using by default this path
/lsst7/releaseW13EP/
and that is another 194GB. At this point I am totally lost.
I generated some catalog based on a subset of
/raid/lauren/rerun/LSST/STRIPE82L/v2
when producing the catalog I used something from _parent, which
goes to CALIB, and in order to display images we told imgserv
to fetch images from /lsst7/releaseW13EP
That feels so convoluted I have no clue which subset I should
be copying over. K-T, do you know? Should I ask Laurie or Paul?
Should I just give up and run the whole thing on lsst-dev?
Or maybe attached 1 TB to that VM and copy everything?
Jacek
On 01/21/2016 03:18 PM, Pease, Nathan wrote:
> if there's a registry (file name registry.sqlite3) you'll want that.
> if there's a _policy file you'll want that too, but if the dataset was used at Bremerton then it won't be there; we didn't support in-repo policy back then.
>
> otherwise, you just won't be able to get things from the datasets that you don't copy down. This may be a problem; I don't know what will be needed from the calibrations (CALIB datasets). And the registry might have a record of what's in the missing files (evidently the registry knows what "might" be there, not what "is" there - this is something I don't entirely grok).
> But grab that stuff named above, and give it a try. Let me know how it goes...
>
> n
>
> ________________________________________
> From: Becla, Jacek
> Sent: Thursday, January 21, 2016 3:06 PM
> To: Pease, Nathan; qserv-l
> Subject: Re: [QSERV-L] webserv and butler issue
>
> Ah, that is what it is. So, on lsst-dev (where things were working)
> _parent has lots of files. Per hipchat discussion with K-T, I only
> brought _mapper, and here is how I did that:
>
>
> ls -l STRIPE82L/v2/
> total 0
> drwxrwsr-x. 6 centos centos 55 Aug 18 21:06 deepCoadd-results
> drwxrwxr-x. 2 centos centos 20 Jan 19 23:05 _parent
>
> ls -l STRIPE82L/v2/_parent/
> total 4
> -rw-rw-r--. 1 centos centos 23 Aug 5 18:07 _mapper
>
> I can't just blindly copy the whole thing because _parent has
> a symlink inside to CALIB directory which is over 300 GB,
> and I am trying to come up with a nice small data set here
> that will fit into a small VM. So how do I know what really
> needs to be copied?
>
>
> Jacek
>
>
>
>
>
>
> On 01/21/2016 02:48 PM, Pease, Nathan wrote:
>> a mapper instance can be passed into Butler at init time, but that's not being done here:
>> "/home/centos/stack/repos/dax_imgserv/python/lsst/dax/imgserv/locateImage.py",
>> line 214, in _getImageButler
>> butler = lsst.daf.persistence.Butler(self._dataRoot)
>>
>> (mapper would be a 2nd arg).
>>
>> it's possible there was a parent repo specified in the repo dir you copied over. you could look for a symlink inside the dir called _parent. If you find it, it would point to another repo on the originating filesystem. Look there for a file called _mapper. If you find that, copy it to the repo you have on your system, or copy the whole parent repo (and recreate the _parent symlink to be accurate if necessary). You may have to follow several _parent links. (Even if you find the _mapper right away, you may need to keep following _parent to make sure you have all the data needed to run the script)
>>
>> ping me if you need help or clarification.
>>
>> n
>>
>> ________________________________________
>> From: [log in to unmask] <[log in to unmask]> on behalf of Jacek Becla <[log in to unmask]>
>> Sent: Thursday, January 21, 2016 2:26 PM
>> To: qserv-l
>> Subject: [QSERV-L] webserv and butler issue
>>
>> I need some here, probably Nate, Serge or John will be able to help.
>>
>> So I am trying to setup webserv for ipac... I copied this directory over:
>>
>> /raid/lauren/rerun/LSST/STRIPE82L/v2/deepCoadd-results
>>
>> from lsst-dev, and I ran Serge's ingest script, ending up
>> with some tables in the database becla_bremerton on our
>> lsst10 server.
>>
>> I pointed webserv to lsst10 (like we did at Bremerton),
>> and tried to run some queries we used to run at Bremerton, eg
>>
>>
>> curl
>> 'http://localhost:8661/image/v0/raw?ra=359.195&dec=-0.1055&filter=r' > 1.
>>
>> (see full list at:
>>
>> https://confluence.lsstcorp.org/display/DM/Webserv+for+SUI+Tests#WebservforSUITests-ExampleImageQueries
>> )
>>
>>
>> the imgserv is failing with
>>
>>
>> 2819111 [0x7f7b4179c740] INFO root null - raw ra=359.195 dec=-0.1055 filt=r
>> 2819125 [0x7f7b4179c740] INFO root null - SET time_zone = '+0:00'
>> 2819126 [0x7f7b4179c740] INFO root null - SELECT
>> ra,decl,run,camcol,field,filterName,(power((ra - 359.195),2) +
>> power((decl - -0.1055),2)) as distance FROM Science_Ccd_Exposure WHERE
>> scisql_s2PtInBox(ra, decl, 359.194305556, -0.106194444444,
>> 359.195694444, -0.104805555556) = 1 order by distance LIMIT 1
>> 01/21/2016 10:18:05 __main__ ERROR: Exception on /image/v0/raw [GET]
>> Traceback (most recent call last):
>> File
>> "/home/centos/stack/Linux64/flask/0.10.1+11/lib/python/Flask-0.10.1-py2.7.egg/flask/app.py",
>> line 1817, in wsgi_app
>> response = self.full_dispatch_request()
>> File
>> "/home/centos/stack/Linux64/flask/0.10.1+11/lib/python/Flask-0.10.1-py2.7.egg/flask/app.py",
>> line 1477, in full_dispatch_request
>> rv = self.handle_user_exception(e)
>> File
>> "/home/centos/stack/Linux64/flask/0.10.1+11/lib/python/Flask-0.10.1-py2.7.egg/flask/app.py",
>> line 1381, in handle_user_exception
>> reraise(exc_type, exc_value, tb)
>> File
>> "/home/centos/stack/Linux64/flask/0.10.1+11/lib/python/Flask-0.10.1-py2.7.egg/flask/app.py",
>> line 1475, in full_dispatch_request
>> rv = self.dispatch_request()
>> File
>> "/home/centos/stack/Linux64/flask/0.10.1+11/lib/python/Flask-0.10.1-py2.7.egg/flask/app.py",
>> line 1461, in dispatch_request
>> return self.view_functions[rule.endpoint](**req.view_args)
>> File
>> "/home/centos/stack/repos/dax_imgserv/python/lsst/dax/imgserv/imageREST_v0.py",
>> line 94, in getRaw
>> return _getIFull(request, W13RawDb)
>> File
>> "/home/centos/stack/repos/dax_imgserv/python/lsst/dax/imgserv/imageREST_v0.py",
>> line 149, in _getIFull
>> imgFull = w13db.getImageFull(ra, dec)
>> File
>> "/home/centos/stack/repos/dax_imgserv/python/lsst/dax/imgserv/locateImage.py",
>> line 74, in getImageFull
>> img, metadata = self.getImageFullWithMetadata(ra, dec)
>> File
>> "/home/centos/stack/repos/dax_imgserv/python/lsst/dax/imgserv/locateImage.py",
>> line 85, in getImageFullWithMetadata
>> img, butler = self._getImageButler(res)
>> File
>> "/home/centos/stack/repos/dax_imgserv/python/lsst/dax/imgserv/locateImage.py",
>> line 214, in _getImageButler
>> butler = lsst.daf.persistence.Butler(self._dataRoot)
>> File
>> "/home/centos/stack/Linux64/daf_persistence/2015_10.0-5-g4063539+6/python/lsst/daf/persistence/butler.py",
>> line 137, in __init__
>> cls = Butler.getMapperClass(root)
>> File
>> "/home/centos/stack/Linux64/daf_persistence/2015_10.0-5-g4063539+6/python/lsst/daf/persistence/butler.py",
>> line 103, in getMapperClass
>> (mapperFile,))
>> RuntimeError: No mapper provided and no _mapper available
>> 01/21/2016 10:18:05 werkzeug INFO: 127.0.0.1 - - [21/Jan/2016 22:18:05]
>> "GET /image/v0/raw?ra=359.195&dec=-0.1055&filter=r HTTP/1.1" 500 -
>>
>> ---
>>
>> So why am I missing _mapper?
>>
>> BTW, I suspect we need to redo the sample image queries (I'll need
>> help with that :).
>>
>> ---
>>
>> Oh, and I just noticed that imgserv has this hardcoded:
>>
>> database="DC_W13_Stripe82",
>> table="Science_Ccd_Exposure",
>>
>> that is pretty ugly. We need to sort it all out, but right now I'd
>> like to just get to where we were during Bremerton meeting.
>>
>> J.
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the QSERV-L list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
>>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
|