On 3/27/19 12:51 PM, Michal Kamil Simon wrote:
> Hi Adrian,
Hi!
> Regarding 1) From what I see, unfortunately the API for specifying the
> number of parallel copy jobs is
> not exposed in python bindings.
>
> Regarding 2) Although there is the /sourcelimit/ argument in the python
> bindings, from what I checked
> in the source code it is ignored. So in practice this functionality is
> also not exposed in python bindings.
>
> If you wish I can expose those features in python bindings and they
> potentially could be released in 4.10.0.
> Could you please create respective feature request in our GitHub
> (https://github.com/xrootd/xrootd) so
> I don't forget?
Sure! I think that is best to file 2 different issues ...one about the
number of copy jobs in a batch and one about the multiple sources of a file
so first one :
https://github.com/xrootd/xrootd/issues/948
Also, given your feedback, i would like to be more specific of my(ALICE)
use case about multiple sources :
We have a translation layer, and the name of the physical files
(actually on the servers) is different on each server (it has a GUID)
and these GUIDs in the central catalogue are mapped to a Logical File
Name. Moreover each GUID have a specific authorization envelope
so for each server there is need for a full link specification
So i think that just a switch is not enough ... maybe the usage of comma
[,] could be implemented? for something like :
xrdcp my_local_file
remote_link_1?authz=LONG_ENVELOPE_1&var1=value,remote_link_2?authz=LONG_ENVELOPE_2&var1=value,..
remote_link_N?authz=LONG_ENVELOPE_N&var1=value
Also, for python that would make the signature of add_job to take as
source/destination a list []
What do you think?
For this second part, should i open an issue now, or after we discuss
more on this?
Btw, do you have any idea why i get this on a fedora 27?
pip3_install_user xrootd
Collecting xrootd
Using cached
https://files.pythonhosted.org/packages/d5/9a/33a68c3efd226aff30535790b66b3340c91b67d6418d264cf2778a6941df/xrootd-4.9.0.tar.gz
Building wheels for collected packages: xrootd
Building wheel for xrootd (setup.py) ... error
Complete output from command /usr/bin/python3 -u -c "import
setuptools,
tokenize;__file__='/home.hdd/adrian/tmp/pip-install-fw69kbgt/xrootd/setup.py';f=getattr(tokenize,
'open', open)(__file__);code=f.read().replace('\r\n',
'\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d
/home.hdd/adrian/tmp/pip-wheel-fn58p_u9 --python-tag cp36:
[I] No git repository info found. Trying to interpret VERSION_INFO
[!] VERSION_INFO file invalid. Unable to determine the version. Using
"unknown"
/home/adrian/.local/lib/python2.7/site-packages/setuptools/dist.py:475:
UserWarning: Normalizing '4.9.0
' to '4.9.0'
normalized_version,
usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: -c --help [cmd1 cmd2 ...]
or: -c --help-commands
or: -c cmd --help
error: option --python-tag not recognized
----------------------------------------
Failed building wheel for xrootd
Running setup.py clean for xrootd
Failed to build xrootd
Installing collected packages: xrootd
Running setup.py install for xrootd ... done
Successfully installed xrootd
the thing is that on another machine, also fedora27, with the same
xrootd rpms installed, and in principle with the same python packages
installed in the user, i could install it successfully ...
Thank you!!
Adrian
>
> Cheers,
> Michal
> ________________________________________
> From: [log in to unmask] [[log in to unmask]] on behalf
> of Adrian Sevcenco [[log in to unmask]]
> Sent: 26 March 2019 13:19
> To: xrootd-l
> Subject: python xrootd :: CopyProcess batch size + multiple sources problem
>
> Hi! I want to use the CopyProcess to copy multiple files
> and i seen that all(?) copies are done in parralell with
> each copy with a default of parallelchunks=8
>
> I would have some questions :
> 1. how can i set the size of parralel copies?
> if i add 100 copy jobs will all run in parralel?
>
> 2. for each file i have multiple sources ..
> is there a way to download the file using a list of
> sources?
>
> or shoudld just add to CopyProcess all sources with the same destination
> name and with the default force=False all subseqvent copy jobs will just
> fail?
> but in this case i do not want the parralel download feature of
> CopyProcess, do i?
>
> Thank you for any ideas!
> Adrian
>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
