From ATLAS-TRIUMF

[edit] Don Quijote (DQ2) to get files from the GRID

NOTE: If you are running on a machine in the ATLAS cluster at TRIUMF, you do not need to install gLite or DQ2 locally; just make sure you automount /atlas/ATLASLocalRootBase and do the following setup each time you want to use gLite in a login session:

[edit] Setup and DQ2 on ATLAS Cluster

export ATLAS_LOCAL_ROOT_BASE=/atlas/ATLASLocalRootBase
source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh
localSetupGLite (only needed if you are on a 64-bit machine, otherwise skip)
localSetupDQ2  (answer "yes")
voms-proxy-init -voms atlas:/atlas/ca

and then use it as explained below, but if you really want to do it yourself, the following has worked for me in the past:

[edit] Installing DQ2 End-User Tools

See CurrentStatusOfDQ2.

It is not necessary to install the whole of Don Quijote (DQ2) on the UI that you plan to use. Conceivably all you need is the setup script, but what I actually did (that works for me now) is the following:

Follow these instructions. The exciting bit is the innocent line "configure setup script". As far as I know, there is no yum / apt / pacman / rpm-like installation, so you have to keep these up-to-date by hand... Here are the variables that I had to change / add in the setup.sh.any that I am using on my UI at TRIUMF:

export DQ2_LOCAL_ID=TRIUMF
export DQ2_URL_LOCAL=http://vobox.triumf.ca:8000/dq2/
export DQ2_STORAGE_ROOT=  #(do we have a special mount point?)
export DQ2_SRM_HOST='srm://sc4.triumf.ca:8443/atlas' #(not at all sure about that)
export DQ2_GSIFTP_HOST= #(again, not so sure)
export LCG_CATALOG_TYPE=lfc
export DQ2_LFC_HOME=/grid/atlas

I add this line:

export LFC_HOST=lfc-atlas-test.cern.ch

I want to use my own version of the tools, not the one over afs at BNL:

# PATH
#export PATH=/afs/usatlas.bnl.gov/Grid/Don-Quijote/dq2_user_client:$PATH
#imt switch to local:
export PATH=$PATH:/data/donquijote

At one point I was using zsh, so I had to comment out the section:

## prevent the wild-card expansion
#_dq2_ls_helper () {
#  unalias dq2_ls 
#  dq2_ls $*
#  set +f
#  alias dq2_ls='set -f; _dq2_ls_helper'
#}
#alias dq2_ls='set -f; _dq2_ls_helper'

and replace it with:

alias dq2_ls='noglob dq2_ls'

Lately I have switched from zsh to bash, and find life generally easier.

[edit] Setting up locally installed DQ2

When I actually want to use this, here is what I do:

• open a login terminal
• setup gLite UI
• grid-proxy-init or voms-proxy-init -voms atlas:/atlas/ca
• source setup.sh.any

[edit] Using DQ2

having done that I am able to run commands like

• lfc-ls /grid/atlas/datafiles/mc11/digit/mc11.017702.Bs_Jpsi_mu6mu3_phi_KplusKminus.digit.RDO.v11004101/

Note that:

• dq2_ls mc11*

will list available files (if you picked the correct version of wildcard expansion for your shell), but this doesn't include data produced on LCG prior to DQ2 integration (before 11.0.42). However the dq2_get command will internally look in the lfc to list datasets, so that's OK. And I can use lfc-ls directly to check what is available, and then get the files with

• dq2_get -l -r -d /data/Simfiles mc11.004202.ZmumuJimmy.digit.RDO.v11000401

(where /data/Simfiles is the local directory where I want to put them).

--Isabel 08:34, 25 April 2006 (PDT)

[edit] Installing the full DQ2 client on my UI

For some fancier things, you may need the full DQ2 client. This installation finally seems to have worked: Official instructions Basically, you just copy the installation file and put it in the directory where you want to install dq2, and make it executable. Then you run it,

./install_dq2_client.sh

The only problem I had is that my user path has an alternate (LCG) python path defined. So I installed it from the root account, where python is /usr/bin/python, and it ran with no problems. Then from my user account (with a valid grid proxy) I was able to do

./dq2 unittest

and it returned:

Ran 21 tests in 336.381s
OK

None of the other things I tried really helped much, but for completeness I include them below...

The official instructions for downloading and installing Don Quijote are here.

My most recent attempt was for DQ2_0_2_3. I used wget to download the client tar file. I untarred it and cd'd to dq2-client-0.2.3. Then I ran into problems because I seemed to have several versions of python lying around and none of them was the right version... The one in my default PATH is /home/itrigger/LCG/edg/bin/python, which is version 2.2.3. Various athena releases use 2.3.4 and 2.4.2.

I managed to run the command python setup.py install dq2-client apparently successfully using 2.4.2 (borrowed from athena 11.4.0). But ./dq2 unittest then fails with a message "RuntimeWarning: Python C API version mismatch for module pycurl: This Python has API version 1012, module pycurl has version 1011."

Python 2.3.4 from athena release 11.3.0 also seems to install dq2 successfully, but unittest gives the same error.

If I leave the installation done with python 2.3.4, and try to run unittest with python 2.2.3, I get a different error,

File "./common/client/testcase/DQ2TestCase.py", line 35, in ? from common.client.testcase import client_conf
ImportError: cannot import name client_conf

If I set up a soft link in from common/client/tool/client_conf.py to common/client/testcase/client_conf.py and try again, I fail with

 File "./common/client/testcase/DQ2TestCase.py", line 36, in ? from common.testcase.DQTestCase import DQTestCase
 ImportError: No module named testcase.DQTestCase

Probably I should also edit client_conf.py to point to something other than atldq02.cern.ch, but what? I tried vobox.triumf.ca, but got the same error as above.

This is fairly frustrating... --Isabel 16:25, 12 April 2006 (PDT)

This is how far I got for DQ2_0_1_3:

I followed the official instructions, except that for step 1, where it says to use wget, I downloaded the four tar.gz files directly from this webpage linked from this one.

Since my path does not include the current directory for some reason, I have to do "./dq2" wherever the instructions say to do "dq2" (yes, I could change my default path...).

You have to do grid-proxy-init before you run any of the dq2 commands. The unittest takes quite a long time (1608 seconds for me). In my case, 5 of the 13 tests failed.

Then maybe we want to install local site services (not sure about that for my local workstation!). This is explained here. Be careful to edit everything that points to afs.

I gave up at this point; the documentation all points to something called dms3, which no longer seems to be available for download, while there do not appear to be detailed instructions yet for dq2, which I did manage to download and partially install.

However, note that on our grid UI machine, lcgui01, it is possible to do just grid-proxy-init, and then lcg-cp -v --vo atlas lfn:rome.004122.recov10NONT.Sherpa_ZJ1mumu._00001.AOD.pool.root file:///tmp/templocalfile.pool.root and there the logical file name works, and get traced (in this instance) to a location on castor at CERN.

--Isabel 14:08, 7 Sep 2005 (PDT)