cannot attach a volume to an instance

This is an all-in-one install. cloud-debug excerpt:

20:39:22 DEBUG [Context:New I/O server worker #1-5] :1281400762553:Context:CONTEXT_CREATE:5b2eaee5-20f1-40e8-a75b-4e22159a5962:[id: 0x7d26c5d4, /10.55.55.2:51634 => /10.55.55.2:8773]:NioHttpDecoder.decode.129
20:39:22 DEBUG [TimedEvictionSet:New I/O server worker #1-5] Evicting previous entry: FMd6P2GxVBMlg7PpCHqRt1/5x6OOUaW3TQubZSzsRdQ 5940707576109
20:39:22 DEBUG [Context:New I/O server worker #1-5] :1281400762573:Context:CONTEXT_USER:5b2eaee5-20f1-40e8-a75b-4e22159a5962:admin:NativeMethodAccessorImpl.invoke0.-2
20:39:22 DEBUG [Context:New I/O server worker #1-5] :1281400762574:Context:CONTEXT_SUBJECT:5b2eaee5-20f1-40e8-a75b-4e22159a5962:[DatabaseWrappedUser [ user = UserEntity [ administrator = true, enabled = true, name = admin, password = 21232f297a57a5a743894a0e4a801fc3, queryId = WKy3rMzOWPouVOxK1p3Ar1C2uRBwa2FBXnCw, s
ecretKey = DteXlXLRBPjDIVWaAQQ7nWaGiPjbamP1K6FIsA, token = YR7SQsocoiC5JH4C7ltUOPMOdtgY36GnkbSsiXrejIFCoS9N6at9hXVMkFVdQ7bk2nWJBdTC009jFdX1Tg, certificates = 33908414070706 null LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURGVENDQWYyZ0F3SUJBZ0lHSHRicStiK3lNQTBHQ1NxR1NJYjNEUUVCRFFVQU1FRXhDekFKQmdOVkJBWVQKQWxWVE1RMHdDd1l
EVlFRS0V3UlZjMlZ5TVJNd0VRWURWUVFMRXdwRmRXTmhiSGx3ZEhWek1RNHdEQVlEVlFRRApFd1ZoWkcxcGJqQWVGdzB4TURBNE1Ea3lNekk0TkRWYUZ3MHhOVEE0TURreU16STRORFZhTUVFeEN6QUpCZ05WCkJBWVRBbFZUTVEwd0N3WURWUVFLRXdSVmMyVnlNUk13RVFZRFZRUUxFd3BGZFdOaGJIbHdkSFZ6TVE0d0RBWUQKVlFRREV3VmhaRzFwYmpDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dn
RUJBS0tvWG9pYgp6cU5FdjJ6QVNQd1BDN1JtMGpDcEl0SEd1dStObm9tQlpuKzZ0WmhGVy9CdEJFRWZEYy92aERmcm9mZ0FONDBJCkRJejE3UW8wVXVyeEFlaXN3czVadVoySHNFQnlGdUZFSHlPSnk4aXVqMVBYbmRza3BSOHRxTzZCK0JhT1RhVnAKN0dVVmtRL09hMlZvWk5TWVR2Ti9NbU5Ud2JRSklIVUl1Y2VqUXRwbGFOZDdiQzB4QW03OEVvTkhqNHVRQVdMeAp2dmFFVWZsRXV0UTAzTGlaUityekFPNTFqVjgxakhLR
HVOb0FFdTR1QTBiajFGZ0k4V25ibUhORXJrVVFGazZHCnQ0Z1FwbVlRaE9hQ3d0aDYyWVJ6UjFia0hUT0JpUjNZL25QN0hDRFJ1bm9HOFBPSXVieEhMbjJZbXlDNTdkNFMKNWYxcGFvckRnekZWd1YwQ0F3RUFBYU1UTUJFd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBTkJna3Foa2lHOXcwQgpBUTBGQUFPQ0FRRUFNTVRoekNiS0pWZmgycTRuNDQ4SHNaa2ROZlNnRGJKZzJEY0FONXVpL2dPdjRCMG1BODdxCjJxVFlaSXQza0
9zcTM5OURBUzdHRS9KcWgwODMvSjgzTWt6TjY5QURrS3NzV2Z6SmExR0JvTzRYU2Y3R1g5RUoKaEI5ZlhpUnV3ZlVBV3RCTW01aVd6UjlWbFh6a0pqT0VtTUZYUWlsTXB3cmtCeWM2Vm5DMk96LzhnczBqbjc2OApsY0xGblFlWTR6TU90WXlEZk1GMFdrWGtlL3E3dHBoNjZHY3hFMFBINXNodElpdDVRMDJkTjJYWTdxTHlFU1I1CjFPdVhFWHdTWlFjemtPdE9mekl5MFVaVmdGRHRhSHQxeCtxTndZeVBCQmxzZVROK2JVbGl
vRjgyblFTdFZxOVMKT2M3dEpBZUEvbERycm9WVm8yU2p3b0IrQ3JhODJKQjBJQT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K, eucaGroupIds = ], ], com.eucalyptus.auth.DatabaseWrappedGroup@423dca93]:NativeMethodAccessorImpl.invoke0.-2
20:39:22 DEBUG [RequestQueue:RequestQueue.2] :1281400762600:RequestQueue:5b2eaee5-20f1-40e8-a75b-4e22159a5962:admin:MSG_RECEIVED:5b2eaee5-20f1-40e8-a75b-4e22159a5962:CreateVolumeType:NativeMethodAccessorImpl.invoke0.-2
20:39:22 INFO [VolumeManager:Volume.2] :1281400762711:VolumeManager:5b2eaee5-20f1-40e8-a75b-4e22159a5962:admin:VOLUME_CREATE:user=admin:volume=vol-5A280632:size=1:cluster=UEC-TEST1:snapshot=null:NativeMethodAccessorImpl.invoke0.-2
20:39:22 DEBUG [SystemUtil:pool-8-thread-1] Running comma...

ubuntu@cempedak:/var/log/eucalyptus$ sudo losetup -a
/dev/loop0: [6801]:3146674 (/var/lib/eucalyptus/volumes/vol-5A280632)
ubuntu@cempedak:/var/log/eucalyptus$ sudo vg
vgcfgbackup vgcfgrestore vgchange vgck vgconvert vgcreate vgdisplay vgexport vgextend vgimport vgimportclone vgmerge vgmknodes vgreduce vgremove vgrename vgs vgscan vgsplit
ubuntu@cempedak:/var/log/eucalyptus$ sudo vgscan
  Reading all physical volumes. This may take a while...
  Found volume group "vg-E1rO0g.." using metadata type lvm2
ubuntu@cempedak:/var/log/eucalyptus$ sudo lvscan
  ACTIVE '/dev/vg-E1rO0g../lv-rNkFoQ..' [1.00 GiB] inherit
ubuntu@cempedak:/var/log/eucalyptus$

Setting Importance to critical -- this has to be looked at before feature freeze, and it is a major issue on using the cloud.

Actually HIGH. No need to poke the release team...

With r1224, still a problem. We *did* advance a bit:

18:39:41 INFO [VolumeManager:Volume.16] :1281566381374:VolumeManager:89a38b73-8783-4376-a403-fa0c569f193f:admin:VOLUME_CREATE:user=admin:volume=vol-5A010635:size=1:cluster=UEC-TEST1:snapshot=null:NativeMethodAccessorImpl.invoke0.-2
18:39:41 DEBUG [SystemUtil:pool-8-thread-1] Running command: ///usr/lib/eucalyptus/euca_rootwrap dd if=/dev/zero of=//var/lib/eucalyptus/volumes/vol-5A010635 count=1 bs=1M seek=1027
18:39:41 INFO [OverlayManager:pool-8-thread-1] losetup /dev/loop0 //var/lib/eucalyptus/volumes/vol-5A010635
18:40:11 DEBUG [QueuedEventCallback:Volume.16] :1281566411345:QueuedEventCallback:bc6d011b-594b-4150-a9c3-485998fc03d4:admin:QUEUE:class edu.ucsb.eucalyptus.msgs.AttachVolumeType:[AttachVolumeType volumeId=vol-5A010635 instanceId=i-48AF0842 device=/dev/sdb remoteDevice=//,10.55.55.2,iqn.2009-06.com.eucalyptus.UEC-TEST1:store1,AG0f08WwmtrqkqS8+fHiasNQKaWqp3Rbgv9NqSfdWfbEu6RX5UFk9OOE5gGKNlj68/1h3dXXXRYIv3kTtNRTBw+nl0l3M0owNm9CWU0+IMmjbitjdRfvjQkTZg5Zz/eK7PjSnoGbRHKrwvnC/FH82rAuf6enq8tTeM+EdVXkeLKC2KGcrCIGR7V6ec1Snzcu33KClX5+A7DG24V7DEe1Jwos/zES1ciOJaS9K6XbiozjtXIhj7zu9jNcj6twS3uZuHzveCZcK9ZmWJygnELDNvg0pRy5eZ/o+6zzsj/Pk6G0wM6tLnLNs828I0h0uOHpZjOUhUoTGkjG4r2C/CY8wg== correlationId=bc6d011b-594b-4150-a9c3-485998fc03d4 userId=admin effectiveUserId=eucalyptus _return=false statusMessage=null]:NativeMethodAccessorImpl.invoke0.-2
18:40:11 INFO [VolumeManager:Volume.16] :1281566411346:VolumeManager:bc6d011b-594b-4150-a9c3-485998fc03d4:admin:VOLUME_ATTACH:user=admin:volume=vol-5A010635:instance=i-48AF0842:cluster=UEC-TEST1:NativeMethodAccessorImpl.invoke0.-2
  <volumeId>vol-5A010635</volumeId>
18:40:11 DEBUG [VolumeAttachCallback:New I/O client worker #2-7] Trying to remove invalid volume attachment vol-5A010635 from instance i-48AF0842
18:40:11 DEBUG [VolumeAttachCallback:New I/O client worker #2-7] Removed failed attachment: vol-5A010635 -> i-48AF0842
  <volumeId>vol-5A010635</volumeId>
18:40:11 DEBUG [VolumeAttachCallback:New I/O client worker #2-7] Trying to remove invalid volume attachment vol-5A010635 from instance i-48AF0842
18:40:11 DEBUG [VolumeAttachCallback:New I/O client worker #2-7] Removed failed attachment: vol-5A010635 -> i-48AF0842

The problem is that the iscsi related commands and scripts are not listed in the /etc/eucalyptus/wrappers.conf on NC. I guess there is some misunderstanding between both our sides. We assume you know that all the original sudoers config needs to be moved into wrappers.conf. But obviously this information was not conveyed clearly.

So, here is the stuff you need to add to the wrappers.conf on NC:

get_iscsitarget.pl /usr/share/eucalyptus/get_iscsitarget.pl 0
connect_iscsitarget.pl /usr/share/eucalyptus/connect_iscsitarget.pl 0
disconnect_iscsitarget.pl /usr/share/eucalyptus/disconnect_iscsitarget.pl 0

And put the following in wrappers.conf on SC:

tgtadm /usr/sbin/tgtadm 0

Or maybe just list the 4 items in all wrappers.conf for easiness.

Also, remember to attach a device using "euca-attach-volume ... -d /dev/vdx" since SCSI hotplug is not working on Maverick anymore.

I have successfully tested the process in our Maverick daily build installation. Please give it try again by following the procedure I listed above and let us know if you still have problems with EBS attaching.

Given there is no work from Eucalyptus side, and all you have to do is a change in the config, I'm closing this bug as fixed now. Please reopen it if there is still problem. But please open another bug if the problem is different.

@Ye, Thanks for that update.. i'll get that integrated now!

This bug was fixed in the package eucalyptus - 2.0~bzr1230-0ubuntu1

---------------
eucalyptus (2.0~bzr1230-0ubuntu1) maverick; urgency=low

  * New upstream merge, r1230 - bug fixes:
    - (LP: #617127, #611769, #617032, #613033, #613033, #584497)
  * debian/patches/18-priv_security.patch: Added *_iscsitarget.pl scripts to
    the wrappers.conf whitelist. (LP: #615646)
  * debian/patches/15-fix-default-ramdisk.patch: Dropped, fixed applied upstream.
 -- Dave Walker (Daviey) <email address hidden> Fri, 13 Aug 2010 11:25:25 +0100

Hi, I installed a maverick UEC. I have 1 system with clc,cc,sc,walrus and another with nc. when attach a volume to a running instance no error is show but a can't see the second disk with fdisk -l. I verify wrappers.conf file and everything is ok. I need to open a new bug?

las line of cc.log

print_ccInstance(): refresh_instances(): instanceId=i-402607EE reservationId=r-3F800688 emiId=emi-DF71106F kernelId=eki-F5B510F4 ramdiskId=eri-0A2C115D emiURL=http://192.168.3.252:8773/services/Walrus/image-store-1295465260/image.manifest.xml kernelURL=http://192.168.3.252:8773/services/Walrus/image-store-1295465260/kernel.manifest.xml ramdiskURL=http://192.168.3.252:8773/services/Walrus/image-store-1295465260/ramdisk.manifest.xml state=Extant ts=1295546050 ownerId=fabramowitz keyName=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyso3nDkuOJX2S+g6Av2npyM10LNemRWR2CLRmLYNguLHxsAQznATaQqgGXaxV/now4vY22t++ARMSEW3eQM5wgHbawIbRm40JMzU021nzsE1lI9SF9whQkzGn7FAFI6wezRV35f2PA/10t05WanE8zn31unUo1aO5WExiWoeNDSNfD19wBpHLk0tG3ZLs3kxPhYK+qfVSLJdNmztYIRPoJsDdFlIogoXycDb2iy/pq0HbO6E8MRFa61VNRTs+5HdZCeQsTY/AZG3nmk5vlOIAntZ2IV3pvfwNVzBZncEE4rb4Qd5pIwJvBrpnF1Y91NDmStAuH9ddrepH1lOgdsAB fabramowitz@eucalyptus ccnet={privateIp=172.19.1.2 publicIp=192.168.3.50 privateMac=D0:0D:40:26:07:EE vlan=10 networkIndex=2} ccvm={cores=2 mem=512 disk=10} ncHostIdx=0 serviceTag=http://10.98.98.2:8775/axis2/services/EucalyptusNC userData= launchIndex=0 volumesSize=1 volumes={vol-A37808CF,//,192.168.3.252,iqn.2009-06.com.eucalyptus.euca-flacso-cluster1:store5,W407KmAb1yQLkoICLzOglSfB69dfaqVG2ZVYsmucRmPz4/+Uggkr1iAwk8VKkrGs3mW1USBDnzG02/0/ic8SYxnRINYhP2Oq2s16V0OiaMsQm4Mx2wXOUdwzjn50KEfqo8UYAslEj++1T0WLpAqMQAs3G7CDgi/UBuYG8LlJptqL7b19liIfgG9ifwO3T8YUxqda2JNnWKtLDs2z/CTrMBt0R6sMGGVUzOxw6Uk/yll/2mfFLk6bql6cdkT+zPDG7xGNcHs64Xu8FLiWjjnrwCNNKvZ53rqCUbqZ4AIdr5FtYQXepUPazqpRptRfZZdr2B6AkqExsdd9GGCCxJx9AA==,unknown,requested:/dev/vdb,attached } groupNames={ssh }
[Thu Jan 20 22:21:32 2011][012488][EUCADEBUG ] refresh_instances(): done
[Thu Jan 20 22:21:32 2011][012488][EUCADEBUG ] monitor_thread(): done

Thanks!