#ceph IRC Log

Index

IRC Log for 2012-02-09

Timestamps are in GMT/BST.

[0:09] * aa (~aa@r200-40-114-26.ae-static.anteldata.net.uy) Quit (Remote host closed the connection)
[0:23] * vodka (~paper@85.Red-83-41-151.dynamicIP.rima-tde.net) Quit (Quit: Leaving)
[0:53] * adjohn is now known as Guest1977
[0:53] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) has joined #ceph
[0:54] * Guest1977 (~adjohn@rackspacesf.static.monkeybrains.net) Quit (Read error: Connection reset by peer)
[0:55] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Quit: Leaving.)
[0:57] * joao (~joao@89-181-154-123.net.novis.pt) Quit (Quit: joao)
[0:58] * MarkDude (~MT@sjc-static-208.57.178.24.mpowercom.net) has joined #ceph
[0:59] * al_ (d@niel.cx) Quit (Remote host closed the connection)
[0:59] * al_ (quassel@niel.cx) has joined #ceph
[1:01] * al_ (quassel@niel.cx) Quit (Remote host closed the connection)
[1:01] * al_ (quassel@niel.cx) has joined #ceph
[1:09] * al_ (quassel@niel.cx) Quit (Remote host closed the connection)
[1:10] * al_ (quassel@niel.cx) has joined #ceph
[1:11] * al_ (quassel@niel.cx) Quit ()
[1:11] * al_ (quassel@niel.cx) has joined #ceph
[1:25] * al_ (quassel@niel.cx) Quit (Remote host closed the connection)
[1:25] * al_ (quassel@niel.cx) has joined #ceph
[1:27] * al_ (quassel@niel.cx) Quit ()
[1:27] * al_ (d@niel.cx) has joined #ceph
[1:29] * yoshi (~yoshi@p8031-ipngn2701marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[1:44] * Tv|work (~Tv|work@aon.hq.newdream.net) Quit (Ping timeout: 480 seconds)
[1:58] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) Quit (Remote host closed the connection)
[1:59] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) has joined #ceph
[2:19] * izdubar (~MT@sjc-static-208.57.178.24.mpowercom.net) has joined #ceph
[2:22] * izdubar (~MT@sjc-static-208.57.178.24.mpowercom.net) Quit ()
[2:33] * bchrisman (~Adium@108.60.121.114) Quit (Quit: Leaving.)
[2:52] * MarkDude (~MT@sjc-static-208.57.178.24.mpowercom.net) Quit (Ping timeout: 480 seconds)
[3:30] * adjohn is now known as Guest1985
[3:30] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) has joined #ceph
[3:31] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) Quit ()
[3:32] * Guest1985 (~adjohn@rackspacesf.static.monkeybrains.net) Quit (Read error: Operation timed out)
[3:34] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) has joined #ceph
[3:35] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) Quit ()
[3:46] * dmick (~dmick@aon.hq.newdream.net) Quit (Quit: Leaving.)
[4:26] * Mike (~Mike@awlaptop1.esc.auckland.ac.nz) has joined #ceph
[4:26] * Mike is now known as Guest1990
[4:26] <Guest1990> Hi all, am having some mount issues with ceph
[4:26] <Guest1990> I had to manually rebuild the btrfs on two of the OSDs to get them back in the cluster
[4:26] <Guest1990> This is now working and ceph is healthy
[4:26] <Guest1990> But I can't mount...
[4:27] <Guest1990> mount error 5 = Input/output error
[4:27] <Guest1990> Any ideas?
[4:27] * joshd (~joshd@aon.hq.newdream.net) Quit (Quit: Leaving.)
[4:27] <Guest1990> Here is the ceph status
[4:27] <Guest1990> lp0:~/ogg # ceph -s
[4:27] <Guest1990> 2012-02-09 16:24:36.337850 pg v33030: 2376 pgs: 2376 active+clean; 35702 MB data, 96693 MB used, 22067 GB / 22340 GB avail
[4:27] <Guest1990> 2012-02-09 16:24:36.353389 mds e37: 1/1/1 up {0=1=up:replay}, 1 up:standby
[4:27] <Guest1990> 2012-02-09 16:24:36.353524 osd e490: 12 osds: 12 up, 12 in
[4:27] <Guest1990> 2012-02-09 16:24:36.353841 log 2012-02-09 16:23:29.448276 osd9 10.19.99.122:6811/20116 1505 : [INF] 1.76 scrub ok
[4:27] <Guest1990> 2012-02-09 16:24:36.354085 mon e1: 2 mons at {0=10.19.99.123:6789/0,1=10.19.99.124:6789/0}
[5:33] * Meths (rift@2.25.213.150) Quit (Ping timeout: 480 seconds)
[5:37] * axisys (~axisys@ip68-98-189-233.dc.dc.cox.net) has left #ceph
[6:00] * chutzpah (~chutz@216.174.109.254) Quit (Quit: Leaving)
[6:05] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) has joined #ceph
[6:11] <ajm> your mds'es are up:replay not up:active
[6:11] <ajm> check the mds log, is it stuck or restarting
[7:08] * MarkDude (~MT@c-71-198-138-155.hsd1.ca.comcast.net) has joined #ceph
[7:21] * MarkDude (~MT@c-71-198-138-155.hsd1.ca.comcast.net) Quit (Quit: Leaving)
[8:30] * Guest1990 (~Mike@awlaptop1.esc.auckland.ac.nz) Quit (Read error: Connection reset by peer)
[8:31] * Mike__ (~Mike@awlaptop1.esc.auckland.ac.nz) has joined #ceph
[8:56] * The_Bishop (~bishop@cable-89-16-138-109.cust.telecolumbus.net) Quit (Quit: Wer zum Teufel ist dieser Peer? Wenn ich den erwische dann werde ich ihm mal die Verbindung resetten!)
[9:15] * vodka (~paper@85.Red-83-41-151.dynamicIP.rima-tde.net) has joined #ceph
[9:21] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[9:33] * fghaas (~florian@85-127-86-65.dynamic.xdsl-line.inode.at) has joined #ceph
[9:34] * fghaas (~florian@85-127-86-65.dynamic.xdsl-line.inode.at) Quit ()
[9:36] <Mike__> Thanks, will check the log
[9:37] <Mike__> Was previously Guest1990
[9:38] * al_ (d@niel.cx) Quit (Remote host closed the connection)
[9:38] * al_ (d@niel.cx) has joined #ceph
[10:00] * fronlius (~fronlius@e176053242.adsl.alicedsl.de) has joined #ceph
[10:05] * yoshi (~yoshi@p8031-ipngn2701marunouchi.tokyo.ocn.ne.jp) Quit (Remote host closed the connection)
[10:23] * fronlius (~fronlius@e176053242.adsl.alicedsl.de) Quit (Quit: fronlius)
[10:31] * fghaas (~florian@85-127-86-65.dynamic.xdsl-line.inode.at) has joined #ceph
[10:33] * __jt__ (~james@jamestaylor.org) Quit (Read error: Operation timed out)
[10:34] * __jt__ (~james@jamestaylor.org) has joined #ceph
[10:46] * andret (~andre@pcandre.nine.ch) Quit (Remote host closed the connection)
[10:47] * andret (~andre@pcandre.nine.ch) has joined #ceph
[10:47] * andret (~andre@pcandre.nine.ch) Quit (Remote host closed the connection)
[10:57] * vodka (~paper@85.Red-83-41-151.dynamicIP.rima-tde.net) Quit (Quit: Leaving)
[11:05] * andret (~andre@pcandre.nine.ch) has joined #ceph
[11:09] * joao (~joao@89.181.154.123) has joined #ceph
[11:32] * fronlius (~fronlius@e176053242.adsl.alicedsl.de) has joined #ceph
[12:03] * mtk (~mtk@ool-44c35967.dyn.optonline.net) Quit (Remote host closed the connection)
[13:02] * lollercaust (~paper@85.Red-83-41-151.dynamicIP.rima-tde.net) has joined #ceph
[13:42] * MarkDude (~MT@c-71-198-138-155.hsd1.ca.comcast.net) has joined #ceph
[13:42] * izdubar (~MT@c-71-198-138-155.hsd1.ca.comcast.net) has joined #ceph
[13:43] * MarkDude (~MT@c-71-198-138-155.hsd1.ca.comcast.net) Quit ()
[13:43] * izdubar (~MT@c-71-198-138-155.hsd1.ca.comcast.net) Quit ()
[13:44] * MarkDude (~MT@c-71-198-138-155.hsd1.ca.comcast.net) has joined #ceph
[13:45] * MarkDude (~MT@c-71-198-138-155.hsd1.ca.comcast.net) Quit ()
[14:41] * nhorman (~nhorman@nat-pool-rdu.redhat.com) has joined #ceph
[15:00] * mtk (~mtk@ool-44c35967.dyn.optonline.net) has joined #ceph
[15:06] * fghaas (~florian@85-127-86-65.dynamic.xdsl-line.inode.at) Quit (Read error: No route to host)
[15:07] * fghaas (~florian@85-127-86-65.dynamic.xdsl-line.inode.at) has joined #ceph
[15:12] * lollercaust (~paper@85.Red-83-41-151.dynamicIP.rima-tde.net) Quit (Quit: Leaving)
[15:40] <jdwilson> would it be a bad idea to move data around between ceph pools? specifically i'd like to be able to copy stuff from radosgw to rbd storage directly rather than having to download it via the gateway interface into a filesystem mount of ceph...
[15:51] * vodka (~paper@85.Red-83-41-151.dynamicIP.rima-tde.net) has joined #ceph
[16:00] * jeffhung (~jeffhung@60-250-103-120.HINET-IP.hinet.net) Quit (Read error: Connection reset by peer)
[16:16] * vodka (~paper@85.Red-83-41-151.dynamicIP.rima-tde.net) Quit (Quit: Leaving)
[16:18] * vodka (~paper@85.Red-83-41-151.dynamicIP.rima-tde.net) has joined #ceph
[16:19] * vodka (~paper@85.Red-83-41-151.dynamicIP.rima-tde.net) Quit (Max SendQ exceeded)
[16:21] * vodka (~paper@85.Red-83-41-151.dynamicIP.rima-tde.net) has joined #ceph
[17:22] * aa (~aa@r200-40-114-26.ae-static.anteldata.net.uy) has joined #ceph
[17:24] * vodka (~paper@85.Red-83-41-151.dynamicIP.rima-tde.net) Quit (Read error: Connection reset by peer)
[17:32] * vodka (~paper@85.Red-83-41-151.dynamicIP.rima-tde.net) has joined #ceph
[17:45] <jdwilson> how can i delete failed radosgw multipart uploads? it seems that they final key doesn't exist until the upload is completed and the objects listed in rados seem to have a tab character in them so i can't figure out how to delete them
[17:59] * fghaas (~florian@85-127-86-65.dynamic.xdsl-line.inode.at) Quit (Ping timeout: 480 seconds)
[18:10] * vodka (~paper@85.Red-83-41-151.dynamicIP.rima-tde.net) Quit (Quit: Leaving)
[18:15] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Ping timeout: 480 seconds)
[18:16] * Tv|work (~Tv|work@aon.hq.newdream.net) has joined #ceph
[18:18] * The_Bishop (~bishop@e177091002.adsl.alicedsl.de) has joined #ceph
[18:29] * vodka (~paper@85.Red-83-41-151.dynamicIP.rima-tde.net) has joined #ceph
[18:40] * fronlius_ (~fronlius@testing78.jimdo-server.com) has joined #ceph
[18:46] * fronlius (~fronlius@e176053242.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[18:46] * fronlius_ is now known as fronlius
[18:47] <gregaf1> jdwilson: failed uploads can be deleted with the intent log trimmer; it's part of radosgw-admin and should be run on a regular basis to clean up any "lost" objects"
[18:48] <gregaf1> there's not yet a good way to copy stuff between pools since copying between pools requires network transit anyway, and would open up a big can of worms in terms of how the actual transfer is completed
[18:54] * joshd (~joshd@aon.hq.newdream.net) has joined #ceph
[18:56] * fghaas (~florian@85-127-86-65.dynamic.xdsl-line.inode.at) has joined #ceph
[18:57] * vodka (~paper@85.Red-83-41-151.dynamicIP.rima-tde.net) Quit (Quit: Leaving)
[18:59] <yehudasa_> jdwilson: there's a cancel multipart upload api call
[19:09] <joshd> jmlowe1: your problem with pgs stuck in backfill should be fixed by f0334673ab8547807b961aae19a8e53531585e3f
[19:12] <jmlowe1> awesome
[19:13] <jmlowe1> afaik my raid controller problem from 3.2.1 goes away in 3.2.5
[19:14] <jmlowe1> and is f0334673ab8547807b961aae19a8e53531585e3f slated for 0.42?
[19:14] * fronlius (~fronlius@testing78.jimdo-server.com) Quit (Ping timeout: 480 seconds)
[19:15] * chutzpah (~chutz@216.174.109.254) has joined #ceph
[19:28] * dmick (~dmick@aon.hq.newdream.net) has joined #ceph
[19:30] <jdwilson> yehudasa_: sometimes calling (python boto) cancel_upload() works and sometimes it does not. but it actually /never/ works if, as i do on s3, i loop over bucket.list_multipart_uploads() and call cancel_upload() on each upload ... i see the problem too: the key name is preceded by '_multipart_' which does not exist
[19:31] <jdwilson> gregaf1: thanks! i'll checkout the log trimmer
[19:31] <joshd> jmlowe: yeah, it should be in 0.42
[19:33] * nhorman (~nhorman@nat-pool-rdu.redhat.com) Quit (Quit: Leaving)
[19:34] * fghaas (~florian@85-127-86-65.dynamic.xdsl-line.inode.at) Quit (Ping timeout: 480 seconds)
[19:55] <yehudasa_> jdwilson: you mean that when you list the pending uploads it shows a key preceded with _multipart_?
[19:56] <jdwilson> yes
[19:56] <yehudasa_> jdwilson: that sounds like a bug
[19:56] <yehudasa_> jdwilson: if you trim that prefix, does it cancel the upload?
[19:58] <yehudasa_> jdwilson: issue #2048
[20:00] * ninkotech_lite (~dp@ip-85-160-202-232.eurotel.cz) has joined #ceph
[20:01] <jdwilson> trimming it does let me cancel most of the time ... if i haven't uploaded any parts it still fails
[20:01] * joao (~joao@89.181.154.123) Quit (Quit: joao)
[20:02] <yehudasa_> jdwilson: what version are you running?
[20:03] <yehudasa_> I wouldn't expect trimming to do anything with it, it's not a temporary object
[20:06] <jdwilson> 0.41
[20:11] <jdwilson> also, the first part i try to upload always gives me an error response -- I get a 400 bad request response that says BadDigest ... if i try uploading the first part again it works fine
[20:11] <jdwilson> since it's recoverable i don't really care, but it is 100% of the time
[20:20] * nhorman (~nhorman@99-127-245-201.lightspeed.rlghnc.sbcglobal.net) has joined #ceph
[20:25] <yehudasa_> jdwilson: if you could send your rgw log it'd be helpful
[20:29] <nhm> jmlowe: glad to hear that your raid drivers got fixed!
[20:31] <jmlowe1> yeah, that was troubling, I'm guessing they put in some sort of broken irq throttling
[20:32] * Meths (rift@2.25.214.237) has joined #ceph
[20:44] * fghaas (~florian@85-127-86-65.dynamic.xdsl-line.inode.at) has joined #ceph
[20:53] * al_ (d@niel.cx) Quit (Remote host closed the connection)
[20:54] * al_ (quassel@niel.cx) has joined #ceph
[21:07] * al_ (quassel@niel.cx) Quit (Remote host closed the connection)
[21:07] * al_ (quassel@niel.cx) has joined #ceph
[21:07] * al_ (quassel@niel.cx) Quit (Remote host closed the connection)
[21:07] * al_ (quassel@niel.cx) has joined #ceph
[21:24] * mosu001 (~mosu001@awlaptop1.esc.auckland.ac.nz) has joined #ceph
[21:24] <mosu001> I have been trying to get my ceph system
[21:24] <mosu001> back up and running for a while now
[21:24] <mosu001> I have rebuilt some OSDs and they have rejoined
[21:25] <mosu001> but now one of my MDSs seems stuck on replay
[21:25] <mosu001> ss4:/cephlog # ceph -s
[21:25] <mosu001> 2012-02-10 09:25:20.608070 pg v33209: 2376 pgs: 2376 active+clean; 35702 MB data, 96728 MB used, 22066 GB / 22340 GB avail
[21:25] <mosu001> 2012-02-10 09:25:20.613644 mds e64: 1/1/1 up {0=1=up:replay}, 1 up:standby
[21:25] <mosu001> 2012-02-10 09:25:20.613698 osd e527: 12 osds: 12 up, 12 in
[21:25] <mosu001> 2012-02-10 09:25:20.613824 log 2012-02-10 09:20:20.054139 mon0 10.19.99.123:6789/0 16 : [INF] mds? 10.19.99.123:6800/20597 up:boot
[21:25] <mosu001> 2012-02-10 09:25:20.613945 mon e1: 2 mons at {0=10.19.99.123:6789/0,1=10.19.99.124:6789/0}
[21:25] <mosu001> Any suggestions?
[21:26] <mosu001> I have tried restarting cmds on both my MDS servers, but no change...
[21:30] <mosu001> I,m not sure where to look to find the steps to restart MDS servers nicely
[21:52] <jdwilson> yehudasa_: how do i turn on rgw logging? i've got a /var/log/radosgw dir but no files in it
[22:03] * nhorman (~nhorman@99-127-245-201.lightspeed.rlghnc.sbcglobal.net) Quit (Quit: Leaving)
[22:19] * lollercaust (~paper@85.Red-83-41-151.dynamicIP.rima-tde.net) has joined #ceph
[22:22] * fghaas (~florian@85-127-86-65.dynamic.xdsl-line.inode.at) Quit (Ping timeout: 480 seconds)
[22:54] * verwilst (~verwilst@d51A5B5DF.access.telenet.be) has joined #ceph
[23:13] * Mike__ (~Mike@awlaptop1.esc.auckland.ac.nz) Quit (Quit: Leaving)
[23:51] * fronlius (~fronlius@f054184068.adsl.alicedsl.de) has joined #ceph
[23:59] * fronlius (~fronlius@f054184068.adsl.alicedsl.de) Quit (Quit: fronlius)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.