#ceph IRC Log


IRC Log for 2012-01-27

Timestamps are in GMT/BST.

[0:00] <iggy> kirby_: I would think it'd be in the ceph dir
[0:03] <iggy> none of the branches look like they've been touched in months
[0:20] * joao (~joao@89-181-146-42.net.novis.pt) has joined #ceph
[0:21] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) Quit (Remote host closed the connection)
[0:21] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) has joined #ceph
[0:31] * aliguori (~anthony@ Quit (Remote host closed the connection)
[0:50] * aa (~aa@r200-40-114-26.ae-static.anteldata.net.uy) Quit (Remote host closed the connection)
[0:55] * _adjohn (~adjohn@rackspacesf.static.monkeybrains.net) has joined #ceph
[0:55] * adjohn is now known as Guest636
[0:55] * _adjohn is now known as adjohn
[0:55] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) Quit (Remote host closed the connection)
[0:56] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) has joined #ceph
[0:56] * Guest636 (~adjohn@rackspacesf.static.monkeybrains.net) Quit (Read error: Connection reset by peer)
[0:58] * aa (~aa@r200-40-114-26.ae-static.anteldata.net.uy) has joined #ceph
[1:21] * ceph (~hylick@ Quit (Quit: Leaving.)
[1:24] * izdubar (~MT@c-71-198-138-155.hsd1.ca.comcast.net) Quit (Quit: Leaving)
[1:27] * aa (~aa@r200-40-114-26.ae-static.anteldata.net.uy) Quit (Remote host closed the connection)
[1:31] * yoshi (~yoshi@p11133-ipngn3402marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[1:51] * joao (~joao@89-181-146-42.net.novis.pt) Quit (Quit: joao)
[2:26] * lollercaust (~paper@40.Red-88-19-215.staticIP.rima-tde.net) Quit (Quit: Leaving)
[2:36] * bchrisman (~Adium@ Quit (Quit: Leaving.)
[3:53] * gregphone (~gregphone@cpe-24-24-170-80.socal.res.rr.com) has joined #ceph
[3:54] <gregphone> kirby_: I'm afraid we aren't maintaining the backports right now due to lack of time, and 2.6.32 is old enough I'm not sure it ever worked... :(
[3:54] <gregphone> if you want to use ceph without a current kernel I recommend the FUSE client, which you should have already built and performs similarly :)
[3:55] <ajm> 2.6.32 definitely worked
[3:55] <ajm> not that well though
[3:55] * adjohn is now known as Guest645
[3:55] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) has joined #ceph
[3:55] * Guest645 (~adjohn@rackspacesf.static.monkeybrains.net) Quit (Read error: Connection reset by peer)
[4:04] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) has joined #ceph
[4:07] * chutzpah (~chutz@ Quit (Quit: Leaving)
[4:08] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) Quit (Remote host closed the connection)
[4:08] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) has joined #ceph
[4:28] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) Quit (Quit: adjohn)
[4:34] * gregphone (~gregphone@cpe-24-24-170-80.socal.res.rr.com) Quit (Quit: Rooms • iPhone IRC Client • http://www.roomsapp.mobi)
[4:59] * The_Bishop (~bishop@e179006230.adsl.alicedsl.de) has joined #ceph
[5:21] * adjohn (~adjohn@70-36-197-222.dsl.dynamic.sonic.net) has joined #ceph
[5:21] * adjohn (~adjohn@70-36-197-222.dsl.dynamic.sonic.net) Quit ()
[6:21] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) Quit (resistance.oftc.net synthon.oftc.net)
[6:21] * kirby_ (~kfiles@pool-151-199-52-228.bos.east.verizon.net) Quit (resistance.oftc.net synthon.oftc.net)
[6:21] * aneesh (~aneesh@ Quit (resistance.oftc.net synthon.oftc.net)
[6:21] * Meths (rift@ Quit (resistance.oftc.net synthon.oftc.net)
[6:21] * acaos (~zac@209-99-103-42.fwd.datafoundry.com) Quit (resistance.oftc.net synthon.oftc.net)
[6:21] * eternaleye___ (~eternaley@ Quit (resistance.oftc.net synthon.oftc.net)
[6:21] * jpieper_ (~josh@209-6-86-62.c3-0.smr-ubr2.sbo-smr.ma.cable.rcn.com) Quit (resistance.oftc.net synthon.oftc.net)
[6:21] * nolan (~nolan@phong.sigbus.net) Quit (resistance.oftc.net synthon.oftc.net)
[6:21] * mtk (~mtk@ool-44c35967.dyn.optonline.net) Quit (resistance.oftc.net synthon.oftc.net)
[6:21] * sage (~sage@cpe-76-94-40-34.socal.res.rr.com) Quit (resistance.oftc.net synthon.oftc.net)
[6:21] * yehudasa (~yehudasa@aon.hq.newdream.net) Quit (resistance.oftc.net synthon.oftc.net)
[6:21] * sjust (~sam@aon.hq.newdream.net) Quit (resistance.oftc.net synthon.oftc.net)
[6:21] * jeffhung (~jeffhung@60-250-103-120.HINET-IP.hinet.net) Quit (resistance.oftc.net synthon.oftc.net)
[6:21] * gohko (~gohko@natter.interq.or.jp) Quit (resistance.oftc.net synthon.oftc.net)
[6:21] * svenx_ (92744@diamant.ifi.uio.no) Quit (resistance.oftc.net synthon.oftc.net)
[6:21] * yoshi (~yoshi@p11133-ipngn3402marunouchi.tokyo.ocn.ne.jp) Quit (resistance.oftc.net synthon.oftc.net)
[6:21] * cclien (~cclien@ec2-50-112-123-234.us-west-2.compute.amazonaws.com) Quit (resistance.oftc.net synthon.oftc.net)
[6:21] * sagewk (~sage@aon.hq.newdream.net) Quit (resistance.oftc.net synthon.oftc.net)
[6:21] * u3q (~ben@uranus.tspigot.net) Quit (resistance.oftc.net synthon.oftc.net)
[6:21] * darkfader (~floh@ Quit (resistance.oftc.net synthon.oftc.net)
[6:21] * kirkland (~kirkland@ Quit (resistance.oftc.net synthon.oftc.net)
[6:21] * iggy (~iggy@theiggy.com) Quit (resistance.oftc.net synthon.oftc.net)
[6:21] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) has joined #ceph
[6:21] * yoshi (~yoshi@p11133-ipngn3402marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[6:21] * kirby_ (~kfiles@pool-151-199-52-228.bos.east.verizon.net) has joined #ceph
[6:21] * aneesh (~aneesh@ has joined #ceph
[6:21] * Meths (rift@ has joined #ceph
[6:21] * acaos (~zac@209-99-103-42.fwd.datafoundry.com) has joined #ceph
[6:21] * cclien (~cclien@ec2-50-112-123-234.us-west-2.compute.amazonaws.com) has joined #ceph
[6:21] * eternaleye___ (~eternaley@ has joined #ceph
[6:21] * jpieper_ (~josh@209-6-86-62.c3-0.smr-ubr2.sbo-smr.ma.cable.rcn.com) has joined #ceph
[6:21] * nolan (~nolan@phong.sigbus.net) has joined #ceph
[6:21] * mtk (~mtk@ool-44c35967.dyn.optonline.net) has joined #ceph
[6:21] * sage (~sage@cpe-76-94-40-34.socal.res.rr.com) has joined #ceph
[6:21] * svenx_ (92744@diamant.ifi.uio.no) has joined #ceph
[6:21] * yehudasa (~yehudasa@aon.hq.newdream.net) has joined #ceph
[6:21] * sjust (~sam@aon.hq.newdream.net) has joined #ceph
[6:21] * jeffhung (~jeffhung@60-250-103-120.HINET-IP.hinet.net) has joined #ceph
[6:21] * gohko (~gohko@natter.interq.or.jp) has joined #ceph
[6:21] * sagewk (~sage@aon.hq.newdream.net) has joined #ceph
[6:21] * darkfader (~floh@ has joined #ceph
[6:21] * iggy (~iggy@theiggy.com) has joined #ceph
[6:21] * u3q (~ben@uranus.tspigot.net) has joined #ceph
[6:21] * kirkland (~kirkland@ has joined #ceph
[7:34] * fghaas (~florian@85-127-92-127.dynamic.xdsl-line.inode.at) has joined #ceph
[8:06] * The_Bishop_ (~bishop@e179009037.adsl.alicedsl.de) has joined #ceph
[8:06] * fghaas (~florian@85-127-92-127.dynamic.xdsl-line.inode.at) Quit (Ping timeout: 480 seconds)
[8:13] * The_Bishop (~bishop@e179006230.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[8:30] * MarkDude (~MT@76-218-98-3.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[8:47] * izdubar (~MT@76-218-98-3.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[8:51] * MarkDud (~MT@76-218-98-3.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[8:52] * MarkDude (~MT@76-218-98-3.lightspeed.sntcca.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[8:57] * fghaas (~florian@85-127-92-127.dynamic.xdsl-line.inode.at) has joined #ceph
[8:58] * izdubar (~MT@76-218-98-3.lightspeed.sntcca.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[9:02] * MarkDude (~MT@76-218-98-3.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[9:06] * izdubar (~MT@76-218-98-3.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[9:07] * MarkDud (~MT@76-218-98-3.lightspeed.sntcca.sbcglobal.net) Quit (Read error: Operation timed out)
[9:12] * MarkDude (~MT@76-218-98-3.lightspeed.sntcca.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[9:14] * fronlius (~fronlius@testing78.jimdo-server.com) has joined #ceph
[9:17] * izdubar (~MT@76-218-98-3.lightspeed.sntcca.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[9:30] * verwilst (~verwilst@d51A5B145.access.telenet.be) has joined #ceph
[10:00] * yoshi (~yoshi@p11133-ipngn3402marunouchi.tokyo.ocn.ne.jp) Quit (Remote host closed the connection)
[10:23] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[10:57] * joao (~joao@ has joined #ceph
[11:43] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Quit: Leaving.)
[11:44] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[11:48] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit ()
[11:49] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[13:26] * fronlius (~fronlius@testing78.jimdo-server.com) Quit (synthon.oftc.net oxygen.oftc.net)
[13:26] * cclien (~cclien@ec2-50-112-123-234.us-west-2.compute.amazonaws.com) Quit (synthon.oftc.net oxygen.oftc.net)
[13:29] * fronlius (~fronlius@testing78.jimdo-server.com) has joined #ceph
[13:29] * cclien (~cclien@ec2-50-112-123-234.us-west-2.compute.amazonaws.com) has joined #ceph
[13:31] * kirkland (~kirkland@ Quit (synthon.oftc.net weber.oftc.net)
[13:31] * darkfader (~floh@ Quit (synthon.oftc.net weber.oftc.net)
[13:31] * sagewk (~sage@aon.hq.newdream.net) Quit (synthon.oftc.net weber.oftc.net)
[13:31] * iggy (~iggy@theiggy.com) Quit (synthon.oftc.net weber.oftc.net)
[13:31] * u3q (~ben@uranus.tspigot.net) Quit (synthon.oftc.net weber.oftc.net)
[13:40] * aliguori (~anthony@cpe-70-123-132-139.austin.res.rr.com) has joined #ceph
[13:43] * sagewk (~sage@aon.hq.newdream.net) has joined #ceph
[13:43] * darkfader (~floh@ has joined #ceph
[13:43] * iggy (~iggy@theiggy.com) has joined #ceph
[13:43] * kirkland (~kirkland@ has joined #ceph
[13:43] * u3q (~ben@uranus.tspigot.net) has joined #ceph
[14:03] * fghaas (~florian@85-127-92-127.dynamic.xdsl-line.inode.at) Quit (Read error: No route to host)
[14:12] * Kioob (~kioob@ has joined #ceph
[15:47] * lollercaust (~paper@40.Red-88-19-215.staticIP.rima-tde.net) has joined #ceph
[15:47] * grape_ (~grape@ has joined #ceph
[15:49] * grape (~grape@ Quit (Ping timeout: 480 seconds)
[15:55] * Kioob (~kioob@ Quit (Quit: Leaving.)
[16:13] <wido> hi
[16:14] <wido> http://pastebin.com/95wGPzXP is this one known yet? Didn't see anything in the tracker
[16:14] <wido> I upgraded from 0.39 to 0.40 and everything started crashing
[16:17] * Habitual (~JohnJones@dynamic-acs-24-101-150-38.zoominternet.net) has joined #ceph
[16:19] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Quit: Leaving.)
[16:35] * aa (~aa@r200-40-114-26.ae-static.anteldata.net.uy) has joined #ceph
[16:38] * aa (~aa@r200-40-114-26.ae-static.anteldata.net.uy) Quit (Read error: Connection reset by peer)
[16:39] * aa (~aa@r200-40-114-26.ae-static.anteldata.net.uy) has joined #ceph
[17:08] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[17:14] * ceph (~hylick@ has joined #ceph
[17:15] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[17:17] * verwilst (~verwilst@d51A5B145.access.telenet.be) Quit (Quit: Ex-Chat)
[17:42] <Habitual> Hello #ceph. I am following some recent instructions from http://ceph.newdream.net/w/index.php?title=User:Grape&oldid=2989 and I get the GPG 'error' where it added the key but apt-get update barfs. Is the Ubu11.10 stock repo for ceph okay to use?
[17:50] <Habitual> do while true ; http://search.gmane.org/?query=oneric&group=gmane.comp.file-systems.ceph.devel ; done
[17:58] * fred__ (~fred@80-219-180-134.dclient.hispeed.ch) has joined #ceph
[17:58] <fred__> yehudasa, around?
[18:00] <fred__> yehudasa, tested http://pastebin.com/Uw05TEgG and it works if I set SCRIPT_NAME to '' (was '/' in my case), which is fine.
[18:14] * fronlius (~fronlius@testing78.jimdo-server.com) Quit (Quit: fronlius)
[18:16] <yehudasa> fred_: cool, thanks
[18:19] <yehudasa> fred_: I'll commit that patch then
[18:25] <gregaf> wido: saw you already created a bug, but I believe the answer is "no", we haven't seen that one yet
[18:25] <gregaf> Habitual: you're going to have to give me more to go on than "it added the key but apt-get update barfs" :)
[18:26] <Habitual> gregaf: 1 sec
[18:28] <Habitual> gregaf: "wget -q -O- https://raw.github.com/NewDreamNetwork/ceph/master/keys/autobuild.asc | sudo apt-key add - " came back OK. Next instruction is modify the /etc/apt/sources.list.d/ceph.list - then apt-get update barfs with "W: GPG error: http://ceph.newdream.net oneiric InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY DA4420ED288995C8"
[18:28] * ceph (~hylick@ Quit (Read error: Connection reset by peer)
[18:29] <Habitual> Thank you.
[18:31] <gregaf> Habitual: "echo /etc/apt/sources.list.d/ceph.list" ?
[18:32] <Habitual> deb http://ceph.newdream.net/debian/ oneiric main
[18:37] * ceph (~hylick@ has joined #ceph
[18:40] * joshd (~joshd@aon.hq.newdream.net) has joined #ceph
[18:42] <sagewk> habitual: ah, you imported the wrong key. autobuild.asc is for the autobuilds against git.
[18:42] * ceph (~hylick@ Quit (Read error: Connection reset by peer)
[18:42] <sagewk> releases are signed with my private key
[18:42] <sagewk> http://newdream.net/~sage/pubkey.asc (288995c8)
[18:42] <sagewk> er, s/private/personal/
[18:46] <wido> gregaf: Yes, just created a bug
[18:46] <Habitual> Sweet! Thanks. wget -q -O- http://newdream.net/~sage/pubkey.asc | sudo apt-key add - and apt-get update both now work.
[18:46] <Habitual> ..."work"
[18:47] <wido> but it seems I'm hitting a btrfs one as well, running 3.0 and the remaining OSD's are in status D and won't respond nor shutdown
[18:51] * fred__ (~fred@80-219-180-134.dclient.hispeed.ch) Quit (Quit: Leaving)
[18:51] * ceph (~hylick@ has joined #ceph
[18:53] <gregaf> wido: that's not impossible, btrfs has had a lot of churn and right now the newer the code the more stable it's likely to be :/
[18:56] <wido> gregaf: Yeah, I think so. That's going to be a hard reboot then which will probably kill the filesystems, open_ctree failed :)
[18:56] <gregaf> heh
[19:01] <gregaf> grape_: did you see that conversation about gpg keys and your install docs above?
[19:02] * jmlowe (~Adium@129-79-195-139.dhcp-bl.indiana.edu) has joined #ceph
[19:03] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) has joined #ceph
[19:05] * chutzpah (~chutz@ has joined #ceph
[19:06] <jmlowe> I'm missing something here, all of my rbd commands timeout and I can't figure out why
[19:08] <gregaf> jmlowe: what's ceph -s report?
[19:10] <jmlowe> checking...
[19:11] <jmlowe> on a healthy client:
[19:11] <jmlowe> root@----:/var/log/ceph# ceph -s
[19:11] <jmlowe> 2012-01-27 13:11:03.052271 pg v603043: 2376 pgs: 2376 active+clean; 354 GB data, 711 GB used, 20175 GB / 22004 GB avail
[19:11] <jmlowe> 2012-01-27 13:11:03.058944 mds e35: 1/1/1 up {0=beta=up:active}, 1 up:standby
[19:11] <jmlowe> 2012-01-27 13:11:03.060339 osd e110: 12 osds: 12 up, 12 in
[19:11] <jmlowe> 2012-01-27 13:11:03.064554 log 2012-01-27 06:20:04.589745 osd.7 1410 : [INF] 2.27 scrub ok
[19:11] <jmlowe> 2012-01-27 13:11:03.074223 mon e1: 3 mons at {alpha=,beta=,gw48=}
[19:12] <jmlowe> unhealthy client hasn't returned yet
[19:12] <gregaf> okay, does the healthy client succeed at an rbd command?
[19:12] <jmlowe> yes
[19:13] <gregaf> ah, so there's a connection problem on the bad client presumably
[19:13] <jmlowe> wait, I figured it out
[19:13] <gregaf> cancel that request and try re-running it with the messenger debugging on to stdout?
[19:13] <gregaf> heh, my favorite result!
[19:14] <gregaf> bbiab
[19:14] <jmlowe> *sigh*, our networking guys haven't set the mtu for jumbo frames
[19:28] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[19:28] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit ()
[19:34] * cp (~cp@ has joined #ceph
[19:40] <sagewk> anybody wanna take a quick look at wip-filestore-errors?
[19:41] <joshd> sagewk: sure
[19:45] <sagewk> also forgot to mention earlier: going to merge in these last couple fixes and build v0.41. and then merge all the stuff into master.
[19:47] * fronlius (~fronlius@f054101115.adsl.alicedsl.de) has joined #ceph
[19:47] <lxo> scrubbing is reporting a bunch of errors with files that should have data but somehow got size zero in one of the replicas (easy to fix), and others that got data (and the right data) but that scrub thinks should have size zero
[19:47] <gregaf> lxo: you mean objects, right?
[19:48] <lxo> I can locate the files in the ceph filesystem and check their contents just fine, and rados reports the expected nonzero size, but... how do I tell scrub that the nonzero size is the correct one?
[19:50] <lxo> well, I guess. all files so far were small ones (<4KB), and they are apparently temporally correlated. probably some filesystem glitch that corrupted pg info/logs as well (ceph-osd complained and moved aside some corrupted files, indeed)
[19:51] <lxo> I could just rewrite the files and be done with it, but some of them are in snapshots that would be a bit painful to recreate in the trunk, so I'd love to somehow tweak the metadata so that scrub doesn't complain about the inconsistency any more
[19:51] <gregaf> okay, so you made small files in the fs and the actual rados data sizes don't match the rados logs
[19:51] <gregaf> can you paste an example warning?
[19:55] <joshd> sagewk: made a comment on github, otherwise looks good - much easier to read now
[19:56] <lxo> 2012-01-27 15:17:03.703627 7fac6b7f6700 osd.2 1621 pg[6.ae( v 181'2539 (0'0,181'2539] n=2526 ec=33 les/c 1621/1621 1612/1612/1139) [2,0,6] r=0 lpr=1612 lcod 0'0 mlcod 0'0 !hml active+clean+scrubbing+inconsistent] on disk size (0) does not match object info size (267) for 1000016fdc6.00000000/head/a04964ae
[19:56] <sagewk> joshd: thanks
[19:56] * jmlowe (~Adium@129-79-195-139.dhcp-bl.indiana.edu) has left #ceph
[19:56] <lxo> it looks like this, except that this one repair *can* fix. the ones it can't are those in which the disk size is nonzero and the object info size is zero
[19:57] <lxo> unfortunately all of them seem to have scrolled by by now, and I can't find them in the osd logs either. they're kind of rare
[19:57] <gregaf> okay, just wanted to check that they were log versus disk inconsistencies and not inter-node inconsistencies
[19:58] <sagewk> i wonder if this could be side-effects of the truncate workaround
[19:58] <gregaf> doesn't that ignore it way before it should be updating the Infos?
[19:58] <gregaf> sjust?
[19:58] <lxo> ah, found it: 2012-01-19 13:08:21.241750 7fb8bbff7700 log [ERR] : 7.54
[19:58] <lxo> osd.0: soid 1000005fc74.00000000/head/38a94154size 3402 != known size 0
[19:59] <lxo> this one is current: 2012-01-27 15:14:49.831149 7fac6b7f6700 log [ERR] : 6.ae osd.0: soid 1000016fdc6.00000000/head/a04964aesize 267 != known size 0
[20:00] <sagewk> lxo: i assume these scrub errors just started appearing? what version of the code is it?
[20:00] <gregaf> ah, actually that's an inter-node inconsistency I think
[20:00] <lxo> # rados --pool=tmp stat 1000016fdc6.00000000
[20:00] <lxo> tmp/1000016fdc6.00000000 mtime 1252280315, size 267
[20:00] <gregaf> the primary says size is zero and a replica says differently, unless I'm very confused about the context
[20:01] <lxo> sagewk, IIRC I created the cluster anew with 0.40, so it's kind of recent
[20:01] <sagewk> ok
[20:02] <lxo> aah, then this one should be fixable by just replacing the file in the primary, eh?
[20:02] <lxo> sorry, I'd somehow got the impression that this was the same kind of “does not match” error
[20:02] <gregaf> lxo: well I'm a little confused about how you could be getting the correct data if the primary says it's length 0
[20:02] <gregaf> unless it's cached locally already
[20:03] <lxo> that's odd indeed
[20:04] <lxo> it might be. or it might be that I brought down the osd that had the faulty copy at the time I ran the test. or maybe I just inspected the osd object directly. or maybe with rados. sorry, I don't recall, it was the end of a veeery long day
[20:04] <lxo> come to think of it... I'm pretty sure I got the *wrong* data when using the filesystem interface, and then noticed that I had the correct data in place at one of the osds. which now I'm guessing was not the primary
[20:08] <lxo> fortunately I'd taken snapshots, for attempts to pg repair it have now propagated the zero-sized file to the replicas
[20:14] <lxo> is this object size mismatch a new test in the scrubber? I don't recall seeing that one before 0.40
[20:17] <sjust> Ixo: did you see a message containing "does not match object info size" in your ceph -w output for the same pg and scrub?
[20:17] <sjust> Ixo: that message is quite old, it's just hard to trigger
[20:17] * lollercaust (~paper@40.Red-88-19-215.staticIP.rima-tde.net) Quit (Quit: Leaving)
[20:18] <lxo> there's another problem I've come across recently, after I finished initializing the cluster with 1TB of data (some 6M objects) over 2 osds, I brought up the other osds, and the heavy synchronization and lots of snapshot creation and removal apparently triggered a btrfs bug:
[20:19] <lxo> it would get in a state in which both current and the latest snap_* would be inaccessible; umounting and remounting would fix it; trying to btrfs del current would fail before that. later, I found out that removing the snapshot alone was enough for current to become accessible again, without remounting. just FYI
[20:21] <lxo> sjust, AFAICT I did, but I may not have realized that it was about the same object mentioned in the “!= known size 0” message
[20:22] <gregaf> oh, that sounds to me like it could in fact be btrfs causing the scrub bugs
[20:24] <lxo> yeah, I'm pretty sure the inaccessible snap is a btrfs problem; as for the zero-sized files, I'm more inclined to suspect btrfs than ceph on this one too, but the jury is still out ;-)
[20:24] <sjust> Ixo: that might be related to a btrfs bug Sage is looking into now
[20:25] <sjust> Ixo: the "does not match object info size" would have occurred right after the other message, if it were related
[20:25] <lxo> yep. that's where it did today. I'm not sure it did the other day
[20:25] <lxo> it probably did, but I didn't realize it. long day and all ;-)
[20:26] <sjust> Ixo: from this information it's either a bug in osd recovery or a btrfs bug which resulted in an incorrectly truncated file
[20:26] <lxo> I'm using btrfs straight from v3.2.1
[20:26] <sjust> has there been a lot of osd cluster churn?
[20:27] <lxo> you mean lots of changes to the osd data or to the osd active set?
[20:27] <sjust> yeah
[20:28] <lxo> data changed a lot while uploading 1TB, but all of the problems are in a single pool whose ceph directory was uploaded over a day or two, with tons of small files (my sources and builds tree ;-)
[20:29] <sjust> but not a lot of osds going up and down?
[20:29] <lxo> as for the osd active set, it was supposed to have remained unchanged throughout the operation, but I don't recall whether there were any unexpected changes. lemme check the logs
[20:34] <lxo> no churn whatsoever. the osds were only restarted some 12 hours after that, when the upload was done and a snapshot was taken. the snapshot already had wrong files
[20:35] <lxo> the faulty files had all been created while running v3.1.9, but the btrfs module was from cmason's for-linus tree from just before 3.2
[20:36] <lxo> (the one that was meant to run on 3.1.*)
[20:37] <lxo> anyhow, thanks for drawing to my attention that the metadata was indeed correct, just the error message was different meaning I had a different error in my hands. I'd wasted a lot of time trying to track down the location of the wrong metadata, for the only one I could find was correct ;-D
[20:38] <sjust> Ixo: no problem, it could still be an osd error though, but it seems more likely to be a btrfs error
[20:38] <gregaf> wido: do you have some logs for #1992?
[20:59] * aliguori (~anthony@cpe-70-123-132-139.austin.res.rr.com) Quit (Remote host closed the connection)
[21:08] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) Quit (Quit: adjohn)
[21:26] * aliguori (~anthony@ has joined #ceph
[21:55] * fronlius (~fronlius@f054101115.adsl.alicedsl.de) Quit (Quit: fronlius)
[22:07] * aliguori (~anthony@ Quit (Quit: Ex-Chat)
[22:08] * vodka (~paper@40.Red-88-19-215.staticIP.rima-tde.net) has joined #ceph
[22:20] <NaioN> does anybody know if the writeback for rbd for kvm/qemu also has been implemented in the kernel rbd client?
[22:20] <NaioN> I'm now testing with a 3.2.1 kernel and it seems I get a lot more performance than with older kernels
[22:21] <gregaf> NaioN: nope, nothing new there
[22:32] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) has joined #ceph
[22:33] * aa (~aa@r200-40-114-26.ae-static.anteldata.net.uy) Quit (Remote host closed the connection)
[22:36] <NaioN> hmmm has it been planned for a future kernel?
[22:36] <NaioN> I get real good performance at the moment with the rbd kernel client
[22:37] * mtk (~mtk@ool-44c35967.dyn.optonline.net) Quit (Quit: Leaving)
[22:38] <NaioN> but it could even be better with the writeback I assume
[22:43] * ceph (~hylick@ Quit (Quit: Leaving.)
[22:43] <gregaf> It'll make it in eventually; I don't think there's a specific time in mind yet, though
[23:07] <Habitual> what are the implications (summary Please) of ./configure --without-tcmalloc - following http://ceph.newdream.net/wiki/Installing_on_RedHat_or_CentOS
[23:08] <gregaf> Habitual: we've found that the ceph-osd and ceph-mds daemons are considerably more memory-efficient using tcmalloc
[23:08] <gregaf> however, we haven't tested without it lately
[23:09] <Habitual> I'm about to test it. :)
[23:10] <gregaf> it's just a new memory allocator though, so there aren't any correctness concerns, and if it doesn't impact performance on your box then who cares ;)
[23:11] <Habitual> true dat
[23:11] <Habitual> re: performance
[23:12] <Habitual> now I get ""Can't find boost statechart headers; need 1.34 or later""
[23:14] <gregaf> ungh, I thought all those dependencies were resolved a long time ago
[23:16] <gregaf> I think you just need libboost-dev, actually
[23:16] <Habitual> I'm curious about "Switch to the rc branch" at http://ceph.newdream.net/wiki/Installing_on_RedHat_or_CentOS#Building_Ceph_itselfsec
[23:16] <Habitual> doh
[23:16] <Habitual> sec
[23:16] <gregaf> I don't think that's been updated in a while
[23:17] <Habitual> yum can't find it. :(
[23:17] <gregaf> rc being "not latest master" because that's our development branch
[23:17] <Habitual> ignore that step then? :)
[23:17] <gregaf> so master's stability varies, although we're starting to improve it significantly
[23:18] <Habitual> I'm taking your word for that. :)
[23:18] <gregaf> choose either "next", "master", or "stable", depending on your tolerance for weird bugs, I guess
[23:18] <Habitual> hack-mode, got it.
[23:19] <gregaf> Habitual: I'm just googling, but try boost-devel or boost-statechart-devel?
[23:20] <Habitual> I'm trying to find my EPEL repo sticky note.
[23:33] * joao is now known as Guest721
[23:33] * joao (~joao@ has joined #ceph
[23:39] * Guest721 (~joao@ Quit (Ping timeout: 480 seconds)
[23:42] <Habitual> gregaf: nearest 'match' I can see would be boost141-filesystem.x86_64 for there's no libboost-dev on CentOS 5.7 even with EPEL repos.
[23:42] <gregaf> no other boost prefixed stuff?
[23:42] <Habitual> tons
[23:42] <gregaf> boost-devel?
[23:42] <gregaf> or boost-statechart-devel?
[23:43] <Habitual> gregaf: http://susepaste.org/80701953
[23:44] <gregaf> boost-devel probably has it
[23:45] <gregaf> yep: http://rpmfind.net//linux/RPM/fedora/updates/16/i386/boost-devel-1.47.0-6.fc16.i686.html
[23:46] <Habitual> installed.
[23:54] <Habitual> bah hum bugger me. Tomorrow is another day. :(
[23:55] <gregaf> it didn't work?
[23:55] <Habitual> no.
[23:56] <gregaf> crap
[23:56] <gregaf> we might have broken our checks for centos again :/
[23:57] <Habitual> rpm -Uvh ./boost-1.47.0-6.fc16.src.rpm warning: group mockbuild does not exist - using root
[23:57] <Habitual> ########################################### [100%]
[23:57] <Habitual> error: unpacking of archive failed on file /usr/src/redhat/SOURCES/boost-1.47.0-cmakeify-full.patch;4f23296a: cpio: MD5 sum mismatch
[23:58] <Habitual> maybe this?
[23:58] <Habitual> boost-devel-1.33.1-10.el5.i386.html The Boost C++ headers and development libraries CentOS 5.7 for x86_64
[23:58] <gregaf> yeah, looks like it didn't actually install it?
[23:59] <Habitual> it did not.
[23:59] <gregaf> okay, so Not Our Fault ;)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.