#ceph IRC Log


IRC Log for 2011-09-22

Timestamps are in GMT/BST.

[0:08] * jeffhung (~jeffhung@60-250-103-120.HINET-IP.hinet.net) Quit (Ping timeout: 480 seconds)
[0:15] * jeffhung (~jeffhung@60-250-103-120.HINET-IP.hinet.net) has joined #ceph
[0:15] * Nightdog (~karl@190.84-48-62.nextgentel.com) Quit (Quit: Ex-Chat)
[0:50] * lx0 crosses fingers in preparation to the 0.35 upgrade ;-)
[0:51] * bchrisman (~Adium@ Quit (Quit: Leaving.)
[0:53] <cp> anyone else seeing crashes on 0.35 when trying to do things like "rbd list"?
[1:03] * greglap (~Adium@ Quit (Read error: Connection reset by peer)
[1:17] <Tv|work> cp: make sure your libraries match the ceph version
[1:17] <Tv|work> we've seen that when running new binary against old libs
[1:17] <cp> TV|work: That may have been it. A fresh install mostly works now
[1:18] <cp> Still having some trouble with custom crush rules for pools which worked before though.
[1:28] * danielf (~df@ Quit (Quit: danielf)
[1:28] * cp (~cp@ Quit (Quit: cp)
[1:38] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) has joined #ceph
[1:45] * jojy (~jojyvargh@70-35-37-146.static.wiline.com) Quit (Quit: jojy)
[1:50] * Nightdog (~karl@190.84-48-62.nextgentel.com) has joined #ceph
[1:52] <Nightdog> To whoever that is able to fix it: On http://ceph.newdream.net/docs/latest/ops/install/mkcephfs/ at the 2nd footnote, there is a missing space after the ":". Correct syntax on that line (at least in ubuntu) is "%admin ALL=(ALL) NOPASSWD: ALL". And thanks for making a cool filesystem :)
[1:54] <Tv|work> Nightdog: that space is optional
[1:56] <Nightdog> Tv|work: Ok, then I stand corrected. Only tested it on this one box.
[1:56] <Tv|work> oh, and it didn't work? weird
[1:56] <Tv|work> i have some dozens of boxes happily using that exact line in sudoers
[1:56] * greglap (~Adium@ has joined #ceph
[2:01] <Nightdog> Yea, I retested it now, and it seems to be working. Must have been some other typo :/
[2:05] * sjustlaptop (~sam@aon.hq.newdream.net) Quit (Quit: Leaving.)
[2:07] * adjohn (~adjohn@ has joined #ceph
[2:09] <stass> any developers here?
[2:12] <greglap> stass: what's up?
[2:13] <stass> greglap: hey, I was asking yesterday. I'm porting ceph to freebsd, and I was wondering what is the best way to go about submitting patches?
[2:13] <stass> greglap: I'm trying to keep changes localized and minimal now, but some things will benefit from refactoring later
[2:14] <stass> greglap: and I'm wondering whether it will be acceptable to merge into the main tree at all
[2:14] <greglap> so far we've just been taking stuff off the mailing list
[2:14] <greglap> we're on github now too, so apparently there's a good interface for submitting merge requests?
[2:14] <greglap> well, we'll have to see how invasive the changes are
[2:14] <stass> greglap: so I just submit push request?
[2:15] <stass> s,push,pull,g
[2:15] <greglap> we haven't done that yet, but my understanding is yes
[2:15] <stass> greglap: they're not really invasive, mostly extra includes/ifdefs
[2:15] <greglap> (we just moved our main repo at the end of last week)
[2:15] <stass> greglap: yep, I already benefited from this:)
[2:15] <greglap> we're not ideologically opposed to making other OSes work, certainly — nothing else does right now but there's still scattered #if DARWIN stuff in places
[2:16] <stass> greglap: yep, I used some of the darwin ifdefs
[2:16] <stass> greglap: as they're relevant for freebsd as well
[2:16] <greglap> yep :)
[2:16] <stass> I still need to implement the extattr support for freebsd
[2:17] <stass> I just want to keep the changes minimal, so I won't end up submitting a huge patch nobody can review:))
[2:17] <greglap> sounds good!
[2:17] <Tv|work> stass: forking ceph.git on github and using branches smartly is your best bet forward
[2:17] <Tv|work> stass: note the Sign-off-by etc conventions
[2:18] <greglap> and now if there's nothing else, I have a brand-new TV and a Star Wars blu-ray calling to me :D
[2:18] <Tv|work> stass: try to separate refactorings from porting efforts as much as possible, etc
[2:18] <stass> Tv|work: yep, that's what I'm doing
[2:18] <stass> Tv|work: so I'm on the right track, thanks
[2:20] <stass> my plan is to get the userland parts working well, and then maybe focus on making the GEOM based rbd layer and adding ZFS based transactions support similar to what you're doing with btrfs
[2:26] * Tv|work (~Tv|work@aon.hq.newdream.net) Quit (Ping timeout: 480 seconds)
[2:46] * greglap (~Adium@ Quit (Read error: Connection reset by peer)
[3:05] * Nightdog (~karl@190.84-48-62.nextgentel.com) Quit (Quit: Ex-Chat)
[3:07] * DanielFriesen (~dantman@S0106001731dfdb56.vs.shawcable.net) Quit (Remote host closed the connection)
[3:09] * Dantman (~dantman@S0106001731dfdb56.vs.shawcable.net) has joined #ceph
[3:16] * joshd (~joshd@aon.hq.newdream.net) Quit (Quit: Leaving.)
[3:21] * lx0 (~aoliva@9YYAABNFN.tor-irc.dnsbl.oftc.net) Quit (Quit: later)
[3:21] * lx0 (~aoliva@lxo.user.oftc.net) has joined #ceph
[3:21] * lx0 is now known as lxo
[3:48] * jojy (~jojyvargh@75-54-231-2.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[3:49] * jojy (~jojyvargh@75-54-231-2.lightspeed.sntcca.sbcglobal.net) Quit ()
[3:58] * yoshi (~yoshi@p9224-ipngn1601marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[4:22] * greglap (~Adium@ has joined #ceph
[5:21] * greglap (~Adium@ Quit (Read error: Connection reset by peer)
[5:27] * adjohn (~adjohn@ Quit (Quit: adjohn)
[6:38] * adjohn (~adjohn@50-0-92-177.dsl.dynamic.sonic.net) has joined #ceph
[6:46] * adjohn (~adjohn@50-0-92-177.dsl.dynamic.sonic.net) Quit (Ping timeout: 480 seconds)
[7:15] * gregaf1 (~Adium@aon.hq.newdream.net) has joined #ceph
[7:20] * gregaf (~Adium@aon.hq.newdream.net) Quit (Ping timeout: 480 seconds)
[8:04] * The_Bishop (~bishop@port-92-206-29-232.dynamic.qsc.de) Quit (Quit: Wer zum Teufel ist dieser Peer? Wenn ich den erwische dann werde ich ihm mal die Verbindung resetten!)
[9:17] * greglap (~Adium@ has joined #ceph
[9:23] * pruby (~tim@leibniz.catalyst.net.nz) Quit (Ping timeout: 480 seconds)
[9:28] * pruby (~tim@leibniz.catalyst.net.nz) has joined #ceph
[9:33] * greglap (~Adium@ Quit (Read error: Connection reset by peer)
[10:01] * pruby (~tim@leibniz.catalyst.net.nz) Quit (Ping timeout: 480 seconds)
[10:03] * pruby (~tim@leibniz.catalyst.net.nz) has joined #ceph
[10:27] * yehudasa (~yehudasa@aon.hq.newdream.net) Quit (Ping timeout: 480 seconds)
[10:35] * yehudasa (~yehudasa@aon.hq.newdream.net) has joined #ceph
[10:37] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[10:37] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[11:33] * macan_ (~chatzilla@ has joined #ceph
[11:39] * macan (~chatzilla@ Quit (Ping timeout: 480 seconds)
[11:49] * yoshi (~yoshi@p9224-ipngn1601marunouchi.tokyo.ocn.ne.jp) Quit (Remote host closed the connection)
[12:01] * pruby (~tim@leibniz.catalyst.net.nz) Quit (Ping timeout: 480 seconds)
[12:03] * pruby (~tim@leibniz.catalyst.net.nz) has joined #ceph
[12:12] * pruby (~tim@leibniz.catalyst.net.nz) Quit (Ping timeout: 480 seconds)
[12:13] * pruby (~tim@leibniz.catalyst.net.nz) has joined #ceph
[12:22] * pruby (~tim@leibniz.catalyst.net.nz) Quit (Ping timeout: 480 seconds)
[12:22] * pruby (~tim@leibniz.catalyst.net.nz) has joined #ceph
[12:47] * pruby (~tim@leibniz.catalyst.net.nz) Quit (Ping timeout: 480 seconds)
[12:48] * pruby (~tim@leibniz.catalyst.net.nz) has joined #ceph
[12:57] * pruby (~tim@leibniz.catalyst.net.nz) Quit (Ping timeout: 480 seconds)
[12:59] * pruby (~tim@leibniz.catalyst.net.nz) has joined #ceph
[14:21] * huangwei (~huangwei@ has joined #ceph
[14:30] * huangwei (~huangwei@ Quit (Quit: Leaving)
[15:04] * macan (~chatzilla@ has joined #ceph
[15:05] * macan_ (~chatzilla@ Quit (Ping timeout: 480 seconds)
[15:17] * pruby (~tim@leibniz.catalyst.net.nz) Quit (Ping timeout: 480 seconds)
[15:19] * pruby (~tim@leibniz.catalyst.net.nz) has joined #ceph
[15:27] * macan (~chatzilla@ Quit (Ping timeout: 480 seconds)
[15:28] * macan (~chatzilla@ has joined #ceph
[15:47] * pruby (~tim@leibniz.catalyst.net.nz) Quit (Ping timeout: 480 seconds)
[15:49] * pruby (~tim@leibniz.catalyst.net.nz) has joined #ceph
[16:57] * adjohn (~adjohn@50-0-92-177.dsl.dynamic.sonic.net) has joined #ceph
[17:04] <lxo> wow, osd format change still running on my 3 osds!
[17:07] <jantje_> better running than not running at all ;-)
[17:08] <jantje_> I don't know why they even convert, it's still not a 'stable' release.
[17:44] <gregaf1> lxo: how long has it been running for?
[17:45] * greglap (~Adium@aon.hq.newdream.net) has joined #ceph
[17:45] * greglap (~Adium@aon.hq.newdream.net) Quit ()
[17:47] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[17:54] * Tv|work (~Tv|work@aon.hq.newdream.net) has joined #ceph
[18:17] <lxo> gregaf1, 17 hours so far. 4.4M files in each OSD. I can tell they're making progress with strace
[18:18] <sjust> Ixo: it should be spamming the log with conversion progress
[18:18] <lxo> ah, nice, it is indeed
[18:21] <lxo> hey, it even tells me how far it is into completion. awesome!
[18:22] <lxo> one is 4850/5179, another is at 1105/5269 and the other is at 3350/5269
[18:23] <lxo> odd, shouldn't the numbers after the slash be the same for all of them? and shouldn't machines with pretty much the same hw configuration and workload (like the first two) proceed about as fast?
[18:24] <sjust> Ixo: the pgs are pseudo randomly distributed, so there will be small variations
[18:24] <sjust> but yeah, seems like they should be moving towards completion at roughly the same rate
[18:24] <lxo> sjust, not for me, no. all OSDs should have exactly the same data, it's all replicated x3
[18:25] <sjust> oh
[18:25] <sjust> strange
[18:25] <lxo> very
[18:25] <lxo> I wonder if it has to do with deleted snapshots or something
[18:25] * aliguori (~anthony@cpe-70-123-132-139.austin.res.rr.com) Quit (Quit: Ex-Chat)
[18:26] <sjust> oh, that could do it
[18:26] <sjust> were you using the rados level snapshots?
[18:26] * joshd (~joshd@aon.hq.newdream.net) has joined #ceph
[18:27] * jmlowe (~Adium@140-182-230-125.dhcp-bl.indiana.edu) has joined #ceph
[18:28] <jmlowe> quick sanity check about moving from 0.34->0.35, if it's a single node install I can just shutdown, upgrade, restart?
[18:28] <sjust> yeah, just make sure the pgs are clean and kill cmon first
[18:37] <wonko_be> hey, just updated to .35 - got this after a second restart of the osds: http://pastie.org/2574675
[18:37] <wonko_be> should I drop it in the bugtracker, or is it a know issue?
[18:38] <sjust> ah...
[18:38] <sjust> one sec
[18:38] <wonko_be> i have it on a couple of osds
[18:40] <sjust> I'm going to push a fix in a minute or two
[18:40] <wonko_be> okay, let me know
[18:46] * pruby (~tim@leibniz.catalyst.net.nz) Quit (Ping timeout: 480 seconds)
[18:48] <lxo> sjust, mds snapshots
[18:48] * pruby (~tim@leibniz.catalyst.net.nz) has joined #ceph
[18:48] <lxo> osd0 finished conversion, and got the same error as wonko_be on restart
[18:52] <lxo> all right, I'm going to save mon/osd state and start the new filesystem as planned, because of various corruptions and inconsistencies in the old one
[18:52] <sjust> 5503d4501742fb6e8aee3e4096e75c933edc8e8f in stable should fix that assert failure
[18:53] <wonko_be> ah, let me try that one
[18:53] <wonko_be> is it already packaged somewhere, or should i build it myself?
[18:54] * bchrisman (~Adium@70-35-37-146.static.wiline.com) has joined #ceph
[18:55] * aliguori (~anthony@ has joined #ceph
[18:57] <sjust> wonko_be: you'll have to build it yourself
[18:58] <wonko_be> its building already :-)
[18:58] <wonko_be> had to scout around a bit to find the new repo... on github now
[18:59] <Tv|work> sagewk, yehudasa: so what's the story with ceph.newdream.net:/home/sage/ceph.newdream.net/debian-rgw
[18:59] <lxo> wonko_be, been meaning to look for it myself. care to share the git url?
[18:59] <Tv|work> sagewk, yehudasa: should that perhaps come from a gitbuilder, instead?
[18:59] <sagewk> tv: old repo with old builds.. we can hose it
[19:00] <wonko_be> lxo git://github.com/NewDreamNetwork/ceph.git
[19:00] <lxo> thanks
[19:00] <wonko_be> make sure to checkout the stable
[19:00] <Tv|work> sagewk: so where's sepia supposed to get our custom apache2 & fcgi debs?
[19:05] <Tv|work> sagewk: off the ndn gitbuilders, i guess?
[19:05] <Tv|work> yeah that seems like the way to go
[19:10] <gregaf1> lxo: is your metadata pool maybe at a different replication level than your data pool?
[19:11] <wonko_be> sjust: fixes it!
[19:12] <sjust> wonko_be: cool
[19:13] <wonko_be> thx
[19:16] <lxo> gregaf1, nope, I'm pretty sure I had all pools at x3
[19:17] <lxo> sjust, I take it that corrupt logs aren't all that common since this wasn't caught before, eh?
[19:18] <wonko_be> sjust: all data is still there also, so i'm happy...
[19:18] <lxo> I wonder, do you guys use kernel.org ceph.ko for testing, some newer ceph.ko, or cfuse?
[19:19] <wonko_be> the one that comes with debian 3.0.0 packages
[19:21] <lxo> hmm, I'm still kind of stuck on 2.6.38, as 3.0's btrfs becomes ridiculously inefficient under some loads, including mine :-(
[19:23] <lxo> and I'm beginning to suspect that it may be 2.6.38's ceph.ko that causes some disk corruption and thus ceph inconsistencies
[19:24] <lxo> so this time I'm going cfuse all the way
[19:29] <sjust> Ixo: actually, the corrupt logs part of the upgrade process, we just hadn't seen any where the pg already had a backlog
[19:29] <sjust> Ixo: sorry about that
[19:31] * adjohn (~adjohn@50-0-92-177.dsl.dynamic.sonic.net) Quit (Quit: adjohn)
[19:32] <lxo> sjust, no worries. glad it was an easy fix
[19:35] <lxo> I'm just trying to figure out where my fs corruptions are coming from, 'cause it's kind of annoying to start from scratch every now and again
[19:36] <sjust> did that patch allow the osds to start up?
[19:38] <lxo> I didn't try it, but it looked reasonable to me
[19:39] * cp (~cp@c-98-234-218-251.hsd1.ca.comcast.net) has joined #ceph
[19:39] * jmlowe (~Adium@140-182-230-125.dhcp-bl.indiana.edu) Quit (Ping timeout: 480 seconds)
[19:39] <lxo> I had already planned to start from scratch with 0.35 because my filesystem was all messed up, but I figured I'd run the conversion just for the sake of it (like, reporting bugs and stuff)
[19:40] <sjust> Ixo: ah
[19:40] <lxo> but if it would be incredibly useful, I could be talked into building a tree with that patch and using a saved btrfs snashot of mon and osd to see if it advances further
[19:41] <sjust> Ixo: probably not worth it
[19:44] <lxo> 'k
[19:46] <wonko_be> lxo: the building isn't rocket science
[19:46] <wonko_be> if you run debian, I can give you my packages
[19:47] <lxo> wonko_be, oh, I know, I've been building rpms myself for quite a while. also building the git tree. it's just the re-enabling the snapshots would be take a bit of manual labor
[19:47] <lxo> thanks for the offer though
[19:47] <wonko_be> k, np
[20:00] <lxo> hmm, interesting... mount -t ceph server:/ /mnt/point no longer works, I now have to use server:port:/ instead
[20:21] <Tv|work> lxo: hrmm.. unfortunately, our qa suite potentially runs multiple mons per ip, so it always uses ports
[20:21] <Tv|work> lxo: but if that is true, getting a but report with the error message would be good
[20:35] <lxo> eeek, backslashes in filenames? osd0/current/meta/pginfo\\u10.* for pool 10's PGs
[20:36] <lxo> was that intentional?
[20:36] <sjust> yeah, \u replaces underscore
[20:37] <lxo> but but... in filenames?
[20:37] <lxo> why not just use the underscores?
[20:38] <lxo> (which are used elsewhere in the pathnames, anyway)
[20:38] <sjust> they are used to seperate fields in the filename encoding <object_name>_<object_key>_<hash>_<snap>
[20:38] <lxo> e.g. I have pginfo\u10.6f__0_A098719D
[20:38] <lxo> in current/meta
[20:38] <sjust> yeah
[20:39] <lxo> so object name is pginfo\u10.6f, no key, 0 hash, A098719D snap?!?
[20:41] <lxo> my only dislike is the backslash in filenames, that's going to be a pain to deal with if shell-scripting stuff
[20:41] <sjust> object name is pginfo_10.6f, no key, hash is A098719D, snap is 0, (sorry, hash is at the end apparently)
[20:42] <lxo> aah, I see, so the _ is encoded as \u to disambiguate
[20:42] <lxo> I wish it was a dot or something more usual for filenames, rather than a shell meta-character
[20:43] <lxo> I guess it's too late now. oh well. it's not like we're supposed to delve into the osd tree with shell scripts anyway, but still...
[20:44] <lxo> I just mentioned it because it looked like a bug, for I didn't see that for any of the single-digit pools. turned out they were all in subDIRs :-)
[20:45] * alexxy (~alexxy@ Quit (Remote host closed the connection)
[20:48] * DanielFriesen (~dantman@S0106001731dfdb56.vs.shawcable.net) has joined #ceph
[20:48] * Dantman (~dantman@S0106001731dfdb56.vs.shawcable.net) Quit (Read error: Connection reset by peer)
[21:00] * macan (~chatzilla@ Quit (Read error: Connection reset by peer)
[21:09] * jojy (~jojyvargh@70-35-37-146.static.wiline.com) has joined #ceph
[21:20] * jmlowe (~Adium@140-182-230-125.dhcp-bl.indiana.edu) has joined #ceph
[21:24] * alexxy (~alexxy@ has joined #ceph
[21:28] * adjohn (~adjohn@ has joined #ceph
[21:48] * cp (~cp@c-98-234-218-251.hsd1.ca.comcast.net) Quit (Quit: cp)
[21:49] * votz_ (~votz@pool-108-52-121-23.phlapa.fios.verizon.net) has joined #ceph
[21:55] * sjustlaptop (~sam@aon.hq.newdream.net) has joined #ceph
[21:56] * votz (~votz@pool-108-52-121-23.phlapa.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[22:04] * jmlowe (~Adium@140-182-230-125.dhcp-bl.indiana.edu) Quit (Quit: Leaving.)
[22:58] * Nightdog (~karl@190.84-48-62.nextgentel.com) has joined #ceph
[23:21] * pruby (~tim@leibniz.catalyst.net.nz) Quit (Ping timeout: 480 seconds)
[23:23] * pruby (~tim@leibniz.catalyst.net.nz) has joined #ceph
[23:25] * alexxy (~alexxy@ Quit (Remote host closed the connection)
[23:26] * alexxy (~alexxy@ has joined #ceph
[23:27] * greglap (~Adium@aon.hq.newdream.net) has joined #ceph
[23:45] * pruby (~tim@leibniz.catalyst.net.nz) Quit (Ping timeout: 480 seconds)
[23:46] * pruby (~tim@leibniz.catalyst.net.nz) has joined #ceph
[23:54] * pruby (~tim@leibniz.catalyst.net.nz) Quit (Ping timeout: 480 seconds)
[23:55] * pruby (~tim@leibniz.catalyst.net.nz) has joined #ceph

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.