#ceph IRC Log


IRC Log for 2011-02-02

Timestamps are in GMT/BST.

[0:09] * DJLee (82d8d198@ircip2.mibbit.com) has joined #ceph
[0:10] <DJLee> Is it normal for OSDs to have lots of PGs to start with, initially?
[0:11] <cmccabe> DJLee: it depends on your configuration. Also, how many is lots?
[0:12] <DJLee> I got... hm... (just running 6 OSDs now, starting 'fresh') 12336 !
[0:12] <DJLee> I have a feeling that the ceph isn't (fully) restarting properly..
[0:12] <DJLee> When restarting, I stopped all ceph, remount, delete all remaining files, delete the osdmap and monmap tmps, restart the ceph with clobber old data.
[0:13] <cmccabe> DJLee: what do you get for ./cconf osd_pg_bits
[0:13] <cmccabe> and cconf osd_pgp_bits
[0:14] <DJLee> 9
[0:15] <DJLee> I also saw 'degrading' messages popping up in the ceph -w
[0:15] <cmccabe> that's the default I think for osd_pg_bits
[0:15] <cmccabe> what does ./ceph health say?
[0:16] * MarkN (~nathan@ has left #ceph
[0:16] <DJLee> I find that I had lots of osdmap and monmap in the tmp for both osd node and mon node, so I decided to delete all of them and hope for a clean start,
[0:17] <DJLee> so after that, I think it got better, now, the healt says ok
[0:17] <DJLee> so I think the only difference was that before, I wasn't forcly removing the /tmp/osd and monmap, but I doubt this'd caused the problem;
[0:18] <cmccabe> I might be missing something here, but I don't know what you mean by "the /tmp/osd"
[0:18] <DJLee> but surely before, after the makecephfs and all started up, each osds were 100% load, and showing me lots of degradings
[0:18] <DJLee> there's the two file generated for each time I do mkcephfs
[0:19] <cmccabe> DJLee: oh, actually I wasn't aware of that.
[0:19] <DJLee> osdmap.XXX.. and monmap.XXX.. (I had so many of those from multiple cepoh running in the past, so I just cleaned them up)
[0:19] <cmccabe> DJLee: and look, our script is not properly honoring the value of $TMPDIR either. sigh.
[0:20] <DJLee> hehe
[0:20] <cmccabe> DJLee: yeah, those files really don't matter. The script should be changed to clean all that up when it exists too
[0:20] <cmccabe> DJLee: not that it's really a big deal
[0:20] <DJLee> right.. so those ones must be the one that I 'hard-restart' in the past, when the mount hanged,
[0:21] <DJLee> ok, let me try to restart the ceph, and see the message,
[0:21] <DJLee> btw, ceph -w shows srub okay, is it normal ?
[0:21] <DJLee> I wasnt seeing that in 0.24, but im using the unstable 20th jan
[0:21] <cmccabe> scrub is normal
[0:21] <cmccabe> and expected
[0:22] <DJLee> even with no files at all in any of the osds? (well except some of ceph-related files)
[0:22] <cmccabe> DJLee: I think when creating the FS itself, the MDSes create some objects
[0:27] <DJLee> hm, it restarts, but I've got lots of messages for ceph -w
[0:27] <DJLee> 2011-02-03 01:27:59.613751 mds e3: 1/1/1 up {0=up:creating} 2011-02-03 01:27:59.613791 osd e32: 6 osds: 6 up, 6 in 2011-02-03 01:27:59.613842 log 2011-02-03 01:26:54.370649 mon0 23 : [INF] osd4 failed (by osd3 2011-02-03 01:27:59.613896 mon e1: 1 mons at {0=} 2011-02-0
[0:28] * greglap (~Adium@ has joined #ceph
[0:28] <DJLee> nothing should be 'failed' because there's nothing in the disk, except the restart
[0:36] <DJLee> erm, now the health is okay, at least, so it took about..9minutes
[0:36] <DJLee> for the ceph to get back to health status;
[0:37] <cmccabe> DJLee: sometimes when coming up it takes a while for things to settle down
[0:37] <DJLee> and ceph finds it initially 'degrading' ?
[0:40] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) Quit (Quit: Yoric)
[0:40] <cmccabe> DJLee: I think the fact that different cosd instances come up at slightly different times leads to a temporary 'degraded' status
[0:41] <cmccabe> DJLee: I do find it frustrating and I wish we could shorten this period or eliminate it. I'm not sure if that's possible or not
[0:41] <DJLee> right, cuz I don't remember seeing that on our 6ssd test; heh
[0:42] <DJLee> those hdd we test now are those slow ones
[0:42] <cmccabe> hmm, I just asked someone else in the office and he said he hadn't seen the degraded-on-startup thing
[0:42] <DJLee> the 54rpm green power stuff
[0:42] <DJLee> hmm
[0:44] <cmccabe> I hope your HD is more than 54 rpm... you'll be waiting for hours at that speed!
[0:44] <DJLee> well, its definietely because the hdds are busy like 100% (iostat), and doing something at the start;
[0:44] <cmccabe> maybe you mean 5400 B)
[0:45] <DJLee> forgot the k :p
[0:45] <cmccabe> well, I would expect I/O activity at the start because the OSDs are peering
[0:45] <DJLee> judging from the iosstat, those 6 osd disks, each of them take turn to get 100% busy, sometimes two or more gets busy;;
[0:45] <DJLee> right
[0:45] <cmccabe> how long do they stay at 100%
[0:45] <DJLee> it is indeed saying 'peering'
[0:46] <DJLee> about,, 3 to 10s, or more than 20s,
[0:46] <greglap> DJLee: when are you setting the number of PGs to use?
[0:48] <DJLee> and still going on, (it will go on for about 8min+); hm, I'd thought num_pgs are set by default value
[0:48] <greglap> yeah, but I think they default to 24 and you need to turn them up if you want more
[0:48] <greglap> perhaps I'm misremembering our default setup for those
[0:49] <DJLee> well i got 12336 pgs here
[0:49] <DJLee> ok, all okay, so took less than 8min this time
[0:50] <greglap> oh, nope, I'm mistaken
[0:52] <DJLee> would it be okay, if you'd take a quick look at this start up log and ceph-w :)
[0:52] <DJLee> http://twiki.esc.auckland.ac.nz/twiki/pub/NDSG/Cephtest/ceph-start_ver.20-jan_unstable
[0:52] <DJLee> http://twiki.esc.auckland.ac.nz/twiki/pub/NDSG/Cephtest/ceph-w_ver._20-jan_unstable
[0:53] <DJLee> took ~7minutes before ceph can be usable?
[0:54] <greglap> DJLee: you've got 6 OSDs?
[0:54] <DJLee> yep
[0:55] <DJLee> 6disks
[0:55] <DJLee> iirc, more the disk more time;;
[0:58] <greglap> yeah, that would make sense
[1:02] <greglap> DJLee: looks like we just aren't setting very good defaults and so you're getting too many PGs
[1:02] <DJLee> pg sizes set with the size of disks?
[1:02] <cmccabe> greglap: I think earlier we deliberately agreed to bump up the number of PGs set by default because trying to increase PG size later was painful
[1:03] <DJLee> those disks are 2tb, and so 12tb;
[1:03] <greglap> cmccabe: yeah, but the numbers right now are ridiculous
[1:03] <DJLee> i see
[1:03] <greglap> DJLee: it's actually based on OSD number, not the disk size
[1:03] <cmccabe> greglap: I think the problem we have now is that PGs are just too heavyweight; the code needs to go on a diet
[1:03] <greglap> DJLee: you can reduce it better by putting in the global section of your ceph.conf
[1:04] <greglap> osd_pg_bits = 7
[1:04] <greglap> osd_pgp_bits = 5
[1:04] <cmccabe> greglap: but I dunno. Until we have at least a few system tests monitoring performance this all seems kind of hypothetical
[1:04] <greglap> and then the next time you run mkcephfs you'll get fewer PGs and they will peer much faster
[1:05] <greglap> I'm not sure exactly why some of your PGs/objects are getting degraded, that'll take some more looking at
[1:05] <greglap> wido has reported similar corruption for unknown reasons so I suspect we've introduced an issue recently that is making it detect incorrectly or something
[1:05] <cmccabe> greglap, DJLee: are the objects staying degraded, or do they just appear degraded for a few seconds/minutes
[1:06] <DJLee> thanks a lot, mines are set for 9 and 6, but pgp is for per-pg ?
[1:06] <greglap> cmccabe: if you look at his log they're getting fixed, it's just not clear why they're degraded to begin with
[1:07] <greglap> same with wido's issues; if he runs a repair they fix successfully — which repair really shouldn't be capable of doing in most caes
[1:07] <greglap> DJLee: 9 and 6 are the defaults and are how many bits to shift your number of OSDs to derive the number of placement groups, and how to place them
[1:08] <greglap> reducing those numbers will give you fewer placement groups, which is good since you definitely don't need 2k/OSD!
[1:08] <DJLee> cmccabe, they come back to normal, hence takes about~ 7minutes, I must not mount ceph during this time!
[1:08] * greglap (~Adium@ Quit (Quit: Leaving.)
[1:11] <cmccabe> DJLee: we'll keep this in mind when we start doing perf testing
[1:11] <cmccabe> DJLee: I definitely think it should be ready in a shorter time than that
[1:13] <DJLee> thanks alot,
[1:13] <DJLee> i jsut did the perf dd test on that 6 osds
[1:13] <DJLee> 20GB dd (&& sync) to 6 osd, made 190MB/s
[1:13] <DJLee> but the reading back to /dev/null, made 67MB/s
[1:14] <DJLee> I did find that write is always faster, so this is basically writing in parelle, but reading is from not?
[1:16] <cmccabe> DJLee: how does the load look from top (on each node) when you do the read test?
[1:21] * greglap (~Adium@cpe-76-90-239-202.socal.res.rr.com) has joined #ceph
[1:29] * tjikkun (~tjikkun@195-240-122-237.ip.telfort.nl) Quit (Read error: Operation timed out)
[1:39] <DJLee> cmccabe, when writing, all of the disks were near 90 to 100%, when reading all were about 10 to 20%
[1:40] <DJLee> do you think we need to tune the pagecaches in mds or clients? e.g., http://www.westnet.com/~gsmith/content/linux-pdflush.htm
[1:53] <cmccabe> DJLee: it sounds like a good idea in general. We haven't done much performance tuning yet.
[1:54] <cmccabe> performance optimization is kind of an iterative thing and you need to have a good test suite to make progress
[1:54] <cmccabe> but thanks for the idea
[1:58] <DJLee> yeah, i've been trying to nail down the problems of 'scalability', i.e., increasing osds, doesn't increase performance; or at least in some explainable manner.
[1:59] <cmccabe> DJLee: have you ruled out the network as bottleneck?
[1:59] <cmccabe> DJLee: does ethtool tell you it's running gigabit?
[1:59] <DJLee> yep, they are 2GB
[1:59] <DJLee> iperf give me 1.9Gb
[1:59] <cmccabe> k
[2:00] <DJLee> I use 1 to 6 ssds, in 1x replication mode, I see dd maxout !220MB/s,
[2:00] <DJLee> but when set to 2x replication mode, only by 6osds, I get about 200mb/s finally,
[2:01] <DJLee> but sometimes, those increase isnt obvious either; sometimes fewer osds = better
[2:25] * Tv|work (~Tv|work@ip-66-33-206-8.dreamhost.com) Quit (Read error: Operation timed out)
[2:53] * baldben (~bencheria@ip-66-33-206-8.dreamhost.com) Quit (Ping timeout: 480 seconds)
[2:54] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) has joined #ceph
[2:54] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) Quit ()
[2:55] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) has joined #ceph
[4:07] * joshd (~joshd@ip-66-33-206-8.dreamhost.com) Quit (Quit: Leaving.)
[5:09] * baldben (~bencheria@adsl-76-236-76-143.dsl.lsan03.sbcglobal.net) has joined #ceph
[7:57] * votz (~votz@dhcp0020.grt.resnet.group.upenn.edu) Quit (Quit: Leaving)
[8:05] * tjikkun (~tjikkun@2001:7b8:356:0:204:bff:fe80:8080) has joined #ceph
[8:36] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) has joined #ceph
[9:04] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) Quit (Quit: Yoric)
[9:40] * gregorg_taf (~Greg@ has joined #ceph
[9:47] * gregorg (~Greg@LPuteaux-156-15-57-183.w82-127.abo.wanadoo.fr) Quit (Ping timeout: 480 seconds)
[9:54] * shdb (~shdb@gw.ptr-62-65-159-122.customer.ch.netstream.com) Quit (Quit: Lost terminal)
[10:03] * verwilst (~verwilst@router.begen1.office.netnoc.eu) has joined #ceph
[10:05] * Yoric (~David@ has joined #ceph
[10:23] * gregorg_taf (~Greg@ Quit (Quit: Quitte)
[10:23] * gregorg (~Greg@ has joined #ceph
[11:39] * alexxy[home] (~alexxy@ Quit (Quit: No Ping reply in 180 seconds.)
[11:40] * alexxy (~alexxy@ has joined #ceph
[11:42] * sunech (~felix@ has left #ceph
[13:27] * Yoric_ (~David@ has joined #ceph
[13:27] * Yoric (~David@ Quit (Read error: Connection reset by peer)
[13:27] * Yoric_ is now known as Yoric
[13:30] * julienhuang (~julienhua@pasteur.dedibox.netavenir.com) has joined #ceph
[13:30] * julienhuang (~julienhua@pasteur.dedibox.netavenir.com) Quit ()
[13:32] * julienhuang (~julienhua@pasteur.dedibox.netavenir.com) has joined #ceph
[14:45] * Yoric_ (~David@ has joined #ceph
[14:45] * Yoric (~David@ Quit (Read error: Connection reset by peer)
[14:45] * Yoric_ is now known as Yoric
[15:28] * greglap (~Adium@cpe-76-90-239-202.socal.res.rr.com) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * chrisrd (~chrisrd@ Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * DeHackEd (~dehacked@dhe.execulink.com) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * sagewk (~sage@ip-66-33-206-8.dreamhost.com) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * bbigras__ (quasselcor@ Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * ghaskins (~ghaskins@66-189-113-47.dhcp.oxfr.ma.charter.com) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * nolan (~nolan@phong.sigbus.net) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * cclien (~cclien@ec2-175-41-146-71.ap-southeast-1.compute.amazonaws.com) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * baldben (~bencheria@adsl-76-236-76-143.dsl.lsan03.sbcglobal.net) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * cmccabe (~cmccabe@ Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * Jiaju (~jjzhang@ Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * sage (~sage@dsl092-035-022.lax1.dsl.speakeasy.net) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * yehudasa (~yehudasa@ip-66-33-206-8.dreamhost.com) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * gregaf (~Adium@ip-66-33-206-8.dreamhost.com) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * sjust (~sam@ip-66-33-206-8.dreamhost.com) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * atg (~atg@please.dont.hacktheinter.net) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * Meths (rift@ Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * pruby (~tim@leibniz.catalyst.net.nz) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * iggy (~iggy@theiggy.com) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * MK_FG (~MK_FG@ Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * __jt__ (~james@jamestaylor.org) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * eternaleye (~eternaley@ Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * jeffhung (~jeffhung@60-250-103-120.HINET-IP.hinet.net) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * johnl_ (~johnl@ Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * darkfader (~floh@ Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * monrad-51468 (~mmk@domitian.tdx.dk) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * Yoric (~David@ Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * alexxy (~alexxy@ Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * verwilst (~verwilst@router.begen1.office.netnoc.eu) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * DJLee (82d8d198@ircip2.mibbit.com) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * jantje (~jan@paranoid.nl) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * s15y (~s15y@sac91-2-88-163-166-69.fbx.proxad.net) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * raso (~raso@debian-multimedia.org) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * Anticimex (anticimex@netforce.csbnet.se) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * DLange (~DLange@dlange.user.oftc.net) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * morse (~morse@supercomputing.univpm.it) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * stingray (~stingray@stingr.net) Quit (reticulum.oftc.net charon.oftc.net)
[15:28] * tjikkun (~tjikkun@2001:7b8:356:0:204:bff:fe80:8080) Quit (reticulum.oftc.net charon.oftc.net)
[15:31] * greglap (~Adium@cpe-76-90-239-202.socal.res.rr.com) has joined #ceph
[15:31] * chrisrd (~chrisrd@ has joined #ceph
[15:31] * DeHackEd (~dehacked@dhe.execulink.com) has joined #ceph
[15:31] * sagewk (~sage@ip-66-33-206-8.dreamhost.com) has joined #ceph
[15:31] * bbigras__ (quasselcor@ has joined #ceph
[15:31] * ghaskins (~ghaskins@66-189-113-47.dhcp.oxfr.ma.charter.com) has joined #ceph
[15:31] * nolan (~nolan@phong.sigbus.net) has joined #ceph
[15:31] * cclien (~cclien@ec2-175-41-146-71.ap-southeast-1.compute.amazonaws.com) has joined #ceph
[15:31] * baldben (~bencheria@adsl-76-236-76-143.dsl.lsan03.sbcglobal.net) has joined #ceph
[15:31] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) has joined #ceph
[15:31] * cmccabe (~cmccabe@ has joined #ceph
[15:31] * yehudasa (~yehudasa@ip-66-33-206-8.dreamhost.com) has joined #ceph
[15:31] * Meths (rift@ has joined #ceph
[15:31] * gregaf (~Adium@ip-66-33-206-8.dreamhost.com) has joined #ceph
[15:31] * sjust (~sam@ip-66-33-206-8.dreamhost.com) has joined #ceph
[15:31] * pruby (~tim@leibniz.catalyst.net.nz) has joined #ceph
[15:31] * eternaleye (~eternaley@ has joined #ceph
[15:31] * iggy (~iggy@theiggy.com) has joined #ceph
[15:31] * MK_FG (~MK_FG@ has joined #ceph
[15:31] * atg (~atg@please.dont.hacktheinter.net) has joined #ceph
[15:31] * Jiaju (~jjzhang@ has joined #ceph
[15:31] * sage (~sage@dsl092-035-022.lax1.dsl.speakeasy.net) has joined #ceph
[15:31] * __jt__ (~james@jamestaylor.org) has joined #ceph
[15:31] * darkfader (~floh@ has joined #ceph
[15:31] * monrad-51468 (~mmk@domitian.tdx.dk) has joined #ceph
[15:31] * johnl_ (~johnl@ has joined #ceph
[15:31] * jeffhung (~jeffhung@60-250-103-120.HINET-IP.hinet.net) has joined #ceph
[15:31] * DLange (~DLange@dlange.user.oftc.net) has joined #ceph
[15:31] * morse (~morse@supercomputing.univpm.it) has joined #ceph
[15:31] * stingray (~stingray@stingr.net) has joined #ceph
[15:32] * tjikkun (~tjikkun@2001:7b8:356:0:204:bff:fe80:8080) has joined #ceph
[15:32] * Yoric (~David@ has joined #ceph
[15:32] * alexxy (~alexxy@ has joined #ceph
[15:32] * verwilst (~verwilst@router.begen1.office.netnoc.eu) has joined #ceph
[15:32] * DJLee (82d8d198@ircip2.mibbit.com) has joined #ceph
[15:32] * jantje (~jan@paranoid.nl) has joined #ceph
[15:32] * s15y (~s15y@sac91-2-88-163-166-69.fbx.proxad.net) has joined #ceph
[15:32] * raso (~raso@debian-multimedia.org) has joined #ceph
[15:32] * Anticimex (anticimex@netforce.csbnet.se) has joined #ceph
[15:35] * sage (~sage@dsl092-035-022.lax1.dsl.speakeasy.net) Quit (charon.oftc.net resistance.oftc.net)
[15:35] * Jiaju (~jjzhang@ Quit (charon.oftc.net resistance.oftc.net)
[15:35] * cmccabe (~cmccabe@ Quit (charon.oftc.net resistance.oftc.net)
[15:35] * baldben (~bencheria@adsl-76-236-76-143.dsl.lsan03.sbcglobal.net) Quit (charon.oftc.net resistance.oftc.net)
[15:35] * sjust (~sam@ip-66-33-206-8.dreamhost.com) Quit (charon.oftc.net resistance.oftc.net)
[15:35] * gregaf (~Adium@ip-66-33-206-8.dreamhost.com) Quit (charon.oftc.net resistance.oftc.net)
[15:35] * yehudasa (~yehudasa@ip-66-33-206-8.dreamhost.com) Quit (charon.oftc.net resistance.oftc.net)
[15:35] * atg (~atg@please.dont.hacktheinter.net) Quit (charon.oftc.net resistance.oftc.net)
[15:35] * __jt__ (~james@jamestaylor.org) Quit (charon.oftc.net resistance.oftc.net)
[15:35] * MK_FG (~MK_FG@ Quit (charon.oftc.net resistance.oftc.net)
[15:35] * Meths (rift@ Quit (charon.oftc.net resistance.oftc.net)
[15:35] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) Quit (charon.oftc.net resistance.oftc.net)
[15:35] * iggy (~iggy@theiggy.com) Quit (charon.oftc.net resistance.oftc.net)
[15:35] * pruby (~tim@leibniz.catalyst.net.nz) Quit (charon.oftc.net resistance.oftc.net)
[15:35] * eternaleye (~eternaley@ Quit (charon.oftc.net resistance.oftc.net)
[15:35] * jeffhung (~jeffhung@60-250-103-120.HINET-IP.hinet.net) Quit (charon.oftc.net resistance.oftc.net)
[15:35] * johnl_ (~johnl@ Quit (charon.oftc.net resistance.oftc.net)
[15:35] * darkfader (~floh@ Quit (charon.oftc.net resistance.oftc.net)
[15:35] * monrad-51468 (~mmk@domitian.tdx.dk) Quit (charon.oftc.net resistance.oftc.net)
[15:35] * baldben (~bencheria@adsl-76-236-76-143.dsl.lsan03.sbcglobal.net) has joined #ceph
[15:35] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) has joined #ceph
[15:35] * cmccabe (~cmccabe@ has joined #ceph
[15:35] * yehudasa (~yehudasa@ip-66-33-206-8.dreamhost.com) has joined #ceph
[15:35] * Meths (rift@ has joined #ceph
[15:35] * gregaf (~Adium@ip-66-33-206-8.dreamhost.com) has joined #ceph
[15:35] * sjust (~sam@ip-66-33-206-8.dreamhost.com) has joined #ceph
[15:35] * pruby (~tim@leibniz.catalyst.net.nz) has joined #ceph
[15:35] * eternaleye (~eternaley@ has joined #ceph
[15:35] * iggy (~iggy@theiggy.com) has joined #ceph
[15:35] * MK_FG (~MK_FG@ has joined #ceph
[15:35] * atg (~atg@please.dont.hacktheinter.net) has joined #ceph
[15:35] * Jiaju (~jjzhang@ has joined #ceph
[15:35] * sage (~sage@dsl092-035-022.lax1.dsl.speakeasy.net) has joined #ceph
[15:35] * __jt__ (~james@jamestaylor.org) has joined #ceph
[15:35] * darkfader (~floh@ has joined #ceph
[15:35] * monrad-51468 (~mmk@domitian.tdx.dk) has joined #ceph
[15:35] * johnl_ (~johnl@ has joined #ceph
[15:35] * jeffhung (~jeffhung@60-250-103-120.HINET-IP.hinet.net) has joined #ceph
[16:15] * greglap (~Adium@cpe-76-90-239-202.socal.res.rr.com) Quit (Quit: Leaving.)
[16:29] * greglap (~Adium@ has joined #ceph
[17:17] * greglap (~Adium@ Quit (Quit: Leaving.)
[17:38] * greglap (~Adium@ip-66-33-206-8.dreamhost.com) has joined #ceph
[17:38] * baldben (~bencheria@adsl-76-236-76-143.dsl.lsan03.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[17:40] * greglap (~Adium@ip-66-33-206-8.dreamhost.com) Quit ()
[17:48] * verwilst (~verwilst@router.begen1.office.netnoc.eu) Quit (Quit: Ex-Chat)
[17:48] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[17:54] * Yoric_ (~David@ has joined #ceph
[17:54] * Yoric (~David@ Quit (Read error: Connection reset by peer)
[17:54] * Yoric_ is now known as Yoric
[18:21] * Tv|work (~Tv|work@ip-66-33-206-8.dreamhost.com) has joined #ceph
[18:32] * Yoric (~David@ Quit (Quit: Yoric)
[18:42] * bchrisman (~Adium@70-35-37-146.static.wiline.com) has joined #ceph
[18:45] <Tv|work> cmccabe: RoundTrip.RandomRoundTrips unittest seems to segfault consistently..
[18:56] <Tv|work> ah heh, found it
[19:01] * joshd (~joshd@ip-66-33-206-8.dreamhost.com) has joined #ceph
[19:26] <Tv|work> whee found a bug in ceph_unarmor, fixing
[19:27] <sagewk> let's meet!
[19:51] * julienhuang (~julienhua@pasteur.dedibox.netavenir.com) Quit (Quit: julienhuang)
[20:00] <cmccabe> I think it's ironic there's an email about "Speling fixes"
[20:00] <sagewk> yeah made me laugh :)
[20:02] <gregaf> I presume he titled the subject on purpose
[20:02] <gregaf> it was a little mean putting it into the commit message ;)
[20:02] <Tv|work> that's a classic
[20:03] <Tv|work> I usually do git commit -m 'Tpyo.'
[20:03] * DLange (~DLange@dlange.user.oftc.net) Quit (Remote host closed the connection)
[20:09] <cmccabe> sage, should I push objecter_balance_reads now?
[20:09] * ghaskins (~ghaskins@66-189-113-47.dhcp.oxfr.ma.charter.com) Quit (Remote host closed the connection)
[20:09] <cmccabe> I verified that writes show up in both OSDs logs when doing csyn with localize_reads
[20:10] * DLange (~DLange@dlange.user.oftc.net) has joined #ceph
[20:11] <Tv|work> fyi the gitbuilder g++ is 4.4.5
[20:12] <cmccabe> tv: I'm at 4.4.5 too
[20:12] <cmccabe> tv: I guess that makes sense since ubuntu is based on debian and we're both running the latest
[20:16] * baldben (~bencheria@ip-66-33-206-8.dreamhost.com) has joined #ceph
[20:16] <Tv|work> so what's the story between ceph-client.git and ceph-client-standalone.git -- is one updated based on the other, which one is authoritative, etc?
[20:16] <Tv|work> should i make independent commits to both?
[20:17] <gregaf> ceph-client-standalone is based on ceph-client
[20:17] <gregaf> ceph-client is based on Linus' tree and is the one to do dev in
[20:17] <gregaf> I think sage has scripts to update standalone from ceph-client
[20:17] <Tv|work> ok so commits go to ceph-client
[20:17] <gregaf> and standalone also includes some backport macros etc so it works on older kernels
[20:17] <gregaf> yeah, do your dev in ceph-client
[20:18] * ghaskins (~ghaskins@66-189-113-47.dhcp.oxfr.ma.charter.com) has joined #ceph
[20:21] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) has joined #ceph
[20:22] <Tv|work> it's interesting that you guys chose to do base64 in kernel for the ceph client..
[20:22] <Tv|work> oh well ;)
[20:24] <gregaf> ?
[20:24] <Tv|work> just usually one puts minimum amount of code in the kernel
[20:24] <Tv|work> you could just hand the pre-base64decoded key into kernel
[20:26] <gregaf> how would you do that?
[20:26] <gregaf> it needs to be encoded on the client side
[20:26] <Tv|work> gregaf: does the wire protocol specify base64?-o
[20:26] * bcherian (~bencheria@ip-66-33-206-8.dreamhost.com) has joined #ceph
[20:26] <gregaf> and if you're using the kernel client there isn't any other ceph code running on the machine
[20:26] <Tv|work> gregaf: oh i expected there to be a mount helper
[20:26] <cmccabe> gregaf: mount.ceph?
[20:26] <gregaf> oh, well, I guess there's that
[20:27] <Tv|work> yup
[20:27] <gregaf> but base64 is used for the keys, I don't even know if it's used in transmission or not
[20:27] <gregaf> the keys need to stay encoded
[20:27] <Tv|work> gregaf: why?
[20:28] <gregaf> maybe I'm misunderstanding how this bit actually relates to the code
[20:28] * baldben (~bencheria@ip-66-33-206-8.dreamhost.com) Quit (Ping timeout: 480 seconds)
[20:29] <Tv|work> gregaf: keys stored in files are apparently base64'ed, handed to kernel uninterpreted, and then decoded in there; could as well decode in userspace before handing to kernel
[20:29] <Tv|work> but i'm not fluent enough with that whole thing yet to talk about specifics
[20:33] <gregaf> Tv|work: so you're allowed to pass in the key as a command-line option when mounting
[20:34] <gregaf> and if it's not base64 at that point then you can have illegal characters and junk
[20:34] <Tv|work> gregaf: urrr, keys in mount options?
[20:34] <Tv|work> gregaf: but those are *public*
[20:35] <gregaf> yeah, but the client machine is already trusted
[20:35] <Tv|work> gregaf: do you have a ceph client handy (i don't) -- run grep ceph /proc/mounts
[20:35] <gregaf> not atm
[20:35] <Tv|work> gregaf: client machine's *root* may be trusted -- that doesn't mean every user account is
[20:35] <gregaf> I guess it would be a bad idea to do it on a machine where the machine's users are untrusted then ;)
[20:35] <gregaf> nevertheless the capability exists
[20:36] <Tv|work> oh wow
[20:36] <gregaf> it's not the common case or anything
[20:36] <Tv|work> o hope you you realize that really needs to be fixed before ceph gets popular
[20:36] <cmccabe> gregaf: I think maybe what tv is referring to is that other users could use ps to find the key
[20:36] <Tv|work> can't have weaker security than nfs, really
[20:36] <Tv|work> cmccabe: cat /proc/mounts
[20:36] <gregaf> you don't have to pass it in via the command-line
[20:36] <cmccabe> tv: oh... the options stay there?
[20:37] <Tv|work> cmccabe: or /etc/mtab, but yes
[20:37] <Tv|work> or even just race vs the mount and ps a lot
[20:37] <gregaf> the common/default case is to pass it in as just a filename, which can be inaccessible to other users
[20:37] <Tv|work> gregaf: i wish to kill uncommon insecure cases ;)
[20:37] <gregaf> well, make a bug about it then and see what happens :p
[20:37] <Tv|work> but perhaps this should use the kernels built-in key management, these days..
[20:41] <Tv|work> sagewk, *: i'm being extra careful about ceph-client.git for now, the base64 fix is in branch base64-fix there, somebody should vet it quickly for style
[20:41] <wido> gregaf: fyi, haven't seen any inconsistent pg's. Always the same, you change the logging and then the problems stays away :)
[20:42] <gregaf> well bah humbug
[20:42] <Tv|work> anyone looking at gitbuilder: the "git ls-files --modified" error is not your problem, i'm on it
[20:43] <gregaf> I didn't realize the web frontend was working again
[20:44] <Tv|work> gregaf: yeah i'm not advertising it hugely as long as it still has non-ceph trouble
[20:44] * bencherian (~bencheria@ip-66-33-206-8.dreamhost.com) has joined #ceph
[20:51] * bcherian (~bencheria@ip-66-33-206-8.dreamhost.com) Quit (Ping timeout: 480 seconds)
[20:54] <Tv|work> alright the ls-files thing is fixed, any gitbuilder errors should be "real" now
[20:54] <Tv|work> many of the non-master branches are forked at a point when master was buggy, so they'll see fails for things already fixed in master
[20:55] <bchrisman> have there been ideas floating around on how to fix the ceph export interface for NFS?
[20:59] <gregaf> bchrisman: we haven't talked about it in a long time,
[20:59] <gregaf> I don't think there are too many users
[21:00] <gregaf> if you're interested in it then you'll probably be the one driving any improvements we embark upon
[21:02] <wido> about the "rbd" tool, right now it rounds the image size
[21:03] <wido> when resizing that is not so useful, it reports 3100GB for my image now, in 793625 objects
[21:03] <wido> 793625*4=3174500MB
[21:03] <wido> 3100GB=3174400MB
[21:04] <wido> since the "--size" argument takes MB's, it would be easier if it reported MB's, doesn't it? I see that prettybyte_t() is used to make it more human readable
[21:06] <cmccabe> debian puts gcc manpages in "non-free"
[21:06] <cmccabe> fail.
[21:07] <bchrisman> gregaf: yeah.. that's what I'm thinking… wondering if the implementation has been spec'ed out such that it's a matter of work to implement.. or whether a solution hasn't really been designed yet?
[21:34] * eternaleye_ (~eternaley@ has joined #ceph
[21:34] * eternaleye (~eternaley@ Quit (Remote host closed the connection)
[21:44] <Tv|work> cmccabe: i recall fsf being a bitch about their documentation, though..
[21:45] <cmccabe> tv: there was some complaint about an aspect of their documentation license
[21:45] <Tv|work> oh good old GFDL
[21:45] <gregaf> bchrisman: not sure; I don't know that we've even talked about it since I started working here :p
[21:46] <Tv|work> the invariant thing in GFDL is really annoying
[21:46] <Tv|work> i've seen docs with multi-page invariant sections; i'd rather not live in that world
[21:47] <cmccabe> gregaf: what's the deal with Client.cc:4263
[21:47] <cmccabe> gregaf: while (off - dirp->this_offset >= 0 && off - dirp->this_offset < dirp->buffer->size())
[21:48] <cmccabe> gregaf: I think we need to test (off >= dirp->this_offset) instead
[21:49] * eternaleye_ (~eternaley@ Quit (Quit: eternaleye_)
[21:49] * eternaleye_ (~eternaley@ has joined #ceph
[21:49] <Tv|work> sagewk, *: i'm being extra careful about ceph-client.git for now, the base64 fix is in branch base64-fix there, somebody should vet it quickly for style
[21:49] <cmccabe> gregaf: just asking you since you touched it last, and I don't have much insight into readdir_r_cb
[21:49] <gregaf> yeah, I'm looking at it
[21:50] <gregaf> I didn't write that code, why do you want to make the change?
[21:50] <cmccabe> off and dirp->this_offset are both unsigned
[21:51] <gregaf> all right, that would cause trouble with that code snippet
[21:51] <cmccabe> and one is a u64. So I don't think it gets promoted to int
[21:51] <cmccabe> I think it stays unsigned, and therefore is always >= 0
[21:52] <Tv|work> cmccabe: sounds right
[21:52] <Tv|work> a unit test would be good ;)
[21:53] <cmccabe> tv: ah, apparently integer types smaller than int are promoted to int
[21:54] <Tv|work> cmccabe: but those aren't smaller than int, are they?
[21:54] <cmccabe> tv: no
[21:54] <gregaf> cmccabe: yeah, definitely need to substitute off >= dirp->this_offset for the first condition there
[21:54] * tjikkun (~tjikkun@2001:7b8:356:0:204:bff:fe80:8080) Quit (Ping timeout: 480 seconds)
[21:54] <Tv|work> yeah i was worried even about uint32_t off, until i realized it's about directories not about files
[21:54] <Tv|work> 4GB limit on files would be bad
[21:55] * DJLee (82d8d198@ircip2.mibbit.com) Quit (Ping timeout: 480 seconds)
[21:55] <Tv|work> gregaf: test test test ;)
[21:55] <cmccabe> :)
[21:55] <gregaf> I'm not sure if it would suffice to just drop the first condition and rely on the second or not, though
[21:55] <gregaf> how on earth did you run across that code?
[21:55] * tjikkun (~tjikkun@2001:7b8:356:0:204:bff:fe80:8080) has joined #ceph
[21:56] <cmccabe> gregaf: fixing warnings
[21:56] <gregaf> ah, right
[21:56] <Tv|work> yay for useful warnings!
[21:56] <cmccabe> yeah, this one was a good one
[21:56] <gregaf> this is from gcc?
[21:56] <cmccabe> just gcc with Wall, Wextra
[21:56] <gregaf> excellent
[21:57] <jantje> sagewk: did you read the pm?
[21:57] * f4m8 (~f4m8@lug-owl.de) Quit (Remote host closed the connection)
[21:57] * f4m8 (~f4m8@lug-owl.de) has joined #ceph
[21:58] <gregaf> he's been out of the office for a couple hours now
[22:12] * alexxy (~alexxy@ Quit (Ping timeout: 480 seconds)
[22:15] <Tv|work> yehudasa, gregaf: would you mind quickly checking ceph-client.git branch base64-fix that you're happy with the style etc? my first kernel-side commit, not sure if you have "linus criteria" that's stricter or something
[22:16] * alexxy (~alexxy@ has joined #ceph
[22:18] <gregaf> Tv|work: I've only git 5 kernel commits or something and still like to get mine reviewed; you'll have to wait for yehuda or sage :)
[22:18] <Tv|work> gregaf: i'm confident about the change itself, it's more Signed-Off-By etc that i want a doublecheck on
[22:20] <gregaf> Tv|work: oh, yeah, that looks like the correct if-block style, if that's what you're worried about
[22:20] <Tv|work> commit message wording etc
[22:21] <Tv|work> does it belong on master
[22:21] <Tv|work> etc
[22:22] <gregaf> looks fine to me, and yes it belongs on master
[22:22] <yehudasa> tv: it looks ok, I can't really see if the tabs that you used are actual tabs or space expansions, but other than that it's pretty trivial
[22:23] <Tv|work> they are tabs; my emacs is good about picking up the local style ;)
[22:24] <yehudasa> one of these days I'll manage to configure my vim to do that, but in the mean time I'm doing it the wrong manual way
[22:24] <Tv|work> merging it to master
[22:24] <gregaf> there's a kernel Coding Style page that Linus wrote that you can check out too if you're interested
[22:24] <gregaf> http://www.kernel.org/doc/Documentation/CodingStyle
[22:24] <Tv|work> yeah i'm familiar with the C stuff itself
[22:24] <gregaf> the kernel uses the standard git conventions as that's where those conventions came from ;)
[22:25] <Tv|work> i'm not sure about e.g. pending pull requests, how you really separate ceph vs libceph, etc
[22:25] <gregaf> I'm afraid I don't have a good link for those off-hand though
[22:25] <Tv|work> gregaf: you mean git uses kernel standards ;)
[22:25] <gregaf> yeah, that's what I was trying to say
[22:26] <gregaf> sage manages the submission stuff via the for-linus branch; I don't think you'll need to worry about that part of it
[22:26] <yehudasa> via the for-linus branch on git.kernel.org
[22:27] <cmccabe> yehudasa: I have my vim configured to use kernel coding standards on .c files, ceph standards on .cc/.cpp
[22:27] <yehudasa> oh, then I should take your .vimrc
[22:27] <Tv|work> <3 emacs for noticing tab/space things file by file
[22:28] <Tv|work> i can grab just about any random source tree and rely on tab doing the right thing
[22:28] <cmccabe> yehudasa: I use a few different files... some under .vim/ftdetect/ and some under .vim/indent/
[22:31] <yehudasa> cmccabe: your default font is atrocious
[22:31] <cmccabe> yehudasa: hehe. The guifont thing.
[22:32] <cmccabe> yehudasa: yeah, I stopped using gvim at some point. Also, I think guifont=fixed does different things depending on what fonts you have installed on your system
[22:32] <gregaf> ….wow
[22:32] <yehudasa> yeah, I'll have to disable this part
[22:33] <cmccabe> yehudasa: you can see the other config files I have at http://www.senchas.com/git/cmccabe-etc/
[22:33] <cmccabe> yehudasa: for some reason the gitweb interface isn't working at the moment, but cloning cmccabe-etc works just fine
[22:33] <yehudasa> I just copied your .vim* from fatty
[22:34] <cmccabe> yehudasa: k
[22:34] <yehudasa> and disabled that guifont line, it looks ok
[22:34] <cmccabe> yehudasa: you probably also want to remove imap <BS> <ESC>
[22:34] <yehudasa> I might get rid of the status line at the bottom, it flicks a bit
[22:34] <cmccabe> it's... just something I'm used to. But most people hate it.
[22:34] <cmccabe> I mean the backspace = escape thing
[22:35] <yehudasa> oh
[22:35] <yehudasa> that's really unusual
[22:35] <cmccabe> also, you can hit \B to get a shell
[22:36] <cmccabe> oh yeah... try doing :Rgrep foo from command mode. It's really handy.
[22:37] <cmccabe> or I guess that's called ex mode. Anyway, Rgrep is often handy because it can do regexp
[22:37] <yehudasa> yeah, cool
[22:38] <cmccabe> yehudasa: actually for kernel stuff the cscope plugin is probably the best way to search. I have some nice bindings for inside vim...
[22:39] <cmccabe> yehudasa: if your cursor is on a word, \s does a cscope find uses, and \g does a cscope find definition.
[22:39] <cmccabe> yehudasa: you also need to do :cscope add cscope.out to add the database first
[22:39] <yehudasa> cmccabe: nice!
[22:40] <cmccabe> glad you like it
[22:41] <yehudasa> cmccane: the only annoying thing is that it flashes whenever it refreshes the page, somewhat like an eink display
[22:41] <yehudasa> do you have any idea how to disable it?
[22:41] <cmccabe> yehudasa: when you do pageup/pagedown, or what?
[22:42] <yehudasa> actually it's ok for pageup/pagedown
[22:42] <yehudasa> e.g., when you undo and there's no action
[22:42] <yehudasa> or pageup when at the top of the file
[22:43] <cmccabe> I think that's the visual bell
[22:43] <cmccabe> probably comment out visualbell
[22:43] <yehudasa> I assumed it's the visual bell, but I disabled it and it didn't help
[22:43] <cmccabe> maybe try getting rid of set vb t_vb="."
[22:44] <yehudasa> yeah, that did it
[22:44] <cmccabe> I have only a vague idea of what that does. I just kept tweaking it until it didn't use any kind of bell in the text-mode vim
[22:44] <yehudasa> it's a kindle emulation apparently
[22:44] <cmccabe> lol
[22:45] <cmccabe> I think I did that because it allowed me to set no beep or flash at all
[22:45] <cmccabe> but I don't use the gui so I didn't see that particular flicker issue
[22:46] <yehudasa> it was really annoying, but it's ok now
[22:46] <cmccabe> Apparently you can do stuff like "au! GuiEnter * set vb t_vb="
[22:46] <cmccabe> to run configuration commands only when in gui mode
[22:46] <cmccabe> if it's important
[23:20] * DJLee (82d8d198@ircip3.mibbit.com) has joined #ceph
[23:27] * bcherian (~bencheria@ip-66-33-206-8.dreamhost.com) has joined #ceph
[23:28] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) Quit (Quit: Yoric)
[23:34] * bencherian (~bencheria@ip-66-33-206-8.dreamhost.com) Quit (Ping timeout: 480 seconds)
[23:52] <DJLee> I find that (sequential) writing to OSDs is relatively fast, but (sequential) reading from it, is relatively slow, anyone knows why reading < writing..?

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.