#ceph IRC Log


IRC Log for 2011-04-15

Timestamps are in GMT/BST.

[0:06] * aliguori (~anthony@ Quit (Quit: Ex-Chat)
[0:31] <Tv> yehudasa: hey do you know enough to be able to run the s3 tests by yourself? e.g. the acl fix you just made is now just bringing out the next bug..
[0:31] * sjustlaptop (~sam@ip-66-33-206-8.dreamhost.com) Quit (Quit: Leaving.)
[0:31] <Tv> yehudasa: there's no point for me to be in the loop, filing tickets
[0:32] <yehudasa> Tv: I know how to run that s3-test of yours
[0:32] <yehudasa> Tv: is there a new bug that you see?
[0:33] <Tv> yehudasa: well more like the test now runs one notch further, hits the next problem
[0:33] <Tv> yehudasa: i'm cleaning up the 'fails_on_rgw' decorators based on the bugs fixed, and see some of those just hit the next problem
[0:34] <yehudasa> Tv: I just ran the specific tests that failed on s3-test
[0:35] <Tv> yehudasa: yeah, and those fail for me...
[0:35] <Tv> at commit 22dbbe85186ee71c3f6eba12a98be116f41ca398
[0:35] <yehudasa> Tv: is there a new one that you haven't opened a bug for?
[0:35] <Tv> well i'm just seeing those
[0:35] <Tv> and don't see the value in me filing tickets for you
[0:35] <Tv> when you can run the tests yourself
[0:36] <yehudasa> yeah, thanks
[0:36] <Tv> yehudasa: so.. e.g. test_bucket_acl_grant_userid still fails, just differently
[0:36] <Tv> yehudasa: please confirm you see the new failure
[0:38] <yehudasa> if that's #982, than I'd expect it to fail
[0:38] <yehudasa> *then
[0:38] <Tv> no i mean a *new* failure
[0:38] <yehudasa> yeah..
[0:38] <Tv> something where it's def test_foo(): one(); two() and ealier one() raised an error; now it fails at two()
[0:39] <yehudasa> I looked at #982 yesterday and figured there was some basic issue, so I moved to #981 which had the same issue
[0:39] <yehudasa> but when I looked at #982 I also saw that we don't handle it correctly anyway
[0:39] <yehudasa> but that one should be easy to fix I think
[0:41] <Tv> yehudasa: i just pushed commit 9b39add43f5542e9cb332bf60cb0ab7d20ae1a07 to s3-client.git, see that
[0:41] <Tv> maybe that'll clarify what i mean
[0:44] <cmccabe> yehudasa: just FYI I'm working on #977
[0:44] <cmccabe> yehuda: after I resolve one other thing today
[0:45] <yehudasa> cmccabe: it looks like it needs to call abort_early() with the correct parameter or something like that
[0:46] <cmccabe> yehudasa: I'm looking at it
[0:52] * verwilst (~verwilst@dD576FAAE.access.telenet.be) has joined #ceph
[0:53] <cmccabe> ok, I can confirm that the ordering of config file overrides is 100% correct
[0:53] <sjust> did you by any chance test it on wido's log?
[0:54] <cmccabe> you can test it on that pretty trivially by running cconf
[0:54] <cmccabe> I'll give it a try
[0:55] <cmccabe> what am I testing the overriding of?
[0:55] <sjust> log dir and log file
[0:55] <cmccabe> sjust: they don't appear in the configuration
[0:56] <sjust> ah, he overwrote the older one
[0:56] <sjust> oh well
[0:56] <sjust> sorry about that
[0:56] <cmccabe> sjust: for daemons, they do have a default value
[0:56] <cmccabe> sjust: default log dir is /var/log/ceph
[0:56] <sjust> I know, he uploaded a second conf file after he set all to log locally
[0:57] <sjust> originally [global] contained log dir = and log file =
[0:57] <sjust> to prevent local logging
[0:57] <sjust> he added to some of the individual osd log dir = /var/log/ceph
[0:57] <sjust> and log file
[0:57] <sjust> I think
[0:58] <cmccabe> seems to work fine in my tests.
[0:58] <sjust> ok
[0:58] <sjust> must have been something else
[0:58] <cmccabe> let me know if you have any problems. If you have any doubts, try cconf
[0:58] <sjust> ok
[1:17] * bchrisman (~Adium@sjs-cc-wifi-1-1-lc-int.sjsu.edu) has joined #ceph
[1:39] <yehudasa> Tv: I have some trouble with test_s3.test_access_bucket_private_object_private
[1:39] <Tv> yehudasa: i'll walk over
[1:40] * verwilst (~verwilst@dD576FAAE.access.telenet.be) Quit (Quit: Ex-Chat)
[1:42] * bchrisman (~Adium@sjs-cc-wifi-1-1-lc-int.sjsu.edu) Quit (Quit: Leaving.)
[2:13] * lxo (~aoliva@ has joined #ceph
[2:20] * cmccabe (~cmccabe@ has left #ceph
[2:29] * Meths (rift@customer16161.pool1.unallocated-106-128.orangehomedsl.co.uk) Quit (Ping timeout: 480 seconds)
[2:33] * Meths (rift@ has joined #ceph
[2:36] * Tv (~Tv|work@ip-66-33-206-8.dreamhost.com) Quit (Ping timeout: 480 seconds)
[2:48] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) has joined #ceph
[3:01] * joshd (~joshd@ip-66-33-206-8.dreamhost.com) Quit (Quit: Leaving.)
[3:39] * greglap (~Adium@ip-66-33-206-8.dreamhost.com) Quit (Quit: Leaving.)
[4:08] * greglap (~Adium@ has joined #ceph
[4:56] * sjustlaptop (~sam@ has joined #ceph
[5:03] * Dantman (~dantman@S0106001eec4a8147.vs.shawcable.net) Quit (Ping timeout: 480 seconds)
[5:03] * greglap (~Adium@ Quit (Read error: Connection reset by peer)
[5:12] * greglap (~Adium@cpe-76-170-84-245.socal.res.rr.com) has joined #ceph
[5:26] * lxo (~aoliva@ Quit (Ping timeout: 480 seconds)
[5:53] * Dantman (~dantman@S0106001eec4a8147.vs.shawcable.net) has joined #ceph
[6:42] * lidongyang (~lidongyan@ Quit (Remote host closed the connection)
[6:49] * neurodrone (~neurodron@cpe-76-180-162-12.buffalo.res.rr.com) Quit (Quit: zzZZZZzz)
[6:52] * lidongyang (~lidongyan@ has joined #ceph
[6:54] * Dantman (~dantman@S0106001eec4a8147.vs.shawcable.net) Quit (Remote host closed the connection)
[6:59] * Meths (rift@ Quit (resistance.oftc.net oxygen.oftc.net)
[6:59] * votz (~votz@dhcp0020.grt.resnet.group.upenn.edu) Quit (resistance.oftc.net oxygen.oftc.net)
[6:59] * pruby (~tim@leibniz.catalyst.net.nz) Quit (resistance.oftc.net oxygen.oftc.net)
[6:59] * yehudasa (~quassel@ip-66-33-206-8.dreamhost.com) Quit (resistance.oftc.net oxygen.oftc.net)
[6:59] * cclien_ (~cclien@ec2-175-41-146-71.ap-southeast-1.compute.amazonaws.com) Quit (resistance.oftc.net oxygen.oftc.net)
[6:59] * johnl (~johnl@johnl.ipq.co) Quit (resistance.oftc.net oxygen.oftc.net)
[7:00] * Meths (rift@ has joined #ceph
[7:00] * votz (~votz@dhcp0020.grt.resnet.group.upenn.edu) has joined #ceph
[7:00] * pruby (~tim@leibniz.catalyst.net.nz) has joined #ceph
[7:00] * yehudasa (~quassel@ip-66-33-206-8.dreamhost.com) has joined #ceph
[7:00] * johnl (~johnl@johnl.ipq.co) has joined #ceph
[7:00] * cclien_ (~cclien@ec2-175-41-146-71.ap-southeast-1.compute.amazonaws.com) has joined #ceph
[7:00] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) has joined #ceph
[7:09] * Dantman (~dantman@S0106001eec4a8147.vs.shawcable.net) has joined #ceph
[8:14] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) Quit (Quit: Yoric)
[8:16] * Dantman (~dantman@S0106001eec4a8147.vs.shawcable.net) Quit (Remote host closed the connection)
[8:22] * sjustlaptop (~sam@ Quit (Quit: Leaving.)
[8:24] * sjustlaptop (~sam@adsl-76-208-183-201.dsl.lsan03.sbcglobal.net) has joined #ceph
[8:40] * Dantman (~dantman@S0106001eec4a8147.vs.shawcable.net) has joined #ceph
[8:51] * Dantman (~dantman@S0106001eec4a8147.vs.shawcable.net) Quit (Remote host closed the connection)
[8:57] * Meths (rift@ Quit (Read error: Connection reset by peer)
[8:58] * Meths (rift@ has joined #ceph
[9:06] * verwilst (~verwilst@dD576FAAE.access.telenet.be) has joined #ceph
[9:13] * Dantman (~dantman@S0106001eec4a8147.vs.shawcable.net) has joined #ceph
[9:36] * Yoric (~David@dhcp26-111.enst.fr) has joined #ceph
[9:56] * allsystemsarego (~allsystem@ has joined #ceph
[10:29] * verwilst (~verwilst@dD576FAAE.access.telenet.be) Quit (Quit: Ex-Chat)
[10:40] <chraible> hi @all
[10:40] <chraible> i compiled new kernel on CentOS with CEPH support CONFIG_CEPH_LIB=y
[10:40] <chraible> # CONFIG_CEPH_LIB_PRETTYDEBUG is not set
[10:40] <chraible> CONFIG_CEPH_FS=y
[10:41] <chraible> now when I want to mount a Ceph filesystem with mount -t ceph /mnt/ceph
[10:41] <chraible> I got following error: "mount.ceph: modprobe failed, exit status 127"
[10:42] <chraible> have anyone any ideas whats I do wrong?
[10:51] <chraible> "dmesg | grep ceph" sends following output: http://pastebin.com/rCzWdhVV
[11:00] * Yoric (~David@dhcp26-111.enst.fr) Quit (Quit: Yoric)
[12:25] * Hugh (~hughmacdo@soho-94-143-249-50.sohonet.co.uk) Quit (Quit: Ex-Chat)
[13:12] * midnightmagic (~midnightm@S0106000102ec26fe.gv.shawcable.net) Quit (Ping timeout: 480 seconds)
[13:15] * midnightmagic (~midnightm@S0106000102ec26fe.gv.shawcable.net) has joined #ceph
[14:16] * votz (~votz@dhcp0020.grt.resnet.group.upenn.edu) Quit (Quit: Leaving)
[14:17] * votz (~votz@dhcp0020.grt.resnet.group.upenn.edu) has joined #ceph
[14:55] * aliguori (~anthony@cpe-70-123-132-139.austin.res.rr.com) has joined #ceph
[16:45] * Yoric (~David@dhcp26-111.enst.fr) has joined #ceph
[16:45] * Yoric (~David@dhcp26-111.enst.fr) Quit ()
[17:07] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[17:21] * Hugh (~hughmacdo@soho-94-143-249-50.sohonet.co.uk) has joined #ceph
[17:34] * sjustlaptop (~sam@adsl-76-208-183-201.dsl.lsan03.sbcglobal.net) Quit (Quit: Leaving.)
[18:01] * Yoric (~David@87-231-38-145.rev.numericable.fr) has joined #ceph
[18:02] * Yoric (~David@87-231-38-145.rev.numericable.fr) Quit (Read error: Connection reset by peer)
[18:03] * Yoric (~David@87-231-38-145.rev.numericable.fr) has joined #ceph
[18:05] * bchrisman (~Adium@70-35-37-146.static.wiline.com) has joined #ceph
[18:20] * __jt___ (~james@jamestaylor.org) has joined #ceph
[18:22] * __jt__ (~james@jamestaylor.org) Quit (Ping timeout: 480 seconds)
[18:28] * Tv (~Tv|work@ip-66-33-206-8.dreamhost.com) has joined #ceph
[18:30] <bchrisman> chraible: I think exit 127 is traditionally "can't find" I wouldn't worry about that message as ceph is already loaded from your dmesg output.. are you sure it didn't mount?
[18:31] <Tv> yeah shell claims exit status 127 if the command to run was not found
[18:32] <bchrisman> I've been noticing that error from the "system" call to execute modprobe in ceph's mount program
[18:32] <Tv> oh that's weird
[18:33] <Tv> bchrisman: yet "modprobe ceph; echo $?" works and says 0, when run manually?
[18:33] <bchrisman> There's some environmental dependency… because executing it with sh -c works which is what "system" is supposed to be doing.
[18:34] <bchrisman> Tv: yeah…
[18:34] <bchrisman> I haven't been watching for the issue lately because it doesn't actually cause a problem (ceph module is already loaded or .. welll.. something gets it loaded)
[18:34] <bchrisman> but I'll keep an eye out for it.
[18:35] * sjust (~sam@ip-66-33-206-8.dreamhost.com) Quit (Read error: Connection reset by peer)
[18:50] * Yoric (~David@87-231-38-145.rev.numericable.fr) Quit (Quit: Yoric)
[18:51] * Juul (~Juul@c-76-21-88-119.hsd1.ca.comcast.net) has joined #ceph
[18:55] * sjust (~sam@ip-66-33-206-8.dreamhost.com) has joined #ceph
[18:57] <greglap> sagewk: everybody else: running late today, will be in ~11:20
[18:57] * greglap (~Adium@cpe-76-170-84-245.socal.res.rr.com) Quit (Quit: Leaving.)
[18:57] <sagewk> k
[18:57] <sagewk> let's shoot for 1130 skype
[19:04] * joshd (~joshd@ip-66-33-206-8.dreamhost.com) has joined #ceph
[19:05] * cmccabe (~cmccabe@ has joined #ceph
[19:13] <cmccabe> I think I thought of a way to squash the SIGHUP requirement
[19:14] <cmccabe> when changing logging settings
[19:19] * greglap (~Adium@ has joined #ceph
[19:25] <bchrisman> got samba vfs layer to attempt to mount through libceph… getting those fault messages… can I pass debug flags into libceph like the other clients?
[19:28] <greglap> you used to be able to, but config stuff went through some changes recently...
[19:28] <greglap> cmccabe: is there still a good way to pass debug info into the libraries?
[19:29] <cmccabe> greplap: for librados, use use the conf set/get
[19:29] <bchrisman> greglap: Here's the libceph messages out of samba: http://pastebin.com/ZJZDZxEV
[19:29] <bchrisman> might be working for all I know.. :)
[19:29] <bchrisman> will debug from samba side
[19:30] <bchrisman> but doesn't look right being that it's 'hunting for new mon'
[19:30] <cmccabe> for libceph you can do the same but you have to use g_conf
[19:30] <greglap> cmccabe: how would that work for an external user?
[19:30] <greglap> bchrisman: hmm, the client thinks it's getting repeat messages from the other end and so it's just ignoring them
[19:31] <bchrisman> cmccabe: I guess I can implement calls to g_conf in libceph
[19:31] <cmccabe> greglap: libceph needs some work definitely. I think it might still be parsing argv though
[19:31] <bchrisman> yeah… it does parse argv.. I format that and send it in
[19:31] <greglap> oh, that's still there
[19:31] <cmccabe> bchrisman: the best way would be to implement something like set_conf_val in libceph
[19:31] <bchrisman> so I can put those debug flags in the argv?
[19:31] <greglap> yeah
[19:31] <bchrisman> cmccabe: gotchya…
[19:31] <greglap> it processes it just like the CLI args are processed
[19:31] * pombreda (~Administr@ has joined #ceph
[19:32] <bchrisman> I'll pass argv flags for now… and look at implementing set_conf_val on the side
[19:32] <sjust> greglap: http://tracker.newdream.net/issues/996 should be resolved, right?
[19:32] <greglap> sjust: yeah, think so
[19:33] <greglap> I didn't want to close it until we got confirmation from wido
[19:33] <bchrisman> greglap: I didn't archive the cfuse mega-debug line you sent me a while back… can you give me those args again so I don't have to chase them down? I'll put it up on the wiki in debug.
[19:33] <pombreda> greglap, sage: fwiw, the ceph fs is not mounted on the playground, just in case you do not know already
[19:33] <greglap> pombreda: you ran us out of space on one of our OSDs! :p
[19:33] <pombreda> greglap: :D
[19:33] <bchrisman> (and I'll pass that in to the samba vfs… actually.. I'll put in a samba conf directive that will get pass through to libceph transparently
[19:33] <greglap> bchrisman: uh, I think I usually run with debug_client 20 debug_ms 1
[19:34] <pombreda> greglap: glad to indulge ;)
[19:34] <greglap> depending on what you're doing you might also want debug_objecter or debug objectcacher
[19:34] <bchrisman> greglap: are any of those inclusive of the others or are they mainly orthogonal?
[19:35] <pombreda> greglap: beta testing is about pushing a bit the limits, right? ;)
[19:35] <bchrisman> ios that 'debug_objecter', 'debug_objectcacher', 'debug_client' and 'debug_ms' then?
[19:35] <greglap> pombreda: anyway, Ceph handles out-of-space errors about as well as btrfs does
[19:35] <greglap> I think Sage was going to do something with it today but he just noticed last night and we haven't gotten around to it yet
[19:36] <greglap> bchrisman: they're all separate
[19:36] <greglap> the debug mechanism isn't set up in a way that lets stuff overlap
[19:36] <bchrisman> ok cool
[19:36] <greglap> you're probably not going to want to run with the objecter or objectcacher debugging most of the time though — I only turn it on if I suspect there's something specifically wrong in them
[19:37] <pombreda> greglap, sage: no rush, I'll just monitor when it is back up
[19:37] <bchrisman> yeah… what's the 'objecter' do?
[19:37] <greglap> in this particular case there's something weird going on with the messenger so you probably want more like debug_ms 10
[19:37] <greglap> the objecter is what talks to RADOS
[19:37] <bchrisman> greglap: yup
[19:37] <greglap> librados is a wrapper for the Objecter
[19:37] <bchrisman> ahh okie
[19:38] * pombreda (~Administr@ Quit (Quit: Leaving.)
[19:38] <greglap> the Filer class wraps the Objecter for the client and converts from files to objects, but the Objecter is what handles the actual writes
[19:39] <greglap> ObjectCacher caches objects for the Filer so it doesn't need to go out to the cluster for every read
[19:39] <bchrisman> ahh so that's the underlying mechanism identifying where and in which osd to put the bits themselves.
[19:40] <greglap> yeah
[19:41] <greglap> Filer's pretty thin though; it just needs to convert from inode numbers and offsets to stripe locations
[19:42] * Juul (~Juul@c-76-21-88-119.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[19:42] <bchrisman> thin but extremely core/critical code :)
[19:42] <bchrisman> bug in that would be nasty :)
[19:43] <greglap> yeah
[19:43] <greglap> like I said it's pretty stable though
[19:43] <greglap> 4 commits so far this year and 8 last year, and most of them are interface changes
[19:45] <bchrisman> yeah… backend on this is rock solid… impressive in resiliency in comparison to GPFS… haven't verified performance though..
[19:49] <Tv> a single dbench against two osds gets ~1.5MB/s.. 4 dbenches against 1 osd get ~0.60 MB/s each.. 4*0.60=2.4MB/s.. confused now
[19:50] <sjust> Tv: more concurrent IO threads?
[19:50] <Tv> sjust: yeah but why is 1 osd faster than 2..
[19:50] <sjust> it's not the 1 osd vs 2 osd, it's probably 1 dbench vs 4 dbench
[19:51] <Tv> sjust: dbench is internally 2 threads always, too...
[19:51] <Tv> yeah, i guess
[19:51] <greglap> doesn't matter, it's the latency
[19:51] <sjust> greglap: hmm?
[19:51] <greglap> what you were saying — one dbench on 2 OSDs has way fewer thread than 4 dbench on 1 OSD
[19:51] <greglap> so the latency hurts it more
[19:52] <sjust> greglap: ah, thats what I meant
[19:52] <bchrisman> does cfuse respect the client section in /etc/ceph/ceph.conf then? same with libceph?
[19:52] <cmccabe> greplap: latency between what and what?
[19:53] <Tv> they speak English in what?
[19:53] <greglap> bchrisman: yes
[19:53] <greglap> cmccabe: dbench will only run one IO at a time by default, so the latency for an IO request on Ceph to finish makes it pretty slow...
[19:54] <cmccabe> greplap: so latency between initiating the I/O and getting the completion
[19:54] <greglap> it's like we were talking about yesterday with librbd being slow because it's all synchronous IO, although not quite so bad since dbench doesn't have to wait for the commits I don't think
[19:54] <Tv> unless something does explicit syncs, i would expect dbench to be IO bound though
[19:54] <Tv> i don't know whether it syncs or not
[19:54] <greglap> Tv: I think sync is an option
[19:54] <greglap> but it's still only going to dispatch one IO at a time per thread
[19:55] <greglap> and that single IO won't return from the Ceph client until it gets an ack from the OSDs
[19:55] <Tv> greplap: i don't see anything passing in the -s as in synchronous option
[19:56] <greglap> no, I mean even without sync it's still only going to dispatch one IO at a time
[19:56] <Tv> hmm
[19:56] <cmccabe> greplap: I remember discussing ACK that meant "the data has been received" vs. ACKs that meant "the data is on the disk"
[19:56] <cmccabe> greplap: I'm assuming that we're going with option #2
[19:56] <Tv> greglap: i thought that'd hit the local vfs and get flushed at some point
[19:56] <greglap> on a local FS that single IO would take the time required to send the data from memory to disk cache
[19:57] <Tv> greglap: but but... if i do a write, nothing gets sent to disk quite yet; it's buffered in page cache
[19:57] <greglap> here it's the time it takes to send the data from memory across the network to an OSD, then to another OSD, then to get an ack from second OSD to first OSD, then an ack from first OSD to client
[19:57] <greglap> oh, right
[19:57] <greglap> ugh, sorry, still really tired
[19:58] <greglap> but it's got to be something about that, nothing else should make it slow
[19:58] <greglap> what's dbench do again?
[19:58] <Tv> pagecache converts the inherently synchronous write syscalls into a pipelined stream of data
[19:58] <greglap> is it getting hung up on inode creations, maybe?
[19:58] <cmccabe> tv: a good way to put it
[19:58] <greglap> I forget which fs test does what
[19:58] <cmccabe> greglap: bonnie?
[19:58] * morse_ (~morse@supercomputing.univpm.it) Quit (Remote host closed the connection)
[19:59] <Tv> greglap: looks like it hits a single file for all operations
[19:59] <cmccabe> greglap: I remember sage saying that one FS test just boiled down to how fast you could shovel out inodes
[19:59] <Tv> for one thread
[20:01] <Tv> ahhh found it.. dbench has an ascii file with operations to perform
[20:01] <greglap> what's it doing?
[20:02] <Tv> it looks like a filesystem trace from an NT machine
[20:03] <Tv> it creates files in a dir and writes to a bunch of offsets, etc
[20:03] <Tv> NTCreateX "\clients\client1\~dmtmp\COREL\ROSE.CPT" 0x40 0x2 9966 NT_STATUS_OK
[20:03] <Tv> WriteX 9966 65534 1 1 NT_STATUS_OK
[20:03] <Tv> WriteX 9966 0 65536 65536 NT_STATUS_OK
[20:03] <Tv> WriteX 9966 65536 65536 65536 NT_STATUS_OK
[20:03] <Tv> WriteX 9966 131072 65536 65536 NT_STATUS_OK
[20:03] <Tv> WriteX 9966 196608 63488 63488 NT_STATUS_OK
[20:03] <Tv> Close 9966 NT_STATUS_OK
[20:03] <Tv> that's representative though short
[20:03] <Tv> 25MB of those events are played back
[20:03] <Tv> in several threads
[20:04] <Tv> so yeah it might be bottlenecked by inode operations
[20:04] <Tv> also, CorelDraw, yuck
[20:05] <greglap> ah, the man page says it's supposed to simulate the fs workload of a Samba server
[20:05] <greglap> serving 60/150 clients
[20:05] <greglap> so yeah, probably metadata ops
[20:05] <greglap> which the the threading would help hide latency on
[20:08] <Tv> yeah funny that it reports its results in MB/s
[20:08] * greglap (~Adium@ Quit (Read error: Connection reset by peer)
[20:08] <Tv> heh.. are the tunnels his train goes through?-)
[20:08] <Tv> *there
[20:09] <yehudasa> Tv: fixed a few issues, there are now 3 errors, 4 failures out of 45, not sure whether there's any regression
[20:09] <Tv> run the test with -a '!fails_on_rgw'
[20:09] <Tv> all of those succeeded earlier
[20:11] <yehudasa> right, no regressions
[20:11] -magnet.oftc.net- *** Looking up your hostname...
[20:11] -magnet.oftc.net- *** Checking Ident
[20:11] -magnet.oftc.net- *** No Ident response
[20:11] -magnet.oftc.net- *** Found your hostname
[20:11] * CephLogBot (~PircBot@rockbox.widodh.nl) has joined #ceph
[20:11] <yehudasa> so most of the ones marked as fail work now
[20:12] <Tv> yehudasa: i took out some of them in the last few commits; feel free to take out the ones that are now unneeded
[20:12] <Tv> yehudasa: ideally, there'll be none left..
[20:12] <cmccabe> yehudasa: you're working on master?
[20:12] <yehudasa> cmccabe: yes
[20:12] <cmccabe> yehudasa: ok great, that means my changes of yesterday are good then
[20:12] <yehudasa> cmccabe: I modified your changes a bit actually
[20:12] <cmccabe> yehudasa: as far as not causing regressions
[20:13] <yehudasa> cmccabe: there was a mixup between the s3 error code and the s3 message
[20:13] <yehudasa> it also prevented of sending a successful http response that was not 200 (e.g., 204)
[20:14] <cmccabe> yehudasa: looks ok
[20:14] <cmccabe> yehudasa: yeah, I had not considered 204
[20:14] <cmccabe> yehudasa: actually, just learned about 204... weird thing
[20:15] <yehudasa> we'll need to look at the swift implementation too.. we probably broke a few things there too
[20:15] <cmccabe> yehudasa: should we perhaps consider all 2XX as success?
[20:15] <Tv> cmccabe: that's the definition of 2xx
[20:15] <yehudasa> yes
[20:16] <yehudasa> Tv: is there an easy way to know which tests succeeded?
[20:17] * morse (~morse@supercomputing.univpm.it) has joined #ceph
[20:17] <Tv> yehudasa: probably -v is what you want
[20:17] <Tv> yehudasa: there's more programmatic output too but -v is often good enough to eyeball
[20:17] <yehudasa> yeah, thanks
[20:19] <cmccabe> yehudasa: so we could change rgw_err::is_clear to be like >= 200 and <= 299
[20:19] <cmccabe> yehudasa: not that it really matters if we don't use those other 2XX codes
[20:19] <yehudasa> cmccabe: yeah, sure
[20:20] <cmccabe> yehudasa: I guess you can see what I was going for in that change. I was trying to make it easy to go from our internal err_no code to a nice HTTP error code and message
[20:21] <cmccabe> yehudasa: at the same time, I wanted to let certain code paths continue to set the error code and message manually in case that was necessary
[20:22] <cmccabe> yehudasa: If nobody needs to set the message manually, we could just put all errors in the RGW_HTML_ERRORS table and then every error would be a nice easy to understand int :)
[20:23] <yehudasa> cmccabe: right, probably a needed cleanup. We should try to make a habit of running s3-tests to make sure there are no regressions after introducing any gateway changes
[20:23] <cmccabe> yehudasa: yeah
[20:24] * greglap (~Adium@ip-66-33-206-8.dreamhost.com) has joined #ceph
[20:25] <Tv> yehudasa: and please make sure new functionality is covered by tests
[20:25] <Tv> yehudasa: the current selection of tests is just what Wes got together in the time he had; all I did was clean up
[20:26] <cmccabe> tv: how do you run the tests? is there a wiki page?
[20:26] <Tv> cmccabe: have you read the readme?
[20:29] <cmccabe> tv: taking a look
[20:33] -kilo.oftc.net- *** Looking up your hostname...
[20:33] -kilo.oftc.net- *** Checking Ident
[20:33] -kilo.oftc.net- *** No Ident response
[20:33] -kilo.oftc.net- *** Found your hostname
[20:33] * CephLogBot (~PircBot@rockbox.widodh.nl) has joined #ceph
[20:35] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) has joined #ceph
[20:50] <stefanha> Does it make sense to run RBD without replication? For example in order to reduce fragmentation across a number of machines that have local storage.
[20:51] * Juul (~Juul@slim.dhcp.lbl.gov) has joined #ceph
[20:53] <stefanha> (I'm thinking of adding all hosts into a pool, no replication, and being able to create RBD devices that are larger than the storage a single host has)
[20:54] <wido> stefanha: Sure, you can do that
[20:54] <wido> but the failure of just one device will cause a total failure
[20:54] <wido> imho RBD/Ceph isn't really designed to do so
[21:08] <stefanha> wido: Is there a way to replace the missing device?
[21:09] * gregaf (~Adium@ip-66-33-206-8.dreamhost.com) Quit (Remote host closed the connection)
[21:10] <sjust> wido: it seems that only atom0 has logs in /var/log/ceph
[21:11] <sjust> oops, nvm
[21:12] <wido> sjust: ok :) I didn't get a chance yet to check the weirdness I saw yesterday
[21:12] <sjust> wido: nvm found some logs with a backtrace!
[21:12] <wido> but while playing with cconf I saw this in the cconf help
[21:12] <wido> cconf --name client.cconf -c /etc/ceph/ceph.conf -t mon -i 0 'mon addr'
[21:12] <wido> that's not valid, -t and -i are not valid arguments
[21:13] <wido> this should print a 'mon addr' value from the conf. Now, I'd like to fix the docs/manpage, but I can't figure out the options which are right now
[21:13] * gregaf (~Adium@ip-66-33-206-8.dreamhost.com) has joined #ceph
[21:13] <wido> cconf --name client.cconf -c /etc/ceph/ceph.conf -s mon 'mon addr'
[21:13] <wido> that doesn't work, unless I supply '-n mon.dga'
[21:14] <wido> well, it works, but doesn't print anything
[21:14] <cmccabe> wido: yeah, that usage need to be fixed
[21:14] <cmccabe> wido: you probably want something like cconf --name mon.0 -c /etc/ceph/ceph.conf 'mon addr'
[21:16] <wido> but then you are still supplying a mon's name, in this case .0
[21:16] <cmccabe> name=type.id
[21:16] <cmccabe> 0 is an id
[21:17] <wido> well, that is broken then as well ;) -n mon.dga works, mon.0 doesn't
[21:17] <cmccabe> what is dga
[21:17] <wido> oh, my monitor name
[21:17] <wido> [mon.dga] in the ceph.conf
[21:18] <cmccabe> well... sounds reasonable then
[21:18] <cmccabe> actually we have a habit of naming monitors a,b,c,d,etc.
[21:18] <wido> Yes, but the example says: "Find out if there is a 'mon addr' defined in /etc/ceph/ceph.conf"
[21:19] <cmccabe> I don't know what happens if you name a monitor a multi-letter combo. It might not be good though.
[21:19] <gregaf> it should be fine
[21:19] <gregaf> if it's not that's a bug
[21:20] <cmccabe> I vaguely remember a few places doing something cringe-worthy with the digit they found after the period
[21:20] <cmccabe> I haven't looked at it in a while though
[21:20] <cmccabe> that is all in the monitor code itself, not in the configuration code.
[21:20] <cmccabe> id's just a string to the config code.
[21:21] <cmccabe> wido: yeah, I will clarify the example a bit.
[21:21] <cmccabe> wido: it could be reworded better of course.
[21:21] <cmccabe> brb, lunch
[21:22] * allsystemsarego (~allsystem@ Quit (Quit: Leaving)
[21:34] <Tv> wido: there's a few tests for cconf that are known to work, in src/test/cli/cconf/
[21:34] <Tv> wido: e.g. option.t looks like what you want
[21:34] <Tv> well, those don't use the type.id magic section name thing
[21:35] <Tv> probably because it was a mess when i added the tests.. :-/
[21:36] <wido> Tv: What I mean, the current example ays "Find out if there is a 'mon addr' ", but it doesn't seem possible without actually naming a section
[21:37] <Tv> wido: ./cconf --name mon.a --lookup 'mon addr' ?
[21:38] <Tv> wido: i'm not sure i understand
[21:38] <Tv> wido: there's no "mon addr" without talking about one specific mon
[21:41] <wido> a simple bash script which wants to extract 'mon addr' from a conf without the knowledge of the monitor names won't work
[21:41] <wido> you would have to do cconf -l mon first, and use that output afterwards
[21:42] <Tv> wido: you mean "get *all* mon addr's"?
[21:42] <Tv> wido: there's no one mon addr
[21:42] <wido> Tv: yes, that's what I mean. But then, the example is wrong :-)
[21:42] <wido> try: cconf --help
[21:43] <Tv> ./cconf --list-sections mon.|while read s; do ./cconf -s "$s" --lookup 'mon addr'; done
[21:43] <Tv> wido: man pages are usages are really wrong, most of the time, right now
[21:43] <Tv> wido: i want to fix all that, but good tests are more critical, right now
[21:43] <wido> Tv: Correct :-) I'll create some patch with a few better examples and fix the man
[21:44] <Tv> wido: please add tests in src/test/cli/ so we don't keep screwing up
[21:44] <wido> ok, not so familair with autotest, but I'll give it a try
[21:44] <Tv> it's not autotest
[21:44] <Tv> wido: it's a thing called "cram", just run "make check" at toplevel
[21:45] <Tv> actually even that loop i pasted is flawed, because you could set mon addr in [mon] with variables in it :(
[21:45] <wido> k. But still, some good manpages are useful for new users, so I'll give them some attention
[21:46] <Tv> ./cconf --list-sections mon.|sed 's/^mon\.//'|while read id; do ./cconf --name="mon.$id" --lookup 'mon addr'; done
[21:47] <Tv> apparently that is the way to get [mon] and [global] searched too
[21:47] <Tv> oh the sed is unnecessary now
[21:47] <Tv> ./cconf --list-sections mon.|while read name; do ./cconf --name="$name" --lookup 'mon addr'; done
[21:47] <Tv> and *now* that will fail if you put [mona] in config, which is still allowed and one of my pet peeves... :-|
[21:47] * Tv looks at sagewk
[21:48] <sagewk> my only goal originally was to make [mon0] still work. [mona] doesn't need to.. and really i'm happy dropping that if everyone else is
[21:48] <Tv> mon0 is just as much a special case
[21:49] <sagewk> dropping [mon0] too i mean
[21:49] <Tv> yay i can rank up my negative code balance ;)
[21:49] <Tv> anyone want to really defend not having period in the type.id names? yehudasa gregaf?
[21:49] <bchrisman> I need to implement realpath() in libceph… is there something I can call to get that already in the code… I notic ea few refs to realpath in source… one in mount...
[21:50] <bchrisman> something in the client with path_walk?
[21:50] <Tv> bchrisman: why does libceph need realpath?
[21:51] <bchrisman> hmm.. maybe it doesn't.
[21:51] <bchrisman> I was implementing that in the samba vfs module
[21:51] <bchrisman> yeah.. I think it will
[21:51] <sagewk> what does realpath() do? fd -> path?
[21:51] <bchrisman> it calls it on the export directory right now.. but I didn't implement so I return ENOTSUP
[21:51] <Tv> sagewk: rewrite path to get rid of all symlinks
[21:51] <Tv> sagewk: string to string
[21:52] <sagewk> ah
[21:52] <bchrisman> takes a relative path
[21:52] <Tv> it's a fugly wart of posix
[21:52] <Tv> oh and don't confuse realpath with abspath
[21:52] <bchrisman> yeah.. smb vfs is going to need it in libceph… no real way around it I think.
[21:52] <Tv> realpath is expensive and can trigger unwanted behavior
[21:53] <bchrisman> yeah.. not sure how much it calls it.. but it's calling it on mount.. which is reasonable.
[21:53] <bchrisman> it's goign ;through vfs.. so it must be expecting the filesystem to support it.
[21:53] <Tv> bchrisman: realpath is not a filesystem operation
[21:54] <sagewk> just return the argument for now i'd say?
[21:54] <bchrisman> heh.. yeah.. actually that's what I did first.. but undid it because I thought it was too kludgy… at least I know it's getting called then.. will do...
[21:55] <Tv> the reason mount does realpath is to canonicalize common funky typos & shell script variable expansion like /mnt//././
[21:56] <Tv> the symlink following is somewhat usable too, so you know that with symlinks /a -> /c and /b -> /c, mount /a/x is really mounting /c/x
[21:56] <bchrisman> yeah
[21:56] <Tv> but normal programs really don't need/want realpath
[21:56] <Tv> samba may need something special
[21:56] <Tv> but that cost should be in samba ;)
[22:00] <bchrisman> heh
[22:05] <yehudasa> Tv: we can probably omit the period if there's no name after it and treat it as a wildcard
[22:06] <Tv> yehudasa: i'm sorry i can't parse that
[22:06] <Tv> i'm talking about how ceph.conf now handles [osd.0] and [osd0] as the same thing
[22:06] <Tv> and that makes programmatic handling of the config harder in many places
[22:09] <cmccabe> realpath also can't handle path names longer than 255 characters
[22:09] <cmccabe> that's a real problem since PATH_MAX is 4096
[22:10] <Tv> cmccabe: you mean myrealpath, as it is in mount.ceph
[22:10] <Tv> that's just an implementation bug
[22:10] <Tv> but the whole concept is full of traps
[22:10] <Tv> i've actually seen cases where the input is <PATH_MAX but the output, with symlinks resolved, must be >PATH_MAX
[22:11] <Tv> bag of fail to be avoided unless you *know* you are special and need it
[22:11] <cmccabe> actually, now that I look at it, it looks like it has PATH_MAX as the limit
[22:12] <cmccabe> it looks like we want it for mtab
[22:13] <cmccabe> as far as dropping osd0/mona support, that's trivial
[22:13] <Tv> the world will be a happier place once mtab finally dies
[22:13] <cmccabe> it's probably like a 5-line change, not counting the test code that would need to change
[22:14] <cmccabe> to be honest, I'm not sure the gain would be worth the pain, though
[22:14] <cmccabe> people get frustrated when their old configurations stop working
[22:15] <cmccabe> anyway, I can do it in a branch if anyone really feels strongly about it
[22:15] <wido> imho you could change as much as you want until 1.0
[22:15] <Tv> wido: yup, that's how i feel
[22:16] <Tv> better now than later
[22:16] <wido> currently almost every user is in a test-phase
[22:16] <Tv> and the more we can simplify the setup & configuration, the better
[22:16] <wido> most users have their Ceph cluster running for about 2 weeks
[22:16] <cmccabe> true
[22:16] <wido> and come by a few months later again to check the status
[22:16] <wido> and build a new one
[22:17] <wido> but I'm going afk, I'll give the manpages some attention tomorrow, see if the examples still work, etc, etc
[22:17] <wido> ttyl
[22:18] <sagewk> tv, cmccabe as for [mon0], let's kill it now
[22:18] <cmccabe> sagewk: k
[22:19] * lxo (~aoliva@ has joined #ceph
[22:27] <Tv> yay
[22:27] <Tv> cmccabe: it seems you have a better grasp of what's needed
[22:28] <Tv> cmccabe: as far as tests, nothing i wrote ever relied on that, as i've wanted to kill it for a while now ;)
[22:28] <cmccabe> tv: I'll kill it in just a moment, want to finish this commit
[22:28] <Tv> clitests / autotest etc are all careful to use name.id
[22:28] <cmccabe> tv: yeah, I forget if I put it in any tests or not
[22:28] <cmccabe> tv: and obviously we need to audit the example configs / help messages to make sure they don't lead people astray
[22:28] <Tv> yeah, though they're already wrong ;)
[22:30] <cmccabe> tv: in this case I don't think past precedent is a good guide
[23:03] <cmccabe> ok, old-style section names are history.
[23:03] <cmccabe> will check the docs for bad examples in a minute, brb
[23:04] * lxo (~aoliva@ Quit (Ping timeout: 480 seconds)
[23:08] * verwilst (~verwilst@dD576FAAE.access.telenet.be) has joined #ceph
[23:27] * yehuda (~yehuda@99-48-179-68.lightspeed.irvnca.sbcglobal.net) has joined #ceph
[23:28] * yehuda is now known as yehuda_hm
[23:30] * lxo (~aoliva@ has joined #ceph

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.