#ceph IRC Log


IRC Log for 2011-06-03

Timestamps are in GMT/BST.

[0:11] * verwilst (~verwilst@dD576F882.access.telenet.be) has joined #ceph
[0:12] * Juul_ (~Juul@slim.dhcp.lbl.gov) Quit (Remote host closed the connection)
[0:14] <yoshi> alright. installed and restarted. confirmed rbd still hangs.
[0:15] <sjust> ok, can you post the osd log?
[0:22] <yoshi> sjust: sure. here is a full dump removing heartbeat. https://gist.github.com/7e627e7074a0d18937be
[0:25] <joshd> yoshi: did your logs rotate already? there's no startup there - the output after 'load_pgs' is what we're after
[0:26] <yoshi> joshd: my bad. probably skipped the beginning. just a sec.
[0:27] * aliguori (~anthony@cpe-70-123-132-139.austin.res.rr.com) Quit (Quit: Ex-Chat)
[0:30] <yoshi> joshd: here is the start up log. https://gist.github.com/346e7dd8c63cb3359ae8
[0:32] * greglap1 (~Adium@ip-66-33-206-8.dreamhost.com) Quit (Quit: Leaving.)
[0:36] <joshd> yoshi: are you sure you got the filestore_debugging packages, and not the regular 0.28.2 packages?
[0:38] <yoshi> joshd: I removed the regular deb src but let me confirm from apt-cache.
[0:39] <yoshi> joshd: https://gist.github.com/e20f07ba81c3efd8ad58
[0:43] * greglap (~Adium@ip-66-33-206-8.dreamhost.com) has joined #ceph
[0:44] <joshd> ah, I think the problem may be xfs after all
[0:44] <yoshi> joshd: really? why do you think so?
[0:45] * aliguori (~anthony@cpe-70-123-132-139.austin.res.rr.com) has joined #ceph
[0:46] <joshd> we're detecting directories using a dirent's d_type, which may not be supported on xfs
[0:47] <yoshi> joshd: ah. ic. but interesting that it seemed to work at beginning.
[0:47] <yoshi> joshd: then suddenly stopped.
[0:48] <joshd> well, this code is only run when the osd restarts and there's existing data
[0:48] <yoshi> that makes sense.
[0:48] <joshd> we can probably fix it to work on xfs pretty easily
[0:49] <yoshi> cool. let me try once the patch is ready. it was interesting to investigate. btw, which fs will you use for production system? btrfs or ext4?
[0:50] * greglap (~Adium@ip-66-33-206-8.dreamhost.com) Quit (Quit: Leaving.)
[0:51] <joshd> I'm pretty sure btrfs, but sagewk1 knows for sure
[0:52] <Tv> Currently, only some file systems (among them: Btrfs, ext2, ext3, and ext4) have full
[0:52] <Tv> support for returning the file type in d_type. All applications must properly handle a
[0:52] <Tv> return of DT_UNKNOWN.
[0:52] <Tv> says getdents(2)
[0:53] <Tv> i'm still disturbed by seeing >>12 etc in the code, that looks odd
[0:54] <joshd> yeah, that's what made me look it up
[0:55] <Tv> ooh ooh http://www.delorie.com/gnu/docs/glibc/libc_270.html
[0:55] <Tv> those smell like proper macros for the >>12 stuff
[0:55] <Tv> or just use DT_REG etc
[0:55] <Tv> instead of ST_REG
[0:55] <Tv> like the lovely manpage shows
[0:56] <Tv> s/ST_REG/S_IFREG/
[0:58] <yoshi> joshd, sjust, Tv: thanks for your help. glad that we found a clear issue.
[0:58] <Tv> http://tracker.newdream.net/issues/1135
[0:59] <Tv> joshd: i trust you'll deal with the DT_UNKNOWN issue
[0:59] <Tv> i have a meeting starting, somewhere...
[0:59] <joshd> Tv: yeah, I'll fix it
[1:00] <joshd> yoshi: thanks for helping us debug it!
[1:01] <yoshi> joshd: NP. It was fun! ttyl.
[1:01] <joshd> yoshi: see ya
[1:02] * yoshi (~yoshi@KD027091032046.ppp-bb.dion.ne.jp) has left #ceph
[1:06] * mtk (~mtk@ool-182c8e6c.dyn.optonline.net) Quit (Ping timeout: 480 seconds)
[1:06] * votz (~votz@dhcp0020.grt.resnet.group.upenn.edu) has joined #ceph
[1:09] * Nadir_Seen_Fire (~dantman@S0106001731dfdb56.vs.shawcable.net) has joined #ceph
[1:11] * greglap (~Adium@ip-66-33-206-8.dreamhost.com) has joined #ceph
[1:12] * mtk (~mtk@ool-182c8e6c.dyn.optonline.net) has joined #ceph
[1:14] * DanielFriesen (~dantman@S0106001731dfdb56.vs.shawcable.net) Quit (Ping timeout: 480 seconds)
[1:21] * yoshi (~yoshi@KD027091032046.ppp-bb.dion.ne.jp) has joined #ceph
[1:32] * yoshi (~yoshi@KD027091032046.ppp-bb.dion.ne.jp) Quit (Remote host closed the connection)
[2:08] * Tv (~Tv|work@ip-66-33-206-8.dreamhost.com) Quit (Ping timeout: 480 seconds)
[2:09] * neurodrone (~neurodron@cpe-76-180-162-12.buffalo.res.rr.com) has joined #ceph
[2:14] * verwilst (~verwilst@dD576F882.access.telenet.be) Quit (Quit: Ex-Chat)
[2:50] * cmccabe (~cmccabe@c-24-23-254-199.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[3:02] * bchrisman (~Adium@70-35-37-146.static.wiline.com) Quit (Quit: Leaving.)
[3:46] * joshd (~joshd@ip-66-33-206-8.dreamhost.com) Quit (Quit: Leaving.)
[3:49] * greglap (~Adium@ip-66-33-206-8.dreamhost.com) Quit (Quit: Leaving.)
[4:00] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) has joined #ceph
[4:11] * greglap (~Adium@mobile-198-228-209-038.mycingular.net) has joined #ceph
[4:25] * Nadir_Seen_Fire (~dantman@S0106001731dfdb56.vs.shawcable.net) Quit (Ping timeout: 480 seconds)
[4:50] * greglap (~Adium@mobile-198-228-209-038.mycingular.net) Quit (Ping timeout: 480 seconds)
[5:03] * ajm- (adam@adam.gs) has joined #ceph
[5:03] * ajm (adam@adam.gs) Quit (Read error: Connection reset by peer)
[5:14] * hachiya (~hachiya@encyclical.net) Quit (Remote host closed the connection)
[5:14] * hachiya (~hachiya@encyclical.net) has joined #ceph
[5:37] * Dantman (~dantman@S0106001731dfdb56.vs.shawcable.net) has joined #ceph
[5:54] * greglap (~Adium@cpe-76-170-84-245.socal.res.rr.com) has joined #ceph
[6:02] * greglap (~Adium@cpe-76-170-84-245.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[6:45] * yehuda_hm (~yehuda@99-48-179-68.lightspeed.irvnca.sbcglobal.net) has joined #ceph
[6:57] * Dantman (~dantman@S0106001731dfdb56.vs.shawcable.net) Quit (Ping timeout: 480 seconds)
[7:57] * neurodrone (~neurodron@cpe-76-180-162-12.buffalo.res.rr.com) Quit (Quit: zzZZZZzz)
[8:48] * jbd (~jbd@ks305592.kimsufi.com) has joined #ceph
[9:02] * allsystemsarego (~allsystem@ has joined #ceph
[9:07] * greglap (~Adium@cpe-76-170-84-245.socal.res.rr.com) has joined #ceph
[9:10] * jbd (~jbd@ks305592.kimsufi.com) has left #ceph
[9:22] * votz (~votz@dhcp0020.grt.resnet.group.upenn.edu) Quit (Quit: Leaving)
[9:34] * votz (~votz@dhcp0020.grt.resnet.group.upenn.edu) has joined #ceph
[9:36] * votz (~votz@dhcp0020.grt.resnet.group.upenn.edu) Quit ()
[9:37] * votz (~votz@dhcp0020.grt.resnet.group.UPENN.EDU) has joined #ceph
[11:56] * squig (~bendeluca@soho-94-143-249-50.sohonet.co.uk) Quit (Read error: Connection reset by peer)
[12:05] * macana (~ml.macana@ Quit (Ping timeout: 480 seconds)
[12:05] * macana (~ml.macana@ has joined #ceph
[12:10] * mtk (~mtk@ool-182c8e6c.dyn.optonline.net) Quit (Ping timeout: 480 seconds)
[12:15] * mtk (~mtk@ool-182c8e6c.dyn.optonline.net) has joined #ceph
[13:35] * jbd (~jbd@ks305592.kimsufi.com) has joined #ceph
[14:09] * ico (~ico@isper-224-23.isper.sk) has joined #ceph
[14:09] <ico> helo
[14:10] <ico> i have a problem compiling ceph client
[14:10] <ico> ceph-0.28.2 on slackware 13.37, 64 bit
[14:10] <ico> i saw mentioned it here in logs
[14:10] <ico> but not the solution :)
[14:10] <ico> configure: error: no suitable crypto library found
[14:11] <ico> does anyone know which library could solve this?
[14:11] <ico> thanks
[14:27] * hijacker (~hijacker@ Quit (Read error: Connection reset by peer)
[14:27] * hijacker (~hijacker@ has joined #ceph
[14:30] <nms> ico: nss solved that problem for me on Gentoo, but there might be alternatives
[14:32] * hijacker (~hijacker@ Quit (Read error: Connection reset by peer)
[14:33] * hijacker (~hijacker@ has joined #ceph
[14:37] <ico> nms: you mean mozilla's nss package?
[14:37] <nms> ico: i do
[14:55] * morse (~morse@supercomputing.univpm.it) Quit (Remote host closed the connection)
[15:07] * squig (~bendeluca@soho-94-143-249-50.sohonet.co.uk) has joined #ceph
[15:14] * hijacker (~hijacker@ Quit (Read error: Connection reset by peer)
[15:14] * hijacker (~hijacker@ has joined #ceph
[15:16] * damoxc (~damien@andromeda.digitalnetworks.co.uk) Quit (Quit: WeeChat 0.3.4)
[15:29] * jbd (~jbd@ks305592.kimsufi.com) has left #ceph
[15:35] * hijacker (~hijacker@ Quit (Read error: Connection reset by peer)
[15:35] * hijacker (~hijacker@ has joined #ceph
[15:54] * andret (~andre@pcandre.nine.ch) Quit (Remote host closed the connection)
[15:57] * jbd (~jbd@ks305592.kimsufi.com) has joined #ceph
[16:05] * jbd (~jbd@ks305592.kimsufi.com) has left #ceph
[16:28] * neurodrone (~neurodron@cpe-76-180-162-12.buffalo.res.rr.com) has joined #ceph
[16:44] * morse (~morse@supercomputing.univpm.it) has joined #ceph
[16:45] * morse (~morse@supercomputing.univpm.it) Quit (Remote host closed the connection)
[16:48] * morse (~morse@supercomputing.univpm.it) has joined #ceph
[17:07] * jbd (~jbd@ks305592.kimsufi.com) has joined #ceph
[17:18] * Tv (~Tv|work@ip-66-33-206-8.dreamhost.com) has joined #ceph
[17:28] * ico (~ico@isper-224-23.isper.sk) Quit (Ping timeout: 480 seconds)
[17:40] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[17:42] * greglap (~Adium@cpe-76-170-84-245.socal.res.rr.com) Quit (Quit: Leaving.)
[17:52] * greglap (~Adium@mobile-198-228-211-072.mycingular.net) has joined #ceph
[17:53] * jbd (~jbd@ks305592.kimsufi.com) has left #ceph
[18:08] * jbd (~jbd@ks305592.kimsufi.com) has joined #ceph
[18:30] * bchrisman (~Adium@70-35-37-146.static.wiline.com) has joined #ceph
[18:50] * greglap (~Adium@mobile-198-228-211-072.mycingular.net) Quit (Ping timeout: 480 seconds)
[18:58] * cmccabe (~cmccabe@ has joined #ceph
[19:00] * joshd1 (~joshd@ip-66-33-206-8.dreamhost.com) has joined #ceph
[19:00] * greglap (~Adium@ip-66-33-206-8.dreamhost.com) has joined #ceph
[19:31] <cmccabe> so I'm confused about g_conf.auth_supported
[19:32] <cmccabe> do we really want it to default to "none"?
[19:36] <gregaf> using auth requires extra work on the user's part
[19:36] <cmccabe> so basically if it's set to something other than none, it will error out unless the setup is correct
[19:36] <gregaf> we can talk about changing it, but it makes things easier on first-time users
[19:37] <cmccabe> ic
[19:37] <gregaf> and I don't think that we want to make our defaults work for a production cluster since that's too wide a spread
[19:38] <gregaf> so I for one prefer to make the defaults as easy to use as possible so there's less of a learning curve for people to play around with it
[19:39] <Tv> gregaf: i would argue that 1) crypto should be very hard to disable 2) mkcephfs is meant to make key management simpler
[19:39] * aliguori (~anthony@cpe-70-123-132-139.austin.res.rr.com) Quit (Quit: Ex-Chat)
[19:39] <Tv> the monmap/osdmap stuff is already difficult enough, users need that wrapped in comfy cushions anyway
[19:39] <gregaf> *shrug*
[19:39] <gregaf> I'm not set on it or anything, and if crypto management isn't a problem anymore then I don't have any argument at all :)
[19:40] <gregaf> it just seems that every time somebody turns on crypto everything breaks
[19:40] <cmccabe> haha
[19:40] <gregaf> because it's still difficult to use
[19:40] <cmccabe> well, I just did some refactoring on g_keyring
[19:40] <Tv> yes it should be improved
[19:40] <Tv> no it should not be disabled ;)
[19:40] <cmccabe> what should I test
[19:40] <cmccabe> does mkcephfs have like a crypto enabled mode or something?
[19:41] <cmccabe> if so, I'm not seeing it...
[19:41] <gregaf> look through the wiki?
[19:41] <Tv> i see crypto things in mkcephfs, unconditional
[19:41] <gregaf> I think it does but I'm not sure, and one of the best ways to find out why things break is for somebody who doesn't know how it's done to try and do it and document their troubles :)
[19:42] <gregaf> Tv: iirc it generates keys and things but doesn't actually use them?
[19:42] <Tv> as in, it'll create the keys, your ceph.conf can choose not to use them
[19:42] <Tv> gregaf: mkcephfs doesn't dictate your ceph.conf
[19:42] <cmccabe> tv: I think maybe we're getting confused
[19:42] <cmccabe> the keyring doesn't mean that cephx is enabled, does it?
[19:42] <Tv> it's just a file
[19:42] <Tv> auth supported = cephx
[19:42] <Tv> that's what tells ceph to use the file
[19:43] <cmccabe> well, keyring = filename is what tells ceph to use the file
[19:43] <Tv> and probably a bunch of keyring = for the client side
[19:43] <Tv> well, more like.. keyring= tells where the file is
[19:43] <cmccabe> are you saying that with auth supported = none, I can skip the keyring altogether?
[19:44] <Tv> i don't know if some code activates just on presence of keyring= in config
[19:44] <Tv> but barring silly things, yes
[19:44] <Tv> keyring merely makes the filename to read configurable
[19:45] <cmccabe> I think I would prefer to disable crypto with keyring=none then
[19:45] <cmccabe> having two options that sort of do the same thing isn't great
[19:45] <Tv> please no magic filenames
[19:45] <Tv> they don't do the same thing
[19:45] <Tv> one toggles it on or off
[19:45] <cmccabe> well, then an empty filename
[19:46] <cmccabe> I don't think anyone is going to store his key in a file with an empty filename :)
[19:46] <Tv> another can be used to override the default path, e.g. to have osds and mds's use different keys
[19:46] <gregaf> there are default places to look for the keyring, iirc....
[19:46] <cmccabe> you can still do that with a single option
[19:46] <Tv> empty filename i might otherwise go with, but i like the extra safety of a simple boolean flag
[19:46] <Tv> you don't want to accidentally disable this
[19:46] <gregaf> anyway, generally the problems occur because people try to turn on cephx and the keyring file hasn't been properly distributed or located or something
[19:46] <gregaf> iirc
[19:47] <Tv> gregaf: yeah, manually setting up a ceph cluster / modifying an existing one is truly painful
[19:47] <cmccabe> btw, auth_supported isn't a simple boolean flag
[19:48] <cmccabe> it seems to take a list of auth formats
[19:48] <Tv> cmccabe: sure, the auth system was built to be extensible, in practise it isn't
[19:48] <cmccabe> why you would want to do this, as opposed to deducing it from the keyfile format, is not at all clear
[19:48] <Tv> cmccabe: keyring = $foo # but foo is empty! oops!
[19:48] <cmccabe> well, then issue an error message and quit.
[19:49] <cmccabe> I think maybe the true reason is that keyfiles used to be binary
[19:49] <cmccabe> and I'm not sure if there was a header identifying the file
[19:49] <cmccabe> file format
[19:49] <Tv> how is that different from "keyring=" to disable it, if the "auth supported" is removed?
[19:49] <cmccabe> so we might just be screwed
[19:49] <cmccabe> the user will have to manually specify the keyfile type forever as long as we want backwards compatibility
[19:49] <Tv> send an email to mailing list, remove binary keyring format support, move on ;)
[19:50] <cmccabe> heh
[19:50] <cmccabe> you make it sound so easy
[19:50] <Tv> i'm actually serious
[19:50] <cmccabe> well, at least we're generating all ascii keys these days in new clusters
[19:50] <gregaf> I'm not sure why you think different auth schemes will require different keyrings
[19:51] <Tv> oh on top of that we're confusing envelopes vs actual data stored ;)
[19:51] <cmccabe> I guess I was thinking in terms of different crypto methods, etc.
[19:51] <Tv> but just kill all the extra that is never used
[19:51] <Tv> it'll be better that way
[19:51] <cmccabe> but I suppose we could support different schemes with the same key files
[19:51] <Tv> cmccabe: sometimes, rarely
[19:52] <cmccabe> anyway... we should produce an error or at least log a warning when the user specifies a keyfile, but we ignore it
[19:52] <Tv> but the envelope can stay the same
[19:52] <cmccabe> because auth supported = none
[19:53] <cmccabe> ok, confirmed that no keyring works fine with auth=none
[19:54] <cmccabe> now I just have to confirm that keyring+auth supported=cephx still works
[20:02] <cmccabe> ok... cephx seems to still be working under vstart at least
[20:07] <cmccabe> so... vstart.sh seems to create keys in dev/mds.$name.keyring and dev/osd$osd/keyring
[20:12] * aliguori (~anthony@ has joined #ceph
[20:16] * jbd (~jbd@ks305592.kimsufi.com) has left #ceph
[20:32] * Dantman (~dantman@S0106001731dfdb56.vs.shawcable.net) has joined #ceph
[20:53] * MarkN (~nathan@ has joined #ceph
[21:29] * jbd (~jbd@ks305592.kimsufi.com) has joined #ceph
[21:44] * ico (~ico@isper-224-34.isper.sk) has joined #ceph
[22:04] * jbd (~jbd@ks305592.kimsufi.com) has left #ceph
[22:17] <ico> nms: hello again, nss seems to be the library, i got a mile further, but then i got stuck...
[22:18] <ico> does someone know what is needed from package nss to compile ceph client?
[22:19] <Tv> ico: pretty much the development files for the NSS crypto library, and everything it needs, not sure what else to say there; your alternative is using the Crypto++ library
[22:19] <ico> i had to compile nss (slackware doesn't have a build script for it), but don't know, whal should be installed
[22:19] <Tv> e.g. the debian package is called libnss3-dev
[22:20] <ico> Tv: hm i tried Crypto++ and got some error, i'll check history what was the problem
[22:22] <ico> ok i got it: configure script tried to find libcrypto++ but i have libcryptopp
[22:22] * darkfader (~floh@ Quit (Ping timeout: 480 seconds)
[22:22] <Tv> it tries both names
[22:23] <ico> nss solved that problem, i successfully run configure, but make throws error on me: undefined reference to `PK11_GetBestSlot'... etc
[22:24] <ico> well doesn't seem to try both names
[22:24] <ico> in config.log i got twice the same line: $PKG_CONFIG --exists --print-errors "libcrypto++"
[22:25] <Tv> that's check #1
[22:25] <Tv> there's a check #2 without using pkg-config, for older versions
[22:26] <ico> you mean "checking for library containing _ZTIN8CryptoPP14CBC_EncryptionE" ?
[22:26] <Tv> yup
[22:26] <ico> hm
[22:26] * darkfader (~floh@ has joined #ceph
[22:28] <ico> that also tries to use libcrypto++: ....../x86_64-slackware-linux/bin/ld: cannot find -lcrypto++
[22:28] * yehuda_hm (~yehuda@99-48-179-68.lightspeed.irvnca.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[22:28] <Tv> [AC_SEARCH_LIBS([_ZTIN8CryptoPP14CBC_EncryptionE], [crypto++ cryptopp],
[22:28] <Tv> that lists two libraries to look in
[22:29] <ico> Tv: oh... sorry, yes it's there :(
[22:31] <ico> going to try to compile crypto++ by hand, the installed one has some undefined references..
[22:31] <ico> Tv: thanks for now :)
[22:34] * aliguori (~anthony@ Quit (Quit: Ex-Chat)
[22:34] * aliguori (~anthony@ has joined #ceph
[22:44] * jbd (~jbd@ks305592.kimsufi.com) has joined #ceph
[22:56] <ico> Tv: so i compiled & installed newest crypto++, test passed, but ceph's configure can't use it
[22:57] <ico> got some undefined references in test program
[23:02] <Tv> gregaf: i will very much agree with you on one thing.. ceph auth is painful to debug
[23:02] <Tv> now my osd can't talk to mon :(
[23:03] <gregaf> :(
[23:03] <gregaf> could it ever talk to the mon?
[23:03] <Tv> yeah, before i shuffled ~200 lines of code that sets up the cluster
[23:04] <gregaf> oh
[23:04] <Tv> it's *supposed* to act identical, but something slipped
[23:04] <gregaf> did you see why the OSD and mon aren't talking?
[23:04] <gregaf> is the OSD failing the auth handshake on connect?
[23:06] <Tv> 2011-06-03 13:55:14.453597 7f61cc709700 -- <== mon1 2 ==== auth_reply(proto 2 -1 Operation not permitted) v1 ==== 24+0+0 (1161288937 0 0) 0x2431a00 con 0x242f3c0
[23:06] <Tv> fiddling..
[23:06] <gregaf> so probably the OSD couldn't find its key?
[23:07] <Tv> oh hey
[23:08] <Tv> 2011-06-03 13:55:01.039168 7f3ba2994700 cephx: verify_authorizer_reply exception in decode_decrypt with AQAlSulNQN9RAhAA1CNlsP+9/+v8eom8Tjh7+A==
[23:08] <Tv> exception you say
[23:08] * aliguori_ (~anthony@ has joined #ceph
[23:09] <Tv> no key i can see matches that base64
[23:10] <Tv> i may have ghost processes haunting
[23:12] * jbd (~jbd@ks305592.kimsufi.com) has left #ceph
[23:12] <sjust> ceph meeting
[23:13] * aliguori (~anthony@ Quit (Ping timeout: 480 seconds)
[23:13] <cmccabe> now?
[23:13] <sjust> I think so
[23:13] <cmccabe> ok, give me a skype when it happens
[23:38] <Tv> yay found my bug
[23:38] * aliguori_ (~anthony@ Quit (Quit: Ex-Chat)
[23:40] <Tv> it was a search-replace slip :(
[23:41] * Dantman (~dantman@S0106001731dfdb56.vs.shawcable.net) Quit (Remote host closed the connection)
[23:51] <Tv> https://github.com/tv42/teuthology <-- the test framework, because that was quick to slap up
[23:51] <Tv> if you try it, don't steal my sepia machines though ;)
[23:51] <Tv> i'll write docs soon
[23:51] <Tv> but before that
[23:51] <Tv> yehudasa: talk to me about boto versions
[23:51] <yehudasa> Tv: verifying
[23:55] <yehudasa> Tv: min boto version is 2.0
[23:57] <yehudasa> Tv: the problem is that 2.0 is not out yet
[23:57] <yehudasa> it's in rc1
[23:57] <Tv> boto >=2.0b4
[23:57] <Tv> that's what we currently use
[23:57] <yehudasa> is it?
[23:57] <Tv> i'll happily bump that to rc1
[23:58] <yehudasa> 2.0b4 is enough
[23:58] <Tv> well what's the problem then
[23:59] <yehudasa> could it be that the only thing that checks for this dependency is in the debian packages?
[23:59] <Tv> how are you running s3-tests?

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.