#ceph IRC Log


IRC Log for 2011-05-18

Timestamps are in GMT/BST.

[0:02] <cmccabe> sagewk: did you see http://tracker.newdream.net/issues/943
[0:03] <sagewk> yeah
[0:03] <cmccabe> sagewk: looks like someone reproduced it
[0:03] <sagewk> i'll follow up
[0:03] <cmccabe> sagewk: well, lxo reproduced it, specifically
[0:05] <sagewk> oh hi lxo
[0:05] <sagewk> posted an update on the bug
[0:09] * aliguori (~anthony@ Quit (Quit: Ex-Chat)
[0:14] * verwilst (~verwilst@dD5769423.access.telenet.be) has joined #ceph
[0:37] <Tv> uhhh ceph.newdream.net has not written to kern.log since 2008? i highly doubt that...
[0:38] <lxo> oh, hi sagewk ;-)
[0:38] <Tv> lxo: i think he just went into a meeting
[0:39] <lxo> np, I was just responding his greeting
[0:42] <u3q> I GO TOO USERNETT AND B
[0:42] <u3q> oops ww
[0:42] <Tv> i'm setting timeouts for git-daemon.. 1 minute to say "hi", 1 day to complete transfer, those should be huge enough to never cause real problems
[0:46] <Tv> yay timeout works
[0:49] <Tv> sagewk: in case there was something dh-managed about /etc/inetd.conf, please take care of that or teach me how; i added --init-timeout=60 --timeout=86400
[1:05] <sagewk> tv: no magic there, that one i did by hand
[1:05] <sagewk> cool
[1:05] <sagewk> was that gitbuilder's issue?
[1:05] <Tv> i don't know why but a git fetch was hanging
[1:05] <Tv> as soon as i killed it, it started making progress
[1:06] <Tv> and then i cleaned the other fetches hanging on ceph.newdream.net, and with the timeouts that can't happen again
[1:11] * allsystemsarego (~allsystem@ Quit (Quit: Leaving)
[1:40] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) has joined #ceph
[1:50] <lxo> oh, now this is new: ceph -w reports there's one PG in inconsistent state
[1:52] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[1:56] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) has joined #ceph
[2:27] * verwilst (~verwilst@dD5769423.access.telenet.be) Quit (Quit: Ex-Chat)
[2:35] * Tv (~Tv|work@ip-66-33-206-8.dreamhost.com) Quit (Ping timeout: 480 seconds)
[3:12] * sagelap (~sage@ip-66-33-206-8.dreamhost.com) has joined #ceph
[3:12] * joshd (~joshd@ip-66-33-206-8.dreamhost.com) Quit (Quit: Leaving.)
[3:16] * djlee__ (~dlee064@des152.esc.auckland.ac.nz) has joined #ceph
[3:23] * djlee_ (~dlee064@des152.esc.auckland.ac.nz) Quit (Ping timeout: 480 seconds)
[3:26] * djlee_ (~dlee064@des152.esc.auckland.ac.nz) has joined #ceph
[3:33] * djlee__ (~dlee064@des152.esc.auckland.ac.nz) Quit (Ping timeout: 483 seconds)
[3:35] * djlee__ (~dlee064@des152.esc.auckland.ac.nz) has joined #ceph
[3:39] * sagelap (~sage@ip-66-33-206-8.dreamhost.com) has left #ceph
[3:42] * djlee_ (~dlee064@des152.esc.auckland.ac.nz) Quit (Ping timeout: 480 seconds)
[5:35] * cmccabe (~cmccabe@c-24-23-254-199.hsd1.ca.comcast.net) has left #ceph
[9:03] * yehuda_hm (~yehuda@bzq-79-182-97-171.red.bezeqint.net) Quit (Ping timeout: 480 seconds)
[9:10] * neurodrone (~neurodron@cpe-76-180-162-12.buffalo.res.rr.com) Quit (Quit: zzZZZZzz)
[9:46] * allsystemsarego (~allsystem@ has joined #ceph
[9:57] * votz (~votz@dhcp0020.grt.resnet.group.upenn.edu) Quit (Read error: Connection reset by peer)
[9:58] * votz (~votz@dhcp0020.grt.resnet.group.UPENN.EDU) has joined #ceph
[9:59] * votz_ (~votz@dhcp0020.grt.resnet.group.UPENN.EDU) has joined #ceph
[10:02] * yehuda_hm (~yehuda@ has joined #ceph
[10:06] * votz (~votz@dhcp0020.grt.resnet.group.UPENN.EDU) Quit (Ping timeout: 480 seconds)
[11:35] * billy (~billy@ has joined #ceph
[11:41] * MarkN (~nathan@ has joined #ceph
[11:41] <billy> on a fresh, clean ubuntu natty server installation, which is the best way for installing ceph?
[11:41] <billy> I'm leaning towards the Building From Source section of the wiki http://ceph.newdream.net/wiki/Debian
[11:41] <billy> am I right?
[12:10] <billy> can anyone help with a two host test installation?
[14:11] * aliguori (~anthony@cpe-70-123-132-139.austin.res.rr.com) has joined #ceph
[15:09] <bchrisman> billy: I'm not on ubuntu/debian, but I think they have packages for it that are generally pretty current.
[15:09] <bchrisman> billy: probably best to start there???
[15:24] <billy> k, let's say I'm past the point of getting the packages
[15:24] <billy> I've made the pkg's and am ready to install
[15:24] <billy> but I have a couple of questions before I start
[15:25] <billy> I want to start with two machines, preferably mirroring the data between them
[15:26] <billy> and then I want to add a third host and convert the replication to raid5 style between the 3 machines
[15:26] <billy> is that possible?
[15:28] <bchrisman> raid 5 across machines is not currently supported.
[15:29] <bchrisman> I think they were considering raid 4 across machines at some point, but I don't think it's particularly high priority
[15:29] <bchrisman> it's mainly mirrored data right now.
[15:29] <bchrisman> (some people use per-host raid underneath, but that's a completely separate issue)
[15:37] <billy> so how do I go about setting up the two hosts for a mirrored setup?
[15:42] <bchrisman> they'll be in mirrored mode by default.
[15:43] <bchrisman> should be the default crushmap, which you wont have to adjust to start out.
[15:44] <bchrisman> the instructions should be pretty straightforward for a basic setup.
[16:27] * neurodrone (~neurodron@cpe-76-180-162-12.buffalo.res.rr.com) has joined #ceph
[16:35] * billy (~billy@ Quit (Quit: Leaving)
[17:10] * neurodrone (~neurodron@cpe-76-180-162-12.buffalo.res.rr.com) Quit (Quit: zzZZZZzz)
[17:31] * Tv (~Tv|work@ip-66-33-206-8.dreamhost.com) has joined #ceph
[17:34] * yehuda_hm (~yehuda@ Quit (Ping timeout: 480 seconds)
[17:39] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[17:59] * neurodrone (~neurodron@dhcp212-168.wireless.buffalo.edu) has joined #ceph
[18:28] * bchrisman (~Adium@70-35-37-146.static.wiline.com) has joined #ceph
[18:51] * sjust (~sam@ip-66-33-206-8.dreamhost.com) Quit (Read error: Connection reset by peer)
[18:53] * sjust (~sam@ip-66-33-206-8.dreamhost.com) has joined #ceph
[18:54] * neurodrone (~neurodron@dhcp212-168.wireless.buffalo.edu) Quit (Quit: zzZZZZzz)
[18:58] * joshd (~joshd@ip-66-33-206-8.dreamhost.com) has joined #ceph
[19:04] * yehuda_hm (~yehuda@bzq-79-182-97-171.red.bezeqint.net) has joined #ceph
[19:31] <sagewk> morning everyone! let's do the skype
[19:36] * neurodrone (~neurodron@dhcp212-168.wireless.buffalo.edu) has joined #ceph
[19:40] * yehuda_hm (~yehuda@bzq-79-182-97-171.red.bezeqint.net) Quit (Ping timeout: 480 seconds)
[19:46] * cmccabe (~cmccabe@c-24-23-254-199.hsd1.ca.comcast.net) has joined #ceph
[19:53] <bchrisman> sagewk: one problem (possibly superfluous) is that ceph_telldir is reporting Client::telldir()'s return code, which returns that magic value (thinking it's the dirp offset). It should be returning the current location of the directory stream.
[19:53] <Tv> bchrisman: telldir values are all magic
[19:53] <Tv> bchrisman: all that's guaranteed is, if you feed them back to seekdir, you get to go back to the same spot
[19:54] <bchrisman> Tv: yeah.. doesn't have to be an ordinal number??? part of this problem is that samba's vfs layer expects telldir to return a long instead of a long long...
[19:55] <sagewk> bchrisman: aie.. that sucks. is that something that's fixable?
[19:57] <bchrisman> in samba? talked with richard about it who's more clued in on samba dev ??? he winced??? was thinking of working around it in the vfs layer
[19:59] <Tv> sagewk: didn't ceph use the high bits for something like fragmented dirs?
[20:00] <sagewk> hmm yeah that might work.
[20:00] <sagewk> tv: exactly yeah
[20:00] <Tv> so there's going to be >long offsets anyway
[20:00] <bchrisman> yeah??? I saw the encoding and it makes sense.
[20:01] <bchrisman> there's a data structure available in the vfs layer we could use to track a counter with readdir calls??? one possibility.
[20:01] <bchrisman> there are probably better ways than that.
[20:02] <bchrisman> manpage specifies off_t for telldir.. is that going to always be 32-bit or is that 32/64 variable?
[20:02] <bchrisman> I see we're using loff_t as the value in libceph for telldir...
[20:03] <bchrisman> It might make more sense to wrap that up in the libceph layer if other applications are going to expect 32-bit returns from ceph_telldir()
[20:03] <sagewk> bleh i hate [l]off_t.. we should use explicit 64-bit types probably. we define _FILE_OFFSET_BITS=64 but maybe samba doesn't?
[20:04] <bchrisman> well the samba vfs explicitly calls it a 'long' too :)
[20:07] <bchrisman> odd.. even if you have telldir64, it still casts the result to a long.. though I guess that shouldn't matter.
[20:07] <bchrisman> #if defined(HAVE_EXPLICIT_LARGEFILE_SUPPORT) && defined(HAVE_TELLDIR64)
[20:07] <bchrisman> return (long)telldir64(dirp);
[20:07] <bchrisman> (that's lib/system.c, called by the default vfs)
[20:12] <cmccabe> bchrisman: glibc doesn't seem to have telldir64 at all
[20:12] <sagewk> long is 64-bit on x86_64 at least... :/
[20:12] <bchrisman> oh.. hmm
[20:16] <bchrisman> wasn't clear to me that this is part of my current issue, I just noticed the type ugliness??? still debugging
[20:17] <sagewk> k
[20:20] <Tv> oh i went through that a while ago
[20:20] <Tv> there's two posix apis for 64-bit file access
[20:20] <Tv> the version with telldir64 etc is the one that looks like it's less common; it allows you to access both 32-bit and 64-bit versions from the same app
[20:21] <Tv> samba uses that
[20:21] <Tv> the other one just makes off_t etc be 64-bit; no way to access the 32-bit api anymore
[20:21] <Tv> libceph models that api
[20:23] <cmccabe> tv: I can't get a program to compile with telldir64 in it
[20:23] <Tv> cmccabe: defines
[20:23] <cmccabe> tv: and it looks like telldir64 doesn't exist in my include directory
[20:23] <cmccabe> /usr/include
[20:23] <cmccabe> tv: and there's no man page
[20:23] <cmccabe> tv: AIX or solaris might have it, but so far I haven't seen any evidence of it in Linux
[20:24] <cmccabe> tv: I mean readdir64 and the rest exist on linux, just not sure about telldir
[20:24] <Tv> cmccabe: did you -D_LARGEFILE64_SOURCE=1 ?
[20:25] <cmccabe> tv: yep
[20:25] <Tv> hmm i wonder if the LFS thing didn't actually touch directory seeking, directly
[20:25] <cmccabe> tv: It seems that long is 8 bytes on x86_64 anyway
[20:26] <cmccabe> tv: so telldir returning a long might be enough
[20:26] <Tv> on x86_64..
[20:26] <cmccabe> tv: I wonder what the maximum number of files you can have in a directory is on ext4
[20:26] <Tv> " The LFS does in fact not specify any
[20:26] <Tv> equivalent seekdir64/telldir64, and most implementations of *NIX don't
[20:26] <Tv> support them. "
[20:26] <cmccabe> tv: those numbers might be related
[20:26] <Tv> http://lkml.indiana.edu/hypermail/linux/kernel/0103.1/0521.html
[20:27] <Tv> extN with dir hashing might actually need to use cookies to even support the whole damn api
[20:27] <Tv> telldir/seekdir should just die
[20:27] <cmccabe> tv: it would be nice
[20:28] <sagewk> i wonder if samba really needs it?
[20:28] <Tv> something make make telldir() always return 0, and just live with what breaks..
[20:28] <Tv> s/make/like/
[20:28] <Tv> i wish we (= the linux community) could just do that
[20:30] <sagewk> seriously
[20:31] <cmccabe> it looks to me like 64 bit offsets will work on x86_64 under glibc
[20:31] <cmccabe> glibc has long int telldir(DIR *dirp) { return dirp->filepos; }
[20:31] <cmccabe> which is an off_t
[20:32] <cmccabe> on 32-bit, looks like it would chop off the high 32 bits when converting to long
[20:35] <cmccabe> I guess one can only hope that you'll never have a directory that large on a 32-bit system
[20:37] <Tv> one of my rules of api design, learned about a decade ago, was always be explicit about bitsizes of every variable; no int etc ever
[20:37] <bchrisman> yeah??? sorry, I assumed that long was 32-bit??? makes me wonder what 'long long' is on x86_64.. probably 64 bit as well.
[20:37] <cmccabe> I still haven't really gotten used to 64-bit long
[20:37] <Tv> we ran across 32-bit intel, sparc, powerpc and s/390..
[20:37] <bchrisman> Tv: agreed on that.. it's like writing XML without a schema otherwise.
[20:37] <cmccabe> 64-bit long long isn't surprising to me
[20:38] <cmccabe> tv: I have to disagree slightly.
[20:38] <cmccabe> tv: If you really don't care about the number of bits, it can be an int.
[20:38] <cmccabe> tv: like if you have a number you know will only be between 0 and 10.
[20:38] <Tv> as a lone variable, sure
[20:39] <cmccabe> tv: that allows the compiler to select the type which is most efficient
[20:39] <Tv> put into a struct, be wary about unpredictable cache lines etc
[20:39] <cmccabe> tv: there is a reason why C has int; it's not just to confuse newbies who think it's always 32 bits
[20:40] <cmccabe> bbl
[20:40] * cmccabe (~cmccabe@c-24-23-254-199.hsd1.ca.comcast.net) has left #ceph
[20:44] <Tv> look what google dug up: http://www.vnode.ch/fixing_seekdir
[20:51] <Tv> so it's even been historically broken, and nobody really cared
[20:58] <bchrisman> Tv: that's an amusing article :)
[21:01] * Mibbit_ (~lol@82VAABM8Y.tor-irc.dnsbl.oftc.net) has joined #ceph
[21:07] * Mibbit_ (~lol@82VAABM8Y.tor-irc.dnsbl.oftc.net) Quit (Quit: leaving)
[22:09] * Yulya_ (~Yu1ya_@ip-95-220-238-246.bb.netbynet.ru) has joined #ceph
[22:16] * Yulya__ (~Yu1ya_@ip-95-220-188-224.bb.netbynet.ru) Quit (Ping timeout: 480 seconds)
[22:18] * neurodrone (~neurodron@dhcp212-168.wireless.buffalo.edu) Quit (Quit: zzZZZZzz)
[22:30] * Yulya__ (~Yu1ya_@ip-95-220-138-132.bb.netbynet.ru) has joined #ceph
[22:37] * Yulya_ (~Yu1ya_@ip-95-220-238-246.bb.netbynet.ru) Quit (Ping timeout: 480 seconds)
[23:23] * Yulya_ (~Yu1ya_@ip-95-220-174-84.bb.netbynet.ru) has joined #ceph
[23:25] * neurodrone (~neurodron@cpe-76-180-162-12.buffalo.res.rr.com) has joined #ceph
[23:30] * Yulya__ (~Yu1ya_@ip-95-220-138-132.bb.netbynet.ru) Quit (Ping timeout: 480 seconds)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.