#ceph IRC Log


IRC Log for 2011-03-08

Timestamps are in GMT/BST.

[0:08] * bbigras (quasselcor@bas11-montreal02-1128535815.dsl.bell.ca) has joined #ceph
[0:08] * bbigras is now known as Guest3918
[0:11] * Guest3097 (quasselcor@bas11-montreal02-1128535815.dsl.bell.ca) Quit (Remote host closed the connection)
[0:12] * allsystemsarego (~allsystem@ Quit (Quit: Leaving)
[0:29] <Tv> hey everyone, feedback please: NSS requires initialization, which I put in the CryptoAES constructor.. but it can fail. right now i have a dout and exit(1) in there; what should it do?
[0:30] <cmccabe> tv: don't put initialization in global constructors
[0:30] <Tv> cmccabe: yeah. so where?
[0:30] <cmccabe> tv: if it's only required by specific programs, you could add it to common_init with a flag saying whether to do it
[0:31] <cmccabe> tv: similar to STARTUP_FLAG_INIT_KEYS
[0:31] <cmccabe> tv: in fact, I wonder if STARTUP_FLAG_INIT_KEYS implies initialize NSS?
[0:32] <cmccabe> tv: if you find cases where we don't need INIT_KEYS, but do need init nss, maybe create something like STARTUP_FLAG_INIT_NSS
[0:39] <Tv> i think i need to test what happens if you don't init it
[0:39] <Tv> yet still call the functions
[0:40] <Tv> perhaps i'll add a global and init on first call
[0:40] <Tv> s/global/static/
[0:40] <cmccabe> please don't
[0:40] <cmccabe> I really hate seeing those if (not_init) in the fastpath
[0:40] <Tv> not visible to others but remembered across calls
[0:40] <cmccabe> it really slows everything down
[0:41] <cmccabe> it also tends to push initialization errors off to the point where they're far from initialization, which tends to confuse the issue
[0:43] <Tv> well at least it fails reliably when not initialized
[0:43] <Tv> at least at this version
[0:43] <Tv> so i guess that's good enough
[0:59] <Tv> though i am concerned that it'll be hard to discover that you need a magic flag
[0:59] <Tv> unless i add an explicit test that it's initialized, which is as "slow" as your earlier mention
[1:00] <Tv> hell, just about all crypto use goes through an extra layer of indirection with a switch
[1:00] <Tv> i don't think i care about an extra if, at this point
[1:00] <Tv> oh it's not just a switch, it's a switch with an if after it to check that it found a suitable branch
[1:00] <Tv> etc etc
[1:07] <cmccabe> tv: most people will just copy the code from the previous crypto-using programs, including the initialization
[1:07] <Tv> eww copy-paste
[1:07] <cmccabe> tv: as long as it's obvious what's going on, I wouldn't agonize over it
[1:07] <cmccabe> tv: and as long as failure to init causes a clear error, which seems to be the case here
[1:07] <Tv> haha
[1:08] <Tv> there's nothing clear about NSS
[1:08] <Tv> 2011-03-07 16:03:22.163800 7f3169cdf720 cannot find NSS slot to use: -8127
[1:08] <Tv> that is not clear
[1:10] <cmccabe> tv: the truth is, almost every library requires some kind of initialization
[1:10] <cmccabe> tv: when you do enough systems programming, you come to expect it
[1:11] <cmccabe> tv: they could have made the error message clearer, but that might have also made it less efficient.
[1:14] <cmccabe> tv: I guess we could store the init flags somewhere, and check to see if INIT_NSS was in them after an NSS error
[1:16] <cmccabe> tv: I guess that would force you to wrap every NSS call, which seems like a lot of work to go through just to catch a fairly obvious and trivial mistake
[1:17] <cmccabe> tv: it is kind of disappointing that NSS itself didn't do that. I mean, they could have set a flag in init() and checked that in the error handling pathways. The error handling pathways don't have to be super-fast
[1:34] <cmccabe> tv: I dunno why you think libNSS sucks.
[1:34] <cmccabe> tv: I'm looking at the code now and it doesn't seem that bad.
[1:34] <cmccabe> tv: functions are short and seem to do only one thing.
[1:35] <cmccabe> tv: there is some clutter, like multiple init functions, but isn't this thing like a decade old?
[1:35] <cmccabe> tv: also it's kind of lame that they haven't split it out from the mozilla sources... and that they still use the ancient CVS system
[1:37] <Tv> and calling aes encrypt takes 81 lines of code
[1:38] <Tv> with several operations that might fail if you weren't talking about AES, but that cannot be skipped
[1:38] <Tv> like, i need to pick the right "slot number", because smart card hardware has slots
[1:41] <cmccabe> tv: I think slot can also represent a database
[1:41] <Tv> of which AES uses none
[1:42] <cmccabe> tv: well, yeah. AES is a symmetric cipher, so it wouldn't have any use for secret keys
[1:42] <cmccabe> tv: yeah, in general the codebase doesn't seem very well separated from mozilla's use case
[1:42] <Tv> umm, aes keys are all secret..
[1:43] <Tv> and using a db of keys is completely separate from priv/pub key crypto
[1:43] <cmccabe> tv: yes and no.
[1:43] <Tv> i can run RSA operations through NSS much like the AES ones, without a database, handling keys manually
[1:43] <cmccabe> tv: having a stored public and private key that you use is something that most public key systems have to do
[1:43] <cmccabe> tv: it makes sense to allow a database to hold this key (although I like flatfiles, personally)
[1:44] <cmccabe> tv: yeah, you could contrive some kind of use for a database in conjunction with symmetric crypto, but it seems like a bit of a stretch.
[1:44] <cmccabe> tv: and yes, you don't need a db to do public crypto... ssh doesn't use one
[1:45] <cmccabe> tv: it's just a feature. I don't know how useful it is since I haven't played with it. see http://lwn.net/Articles/428399/
[1:48] <cmccabe> tv: I wonder why RedHat can't persuade these guys to at least split libnss off from the mozilla sources? The way it is now is kind of messy.
[1:48] <cmccabe> tv: you would think they would fork it or something
[1:49] <cmccabe> tv: I guess nobody wanted to modify crypto code and potentially make a debian-style screwup
[2:36] * Tv (~Tv|work@ip-66-33-206-8.dreamhost.com) Quit (Ping timeout: 480 seconds)
[2:44] * bchrisman (~Adium@70-35-37-146.static.wiline.com) Quit (Quit: Leaving.)
[2:59] * joshd (~joshd@ip-66-33-206-8.dreamhost.com) Quit (Quit: Leaving.)
[2:59] * greglap (~Adium@ has joined #ceph
[3:47] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) has joined #ceph
[3:49] * greglap (~Adium@ Quit (Quit: Leaving.)
[4:54] * sage (~sage@dsl092-035-022.lax1.dsl.speakeasy.net) has joined #ceph
[6:31] * lxo (~aoliva@ Quit (Read error: Connection reset by peer)
[6:31] * lxo (~aoliva@ has joined #ceph
[6:33] <lxo> I'm stil on 0.24.3. The btrfs-freeze issues with 2.6.35 (and also 2.6.37 and 2.6.38-rc6) seem to be related with the use of journals within the osd filesystem. I suppose some mishandling of transactions or somesuch that makes btrfs deadlock. without osd journals, it seems to have worked fine, at least until mdses started failing to recover, erroring out with "mds0.journaler try_read_entry got 0 len entry at offset 1209546579". does this error have to do with t
[6:33] <lxo> he lack of osd journals? I thought they weren't needed for btrfs :-(
[6:37] <sage> the mds error shouldn't be related.. something else went awry.
[6:37] <lxo> ouch. I just destroyed the filesystem to start over with different journal configuration
[6:38] <lxo> :-(
[6:38] <sage> we haven't seen teh btrfs freeze issue. can you reproduce the problem with 'debug osd = 20' 'debug filestore = 20' and 'debug journal = 20' in your [osd] section? the tracker lets you attach log files
[6:38] <sage> i assume you've checked dmesg and there are not btrfs kernel warnings/errors?
[6:39] <lxo> there are only syscall-blocked-for-longer-than-120-seconds backtraces
[6:39] <sage> what are they blocked on?
[6:39] <lxo> the btrfs is shared with the root filesystem, and I observe packagekit and sometimes other processes freeze. at which point I can't tell what they're blocked on any more
[6:39] <sage> are you mounting ceph on the same hosts the osds are running on?
[6:40] <lxo> yep
[6:40] <sage> ah.. that can cause deadlock under load.
[6:40] <lxo> btrfs deadlock?
[6:40] <sage> if you post the backtraces i can confirm
[6:40] <lxo> kernel backtraces?
[6:40] <sage> yeah, the syscall blocked for 120 seconds ones
[6:41] <lxo> Mar 8 01:34:05 fri kernel: [ 320.090885] Pid: 2437, comm: cosd Not tainted #1
[6:41] <lxo> Mar 8 01:34:05 fri kernel: [ 320.091018] Call Trace:
[6:41] <lxo> Mar 8 01:34:05 fri kernel: [ 320.091033] [<ffffffff8104d9ad>] warn_slowpath_common+0x85/0x9d
[6:41] <lxo> Mar 8 01:34:05 fri kernel: [ 320.091042] [<ffffffff8104d9df>] warn_slowpath_null+0x1a/0x1c
[6:41] <lxo> Mar 8 01:34:05 fri kernel: [ 320.091081] [<ffffffffa0153fb1>] btrfs_orphan_commit_root+0x86/0xa4 [btrfs]
[6:41] <lxo> Mar 8 01:34:05 fri kernel: [ 320.091116] [<ffffffffa014f06a>] commit_fs_roots.clone.17+0x8b/0xf1 [btrfs]
[6:41] <sage> it's not btrfs specific. VM writeback on the ceph fs relies on a userspace process to do work, and that process may need to allocate memory.
[6:41] <lxo> Mar 8 01:34:05 fri kernel: [ 320.091131] [<ffffffff8103c14a>] ? need_resched+0x23/0x2d
[6:41] <lxo> Mar 8 01:34:05 fri kernel: [ 320.091142] [<ffffffff8103c162>] ? should_resched+0xe/0x2e
[6:41] <lxo> Mar 8 01:34:05 fri kernel: [ 320.091152] [<ffffffff81468a44>] ? _cond_resched+0xe/0x22
[6:41] <lxo> Mar 8 01:34:05 fri kernel: [ 320.091188] [<ffffffffa014fec7>] btrfs_commit_transaction+0x341/0x5e6 [btrfs]
[6:41] <lxo> Mar 8 01:34:05 fri kernel: [ 320.091200] [<ffffffff81066633>] ? autoremove_wake_function+0x0/0x39
[6:41] <lxo> Mar 8 01:34:05 fri kernel: [ 320.091227] [<ffffffffa013dff8>] ? block_rsv_migrate_bytes+0x5d/0x69 [btrfs]
[6:41] <lxo> Mar 8 01:34:05 fri kernel: [ 320.091270] [<ffffffffa017400c>] btrfs_mksubvol+0x1c1/0x2d5 [btrfs]
[6:41] <lxo> Mar 8 01:34:05 fri kernel: [ 320.091309] [<ffffffffa0174231>] btrfs_ioctl_snap_create+0x111/0x136 [btrfs]
[6:41] <lxo> Mar 8 01:34:05 fri kernel: [ 320.091349] [<ffffffffa0176069>] btrfs_ioctl+0x3ad/0x800 [btrfs]
[6:41] <lxo> Mar 8 01:34:05 fri kernel: [ 320.091363] [<ffffffff81116d36>] ? do_sync_write+0xcb/0x108
[6:41] <lxo> Mar 8 01:34:05 fri kernel: [ 320.091374] [<ffffffff811242f3>] vfs_ioctl+0x36/0xa7
[6:41] <lxo> Mar 8 01:34:05 fri kernel: [ 320.091382] [<ffffffff81124c54>] do_vfs_ioctl+0x468/0x49b
[6:42] <lxo> Mar 8 01:34:05 fri kernel: [ 320.091392] [<ffffffff81116c12>] ? fsnotify_modify+0x6c/0x74
[6:42] <lxo> Mar 8 01:34:05 fri kernel: [ 320.091402] [<ffffffff81118409>] ? fput+0x22/0x1ed
[6:42] <lxo> Mar 8 01:34:05 fri kernel: [ 320.091411] [<ffffffff81124cdd>] sys_ioctl+0x56/0x79
[6:42] <lxo> Mar 8 01:34:05 fri kernel: [ 320.091420] [<ffffffff8111771c>] ? sys_pwrite64+0x69/0x76
[6:42] <lxo> Mar 8 01:34:05 fri kernel: [ 320.091431] [<ffffffff81009cf2>] system_call_fastpath+0x16/0x1b
[6:42] <lxo> Mar 8 01:34:05 fri kernel: [ 320.091438] ---[ end trace 47c6fe79dfb97683 ]---
[6:42] <sage> hmm that's not the usual one i would expect. is there another that's not in btrfs code?
[6:42] <lxo> nope
[6:42] <sage> (btw using something like fpaste.org is pbly easier)
[6:42] <sage> hmm
[6:43] <lxo> there are others indicating corruption in list_add
[6:44] <lxo> within btrfs_unlink
[6:44] <lxo> but I don't seem to get these without journals
[6:46] <lxo> err... actually, I do
[6:46] <sage> is the journal a file in btrfs, or a separate partition/device?
[6:46] <sage> the list_add sounds like a real btrfs bug.
[6:46] <lxo> but they don't seem to bring the filesystem to a halt like they do when there is a journal
[6:46] <lxo> yep
[6:46] <lxo> (for the latter statement)
[6:46] <sage> post to the btrfs list (and cc ceph-devel)?
[6:46] <lxo> journal is a file in btrfs
[6:47] <lxo> I should probably not bother the btrfs folks with a bug on such an old kernel. I'll probably only report it if I still get it with 2.6.37+
[6:47] <sage> you might try putting the journal on a different device, if you can, and see if that changes it. the jouranl file is fsynced regularly, so it's exercising the btrfs tree logging code. (we don't usually do that.)
[6:48] <sage> yeah definitely. see if you have better luck with the latest mainline
[6:48] <sage> i'm off to bed, ttyl!
[6:48] <lxo> thanks! have a good one
[7:09] * neurodrone (~neurodron@cpe-76-180-162-12.buffalo.res.rr.com) Quit (Quit: neurodrone)
[8:18] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) has joined #ceph
[9:09] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) Quit (Quit: Yoric)
[9:10] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) has joined #ceph
[9:10] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) Quit ()
[10:07] * allsystemsarego (~allsystem@ has joined #ceph
[10:07] * Yoric (~David@ has joined #ceph
[10:38] * lxo (~aoliva@ Quit (Read error: Connection reset by peer)
[10:38] * lxo (~aoliva@ has joined #ceph
[10:41] * Jiaju (~jjzhang@ Quit (Remote host closed the connection)
[10:54] * Jiaju (~jjzhang@ has joined #ceph
[11:11] * Yoric_ (~David@ has joined #ceph
[11:11] * Yoric (~David@ Quit (Read error: Connection reset by peer)
[11:11] * Yoric_ is now known as Yoric
[11:55] * todin (tuxadero@kudu.in-berlin.de) has joined #ceph
[12:33] <todin> wido: you are here?
[12:34] <todin> do you have any power consumption numbers for the atom machine?
[13:51] * Yoric (~David@ Quit (Read error: Connection reset by peer)
[13:51] * Yoric (~David@ has joined #ceph
[14:33] * Yoric (~David@ Quit (Ping timeout: 480 seconds)
[14:38] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) has joined #ceph
[14:48] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) Quit (Quit: This computer has gone to sleep)
[15:01] * neurodrone (~neurodron@dhcp210-211.wireless.buffalo.edu) has joined #ceph
[15:14] * upendra (ca419f6a@ircip1.mibbit.com) has joined #ceph
[15:20] * upendra (ca419f6a@ircip1.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[15:56] * Yoric (~David@ has joined #ceph
[15:56] * Yoric_ (~David@ has joined #ceph
[15:56] * Yoric (~David@ Quit (Read error: Connection reset by peer)
[15:56] * Yoric_ is now known as Yoric
[16:30] * morse (~morse@supercomputing.univpm.it) Quit (Remote host closed the connection)
[16:35] * morse (~morse@supercomputing.univpm.it) has joined #ceph
[16:35] * morse (~morse@supercomputing.univpm.it) Quit (Remote host closed the connection)
[16:38] * morse (~morse@supercomputing.univpm.it) has joined #ceph
[17:23] * verwilst (~verwilst@router.begen1.office.netnoc.eu) has joined #ceph
[17:38] * neurodrone (~neurodron@dhcp210-211.wireless.buffalo.edu) Quit (Quit: neurodrone)
[17:44] * sjusthm (~sam@adsl-76-208-183-201.dsl.lsan03.sbcglobal.net) has joined #ceph
[17:44] * Yoric_ (~David@ has joined #ceph
[17:45] * greglap (~Adium@cpe-76-90-239-202.socal.res.rr.com) has joined #ceph
[17:48] * Yoric (~David@ Quit (Ping timeout: 480 seconds)
[17:48] * Yoric_ is now known as Yoric
[17:52] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[17:52] * neurodrone (~neurodron@dhcp208-002.wireless.buffalo.edu) has joined #ceph
[18:09] * sjusthm (~sam@adsl-76-208-183-201.dsl.lsan03.sbcglobal.net) Quit (Remote host closed the connection)
[18:37] * Tv (~Tv|work@ip-66-33-206-8.dreamhost.com) has joined #ceph
[18:38] * bchrisman (~Adium@70-35-37-146.static.wiline.com) has joined #ceph
[18:50] * Yoric (~David@ Quit (Quit: Yoric)
[18:59] * cmccabe1 (~cmccabe@c-24-23-253-6.hsd1.ca.comcast.net) has joined #ceph
[19:05] * sjusthm (~sam@adsl-76-208-183-201.dsl.lsan03.sbcglobal.net) has joined #ceph
[19:09] * joshd1 (~joshd@ip-66-33-206-8.dreamhost.com) has joined #ceph
[19:17] * hijacker (~hijacker@ Quit (Ping timeout: 480 seconds)
[19:18] * hijacker (~hijacker@ has joined #ceph
[19:37] * neurodrone (~neurodron@dhcp208-002.wireless.buffalo.edu) Quit (Quit: neurodrone)
[20:00] <cmccabe1> figured out the problem with my microphone
[20:01] <cmccabe1> sorry it's just another one of those things... I guess I need to make a checklist of all these possible problems or something
[20:01] <Tv> call the echo service
[20:01] <Tv> set it up once, don't fuck with it
[20:01] <cmccabe1> different applications can change settings when you run them
[20:02] <cmccabe1> so I usually end up running xfce4-mixer, since it exposes all the controls.
[20:02] <cmccabe1> with my previous sound card, there was this one particular checkbox that had to be checked, or else you'd never hear a thing... I think it was ADC2
[20:02] <Tv> then you need to call echo every morning before the call
[20:03] <cmccabe1> but this checkbox wasn't exposed by most of the pretty gui interfaces
[20:03] <cmccabe1> actually the problem here was a lot simpler... a hardswitch
[20:03] <cmccabe1> I generally do test the mic each morning... I forgot today
[20:30] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) has joined #ceph
[20:51] <Tv> NSS as it is right now (forced to fit existing apis, not sharing initialization code across multiple runs) takes ~3.5x as long for AES
[20:53] <cmccabe1> tv: hmm, is there a way to share initialization across multiple runs?
[21:07] <greglap> Tv: how often do we need to run that?
[21:20] * sjusthm (~sam@adsl-76-208-183-201.dsl.lsan03.sbcglobal.net) Quit (Remote host closed the connection)
[21:22] * sjusthm (~sam@adsl-76-208-183-201.dsl.lsan03.sbcglobal.net) has joined #ceph
[21:23] <Tv> cmccabe1: yes, but that means changing the existing api
[21:23] <Tv> greglap: like, for every message sent/received, if you use authx
[21:24] <greglap> Tv: yeah, that's what I thought
[21:24] <Tv> and why i feel it needs to be improved ;)
[21:24] <greglap> how significant is the time?
[21:24] <greglap> and by "existing APIs", you mean ours, right?
[21:24] <Tv> i'm still trying to understand the protocol
[21:24] <Tv> yeah
[21:24] <Tv> it's all about ceph internals
[21:24] <greglap> so we can adjust how the authentication works and cache results or whatever
[21:25] <Tv> yeah, have a "context" open for a specific secret key, etc
[21:25] <Tv> i'll do that, just needs more time
[21:26] <Tv> frankly, i'm not sure about the wire protocol at all, yet
[21:26] <Tv> but sage did assure me it's not mitm hijackable = it runs aes on every block transmitted
[21:27] <greglap> yep, I believe so
[21:28] * neurodrone (~neurodron@cpe-76-180-162-12.buffalo.res.rr.com) has joined #ceph
[21:56] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) has joined #ceph
[21:56] * ghaskins_mobile (~ghaskins_@66-189-113-47.dhcp.oxfr.ma.charter.com) Quit (Remote host closed the connection)
[22:28] * allsystemsarego (~allsystem@ Quit (Quit: Leaving)
[23:10] <Tv> you know what totally made my day?
[23:10] <Tv> the file /usr/include/nss.h belongs to the Name Service Switch
[23:10] <Tv> *headdesk*
[23:13] <Tv> (nss the crypto lib has /usr/include/nss/nss.h, which you are supposed to use with -I/usr/include/nss)
[23:14] <cmccabe1> heh
[23:16] * bchrisman (~Adium@70-35-37-146.static.wiline.com) Quit (Quit: Leaving.)
[23:30] * frank_ (frank@november.openminds.be) Quit (Server closed connection)
[23:30] * frank_ (frank@november.openminds.be) has joined #ceph
[23:31] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) Quit (Quit: Yoric)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.