#ceph IRC Log


IRC Log for 2012-03-15

Timestamps are in GMT/BST.

[0:02] <sagewk> elder: so the hard part is to figure out how to make the kdump kernel copy the kernel somewhere else instead of to /var/crash
[0:02] <Tv|work> err
[0:02] <elder> I don't even think it's that hard.
[0:02] <elder> It's configuratino.
[0:02] <Tv|work> you mean use the send-udp thingie instead of the write-to-disk thingie
[0:02] <elder> The hard part is putting all those pieces together and getting them all working in unison.
[0:03] <elder> We need a server with enough storage to hold the dumps
[0:03] <sagewk> well, whatever the kexec'd initrd does for you. i don't see easy instructions on making kdump copy over the network
[0:03] <Tv|work> elder: step 1: document the command that receives the dump
[0:03] <sagewk> only how t specify the initrd it uses when the dump kernel is run
[0:03] <elder> We need to take the instructions I worked out and translate them to the specifics of a sepia/whatever system.
[0:04] <elder> And then, 3) profit!
[0:04] <Tv|work> elder: do you have it using udp?
[0:04] <elder> I haven't used netdump yet, it was a pain in the ass to get the server built on Ubuntu.
[0:04] <Tv|work> elder: can you, please/
[0:04] <elder> So I used ssh to push the dump to another machine.
[0:04] <Tv|work> ?
[0:04] <sagewk> i don't think it has to be (or even should be) udp
[0:05] <Tv|work> dump over udp is going to much nicer to manage than needing to recover the box just to get the dump extracted
[0:05] <Tv|work> sagewk: oh i just recall someone implementing it like the netconsole thing
[0:05] <elder> Is this because you want to scrub the box when it reboots?
[0:05] <sagewk> i mean, it could scp it somewhere. the crash kernel initrd can do whatever you want with /proc/vmcore
[0:05] <Tv|work> elder: imagine the box coming up even
[0:05] <Tv|work> *not
[0:06] <sagewk> elder: and minimize moving parts (e.g., having to reboot before we get the dump)
[0:06] <Tv|work> i'd hate to write automation that reboots-and-waits-and-hopes-to-hear-back
[0:06] <Tv|work> that's just too fiddly
[0:06] <elder> Sage is right though. The whole point of the kexec'd crash kernel is so you have a clean working environment with full access to the memory of hte other kernel.
[0:06] <elder> I believe that right now it simply reboots when it has done it's work--back to the original kernel again (using a full BIOS startup)
[0:07] <sagewk> in that case, i think the hard part is making a crash initrd that gets an ip and copies the core to a known location
[0:07] <Tv|work> that work has been done by others
[0:07] <Tv|work> use it
[0:07] <sagewk> instead of copying to /var/crash/$whatever
[0:07] <elder> Again, I think it's just configuratino.
[0:07] <elder> And having a server to receive them.
[0:07] <sagewk> let's find the configuration, then.. i wasn't able to
[0:07] <elder> I don't think it's that hard, once we've decided what the infrastructure looks like (and once we get the server built, which is not *that* hard)
[0:08] <Tv|work> http://wiki.wireshark.org/NetDump
[0:08] <Tv|work> ^ THAT
[0:08] <sagewk> just assume there is some machine/ip we can scp the kernel to. that part is easy to set up.
[0:08] <Tv|work> now stop saying it's just configuration -- there's a pre-existing solution, figure it out
[0:08] <Tv|work> otherwise i will, once i get to it, and it'll take longer
[0:08] <elder> No, I mean it's a matter of specifying the right things in the config file.
[0:09] <elder> I'll work that out.
[0:09] <Tv|work> elder: please discover & document, thank you
[0:09] <elder> I will.
[0:09] <elder> My main aim was to get a working example. Now that I believe I have that I can start looking a little deeper at getting the server to build. And once I have that I can try that out.
[0:09] * BManojlovic (~steki@ Quit (Remote host closed the connection)
[0:10] <Tv|work> https://twiki.cern.ch/twiki/bin/view/LinuxSupport/NetDump
[0:10] <sagewk> elder: as long as it goes over the network and avoids a second reboot to recovery from /var/crash :)
[0:10] * verwilst (~verwilst@dD5769628.access.telenet.be) Quit (Quit: Ex-Chat)
[0:10] <Tv|work> that says they might not support it anymore
[0:11] <elder> Yes. I fully expect it should be possible to skip the /var/crash thing.
[0:11] <Tv|work> yeah, if it's a proper initrd it can just use tcp
[0:11] <elder> All this kdump stuff is pretty sketchily documented, with lots of it being years old and referring to pretty old distro or kernel release numbers.
[0:11] <sagewk> tv|work: right, netdump is old and not supported on ubuntu, that's why he's looking at (newer) kexec
[0:11] <Tv|work> now, i'm surprised if the netdump stuff died away completely..
[0:11] <sagewk> i think kexec is way more flexible/robust
[0:12] <Tv|work> oh yes
[0:12] <elder> I don't think there's anything wrong with netdump as a service.
[0:12] <Tv|work> i'm not stuck on the specific udp protocol & old kernel code
[0:12] <elder> But the client side using kexec/kdump is better.
[0:13] <Tv|work> but i'm saying, there's a bunch of people who use the "kernel crashes go to network" stuff, so we should be able to use someone else's code for it
[0:13] <elder> I think RedHat maintains this stuff for their own releases.
[0:19] <elder> OK, correction, the /usr/sbin/kdump-config script that manages all this stuff only supports write to local disk. And it notes:
[0:19] <elder> # TODO: implement different transport options other than
[0:19] <elder> # store to local disk
[0:20] <elder> Still, if we're willing to tweak that script it shouldn't be too hard to copy the file somewhere. Or write to a pipe that has a netdump server on the other end.
[0:21] <elder> But I agree with Tv it would be nice if we could just stand on the shoulders of someone who already has this sort of thing set up.
[0:24] * lxo (~aoliva@lxo.user.oftc.net) Quit (Quit: later)
[0:26] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[0:28] * LarsFronius (~LarsFroni@f054109189.adsl.alicedsl.de) Quit (Quit: LarsFronius)
[0:37] <nhm> elder: not that I know anything really about what the latest in kernel dump technology is, but SGI used kdump on our big Altix systems.
[0:38] <nhm> 3TB of memory!
[0:38] <elder> Yes. And I never really figured out how to set it up. (Or rather, I may have set it up once on my main test machine, and then forgot the rest).
[0:38] <elder> But they saved to disk, right, not to a server over a network?
[0:38] <elder> Too slow over the network for that much data.
[0:38] <Tv|work> nhm: there's an urban legend of someone with some preposterous amount of ram like 64GB accidentally triggering a hibernate..
[0:39] <nhm> elder: correct.
[0:39] <nhm> Tv|work: s/GB/TB I assume? :)
[0:39] <Tv|work> nhm: no, just 64GB.. takes a sweet while to write to disk, still
[0:40] <dmick> one wonders if you can configure which pages get dumped and compress; both those help a lot
[0:40] <elder> I believe one can.
[0:40] <nhm> Tv|work: yeah, it was a huge pain every time we had to save a dump from that SGI Altix UV.
[0:40] <dmick> i.e. maybe "screw the buffer cache"
[0:41] <Tv|work> dmick: yeah, that just means you trust the state of the crashed system more and more.. as in, it's a tradeoff, where sometimes you get a dump with crucial info missing
[0:41] <elder> Or rather, one can request that optionally filter out: zero pages, caches with or without private (mappings?), user data, and free pages
[0:41] <dmick> yup
[0:41] <dmick> no free lunches here
[0:41] <Tv|work> dmick: actually we're getting one tomorrow ;)
[0:42] <elder> Except if you save it to "the cloud." That's free lunch.
[0:42] <dmick> oh yeah wait, maybe we should set up a Ceph cluster to store crashdu...
[0:42] <Tv|work> dmick: and run the ceph qa in vms on rbd
[0:42] <elder> Nah, use S3.
[0:42] * yoshi (~yoshi@p1062-ipngn1901marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[0:44] * yoshi_ (~yoshi@p1062-ipngn1901marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[0:44] * yoshi (~yoshi@p1062-ipngn1901marunouchi.tokyo.ocn.ne.jp) Quit (Read error: Connection reset by peer)
[0:44] * yoshi_ (~yoshi@p1062-ipngn1901marunouchi.tokyo.ocn.ne.jp) Quit (Remote host closed the connection)
[0:44] * yoshi (~yoshi@p1062-ipngn1901marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[0:44] <nhm> elder: if we wrote S3 support into kdump, we could store it on S3 or via rados. :P
[0:45] <elder> Great idea.
[0:45] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Quit: Leaving.)
[0:45] <nhm> then companies could have their own private cloud kdump storage servers.
[0:47] <sagewk> gregaf: wip-msgr4 look ok? first step before breaking out the new implemenation
[0:48] <nhm> elder: having said all of that, I'm pretty sure we were storing our kernel dumps on cxfs.
[0:48] <elder> It looks like makedumpfile (which generates the file that crash(8) can use) is oriented toward writing to a file descriptor. So if we're going to send right to the net and avoid a save to disk (and use makedumpfile) we're going to have to set up a pipeline to a server somewhere.
[0:48] <elder> Well if you set it up right CXFS can deliver very high bandwidth.
[0:49] <elder> I have to quit. I'll be back later.
[0:50] <nhm> elder: sort of. The bug in XFS that causes corruption with non-standard extent sizes kind of limites the ind of writes you can do.
[0:50] <nhm> elder: ttyl
[0:56] <gregaf> sagewk: I'm not sure that bind() belongs in the generic Messenger interface, but then I'm not real familiar with non-IP networking protocols
[1:02] * Tv|work (~Tv_@aon.hq.newdream.net) Quit (Ping timeout: 480 seconds)
[1:10] * ghaskins (~ghaskins@68-116-192-32.dhcp.oxfr.ma.charter.com) has joined #ceph
[1:23] * tnt_ (~tnt@148.47-67-87.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[1:24] * cp (~cp@ Quit (Quit: cp)
[1:26] * cp (~cp@ has joined #ceph
[1:27] * cp (~cp@ Quit ()
[1:49] * lofejndif (~lsqavnbok@83TAAD31Y.tor-irc.dnsbl.oftc.net) Quit (Quit: Leaving)
[1:52] * joao (~JL@89-181-145-13.net.novis.pt) Quit (Ping timeout: 480 seconds)
[1:54] * adjohn (~adjohn@ Quit (Quit: adjohn)
[2:14] * joshd (~joshd@aon.hq.newdream.net) Quit (Quit: Leaving.)
[2:17] * Theuni (~Theuni@c-24-5-66-156.hsd1.ca.comcast.net) has joined #ceph
[2:32] * softcrack (ca55d12f@ircip4.mibbit.com) has joined #ceph
[2:34] <softcrack> hello, is Greg Farnum here?
[2:39] <dmick> it appears as though he's gone home
[2:42] <softcrack> Thanks dmick. He is responsable for bug 2173. I'm looking for a workaround. Anybody can help me?
[2:44] <dmick> I'm pretty new with ceph, but it would be interesting to know what ceph health reports
[2:44] <softcrack> I am from china. Perhaps it not a good time, as most of you get off work.
[2:45] <softcrack> mon.0 -> 'HEALTH_WARN 1 pgs degraded; There are lagging MDSes: 1(rank 0)' (0)
[2:45] <softcrack> This is the bug report. http://tracker.newdream.net/issues/2173
[2:47] <dmick> yeah. I'm sorry, I just don't know enough to be very helpful
[2:48] <dmick> I'm sure Greg will respond further in the morning; sorry I can't do more
[2:48] <softcrack> Thank you, dmick.
[2:56] * softcrack (ca55d12f@ircip4.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[3:28] * Theuni (~Theuni@c-24-5-66-156.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[3:55] * Tv__ (~tv@cpe-24-24-131-250.socal.res.rr.com) has joined #ceph
[4:34] * chutzpah (~chutz@ Quit (Quit: Leaving)
[4:45] * ghaskins (~ghaskins@68-116-192-32.dhcp.oxfr.ma.charter.com) has left #ceph
[4:50] * cattelan is now known as cattelan_away
[5:01] * dmick (~dmick@aon.hq.newdream.net) Quit (Quit: Leaving.)
[5:49] * Tv__ (~tv@cpe-24-24-131-250.socal.res.rr.com) Quit (Read error: Operation timed out)
[6:13] * eternaleye____ (~eternaley@tchaikovsky.exherbo.org) Quit (Remote host closed the connection)
[6:15] * eternaleye (~eternaley@tchaikovsky.exherbo.org) has joined #ceph
[6:45] * adjohn (~adjohn@50-0-164-119.dsl.dynamic.sonic.net) has joined #ceph
[8:12] * lxo (~aoliva@lxo.user.oftc.net) Quit (Quit: later)
[8:17] * tnt_ (~tnt@148.47-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[8:20] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[8:22] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[8:35] * lxo (~aoliva@lxo.user.oftc.net) Quit (Quit: later)
[8:36] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[9:36] * tnt_ (~tnt@148.47-67-87.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[9:46] * tnt_ (~tnt@212-166-48-236.win.be) has joined #ceph
[10:07] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) has joined #ceph
[10:30] * edwardw`away (~edward@ec2-50-19-100-56.compute-1.amazonaws.com) Quit (Ping timeout: 480 seconds)
[10:46] * edwardw`away (~edward@ec2-50-19-100-56.compute-1.amazonaws.com) has joined #ceph
[11:01] * adjohn (~adjohn@50-0-164-119.dsl.dynamic.sonic.net) Quit (Quit: adjohn)
[11:04] * yoshi (~yoshi@p1062-ipngn1901marunouchi.tokyo.ocn.ne.jp) Quit (Ping timeout: 480 seconds)
[11:05] * guilhem1 (~spectrum@sd-20098.dedibox.fr) has joined #ceph
[11:10] * ivan\ (~ivan@108-213-76-179.lightspeed.frokca.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[11:16] * ivan\ (~ivan@ has joined #ceph
[12:14] * ghaskins (~ghaskins@68-116-192-32.dhcp.oxfr.ma.charter.com) has joined #ceph
[12:18] * ghaskins (~ghaskins@68-116-192-32.dhcp.oxfr.ma.charter.com) has left #ceph
[13:15] * joao (~JL@ has joined #ceph
[14:09] * cattelan_away is now known as cattelan
[14:51] * LarsFronius_ (~LarsFroni@testing78.jimdo-server.com) has joined #ceph
[14:51] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) Quit (Read error: Connection reset by peer)
[14:51] * LarsFronius_ is now known as LarsFronius
[14:54] * LarsFronius_ (~LarsFroni@p578b21b6.dip0.t-ipconnect.de) has joined #ceph
[14:55] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) Quit (Read error: No route to host)
[14:55] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) has joined #ceph
[15:03] * LarsFronius_ (~LarsFroni@p578b21b6.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[15:35] <sage> elder: sounds reasonable on #2174, including the plana concurrency theory. were you able to reproduce? fix seems pretty obvious (init in allocator)?
[15:49] <elder> I'm trying to reproduce using for-linus.
[15:49] <elder> I may have reproduced it. Last night I got a trace from the unitialized lock warning and that led me to this explanation.
[15:50] <elder> So now I'm trying with for-linus to verify it's nothing new.
[16:47] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[16:56] * perplexed (~ncampbell@ has joined #ceph
[16:56] * joshd (~joshd@aon.hq.newdream.net) has joined #ceph
[16:56] * perplexed (~ncampbell@ Quit ()
[17:04] * Tv|work (~Tv_@aon.hq.newdream.net) has joined #ceph
[17:12] <joao> oh, joy to the world
[17:13] <joao> I can directly access sepia16 again
[17:14] <Tv|work> joao: do not use sepia anymore!
[17:14] <Tv|work> joao: ... i guess you're not on the CephBiz mailing list.. old sepia hardware is getting scrapped to free up the rack space
[17:14] <joao> nop, not on the ml
[17:15] <joao> but in any case, what I'm doing may very well be interrupted by someone pulling the plug on the machine
[17:15] <joao> I just want to run a test
[17:15] <Tv|work> joao: can you use plana? use them, lock a machine with "teuthology-lock --lock-many 1" if you need it outside of teuthology
[17:15] <joao> ok
[17:15] <joao> I can do that oo
[17:15] <joao> *too
[17:16] <Tv|work> joao: remember to unlock when you're done, teuthology-lock --unlock ubuntu@planaXX...
[17:16] <Tv|work> "teuthology-lock --list" tells what you have locked
[17:16] <joao> very well then :)
[17:18] <Tv|work> hmm 92 entries for 96 machines
[17:24] <joao> I really have to learn how to properly code in python
[17:25] <joao> everytime I look at teuthology I get all pumped up
[17:26] <Tv|work> http://www.youtube.com/watch?v=HHZhw94C5vQ
[17:35] <joao> Tv|work, what's the url for the lockserver?
[17:35] <Tv|work> $ cat /home/tv/.teuthology.yaml
[17:35] <Tv|work> lock_server: http://teuthology.ceph.dreamhost.com/locker/lock
[17:35] <Tv|work> queue_host: teuthology.ceph.dreamhost.com
[17:35] <Tv|work> queue_port: 11300
[17:36] <joao> thanks
[17:43] * groovious (~Adium@64-126-49-62.dyn.everestkc.net) has joined #ceph
[17:44] <joao> do the planas have any btrfs device available by default?
[17:48] <nhm> joao: if you use teuthology to deploy I think you can format btrfs on one of the non-primary HDs by specifying "fs: btrfs" in your ceph overrides.
[17:51] <joao> nhm, all I want is to lock a node so I can make a few tests on a btrfs subvolume
[17:51] <joao> not sure if that qualifies as "deploying"
[17:51] <joao> (honestly, no clue at all)
[17:55] * BManojlovic (~steki@ has joined #ceph
[17:58] * bchrisman (~Adium@ has joined #ceph
[17:58] <Tv|work> joao: nope
[17:58] <Tv|work> joao: but you're a mkfs.btrfs away from what you want, so shouldn't be too hard
[18:01] <joao> Tv|work, just wasn't sure if messing with the available devices was an option :)
[18:04] * dmick (~dmick@aon.hq.newdream.net) has joined #ceph
[18:04] <joao> so, the new vpn does not allow access to the teuthology server, and the old vpn doesn't allow access to planas, is that right? :p
[18:06] <nhm> joao: hrm, don't know, I'm using both still..
[18:06] <dmick> new vpn doesn't reach teuthology lockserver?...
[18:07] <dmick> that could be
[18:07] <dmick> yeah, that's on the old net still
[18:07] <dmick> you can service openvpn start and it should start both
[18:07] <dmick> (or, you know, the non upstart version of that)
[18:08] <joao> nhm, I've been using gnome-network-manager to handle the vpn connection, and still haven't found a way to keep them both connected (mainly because until now I hadn't found a reason to)
[18:08] <dmick> g-n-m! Didn't even know it thought about vpns
[18:09] <joao> dmick, it loads the openvpn's config file and then magic happens
[18:09] <joao> :)
[18:09] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Quit: Leaving.)
[18:09] <dmick> shudder
[18:10] <nhm> joao: interesting, I hadn't even thought about gnome-network-manager.
[18:10] <nhm> joao: I'm just using the standard openvpn service with /etc configurations.
[18:16] * tnt_ (~tnt@212-166-48-236.win.be) Quit (Ping timeout: 480 seconds)
[18:25] * tnt_ (~tnt@148.47-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[18:28] <sagewk> elder: are you by chance doing the thrashing job that failed in qa to test that, or something else?
[18:30] <elder> I am running that job.
[18:30] <sagewk> great
[18:31] <elder> One time through, no problems. Re-started with three consecutive runs. We'll see how that goes.
[18:32] <sagewk> great. i'll reenable those tests in qa
[18:39] <elder> I've got the testing branch ready to push (wip-testing)
[18:39] <sjust> sagewk: wip_omap_xattrs is ready for review, it does not appear to have caused snapshot/recovery bug
[18:39] <elder> I'm OK with doing it now if you want.
[18:39] <sagewk> elder: go for it
[18:39] <sagewk> so the noon run will test it
[18:40] <sjust> sagewk: though I'm not going to merge it until I have fixed the bug
[18:40] <sagewk> sjust: ok
[18:51] * adjohn (~adjohn@50-0-164-119.dsl.dynamic.sonic.net) has joined #ceph
[18:54] <elder> sagewk, done.
[19:02] * guilhem1 (~spectrum@sd-20098.dedibox.fr) has left #ceph
[19:04] * The_Bishop (~bishop@178-17-163-220.static-host.net) Quit (Ping timeout: 480 seconds)
[19:11] * The_Bishop (~bishop@178-17-163-220.static-host.net) has joined #ceph
[19:19] * The_Bishop (~bishop@178-17-163-220.static-host.net) Quit (Ping timeout: 480 seconds)
[19:21] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[19:25] <joao> sagewk, by the looks of it, either triggering the bug is a fluke every now and then, or there's something else needed besides rm -R the collection
[19:29] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[19:34] <elder> http://www.datacenterknowledge.com/archives/2012/03/14/estimate-amazon-cloud-backed-by-450000-servers/
[19:36] <wido> Does anyone of you know a Max Dunn from CA?
[19:36] <wido> I just got an e-mail from Github I got write access to a repo from that guy, no idea who he is
[19:37] <Tv|work> hah
[19:37] <wido> Probably spam
[19:37] <wido> some kind of Wiki
[19:37] * The_Bishop (~bishop@178-17-163-220.static-host.net) has joined #ceph
[19:37] <Tv|work> wido: it's also very easy to pick wrong github username there
[19:38] <wido> Tv|work: could be :) But I'm being plagued lately with people following me on Twitter, getting friend requests on Facebook, etc
[19:38] <wido> follow*
[19:39] <Tv|work> wido: don't get me started about the @tv twitter mentions...
[19:39] <wido> Tv|work: ... I get people following me with the hope I read their profile, since their profile is just full of spam
[19:40] <wido> elder: About Amazon... Wow
[19:41] <gregaf> I thought estimates placed them there 2 years ago, though
[19:42] * nhorman (~nhorman@99-127-245-201.lightspeed.rlghnc.sbcglobal.net) has joined #ceph
[19:44] <Tv|work> that math has a high fudge factor
[19:44] <Tv|work> it assumes how many vms are in a rack, it doesn't account for non-public vms, it doesn't account for amazon's internal use, ....
[19:44] <Tv|work> he does point out many of those things in the article, just.. don't just tl;dr that number
[19:49] <wido> It's indeed a wild guess, but still, we can safely assume that Amazon has some hardware running
[19:49] <dmick> lol
[19:53] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[19:53] <joao> sagewk, looks like I triggered another warning
[19:53] <joao> but no joy on #1975
[19:54] <Tv|work> wido: i heard it's virtual all the way down
[20:42] <sagewk> joao: :( well, we can reproduce w/ the ceph workload, so instrumenting btrfs to diagnose where things went wrong is another possibility. just much longer to trigger.
[20:43] <Tv|work> omg the install is *still* going
[20:43] <sagewk> joao: alternatively, we can use your new tool to generate transactions that populate and then remove collections and see if that triggers it too
[20:44] <nhm> joao: What kind of workload does your tool generate btw?
[20:44] <Tv|work> oh, crap, it rebooted back into the installer
[20:45] <Tv|work> well ain't that just a nice antifeature
[20:47] * cattelan (~cattelan@c-66-41-26-220.hsd1.mn.comcast.net) Quit (Quit: Terminated with extreme prejudice - dircproxy 1.2.0)
[20:49] <joao> sagewk, how long is "much longer"? :)
[20:49] <joao> nhm, trying to mimic OSD behavior
[20:49] <sagewk> it took on the order of a day to hit i think? I can't remember.
[20:49] <joao> ouch
[20:50] <joao> well, currently I've tried pretty much everything
[20:50] <nhm> joao: neat!
[20:50] <joao> directory creation/removal while taking snapshots and another process deleting them
[20:51] <sagewk> did you include populating the directory with lots of files before removing?
[20:51] <joao> and another process appending to files
[20:51] <joao> sage, yeah, 100's of directories with 6k files each
[20:51] <joao> just to cause mayhem
[20:51] <sagewk> well, we can continue to debug as before then..
[20:52] * eternaleye (~eternaley@tchaikovsky.exherbo.org) Quit (Quit: eternaleye)
[20:52] <sagewk> if you have theories about what is going on inside btrfs, add printk's, and reproduce with the full test
[20:52] * eternaleye (~eternaley@tchaikovsky.exherbo.org) has joined #ceph
[20:52] * cattelan (~cattelan@c-66-41-26-220.hsd1.mn.comcast.net) has joined #ceph
[20:52] <joao> yeah, that's probably going to be my next approach
[20:53] <sagewk> joao: i reproduced it with metropolis:~sage/src/teuthology/btrfs.yaml
[20:53] <joao> meanwhile, I triggered a lockdep warning :p
[20:53] <sagewk> you can push whatever you want to a wip-something branch in ceph-client.git, and specify that kernel in the job
[20:53] <sagewk> joao: interesting, can you pastebin it?
[20:53] <joao> sure
[20:54] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) Quit (Quit: LarsFronius)
[20:54] <joao> http://pastebin.com/gWtttFps
[20:54] <joao> there
[20:56] <Tv|work> plana reinstalling has reached feature parity with old sepia
[20:56] <joao> sagewk, the funniest thing about yesterday's dmesg is that the orphan_cleanup warning was the very first thing btrfs outputted after being mounted
[20:57] <joao> and that just boggles my mind, specially because rmdir causes a lot of garbage
[20:58] <joao> ah no
[20:58] <joao> no no
[20:58] <joao> I was looking at warning caused by mounting the FS, *after* it was initially triggered :)
[21:07] <Tv|work> and not plana reinstalling has surpassed old sepia in features ;)
[21:07] <Tv|work> *now
[21:08] <Tv|work> PSA: it is now safe to wreck the base installs on plana as much as you want
[21:10] <Tv|work> except for plana01, which is a server for now ;)
[21:12] * nhorman (~nhorman@99-127-245-201.lightspeed.rlghnc.sbcglobal.net) Quit (Quit: Leaving)
[21:23] <nhm> Tv|work: Let them eat cake!
[21:23] <dmick> <whooooooosh THUMP>
[21:53] * imjustmatthew (~imjustmat@pool-96-228-59-130.rcmdva.fios.verizon.net) has joined #ceph
[21:56] * Kioob (~kioob@luuna.daevel.fr) Quit (Quit: Leaving.)
[22:00] <imjustmatthew> I just got a kernel NULL pointer dereference at ceph_d_prune on a kernel client after an OSD rebooted, what do I need to collect from the client?
[22:05] <gregaf> imjustmatthew: whatever you've got from the client — dmesg output and any logs is what's likely
[22:05] <gregaf> if you've got OSD logs you should stash them somewhere but I doubt they'll matter
[22:06] <imjustmatthew> gregaf: k, and is there anything I can do to increase the logging level for the kernel client before I try and reproduce it? It doesn 't seem to log much
[22:07] <gregaf> yes, let me see...
[22:08] * lofejndif (~lsqavnbok@28IAADC3J.tor-irc.dnsbl.oftc.net) has joined #ceph
[22:09] <gregaf> so you need to have the kernel debug stuff turned on, which somebody else will have to talk about (sjust? Tv|work?), but then you can echo stuff into Ceph's dynamic debugs
[22:09] <gregaf> #!/bin/sh -x
[22:09] <gregaf> p() {
[22:09] <gregaf> echo "$*" > /sys/kernel/debug/dynamic_debug/control
[22:09] <gregaf> }
[22:09] <gregaf> echo 9 > /proc/sysrq-trigger
[22:09] <gregaf> p 'module ceph +p'
[22:09] <gregaf> p 'module libceph +p'
[22:09] <gregaf> p 'module rbd +p'
[22:09] <gregaf> p 'file net/ceph/messenger.c -p'
[22:09] <gregaf> p 'file' `grep -- --- /sys/kernel/debug/dynamic_debug/control | grep ceph | awk '{print $1}' | sed 's/:/ line /'` '+p'
[22:09] <gregaf> p 'file' `grep -- === /sys/kernel/debug/dynamic_debug/control | grep ceph | awk '{print $1}' | sed 's/:/ line /'` '+p'
[22:09] <gregaf> is what I use to turn it on
[22:15] * matthew_ (~imjustmat@pool-96-228-59-130.rcmdva.fios.verizon.net) has joined #ceph
[22:16] <matthew_> gregaf: still there? I crashed the client again and missed what you said after "Yes, le me see"
[22:16] * imjustmatthew (~imjustmat@pool-96-228-59-130.rcmdva.fios.verizon.net) Quit (Read error: No route to host)
[22:17] <gregaf> heh
[22:17] <matthew_> gregaf: nm, got it from the logbot
[22:17] <gregaf> ah, cool
[22:18] <matthew_> gregaf: apparently it has nothing to do with the rebooting OSD :)
[22:18] <gregaf> yeah, I would be surprised if it did
[22:35] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Quit: Leaving.)
[22:37] * lofejndif (~lsqavnbok@28IAADC3J.tor-irc.dnsbl.oftc.net) Quit (Ping timeout: 480 seconds)
[22:46] <- *joao* .
[22:47] * lofejndif (~lsqavnbok@04ZAAB2Y4.tor-irc.dnsbl.oftc.net) has joined #ceph
[23:00] * chutzpah (~chutz@ has joined #ceph
[23:38] * BManojlovic (~steki@ Quit (Remote host closed the connection)
[23:59] <Tv|work> well i just cat'ed 100MB of 0x00 to my terminal and wondered why things are so slow..

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.