#ceph IRC Log

Index

IRC Log for 2013-05-02

Timestamps are in GMT/BST.

[0:00] <PerlStalker> I'd like to create a local mirror since my new ceph nodes will only have local network access.
[0:08] * ScOut3R (~ScOut3R@BC065770.dsl.pool.telekom.hu) has joined #ceph
[0:10] <lurbs> PerlStalker: Try using debmirror, you should be able to mirror it via HTTP easily enough.
[0:10] * BillK (~BillK@58-7-104-61.dyn.iinet.net.au) has joined #ceph
[0:10] * jtang1 (~jtang@79.97.135.214) Quit (Quit: Leaving.)
[0:15] * jtang1 (~jtang@79.97.135.214) has joined #ceph
[0:17] * ScOut3R (~ScOut3R@BC065770.dsl.pool.telekom.hu) Quit (Ping timeout: 480 seconds)
[0:17] * lofejndif (~lsqavnbok@83TAAA2SX.tor-irc.dnsbl.oftc.net) Quit (Quit: gone)
[0:21] * vata (~vata@2607:fad8:4:6:221:5aff:fe2a:d1dd) Quit (Quit: Leaving.)
[0:25] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[0:28] * gmason (~gmason@hpcc-fw.net.msu.edu) Quit (Ping timeout: 480 seconds)
[0:36] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit (Quit: Leaving.)
[0:44] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Quit: Leaving.)
[0:46] * jtang1 (~jtang@79.97.135.214) Quit (Quit: Leaving.)
[0:48] * PerlStalker (~PerlStalk@72.166.192.70) Quit (Quit: ..)
[0:59] * alop (~al592b@71-80-139-200.dhcp.rvsd.ca.charter.com) has joined #ceph
[1:00] * diegows (~diegows@190.190.2.126) has joined #ceph
[1:01] * lurbs (user@uber.geek.nz) Quit (Read error: Connection reset by peer)
[1:01] * lurbs (user@uber.geek.nz) has joined #ceph
[1:01] * brady_ (~brady@rrcs-64-183-4-86.west.biz.rr.com) Quit (Quit: Konversation terminated!)
[1:08] * LeaChim (~LeaChim@176.250.181.7) Quit (Read error: Operation timed out)
[1:17] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) has joined #ceph
[1:18] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Quit: Leaving.)
[1:23] * jlogan (~Thunderbi@2600:c00:3010:1:1::40) Quit (Read error: Connection reset by peer)
[1:23] * jlogan (~Thunderbi@2600:c00:3010:1:1::40) has joined #ceph
[1:27] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[1:27] * loicd (~loic@magenta.dachary.org) has joined #ceph
[1:29] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[1:30] * dwt (~dwt@128-107-239-233.cisco.com) Quit (Quit: Leaving)
[1:48] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[1:48] * loicd (~loic@magenta.dachary.org) has joined #ceph
[2:00] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[2:11] * danieagle (~Daniel@186.214.77.43) Quit (Quit: Inte+ :-) e Muito Obrigado Por Tudo!!! ^^)
[2:13] * coyo (~unf@00017955.user.oftc.net) Quit (Ping timeout: 480 seconds)
[2:15] <sage> sjusthm: look at wip-paranoid? any reason not to merge this?
[2:15] <sage> (no word yet on whether this resolves things for wido)
[2:18] * MarkN (~nathan@142.208.70.115.static.exetel.com.au) has joined #ceph
[2:18] * MarkN (~nathan@142.208.70.115.static.exetel.com.au) has left #ceph
[2:21] * davidz (~Adium@ip68-96-75-123.oc.oc.cox.net) Quit (Quit: Leaving.)
[2:21] * davidz (~Adium@ip68-96-75-123.oc.oc.cox.net) has joined #ceph
[2:23] * sjustlaptop (~sam@71-83-191-116.dhcp.gldl.ca.charter.com) has joined #ceph
[2:24] <joao> sage, around?
[2:24] <sage> yeah
[2:25] * coyo (~unf@71.21.193.106) has joined #ceph
[2:25] <joao> got a peon stuck on leveldb's MakeRoomForWrite() for some 3 hours or so
[2:25] <joao> didn't noticed it as the cluster was working fine
[2:25] <joao> 56M dev/mon.c/store.db/001205.log
[2:25] <sage> oh nice, that's what wido saw on #4851
[2:25] <sage> make a copy of that store so we can reproduce :)
[2:25] <joao> this should be leveldb's log file
[2:26] <joao> copying
[2:26] <joao> not sure how to debug this though
[2:26] <joao> poking the leveldb guys
[2:27] <sage> could try --mon-leveldb-paranoid and see what happens
[2:27] <sage> thats what i asked wido to try
[2:27] <sage> (from wip-paranoid)
[2:27] <sage> but definitely poke #leveldb :)
[2:29] * davidz (~Adium@ip68-96-75-123.oc.oc.cox.net) Quit (Quit: Leaving.)
[2:30] * davidz (~Adium@ip68-96-75-123.oc.oc.cox.net) has joined #ceph
[2:31] <joao> 1<rvagg>1 weetabeex: hours... I'd post details to the leveldb mailing list, that's gotta be a bug
[2:37] <dmick> joao: if it's still stuck, strace it?
[2:38] <joao> well, no longer stuck though, but I did attached gdb to it to check where it was stuck
[2:42] <sage> joao: also http://tracker.ceph.com/issues/4851 has an attached bt.txt that shows it stuck there too
[2:42] <joao> well, just triggered it again
[2:43] <dmick> ah, so, foo, not spinning, blocked
[2:46] * tnt (~tnt@109.130.111.118) Quit (Ping timeout: 480 seconds)
[2:53] * alop (~al592b@71-80-139-200.dhcp.rvsd.ca.charter.com) Quit (Quit: alop)
[3:01] <joao> <weetabeex> rvagg, say, I would assume a batch of some 3.5k ops totaling 2MB in size wouldn't be the issue, but I may be dead wrong; any thoughts?
[3:01] <joao> 1<rvagg>1 weetabeex: I couldn't say, 2MB is a large batch and will overflow into cascading compacts, but then again it should be able to handle it
[3:01] <joao> sage, ^
[3:03] <joao> transaction's size is capped at 1 MB by default though; I'm the one who increased the payload size while playing with leveldb
[3:03] <joao> so that might not be it at all
[3:10] * diegows (~diegows@190.190.2.126) Quit (Ping timeout: 480 seconds)
[3:17] * Cube (~Cube@12.248.40.138) Quit (Quit: Leaving.)
[3:18] * MarkN (~nathan@142.208.70.115.static.exetel.com.au) has joined #ceph
[3:18] * MarkN (~nathan@142.208.70.115.static.exetel.com.au) has left #ceph
[3:20] * eagen (~david@c-174-53-214-214.hsd1.mn.comcast.net) has joined #ceph
[3:23] * yehuda_hm (~yehuda@99-48-177-65.lightspeed.irvnca.sbcglobal.net) Quit (Quit: Leaving)
[3:30] * yehuda_hm (~yehuda@2602:306:330b:1410:a455:4b9b:5d5a:bdd9) has joined #ceph
[3:36] * MarkN (~nathan@142.208.70.115.static.exetel.com.au) has joined #ceph
[3:37] * MarkN (~nathan@142.208.70.115.static.exetel.com.au) has left #ceph
[3:38] * portante|ltp (~user@c-24-63-226-65.hsd1.ma.comcast.net) has joined #ceph
[3:38] * dpippenger (~riven@206-169-78-213.static.twtelecom.net) Quit (Remote host closed the connection)
[3:41] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[3:41] * loicd (~loic@2a01:e35:2eba:db10:7dd3:6565:e69f:225f) has joined #ceph
[3:43] * john_barbee__ (~jbarbee@c-98-226-73-253.hsd1.in.comcast.net) has joined #ceph
[3:47] * MarkN (~nathan@142.208.70.115.static.exetel.com.au) has joined #ceph
[3:48] * MarkN (~nathan@142.208.70.115.static.exetel.com.au) has left #ceph
[3:48] * john_barbee_ (~jbarbee@c-98-226-73-253.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[3:49] * rturk is now known as rturk-away
[3:51] * john_barbee__ (~jbarbee@c-98-226-73-253.hsd1.in.comcast.net) Quit (Quit: ChatZilla 0.9.90 [Firefox 21.0/20130425162858])
[3:57] * rustam (~rustam@94.15.91.30) Quit (Remote host closed the connection)
[4:01] * MarkN (~nathan@142.208.70.115.static.exetel.com.au) has joined #ceph
[4:01] * MarkN (~nathan@142.208.70.115.static.exetel.com.au) has left #ceph
[4:17] * rustam (~rustam@94.15.91.30) has joined #ceph
[4:19] * sjustlaptop (~sam@71-83-191-116.dhcp.gldl.ca.charter.com) Quit (Quit: Leaving.)
[4:19] * rustam (~rustam@94.15.91.30) Quit (Remote host closed the connection)
[4:20] * sjustlaptop (~sam@71-83-191-116.dhcp.gldl.ca.charter.com) has joined #ceph
[4:20] * DarkAce-Z (~BillyMays@50.107.54.92) has joined #ceph
[4:20] * DarkAce-Z (~BillyMays@50.107.54.92) Quit (Read error: Connection reset by peer)
[4:21] * john_barbee_ (~jbarbee@c-98-226-73-253.hsd1.in.comcast.net) has joined #ceph
[4:24] * DarkAceZ (~BillyMays@50.107.54.92) Quit (Ping timeout: 480 seconds)
[4:30] * rustam (~rustam@94.15.91.30) has joined #ceph
[4:32] * rustam (~rustam@94.15.91.30) Quit (Remote host closed the connection)
[4:41] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) has joined #ceph
[4:45] * sjusthm (~sam@71-83-191-116.dhcp.gldl.ca.charter.com) Quit (Remote host closed the connection)
[4:46] * coyo (~unf@00017955.user.oftc.net) Quit (Quit: F*ck you, I'm a daemon.)
[4:58] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) has joined #ceph
[5:03] * loicd (~loic@2a01:e35:2eba:db10:7dd3:6565:e69f:225f) Quit (Quit: Leaving.)
[5:03] * loicd (~loic@magenta.dachary.org) has joined #ceph
[5:09] * wschulze1 (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) has joined #ceph
[5:13] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) Quit (Ping timeout: 480 seconds)
[5:36] * rustam (~rustam@94.15.91.30) has joined #ceph
[5:38] * rustam (~rustam@94.15.91.30) Quit (Remote host closed the connection)
[5:45] * ScOut3R (~ScOut3R@BC065770.dsl.pool.telekom.hu) has joined #ceph
[5:54] * ScOut3R (~ScOut3R@BC065770.dsl.pool.telekom.hu) Quit (Ping timeout: 480 seconds)
[5:57] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[5:57] * loicd (~loic@magenta.dachary.org) has joined #ceph
[5:59] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Quit: Leaving.)
[6:04] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[6:16] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[6:16] * loicd (~loic@magenta.dachary.org) has joined #ceph
[6:21] * wschulze1 (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) Quit (Quit: Leaving.)
[6:24] * KindTwo (KindOne@50.96.83.6) has joined #ceph
[6:26] * KindOne (KindOne@0001a7db.user.oftc.net) Quit (Ping timeout: 480 seconds)
[6:26] * KindTwo is now known as KindOne
[6:29] * MarkN (~nathan@142.208.70.115.static.exetel.com.au) has joined #ceph
[6:29] * MarkN (~nathan@142.208.70.115.static.exetel.com.au) has left #ceph
[6:55] * wogri_risc (~wogri_ris@ro.risc.uni-linz.ac.at) has joined #ceph
[6:55] * sjustlaptop (~sam@71-83-191-116.dhcp.gldl.ca.charter.com) Quit (Ping timeout: 480 seconds)
[7:02] * rustam (~rustam@94.15.91.30) has joined #ceph
[7:03] * rustam (~rustam@94.15.91.30) Quit (Remote host closed the connection)
[7:05] * scuttlemonkey (~scuttlemo@c-69-244-181-5.hsd1.mi.comcast.net) Quit (Ping timeout: 480 seconds)
[7:12] * rustam (~rustam@94.15.91.30) has joined #ceph
[7:13] * rustam (~rustam@94.15.91.30) Quit (Remote host closed the connection)
[7:53] * eternaleye_ (~eternaley@cl-43.lax-02.us.sixxs.net) has joined #ceph
[7:54] * eternaleye (~eternaley@c-50-132-41-203.hsd1.wa.comcast.net) Quit (Read error: Connection reset by peer)
[7:54] * eternaleye_ is now known as eternaleye
[8:09] * scuttlemonkey (~scuttlemo@c-69-244-181-5.hsd1.mi.comcast.net) has joined #ceph
[8:09] * ChanServ sets mode +o scuttlemonkey
[8:14] * fridad (~fridad@b.clients.kiwiirc.com) has joined #ceph
[8:14] * eternaleye (~eternaley@cl-43.lax-02.us.sixxs.net) Quit (Remote host closed the connection)
[8:14] <fridad> i've seen a cuttlefish branch in git? Is it released?
[8:16] * eternaleye (~eternaley@c-50-132-41-203.hsd1.wa.comcast.net) has joined #ceph
[8:19] <wogri_risc> fridad: at least there hasn't been an announcement on the Mailinglilst.
[8:21] * scuttlemonkey (~scuttlemo@c-69-244-181-5.hsd1.mi.comcast.net) Quit (Ping timeout: 480 seconds)
[8:25] <fridad> yes i know ;-)
[8:30] * bergerx_ (~bekir@78.188.101.175) has joined #ceph
[8:33] * rustam (~rustam@94.15.91.30) has joined #ceph
[8:34] * rustam (~rustam@94.15.91.30) Quit (Remote host closed the connection)
[8:35] * Vjarjadian (~IceChat77@90.214.208.5) Quit (Quit: For Sale: Parachute. Only used once, never opened, small stain.)
[8:38] * rustam (~rustam@94.15.91.30) has joined #ceph
[8:41] * rustam (~rustam@94.15.91.30) Quit (Remote host closed the connection)
[8:41] * niklas (~niklas@2001:7c0:409:8001::32:115) has joined #ceph
[8:41] * niklas_ (~niklas@2001:7c0:409:8001::32:115) Quit (Read error: Connection reset by peer)
[8:42] * tnt (~tnt@109.130.111.118) has joined #ceph
[8:45] * jtang1 (~jtang@79.97.135.214) has joined #ceph
[9:01] * niklas_ (~niklas@2001:7c0:409:8001::32:115) has joined #ceph
[9:01] * niklas (~niklas@2001:7c0:409:8001::32:115) Quit (Read error: Connection reset by peer)
[9:09] * allsystemsarego (~allsystem@5-12-37-186.residential.rdsnet.ro) has joined #ceph
[9:11] * niklas_ (~niklas@2001:7c0:409:8001::32:115) Quit (Ping timeout: 480 seconds)
[9:13] * ShaunR (~ShaunR@staff.ndchost.com) has joined #ceph
[9:15] * ShaunR- (~ShaunR@staff.ndchost.com) Quit (Read error: Connection reset by peer)
[9:20] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[9:24] * eschnou (~eschnou@85.234.217.115.static.edpnet.net) has joined #ceph
[9:25] * niklas (~niklas@2001:7c0:409:8001::32:115) has joined #ceph
[9:29] * leseb (~Adium@83.167.43.235) has joined #ceph
[9:30] * gucki (~smuxi@HSI-KBW-078-042-028-165.hsi3.kabel-badenwuerttemberg.de) has joined #ceph
[9:56] * illuminatis (~illuminat@0001adba.user.oftc.net) Quit (Ping timeout: 480 seconds)
[9:56] * l0nk (~alex@83.167.43.235) has joined #ceph
[9:56] * LeaChim (~LeaChim@176.250.181.7) has joined #ceph
[9:59] * jbd_ (~jbd_@34322hpv162162.ikoula.com) has joined #ceph
[10:05] * illuminatis (~illuminat@0001adba.user.oftc.net) has joined #ceph
[10:10] * jtang1 (~jtang@79.97.135.214) Quit (Quit: Leaving.)
[10:18] * tnt (~tnt@109.130.111.118) Quit (Ping timeout: 480 seconds)
[10:28] * tnt (~tnt@212-166-48-236.win.be) has joined #ceph
[10:32] * dxd828 (~dxd828@195.191.107.205) has joined #ceph
[10:37] * rustam (~rustam@94.15.91.30) has joined #ceph
[10:39] * rustam (~rustam@94.15.91.30) Quit (Remote host closed the connection)
[10:40] * jtang1 (~jtang@79.97.135.214) has joined #ceph
[10:43] * loicd (~loic@3.46-14-84.ripe.coltfrance.com) has joined #ceph
[10:52] * jtang1 (~jtang@79.97.135.214) Quit (Ping timeout: 480 seconds)
[10:54] * v0id (~v0@91-115-228-104.adsl.highway.telekom.at) has joined #ceph
[11:01] * vo1d (~v0@62-46-172-22.adsl.highway.telekom.at) Quit (Ping timeout: 480 seconds)
[11:09] * ScOut3R (~ScOut3R@212.96.47.215) has joined #ceph
[11:27] * Rocky (~r.nap@188.205.52.204) Quit (Quit: bleh)
[11:27] * Rocky (~r.nap@188.205.52.204) has joined #ceph
[11:43] <wido> joao: The leveldb stuck thing indeed resolved for me in that case
[11:43] <wido> That mon is working again
[11:50] * jtang1 (~jtang@79.97.135.214) has joined #ceph
[11:58] <joao> wido, the paranoid checks?
[11:58] <wido> joao: Indeed. It resolved the single monitor thing
[11:58] * jtang1 (~jtang@79.97.135.214) Quit (Ping timeout: 480 seconds)
[11:58] <wido> I still have 3 monitors which won't elect
[11:58] <wido> But that's a different case, since I tried the paranoid checks there as well
[11:59] <wido> 1 mon is leader, other is electing and other one is synchronizing
[11:59] <joao> hmm
[12:00] <joao> wido, fwiw, the paranoid checks also fixed my dev cluster
[12:00] <joao> but then again, I also adjusted the write buffer size
[12:00] <joao> so not sure which was which
[12:00] <wido> joao: cool. I'll check later with gdb to see in what state the threads are on those 3 mons
[12:02] * Cube (~Cube@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[12:12] <Zethrok> Wild shot: does anyone know if qemu-img convert can create format 2 rbd images (going from qcow2 -> rbd)? Seems to default to format 1.
[12:14] <Zethrok> Or alternatively any way to set a default format level in rbd or well, pretty much anything which doesn't require me to first export and then import image in format 2.
[12:15] <fridad> Zethrok: i really would like to know that too
[12:17] * vanion (~oftc-webi@cm59.zeta117.maxonline.com.sg) has joined #ceph
[12:22] <vanion> hi all. I'm having problems with ceph-deploy on the centos 6... getting lots of errors following the instructions at http://ceph.com/docs/master/start/quick-ceph-deploy/
[12:22] <nigwil> Zethrok: back in Nov-2012 it was asked with this response:
[12:22] <nigwil> [1:00] <benpol> Anyone know if qemu-img can create "format 2" images?
[12:22] <nigwil> [1:00] <joshd> elder: mount an fs on top of an rbd device, and then try to unmap it. it should fail with EBUSY
[12:22] <vanion> seems like ceph-0.56.3 doesn't like the conf file generated by ceph-deploy... anybody has any tips/advice?
[12:22] <nigwil> [1:00] <joshd> benpol: no, it can't
[12:22] <nigwil> [1:02] <elder> Ok.
[12:22] <nigwil> [1:03] <benpol> another way to lose sparseness is in converting a "format 1" image to "format 2" via the export/import path.
[12:22] <nigwil> [1:04] <joshd> benpol: yeah, import supports sparse files, but export doesn't yet
[12:25] <fridad> So there's no way to use v2 with qemu-img?
[12:26] <Zethrok> bummer - was hoping at least the export/import was a possibility - but seems we'll loose sparseness
[12:26] <Zethrok> nigwil: thanks though
[12:27] <fridad> Zethrok: sure? I thought that rbd-import skips zeros.
[12:28] <Zethrok> fridad: ahh, yea - think you're right
[12:29] <fridad> sadly there's also no patch for qemu-img available
[12:29] <wido> I checked the latest qemu source code
[12:29] <wido> It calls rbd_create from librbd which indeed doesn't support v2
[12:29] <wido> only v1
[12:30] <wido> http://git.qemu.org/?p=qemu.git;a=blob;f=block/rbd.c#l364
[12:30] <Zethrok> wido: ok, will see if we can put in layer of exporting then importing with --format 2 after the qemu-img - annoying, but should work I guess. Thanks!
[12:30] * vanion (~oftc-webi@cm59.zeta117.maxonline.com.sg) Quit (Quit: Page closed)
[12:30] * vanion (~vanion@cm59.zeta117.maxonline.com.sg) has joined #ceph
[12:31] <wido> librbd has rbd_create2 and rbd_create3
[12:31] <wido> Zethrok: https://github.com/ceph/ceph/blob/master/src/include/rbd/librbd.h#L72
[12:32] <vanion> back
[12:33] <vanion> hi all, anyone can help me with ceph-deploy?
[12:34] <fridad> wido: yeah so somebody needs to port qmu to rbd_create3 and
[12:35] <wido> fridad: feel free ;)
[12:36] <fridad> wido: i can't ;-( no C dev
[12:36] <vanion> "ceph-deploy mon create ceph1" throws error "pushy.protocol.proxy.ExceptionProxy: Command '['service', 'ceph', 'start', 'mon.ceph1']' returned non-zero exit status 1"
[12:36] <wido> fridad: np :) You can always create an issue for it in the tracker though
[12:36] <wido> vanion: Have you tried starting the mon manually?
[12:36] <wido> I've never used ceph-deploy though
[12:37] <vanion> wido: I would like to... but the installations instructions seems to have shifted
[12:38] <vanion> is ceph-deploy the way to go, or maybe not?
[12:39] <vanion> ceph-deploy doesn't create the needed [mon.ceph1] part of the configuration
[12:39] <vanion> did they change the configuration parameters recently?
[12:42] <wido> vanion: I have no idea, I know the tool exists, but that's all
[12:43] <wido> checked the most recent docs?
[12:47] <vanion> wido, thanks, I think I'll try without the tool then. :)
[13:03] <LeaChim> Hmm. Is there a way to dump the MDS's view of which files are in use? My mountpoint reports 12GB used, but the data pool is currently using 86GB. I'm guessing the MDS isn't removing the objects from deleted files for some reason..
[13:10] * john_barbee_ (~jbarbee@c-98-226-73-253.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[13:14] * jtang1 (~jtang@79.97.135.214) has joined #ceph
[13:22] * jtang1 (~jtang@79.97.135.214) Quit (Ping timeout: 480 seconds)
[13:38] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) has joined #ceph
[13:49] * rustam (~rustam@94.15.91.30) has joined #ceph
[13:51] * rustam (~rustam@94.15.91.30) Quit (Remote host closed the connection)
[13:53] * nhm (~nhm@65-128-150-185.mpls.qwest.net) Quit (Ping timeout: 480 seconds)
[13:57] * mrjack (mrjack@office.smart-weblications.net) Quit (Ping timeout: 480 seconds)
[13:58] * rustam (~rustam@94.15.91.30) has joined #ceph
[13:58] * Cube (~Cube@cpe-76-95-217-129.socal.res.rr.com) Quit (Quit: Leaving.)
[13:59] * rustam (~rustam@94.15.91.30) Quit (Remote host closed the connection)
[14:08] * jtang1 (~jtang@79.97.135.214) has joined #ceph
[14:16] * jtang1 (~jtang@79.97.135.214) Quit (Ping timeout: 480 seconds)
[14:37] * vanion (~vanion@cm59.zeta117.maxonline.com.sg) Quit (Ping timeout: 480 seconds)
[14:38] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) has joined #ceph
[14:41] * DarkAceZ (~BillyMays@50.107.54.92) has joined #ceph
[14:47] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[14:57] * wogri_risc (~wogri_ris@ro.risc.uni-linz.ac.at) Quit (Remote host closed the connection)
[15:02] * jtang1 (~jtang@79.97.135.214) has joined #ceph
[15:04] * danieagle (~Daniel@186.214.77.43) has joined #ceph
[15:05] * scuttlemonkey (~scuttlemo@c-69-244-181-5.hsd1.mi.comcast.net) has joined #ceph
[15:05] * ChanServ sets mode +o scuttlemonkey
[15:10] * jtang1 (~jtang@79.97.135.214) Quit (Ping timeout: 480 seconds)
[15:15] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) Quit (Quit: Leaving.)
[15:18] * TiCPU (~jeromepou@190-130.cgocable.ca) Quit (Ping timeout: 480 seconds)
[15:24] * Vjarjadian (~IceChat77@90.214.208.5) has joined #ceph
[15:26] * rustam (~rustam@94.15.91.30) has joined #ceph
[15:27] * rustam (~rustam@94.15.91.30) Quit (Remote host closed the connection)
[15:29] * TiCPU (~jeromepou@190-130.cgocable.ca) has joined #ceph
[15:31] * TiCPU_ (~jeromepou@190-130.cgocable.ca) has joined #ceph
[15:33] <kylehutson> mikedawson and jmlowe: Sorry, I was home with a sick kid yesterday. Cluster is back to HEALTH_OK. Next time I see either of you, I owe you a beverage of your choice.
[15:34] <mikedawson> kylehutson: nice! glad to hear it
[15:34] <jmlowe> ain't no thang, SC '13 maybe?
[15:34] <kylehutson> I'll be there.
[15:37] * TiCPU (~jeromepou@190-130.cgocable.ca) Quit (Ping timeout: 480 seconds)
[15:41] <Teduardo> any work being done to add location awareness to ceph?
[15:43] <Teduardo> also what is the best devops tool to use to maintain/build a ceph cluster?
[15:44] * portante|ltp (~user@c-24-63-226-65.hsd1.ma.comcast.net) Quit (Ping timeout: 480 seconds)
[15:45] * danieagle (~Daniel@186.214.77.43) Quit (Quit: Inte+ :-) e Muito Obrigado Por Tudo!!! ^^)
[15:47] * tziOm (~bjornar@194.19.106.242) has joined #ceph
[15:52] * gmason (~gmason@hpcc-fw.net.msu.edu) has joined #ceph
[15:52] * wschulze (~wschulze@38.98.115.249) has joined #ceph
[15:52] * tnt (~tnt@212-166-48-236.win.be) Quit (Ping timeout: 480 seconds)
[15:54] * b1tbkt (~Peekaboo@68-184-193-142.dhcp.stls.mo.charter.com) Quit (Ping timeout: 480 seconds)
[15:56] * jtang1 (~jtang@79.97.135.214) has joined #ceph
[15:56] * leseb (~Adium@83.167.43.235) Quit (Read error: Connection reset by peer)
[15:57] * leseb (~Adium@83.167.43.235) has joined #ceph
[15:59] * Vjarjadian (~IceChat77@90.214.208.5) Quit (Quit: I used to think I was indecisive, but now I'm not too sure.)
[16:05] * jtang1 (~jtang@79.97.135.214) Quit (Ping timeout: 480 seconds)
[16:05] * tnt (~tnt@109.130.111.118) has joined #ceph
[16:19] * aliguori (~anthony@cpe-70-112-157-87.austin.res.rr.com) Quit (Quit: Ex-Chat)
[16:20] * portante|ltp (~user@66.187.233.207) has joined #ceph
[16:23] * sean (sean@118.133.96.107) has joined #ceph
[16:23] <sean> nice to everybody
[16:29] * rustam (~rustam@94.15.91.30) has joined #ceph
[16:30] * rustam (~rustam@94.15.91.30) Quit (Remote host closed the connection)
[16:31] * vanion (~vanion@cm59.zeta117.maxonline.com.sg) has joined #ceph
[16:40] * vata (~vata@2607:fad8:4:6:221:5aff:fe2a:d1dd) has joined #ceph
[16:42] * PerlStalker (~PerlStalk@72.166.192.70) has joined #ceph
[16:46] * sean (sean@118.133.96.107) Quit ()
[16:51] * jtang1 (~jtang@79.97.135.214) has joined #ceph
[16:55] * aliguori (~anthony@32.97.110.51) has joined #ceph
[16:59] * jtang1 (~jtang@79.97.135.214) Quit (Ping timeout: 480 seconds)
[16:59] * eschnou (~eschnou@85.234.217.115.static.edpnet.net) Quit (Read error: Connection reset by peer)
[17:02] * Wolff_John (~jwolff@ftp.monarch-beverage.com) has joined #ceph
[17:10] * diegows (~diegows@200.68.116.185) has joined #ceph
[17:11] * bergerx_ (~bekir@78.188.101.175) Quit (Quit: Leaving.)
[17:13] * wschulze (~wschulze@38.98.115.249) Quit (Quit: Leaving.)
[17:19] * wschulze (~wschulze@38.98.115.249) has joined #ceph
[17:20] * glowell1 (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[17:23] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) has joined #ceph
[17:26] * glowell1 (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) has joined #ceph
[17:26] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) Quit (Read error: Connection reset by peer)
[17:28] * rustam (~rustam@94.15.91.30) has joined #ceph
[17:30] * rustam (~rustam@94.15.91.30) Quit (Remote host closed the connection)
[17:31] * loicd (~loic@3.46-14-84.ripe.coltfrance.com) Quit (Ping timeout: 480 seconds)
[17:32] * tziOm (~bjornar@194.19.106.242) Quit (Remote host closed the connection)
[17:35] * ScOut3R (~ScOut3R@212.96.47.215) Quit (Ping timeout: 480 seconds)
[17:36] * leseb (~Adium@83.167.43.235) Quit (Read error: Connection reset by peer)
[17:37] * nhm (~nhm@207.229.236.209) has joined #ceph
[17:38] * leseb (~Adium@83.167.43.235) has joined #ceph
[17:45] * jtang1 (~jtang@79.97.135.214) has joined #ceph
[17:45] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[17:45] * alop (~al592b@71-80-139-200.dhcp.rvsd.ca.charter.com) has joined #ceph
[17:45] * alop (~al592b@71-80-139-200.dhcp.rvsd.ca.charter.com) Quit ()
[17:51] * john_barbee (~jbarbee@23-25-46-97-static.hfc.comcastbusiness.net) Quit (Remote host closed the connection)
[17:53] * jtang1 (~jtang@79.97.135.214) Quit (Ping timeout: 480 seconds)
[17:55] * sagewk (~sage@2607:f298:a:607:b0f4:5462:45db:ca49) Quit (Quit: Leaving.)
[18:03] * sagewk (~sage@2607:f298:a:607:c54c:c777:f195:239a) has joined #ceph
[18:03] <sagewk> joao: ping!
[18:04] <joao> morning sage
[18:04] * vanion (~vanion@cm59.zeta117.maxonline.com.sg) Quit (Remote host closed the connection)
[18:04] <joao> also, sagewk, pong
[18:04] * nhm (~nhm@207.229.236.209) Quit (Ping timeout: 480 seconds)
[18:07] * sagewk (~sage@2607:f298:a:607:c54c:c777:f195:239a) Quit (Remote host closed the connection)
[18:08] * BillK (~BillK@58-7-104-61.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[18:09] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) has joined #ceph
[18:09] * glowell1 (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) Quit (Read error: Connection reset by peer)
[18:09] <benner> what's difference between weight in osd dump and osd tree commands? in osd tree i suppose there is crush weight. what about osd dump?
[18:10] * wschulze (~wschulze@38.98.115.249) Quit (Quit: Leaving.)
[18:22] * eschnou (~eschnou@252.94-201-80.adsl-dyn.isp.belgacom.be) has joined #ceph
[18:28] * aliguori (~anthony@32.97.110.51) Quit (Quit: Ex-Chat)
[18:30] * aliguori (~anthony@32.97.110.51) has joined #ceph
[18:31] * Vjarjadian (~IceChat77@90.214.208.5) has joined #ceph
[18:33] * hybrid512 (~walid@LPoitiers-156-86-25-85.w193-248.abo.wanadoo.fr) Quit (Quit: Leaving.)
[18:33] * sagewk (~sage@2607:f298:a:607:c54c:c777:f195:239a) has joined #ceph
[18:34] * eschnou (~eschnou@252.94-201-80.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[18:34] * scuttlemonkey (~scuttlemo@c-69-244-181-5.hsd1.mi.comcast.net) Quit (Ping timeout: 480 seconds)
[18:34] * loicd (~loic@magenta.dachary.org) has joined #ceph
[18:36] * BManojlovic (~steki@fo-d-130.180.254.37.targo.rs) has joined #ceph
[18:39] * jtang1 (~jtang@79.97.135.214) has joined #ceph
[18:39] * aliguori_ (~anthony@32.97.110.51) has joined #ceph
[18:39] * sagewk (~sage@2607:f298:a:607:c54c:c777:f195:239a) Quit (Quit: Leaving.)
[18:40] * sagewk (~sage@2607:f298:a:607:c54c:c777:f195:239a) has joined #ceph
[18:43] * leseb (~Adium@83.167.43.235) Quit (Quit: Leaving.)
[18:43] * l0nk (~alex@83.167.43.235) Quit (Quit: Leaving.)
[18:45] * sagewk (~sage@2607:f298:a:607:c54c:c777:f195:239a) Quit (Remote host closed the connection)
[18:45] * aliguori (~anthony@32.97.110.51) Quit (Ping timeout: 480 seconds)
[18:46] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Read error: Operation timed out)
[18:47] * jtang1 (~jtang@79.97.135.214) Quit (Ping timeout: 480 seconds)
[18:55] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[18:55] * eschnou (~eschnou@252.94-201-80.adsl-dyn.isp.belgacom.be) has joined #ceph
[18:56] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[18:58] * Vjarjadian (~IceChat77@90.214.208.5) Quit (Quit: IceChat - Keeping PC's cool since 2000)
[19:01] * yehudasa (~yehudasa@2607:f298:a:607:d499:a5eb:5688:9814) Quit (Remote host closed the connection)
[19:02] * gmason_ (~gmason@hpcc-fw.net.msu.edu) has joined #ceph
[19:03] * rturk-away is now known as rturk
[19:03] * yehudasa (~yehudasa@2607:f298:a:607:c018:c2d6:a679:684d) has joined #ceph
[19:04] * gmason (~gmason@hpcc-fw.net.msu.edu) Quit (Ping timeout: 480 seconds)
[19:05] * eschnou (~eschnou@252.94-201-80.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[19:09] * brady (~brady@rrcs-64-183-4-86.west.biz.rr.com) has joined #ceph
[19:13] * eschnou (~eschnou@104.207-201-80.adsl-dyn.isp.belgacom.be) has joined #ceph
[19:15] * Teduardo (~DW-10297@dhcp92.cmh.ee.net) Quit (Ping timeout: 480 seconds)
[19:16] * sagewk (~sage@2607:f298:a:607:c54c:c777:f195:239a) has joined #ceph
[19:17] * nhm (~nhm@207.229.236.209) has joined #ceph
[19:18] * tziOm (~bjornar@ti0099a340-dhcp0870.bb.online.no) has joined #ceph
[19:23] <imjustmatthew> Will a client failover from the mon it initially attaches to as the monitor cluster degrades?
[19:25] <benner> imjustmatthew: yes
[19:25] <imjustmatthew> benner: okay, thanks!
[19:28] * sjustlaptop (~sam@2607:f298:a:697:2015:b217:f5be:3ab1) has joined #ceph
[19:28] * Wolff_John_ (~jwolff@ftp.monarch-beverage.com) has joined #ceph
[19:30] * Wolff_John (~jwolff@ftp.monarch-beverage.com) Quit (Read error: Connection reset by peer)
[19:30] * Wolff_John_ is now known as Wolff_John
[19:33] * jtang1 (~jtang@79.97.135.214) has joined #ceph
[19:36] * sjustlaptop (~sam@2607:f298:a:697:2015:b217:f5be:3ab1) Quit (Ping timeout: 480 seconds)
[19:41] * jtang1 (~jtang@79.97.135.214) Quit (Ping timeout: 480 seconds)
[19:42] * nhm (~nhm@207.229.236.209) Quit (Ping timeout: 480 seconds)
[19:46] * nhm (~nhm@207.229.236.209) has joined #ceph
[19:50] * gillesMo (~gillesMo@00012912.user.oftc.net) has joined #ceph
[19:51] * tkensiski (~tkensiski@209.66.64.134) has joined #ceph
[19:51] * tkensiski (~tkensiski@209.66.64.134) has left #ceph
[19:52] * pconnelly (~pconnelly@71-93-233-229.dhcp.mdfd.or.charter.com) has joined #ceph
[19:55] * nhm (~nhm@207.229.236.209) Quit (Ping timeout: 480 seconds)
[19:58] * rturk is now known as rturk-away
[19:58] * nhm (~nhm@207.229.236.209) has joined #ceph
[19:59] * gillesMo (~gillesMo@00012912.user.oftc.net) Quit (Quit: Konversation terminated!)
[19:59] <sagewk> wido: was mon.3 hung after also being started iwth --mon-leveldb-paranoid?
[20:04] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[20:08] * Cube (~Cube@12.248.40.138) has joined #ceph
[20:08] <wido> sagewk: Yes
[20:08] <wido> I'll be back in about an hour
[20:09] * nhm (~nhm@207.229.236.209) Quit (Ping timeout: 480 seconds)
[20:10] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[20:10] * loicd (~loic@magenta.dachary.org) has joined #ceph
[20:12] <sagewk> wido: damn. thanks!
[20:13] <sjust> sagewk: I'm not sure that option actually causes repair
[20:13] <sagewk> mikedawson: around?
[20:14] <mikedawson> sagewk: yep
[20:14] <sagewk> mikedawson: do you have a bit of time to test something?
[20:14] <mikedawson> sure\
[20:14] <sagewk> mikedawson: wondering if disabling the leveldb compaction, but running with teh smaller block size will avoid the store.db growth
[20:15] <mikedawson> that would be nice... I feel like I'm chewing SSD r/w and bandwidth as-is
[20:15] * jtang1 (~jtang@79.97.135.214) has joined #ceph
[20:15] <mikedawson> what config do you want. What branch/commit?
[20:15] <sagewk> can you upgrade to latest next, and put 'mon compact on trim = false' in your ceph.conf, and restart? the default block size is now 64KB. hopefully that will work
[20:16] <mikedawson> will do
[20:16] <sagewk> er, 32KB actually.. even better.
[20:16] <sjust> sagewk: you need to use the RepairDB method to actually repair the db apparently
[20:20] * tnt_ (~tnt@109.130.111.54) has joined #ceph
[20:26] * tnt (~tnt@109.130.111.118) Quit (Ping timeout: 480 seconds)
[20:27] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) has joined #ceph
[20:31] * nhorman (~nhorman@hmsreliant.think-freely.org) has joined #ceph
[20:36] <mikedawson> sagewk: before I push ceph.conf, do I still need "mon compact on bootstrap = false"?
[20:37] <mikedawson> sagewk: that should be the default for next/cuttlefish, right?
[20:45] * aliguori_ is now known as aliguori
[20:48] * jmlowe (~Adium@c-71-201-31-207.hsd1.in.comcast.net) Quit (Quit: Leaving.)
[20:48] * jmlowe (~Adium@c-71-201-31-207.hsd1.in.comcast.net) has joined #ceph
[20:48] * jmlowe (~Adium@c-71-201-31-207.hsd1.in.comcast.net) Quit ()
[20:50] * eschnou (~eschnou@104.207-201-80.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[20:52] * kiran (~kiran@proxy-rtp-1.cisco.com) has joined #ceph
[20:53] * portante` (~user@66.187.233.206) has joined #ceph
[20:55] * portante|ltp (~user@66.187.233.207) Quit (Read error: Operation timed out)
[20:55] <mikedawson> sjust: sagewk just had me upgrade to test mon issues on next, on four OSDs I hit: http://pastebin.com/raw.php?i=NurC0ibS Is that your area?
[20:56] * portante|afk is now known as portante
[20:59] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) has joined #ceph
[21:07] * jmlowe (~Adium@2001:18e8:2:28cf:f000::dab8) has joined #ceph
[21:10] * caleb_miles (~caleb@65-183-137-95-dhcp.burlingtontelecom.net) has joined #ceph
[21:10] * caleb_miles (~caleb@65-183-137-95-dhcp.burlingtontelecom.net) has left #ceph
[21:12] * eschnou (~eschnou@104.207-201-80.adsl-dyn.isp.belgacom.be) has joined #ceph
[21:16] * stefunel (~stefunel@static.38.162.46.78.clients.your-server.de) Quit (Read error: Operation timed out)
[21:16] <imjustmatthew> I just switched to the "cuttlefish" branch from gitbuilder and am getting stuck with non-responsive mons; they reach quorum and work for a few seconds, then the leader just spams its own log with .accept messages like "2013-05-02 15:15:17.035709 7f2bcd086700 1 -- [2001:470:8:dd9::20]:6789/0 >> :/0 pipe(0x4021500 sd=410 :6789 s=0 pgs=0 cs=0 l=0).accept sd=410 [2001:470:8:dd9::21]:39322/0""
[21:17] <imjustmatthew> Does this look familiar as a known issue?
[21:20] <mikedawson> imjustmatthew: do they stay in quorum or do one or more fall out of quorum? I've seen lots of this leading up to cuttlefish (but I haven't tested that branch). Also, what are the sizes of your monitor store.db dirs?
[21:20] <imjustmatthew> It's weird they don't seem to loose quorum, but after I kill the leader the mon cluster elects a new leader and recovers, but the next leader then dies after a few seocnds.
[21:21] <mikedawson> imjustmatthew: define die? ceph-mon process goes away, segfault, assert, falls out of quorum, etc?
[21:22] <imjustmatthew> Mikedawson: On mon.a store.db is only 67MB, .b is 58MB, .d is 11M
[21:23] <imjustmatthew> mikedawson: Becomes non-responsive, but still runs and keeps spaming its own log
[21:23] * pconnelly (~pconnelly@71-93-233-229.dhcp.mdfd.or.charter.com) Quit (Quit: pconnelly)
[21:23] <imjustmatthew> I'm upping the logging now and will restart them to see if that's more revelaing
[21:24] <sjust> mikedawson: that's really strange
[21:24] <sjust> mikedawson: can you tar up the omap directory from one of those osds and put in in cephdrop?
[21:25] <cjh_> if i nuke the the directories on the monitors is it possible to restore the cluster to a functioning state?
[21:25] <mikedawson> imjustmatthew: ok. mon sizes aren't of concern for me (with the wrong circumstances, mine can grow several GBs/day). With logging up, look for the trends of (electing, probing, sync) instead of (leader, peon)
[21:27] <mikedawson> sjust: one of the others says "Error initializing leveldb: Corruption: checksum mismatch" instead of the pastebin error
[21:27] <sjust> yeah, it's due to the paranoid checking
[21:27] * eschnou (~eschnou@104.207-201-80.adsl-dyn.isp.belgacom.be) Quit (Quit: Leaving)
[21:27] <mikedawson> sjust: I'll let you know when they are uploaded
[21:27] <sjust> k
[21:27] * eschnou (~eschnou@104.207-201-80.adsl-dyn.isp.belgacom.be) has joined #ceph
[21:28] <sagewk> imjustmatthew: attach with gdb and do 'thread apply all bt'. i susepct you will see leveldb doing its WaitForNotFull thing :(
[21:28] <darkfaded> inktankers, something OT: could one of you take a pick of your view during lunch? i forgot to take a pic while there and i'd love to print it out for my little one
[21:28] <darkfaded> s/pick/pic/
[21:29] * davidz (~Adium@ip68-96-75-123.oc.oc.cox.net) Quit (Quit: Leaving.)
[21:29] <imjustmatthew> sagewk: Okay, I'll look at that
[21:30] <dmick> darkfaded: PM me; I may have one already
[21:31] <mikedawson> sjust: mikedawson-osd.42-omap.tar.gz is cephdropped. It is the one with checksum mismatch. I'll get you an example of the other error next
[21:31] <sjust> great, thanks
[21:33] <sjust> hmm
[21:34] <sjust> interestingly, it checks correctly, the osd's structure is fine
[21:35] <mikedawson> sjust: it was working, I upgraded to next 30 mins ago, restarted ceph, then the failure. 3 checksum mismatchs, and one "error in middle of record", 62 osds working properly
[21:35] <sjust> yeah, ok
[21:35] <sjust> did you enable the option?
[21:37] <sjust> try adding 'osd leveldb paranoid = false' to the osd section
[21:37] <sjust> that should get the osds back up
[21:37] <mikedawson> sjust: will do
[21:38] <mikedawson> sjust: mikedawson-osd.64-omap.tar.gz is ready
[21:38] <cjh_> i'm thinking that each osd will need a newly generated key in order to function properly again
[21:39] <imjustmatthew> sagewk: Actually, of the 1151 threads in one of the mons that doesn't seem to be in any of the backtraces, they're mostly in tcp_read_wait
[21:41] <imjustmatthew> sagewk: The gdb backtraces are at: https://docs.google.com/file/d/0B-VPvwEe43FVbkU2U2RkUjZTRVE/edit?usp=sharing
[21:44] <sagewk> sjust: oops, i made paranoid true for the osd
[21:44] * john_barbee (~jbarbee@23-25-46-97-static.hfc.comcastbusiness.net) has joined #ceph
[21:44] <sjust> yeah, probably ought to default to false
[21:44] <mikedawson> sjust: they started with 'osd leveldb paranoid = false'. Is that something that went in recently? Should I run any type of maintenance on those omaps?
[21:44] <sjust> this is odd though, it certainly doesn't act corrupt...
[21:45] <sjust> mikedawson: not sure yet, the one you sent me is at least superficially fine
[21:45] <sjust> I'll have more in about 10 min
[21:46] <sagewk> imjustmatthew: its doing leveldb::DBImpl::CompactRange. is the disk busy?
[21:46] <gregaf> sagewk: I thought it was defaulting to false?
[21:46] <sagewk> the mon was does, but i typoed the osd one
[21:49] <mikedawson> sagewk: my mons are growing unbounded again with next, "mon compact on bootstrap = false", and "mon compact on trim = false". Went from ~300MB to >2GB in an hour
[21:49] <imjustmatthew> sagewk: The OSD disk is busy with reads, but the disk with /var/lib on it is not
[21:50] <gregaf> what are you doing that we're not covering </confusion>
[21:51] <sagewk> mikedawson: can you let it grow a bit bigger (may be 5gb) and then send us a tarball of it? then we (/the leveldb folk) can hopefully track down why it's doing that
[21:51] <sagewk> mikedawson: ...and we'll leave explicit compaction on for now.
[21:52] <mikedawson> sagewk: will do.
[21:52] <sagewk> imjustmatthew: sorry, i forget.. this is with --mon-level-paranoid, right?
[21:53] <imjustmatthew> sagewk: not unless that became a defualt :) I can add that though
[21:53] <mikedawson> gregaf: I'm on raring. libleveldb1:amd64 1.9.0-1 amd64 fast key-value storage library
[21:53] <imjustmatthew> Is it normal that the mon is spitting out a lot of "2013-05-02 15:53:13.329196 7fd96576c700 10 mon.a@0(synchronizing) e11 ms_verify_authorizer"
[21:54] <sagewk> that's everybody trying to reconnect because it is being unresponsive
[21:54] <sagewk> try trestarting with --mon-leveldb-paranoid and lets see what happens
[21:56] <sjust> mikedawson: just pushed a change to next with an option for the ceph-osdomap-tool to use paranoid checking
[21:56] <imjustmatthew> sagewk: It's working, thanks!
[21:57] <sjust> if you can install the next version of the ceph-test package once next finishes rebuilding, I'd like you to run the consistency checker on one of the apparently flaky omap directories
[21:57] <sjust> it actually worked fine here, so I am wondering whether the 12.04-13.04 difference is the problem
[21:58] <sagewk> imjustmatthew: are you on raring too?
[21:58] <imjustmatthew> sagewk: No, these are all precise with the 3.2 series kernel
[21:59] <mikedawson> sjust: so your push disables paranoid checks on the entire osd, but enables paranoid checks on the osdomap? Or did I misunderstand?
[21:59] <sjust> there were two, one was resetting the default to be false for osdsd
[21:59] <sagewk> oh, it was wido hitting this, and he is i think also on precise.
[22:00] <wido> sagewk: I'm back again
[22:00] <sjust> separately, there is a change to a consistency checker to optionally use paranoid checking
[22:00] <wido> it's indeed on Precise
[22:00] <wido> Running 3.5 kernel however
[22:00] <mikedawson> sagewk: I saw tons of mons stuck like imjustmatthew with things like mon.a@0(synchronizing) e11 ms_verify_authorizer
[22:00] <mikedawson> sagewk: on raring with 3.8 kernel
[22:00] <sagewk> ok. so i think #4851 (you) and imjustmatthew's hang #4896 are related (probably same bug).
[22:01] <sagewk> wido, imjustmatthew: you've seen the store.db growth problem too?
[22:01] <wido> sagewk: Well, not as large as imjustmatthew, but on mon3, which is the worst it's 2.2GB now
[22:02] <wido> against ~500MB on the other mons
[22:02] <sagewk> its running the latest next with compction going?
[22:03] <imjustmatthew> sagewk: I haven't noticed a store growth problem, only a memory growth problem, and that was running the older wip-3495 code
[22:04] <wido> sagewk: No, not the latest next. Your fix from yesterday for the paranoid
[22:04] <wido> brb
[22:06] <sagewk> and you're sure that version is on mon3?
[22:06] * eschnou (~eschnou@104.207-201-80.adsl-dyn.isp.belgacom.be) Quit (Quit: Leaving)
[22:09] <imjustmatthew> sagewk: Do you want me to switch back from mon-level-paranoid and try to get logs for #4896?
[22:09] * dpippenger (~riven@206-169-78-213.static.twtelecom.net) has joined #ceph
[22:10] <sagewk> hmm.. if you can reproduce, and then tar of the mon data dir at that point and attach to send us the tarball, that will preserve all the evidence we need
[22:10] * vata (~vata@2607:fad8:4:6:221:5aff:fe2a:d1dd) Quit (Quit: Leaving.)
[22:10] <sagewk> i dont think the cpeh logs are useful in for tracking down the leveldb bug
[22:10] <imjustmatthew> so a tar of the directory from the former leader just after it becomes non-responsive?
[22:12] <sagewk> yeah
[22:12] <imjustmatthew> k, I'll see if I can make it misbehave later
[22:15] <sagewk> imjustmatthew: we're going to prepare a branch that makes a trac eof the leveldb workload so we can replay it here and hopefully reproduce
[22:15] <sagewk> will you be around for the next hour or so?
[22:16] <imjustmatthew> sagewk: no, but I'll check the logbot in about 2 hours and can deploy that branch for testing
[22:16] <sagewk> thanks.
[22:16] <imjustmatthew> np, good luck!
[22:16] <sagewk> same goes for wido, mikedawson: if you can reliably reproduce the hang, that would help us out
[22:19] <mikedawson> sagewk: mine is hanging right now after upgrading to next a minute ago and not having compact on bootup enabled with ~2.5GB mon leveldb
[22:20] <sagewk> with the trim compact off too?
[22:20] <mikedawson> yes
[22:20] * jmlowe (~Adium@2001:18e8:2:28cf:f000::dab8) Quit (Quit: Leaving.)
[22:20] <sagewk> k. sjust is preparing that branch.. hopefully that'll give us what we need
[22:20] * nhorman (~nhorman@hmsreliant.think-freely.org) Quit (Quit: Leaving)
[22:25] <wido> sagewk: I can reproduce the hang as well, it simply doesn't work
[22:26] <wido> mon3 in this case just hangs and no command comes through since the election process doesn't seem to work either
[22:29] * jtang1 (~jtang@79.97.135.214) Quit (Quit: Leaving.)
[22:30] * jtang1 (~jtang@79.97.135.214) has joined #ceph
[22:30] <sagewk> k
[22:31] <wido> sagewk: I'll try the branch from sjust tomorrow
[22:36] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[22:36] * loicd (~loic@magenta.dachary.org) has joined #ceph
[22:39] * BManojlovic (~steki@fo-d-130.180.254.37.targo.rs) Quit (Ping timeout: 480 seconds)
[22:43] * Wolff_John (~jwolff@ftp.monarch-beverage.com) Quit (Quit: ChatZilla 0.9.90 [Firefox 20.0.1/20130409194949])
[22:47] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) Quit (reticulum.oftc.net coulomb.oftc.net)
[22:47] * fridad (~fridad@b.clients.kiwiirc.com) Quit (reticulum.oftc.net coulomb.oftc.net)
[22:47] * jefferai (~quassel@corkblock.jefferai.org) Quit (reticulum.oftc.net coulomb.oftc.net)
[22:47] * Meyer__ (meyer@c64.org) Quit (reticulum.oftc.net coulomb.oftc.net)
[22:47] * sig_wal1 (~adjkru@185.14.185.91) Quit (reticulum.oftc.net coulomb.oftc.net)
[22:47] * stass (stas@ssh.deglitch.com) Quit (reticulum.oftc.net coulomb.oftc.net)
[22:47] * trond_ (~trond@trh.betradar.com) Quit (reticulum.oftc.net coulomb.oftc.net)
[22:47] * mdxi (~mdxi@74-95-29-182-Atlanta.hfc.comcastbusiness.net) Quit (reticulum.oftc.net coulomb.oftc.net)
[22:47] * scalability-junk (uid6422@tooting.irccloud.com) Quit (reticulum.oftc.net coulomb.oftc.net)
[22:47] * SubOracle (~quassel@00019f1e.user.oftc.net) Quit (reticulum.oftc.net coulomb.oftc.net)
[22:47] * wido (~wido@2a00:f10:104:206:9afd:45af:ae52:80) Quit (reticulum.oftc.net coulomb.oftc.net)
[22:47] * dignus (~dignus@bastion.jkit.nl) Quit (reticulum.oftc.net coulomb.oftc.net)
[22:47] * _are_ (~quassel@2a01:238:4325:ca00:f065:c93c:f967:9285) Quit (reticulum.oftc.net coulomb.oftc.net)
[22:47] * barryo (~borourke@cumberdale.ph.ed.ac.uk) Quit (reticulum.oftc.net coulomb.oftc.net)
[22:47] * hijacker (~hijacker@bgva.sonic.taxback.ess.ie) Quit (reticulum.oftc.net coulomb.oftc.net)
[22:47] * Ludo_ (~Ludo@falbala.zoxx.net) Quit (reticulum.oftc.net coulomb.oftc.net)
[22:47] * jochen (~jochen@laevar.de) Quit (reticulum.oftc.net coulomb.oftc.net)
[22:47] * al (d@niel.cx) Quit (reticulum.oftc.net coulomb.oftc.net)
[22:47] <mikedawson> sagewk: do you expect "mon compact on bootstrap = true" to work in ceph version 0.60-795-g444660e (444660ed7714cdd3e00162825e0ad03bbe43f5c4?
[22:48] <sagewk> yeah
[22:48] <sagewk> well, it will compact.
[22:48] <sagewk> on boot. it won't prevent growth
[22:48] <sagewk> or anything else
[22:48] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) has joined #ceph
[22:48] * fridad (~fridad@b.clients.kiwiirc.com) has joined #ceph
[22:48] * jefferai (~quassel@corkblock.jefferai.org) has joined #ceph
[22:48] * Meyer__ (meyer@c64.org) has joined #ceph
[22:48] * sig_wal1 (~adjkru@185.14.185.91) has joined #ceph
[22:48] * stass (stas@ssh.deglitch.com) has joined #ceph
[22:48] * trond_ (~trond@trh.betradar.com) has joined #ceph
[22:48] * mdxi (~mdxi@74-95-29-182-Atlanta.hfc.comcastbusiness.net) has joined #ceph
[22:48] * scalability-junk (uid6422@tooting.irccloud.com) has joined #ceph
[22:48] * SubOracle (~quassel@00019f1e.user.oftc.net) has joined #ceph
[22:48] * wido (~wido@2a00:f10:104:206:9afd:45af:ae52:80) has joined #ceph
[22:48] * barryo (~borourke@cumberdale.ph.ed.ac.uk) has joined #ceph
[22:48] * dignus (~dignus@bastion.jkit.nl) has joined #ceph
[22:48] * _are_ (~quassel@2a01:238:4325:ca00:f065:c93c:f967:9285) has joined #ceph
[22:48] * hijacker (~hijacker@bgva.sonic.taxback.ess.ie) has joined #ceph
[22:48] * Ludo_ (~Ludo@falbala.zoxx.net) has joined #ceph
[22:48] * al (d@niel.cx) has joined #ceph
[22:48] * jochen (~jochen@laevar.de) has joined #ceph
[22:49] <mikedawson> sagewk: not compacting on startup for me right now, and all mons stuck in probing state
[22:50] <cjh_> no takers on how to restore a ceph cluster if the monitor directories get destroyed?
[22:50] <sagewk> sorry, it compacts on bootstrap, which happens when they probe.. so it is eithe rhugn compacting or compacting is slow
[22:51] <sagewk> cjh_: not easily
[22:51] <cjh_> sagewk: if i backed up the keys in the monitor directory would it be easier?
[22:52] * trond_ (~trond@trh.betradar.com) Quit (Ping timeout: 482 seconds)
[22:52] <sagewk> not by much
[22:52] <cjh_> wow
[22:53] <sagewk> you lost all 3?
[22:53] <cjh_> would i still be able to pull the data out of the rados cluster without the monitors?
[22:53] <cjh_> yeah i lost all 3 because i was curious how i would restore them if this happened
[22:53] <cjh_> i just basically rm'ed the files
[22:53] <sagewk> bobtail, or newer?
[22:53] <cjh_> bobtail
[22:54] <cjh_> this is just a test cluster so if i lose it, it's no big deal
[22:54] <sagewk> in theory you can piece together the osdmap files and reconstruct the paxos state
[22:54] <sagewk> tedious manual process.
[22:54] <cjh_> would this be like hours of work or days?
[22:54] <gregaf> you're really not supposed to lose all the copies, that's why you want 3 copies on different machines ;)
[22:54] <sagewk> hours
[22:55] <cjh_> gregaf: i know but i'm just wondering what if :)
[22:55] <cjh_> i have automated repair things that can do bad things
[22:57] * fridad_ (~oftc-webi@p4FC2C08F.dip0.t-ipconnect.de) has joined #ceph
[22:57] <cjh_> i also found that if you really pound on the cluster with rados bench that it's a good idea to break out the monitor to it's own machine with possibly ssd storage
[22:57] <mikedawson> sagewk: backtrace -> http://pastebin.com/raw.php?i=gY7DTZFg seems stuck 2.8G mon leveldb
[23:00] <sagewk> mikedawson: that looks like it's not all the threads?
[23:00] * gucki (~smuxi@HSI-KBW-078-042-028-165.hsi3.kabel-badenwuerttemberg.de) Quit (Remote host closed the connection)
[23:01] * alram (~alram@38.122.20.226) has joined #ceph
[23:02] <mikedawson> sagewk: sorry about that http://pastebin.com/raw.php?i=tGMamgjv
[23:03] <sagewk> and that's without --mon-leveldb-paranoid?
[23:03] <gregaf> cjh_: you shouldn't need SSDs for the monitor, at all
[23:03] * Vjarjadian (~IceChat77@90.214.208.5) has joined #ceph
[23:03] <mikedawson> i don't have paranoid in my conf or init
[23:05] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[23:05] * fridad_ (~oftc-webi@p4FC2C08F.dip0.t-ipconnect.de) Quit (Remote host closed the connection)
[23:06] * trond (~trond@trh.betradar.com) has joined #ceph
[23:07] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit (Quit: Leaving.)
[23:07] <pioto> gregaf: hi, thanks for adding yourself as "interested" in my blueprint. but, i noticed that the "owner" is supposed to be guiding the discussion, and that's apparently me :)
[23:09] <gregaf> that sounds familiar :)
[23:09] <pioto> i think i'll be able to get onto g+ during that time, but i can't say for sure yet. i shoudl be able to be on irc, though
[23:09] <pioto> (sorry, the cephfs security one, if i'm not being specific enough)
[23:10] <sagewk> k
[23:10] <gregaf> I'm finding out about the procedures for this about the same time everybody else is, maybe ask rturk-away when he gets back?
[23:13] <pioto> k, will do. thanks.
[23:15] <cjh_> gregaf: i'm not sure. when i ran 20 rados-bench instances across 20 machines i noticed that the monitor was keeping the disk at near 100% utilization
[23:17] * allsystemsarego (~allsystem@5-12-37-186.residential.rdsnet.ro) Quit (Quit: Leaving)
[23:17] * loicd1 (~loic@2a01:e35:2eba:db10:75bf:fd54:c3ed:54ab) has joined #ceph
[23:17] <gregaf> ah, interesting — that would be the pg stats reporting, but I think it should basically use up all the available disk and then throttle itself
[23:18] <gregaf> if not there's tuning available for those pieces
[23:18] <cjh_> gregaf: ok so i think i should turn down the stats reporting. The monitor appears to be my bottleneck at the moment
[23:19] <gregaf> what's giving you that impression? the disk activity on the monitor really isn't coupled to anything else happening in the cluster
[23:20] <cjh_> well when i run iostat -x 1 i'm not seeing the disks fully utilized. so i thought maybe my bottleneck was network or cpu but those are only at 50% usage.
[23:21] <cjh_> it also looks like it's not evenly using the disks but hammering a few of them. i think maybe i need to adjust the pg's in my pool
[23:21] <gregaf> all of those are more likely to be the problem than the monitor disk usage :)
[23:22] <mikedawson> sagewk: it seems like next has regressed since Monday or Tuesday. is there another branch in gitbuilder I can easily revert to?
[23:22] <mikedawson> or are historical builds retained?
[23:23] <cjh_> gregaf: gotcha. i have the monitors running on the same machines as the osd's. It says on the wiki that i might want to break it out onto a separate machine
[23:23] <gregaf> that *ought* to be more about keeping separate resource domains, but maybe
[23:23] <cjh_> ok
[23:24] <cjh_> the monitor has a separate disk and the osd's are on a jbod
[23:24] <cjh_> maybe i just need to fire up 40 instances of rados bench instead of 20 to see what happens
[23:24] <gregaf> where are their journals?
[23:24] <mikedawson> gregaf: I've recently done parallel rados bench testing with small writes, and I'm seeing quite a bunch more mon<-> traffic than traffic between osds
[23:24] * loicd (~loic@magenta.dachary.org) Quit (Ping timeout: 480 seconds)
[23:24] <mikedawson> seems possibly similar to cjh_
[23:25] <cjh_> that's what i'm seeing also
[23:25] <cjh_> it looks like the monitor is getting hammered and the osd's are sitting there
[23:25] <gregaf> mikedawson: yeah, that's going to be the pg stats reporting, which triggers a full pgmap update
[23:25] <cjh_> relatively speaking
[23:25] <gregaf> but it should all be out-of-band for the actual IO
[23:25] <cjh_> gregaf: where would i set that?
[23:26] <mikedawson> gregaf: do you know if it's possible to grab a "next" build from a few days ago? Current next makes my mons unhappy
[23:26] <gregaf> cjh_: you can try increasing the osd_mon_report_interval_min option on the OSDs (defaults to 5 [seconds])
[23:26] <cjh_> ok i'll see where that takes me :)
[23:26] <gregaf> mikedawson: it depends on how much has been cleaned up; a certain number of older builds are available
[23:27] <gregaf> via their sha1s; I think accessing that is described in teh docs?
[23:30] <sagewk> mikedawson: so far, 'mon leveldb paranoid' has successfully worked around this hang pretty well.. *fingers crossed*
[23:31] <sagewk> 'mon leveldb paranoid = true'
[23:31] <sagewk> would be nice to know if that isn't true before you downgrade.. :)
[23:32] <sagewk> sjust: looks good
[23:32] <sagewk> sjust: i assume it works? :)
[23:35] <sagewk> mikedawson: actually, can you generate a transaction log leading up to the hang? wip_mon_dump_ops is building right now
[23:36] <elder> sagewk, or maybe dmick, is it true that for a particular image the RBD_FEATURE_LAYERING bit will be set iff the image has a parent?
[23:36] <sagewk> elder: it isn't removed on flatten, if that's what you're getting at
[23:36] <elder> Oh.
[23:36] <mikedawson> sagewk: restarted with 'mon leveldb paranoid = true' and ceph version 0.60-795-g444660e (444660ed7714cdd3e00162825e0ad03bbe43f5c4). Still won't compact with 'mon compact on bootstrap = true' and 'mon compact on trim = true', all mons probing
[23:36] <elder> What will "get_parent" return in that case?
[23:37] <sagewk> the basic procedure would be: stop the mon, make a copy of the mon data dir, start it up again (with 'mon debug dump transactions = true' and 'mon leveldb paranoid = false'), wait for it to hang, then tar bzip2 the (starting and ending) data dir and the tdump log file (in /var/log/ceph)
[23:37] <sagewk> just be aware that it will be big...
[23:37] <elder> (And why not clear the LAYERING feature bit if the image is no longer a clone?)
[23:37] <sagewk> it probably could, but i don't think it does. the get_parent cls op will return ENOENT
[23:38] <mikedawson> sagewk: do you want that with this version of next or with wip_mon_dump_ops?
[23:38] <sagewk> wip_mon_dump_ops
[23:38] <elder> OK. ENOENT I think I can deal with. I need to change a little code arround though.
[23:39] <sagewk> imjustmatthew, wido: if you can reproduce and generate a log as well, that'd help!
[23:39] <dmick> elder: the bits aren't "current used features", they're "capabilities", AFAIK
[23:40] <dmick> but every time we have a features discussion I get less clear
[23:40] <elder> The kernel client doesn't treat it that way, and it works...
[23:40] <elder> If "get_parent" returns an error, that's it, it's done.
[23:41] <elder> So now I need to make it distinguish between ENOENT and other errors.
[23:42] <sagewk> elder: i think its only because we only set the bit when we create a clone, and we haven't tested with flatten yet.
[23:42] <elder> Right.
[23:42] <elder> But that is in contrast somewhat to what dmick said.
[23:42] <elder> I.e., at least initially, it also serves as a "current used feature"
[23:42] <elder> And that's all I've dealt with so far.
[23:43] <elder> So with the possibility of layering I have to look for ENOENT as well.
[23:43] * KindOne (KindOne@0001a7db.user.oftc.net) Quit (Ping timeout: 480 seconds)
[23:43] <elder> (I do think a "current used feature" is more useful in this case though. It will never go from "in use" to "not" and then back to "in use again")
[23:43] <elder> (And yet the whole time it's a *possible* feature)
[23:43] * KindOne (KindOne@0001a7db.user.oftc.net) has joined #ceph
[23:44] <sagewk> mikedawson: build is ready, btw.
[23:44] <mikedawson> sagewk: on it
[23:44] <sagewk> elder: yeah, we should probably make it behave that way going forward.. but i don't think it currently does that now.
[23:45] <elder> Either way, I can deal with it.
[23:48] * tziOm (~bjornar@ti0099a340-dhcp0870.bb.online.no) Quit (Remote host closed the connection)
[23:48] * glowell1 (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) has joined #ceph
[23:48] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) Quit (Read error: Connection reset by peer)
[23:49] * nhm (~nhm@ma02536d0.tmodns.net) has joined #ceph
[23:49] <sjust> sagewk: dumper seems to work
[23:50] * jtang1 (~jtang@79.97.135.214) Quit (Quit: Leaving.)
[23:53] * fridad (~fridad@b.clients.kiwiirc.com) Quit (Remote host closed the connection)
[23:53] <dmick> elder: yeah, ignore me, that stuff was swapped out and I was reading garbage. Of course the field is set on the image. I think it's set at creation time, though, so that it can be checked, say, on "clone me" time. that is, the parent snap needs to support layering for it to work, whether or not ther are clones
[23:53] * jtang1 (~jtang@79.97.135.214) has joined #ceph
[23:54] * jtang2 (~jtang@79.97.135.214) has joined #ceph
[23:54] * jtang1 (~jtang@79.97.135.214) Quit (Read error: Connection reset by peer)
[23:56] * diegows (~diegows@200.68.116.185) Quit (Ping timeout: 480 seconds)
[23:57] * jmlowe (~Adium@108.192.77.199) has joined #ceph

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.