#ceph IRC Log

Index

IRC Log for 2013-08-06

Timestamps are in GMT/BST.

[0:01] * sprachgenerator (~sprachgen@130.202.135.214) Quit (Quit: sprachgenerator)
[0:02] <sagewk> what's the irc channel?
[0:03] * Vincent_Valentine (Vincent_Va@49.206.158.155) Quit (Read error: Connection reset by peer)
[0:03] <mikedawson> sage: ceph-summit
[0:03] * Vincent_Valentine (Vincent_Va@49.206.158.155) has joined #ceph
[0:03] * sagewk does /join #ceph-summit
[0:09] * mschiff (~mschiff@port-2854.pppoe.wtnet.de) Quit (Remote host closed the connection)
[0:15] * jeff-YF (~jeffyf@67.23.117.122) Quit (Ping timeout: 480 seconds)
[0:17] * dalgaaf (~dalgaaf@nrbg-4dbfcce3.pool.mediaWays.net) has joined #ceph
[0:20] * diegows (~diegows@200.68.116.185) Quit (Ping timeout: 480 seconds)
[0:22] <cjh_> tuning in :)
[0:23] * _Tassadar (~tassadar@tassadar.xs4all.nl) has joined #ceph
[0:23] <rturk> cjh_: welcome :)
[0:23] <rturk> if you are attending the Ceph Developer Summit, head to #ceph-summit
[0:26] * tnt (~tnt@109.130.80.16) Quit (Ping timeout: 480 seconds)
[0:28] * _Tass4dar (~tassadar@tassadar.xs4all.nl) has joined #ceph
[0:29] <sagewk> anybody wnat to join teh zfs discussion?
[0:29] * szaydel (~szaydel@static-108-45-37-100.washdc.fios.verizon.net) has joined #ceph
[0:31] * dalgaaf (~dalgaaf@nrbg-4dbfcce3.pool.mediaWays.net) Quit (Quit: Konversation terminated!)
[0:32] * BillK (~BillK-OFT@124-148-246-233.dyn.iinet.net.au) has joined #ceph
[0:33] * _Tassadar (~tassadar@tassadar.xs4all.nl) Quit (Ping timeout: 480 seconds)
[0:34] * dalgaaf (~dalgaaf@nrbg-4dbfcce3.pool.mediaWays.net) has joined #ceph
[0:35] * _Tass4dar (~tassadar@tassadar.xs4all.nl) Quit (Remote host closed the connection)
[0:35] * _Tassadar (~tassadar@tassadar.xs4all.nl) has joined #ceph
[0:37] * yehudasa__ (~yehudasa@2607:f298:a:607:ea03:9aff:fe98:e8ff) Quit (Ping timeout: 480 seconds)
[0:39] <mozg> ah
[0:39] <mozg> zfs
[0:41] * gregmark (~Adium@cet-nat-254.ndceast.pa.bo.comcast.net) Quit (Quit: Leaving.)
[0:42] * haomaiwa_ (~haomaiwan@117.79.232.144) Quit (Remote host closed the connection)
[0:43] * haomaiwang (~haomaiwan@li565-182.members.linode.com) has joined #ceph
[0:49] * haomaiwa_ (~haomaiwan@117.79.232.202) has joined #ceph
[0:50] <cmdrk> hey all, hope the summit is going well
[0:50] <cmdrk> "journal aio = true" -- is that the correct syntax for [osd] in ceph.conf ?
[0:51] <cmdrk> or should it be like "osd journal aio = true" ?
[0:51] <Cube> journal aio is correct
[0:51] <cmdrk> alright
[0:53] * haomaiwang (~haomaiwan@li565-182.members.linode.com) Quit (Read error: Operation timed out)
[0:53] <cmdrk> i'm trying to add 'journal aio = true' to OSDs that have already been created, will that work? i'm seeing "directio = 1, aio = 0" in the logs after I update ceph.conf and restart
[0:58] * dalgaaf (~dalgaaf@nrbg-4dbfcce3.pool.mediaWays.net) Quit (Quit: Konversation terminated!)
[0:58] * dalgaaf (~dalgaaf@nrbg-4dbfcce3.pool.mediaWays.net) has joined #ceph
[0:58] * zhu (~quassel@106.120.176.124) has joined #ceph
[1:01] <ccourtaut> used to have my ISP having very bad peering with youtube, might be the case here
[1:02] * davidzlap (~Adium@ip68-5-239-214.oc.oc.cox.net) Quit (Read error: Connection reset by peer)
[1:02] * davidzlap (~Adium@ip68-5-239-214.oc.oc.cox.net) has joined #ceph
[1:08] * ninkotech (~duplo@static-84-242-87-186.net.upcbroadband.cz) Quit (Read error: Connection reset by peer)
[1:08] * ninkotech (~duplo@static-84-242-87-186.net.upcbroadband.cz) has joined #ceph
[1:16] * zhu (~quassel@106.120.176.124) Quit (Remote host closed the connection)
[1:16] * zhu (~quassel@117.79.232.156) has joined #ceph
[1:17] * gentleben (~sseveranc@216.55.31.102) Quit (Quit: gentleben)
[1:18] * devoid (~devoid@130.202.135.215) Quit (Quit: Leaving.)
[1:26] * szaydel (~szaydel@static-108-45-37-100.washdc.fios.verizon.net) Quit (Quit: My MacBook Pro has gone to sleep. ZZZzzz…)
[1:27] <scuttlemonkey> ** If anyone here has thoughts on ceph-deploy and would like to participate in the development discussion please let me know.
[1:27] <scuttlemonkey> ** There is a google hangout starting in ~2m about ceph-deploy development
[1:41] * gentleben (~sseveranc@c-98-207-40-73.hsd1.ca.comcast.net) has joined #ceph
[1:42] * duff_ (~duff@199.181.135.135) Quit (Quit: leaving)
[1:44] * xmltok_ (~xmltok@pool101.bizrate.com) Quit (Remote host closed the connection)
[1:44] * xmltok (~xmltok@relay.els4.ticketmaster.com) has joined #ceph
[1:48] * diegows (~diegows@190.190.11.42) has joined #ceph
[2:00] * mmercer (~kvirc@c-67-180-16-120.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[2:06] * yanzheng (~zhyan@jfdmzpr01-ext.jf.intel.com) Quit (Remote host closed the connection)
[2:07] * gentleben (~sseveranc@c-98-207-40-73.hsd1.ca.comcast.net) Quit (Quit: gentleben)
[2:08] * mmercer (~kvirc@c-67-180-16-120.hsd1.ca.comcast.net) has joined #ceph
[2:09] * gentleben (~sseveranc@c-98-207-40-73.hsd1.ca.comcast.net) has joined #ceph
[2:09] * lxo (~aoliva@lxo.user.oftc.net) Quit (Quit: later)
[2:10] <athrift> Do the ceph OSD's write data sequentially very often? or is it primarily random writes ?
[2:10] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[2:11] <dmick> athrift: the journals are there specifically to increase sequential writing
[2:11] <dmick> (not only, but in large part)
[2:11] * leseb (~leseb@88-190-214-97.rev.dedibox.fr) Quit (Killed (NickServ (Too many failed password attempts.)))
[2:11] <dmick> how well does it work? good question.
[2:11] * leseb (~leseb@88-190-214-97.rev.dedibox.fr) has joined #ceph
[2:11] * xmltok_ (~xmltok@pool101.bizrate.com) has joined #ceph
[2:12] <athrift> dmick: Thanks Dan, just trying to workout if using bcache would work better for our configuration. We have a SSD to Spinner ratio of 1:6 so SSD could become out bottleneck
[2:14] * xmltok_ (~xmltok@pool101.bizrate.com) Quit ()
[2:18] <rturk> ** Live video feed for sessions 6-10 of the Ceph Developer Summit: http://youtu.be/90sc-vvUCfw
[2:19] * xmltok (~xmltok@relay.els4.ticketmaster.com) Quit (Ping timeout: 480 seconds)
[2:20] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[2:22] * rturk is now known as rturk-away
[2:24] * gggg (~Vincent@49.206.158.155) has joined #ceph
[2:25] * LeaChim (~LeaChim@2.122.178.96) Quit (Ping timeout: 480 seconds)
[2:25] * huangjun (~kvirc@111.173.155.201) has joined #ceph
[2:28] * andreask (~andreask@h081217135028.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[2:30] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) has joined #ceph
[2:32] * lx0 (~aoliva@lxo.user.oftc.net) has joined #ceph
[2:35] * lightspeed (~lightspee@fw-carp-wan.ext.lspeed.org) Quit (Ping timeout: 480 seconds)
[2:37] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[2:37] * john (~john@astound-64-85-225-33.ca.astound.net) Quit (Quit: Leaving)
[2:38] * grepory (~Adium@50-115-70-146.static-ip.telepacific.net) Quit (Quit: Leaving.)
[2:38] * rturk-away is now known as rturk
[2:42] * huangjun (~kvirc@111.173.155.201) Quit (Quit: KVIrc 4.2.0 Equilibrium http://www.kvirc.net/)
[2:43] * huangjun (~kvirc@111.173.155.201) has joined #ceph
[2:44] * yanzheng (~zhyan@jfdmzpr06-ext.jf.intel.com) has joined #ceph
[2:48] * lightspeed (~lightspee@81.187.0.153) has joined #ceph
[2:50] <sagewk> annoying/confusing ceph issues ("papercuts") that you want to share? http://pad.ceph.com/p/papercuts
[3:06] * smiley (~smiley@pool-173-73-0-53.washdc.fios.verizon.net) has joined #ceph
[3:06] * mmercer (~kvirc@c-67-180-16-120.hsd1.ca.comcast.net) Quit (Read error: Operation timed out)
[3:09] * gggg (~Vincent@49.206.158.155) Quit (Ping timeout: 480 seconds)
[3:09] * yy-nm (~chatzilla@115.196.74.105) has joined #ceph
[3:16] * Cube1 (~Cube@12.248.40.138) has joined #ceph
[3:16] * Cube1 (~Cube@12.248.40.138) Quit ()
[3:24] * Cube (~Cube@12.248.40.138) Quit (Ping timeout: 480 seconds)
[3:25] * gentleben (~sseveranc@c-98-207-40-73.hsd1.ca.comcast.net) Quit (Quit: gentleben)
[3:30] <paravoid> sagewk: too late?
[3:30] <sagewk> still here, doing CDS
[3:30] <paravoid> I guess so
[3:30] <sagewk> /join #ceph-summit :)
[3:30] <paravoid> no, for papercuts :)
[3:30] <sagewk> ah!
[3:30] <sagewk> can still use the pad
[3:30] <sagewk> or share in teh channel
[3:31] <sagewk> 'http://pad.ceph.com/p/papercuts
[3:32] * rturk is now known as rturk-away
[3:32] * rturk-away is now known as rturk
[3:32] * alram (~alram@38.122.20.226) Quit (Quit: leaving)
[3:34] <paravoid> oh it looks like I'm going to join you for ceph days in London
[3:35] <rturk> paravoid: cool!
[3:35] * bandrus (~Adium@12.248.40.138) Quit (Quit: Leaving.)
[3:35] <paravoid> is there a deadline for a talk proposal?
[3:36] <rturk> Nope, we haven't started assembling the schedule for London yet
[3:36] <paravoid> I'd like to talk about our setup at Wikimedia but I'd like to see how this month is going to go
[3:36] <rturk> drop an email to community@ceph.com - we won't finalize it for a bit
[3:36] <paravoid> I'll ping you later this month
[3:36] <rturk> we can save a spot for you in the meantime
[3:36] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Quit: Leaving.)
[3:40] <rturk> ** The session "rgw: multitenancy" starts in 5 minutes!
[3:40] <rturk> ** Please tell rturk or scuttlemonkey if you want to join the hangout.
[3:40] * diegows (~diegows@190.190.11.42) Quit (Ping timeout: 480 seconds)
[3:53] * mikedawson_ (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) has joined #ceph
[3:55] * rongze_ (~quassel@notes4.com) has joined #ceph
[3:56] <scuttlemonkey> paravoid: they are wishing for your presence in a RGW discussion :)
[3:56] <paravoid> they are?
[3:56] <paravoid> opened the feed
[3:56] <scuttlemonkey> yeah, talking RGW multitenancy and swift api semantics
[3:56] <rturk> want to join the hangout?
[3:56] <paravoid> I could
[3:57] <rturk> let me verify if they're still on that subject
[3:57] <scuttlemonkey> rturk: sent the info
[3:57] <rturk> or that
[3:57] <rturk> :)
[3:57] * dalgaaf (~dalgaaf@nrbg-4dbfcce3.pool.mediaWays.net) Quit (Read error: Operation timed out)
[3:58] <sagewk> if you have any insight about what multitenancy people want/use with swift..
[4:00] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[4:01] * rongze (~quassel@117.79.232.202) Quit (Ping timeout: 480 seconds)
[4:02] * buck (~buck@c-24-6-91-4.hsd1.ca.comcast.net) has left #ceph
[4:03] <scuttlemonkey> Li Wang around by chance?
[4:04] <paravoid> heh, sorry, not hugely experienced with multi-tenancy
[4:04] <paravoid> I'm not the typical swift user I guess :)
[4:04] <sagewk> :)
[4:04] * dalgaaf (~dalgaaf@nrbg-4dbe25d4.pool.mediaWays.net) has joined #ceph
[4:05] * Vincent_Valentine (Vincent_Va@49.206.158.155) Quit (Ping timeout: 480 seconds)
[4:06] * julian (~julianwa@125.69.104.58) has joined #ceph
[4:09] * gentleben (~sseveranc@c-98-207-40-73.hsd1.ca.comcast.net) has joined #ceph
[4:11] * nwat (~nwat@c-50-131-197-174.hsd1.ca.comcast.net) Quit (Quit: leaving)
[4:12] * nwat (~nwat@c-50-131-197-174.hsd1.ca.comcast.net) has joined #ceph
[4:16] * dpippenger (~riven@cpe-76-166-208-83.socal.res.rr.com) Quit (Remote host closed the connection)
[4:32] * gentleben (~sseveranc@c-98-207-40-73.hsd1.ca.comcast.net) Quit (Quit: gentleben)
[4:34] * xmltok (~xmltok@cpe-76-170-26-114.socal.res.rr.com) has joined #ceph
[4:36] * andrewbogott (~andrewbog@113.28.133.117) has joined #ceph
[4:37] * andrewbogott (~andrewbog@113.28.133.117) Quit ()
[4:42] * silversurfer (~jeandanie@124x35x46x12.ap124.ftth.ucom.ne.jp) has joined #ceph
[4:45] <silversurfer> Hi all, I am looking for a way to reduce the verbosity in /var/log/ceph/ceph.log, I would like to reduce all the information lines like 2013-08-06 11:42:21.438323 mon.0 172.23.45.21:6789/0 125130 : [INF] pgmap v367534: 1216 pgs: [...]1252 GB / 1452 GB avail; 3094B/s wr, 0op/s
[4:46] <silversurfer> I am confused with all the different logs settings and having trouble pinpoint the setting that can allow me to reduce the amount of these logs ( 1 line per second in my cluster). Any hint?
[4:46] * grepory (~Adium@c-69-181-42-170.hsd1.ca.comcast.net) has joined #ceph
[4:49] <scuttlemonkey> hey silversurfer: just wanted to make sure you didn't hit radio silence
[4:49] <scuttlemonkey> we're in the middle of our online developer summit...so many brains focused there
[4:50] <dmick> silversurfer: that log is the 'cluster log'
[4:50] * grepory (~Adium@c-69-181-42-170.hsd1.ca.comcast.net) Quit ()
[4:51] * grepory (~Adium@c-69-181-42-170.hsd1.ca.comcast.net) has joined #ceph
[4:53] * grepory (~Adium@c-69-181-42-170.hsd1.ca.comcast.net) Quit ()
[4:53] <dmick> I'm honestly not sure how you affect that verbosity. Generally that logging is pretty small in comparison to everything else
[4:57] * alfredodeza (~alfredode@38.122.20.226) Quit (Remote host closed the connection)
[4:57] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[5:05] * fireD_ (~fireD@93-142-223-18.adsl.net.t-com.hr) has joined #ceph
[5:07] * fireD (~fireD@93-142-207-243.adsl.net.t-com.hr) Quit (Ping timeout: 480 seconds)
[5:11] * xmltok (~xmltok@cpe-76-170-26-114.socal.res.rr.com) Quit (Remote host closed the connection)
[5:11] * xmltok (~xmltok@relay.els4.ticketmaster.com) has joined #ceph
[5:12] <silversurfer> scuttlemonkey: hi alright
[5:13] <silversurfer> dmick: ok will try "debug cluster log 0" in ceph.conf. Thanks for the info
[5:20] <silversurfer> hmm logs are still raining. I found in the documentation that "mon cluster log file" set the location of the log in /var/log/ceph/$cluster.log
[5:21] * gentleben (~sseveranc@c-98-207-40-73.hsd1.ca.comcast.net) has joined #ceph
[5:22] <silversurfer> my cluster name is ceph hence /var/log/ceph/ceph.log but I can't find the option to silence reduce its verbosity
[5:24] <mikedawson_> silversurfer: you can connect to the admin socket (http://ceph.com/docs/master/rados/operations/monitoring/#using-the-admin-socket). Something like 'ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show' will show you all the settings currently in use
[5:24] <joshd> silversurfer: if you send it to syslog instead of the file, you can configure the syslog level with mon_cluster_log_to_syslog_level = "warn" etc
[5:26] <silversurfer> nice! thank you for the info I will dig in!
[5:55] * grepory (~Adium@c-69-181-42-170.hsd1.ca.comcast.net) has joined #ceph
[6:02] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) has left #ceph
[6:05] * rturk is now known as rturk-away
[6:11] * sagelap (~sage@38.122.20.226) Quit (Ping timeout: 480 seconds)
[6:12] * mikedawson_ (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) Quit (Quit: ChatZilla 0.9.90.1 [Firefox 22.0/20130618035212])
[6:13] * scuttlemonkey (~scuttlemo@2607:f298:a:607:94a3:6831:7ee1:8fdd) Quit (Ping timeout: 480 seconds)
[6:16] * joao (~JL@2607:f298:a:607:9eeb:e8ff:fe0f:c9a6) Quit (Ping timeout: 480 seconds)
[6:21] <yanzheng> sigh, it looks like zfsonlinux doesn't flush data when snapshot. I have to call sync_filesytem() before calling zfs_snapshot. otherwise ceph's pg log get corrupted
[6:23] <phantomcircuit> yanzheng, what kind of corruption?
[6:23] <phantomcircuit> the only possible form of corruption should be a shorn write
[6:23] <phantomcircuit> ie the last journal entry is only partial
[6:24] <yanzheng> log bound mismatch, info ....
[6:24] * sagelap (~sage@2600:1012:b025:931a:d5ef:c133:59c8:6e59) has joined #ceph
[6:24] <yanzheng> then a few unfound objects
[6:28] * grepory (~Adium@c-69-181-42-170.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[6:31] * dalgaaf (~dalgaaf@nrbg-4dbe25d4.pool.mediaWays.net) Quit (Read error: Operation timed out)
[6:32] * xmltok (~xmltok@relay.els4.ticketmaster.com) Quit (Quit: Leaving...)
[6:34] * Vincent_Valentine (~Vincent_V@49.206.158.155) has joined #ceph
[6:36] * Vincent_Valentine (~Vincent_V@49.206.158.155) Quit (Read error: Connection reset by peer)
[6:36] * Vincent_Valentine (Vincent_Va@49.206.158.155) has joined #ceph
[6:41] * sagelap (~sage@2600:1012:b025:931a:d5ef:c133:59c8:6e59) Quit (Quit: Leaving.)
[6:45] <silversurfer> I tried these setting http://pastebin.com/L47X3vD1 but the INF logs are still pouring
[6:52] * xmltok (~xmltok@cpe-76-170-26-114.socal.res.rr.com) has joined #ceph
[6:54] * xmltok (~xmltok@cpe-76-170-26-114.socal.res.rr.com) Quit (Remote host closed the connection)
[6:55] * xmltok (~xmltok@pool101.bizrate.com) has joined #ceph
[6:55] * Vincent_Valentine (Vincent_Va@49.206.158.155) Quit (Ping timeout: 480 seconds)
[7:01] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) Quit (Quit: Leaving.)
[7:05] * julian (~julianwa@125.69.104.58) Quit (Quit: afk)
[7:17] <lurbs> ceph-deploy 1.1-1 built from master seems to require python-pushy 0.5.2, but that's not listed in the package dependencies.
[7:17] <lurbs> Or rather the dependency on python-pushy is, just not >= 0.5.2.
[7:18] * Macheske (~Bram@d5152D87C.static.telenet.be) Quit ()
[7:23] * xmltok (~xmltok@pool101.bizrate.com) Quit (Quit: Bye!)
[7:25] * Kioob (~kioob@2a01:e35:2432:58a0:21e:8cff:fe07:45b6) has joined #ceph
[7:41] <dmick> lurbs: there is 4ff5b728b409b000f521aa38170d5cca547169fb, but I see that's not in requirements.txt; maybe it should be. I don't understand how they interact.
[7:48] * tnt (~tnt@109.130.80.16) has joined #ceph
[7:52] <lurbs> Looks like even though the required version was bumped no python-pushy of that version is actually being built - or not put anywhere I can find it, anyway.
[8:08] * davidzlap (~Adium@ip68-5-239-214.oc.oc.cox.net) Quit (Quit: Leaving.)
[8:21] * yanzheng (~zhyan@jfdmzpr06-ext.jf.intel.com) Quit (Remote host closed the connection)
[8:24] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[8:25] * andreask (~andreask@h081217135028.dyn.cm.kabsi.at) has joined #ceph
[8:25] * ChanServ sets mode +v andreask
[8:31] <yy-nm> hi, all. i have aquestion about ceph-deploy. the ceph-deploy tool will gener the osd instance information append to the ceph.conf after add a osd by ceph-deploy?
[8:33] <joelio> yy-nm: no, the current version of ceph-deploy doesn't add to ceph.conf - it uses known file paths to enumerate the osds in the system
[8:33] <joelio> btw I'd update from master on git for ceph-deploy
[8:33] * joelio watched dev conf last night and saw lots of fixes
[8:34] * odyssey4me (~odyssey4m@165.233.71.2) has joined #ceph
[8:34] <joelio> ceph-deploy can be used to create the osds though, it just doesn't add to the ceph.conf file as you may expect if coming from mkcephfs days
[8:34] <yy-nm> you mean i need to add the osd information to ceph.conf file manually?
[8:35] <joelio> no, no need
[8:35] <yy-nm> how can i stop the running osd daemon?
[8:35] <joelio> if you use ceph-deploy it'll handle the osds itself - it doesn't end up in ceph.conf, that just describes initial mon members and has some other tunables in there
[8:36] <joelio> with init scripts
[8:36] <joelio> on ubuntu, upstart.. ie. stop ceph-all; start ceph-all etc..
[8:36] <joelio> stop ceph-osd (iirc)
[8:37] <joelio> I would test the ceph-deploy from master if you can though,
[8:38] <yy-nm> ok, you means the upstart tool ?
[8:38] <joelio> yes
[8:38] <joelio> what distro are you using?
[8:39] <yy-nm> linux? or ceph?
[8:39] <joelio> linux
[8:39] <yy-nm> ubuntu 12.04
[8:39] <joelio> yea, that uses upstart
[8:40] <joelio> yy-nm: http://ceph.com/docs/master/rados/operations/operating/
[8:42] <yy-nm> thx, i get it
[8:50] * allsystemsarego (~allsystem@188.25.130.190) has joined #ceph
[8:51] * glzhao (~glzhao@203.192.156.9) has joined #ceph
[8:51] * glzhao (~glzhao@203.192.156.9) Quit ()
[8:51] * hybrid512 (~walid@LPoitiers-156-86-25-85.w193-248.abo.wanadoo.fr) has joined #ceph
[8:51] * glzhao (~glzhao@203.192.156.9) has joined #ceph
[8:52] * agh (~oftc-webi@gw-to-666.outscale.net) has joined #ceph
[8:52] * glzhao (~glzhao@203.192.156.9) Quit ()
[8:52] * saabylaptop (~saabylapt@2a02:2350:18:1010:150:278a:15:d71f) has joined #ceph
[8:53] <agh> Hello,
[8:53] <agh> i've a question about crushmaps
[8:54] <agh> I've a cluster, composed of 6 hosts, installed in 2 racks
[8:54] <agh> and i want a replica level of 3 (size=3)
[8:54] <agh> I wan the first object in a rack,
[8:54] <agh> the first copy in the other rack
[8:55] <agh> and the second copy on any other host
[8:55] <agh> so, i did the following in my crushmap :
[8:55] <agh> step chooseleaf firstn 2 type rack
[8:55] <agh> step chooseleaf firstn 1 type host
[8:55] <agh> step emit
[8:56] <agh> but... when i apply this. It does not work
[8:56] <agh> (the mon is going down and the map is not applied)
[8:56] <agh> any idea ?
[8:56] * saabylaptop (~saabylapt@2a02:2350:18:1010:150:278a:15:d71f) Quit ()
[8:56] * saabylaptop (~saabylapt@2a02:2350:18:1010:150:278a:15:d71f) has joined #ceph
[9:02] <saabylaptop> hi agh
[9:03] <agh> saabylaptop: hi
[9:03] <saabylaptop> you want the first copy in 'a' rack or in a specific rack?
[9:03] <agh> saabylaptop: in a rack.
[9:03] <saabylaptop> ok
[9:03] <agh> i wan to be able to shutdown a whole rack
[9:03] <saabylaptop> right
[9:04] <saabylaptop> step take firstn 2 type rack
[9:04] <saabylaptop> step chooseleaf firstn 1 type host
[9:04] <saabylaptop> step emit
[9:04] <saabylaptop> that should do it
[9:05] <yy-nm> you maybe write in this way: step take rack1 step chooseleaf firstn
[9:05] <saabylaptop> chooseleaf will choose a leafnode under the node specified
[9:06] <agh> saabylaptop: ok, i'm gonna try it
[9:06] <saabylaptop> good luck.
[9:06] <yy-nm> the chooseleaf can't under the step chooseleaf?
[9:06] <yy-nm> i guess
[9:07] * Vjarjadian (~IceChat77@90.214.208.5) Quit (Quit: On the other hand, you have different fingers.)
[9:07] <saabylaptop> agreed.
[9:12] * tnt (~tnt@109.130.80.16) Quit (Ping timeout: 480 seconds)
[9:22] * agh (~oftc-webi@gw-to-666.outscale.net) Quit (Quit: Page closed)
[9:29] * tnt (~tnt@212-166-48-236.win.be) has joined #ceph
[9:37] * ScOut3R (~ScOut3R@catv-89-133-25-52.catv.broadband.hu) has joined #ceph
[9:44] * dalgaaf (~dalgaaf@nrbg-4dbe25d4.pool.mediaWays.net) has joined #ceph
[9:46] * Vincent_Valentine (~Vincent_V@115.119.113.218) has joined #ceph
[9:49] * lx0 (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[9:52] * sage (~sage@76.89.177.113) Quit (Ping timeout: 480 seconds)
[10:03] * mschiff (~mschiff@p4FD7E178.dip0.t-ipconnect.de) has joined #ceph
[10:08] * dosaboy_alt (~dosaboy_a@faun.canonical.com) has joined #ceph
[10:08] * dalgaaf (~dalgaaf@nrbg-4dbe25d4.pool.mediaWays.net) Quit (Ping timeout: 480 seconds)
[10:08] * dalgaaf-tmp (~dalgaaf@85.214.11.161) has joined #ceph
[10:17] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) Quit (Quit: Leaving.)
[10:17] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[10:20] * dobber (~dobber@213.169.45.222) has joined #ceph
[10:22] * mschiff_ (~mschiff@p4FD7E178.dip0.t-ipconnect.de) has joined #ceph
[10:22] * mschiff (~mschiff@p4FD7E178.dip0.t-ipconnect.de) Quit (Read error: Connection reset by peer)
[10:24] * Meths_ (rift@2.25.214.150) has joined #ceph
[10:27] * julian (~julianwa@125.69.104.58) has joined #ceph
[10:28] * Meths (rift@2.25.189.113) Quit (Ping timeout: 480 seconds)
[10:30] * sage (~sage@76.89.177.113) has joined #ceph
[10:38] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) Quit (Quit: Leaving.)
[10:38] * jaydee (~jeandanie@124x35x46x8.ap124.ftth.ucom.ne.jp) has joined #ceph
[10:43] * silversurfer (~jeandanie@124x35x46x12.ap124.ftth.ucom.ne.jp) Quit (Ping timeout: 480 seconds)
[10:49] * KindTwo (KindOne@h186.48.186.173.dynamic.ip.windstream.net) has joined #ceph
[10:49] * KindOne (KindOne@0001a7db.user.oftc.net) Quit (Ping timeout: 480 seconds)
[10:50] * KindTwo is now known as KindOne
[10:58] * yy-nm (~chatzilla@115.196.74.105) Quit (Quit: ChatZilla 0.9.90.1 [Firefox 22.0/20130618035212])
[11:03] * bergerx_ (~bekir@78.188.101.175) has joined #ceph
[11:18] <loicd> ccourtaut: \o
[11:18] <ccourtaut> loicd, o/
[11:19] * Vincent_Valentine (~Vincent_V@115.119.113.218) Quit (Ping timeout: 480 seconds)
[11:24] * nwat (~nwat@c-50-131-197-174.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[11:27] * julian (~julianwa@125.69.104.58) Quit (Read error: Connection reset by peer)
[11:28] <loicd> ccourtaut: do you know where 'ceph osd pool set <pool> ' is interpreted ? I'd like to understand what it is used for and where it is stored in g_conf but I'm not familiar with this part.
[11:31] <ccourtaut> loicd, no i don't, i can take a look to
[11:33] * silversurfer (~jeandanie@124x35x46x12.ap124.ftth.ucom.ne.jp) has joined #ceph
[11:37] * jaydee (~jeandanie@124x35x46x8.ap124.ftth.ucom.ne.jp) Quit (Ping timeout: 480 seconds)
[11:40] * sage (~sage@76.89.177.113) Quit (Ping timeout: 480 seconds)
[11:41] * andreask (~andreask@h081217135028.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[11:41] <ccourtaut> loicd, looks like librados is called
[11:41] * mozg (~andrei@host109-151-35-94.range109-151.btcentralplus.com) Quit (Ping timeout: 480 seconds)
[11:42] <ccourtaut> loicd, https://github.com/ceph/ceph/blob/master/src/librados/librados.cc#L1936
[11:42] * loicd looking
[11:43] <ccourtaut> and the commands seems to be defined here https://github.com/ceph/ceph/blob/master/src/mon/MonCommands.h
[11:44] <ccourtaut> so that, for example, this command can be trigger by a client, such as here https://github.com/ceph/ceph/blob/master/src/pybind/rados.py#L559
[11:46] * nwat (~nwat@c-50-131-197-174.hsd1.ca.comcast.net) has joined #ceph
[11:51] * sage (~sage@76.89.177.113) has joined #ceph
[11:51] <loicd> ccourtaut: thanks :-)
[11:52] <ccourtaut> loicd, https://github.com/ceph/ceph/blob/master/src/osdc/Objecter.h#L1258
[11:52] <ccourtaut> this seems to be the "real" entry point for what you are looking for :)
[11:52] * mxmln (~maximilia@212.79.49.65) has joined #ceph
[11:54] <loicd> I was kind of hoping that pool set would allow an arbitrary key/value pair
[11:54] <loicd> https://github.com/ceph/ceph/blob/master/src/mon/MonCommands.h#L496
[11:54] <loicd> but it does not... :-)
[11:55] * dalgaaf (~dalgaaf@nrbg-4dbe25d4.pool.mediaWays.net) has joined #ceph
[11:57] * dalgaaf-tmp (~dalgaaf@85.214.11.161) Quit (Read error: Operation timed out)
[11:58] * nwat (~nwat@c-50-131-197-174.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[12:00] <ccourtaut> doesn't seem to be arbitrary indeed :)
[12:01] * andreask (~andreask@h081217135028.dyn.cm.kabsi.at) has joined #ceph
[12:01] * ChanServ sets mode +v andreask
[12:03] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) has joined #ceph
[12:07] * fireD_ is now known as fireD
[12:10] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[12:14] * ScOut3R (~ScOut3R@catv-89-133-25-52.catv.broadband.hu) Quit (Ping timeout: 480 seconds)
[12:18] <huangjun> loicd: it lays on the OSDMonitor.cc::prepare_command and process_command function
[12:19] * LeaChim (~LeaChim@2.122.178.96) has joined #ceph
[12:20] <loicd> huangjun: thanks, thats https://github.com/ceph/ceph/blob/master/src/mon/OSDMonitor.cc#L2578
[12:21] <loicd> right ?
[12:28] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) Quit (Quit: Leaving.)
[12:29] * goldfish (~goldfish@91.215.166.4) has joined #ceph
[12:31] <huangjun> yes,
[12:31] * ScOut3R (~ScOut3R@catv-89-133-17-71.catv.broadband.hu) has joined #ceph
[12:31] <huangjun> all the cmds like "ceph osd" "ceph mds" "ceph pg" are direct to MDSMonitor OSDMonitor PGMonitor
[12:31] * klklkl (~360BUY@221.226.39.82) has joined #ceph
[12:36] * klklkl (~360BUY@221.226.39.82) Quit (Quit: Leaving)
[12:36] * madkiss1 (~madkiss@2001:6f8:12c3:f00f:e1ed:f163:805c:87f7) Quit (Ping timeout: 480 seconds)
[12:46] * agh (~oftc-webi@gw-to-666.outscale.net) has joined #ceph
[12:48] * mozg (~andrei@host217-46-236-49.in-addr.btopenworld.com) has joined #ceph
[12:49] <mozg> wido: sorry to trouble you. do you know if you can resize an rbd guest vm disk once it has been created?
[12:49] * madkiss (~madkiss@2001:6f8:12c3:f00f:b540:54ff:945:9838) has joined #ceph
[12:52] * tziOm (~bjornar@194.19.106.242) has joined #ceph
[12:54] * madkiss (~madkiss@2001:6f8:12c3:f00f:b540:54ff:945:9838) Quit ()
[12:54] * huangjun (~kvirc@111.173.155.201) Quit (Quit: KVIrc 4.2.0 Equilibrium http://www.kvirc.net/)
[12:54] * madkiss (~madkiss@2001:6f8:12c3:f00f:b540:54ff:945:9838) has joined #ceph
[13:04] * andreask (~andreask@h081217135028.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[13:05] * nhm (~nhm@184-97-255-87.mpls.qwest.net) Quit (Ping timeout: 480 seconds)
[13:14] <Kioob`Taff> mozg: yes
[13:14] <mozg> how do I do that?
[13:15] <Kioob`Taff> rbd resize poolname/imgname --size XXXX
[13:15] <Kioob`Taff> be carefull to *not* reduce it
[13:17] <tnt> I think newer ceph have a BFW (Big Fat Warning tm) when sizing down.
[13:30] * diegows (~diegows@190.190.11.42) has joined #ceph
[13:33] * dalgaaf (~dalgaaf@nrbg-4dbe25d4.pool.mediaWays.net) Quit (Ping timeout: 480 seconds)
[13:35] * dalgaaf (~dalgaaf@85.214.11.161) has joined #ceph
[13:48] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[13:53] * infinitytrapdoor (~infinityt@134.95.27.132) has joined #ceph
[14:01] <mozg> thanks
[14:02] * dosaboy_1lt (~dosaboy_a@faun.canonical.com) has joined #ceph
[14:04] * dosaboy_alt (~dosaboy_a@faun.canonical.com) Quit (Ping timeout: 480 seconds)
[14:08] <niklas> How do I report a Bug in ceph?
[14:08] <niklas> Maybe I'm just blinde, but I can't find a way to create a new issue on http://tracker.ceph.com/projects/ceph
[14:10] * markbby (~Adium@168.94.245.2) has joined #ceph
[14:10] <tnt> niklas: are you logged in ?
[14:10] <joelio> niklas: have you registered ?
[14:10] <niklas> yes
[14:10] <joelio> tnt: snap :)
[14:10] <niklas> both
[14:11] <niklas> doh - just found it
[14:11] <niklas> neverminde
[14:11] <tnt> :)
[14:29] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[14:30] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[14:31] * smiley (~smiley@pool-173-73-0-53.washdc.fios.verizon.net) Quit (Quit: smiley)
[14:39] * BillK_ (~BillK-OFT@124-148-246-233.dyn.iinet.net.au) has joined #ceph
[14:41] * BillK (~BillK-OFT@124-148-246-233.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[14:45] * KippiX (~kippix@coquelicot-a.easter-eggs.com) has joined #ceph
[14:47] * jochen (~jochen@laevar.de) Quit (Ping timeout: 480 seconds)
[14:48] * jochen (~jochen@laevar.de) has joined #ceph
[14:50] * alfredodeza (~alfredode@216.1.187.162) has joined #ceph
[14:53] * joelio gets more test kit
[14:54] * yanzheng (~zhyan@134.134.137.75) has joined #ceph
[14:55] * goldfish (~goldfish@91.215.166.4) Quit (Ping timeout: 480 seconds)
[14:58] * zhu (~quassel@117.79.232.156) Quit (Read error: Connection reset by peer)
[14:58] * zhu (~quassel@117.79.232.203) has joined #ceph
[15:03] * aliguori (~anthony@cpe-70-112-157-87.austin.res.rr.com) Quit (Remote host closed the connection)
[15:04] * Vincent_Valentine (~Vincent_V@49.206.158.155) has joined #ceph
[15:05] * zhu_ (~quassel@117.79.232.188) has joined #ceph
[15:07] * zhu (~quassel@117.79.232.203) Quit (Ping timeout: 480 seconds)
[15:10] * tnt (~tnt@212-166-48-236.win.be) Quit (Ping timeout: 480 seconds)
[15:14] * agh (~oftc-webi@gw-to-666.outscale.net) Quit (Quit: Page closed)
[15:14] * nwat (~nwat@c-50-131-197-174.hsd1.ca.comcast.net) has joined #ceph
[15:15] * alfredodeza (~alfredode@216.1.187.162) Quit (Remote host closed the connection)
[15:22] * ScOut3R (~ScOut3R@catv-89-133-17-71.catv.broadband.hu) Quit (Ping timeout: 480 seconds)
[15:23] * andreask (~andreask@h081217135028.dyn.cm.kabsi.at) has joined #ceph
[15:23] * ChanServ sets mode +v andreask
[15:26] * doxavore (~doug@99-89-22-187.lightspeed.rcsntx.sbcglobal.net) has joined #ceph
[15:27] * scuttlemonkey (~scuttlemo@2607:f298:a:607:91ef:d414:2ad6:6a62) has joined #ceph
[15:27] * ChanServ sets mode +o scuttlemonkey
[15:27] * sagelap (~sage@2600:1012:b019:7627:6196:4c82:9a99:6110) has joined #ceph
[15:31] * yanzheng (~zhyan@134.134.137.75) Quit (Remote host closed the connection)
[15:34] * sagelap (~sage@2600:1012:b019:7627:6196:4c82:9a99:6110) Quit (Read error: Connection reset by peer)
[15:34] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[15:38] * scuttlemonkey changes topic to 'TODAY -- Day 2 Ceph Developer Summit: Emperor - http://ceph.com/cds JOIN: #ceph-summit for chat || Latest stable (v0.61.7 "Cuttlefish") -- http://ceph.com/get'
[15:38] * aliguori (~anthony@32.97.110.51) has joined #ceph
[15:46] <sagewk> good morning
[15:47] * sagelap (~sage@2607:f298:a:607:ea03:9aff:febc:4c23) has joined #ceph
[15:51] * rturk-away is now known as rturk
[15:52] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) has joined #ceph
[15:53] <wido> morning!
[15:53] <sagewk> /join #ceph-summit !
[15:54] <scuttlemonkey> ** ======= CDS =======
[15:54] <scuttlemonkey> ** Day 2 Ceph Developer Summit starting shortly
[15:54] <scuttlemonkey> ** /join #ceph-summit for discussion
[15:54] <scuttlemonkey> ** First block of session broadcast "live" at: http://youtu.be/-K8bSHx7zJ0
[15:54] <scuttlemonkey> ** View schedule at: http://ceph.com/cds/
[15:54] <scuttlemonkey> ** ======= CDS =======
[15:59] * markbby (~Adium@168.94.245.2) Quit (Quit: Leaving.)
[16:01] * markbby (~Adium@168.94.245.2) has joined #ceph
[16:02] * davidzlap (~Adium@ip68-5-239-214.oc.oc.cox.net) has joined #ceph
[16:09] * joao (~JL@2607:f298:a:607:9eeb:e8ff:fe0f:c9a6) has joined #ceph
[16:09] * ChanServ sets mode +o joao
[16:17] * alfredodeza (~alfredode@38.122.20.226) has joined #ceph
[16:18] <cfreak201> Has there been any major performance gain grom 2.6.32 -> 3.10 / .. considering xfs & ceph ? If no one knows I'll try to figure that out just want to avoid unneeded time investments..
[16:20] <scuttlemonkey> cfreak201: most of the brains who would know are involved in our Ceph Developer Summit going on right now
[16:20] <scuttlemonkey> but I haven't heard about any meaningful performance gains
[16:21] <cfreak201> scuttlemonkey: i know, i'm watching the livestream ;) Maybe they'll have a chance to answer later
[16:21] <scuttlemonkey> yah, I'll see if I can prod someone
[16:21] <scuttlemonkey> those who aren't involved aren't in the LA office yet
[16:21] <rturk> if you have a question for the folks in the summit, you might want to post it in #ceph-summit - that's where the summit folks are hanging out
[16:22] * Vincent_Valentine (~Vincent_V@49.206.158.155) Quit (Read error: Connection reset by peer)
[16:22] * Vincent_Valentine (Vincent_Va@49.206.158.155) has joined #ceph
[16:22] <scuttlemonkey> but if you're watching I can just inject that question as a part of our catch-all intro Q&A
[16:22] * BillK_ (~BillK-OFT@124-148-246-233.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[16:23] <mikedawson> cfreak201: if you are using the ceph kernel module, you want a new kernel. Otherwise, http://ceph.com/docs/next/install/os-recommendations/#ceph-dependencies
[16:24] <cfreak201> scuttlemonkey: that would be nice, i'm "stuck" on centos 6.4 :/
[16:24] <cfreak201> mikedawson: no kernel rbd... mostly / only qemu with librbd i guess
[16:24] <scuttlemonkey> ahh, yeah...most of the kernel stuff has been stability/feature iirc
[16:24] <scuttlemonkey> but we'll see what Sage says
[16:28] * grepory (~Adium@50-115-70-146.static-ip.telepacific.net) has joined #ceph
[16:31] * CliMz (~CliMz@179-179.62-81.cust.bluewin.ch) has joined #ceph
[16:31] <cfreak201> thank you very much ;-)
[16:32] <scuttlemonkey> np
[16:33] <CliMz> Hi guys ! I'm testing ceph and want to share a block device between two servers. Both can see it but it seems that each one own a version of it. I can write on both but not see the files on other node. To commit changes i have to unmount mount volumes. I'm sure i misunderstood something but what ? Thx for helping
[16:33] * infinitytrapdoor (~infinityt@134.95.27.132) Quit ()
[16:34] <scuttlemonkey> CliMz: yeah, block devices aren't meant to be shared like that
[16:35] <scuttlemonkey> if you want them to share you could mount a block device somewhere and reexport via NFS...or you could try using cephFS
[16:35] <CliMz> well the ultimate goal is be able to share storage between KVM node
[16:35] <CliMz> to achieve live migration
[16:36] <CliMz> and HA
[16:36] <CliMz> what do you think is the best way ?
[16:37] <joelio> use libvirt?
[16:37] <CliMz> yes
[16:37] <joelio> that's what I do using OpenNebula - Live migrations are handled inside libvirt
[16:37] <joelio> I use cephfs-fuse to store the opennebula system datastore
[16:38] <CliMz> ok
[16:39] <CliMz> so i'll try to go further with that and http://ceph.com/docs/master/rbd/libvirt/ but i really thought (reading all about Ceph online) that it could share block devices
[16:40] <joelio> sharing block devices?
[16:40] <joelio> umm, that sounds like a 'bad thing' tm
[16:40] <scuttlemonkey> CliMz: Sebastien touched on this a bit
[16:40] <scuttlemonkey> http://ceph.com/user-story/ceph-from-poc-to-production/
[16:40] <scuttlemonkey> and talked about the NFS option: http://www.sebastien-han.fr/blog/2012/07/06/nfs-over-rbd/
[16:40] <scuttlemonkey> but yeah, libvirt is the more traditional answer
[16:41] <CliMz> ok
[16:41] <CliMz> thanks guy i'll continue testing and come back asking if needed :)
[16:42] <scuttlemonkey> cool
[16:47] <darkfaded> wow, thats a nice way of making a howto
[16:47] <darkfaded> and a ETOOMANYLAYERS
[16:49] <scuttlemonkey> darkfaded: both of those posts are showing their age a bit
[16:52] <darkfaded> scuttlemonkey: i'd still love to be able to produce something like that
[16:53] * bergerx_ (~bekir@78.188.101.175) Quit (Quit: Leaving.)
[16:55] * bergerx_ (~bekir@78.188.204.182) has joined #ceph
[16:55] * junglebells (~junglebel@0001b1b9.user.oftc.net) has joined #ceph
[17:00] * allsystemsarego (~allsystem@188.25.130.190) Quit (Quit: Leaving)
[17:02] * rongze_ (~quassel@notes4.com) Quit (Remote host closed the connection)
[17:02] * tnt (~tnt@109.130.80.16) has joined #ceph
[17:03] * bergerx_ (~bekir@78.188.204.182) has left #ceph
[17:09] * davidzlap (~Adium@ip68-5-239-214.oc.oc.cox.net) Quit (Read error: Operation timed out)
[17:12] * mikedawson_ (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) has joined #ceph
[17:13] * davidzlap (~Adium@ip68-5-239-214.oc.oc.cox.net) has joined #ceph
[17:15] * tziOm (~bjornar@194.19.106.242) Quit (Remote host closed the connection)
[17:17] * saabylaptop (~saabylapt@2a02:2350:18:1010:150:278a:15:d71f) Quit (Quit: Leaving.)
[17:18] * gregmark (~Adium@68.87.42.115) has joined #ceph
[17:19] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[17:20] * BManojlovic (~steki@91.195.39.5) Quit (Quit: Ja odoh a vi sta 'ocete...)
[17:21] * sprachgenerator (~sprachgen@130.202.135.209) has joined #ceph
[17:21] * sjm (~oftc-webi@c73-103.rim.net) has joined #ceph
[17:22] * goldfish (~goldfish@91.215.166.4) has joined #ceph
[17:25] * Mersaul (~oftc-webi@c-98-240-246-30.hsd1.mn.comcast.net) has joined #ceph
[17:27] * sleinen (~Adium@eduroam-4-152.epfl.ch) has joined #ceph
[17:28] * dobber (~dobber@213.169.45.222) Quit (Remote host closed the connection)
[17:31] * CliMz (~CliMz@179-179.62-81.cust.bluewin.ch) Quit (Ping timeout: 480 seconds)
[17:38] * sleinen1 (~Adium@2001:620:0:25:6800:f32b:e259:c2fb) has joined #ceph
[17:40] * Mersaul (~oftc-webi@c-98-240-246-30.hsd1.mn.comcast.net) Quit (Quit: Page closed)
[17:41] * odyssey4me (~odyssey4m@165.233.71.2) Quit (Ping timeout: 480 seconds)
[17:45] * sleinen (~Adium@eduroam-4-152.epfl.ch) Quit (Ping timeout: 480 seconds)
[17:51] * mtl (~Adium@c-67-176-54-246.hsd1.co.comcast.net) has joined #ceph
[17:56] * devoid (~devoid@130.202.135.211) has joined #ceph
[17:58] * nwat (~nwat@c-50-131-197-174.hsd1.ca.comcast.net) Quit (Read error: Operation timed out)
[18:00] * markbby (~Adium@168.94.245.2) Quit (Quit: Leaving.)
[18:02] * Vincent_Valentine (Vincent_Va@49.206.158.155) Quit (Read error: Connection reset by peer)
[18:02] * Vincent_Valentine (Vincent_Va@49.206.158.155) has joined #ceph
[18:10] * mikedawson_ (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[18:10] * Vjarjadian (~IceChat77@90.214.208.5) has joined #ceph
[18:15] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) has joined #ceph
[18:16] * markbby (~Adium@168.94.245.2) has joined #ceph
[18:18] <rturk> Just resumed the live video feed of the Ceph Developer Summit: http://youtu.be/cGosx5zD4FM
[18:18] <rturk> welcome to all, feel free to listen in :)
[18:20] * diegows (~diegows@190.190.11.42) Quit (Ping timeout: 480 seconds)
[18:20] * mikedawson_ (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) has joined #ceph
[18:22] <L2SHO> cfreak201, elrepo.org maintains a mainline kernel for CentOS if that's an option for you
[18:23] <darkfaded> L2SHO: ah, *that's* how people on CentOS6 manage to run ceph and not live in pain
[18:23] <darkfaded> i only had a elrepo kernel once for e1000e bugfixes
[18:24] <L2SHO> darkfaded, well they have separate repo's for their kernel, and kernel modules, but e1000e is the most common reson we use them too
[18:26] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[18:26] * mikedawson_ is now known as mikedawson
[18:29] * sleinen (~Adium@eduroam-4-152.epfl.ch) has joined #ceph
[18:31] * sleinen2 (~Adium@2001:620:0:25:a9be:43c2:feb9:e476) has joined #ceph
[18:32] * odyssey4me (~odyssey4m@41-133-58-101.dsl.mweb.co.za) has joined #ceph
[18:33] * sleinen (~Adium@eduroam-4-152.epfl.ch) Quit (Read error: Operation timed out)
[18:34] * john (~john@2607:f298:a:607:d6be:d9ff:fe04:efa6) has joined #ceph
[18:34] * hybrid512 (~walid@LPoitiers-156-86-25-85.w193-248.abo.wanadoo.fr) Quit (Quit: Leaving.)
[18:34] * sleinen1 (~Adium@2001:620:0:25:6800:f32b:e259:c2fb) Quit (Ping timeout: 480 seconds)
[18:35] * ron-slc (~Ron@173-165-129-125-utah.hfc.comcastbusiness.net) has joined #ceph
[18:36] * xmltok (~xmltok@pool101.bizrate.com) has joined #ceph
[18:41] * erice (~erice@c-98-245-48-79.hsd1.co.comcast.net) Quit (Quit: erice)
[18:49] * saabylaptop (~saabylapt@1009ds5-oebr.1.fullrate.dk) has joined #ceph
[18:49] * saabylaptop (~saabylapt@1009ds5-oebr.1.fullrate.dk) Quit ()
[18:49] * saabylaptop (~saabylapt@1009ds5-oebr.1.fullrate.dk) has joined #ceph
[18:50] <cfreak201> L2SHO: thats what I had in mind if I had to upgrade.. will perfome some benchmarking of my application / setup later today or tomorrow.. maybe it just makes a few MB/s difference but that would be good enough for me :)
[18:52] <L2SHO> cfreak201, I believe that kernel version only matters if you are using the kernel rbd client, but I could be wrong about that
[18:53] * Midnightmyth (~quassel@93-167-84-102-static.dk.customer.tdc.net) has joined #ceph
[18:54] <joshd> cfreak201: if you're creating new fses on your new kernel underneath osds, you'll probably see some benefits - many improvements have been made to xfs, btrfs, and ext4
[18:54] * andreask (~andreask@h081217135028.dyn.cm.kabsi.at) Quit (Read error: Operation timed out)
[18:55] * sleinen2 (~Adium@2001:620:0:25:a9be:43c2:feb9:e476) Quit (Quit: Leaving.)
[18:56] <cfreak201> joshd: thats my expectation that xfs has some fixes/improvements/..
[18:59] * VincentValentine (Vincent_Va@49.206.158.155) has joined #ceph
[19:02] * Vincent_Valentine (Vincent_Va@49.206.158.155) Quit (Ping timeout: 480 seconds)
[19:04] * VincentValentine (Vincent_Va@49.206.158.155) Quit (Read error: Connection reset by peer)
[19:04] * Vincent_Valentine (Vincent_Va@49.206.158.155) has joined #ceph
[19:07] * dosaboy_1lt (~dosaboy_a@faun.canonical.com) Quit (Quit: leaving)
[19:08] * lyncos (~chatzilla@208.71.184.41) has joined #ceph
[19:13] * mozg (~andrei@host217-46-236-49.in-addr.btopenworld.com) Quit (Ping timeout: 480 seconds)
[19:17] * odyssey4me (~odyssey4m@41-133-58-101.dsl.mweb.co.za) Quit (Ping timeout: 480 seconds)
[19:18] * diegows (~diegows@190.190.11.42) has joined #ceph
[19:25] * sleinen1 (~Adium@2001:620:0:25:2142:9ec4:d6ce:f717) has joined #ceph
[19:27] * odyssey4me (~odyssey4m@41-133-58-101.dsl.mweb.co.za) has joined #ceph
[19:27] * saabylaptop (~saabylapt@1009ds5-oebr.1.fullrate.dk) Quit (Quit: Leaving.)
[19:30] * xmltok (~xmltok@pool101.bizrate.com) Quit (Remote host closed the connection)
[19:30] * xmltok (~xmltok@relay.els4.ticketmaster.com) has joined #ceph
[19:37] * bandrus (~Adium@cpe-76-95-217-129.socal.res.rr.com) Quit (Quit: Leaving.)
[19:37] * amatter (~oftc-webi@209.63.136.134) has joined #ceph
[19:37] * buck (~buck@c-24-6-91-4.hsd1.ca.comcast.net) has joined #ceph
[19:37] <amatter> Howdy. One of my OSDs won't start and has the following error: error (39) Directory not empty not handled on operation 21 (6080518.0.15, or op 15, counting from 0). Not sure the best practice for resolving this.
[19:40] <amatter> full log is here: http://pastebin.com/ktrWJGqP
[19:45] <scuttlemonkey> amatter: most of the Ceph folks are involved in the developer summit
[19:45] <amatter> scuttlemonkey: thanks for the heads up
[19:45] <scuttlemonkey> is this a new osd?
[19:48] <amatter> scuttlemonkey: no, it was running then the disk went off line due to io errors while the osd was running, I rebooted the host and the drive is back online but it seems like journal is out of sync with the disk or missing some earlier transactions, perhaps
[19:48] * nhm (~nhm@mf52736d0.tmodns.net) has joined #ceph
[19:49] <scuttlemonkey> may have to run a deep scrub or something...but I'm not the guy to help you with specifics
[19:50] * lyncos (~chatzilla@208.71.184.41) Quit (Remote host closed the connection)
[19:52] <amatter> scuttlemonkey: thanks, i'll check over the docs again, maybe I'll just recreate the OSD and allow it to refill it, but it's a 3TB drive and will take a long time to refill 2TB worth
[19:53] <scuttlemonkey> amatter: gotcha, might be worth tossing the question on the list
[19:53] <scuttlemonkey> then the experts will be able to hit it async...even if it's just post-mortem
[19:54] * The_Bishop (~bishop@2001:470:50b6:0:fdd2:ad52:2d13:596c) has joined #ceph
[19:55] * sleinen1 (~Adium@2001:620:0:25:2142:9ec4:d6ce:f717) Quit (Quit: Leaving.)
[19:55] * sleinen (~Adium@eduroam-4-152.epfl.ch) has joined #ceph
[19:57] * saabylaptop (~saabylapt@1009ds5-oebr.1.fullrate.dk) has joined #ceph
[19:58] * xmltok_ (~xmltok@pool101.bizrate.com) has joined #ceph
[19:58] * saabylaptop (~saabylapt@1009ds5-oebr.1.fullrate.dk) Quit ()
[20:01] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[20:01] * saabylaptop (~saabylapt@1009ds5-oebr.1.fullrate.dk) has joined #ceph
[20:02] * nhm (~nhm@mf52736d0.tmodns.net) Quit (Ping timeout: 480 seconds)
[20:02] * mschiff_ (~mschiff@p4FD7E178.dip0.t-ipconnect.de) Quit (Remote host closed the connection)
[20:03] * sleinen (~Adium@eduroam-4-152.epfl.ch) Quit (Ping timeout: 480 seconds)
[20:04] * xmltok (~xmltok@relay.els4.ticketmaster.com) Quit (Ping timeout: 480 seconds)
[20:05] * alram (~alram@38.122.20.226) has joined #ceph
[20:09] * mmercer (~kvirc@199.127.107.196) has joined #ceph
[20:10] * KevinPerks (~Adium@38.122.20.226) has joined #ceph
[20:11] * nwat (~nwat@156.39.10.22) has joined #ceph
[20:12] * dpippenger (~riven@tenant.pas.idealab.com) has joined #ceph
[20:18] <rturk> Just started up the Ceph Dev Summit sessions 26-29. Live video link: http://youtu.be/hrQuYzyXvVg
[20:22] * nwat (~nwat@156.39.10.22) Quit (Ping timeout: 480 seconds)
[20:25] * andreask (~andreask@h081217135028.dyn.cm.kabsi.at) has joined #ceph
[20:25] * ChanServ sets mode +v andreask
[20:28] * nhm (~nhm@ma30436d0.tmodns.net) has joined #ceph
[20:29] * nhm (~nhm@ma30436d0.tmodns.net) Quit ()
[20:31] * nhm (~nhm@ma30436d0.tmodns.net) has joined #ceph
[20:34] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[20:37] * troug (~troug@c-50-140-187-64.hsd1.il.comcast.net) has joined #ceph
[20:45] * andreask (~andreask@h081217135028.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[20:48] * KevinPerks (~Adium@38.122.20.226) Quit (Quit: Leaving.)
[20:48] * ishkabob (~c7a82cc0@webuser.thegrebs.com) has joined #ceph
[20:49] <ishkabob> does anyone know if there is a way to add an osd manually using a predetermined osd number?
[20:49] <ishkabob> the "ceph osd create" will only output a number which makes deploying with something like puppet a bit of a pain
[20:51] <dmick> ishkabob: generally speaking no
[20:51] <scuttlemonkey> ishkabob: have you taken a look at http://github.com/enovance/puppet-ceph
[20:52] <ishkabob> dmick: thanks for the answer, it seems like ceph-deploy does something like this, as i've deployed a ceph cluster with osds like {osd.001001, osd.001002} it seems to add them as {osd.1001 osd.1002}
[20:53] <ishkabob> scuttlemonkey: i haven't really poked much at the source code, but i'll look again, forgot it was there
[20:53] <scuttlemonkey> gotcha
[20:53] <scuttlemonkey> yeah, those are the most mature puppet modules for ceph that I know of
[20:54] <scuttlemonkey> still could be better, but it might be a good start for you...and I know Sebastien would welcome patches if you figure out a better way
[20:54] * KevinPerks (~Adium@38.122.20.226) has joined #ceph
[20:54] * DarkAce-Z (~BillyMays@50.107.55.36) has joined #ceph
[20:54] <ishkabob> scuttlemonkey: i'm actually using an in-house deployment tool combined with facter facts from puppet. Puppet tells me what drives are available to use as OSDs and then our deployment tools do the rest
[20:54] <ishkabob> i'd rather just do it with puppet though, so this might be better
[20:55] <scuttlemonkey> ahh, gotcha
[20:55] <scuttlemonkey> interesting
[20:55] <dmick> ishkabob: that's a bit surprising.
[20:55] <ishkabob> dmick: why is that?
[20:55] <dmick> because I don't know of an interface to supply a number
[20:57] * DarkAceZ (~BillyMays@50.107.55.36) Quit (Ping timeout: 480 seconds)
[20:57] <ishkabob> dmick: i'm going to look again to verify
[21:00] <ishkabob> dmick: here ya go - http://pastebin.com/EEjpuWtj
[21:01] <ishkabob> dmick: you think the mailing list might have something to say about this? :)
[21:02] * mschiff (~mschiff@85.182.236.82) has joined #ceph
[21:04] <dmick> ishkabob: can't hurt. I just reexamined the current ceph-deploy code and I 1) don't see any place to specify an osdid, and 2) see the code where it's asking the cluster for the next one by executing ceph osd create and taking the output, so, yeah, I'm mystified how you did that with ceph-deploy
[21:05] <ishkabob> dmick: it's possible this was done with mkcephfs, but I don't think so. I'm basically trying to recreate something that someone did using puppet
[21:05] <ishkabob> dmick: thanks for the info, i'm gonna email the mailing list cheers
[21:05] <dmick> ishkabob: what version is this?
[21:06] * saabylaptop (~saabylapt@1009ds5-oebr.1.fullrate.dk) Quit (Quit: Leaving.)
[21:08] <ishkabob> ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60)
[21:08] <ishkabob> sry, dmick: ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60)
[21:09] <ishkabob> the real problem for me is that when I add a new osd, it needs to make it back to the ceph.conf which is generated, stored in a single location, and distributed before deployment (in my setup)
[21:10] <dmick> you can supply UUIDs, and get a map of UUID to ID; that might be useful
[21:10] <ishkabob> i can generate my ceph.conf purely based on facter facts from puppet, and then a script does the rest. but if I have to start hitting the cluster before deployment, then I need one method to bootstrap, and a different method for maintenance and adding/removing components
[21:12] * markbby (~Adium@168.94.245.2) Quit (Remote host closed the connection)
[21:13] * markbby (~Adium@168.94.245.2) has joined #ceph
[21:13] * gentleben (~sseveranc@c-98-207-40-73.hsd1.ca.comcast.net) Quit (Quit: gentleben)
[21:13] <ishkabob> dmick: how do I supply the UUID if I haven't run ceph osd create yet?
[21:13] <dmick> you supply it at create time
[21:13] <dmick> i.e., you can't specify an ID, but you can specify a UUID
[21:13] <dmick> osd create {<uuid>} create new osd (with optional UUID)
[21:14] <ishkabob> dmick: ah, so I could just randomly generate a UUID and then get the map before deployment?
[21:14] <ishkabob> dmick: i think that just might work
[21:25] * ntranger (~ntranger@proxy2.wolfram.com) Quit ()
[21:31] * gentleben (~sseveranc@12.250.97.26) has joined #ceph
[21:31] * houkouonchi_work (~houkouonc@38.122.20.226) has joined #ceph
[21:31] * fooz (~andy@wsip-68-225-89-79.dc.dc.cox.net) has joined #ceph
[21:32] <fooz> ive been tasked with evaluating riak cs vs ceph - has anyone here done such a thing? (use case is object storage)
[21:34] * saabylaptop (~saabylapt@1009ds5-oebr.1.fullrate.dk) has joined #ceph
[21:34] <gentleben> fooz: how are you evaluating?
[21:34] * Cube (~Cube@38.122.20.226) has joined #ceph
[21:35] <fooz> compatibility with common S3 clients, robustness testing, killing servers, etc
[21:35] <fooz> thats the plan at least
[21:37] <fooz> dammit brb
[21:37] * fooz (~andy@wsip-68-225-89-79.dc.dc.cox.net) Quit (Quit: BitchX: often imitated, never duplicated!)
[21:37] * dpippenger (~riven@tenant.pas.idealab.com) Quit (Ping timeout: 480 seconds)
[21:42] * dpippenger (~riven@tenant.pas.idealab.com) has joined #ceph
[21:54] * leseb (~leseb@88-190-214-97.rev.dedibox.fr) Quit (Killed (NickServ (Too many failed password attempts.)))
[21:55] * leseb (~leseb@88-190-214-97.rev.dedibox.fr) has joined #ceph
[21:55] * andreask (~andreask@h081217135028.dyn.cm.kabsi.at) has joined #ceph
[21:55] * ChanServ sets mode +v andreask
[21:58] * odyssey4me2 (~odyssey4m@41-133-58-101.dsl.mweb.co.za) has joined #ceph
[21:58] * odyssey4me (~odyssey4m@41-133-58-101.dsl.mweb.co.za) Quit (Ping timeout: 480 seconds)
[22:01] * bandrus (~Adium@2607:f298:a:697:eca6:666b:2e97:46fa) has joined #ceph
[22:03] * devoid (~devoid@130.202.135.211) Quit (Quit: Leaving.)
[22:05] * mmercer (~kvirc@199.127.107.196) Quit (Ping timeout: 480 seconds)
[22:06] * devoid (~devoid@130.202.135.211) has joined #ceph
[22:06] * rturk is now known as rturk-away
[22:12] * bandrus (~Adium@2607:f298:a:697:eca6:666b:2e97:46fa) Quit (Quit: Leaving.)
[22:14] * odyssey4me2 (~odyssey4m@41-133-58-101.dsl.mweb.co.za) Quit (Ping timeout: 480 seconds)
[22:15] * bandrus (~Adium@38.122.20.226) has joined #ceph
[22:15] * jjgalvez (~jjgalvez@38.122.20.226) has joined #ceph
[22:21] * dosaboy (~dosaboy@host81-152-10-65.range81-152.btcentralplus.com) has joined #ceph
[22:21] * jjgalvez1 (~jjgalvez@38.122.20.226) has joined #ceph
[22:26] * ntranger (~ntranger@proxy2.wolfram.com) has joined #ceph
[22:26] * dalgaaf (~dalgaaf@85.214.11.161) Quit (Quit: Konversation terminated!)
[22:27] * dosaboy_ (~dosaboy@host86-156-252-32.range86-156.btcentralplus.com) Quit (Ping timeout: 480 seconds)
[22:28] * jjgalvez (~jjgalvez@38.122.20.226) Quit (Ping timeout: 480 seconds)
[22:29] * KevinPerks (~Adium@38.122.20.226) Quit (Quit: Leaving.)
[22:32] * KevinPerks (~Adium@2607:f298:a:607:258e:7bbb:7b2f:3ba7) has joined #ceph
[22:33] <ishkabob> hey there aren't any plans to implement an RBD device in windows correct?
[22:34] <scuttlemonkey> ishkabob: not specifically. However we have stuff upstream in Samba as well as a couple of options pending for NFS/CIFS
[22:35] <ishkabob> scuttlemonkey: yeah we are currently installing NFS/CIFS on our ceph nodes and exposing RBDs to them
[22:36] <scuttlemonkey> gotcha
[22:36] <scuttlemonkey> there are also a few tech previews of people doing similar things
[22:36] <scuttlemonkey> one uses Ganesha to expose cephFS
[22:36] <scuttlemonkey> couple others that aren't coming to mind immediately
[22:36] <scuttlemonkey> but yeah, no plans for native RBD-on-windows
[22:37] * davidzlap (~Adium@ip68-5-239-214.oc.oc.cox.net) Quit (Quit: Leaving.)
[22:37] * LeaChim (~LeaChim@2.122.178.96) Quit (Read error: Operation timed out)
[22:38] * Vincent_Valentine (Vincent_Va@49.206.158.155) Quit (Ping timeout: 480 seconds)
[22:48] * LeaChim (~LeaChim@0540749b.skybroadband.com) has joined #ceph
[22:49] * saabylaptop (~saabylapt@1009ds5-oebr.1.fullrate.dk) Quit (Quit: Leaving.)
[22:52] * grepory (~Adium@50-115-70-146.static-ip.telepacific.net) Quit (Quit: Leaving.)
[22:57] * dirk___ (~dirk@nrbg-4dbe3c45.pool.mediaWays.net) has joined #ceph
[22:58] <dmick> ishkabob: there is also an iSCSI gateway for rbd images that can provide connectivity to Windows
[22:58] <dmick> part of stgt
[22:58] <dmick> it has been reported to work by external sources
[22:59] <dirk___> does anyone have an idea on how to "trigger" the peering process?
[22:59] <dirk___> I have a cluster in this state:
[22:59] <dirk___> HEALTH_WARN 191 pgs peering; 192 pgs stuck inactive; 192 pgs stuck unclean
[23:00] <dirk___> and all osd's with essentially zero load (or any relevant log entries with level 10 or higher)
[23:00] <dirk___> it seems it is just waitin for something to happen that doesn't happen
[23:03] <paravoid> which version?
[23:03] <paravoid> I'm not authoritative in any way, but you can't "trigger" the peering process
[23:03] <dirk___> 0.48.1
[23:03] <paravoid> that's probably a bug
[23:03] <paravoid> you can out or restart the osds involved
[23:03] <paravoid> ceph pg dump |grep peering should give you the list of osds for those pgs
[23:04] <dmick> dirk___: that is *ancient*. You probably should upgrade.
[23:05] * Meths_ is now known as Meths
[23:05] <paravoid> yeah this is a bug but I don't think there's any point in debugging this since it's such an old version
[23:05] <dirk___> I totally agree, but there are a few important bits in this ceph cluster that I somehow need to get out of it ..
[23:05] <Vjarjadian> backups?
[23:05] <paravoid> so mitigation would be to try to recover with out/in or restarts
[23:06] <paravoid> try "ceph pg dump |grep peering", see if there's a pattern of 1-2 osds and try restarting them
[23:06] <dirk___> Vjarjadian: no backups
[23:06] <paravoid> it's totally recoverable I think :)
[23:07] <dirk___> paravoid: well, I have 3 osd's, so the pg dump_stuck says 1,2, 2,1 1,0 0,1 and so on, all permutations
[23:07] <dirk___> I've tried already to restart the osd 2, 1 and 0 in any order
[23:08] <dirk___> nothing really happens though :(
[23:08] <Gugge-47527> dirk___: how many hosts?
[23:08] <dirk___> usually when it resyncs it bursts I/O and cpu, but all osd's seem to be essentially idle
[23:08] <dirk___> 3 osd's on 3 hosts, 3 mons (on other hosts)
[23:09] <dmick> was the cluster ever clean? What happened to it to make it unclean>?
[23:09] * amatter (~oftc-webi@209.63.136.134) Quit (Remote host closed the connection)
[23:10] <dirk___> it had a few degraded pg's (usually with 0 objects inside), but other than that active+clean, yes
[23:10] <dirk___> few (like 10 or so)
[23:10] <dirk___> I don't know what happened, other that I went on vacation :)
[23:11] <dirk___> I believe the problem started when all osd's were restarted at the same time due to a config mass-deploy going berserk
[23:11] * mmercer (~kvirc@199.127.107.196) has joined #ceph
[23:13] * sagelap (~sage@2607:f298:a:607:ea03:9aff:febc:4c23) Quit (Ping timeout: 480 seconds)
[23:14] <dirk___> is there anything else other than restarting osd's ?
[23:19] * dpippenger (~riven@tenant.pas.idealab.com) Quit (Quit: Leaving.)
[23:19] * BManojlovic (~steki@fo-d-130.180.254.37.targo.rs) has joined #ceph
[23:19] <scuttlemonkey> dirk___: can you pastebin your ceph -s?
[23:20] <scuttlemonkey> or a ceph health detail
[23:20] <dirk___> scuttlemonkey: http://pastebin.com/1ffdmTnq
[23:21] * The_Bishop (~bishop@2001:470:50b6:0:fdd2:ad52:2d13:596c) Quit (Quit: Wer zum Teufel ist dieser Peer? Wenn ich den erwische dann werde ich ihm mal die Verbindung resetten!)
[23:22] <scuttlemonkey> yeah, grab the health detail too
[23:22] <scuttlemonkey> lets find a specific pg and drill into it to see what the system thinks is wrong
[23:22] <dmick> health detail might not even exist in 0.48.1
[23:23] <scuttlemonkey> erm
[23:23] * sagelap (~sage@2607:f298:a:607:c5e:7bb0:c323:186c) has joined #ceph
[23:23] <dirk___> scuttlemonkey: thats the beginning of health details
[23:23] <scuttlemonkey> I guess we can play w/ variations on a theme of ceph pg dump
[23:23] <dirk___> scuttlemonkey: http://pastebin.com/jBRRdUFm
[23:24] <scuttlemonkey> do 'ceph pg 0.3f query'
[23:25] <dirk___> scuttlemonkey: http://pastebin.com/Y2wQBvye
[23:27] * dosaboy_ (~dosaboy@host109-154-150-215.range109-154.btcentralplus.com) has joined #ceph
[23:29] * LeaChim (~LeaChim@0540749b.skybroadband.com) Quit (Read error: Operation timed out)
[23:30] * aliguori (~anthony@32.97.110.51) Quit (Remote host closed the connection)
[23:32] * dosaboy (~dosaboy@host81-152-10-65.range81-152.btcentralplus.com) Quit (Ping timeout: 480 seconds)
[23:33] * DarkAce-Z is now known as DarkAceZ
[23:39] <sjust> dirk___: can you paste in the output of ceph-osd -v?
[23:40] * LeaChim (~LeaChim@2.122.34.66) has joined #ceph
[23:40] <dirk___> sjust: ceph version 0.48.1argonaut (commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c)
[23:42] * loicd (~loicd@bouncer.dachary.org) Quit (Ping timeout: 480 seconds)
[23:43] <sjust> dirk___: do you have data in the data pool?
[23:43] * loicd (~loicd@bouncer.dachary.org) has joined #ceph
[23:43] <sjust> that is, in cephfs?
[23:43] <sjust> pastebin ceph osd dump?
[23:43] <dirk___> in rbd, yes
[23:44] <dirk___> interesting pg with objects is e.g. 0.3e
[23:44] * markbby (~Adium@168.94.245.2) Quit (Quit: Leaving.)
[23:44] <dirk___> do you want the query of that one?
[23:44] <sjust> not yet, just the output of ceph osd dump
[23:45] <sjust> can you describe how this situation came to be?
[23:45] <sjust> you mentioned a config mass-deploy
[23:46] * BManojlovic (~steki@fo-d-130.180.254.37.targo.rs) Quit (Remote host closed the connection)
[23:46] <dirk___> sjust: http://pastebin.com/hvVdNXaF
[23:47] <sjust> ok, so what happened prior to this situation occuring?
[23:49] <dirk___> all of the osd' hosts were rebooted for a reason unknown to me
[23:49] <sjust> each osd is on its own host?
[23:50] <dirk___> and I think osd.0 was reporting in ceph -w "slow responses".. so osd.0 was reweighted from 1 to 0.8 or so
[23:50] <dirk___> yes
[23:50] <dirk___> 1 drive per host per osd
[23:50] <sjust> can you post the ceph.conf from the osd hosts?
[23:51] <dirk___> can I send that in a query?
[23:51] <dirk___> its not big
[23:51] <sjust> ok
[23:51] * jjgalvez (~jjgalvez@38.122.20.226) has joined #ceph
[23:52] * dpippenger (~riven@tenant.pas.idealab.com) has joined #ceph
[23:55] <sjust> dirk___: july 28 is when this started?
[23:56] * dosaboy_ (~dosaboy@host109-154-150-215.range109-154.btcentralplus.com) Quit (Ping timeout: 480 seconds)
[23:56] * jjgalvez1 (~jjgalvez@38.122.20.226) Quit (Ping timeout: 480 seconds)
[23:57] <sjust> dirk___: have you tried restarting the mon daemons?
[23:58] <dirk___> sjust: either 2013-07-25 14:02 or 2013-07-28
[23:58] <dirk___> all hosts were rebooted on 07-28 (I believe in order to try to mitigate the hickups in ceph)
[23:58] <sjust> can you try restarting the mon daemons/
[23:58] <sjust> ?
[23:58] <dirk___> the prior date is the one that I can find as last_active in the pg's
[23:58] <dirk___> yes, I did that already several dozen times
[23:59] <sjust> ok
[23:59] <dirk___> other than the epoch changing and a new quorum vote, nothing really happening
[23:59] <sjust> ok, restart osd.0 with
[23:59] <sjust> debug osd = 20
[23:59] <sjust> debug ms = 1
[23:59] <sjust> in the [osd] section of the ceph.conf

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.