#ceph IRC Log

Index

IRC Log for 2013-11-20

Timestamps are in GMT/BST.

[0:00] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[0:00] * The_Bishop (~bishop@2001:470:50b6:0:ec49:a07a:74b3:4c77) Quit (Quit: Wer zum Teufel ist dieser Peer? Wenn ich den erwische dann werde ich ihm mal die Verbindung resetten!)
[0:08] <JoeGruher> if i have an object in a pool but i'm not sure what the object is how can i track it down? for example, right now my rbd pool reports 1 object, can i get more info on that object?
[0:08] * Hakisho (~Hakisho@0001be3c.user.oftc.net) Quit (Quit: bye)
[0:08] * The_Bishop (~bishop@2001:470:50b6:0:2d3e:12a2:8f13:d188) has joined #ceph
[0:09] <pmatulis> JoeGruher: would be a start: rados -p <pool> ls
[0:10] <JoeGruher> ah, great, thanks. it gave me "rbd_directory" so maybe that's something that always lives in rbd pool once you've had some rbd activity
[0:12] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[0:15] * tsnider (~tsnider@nat-216-240-30-23.netapp.com) Quit (Ping timeout: 480 seconds)
[0:16] * mtanski (~mtanski@69.193.178.202) has joined #ceph
[0:18] * dmsimard (~Adium@108.163.152.2) Quit (Quit: Leaving.)
[0:18] * nhm (~nhm@wlan-rooms-4019.sc13.org) has joined #ceph
[0:18] * ChanServ sets mode +o nhm
[0:24] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Read error: Operation timed out)
[0:26] * dxd828 (~dxd828@host-92-24-127-29.ppp.as43234.net) Quit (Quit: Computer has gone to sleep.)
[0:32] * nwat (~textual@eduroam-227-103.ucsc.edu) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[0:36] * gregmark (~Adium@68.87.42.115) Quit (Quit: Leaving.)
[0:38] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[0:38] * japuzzo (~japuzzo@ool-4570886e.dyn.optonline.net) has joined #ceph
[0:38] * dpippenger (~riven@66-192-9-78.static.twtelecom.net) Quit (Remote host closed the connection)
[0:38] * rongze (~rongze@117.79.232.205) has joined #ceph
[0:40] * AfC (~andrew@2407:7800:200:1011:6e88:14ff:fe33:2a9c) Quit (Quit: Leaving.)
[0:40] * AfC (~andrew@2407:7800:200:1011:2ad2:44ff:fe08:a4c) has joined #ceph
[0:46] * rongze (~rongze@117.79.232.205) Quit (Ping timeout: 480 seconds)
[0:48] <loicd> I have machines on a network that has ~10ms RTT. I guess it would be good to configure ceph so that it is less sensitive than the default and not too eager to mark an osd down. What is the default timeout for an osd to be marked down ?
[0:49] * AfC (~andrew@2407:7800:200:1011:2ad2:44ff:fe08:a4c) Quit (Quit: Leaving.)
[0:56] <loicd> it's 30seconds, it should not be a problem... something else is going on.
[0:58] <ron-slc> is there a method to disble the rgw log file? I have set debug rgw = 0/5 , this has helped much, but I still see 10's of Thousands of entries like: 2013-11-19 16:51:03.359734 7fe187fcf700 0 setting object write_tag=default.16188.76
[0:58] * Cube (~Cube@66-87-64-40.pools.spcsdns.net) Quit (Read error: Connection reset by peer)
[0:58] <pmatulis> ron-slc: must be another subsystem generating those
[0:59] <pmatulis> isn't there an 'object' subsystem?
[0:59] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[0:59] <pmatulis> hmm, great RFE: each log msg should be labelled according to the subsystem that generated it
[0:59] <ron-slc> I see: objectcacher, objecter, objclass
[1:00] <pmatulis> trial and error i guess
[1:00] <dmick> $ git grep 'setting object write_tag'
[1:00] <dmick> rgw/rgw_rados.cc: ldout(cct, 0) << "setting object write_tag=" << state->write_tag << dendl;
[1:00] <ron-slc> these are log entries appearing as per rgw config section "log file = /var/log/ceph/radosgw.log"
[1:01] <dmick> ldout(cct, 0) means "always" (unless logging is disabled)
[1:01] <dmick> perhaps that was an ill-advised logging level
[1:02] * Cube (~Cube@66-87-66-150.pools.spcsdns.net) has joined #ceph
[1:02] * AfC (~andrew@2407:7800:200:1011:2ad2:44ff:fe08:a4c) has joined #ceph
[1:02] <pmatulis> dmick: cool, thanks
[1:03] <dmick> even if you're not a programmer, searching the source can be a great thing
[1:04] <dmick> ron-slc: if it seems overdone, file a ticket to reduce the log level
[1:06] <ron-slc> OK! will do. Thanks for the guidance. I've been doing web searches, and ceph.com document grepping.. I'll start doing GIT searches, I always have the tree on my notebook.
[1:07] <dmick> np
[1:07] <dmick> tnx for the report
[1:08] * nwat (~textual@c-50-131-197-174.hsd1.ca.comcast.net) has joined #ceph
[1:08] * nhm (~nhm@wlan-rooms-4019.sc13.org) Quit (Ping timeout: 480 seconds)
[1:10] * dpippenger (~riven@66-192-9-78.static.twtelecom.net) has joined #ceph
[1:12] * KevinPerks (~Adium@cpe-066-026-252-218.triad.res.rr.com) Quit (Quit: Leaving.)
[1:16] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[1:18] <ron-slc> dmick: Issue #6804 created
[1:18] <kraken> ron-slc might be talking about: http://tracker.ceph.com/issues/6804 [Overly verbose logging: "setting object write_tag=" - rgw_rados.cc]
[1:21] * mozg (~andrei@host81-151-251-29.range81-151.btcentralplus.com) Quit (Ping timeout: 480 seconds)
[1:21] * scuttlemonkey (~scuttlemo@2601:1:bf00:dda:3434:a7ab:29c2:96c4) has joined #ceph
[1:21] * ChanServ sets mode +o scuttlemonkey
[1:25] * dpippenger (~riven@66-192-9-78.static.twtelecom.net) Quit (Quit: Leaving.)
[1:25] * dpippenger (~riven@66-192-9-78.static.twtelecom.net) has joined #ceph
[1:30] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[1:31] * Midnightmyth (~quassel@93-167-84-102-static.dk.customer.tdc.net) Quit (Ping timeout: 480 seconds)
[1:35] * andreask (~andreask@h081217067008.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[1:36] * mtanski (~mtanski@69.193.178.202) Quit (Read error: Operation timed out)
[1:41] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[1:42] * dpippenger (~riven@66-192-9-78.static.twtelecom.net) Quit (Remote host closed the connection)
[1:43] * KevinPerks (~Adium@cpe-066-026-252-218.triad.res.rr.com) has joined #ceph
[1:44] <jhujhiti> argh, i'm seeing the issue where osds don't go down again. i can't remove them even after shutting the box down
[1:50] <japuzzo> Not sure what this ceph-deploy error means "[ceph_deploy][ERROR ] ClientInitException:
[1:50] <japuzzo> [remote] sudo: sorry, you must have a tty to run sudo"
[1:51] <jhujhiti> japuzzo: Defaults !requiretty in your sudo config
[1:51] <japuzzo> My first two nodes are Ubuntu servers but node 3 & 4 are CentOS. Am I to assume that mixed cluster are not supported?
[1:51] <jhujhiti> it's just a difference in default sudo config
[1:52] <japuzzo> Oh
[1:52] <japuzzo> I just used the normal config will have to look thanks!
[1:53] * AfC (~andrew@2407:7800:200:1011:2ad2:44ff:fe08:a4c) Quit (Quit: Leaving.)
[1:56] * yy-nm (~Thunderbi@122.224.154.38) has joined #ceph
[1:56] <dmick> ron-slc: thanks
[1:56] * KevinPerks (~Adium@cpe-066-026-252-218.triad.res.rr.com) Quit (Ping timeout: 480 seconds)
[2:00] * scuttlemonkey (~scuttlemo@2601:1:bf00:dda:3434:a7ab:29c2:96c4) Quit (Read error: Connection reset by peer)
[2:00] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[2:00] * scuttlemonkey (~scuttlemo@2601:1:bf00:dda:3434:a7ab:29c2:96c4) has joined #ceph
[2:00] * ChanServ sets mode +o scuttlemonkey
[2:01] <jhujhiti> can i kick an osd out of the osd map manually somehow?
[2:01] <jhujhiti> i can't even mark it as lost because it thinks it's up
[2:04] * AfC (~andrew@2407:7800:200:1011:2ad2:44ff:fe08:a4c) has joined #ceph
[2:04] <dmick> maybe you want things like ceph osd crush rm and/or ceph osd rm
[2:04] <dmick> depends on what "kick out of the map" really means
[2:05] <jhujhiti> i marked it out, shut the process down, but the cluster still thinks it's up
[2:05] * xarses (~andreww@64-79-127-122.static.wiline.com) Quit (Ping timeout: 480 seconds)
[2:05] <jhujhiti> so i can't rm it
[2:05] <dmick> cluster will realize it's down after a bit
[2:05] <jhujhiti> what's "a bit"?
[2:05] <dmick> but you should be able to remove it from the crush map before it's down
[2:05] <jhujhiti> it's been like a hour
[2:05] <jhujhiti> an hour
[2:05] <dmick> shorter than that
[2:06] <dmick> pastebin to show your state?
[2:06] <jhujhiti> http://pastie.org/private/x6svyjm4swgceihhocufa
[2:07] <jhujhiti> 0/1/5/6 - all of them are acting the same
[2:07] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[2:07] <dmick> and those osd procs really don't exist?
[2:08] <jhujhiti> the box is powered off
[2:08] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) has joined #ceph
[2:08] <jhujhiti> they're orphaned in osd tree too:
[2:08] <jhujhiti> http://pastie.org/private/z3841n2ia7zn1hmixydua
[2:08] * haomaiwang (~haomaiwan@106.120.121.78) has joined #ceph
[2:10] <dmick> k, that means they're out of the crushmap
[2:11] <jhujhiti> yes, sorry i forgot to mention that. i removed them after marking them down and waiting for the rebalance
[2:11] <jhujhiti> marking them out rather
[2:11] <dmick> um, I don't know why they're still marked up; I could believe the grace for "I can't even contact the host" is longer than "I can't contact the proc on the host", although I don't know, but I wouldn't expect it to be an hour
[2:11] * glzhao (~glzhao@118.195.65.67) has joined #ceph
[2:12] <jhujhiti> oh there they go
[2:12] <jhujhiti> man that took a scarily long time
[2:12] <jhujhiti> e4004: 9 osds: 9 up, 9 in
[2:12] <jhujhiti> nice when things work
[2:13] * alfredodeza (~alfredode@c-24-131-46-23.hsd1.ga.comcast.net) Quit (Remote host closed the connection)
[2:13] <dmick> for future reference, you can actually mark them down too
[2:13] <dmick> (I'd forgotten)
[2:13] <dmick> ceph osd -h showed me that
[2:14] <jhujhiti> hmm i missed that
[2:14] <jhujhiti> next time
[2:15] * yanzheng (~zhyan@134.134.139.72) has joined #ceph
[2:19] * wenjianhn (~wenjianhn@123.118.215.163) has joined #ceph
[2:21] * Tamil1 (~Adium@cpe-76-168-18-224.socal.res.rr.com) Quit (Quit: Leaving.)
[2:23] * sarob_ (~sarob@ip-64-134-225-149.public.wayport.net) Quit (Remote host closed the connection)
[2:23] * sarob (~sarob@ip-64-134-225-149.public.wayport.net) has joined #ceph
[2:24] * linuxkidd (~linuxkidd@2607:f298:a:607:9eeb:e8ff:fe07:6658) Quit (Quit: Konversation terminated!)
[2:25] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Read error: Operation timed out)
[2:25] * nhm (~nhm@c-174-51-178-5.hsd1.co.comcast.net) has joined #ceph
[2:25] * ChanServ sets mode +o nhm
[2:27] * KevinPerks (~Adium@cpe-066-026-252-218.triad.res.rr.com) has joined #ceph
[2:27] * mtanski (~mtanski@cpe-74-65-252-48.nyc.res.rr.com) has joined #ceph
[2:28] * LeaChim (~LeaChim@host86-162-2-255.range86-162.btcentralplus.com) Quit (Ping timeout: 480 seconds)
[2:31] * sarob (~sarob@ip-64-134-225-149.public.wayport.net) Quit (Ping timeout: 480 seconds)
[2:38] <Anticimex> ceph getting picked up in mirantis openstack distribution
[2:38] <Anticimex> inktanks packaging i think it was
[2:38] * The_Bishop (~bishop@2001:470:50b6:0:2d3e:12a2:8f13:d188) Quit (Ping timeout: 480 seconds)
[2:38] <Anticimex> interesting
[2:38] <japuzzo> 'ceph-deploy install node3' where node3 is CentOS 6.4 is trying to install epel-release-6-8.noarch.rpm then fails
[2:41] * xarses (~andreww@c-24-23-183-44.hsd1.ca.comcast.net) has joined #ceph
[2:41] * eternaleye (~eternaley@c-24-17-202-252.hsd1.wa.comcast.net) Quit (Ping timeout: 480 seconds)
[2:41] <Anticimex> oh, this channel has certainly grown since 2006 as well :)
[2:41] * eternaleye_ (~eternaley@c-24-17-202-252.hsd1.wa.comcast.net) Quit (Ping timeout: 480 seconds)
[2:42] <pmatulis> Anticimex: the last time you were here was in 2006?
[2:42] * The_Bishop (~bishop@2001:470:50b6:0:2d3e:12a2:8f13:d188) has joined #ceph
[2:43] <Anticimex> no, but i joined then though
[2:43] <Anticimex> haven't seen /names for a while though, like a couple of years
[2:43] * eternaleye (~eternaley@c-24-17-202-252.hsd1.wa.comcast.net) has joined #ceph
[2:44] <Anticimex> is inktank officially custodians of ceph now?
[2:44] * Anticimex goes to read up a bit
[2:44] * eternaleye_ (~eternaley@c-24-17-202-252.hsd1.wa.comcast.net) has joined #ceph
[2:46] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[2:52] * mtanski (~mtanski@cpe-74-65-252-48.nyc.res.rr.com) Quit (Quit: mtanski)
[2:54] * nhm (~nhm@c-174-51-178-5.hsd1.co.comcast.net) Quit (Ping timeout: 480 seconds)
[2:59] <pmatulis> so changing the number of PGs is a no-go in Bobtail right?
[3:00] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[3:01] * shimo (~A13032@122x212x216x66.ap122.ftth.ucom.ne.jp) Quit (Quit: shimo)
[3:02] <dmick> pmatulis: yes
[3:02] <pmatulis> dmick: thanks
[3:03] * rongze (~rongze@117.79.232.218) has joined #ceph
[3:03] * mattrae (~oftc-webi@softbank126112054050.biz.bbtec.net) has joined #ceph
[3:04] <pmatulis> dmick: recommended only in Cuttlefish or Dumpling?
[3:04] * julian (~julian@125.70.134.54) has joined #ceph
[3:04] <mattrae> hi, ceph -w is giving a warning "osd.2 [WRN] slow request 60.251523 seconds old, recieved at ... currently waiting for pg to exist locally"
[3:04] <mattrae> i'm not finding much telling me what to do about that warning
[3:05] * japuzzo (~japuzzo@ool-4570886e.dyn.optonline.net) Quit (Quit: Leaving)
[3:05] <dmick> pmatulis: my memory is "not working in bobtail, experimental in cuttlefish, supported in dumpling" but I'm not sure
[3:05] <mattrae> i've restarted the osd in question
[3:05] * shimo (~A13032@122x212x216x66.ap122.ftth.ucom.ne.jp) has joined #ceph
[3:05] <pmatulis> hi mattrae :)
[3:05] <mattrae> hey pmatulis :D
[3:06] <mattrae> and julian :D
[3:08] <julian> mattrae, hey matt :)
[3:09] <mattrae> i guess we all are working on ceph today
[3:10] * angdraug (~angdraug@64-79-127-122.static.wiline.com) Quit (Quit: Leaving)
[3:11] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[3:12] <pmatulis> mattrae: you're not looking at Dumpling are you
[3:13] <julian> mattrae, nice, good to know you have workaround by smcroute
[3:19] * mtanski (~mtanski@cpe-74-65-252-48.nyc.res.rr.com) has joined #ceph
[3:21] <mattrae> pmatulis: nope we're still using .56
[3:21] <pmatulis> mattrae: ew
[3:25] <mattrae> julian: yeah we're getting closer to having it working.. but we're still having troulbe with multicast
[3:25] * dpippenger (~riven@cpe-76-166-208-83.socal.res.rr.com) has joined #ceph
[3:26] <julian> mattrae, no worry
[3:29] * sarob (~sarob@2601:9:7080:13a:710a:7533:51e1:6245) has joined #ceph
[3:30] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[3:32] * aliguori (~anthony@74.202.210.82) Quit (Remote host closed the connection)
[3:36] * yanzheng (~zhyan@134.134.139.72) Quit (Remote host closed the connection)
[3:36] * joao|lap (~JL@a79-168-11-205.cpe.netcabo.pt) Quit (Ping timeout: 480 seconds)
[3:37] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[3:37] * mattrae (~oftc-webi@softbank126112054050.biz.bbtec.net) Quit (Remote host closed the connection)
[3:40] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[3:40] * sarob (~sarob@2601:9:7080:13a:710a:7533:51e1:6245) Quit (Remote host closed the connection)
[3:41] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) has joined #ceph
[3:42] * sarob_ (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) has joined #ceph
[3:42] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) Quit (Read error: Connection reset by peer)
[3:45] * nwat (~textual@c-50-131-197-174.hsd1.ca.comcast.net) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[3:47] * sarob_ (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[3:48] * yy-nm (~Thunderbi@122.224.154.38) Quit (Quit: yy-nm)
[3:48] * sarob (~sarob@2601:9:7080:13a:e90f:e3c6:be04:d091) has joined #ceph
[3:49] * nwat (~textual@c-50-131-197-174.hsd1.ca.comcast.net) has joined #ceph
[3:50] * yanzheng (~zhyan@134.134.139.76) has joined #ceph
[3:56] * sarob (~sarob@2601:9:7080:13a:e90f:e3c6:be04:d091) Quit (Ping timeout: 480 seconds)
[4:00] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[4:01] * wschulze1 (~wschulze@cpe-72-229-37-201.nyc.res.rr.com) has joined #ceph
[4:01] * nwat (~textual@c-50-131-197-174.hsd1.ca.comcast.net) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[4:07] * wschulze (~wschulze@cpe-72-229-37-201.nyc.res.rr.com) Quit (Ping timeout: 480 seconds)
[4:07] * wschulze1 (~wschulze@cpe-72-229-37-201.nyc.res.rr.com) Quit (Read error: Connection reset by peer)
[4:08] * wschulze (~wschulze@cpe-72-229-37-201.nyc.res.rr.com) has joined #ceph
[4:13] * yanzheng (~zhyan@134.134.139.76) Quit (Remote host closed the connection)
[4:14] * mtanski (~mtanski@cpe-74-65-252-48.nyc.res.rr.com) Quit (Quit: mtanski)
[4:15] <aarontc> whee, I'm running into the OSD full problem now
[4:16] <aarontc> OSD stats: http://hastebin.com/cekawibixu.css
[4:16] <aarontc> how do I resolve the problem? I know several people have hit this - but I think I have sufficient pgs (over 100 per pool per spindle) and I've enabled the tunables
[4:16] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[4:17] * KevinPerks (~Adium@cpe-066-026-252-218.triad.res.rr.com) Quit (Quit: Leaving.)
[4:18] * haomaiwang (~haomaiwan@106.120.121.78) Quit (Remote host closed the connection)
[4:18] * haomaiwang (~haomaiwan@106.120.121.78) has joined #ceph
[4:20] * wschulze (~wschulze@cpe-72-229-37-201.nyc.res.rr.com) Quit (Read error: Operation timed out)
[4:20] * wschulze (~wschulze@cpe-72-229-37-201.nyc.res.rr.com) has joined #ceph
[4:23] <aarontc> I just ran ceph osd reweight-by-utilization and it seems to be moving a few pgs around
[4:26] * sileht (~sileht@gizmo.sileht.net) Quit (Ping timeout: 480 seconds)
[4:26] * haomaiwang (~haomaiwan@106.120.121.78) Quit (Ping timeout: 480 seconds)
[4:30] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[4:30] * RoddieKieley (~RoddieKie@47.55.80.53) Quit (Quit: Leaving.)
[4:30] * nwat (~textual@c-50-131-197-174.hsd1.ca.comcast.net) has joined #ceph
[4:37] * guerby (~guerby@ip165-ipv6.tetaneutral.net) Quit (Ping timeout: 480 seconds)
[4:39] * nwat (~textual@c-50-131-197-174.hsd1.ca.comcast.net) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[4:39] * yanzheng (~zhyan@134.134.139.76) has joined #ceph
[4:42] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[4:53] <dmick> aarontc: are your weights commensurate with the sizes of the OSDs?
[4:53] <aarontc> dmick: yes
[4:54] <aarontc> after the command I just mentioned, the tree is:
[4:54] <aarontc> http://hastebin.com/yeriyaxedi.md
[4:56] * nwat (~textual@c-50-131-197-174.hsd1.ca.comcast.net) has joined #ceph
[4:59] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[5:05] * fireD (~fireD@93-142-241-115.adsl.net.t-com.hr) has joined #ceph
[5:06] <dmick> seems plausible
[5:06] * fireD_ (~fireD@93-142-206-36.adsl.net.t-com.hr) Quit (Remote host closed the connection)
[5:07] * nwat (~textual@c-50-131-197-174.hsd1.ca.comcast.net) Quit (Quit: My MacBook has gone to sleep. ZZZzzz???)
[5:08] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[5:08] * diegows (~diegows@190.190.11.42) Quit (Ping timeout: 480 seconds)
[5:09] <pmatulis> where's uhura?
[5:09] * rongze (~rongze@117.79.232.218) Quit (Ping timeout: 480 seconds)
[5:09] * yanzheng (~zhyan@134.134.139.76) Quit (Remote host closed the connection)
[5:12] * rongze (~rongze@117.79.232.207) has joined #ceph
[5:13] * AndreyGrebennikov (~Andrey@91.207.132.67) Quit (Ping timeout: 480 seconds)
[5:13] * nwat (~textual@c-50-131-197-174.hsd1.ca.comcast.net) has joined #ceph
[5:15] * john_barbee (~jbarbee@c-98-193-1-69.hsd1.in.comcast.net) has joined #ceph
[5:15] <aarontc> pmatulis: uhura has other responsibilities ;)
[5:22] * nwat (~textual@c-50-131-197-174.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[5:24] <pmatulis> ookayy
[5:24] <aarontc> uhura is a router, since she's a comm officer
[5:24] * AndreyGrebennikov (~Andrey@91.207.132.67) has joined #ceph
[5:25] * nwat (~textual@c-50-131-197-174.hsd1.ca.comcast.net) has joined #ceph
[5:25] <pmatulis> gotcha
[5:25] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Read error: Operation timed out)
[5:25] * nwat (~textual@c-50-131-197-174.hsd1.ca.comcast.net) Quit ()
[5:25] <aarontc> not all the names make sense, but we tried on some of them
[5:32] * nhm (~nhm@mca2636d0.tmodns.net) has joined #ceph
[5:32] * ChanServ sets mode +o nhm
[5:36] * DarkAce-Z (~BillyMays@50.107.53.200) has joined #ceph
[5:39] * DarkAceZ (~BillyMays@50.107.53.200) Quit (Ping timeout: 480 seconds)
[5:39] * DarkAce-Z is now known as DarkAceZ
[5:40] * nwat (~textual@c-50-131-197-174.hsd1.ca.comcast.net) has joined #ceph
[5:45] * wschulze (~wschulze@cpe-72-229-37-201.nyc.res.rr.com) Quit (Quit: Leaving.)
[5:47] * nwat (~textual@c-50-131-197-174.hsd1.ca.comcast.net) Quit (Read error: Operation timed out)
[5:48] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[5:51] * BillK (~BillK-OFT@58-7-109-226.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[5:53] * wschulze (~wschulze@cpe-72-229-37-201.nyc.res.rr.com) has joined #ceph
[5:53] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[5:54] * BillK (~BillK-OFT@58-7-65-132.dyn.iinet.net.au) has joined #ceph
[6:00] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[6:01] * sarob (~sarob@2601:9:7080:13a:355b:cfa9:24ac:70c) has joined #ceph
[6:09] * sarob (~sarob@2601:9:7080:13a:355b:cfa9:24ac:70c) Quit (Ping timeout: 480 seconds)
[6:11] * rongze (~rongze@117.79.232.207) Quit (Quit: Leaving...)
[6:13] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[6:18] * john_barbee (~jbarbee@c-98-193-1-69.hsd1.in.comcast.net) Quit (Quit: ChatZilla 0.9.90.1 [Firefox 25.0/20131025151332])
[6:23] * BillK (~BillK-OFT@58-7-65-132.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[6:24] * BillK (~BillK-OFT@58-7-117-182.dyn.iinet.net.au) has joined #ceph
[6:28] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) has joined #ceph
[6:30] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[6:35] * Siva (~sivat@generalnat.eglbp.corp.yahoo.com) has joined #ceph
[6:39] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[6:41] * The_Bishop (~bishop@2001:470:50b6:0:2d3e:12a2:8f13:d188) Quit (Ping timeout: 480 seconds)
[6:44] * yy-nm (~Thunderbi@122.224.154.38) has joined #ceph
[6:44] * mattrae (~oftc-webi@softbank126112054050.biz.bbtec.net) has joined #ceph
[6:50] * The_Bishop (~bishop@2001:470:50b6:0:bc3e:3cad:c094:8397) has joined #ceph
[6:53] * mwarwick1 (~mwarwick@110-174-133-236.static.tpgi.com.au) has joined #ceph
[6:54] * mwarwick1 (~mwarwick@110-174-133-236.static.tpgi.com.au) Quit ()
[6:54] * mwarwick1 (~mwarwick@110-174-133-236.static.tpgi.com.au) has joined #ceph
[6:55] * nhm (~nhm@mca2636d0.tmodns.net) Quit (Ping timeout: 480 seconds)
[6:56] * mwarwick (~mwarwick@110-174-133-236.static.tpgi.com.au) Quit (Ping timeout: 480 seconds)
[6:59] * dpippenger (~riven@cpe-76-166-208-83.socal.res.rr.com) Quit (Remote host closed the connection)
[7:01] * noahmehl (~noahmehl@cpe-71-67-115-16.cinci.res.rr.com) Quit (Quit: noahmehl)
[7:02] * wschulze (~wschulze@cpe-72-229-37-201.nyc.res.rr.com) Quit (Quit: Leaving.)
[7:03] * The_Bishop (~bishop@2001:470:50b6:0:bc3e:3cad:c094:8397) Quit (Ping timeout: 480 seconds)
[7:04] * dpippenger (~riven@cpe-76-166-208-83.socal.res.rr.com) has joined #ceph
[7:11] * The_Bishop (~bishop@2001:470:50b6:0:2d3e:12a2:8f13:d188) has joined #ceph
[7:13] * mattrae (~oftc-webi@softbank126112054050.biz.bbtec.net) Quit (Remote host closed the connection)
[7:14] * Siva_ (~sivat@vpnnat.eglbp.corp.yahoo.com) has joined #ceph
[7:15] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[7:15] * sarob (~sarob@2601:9:7080:13a:21a5:fd40:c637:3552) has joined #ceph
[7:15] * sarob (~sarob@2601:9:7080:13a:21a5:fd40:c637:3552) Quit (Remote host closed the connection)
[7:16] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) has joined #ceph
[7:18] * yy-nm (~Thunderbi@122.224.154.38) Quit (Quit: yy-nm)
[7:19] * Siva (~sivat@generalnat.eglbp.corp.yahoo.com) Quit (Ping timeout: 480 seconds)
[7:19] * Siva_ is now known as Siva
[7:24] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[7:26] * AfC (~andrew@2407:7800:200:1011:2ad2:44ff:fe08:a4c) Quit (Ping timeout: 480 seconds)
[7:30] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[7:36] * JoeGruher (~JoeGruher@jfdmzpr04-ext.jf.intel.com) Quit (Remote host closed the connection)
[7:39] * mattt_ (~textual@cpc25-rdng20-2-0-cust162.15-3.cable.virginm.net) has joined #ceph
[7:40] * Siva_ (~sivat@vpnnat.eglbp.corp.yahoo.com) has joined #ceph
[7:41] * mattt_ (~textual@cpc25-rdng20-2-0-cust162.15-3.cable.virginm.net) Quit ()
[7:42] * Siva (~sivat@vpnnat.eglbp.corp.yahoo.com) Quit (Ping timeout: 480 seconds)
[7:42] * Siva_ is now known as Siva
[7:46] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) has joined #ceph
[7:47] * toabctl (~toabctl@toabctl.de) Quit (Quit: WeeChat 0.3.7)
[7:47] * mwarwick1 (~mwarwick@110-174-133-236.static.tpgi.com.au) Quit (Ping timeout: 480 seconds)
[7:48] * Cube (~Cube@66-87-66-150.pools.spcsdns.net) Quit (Quit: Leaving.)
[7:48] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[7:49] <bloodice> weird, i am running through my ceph cluster setup document that i created and after i ran the create monitor command, the ceph.log and ceph.conf files changed permissions blocking the ceph user from accessing them
[7:49] * toabctl (~toabctl@toabctl.de) has joined #ceph
[7:49] <bloodice> i adjusted the permissions and ran the mds creation again without issue, so maybe a bug there...
[7:50] <bloodice> lol
[7:50] <bloodice> ok so after the mds create command, it locked the permissions on the ceph.conf file again
[7:50] * rongze (~rongze@117.79.232.196) has joined #ceph
[7:58] * foosinn (~stefan@office.unitedcolo.de) has joined #ceph
[7:59] * odyssey4me (~odyssey4m@41.13.220.47) has joined #ceph
[8:00] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[8:06] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[8:07] * odyssey4me (~odyssey4m@41.13.220.47) Quit (Ping timeout: 480 seconds)
[8:10] <bloodice> anyone know the proper way to shutdown a ceph cluster? the sudo /etc/init.d/ceph -a stop doesnt do anything and it seems to be limited to on server, what if i have 4 ( 3 osd hosts and 1 monitor/admin host )?
[8:11] <Nats_> login to every host
[8:12] <bloodice> is there a particular order?
[8:12] <bloodice> monitors first.. osds second?
[8:13] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[8:14] * Cube (~Cube@12.248.40.138) has joined #ceph
[8:14] <bloodice> this doesnt work on ubuntu: http://eu.ceph.com/docs/wip-msgauth/init/stop-cluster/
[8:19] * scuttlemonkey (~scuttlemo@2601:1:bf00:dda:3434:a7ab:29c2:96c4) Quit (Ping timeout: 480 seconds)
[8:23] <bloodice> seems ceph-deploy installs and runs ceph as a process...
[8:25] <Nats_> osds first then monitors i'd say
[8:25] <Nats_> you might want to set 'noout'
[8:26] <bloodice> on each osd?
[8:28] * sleinen1 (~Adium@2001:620:0:26:d929:19bb:3d9c:d068) Quit (Quit: Leaving.)
[8:28] * sleinen (~Adium@77-58-245-10.dclient.hispeed.ch) has joined #ceph
[8:30] * Sysadmin88 (~IceChat77@94.1.37.151) Quit (Quit: There's nothing dirtier then a giant ball of oil)
[8:30] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[8:35] * topro (~prousa@host-62-245-142-50.customer.m-online.net) Quit (Ping timeout: 480 seconds)
[8:36] * sleinen (~Adium@77-58-245-10.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[8:36] <Nats_> noout is a cluster-wide property
[8:36] <Nats_> ceph osd set noout
[8:38] * topro (~prousa@host-62-245-142-50.customer.m-online.net) has joined #ceph
[8:39] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[8:51] * Siva (~sivat@vpnnat.eglbp.corp.yahoo.com) Quit (Quit: Siva)
[8:53] * guerby (~guerby@ip165-ipv6.tetaneutral.net) has joined #ceph
[8:54] * sileht (~sileht@gizmo.sileht.net) has joined #ceph
[8:58] * mattt_ (~textual@94.236.7.190) has joined #ceph
[8:59] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) has joined #ceph
[9:02] * AfC (~andrew@101.119.14.197) has joined #ceph
[9:04] * shang (~ShangWu@ppp-58-8-107-25.revip2.asianet.co.th) has joined #ceph
[9:07] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[9:07] * AfC (~andrew@101.119.14.197) Quit (Read error: Connection reset by peer)
[9:07] * sleinen (~Adium@2001:620:0:26:99a2:ffc8:722f:b77b) has joined #ceph
[9:12] * alex__ (~quassel@85.14.154.66) Quit (Quit: http://quassel-irc.org - Discuter simplement. Partout.)
[9:23] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[9:24] * mxmln (~mxmln@212.79.49.65) has joined #ceph
[9:30] * ScOut3R (~ScOut3R@catv-89-133-32-3.catv.broadband.hu) has joined #ceph
[9:30] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[9:38] * rendar (~s@host223-180-dynamic.56-79-r.retail.telecomitalia.it) has joined #ceph
[9:44] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[9:45] * haomaiwang (~haomaiwan@117.79.232.196) has joined #ceph
[9:47] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) has joined #ceph
[9:50] * andreask (~andreask@h081217067008.dyn.cm.kabsi.at) has joined #ceph
[9:50] * ChanServ sets mode +v andreask
[9:54] * Siva (~sivat@generalnat.eglbp.corp.yahoo.com) has joined #ceph
[9:55] * alaind (~dechorgna@161.105.182.35) has joined #ceph
[9:55] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Read error: Operation timed out)
[9:55] * Midnightmyth (~quassel@93-167-84-102-static.dk.customer.tdc.net) has joined #ceph
[9:55] * rongze (~rongze@117.79.232.196) Quit (Ping timeout: 480 seconds)
[9:56] * haomaiwang (~haomaiwan@117.79.232.196) Quit (Ping timeout: 480 seconds)
[9:59] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[10:00] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[10:03] * Siva (~sivat@generalnat.eglbp.corp.yahoo.com) Quit (Quit: Siva)
[10:09] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[10:14] * rongze (~rongze@117.79.232.206) has joined #ceph
[10:14] * haomaiwang (~haomaiwan@117.79.232.238) has joined #ceph
[10:15] * Siva (~sivat@generalnat.eglbp.corp.yahoo.com) has joined #ceph
[10:17] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[10:17] * ksingh (~Adium@2001:708:10:10:31f7:d299:cf46:60fd) has joined #ceph
[10:20] * Kioob`Taff (~plug-oliv@local.plusdinfo.com) has joined #ceph
[10:25] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[10:25] <lxo> emperor 0.72.1 init.d/ceph wouldn't start osds before I set ???osd crush update on start = 0??? in the config file; IIRC it had long printed an EINVAL error before actually starting the osd in earlier releases, but that error didn't stop the osd from starting
[10:26] <lxo> could this be because of my custom crushmap? the error hasn't been exactly enlightening :-(
[10:27] * rongze (~rongze@117.79.232.206) Quit (Ping timeout: 480 seconds)
[10:27] * jbd_ (~jbd_@2001:41d0:52:a00::77) has joined #ceph
[10:28] <lxo> the command that fails looks like this: ceph osd crush create-or-move <theosd#> 1.35 root=default host=<thehostname>
[10:28] <lxo> (I was running dumpling before the upgrade to 0.72.1; I skipped 0.72.0)
[10:30] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[10:32] <lxo> the full output is ???Error EINVAL: (22) Invalid argument???; it's not enlightening if you know about that command, but if all you get is that output and a failure to start the osd, it's quite distressing ;-)
[10:34] * Siva_ (~sivat@vpnnat.eglbp.corp.yahoo.com) has joined #ceph
[10:36] * TMM (~hp@sams-office-nat.tomtomgroup.com) has joined #ceph
[10:36] * rongze (~rongze@14.18.203.18) has joined #ceph
[10:36] * Siva (~sivat@generalnat.eglbp.corp.yahoo.com) Quit (Ping timeout: 480 seconds)
[10:36] * Siva_ is now known as Siva
[10:41] * Siva (~sivat@vpnnat.eglbp.corp.yahoo.com) Quit (Quit: Siva)
[10:41] * xdeller (~xdeller@91.218.144.129) has joined #ceph
[10:43] * LeaChim (~LeaChim@host86-162-2-255.range86-162.btcentralplus.com) has joined #ceph
[10:47] * sarob (~sarob@2601:9:7080:13a:881d:d904:f910:dc00) has joined #ceph
[10:49] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[10:51] * alaind (~dechorgna@161.105.182.35) Quit (Ping timeout: 480 seconds)
[10:51] * odyssey4me (~odyssey4m@165.233.205.190) has joined #ceph
[10:52] * odyssey4me (~odyssey4m@165.233.205.190) Quit (Read error: Connection reset by peer)
[10:55] * sarob (~sarob@2601:9:7080:13a:881d:d904:f910:dc00) Quit (Ping timeout: 480 seconds)
[11:00] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[11:00] * Siva (~sivat@vpnnat.eglbp.corp.yahoo.com) has joined #ceph
[11:04] * rongze (~rongze@14.18.203.18) Quit (Remote host closed the connection)
[11:05] * xdeller (~xdeller@91.218.144.129) Quit (Quit: Leaving)
[11:05] * xdeller (~xdeller@91.218.144.129) has joined #ceph
[11:09] * joao (~joao@a79-168-11-205.cpe.netcabo.pt) Quit (Ping timeout: 480 seconds)
[11:12] * Siva (~sivat@vpnnat.eglbp.corp.yahoo.com) Quit (Quit: Siva)
[11:13] * Siva (~sivat@vpnnat.eglbp.corp.yahoo.com) has joined #ceph
[11:13] * andreask (~andreask@h081217067008.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[11:13] * andreask (~andreask@zid-vpnn097.uibk.ac.at) has joined #ceph
[11:13] * ChanServ sets mode +v andreask
[11:14] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[11:30] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[11:34] <leseb> joshd: around?
[11:34] * rongze (~rongze@117.79.232.205) has joined #ceph
[11:39] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[11:46] * rongze (~rongze@117.79.232.205) Quit (Ping timeout: 480 seconds)
[11:47] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) has joined #ceph
[11:52] * diegows (~diegows@190.190.11.42) has joined #ceph
[11:53] * yanzheng (~zhyan@134.134.137.71) has joined #ceph
[11:54] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[11:55] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[11:58] <diegows> does anyone know a test suite to test ceph, specially reliability? something that writes random files, with random size and verifies checksums
[12:00] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[12:05] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[12:09] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Quit: Leaving.)
[12:21] * allsystemsarego (~allsystem@5-12-240-115.residential.rdsnet.ro) has joined #ceph
[12:23] * Cube (~Cube@12.248.40.138) Quit (Ping timeout: 480 seconds)
[12:31] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[12:37] * i_m (~ivan.miro@deibp9eh1--blueice1n2.emea.ibm.com) has joined #ceph
[12:38] * andreask (~andreask@zid-vpnn097.uibk.ac.at) Quit (Read error: Operation timed out)
[12:39] * rongze (~rongze@117.79.232.205) has joined #ceph
[12:41] * julian (~julian@125.70.134.54) Quit (Quit: Leaving)
[12:43] * rongze_ (~rongze@117.79.232.237) has joined #ceph
[12:44] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[12:44] * rongze (~rongze@117.79.232.205) Quit (Read error: Connection reset by peer)
[12:45] * Cube (~Cube@66-87-66-150.pools.spcsdns.net) has joined #ceph
[12:47] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) has joined #ceph
[12:49] * BManojlovic (~steki@91.195.39.5) Quit (Quit: Ja odoh a vi sta 'ocete...)
[12:49] * BillK (~BillK-OFT@58-7-117-182.dyn.iinet.net.au) Quit (Read error: Connection reset by peer)
[12:49] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[12:57] * BillK (~BillK-OFT@124-169-199-250.dyn.iinet.net.au) has joined #ceph
[12:59] * BManojlovic (~steki@91.195.39.5) Quit (Remote host closed the connection)
[13:01] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[13:02] * joao (~joao@a79-168-11-205.cpe.netcabo.pt) has joined #ceph
[13:02] * ChanServ sets mode +o joao
[13:02] * andreask (~andreask@h081217067008.dyn.cm.kabsi.at) has joined #ceph
[13:02] * ChanServ sets mode +v andreask
[13:09] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[13:09] * wenjianhn (~wenjianhn@123.118.215.163) Quit (Read error: Connection reset by peer)
[13:12] * shang (~ShangWu@ppp-58-8-107-25.revip2.asianet.co.th) Quit (Remote host closed the connection)
[13:12] * Hakisho (~Hakisho@0001be3c.user.oftc.net) has joined #ceph
[13:14] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[13:18] * Siva (~sivat@vpnnat.eglbp.corp.yahoo.com) Quit (Quit: Siva)
[13:18] * todin (tuxadero@kudu.in-berlin.de) has joined #ceph
[13:18] * Cube (~Cube@66-87-66-150.pools.spcsdns.net) Quit (Quit: Leaving.)
[13:18] * andreask (~andreask@h081217067008.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[13:19] * andreask (~andreask@zid-vpnn061.uibk.ac.at) has joined #ceph
[13:19] * ChanServ sets mode +v andreask
[13:23] * wenjianhn (~wenjianhn@111.196.86.27) has joined #ceph
[13:24] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[13:29] * dmsimard (~Adium@108.163.152.2) has joined #ceph
[13:31] * Shmouel (~Sam@fny94-12-83-157-27-95.fbx.proxad.net) has joined #ceph
[13:31] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[13:32] * sarob (~sarob@2601:9:7080:13a:59e3:26dc:31f7:8595) has joined #ceph
[13:33] * tsnider (~tsnider@nat-216-240-30-23.netapp.com) has joined #ceph
[13:33] * tsnider (~tsnider@nat-216-240-30-23.netapp.com) has left #ceph
[13:39] * morse (~morse@supercomputing.univpm.it) Quit (Remote host closed the connection)
[13:40] * Siva (~sivat@generalnat.eglbp.corp.yahoo.com) has joined #ceph
[13:40] * Hakisho (~Hakisho@0001be3c.user.oftc.net) Quit (Quit: bye)
[13:40] * sarob (~sarob@2601:9:7080:13a:59e3:26dc:31f7:8595) Quit (Ping timeout: 480 seconds)
[13:43] * Hakisho (~Hakisho@0001be3c.user.oftc.net) has joined #ceph
[13:44] * morse (~morse@supercomputing.univpm.it) has joined #ceph
[13:47] * sarob (~sarob@2601:9:7080:13a:5d56:af74:8375:f332) has joined #ceph
[13:47] * AfC (~andrew@101.119.15.197) has joined #ceph
[13:49] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[13:53] * Shmouel (~Sam@fny94-12-83-157-27-95.fbx.proxad.net) Quit (Quit: Leaving.)
[13:53] * Shmouel (~Sam@fny94-12-83-157-27-95.fbx.proxad.net) has joined #ceph
[13:55] * sarob (~sarob@2601:9:7080:13a:5d56:af74:8375:f332) Quit (Ping timeout: 480 seconds)
[13:56] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Read error: Operation timed out)
[13:58] * mtanski (~mtanski@cpe-74-65-252-48.nyc.res.rr.com) has joined #ceph
[14:02] * aardvark1 (~Warren@2607:f298:a:607:a8d9:5660:cbb1:e25d) has joined #ceph
[14:07] * Shmouel (~Sam@fny94-12-83-157-27-95.fbx.proxad.net) Quit (Quit: Leaving.)
[14:07] * ScOut3R_ (~ScOut3R@catv-89-133-32-3.catv.broadband.hu) has joined #ceph
[14:07] * Shmouel (~Sam@fny94-12-83-157-27-95.fbx.proxad.net) has joined #ceph
[14:09] * WarrenUsui (~Warren@2607:f298:a:607:a8d9:5660:cbb1:e25d) Quit (Ping timeout: 480 seconds)
[14:09] * wusui1 (~Warren@2607:f298:a:607:a8d9:5660:cbb1:e25d) Quit (Ping timeout: 480 seconds)
[14:09] * WarrenUsui (~Warren@2607:f298:a:607:a8d9:5660:cbb1:e25d) has joined #ceph
[14:13] * ScOut3R (~ScOut3R@catv-89-133-32-3.catv.broadband.hu) Quit (Ping timeout: 480 seconds)
[14:15] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[14:17] * peetaur (~peter@CPE788df73fb301-CM788df73fb300.cpe.net.cable.rogers.com) Quit (Ping timeout: 480 seconds)
[14:17] * tsnider (~tsnider@nat-216-240-30-23.netapp.com) has joined #ceph
[14:17] * alfredodeza (~alfredode@c-24-131-46-23.hsd1.ga.comcast.net) has joined #ceph
[14:18] * mschiff (~mschiff@tmo-101-105.customers.d1-online.com) has joined #ceph
[14:19] * Shmouel (~Sam@fny94-12-83-157-27-95.fbx.proxad.net) Quit (Quit: Leaving.)
[14:19] * BillK (~BillK-OFT@124-169-199-250.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[14:20] * BillK (~BillK-OFT@58-7-117-112.dyn.iinet.net.au) has joined #ceph
[14:23] * AfC (~andrew@101.119.15.197) Quit (Ping timeout: 480 seconds)
[14:23] * KevinPerks (~Adium@cpe-066-026-252-218.triad.res.rr.com) has joined #ceph
[14:28] * yanzheng (~zhyan@134.134.137.71) Quit (Remote host closed the connection)
[14:29] * foosinn (~stefan@office.unitedcolo.de) Quit (Read error: Connection reset by peer)
[14:29] * mtanski (~mtanski@cpe-74-65-252-48.nyc.res.rr.com) Quit (Quit: mtanski)
[14:31] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[14:37] * Siva (~sivat@generalnat.eglbp.corp.yahoo.com) Quit (Quit: Siva)
[14:40] * mtanski (~mtanski@cpe-74-65-252-48.nyc.res.rr.com) has joined #ceph
[14:40] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[14:41] * nhm (~nhm@m812636d0.tmodns.net) has joined #ceph
[14:41] * ChanServ sets mode +o nhm
[14:46] * sleinen (~Adium@2001:620:0:26:99a2:ffc8:722f:b77b) Quit (Quit: Leaving.)
[14:46] * sleinen (~Adium@130.59.94.247) has joined #ceph
[14:46] * zoltan (~zoltan@2001:620:20:16:b565:9e49:5de0:2afd) has joined #ceph
[14:46] <zoltan> hi
[14:47] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) has joined #ceph
[14:47] <zoltan> so originally I deployed my cluster with "osd_crush_chooseleaf_type = 0"; is there a way to migrate now to place the copies on different hosts?
[14:47] <zoltan> I've found this blog post: http://jcftang.github.io/2012/09/06/going-from-replicating-across-osds-to-replicating-across-hosts-in-a-ceph-cluster/
[14:48] * Midnightmyth (~quassel@93-167-84-102-static.dk.customer.tdc.net) Quit (Ping timeout: 480 seconds)
[14:48] <zoltan> so should I dump the crushmap, edit it, compile it and then set it as per the blog post?
[14:50] <andreask> hmm .. I think it should be possible by a command ... let me check
[14:52] <zoltan> osd crush chooseleaf type 1? :)
[14:52] <andreask> yeah ;-) ... but I assume you want to set it live?
[14:52] <zoltan> yes
[14:52] <zoltan> can I do that?
[14:52] <zoltan> it's currently in-use by a few VMs :)
[14:52] * rendar (~s@host223-180-dynamic.56-79-r.retail.telecomitalia.it) Quit (Ping timeout: 480 seconds)
[14:54] * markbby (~Adium@168.94.245.2) has joined #ceph
[14:54] * sleinen (~Adium@130.59.94.247) Quit (Ping timeout: 480 seconds)
[14:58] * sleinen (~Adium@130.59.94.247) has joined #ceph
[14:59] * sleinen1 (~Adium@2001:620:0:26:502e:7390:e74a:25a6) has joined #ceph
[15:00] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[15:01] * xcrracer (~xcrracer@fw-ext-v-1.kvcc.edu) has joined #ceph
[15:03] * ScOut3R (~ScOut3R@212.96.46.212) has joined #ceph
[15:05] * ScOut3R_ (~ScOut3R@catv-89-133-32-3.catv.broadband.hu) Quit (Ping timeout: 480 seconds)
[15:06] * sleinen (~Adium@130.59.94.247) Quit (Ping timeout: 480 seconds)
[15:09] * mtanski (~mtanski@cpe-74-65-252-48.nyc.res.rr.com) Quit (Quit: mtanski)
[15:10] <zoltan> andreask, ping?:)
[15:11] <andreask> zoltan: sorry ... customer issue ... looks like editing the chrushmap is your option
[15:12] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[15:12] * wenjianhn (~wenjianhn@111.196.86.27) Quit (Ping timeout: 480 seconds)
[15:17] <zoltan> can't set this via the cli?
[15:18] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[15:20] * wschulze (~wschulze@cpe-72-229-37-201.nyc.res.rr.com) has joined #ceph
[15:21] <andreask> I have not found a command for that
[15:24] <andreask> you can change your default via cli
[15:24] <andreask> ... and in the configuration
[15:25] * clayb (~kvirc@proxy-nj2.bloomberg.com) has joined #ceph
[15:26] <andreask> zoltan: ^^^^
[15:26] <zoltan> interestingly in the decoded crush map this is what I see:
[15:26] <zoltan> step choose firstn 0 type osd
[15:27] <zoltan> I should replace this with
[15:27] <zoltan> step chooseleaf firstn 0 type host
[15:27] <zoltan> right?
[15:27] <kraken> http://i.imgur.com/RvquHs0.gif
[15:27] <andreask> well ... you set it to 0, no
[15:27] <andreask> ?
[15:27] <zoltan> from a git commit message:
[15:27] <zoltan> Add 'osd crush chooseleaf type' option to control what the default
[15:27] <zoltan> CRUSH rule separates replicas across. Default to 1 (host), and set it
[15:27] <zoltan> to 0 in vstart.sh.
[15:27] <andreask> a sorry .. overlooked
[15:27] <zoltan> hmm
[15:28] <andreask> yes .... set it to host
[15:29] * mschiff (~mschiff@tmo-101-105.customers.d1-online.com) Quit (Read error: Connection reset by peer)
[15:29] <zoltan> what I dont understand is why the default isn't "chooseleaf" but "choose"
[15:29] * mschiff (~mschiff@tmo-101-105.customers.d1-online.com) has joined #ceph
[15:30] * BillK (~BillK-OFT@58-7-117-112.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[15:31] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[15:32] <andreask> hmmm
[15:37] * mozg (~andrei@host217-46-236-49.in-addr.btopenworld.com) has joined #ceph
[15:38] * mozg (~andrei@host217-46-236-49.in-addr.btopenworld.com) Quit ()
[15:38] * mozg (~andrei@host217-46-236-49.in-addr.btopenworld.com) has joined #ceph
[15:39] <andreask> zoltan: ah... you can't use chooseleaf for devices ... then it has to be choose
[15:39] * mozg (~andrei@host217-46-236-49.in-addr.btopenworld.com) Quit (Remote host closed the connection)
[15:40] * mozg (~andrei@host217-46-236-49.in-addr.btopenworld.com) has joined #ceph
[15:40] <mozg> hello guys
[15:40] <andreask> zoltan: an osd is already a leaf
[15:40] <mozg> has anyone here configured radosgw service on more than two servers?
[15:41] <mozg> something like to have a failover solution
[15:41] <mozg> so that i can can take one of the radosgw servers down for maintenance and still have access to the service?
[15:41] <andreask> mozg: you mean with an ha-loadbalancer in front?
[15:42] <mozg> andreask, yeah, something like that
[15:42] <mozg> at the moment i have a single instance of radosgw running on one of the servers
[15:42] <mozg> i would like to have at least two instances running on two servers
[15:42] <mozg> and balance requests across them
[15:43] <mozg> can this be done?
[15:43] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[15:43] <ksingh> hello Andreask , i have a query regarding block device and openstack , i read ceph docmuemtation and it says that To use Ceph Block Devices with OpenStack, you must install QEMU, libvirt, and OpenStack first. My question is ??? where do i need to install QEMU and LIBVIRT on my ceph monitor nodes or on my openstack machine
[15:43] <andreask> mozg: sure ... put a loadbalancer like ha-proxy in front of them
[15:44] <ksingh> i am bit confused
[15:44] <andreask> ksingh: on the openstack machines ... on your compute nodes
[15:44] <mozg> andreask, so, in terms of the configuration of the second radosgw, should I do it exactly like i've done with the first one?
[15:44] <mozg> can they use the same buckets, etc?
[15:44] <ksingh> alright thanks :-)
[15:45] <andreask> mozg: yes
[15:46] <mozg> andreask, and in terms of the load balancing policies, what is the best one to pick?
[15:46] <mozg> would a simple round robin do the job
[15:46] <mozg> or should it be session aware?
[15:47] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) has joined #ceph
[15:50] <andreask> mozg: round robin is fine
[15:55] * yanzheng (~zhyan@jfdmzpr04-ext.jf.intel.com) has joined #ceph
[16:01] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[16:03] * japuzzo (~japuzzo@pok2.bluebird.ibm.com) has joined #ceph
[16:08] * yanzheng (~zhyan@jfdmzpr04-ext.jf.intel.com) Quit (Remote host closed the connection)
[16:08] * scuttlemonkey (~scuttlemo@c-174-51-178-5.hsd1.co.comcast.net) has joined #ceph
[16:08] * ChanServ sets mode +o scuttlemonkey
[16:09] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[16:09] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[16:15] * peetaur (~peter@CPE788df73fb301-CM788df73fb300.cpe.net.cable.rogers.com) has joined #ceph
[16:15] <mozg> andreask, thanks!
[16:16] <andreask> yw mosg
[16:16] <andreask> sorry ;-) mozg
[16:19] <zoltan> andreask, hmm
[16:20] <zoltan> andreask, but I can use chooseleaf for host?
[16:20] <andreask> zoltan, yes ... you want chooseleaf for host
[16:20] * Siva (~sivat@117.192.38.118) has joined #ceph
[16:20] <zoltan> ok, so the procedure described in the blog post I linked earlier is correct; dump it, modify it, compile it and then upload
[16:20] <zoltan> and it should rebalance :)
[16:21] <andreask> yes
[16:21] * i_m (~ivan.miro@deibp9eh1--blueice1n2.emea.ibm.com) Quit (Ping timeout: 480 seconds)
[16:25] <zoltan> should I feel confident enough to upgrade my dumpling cluster to emperor?
[16:25] <zoltan> I'm CPU-bound basically now
[16:26] * Siva_ (~sivat@117.192.58.68) has joined #ceph
[16:27] <andreask> if your cluster is a productive system I would wait
[16:28] * Siva (~sivat@117.192.38.118) Quit (Ping timeout: 480 seconds)
[16:28] * Siva_ is now known as Siva
[16:28] <zoltan> it is
[16:30] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[16:31] * shang (~ShangWu@119-46-166-36.static.asianet.co.th) has joined #ceph
[16:32] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) has joined #ceph
[16:33] * Siva_ (~sivat@vpnnat.eglbp.corp.yahoo.com) has joined #ceph
[16:33] <zoltan> 2013-11-20 15:33:08.760129 mon.0 [INF] pgmap v810531: 2128 pgs: 1573 active+clean, 357 active+remapped+wait_backfill, 80 active+recovery_wait, 52 active+remapped+backfilling, 66 active+recovering; 719 GB data, 1439 GB used, 5694 GB / 7134 GB avail; 230KB/s rd, 521KB/s wr, 59op/s; 70621/396048 degraded (17.831%); recovering 297 o/s, 1643MB/s
[16:33] <zoltan> started working :)
[16:38] * Siva (~sivat@117.192.58.68) Quit (Ping timeout: 480 seconds)
[16:38] * Siva_ is now known as Siva
[16:40] <tsnider> pamtulis: u there??
[16:41] * joao|lap (~JL@a79-168-11-205.cpe.netcabo.pt) has joined #ceph
[16:41] * ChanServ sets mode +o joao|lap
[16:43] <tsnider> pmatulis: FYI: WRT my question Sun about rbd mapping -- A flag was disabled which prevented kernel mounted images with older kernels. From Inktank support: "The feature I am thinking of is the hashpspool feature on the pool you created. This is default since V0.64 and is only supported by kernels 3.9 and higher. dmesg will show if this is indeed the error you are getting. You can disable this by setting this in your ceph.conf and creating
[16:47] <pmatulis> tsnider: you got cut off (and creating...)
[16:47] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) has joined #ceph
[16:47] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[16:47] <pmatulis> tsnider: i didn't realize you put in a special feature (hashpool) for your customer pool
[16:47] <pmatulis> *custom
[16:51] * shang (~ShangWu@119-46-166-36.static.asianet.co.th) Quit (Ping timeout: 480 seconds)
[16:56] * gucki (~smuxi@77-56-39-154.dclient.hispeed.ch) has joined #ceph
[16:58] <tsnider> pmatulis: Remainder of inktank response: ..."You can disable this by setting this in your ceph.conf and creating a new pool. osd pool default flag hashpspool = false ". I was creating a pool -- and then an image in the pool --- not doing anything custom (that I'm aware of).
[16:59] <pmatulis> tsnider: good info, thanks for reporting back
[16:59] * diegows (~diegows@190.190.11.42) Quit (Ping timeout: 480 seconds)
[17:00] <tsnider> pmatulis: np --
[17:00] <zoltan> andreask, thanks for the tips, it seems to be working fine. :)
[17:00] <zoltan> see ya later guys
[17:00] * zoltan (~zoltan@2001:620:20:16:b565:9e49:5de0:2afd) Quit (Quit: Leaving)
[17:01] * shang (~ShangWu@119-46-166-36.static.asianet.co.th) has joined #ceph
[17:01] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[17:01] * sarob (~sarob@c-50-161-65-119.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[17:04] <mozg> andreask, one more question about the radosgw if you don't mind
[17:04] <mozg> i see that it created a bunch of pools like it should
[17:04] <mozg> i guess by default it didn't use optimised number of pgs on those pools
[17:04] <mozg> do you know which of those pools contain the actual data so that i can increase the number of pgs on that particular pool
[17:06] <pmatulis> rados -p <pool> ls
[17:08] <andreask> mozg: I think the one with "bucket" in their name
[17:08] * gucki_ (~smuxi@77-56-39-154.dclient.hispeed.ch) has joined #ceph
[17:08] <gucki> hi guys
[17:08] <andreask> ... but I am not sure
[17:08] <gucki> http://tracker.ceph.com/issues/6810
[17:08] <mozg> andreask, so, should I leave the rest of pools with default pgs?
[17:08] <gucki> anybody observed the same?
[17:09] <mozg> these are the pools that I currently have: .rgw.root,12 .rgw.control,13 .rgw,14 .rgw.gc,15 .users.uid,16 .users,17 .users.swift,18 .rgw.buckets.index,19 .rgw.buckets,21 ,
[17:09] <mozg> i guess i need to change the pgs in the .rgw.buckets pool, right?
[17:11] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[17:12] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[17:12] * mtanski (~mtanski@69.193.178.202) has joined #ceph
[17:13] * julian (~julianwa@125.70.134.54) has joined #ceph
[17:14] <andreask> mozg: it does not hurt to change all if the radosgw has not been in use
[17:16] * sleinen1 (~Adium@2001:620:0:26:502e:7390:e74a:25a6) Quit (Quit: Leaving.)
[17:16] * sleinen (~Adium@130.59.94.247) has joined #ceph
[17:22] * sjustlaptop (~sam@24-205-35-233.dhcp.gldl.ca.charter.com) has joined #ceph
[17:23] * andreask (~andreask@zid-vpnn061.uibk.ac.at) Quit (Ping timeout: 480 seconds)
[17:24] * sleinen (~Adium@130.59.94.247) Quit (Ping timeout: 480 seconds)
[17:24] * nhm (~nhm@m812636d0.tmodns.net) Quit (Ping timeout: 480 seconds)
[17:26] * ScOut3R (~ScOut3R@212.96.46.212) Quit (Ping timeout: 480 seconds)
[17:32] * rendar (~s@host223-180-dynamic.56-79-r.retail.telecomitalia.it) has joined #ceph
[17:33] <jcsp> when I increase the PGs in a pool with "ceph osd pool set pg_num", is there any way to get pgp_num set at the same time, without waiting for all the PGs to be created?
[17:34] * shang (~ShangWu@119-46-166-36.static.asianet.co.th) Quit (Quit: Ex-Chat)
[17:35] <pmatulis> jcsp: you need to set it as well
[17:37] <jcsp> the "without waiting" part is what I'm concerned with. The procedure as it stands risks a forgetful admin leaving off the last part (or in my case, makes a script more complex)
[17:38] <jcsp> currently you get the "Error EAGAIN: currently creating pgs, wait" if you try and set them both one after the other
[17:39] <pmatulis> jcsp: oh, you don't want to wait, how may PGs do you currently have?
[17:39] <pmatulis> *many
[17:39] <jcsp> no particular number: I'm scripting the general case
[17:39] <jcsp> so working on the assumption that PG creation can take either a very short or very long time
[17:40] * Siva (~sivat@vpnnat.eglbp.corp.yahoo.com) Quit (Ping timeout: 480 seconds)
[17:41] <pmatulis> jcsp: dunno, i guess you do need to build some intelligence into your script (ex: looping through 'ceph pg stat')
[17:41] <jcsp> If I have to write code to wait for pg creation before setting pgp_num then I can, was just hoping there was a trick??? lazy me :-)
[17:41] * rongze_ (~rongze@117.79.232.237) Quit (Remote host closed the connection)
[17:42] * ksingh (~Adium@2001:708:10:10:31f7:d299:cf46:60fd) has left #ceph
[17:44] * TMM (~hp@sams-office-nat.tomtomgroup.com) Quit (Quit: Ex-Chat)
[17:45] * BManojlovic (~steki@91.195.39.5) Quit (Quit: Ja odoh a vi sta 'ocete...)
[17:47] * sarob (~sarob@2601:9:7080:13a:5d35:e423:3b67:3190) has joined #ceph
[17:50] * mozg (~andrei@host217-46-236-49.in-addr.btopenworld.com) Quit (Quit: Ex-Chat)
[17:52] * nhm (~nhm@wlan-rooms-4019.sc13.org) has joined #ceph
[17:52] * ChanServ sets mode +o nhm
[17:53] * glzhao (~glzhao@118.195.65.67) Quit (Quit: leaving)
[17:54] * sleinen (~Adium@77-58-245-10.dclient.hispeed.ch) has joined #ceph
[17:55] * sleinen1 (~Adium@2001:620:0:25:65c7:86ad:eadb:1f5c) has joined #ceph
[17:56] * sarob (~sarob@2601:9:7080:13a:5d35:e423:3b67:3190) Quit (Ping timeout: 480 seconds)
[17:56] * mschiff (~mschiff@tmo-101-105.customers.d1-online.com) Quit (Remote host closed the connection)
[18:02] * sleinen (~Adium@77-58-245-10.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[18:02] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[18:03] * sleinen1 (~Adium@2001:620:0:25:65c7:86ad:eadb:1f5c) Quit (Quit: Leaving.)
[18:03] <L2SHO> what does the reweight column mean here? http://apaste.info/gPHV
[18:04] * xarses (~andreww@c-24-23-183-44.hsd1.ca.comcast.net) Quit (Read error: Operation timed out)
[18:04] * mattt_ (~textual@94.236.7.190) Quit (Quit: Computer has gone to sleep.)
[18:04] * mozg (~andrei@host217-46-236-49.in-addr.btopenworld.com) has joined #ceph
[18:05] * aliguori (~anthony@74.202.210.82) has joined #ceph
[18:07] * nhm (~nhm@wlan-rooms-4019.sc13.org) Quit (Ping timeout: 480 seconds)
[18:08] <mozg> anyone knows if radosgw with S3 api could be used behind a proxy server? If so, do I need to make any changes to the proxy server settings to make sure it works?
[18:08] * gucki (~smuxi@77-56-39-154.dclient.hispeed.ch) Quit (Remote host closed the connection)
[18:08] * gucki_ (~smuxi@77-56-39-154.dclient.hispeed.ch) Quit (Remote host closed the connection)
[18:10] * mxmln (~mxmln@212.79.49.65) Quit (Quit: mxmln)
[18:15] * Siva (~sivat@vpnnat.eglbp.corp.yahoo.com) has joined #ceph
[18:17] * Underbyte (~jerrad@pat-global.macpractice.net) Quit (Ping timeout: 480 seconds)
[18:18] * Sysadmin88 (~IceChat77@94.1.37.151) has joined #ceph
[18:18] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[18:20] * xdeller (~xdeller@91.218.144.129) Quit (Quit: Leaving)
[18:21] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[18:21] * mschiff (~mschiff@85.182.236.82) has joined #ceph
[18:22] * angdraug (~angdraug@64-79-127-122.static.wiline.com) has joined #ceph
[18:23] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[18:25] * sleinen (~Adium@77-58-245-10.dclient.hispeed.ch) has joined #ceph
[18:26] * sleinen1 (~Adium@2001:620:0:26:195b:1aa6:278c:594b) has joined #ceph
[18:27] * Kioob`Taff (~plug-oliv@local.plusdinfo.com) Quit (Ping timeout: 480 seconds)
[18:29] <Pauline> Bleah. deleted pools stay behind in the "ceph pg dump overview." and showing in "ceph -s", anybody else noticed? (0.72.1)
[18:31] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[18:32] * xarses (~andreww@64-79-127-122.static.wiline.com) has joined #ceph
[18:33] * sleinen (~Adium@77-58-245-10.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[18:35] * mattch (~mattch@pcw3047.see.ed.ac.uk) Quit (Quit: Leaving.)
[18:38] <joao|lap> Pauline, that should be refreshed after a while iirc
[18:39] * sleinen1 (~Adium@2001:620:0:26:195b:1aa6:278c:594b) Quit (Quit: Leaving.)
[18:39] <joao|lap> let me know if they persist forever though
[18:41] * ircolle (~Adium@2601:1:8380:2d9:e407:98b7:92a:669) has joined #ceph
[18:43] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[18:44] * JoeGruher (~JoeGruher@134.134.139.72) has joined #ceph
[18:44] <mozg> hello guys
[18:44] <mozg> i've just increased a number of PGs in a pool from 100 to 500 and i've noticed the data usage jumpted up by a significant margin
[18:44] <mozg> is this normal?
[18:45] <mozg> i am using rados df to check how much is being used
[18:45] <mozg> it jumpted from 52985267KB to 3311426639KB
[18:47] <mozg> so, it jumpted by like 3TB if I am not miscalculating
[18:47] <mozg> and there was no actual data change during this time
[18:47] * sjusthm (~sam@24-205-35-233.dhcp.gldl.ca.charter.com) has joined #ceph
[18:47] <L2SHO> mozg, that should disappear after the backfilling is done and you do a scrub
[18:48] <mozg> L2SHO, it has finished the backfilling already
[18:48] <mozg> the health status is OK and everything is active+clean
[18:48] <L2SHO> mozg: ceph osd scrub "*"
[18:48] <mozg> thanks, will try
[18:49] <davidzlap> L2SHO, mozg: Scrub doesn't remove extra files. It scans for replica corruptions. Replicas should automatically delete after they are moved.
[18:50] <L2SHO> davidzlap, dunno, but I had the same issue last week and as soon as I did a scrub my data usage dropped back to normal
[18:56] * mozg (~andrei@host217-46-236-49.in-addr.btopenworld.com) Quit (Ping timeout: 480 seconds)
[18:57] * sarob (~sarob@nat-dip28-wl-b.cfw-a-gci.corp.yahoo.com) has joined #ceph
[18:59] * scuttlemonkey (~scuttlemo@c-174-51-178-5.hsd1.co.comcast.net) Quit (Ping timeout: 480 seconds)
[18:59] * jbd_ (~jbd_@2001:41d0:52:a00::77) has left #ceph
[18:59] * bcat (~bcat@64-79-127-98.static.wiline.com) has joined #ceph
[19:00] * sarob (~sarob@nat-dip28-wl-b.cfw-a-gci.corp.yahoo.com) Quit (Remote host closed the connection)
[19:00] <bcat> morning guys
[19:00] * sarob (~sarob@nat-dip28-wl-b.cfw-a-gci.corp.yahoo.com) has joined #ceph
[19:00] * Sysadmin88 (~IceChat77@94.1.37.151) Quit (Quit: Beware of programmers who carry screwdrivers.)
[19:00] <bcat> I have one strange question
[19:01] <bcat> I noticed that after I upgrade to emperor (new install), I do feel the performance downgrade, do you guys notice that too?
[19:02] * gregmark (~Adium@68.87.42.115) has joined #ceph
[19:02] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[19:02] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[19:02] * sarob (~sarob@nat-dip28-wl-b.cfw-a-gci.corp.yahoo.com) Quit (Read error: Operation timed out)
[19:04] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) has joined #ceph
[19:06] * xmltok (~xmltok@216.103.134.250) has joined #ceph
[19:09] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[19:11] * sarob (~sarob@nat-dip27-wl-a.cfw-a-gci.corp.yahoo.com) has joined #ceph
[19:12] * rongze (~rongze@14.18.203.18) has joined #ceph
[19:12] * Underbyte (~jerrad@pat-global.macpractice.net) has joined #ceph
[19:15] <sjusthm> bcat: exactly what did you measure?
[19:16] <bcat> hi, sjusthm, thanks for getting back to me,
[19:16] * ircolle (~Adium@2601:1:8380:2d9:e407:98b7:92a:669) Quit (Quit: Leaving.)
[19:16] <bcat> when testing in my house
[19:16] <bcat> I use 4 storage nodes, 2 ssd as journal, 6 3tb as osd
[19:17] <bcat> each node has 4 binded GiE to LAN switch, 4 binded GiE to cluster switch
[19:17] <bcat> when I ran dd command, I was seeing 1.6GB/s write speed,
[19:18] <sjusthm> on rbd or cephfs?
[19:18] <bcat> for rbd, this is the command I tested
[19:18] <bcat> for (( i=0;i<10;i++ )); do dd if=/dev/zero of=test bs=5M count=1k; done
[19:18] <bcat> and
[19:18] <bcat> for (( i=0;i<10;i++ )); do dd if=/dev/zero of=$i bs=5M count=1k; done
[19:19] <bcat> I see 1.6GB/s all the time
[19:19] <bcat> also when performing these command
[19:19] <sjusthm> this is a filesystem on top of rbd?
[19:19] <sjusthm> xfs?
[19:19] <bcat> yes, xfs
[19:19] <bcat> also when prforming these commands, I use iostat 1 to watch the result
[19:20] <L2SHO> bcat, what is the network connection on your rbd client? 1.6GB/s exceeds the network capacity of even a 10G port
[19:20] <sjusthm> L2SHO: he's writing into page cache
[19:20] <L2SHO> sjusthm, yes, thats what I'm getting at
[19:20] * rongze (~rongze@14.18.203.18) Quit (Ping timeout: 480 seconds)
[19:20] <sjusthm> for at least part of it
[19:20] <bcat> I was seeing ssd receive data first, then write to osd
[19:21] <bcat> I use 4x1G binded links
[19:21] <bcat> i used iperf to test the bandwidth, it has around 3.5Gb/s bandwidth
[19:22] <bcat> then after I moved all these servers to the cabinet
[19:22] <bcat> I decided install new emperor version
[19:22] <bcat> same setup, same equipments
[19:22] <bcat> I only see first time dd reach 1.6GB/s
[19:23] <sjusthm> GB/s or Gb/s?
[19:23] <bcat> GB/s
[19:23] <sjusthm> ok, that's writing into page cache
[19:23] <bcat> all other dd are around 200MB/s
[19:23] <sjusthm> you'll have to repeat the test with direct
[19:24] <bcat> direct is around 90MB/s
[19:24] <sjusthm> on both?
[19:24] <bcat> yes
[19:24] <sjusthm> this is using kernel rbd/
[19:24] <sjusthm> ?
[19:24] <bcat> right, kernel rbd
[19:24] <sjusthm> so the initial results with the page cache are likely an artifact of the test
[19:25] <bcat> artifact?
[19:25] <sjusthm> 90MB/s with direct is an issue with a lack of write-back caching in the rbd kernel client
[19:25] <sjusthm> bcat: you weren't measuring quite what you thought you were, I think
[19:25] <bcat> yea, I am trying to
[19:26] <bcat> but is it a normal write speed based on my setup
[19:26] <sjusthm> if you were seeing the same performance with directio, then the rbd part is probably the same
[19:26] * sroy (~sroy@2607:fad8:4:6:3e97:eff:feb5:1e2b) has joined #ceph
[19:26] <bcat> how come I didn't see the benefits of ssd journal
[19:26] * yeled (~yeled@spodder.com) Quit (Ping timeout: 480 seconds)
[19:26] <sjusthm> bcat: you are, but the rbd kernel client doesn't have write-back caching
[19:26] <sjusthm> so you only have 1 io oustanding at a time
[19:26] <sjusthm> *outstanding
[19:27] <bcat> do you have any sugguestions?
[19:27] <sjusthm> yeah, try qemu and with userspace rbd
[19:27] <sjusthm> and write-back caching enabled
[19:28] * Cube (~Cube@66-87-66-150.pools.spcsdns.net) has joined #ceph
[19:28] <bcat> thanks, when you mentioned userspace rbd, what exactly it is
[19:28] <bcat> like rbd-fuse?
[19:29] <sjusthm> http://ceph.com/docs/master/rbd/qemu-rbd/
[19:29] <sjusthm> qemu can talk to the osds directly rather than going through the kernel client
[19:30] <bcat> OK, thanks, it's very help, I will look into it
[19:30] <sjusthm> and it has some more recent rbd features
[19:30] <bcat> cool,
[19:30] <bcat> I am new to ceph community, are you the dev?
[19:31] <tsnider> suddenly I'm getting: " librados: client.bootstrap-osd authentication error" when I resetup my cluster. I've done the ssh-copy-id from controller node to the storage nodes. What else am I missing?
[19:31] <tsnider> BTW: logged in as root
[19:31] <tsnider> on all nodes
[19:31] * sarob (~sarob@nat-dip27-wl-a.cfw-a-gci.corp.yahoo.com) Quit (Remote host closed the connection)
[19:31] <bcat> I also tried to connect some kernel 2.6.32 servers to ceph
[19:32] <aarontc> tsnider: I think that means your cephx keys are missing
[19:32] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[19:32] * sarob (~sarob@nat-dip27-wl-a.cfw-a-gci.corp.yahoo.com) has joined #ceph
[19:32] <sjusthm> bcat: I am a dev
[19:32] <bcat> I tried tgt before, it does the job, but performance is an issue, do you have any suggestions?
[19:32] * Midnightmyth (~quassel@93-167-84-102-static.dk.customer.tdc.net) has joined #ceph
[19:32] <bcat> cool, it's very nice to talk to a dev
[19:33] <sjusthm> if you can install recent qemu on those servers, I think that would work fine?
[19:33] <bcat> they are in openvz or xen kernel already, you meant I can install qemu on the top of current kernel
[19:34] <bcat> and then use iscsi method to access ceph?
[19:34] <sjusthm> bcat: I'm a bit fuzzy on the details there, but qemu uses kvm
[19:34] <sjusthm> I'm not sure what the right way to go would be
[19:34] <sjusthm> not sure whether there's a good way for xen to use rbd
[19:35] <tsnider> aarontc: "gather keys appeared to work"
[19:35] <tsnider> ceph-deploy gatherkeys controller11
[19:35] <tsnider> [ceph_deploy.cli][INFO ] Invoked (1.3.2): /usr/bin/ceph-deploy gatherkeys controller11
[19:35] <tsnider> [ceph_deploy.gatherkeys][DEBUG ] Got ceph.client.admin.keyring key from controller11.
[19:35] <tsnider> [ceph_deploy.gatherkeys][DEBUG ] Got ceph.bootstrap-osd.keyring key from controller11.
[19:35] <tsnider> [ceph_deploy.gatherkeys][DEBUG ] Got ceph.bootstrap-mds.keyring key from controller11.
[19:35] <bcat> i see, thank you for your valuable advice, I will do some tests regarding qemu
[19:36] <sjusthm> bcat: cool, let us know what you find
[19:36] <tsnider> aarontc: it's a scripted install that's worked previously
[19:36] * yeled (~yeled@spodder.com) has joined #ceph
[19:36] <bcat> will do
[19:37] <bcat> also just let you know when I tried to use ceph-deploy to install dumpling, the older version, it throws some errors
[19:37] <bcat> ceph-common : Depends: librbd1 (= 0.67.4-1precise) but 0.72.1-1precise is to be installed
[19:37] <bcat> [mon01][DEBUG ] Depends: python-ceph (= 0.67.4-1precise) but 0.72.1-1precise is to be installed
[19:38] <bcat> just for your information
[19:38] <alfredodeza> bcat: that is not ceph-deploy :)
[19:38] <alfredodeza> that is your package manager complaining
[19:39] <bcat> hi alfredodeza, but it shows on ceph-deploy's output
[19:39] <alfredodeza> it looks like you had installed some ceph packages before so it can't complete now because there are dependency issues
[19:39] <alfredodeza> right, it is ceph-deploy who is giving you the output *of the package manager*
[19:40] * sarob (~sarob@nat-dip27-wl-a.cfw-a-gci.corp.yahoo.com) Quit (Ping timeout: 480 seconds)
[19:40] <bcat> I use apt-get purge and clean and make sure no other newer version of ceph in the system
[19:40] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[19:40] <bcat> I will try it again
[19:40] <bcat> wait, I deleted all other apt source already
[19:41] <alfredodeza> bcat: you have two different versions being pulled in: 0.67.4 and 0.72.1
[19:41] <alfredodeza> so you need to fix that before trying to get ceph-deploy to install ceph for you
[19:42] <bcat> i use ceph-deploy uninstall and purge, remove all other packages by using apt-get remove ceph*
[19:43] <bcat> and apt-get remove python-ceph
[19:43] * jskinner (~jskinner@208.71.93.2) has joined #ceph
[19:43] <alfredodeza> bcat: but you have done this for different versions
[19:43] * jskinner (~jskinner@208.71.93.2) Quit ()
[19:43] <alfredodeza> ceph-deploy will try and remove packages for you, but if your system is pulling different versions in, you need to clean that first
[19:47] * lightspeed (~lightspee@2001:8b0:16e:1:216:eaff:fe59:4a3c) Quit (Ping timeout: 480 seconds)
[19:48] <bcat> OK, I will check, based on my finding, I found ceph-deploy will download its own apt sourcelist and install packages, but when I use aptitude show librbd, I only see version 0.72
[19:49] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[19:49] <alfredodeza> ceph-deploy will set the right repos for the version you are installing
[19:49] <alfredodeza> by default, it will use the latest stable version
[19:49] <alfredodeza> e.g. emperor
[19:49] <bcat> right, that's what I see
[19:50] <bcat> I will go ceph.com/packages to find whether they still have 0.64 librbd
[19:51] * lightspeed (~lightspee@2001:8b0:16e:1:216:eaff:fe59:4a3c) has joined #ceph
[19:51] <bcat> http://ceph.com/debian-dumpling
[19:51] <bcat> i meant here
[19:53] <bcat> Ok, i do find librbd in 0.67, I need to find a way to clean up my system
[19:54] <bcat> thanks for your advice
[19:55] * dxd828 (~dxd828@host-92-24-127-29.ppp.as43234.net) has joined #ceph
[19:56] <pmatulis> tsnider: re the hashpspool thing, did you indeed have any error showing up in dmesg? also, i checked an OSD of mine and that parameter is set to 'false' already
[19:59] <sjusthm> bcat: I think it's in the doc I linked
[19:59] <sjusthm> it's a client side configuration
[20:00] * sarob (~sarob@nat-dip30-wl-d.cfw-a-gci.corp.yahoo.com) has joined #ceph
[20:00] <bcat> Ok, let me check ,thanks again
[20:01] <tsnider> pamtulis: the parameter is apparently enabled in earlier kernels so I had to turn it off. and no to the dmesg thing.. I had to move the cluster to a new lab at the start of teh week and finally got it up this morning. So I just turned off the setting and tried it.
[20:01] <tsnider> So on the current issue: cluster IPs are new. ssh-copy-id was done to all the nodes. cluster tear down and setup is scripted. do now when I set it up gatherkeys 'appears" to work but osd activate gets "librados: client.bootstrap-osd authentication error" during activate. ideas?
[20:02] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[20:03] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[20:03] * andreask (~andreask@h081217067008.dyn.cm.kabsi.at) has joined #ceph
[20:03] * ChanServ sets mode +v andreask
[20:04] * mikedawson_ (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) has joined #ceph
[20:05] <pmatulis> tsnider: well that's the thing, my nodes are running the (old) 3.2 kernel...
[20:08] <tsnider> pamtulis: hmm -- I don't think I'm clever enough to have done anything special.
[20:10] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) Quit (Read error: Operation timed out)
[20:10] * mikedawson_ is now known as mikedawson
[20:10] * esucg0aq47trgb8 (~esucg0aq4@197.195.164.70) has joined #ceph
[20:10] <esucg0aq47trgb8> YOU MAY BE WATCHED
[20:10] <esucg0aq47trgb8> WARNING WARNING WARNING, WARNING
[20:10] <esucg0aq47trgb8> WARNING WARNING WARNING, WARNING WARNING
[20:10] <esucg0aq47trgb8> YOU MAYWATCHED
[20:10] <esucg0aq47trgb8> YOU MAY BE WATCHED
[20:10] <esucg0aq47trgb8> YOU MAY BE WATCHED
[20:10] <esucg0aq47trgb8> )
[20:10] <esucg0aq47trgb8> Do usa&Israel use chat &facebook 2 spy?!?!?!?
[20:10] <esucg0aq47trgb8> Do they record &analyse everything we type?!?!?!?
[20:10] <esucg0aq47trgb8> Do usa&israel use chat&social communication prog(facebook&twitter) to collect informations,,,,can we call that spying!!!!
[20:10] <esucg0aq47trgb8> ???? ???????????? ?????????????????????????? ?????????? ???????????? ?????? ????????????!??!??!??!??!??!??!
[20:10] <esucg0aq47trgb8> Egyptian moslem
[20:11] * esucg0aq47trgb8 (~esucg0aq4@197.195.164.70) Quit (Excess Flood)
[20:11] * esucg0aq47trgb8 (~esucg0aq4@197.195.164.70) has joined #ceph
[20:12] * esucg0aq47trgb8 (~esucg0aq4@197.195.164.70) Quit (Excess Flood)
[20:12] * esucg0aq47trgb8 (~esucg0aq4@197.195.164.70) has joined #ceph
[20:13] <tsnider> well that was nice
[20:13] * esucg0aq47trgb8 (~esucg0aq4@197.195.164.70) Quit (Excess Flood)
[20:13] * esucg0aq47trgb8 (~esucg0aq4@197.195.164.70) has joined #ceph
[20:14] * esucg0aq47trgb8 (~esucg0aq4@197.195.164.70) Quit (Excess Flood)
[20:14] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[20:14] * esucg0aq47trgb8 (~esucg0aq4@197.195.164.70) has joined #ceph
[20:15] * esucg0aq47trgb8 (~esucg0aq4@197.195.164.70) Quit (Excess Flood)
[20:15] * esucg0aq47trgb8 (~esucg0aq4@197.195.164.70) has joined #ceph
[20:16] * esucg0aq47trgb8 (~esucg0aq4@197.195.164.70) Quit (Excess Flood)
[20:16] * esucg0aq47trgb8 (~esucg0aq4@197.195.164.70) has joined #ceph
[20:17] * esucg0aq47trgb8 (~esucg0aq4@197.195.164.70) has left #ceph
[20:26] <japuzzo> Congrads! http://www.sys-con.com/node/2876704
[20:30] <pmatulis> warning, you may be watched! thanks for the news
[20:32] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[20:33] * Siva (~sivat@vpnnat.eglbp.corp.yahoo.com) Quit (Ping timeout: 480 seconds)
[20:33] <dmick> oooh, I'm a spy! I use Facebook and Twitter to collect informations!
[20:33] <aarontc> >.<
[20:34] <aarontc> dmick: so thaaaat explains why you need to store so much data in a distributed fashion...
[20:35] <pmatulis> lol, it's the metadata!
[20:36] <dmick> isn't...the....whole....*point*....of computer applications to collect informations? my brain hurts
[20:36] <aarontc> I thought everyone was just calculating Pi to ridiculous precision...
[20:37] <dmick> no, that's what you do to get the evil computer to stop attacking you. oh, wait, we said FB and Twitter. Hm.
[20:38] * sarob (~sarob@nat-dip30-wl-d.cfw-a-gci.corp.yahoo.com) Quit (Remote host closed the connection)
[20:39] * sarob (~sarob@nat-dip30-wl-d.cfw-a-gci.corp.yahoo.com) has joined #ceph
[20:40] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[20:41] * dxd828 (~dxd828@host-92-24-127-29.ppp.as43234.net) Quit (Quit: Computer has gone to sleep.)
[20:46] * dxd828 (~dxd828@host-92-24-127-29.ppp.as43234.net) has joined #ceph
[20:47] * sarob (~sarob@nat-dip30-wl-d.cfw-a-gci.corp.yahoo.com) Quit (Ping timeout: 480 seconds)
[20:50] * sarob (~sarob@nat-dip30-wl-d.cfw-a-gci.corp.yahoo.com) has joined #ceph
[20:58] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Read error: Operation timed out)
[20:58] * sarob (~sarob@nat-dip30-wl-d.cfw-a-gci.corp.yahoo.com) Quit (Ping timeout: 480 seconds)
[21:01] <Pauline> joao|lap: about 8 hours in and they are still there (the halfway deleted pools)
[21:03] * mxmln (~mxmln@195.222.244.63) has joined #ceph
[21:04] * Underbyte (~jerrad@pat-global.macpractice.net) Quit (Ping timeout: 480 seconds)
[21:07] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[21:10] * alfredodeza (~alfredode@c-24-131-46-23.hsd1.ga.comcast.net) has left #ceph
[21:19] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[21:26] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[21:27] * sjm (~sjm@pool-96-234-124-66.nwrknj.fios.verizon.net) has joined #ceph
[21:29] * mattt_ (~textual@cpc25-rdng20-2-0-cust162.15-3.cable.virginm.net) has joined #ceph
[21:30] * Underbyte (~jerrad@pat-global.macpractice.net) has joined #ceph
[21:33] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[21:33] * tsnider1 (~tsnider@nat-216-240-30-23.netapp.com) has joined #ceph
[21:38] * tsnider (~tsnider@nat-216-240-30-23.netapp.com) Quit (Ping timeout: 480 seconds)
[21:39] * thomnico (~thomnico@2a01:e35:8b41:120:1d0a:e211:da12:d8c1) has joined #ceph
[21:40] * amospalla (~amospalla@0001a39c.user.oftc.net) Quit (Ping timeout: 480 seconds)
[21:44] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[21:54] * musca (musca@tyrael.eu) has joined #ceph
[22:01] * tsnider (~tsnider@nat-216-240-30-23.netapp.com) has joined #ceph
[22:01] * sarob (~sarob@nat-dip28-wl-b.cfw-a-gci.corp.yahoo.com) has joined #ceph
[22:02] * sarob (~sarob@nat-dip28-wl-b.cfw-a-gci.corp.yahoo.com) Quit (Remote host closed the connection)
[22:02] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[22:03] * sarob (~sarob@nat-dip28-wl-b.cfw-a-gci.corp.yahoo.com) has joined #ceph
[22:03] * sarob_ (~sarob@nat-dip28-wl-b.cfw-a-gci.corp.yahoo.com) has joined #ceph
[22:03] * sarob (~sarob@nat-dip28-wl-b.cfw-a-gci.corp.yahoo.com) Quit (Read error: Connection reset by peer)
[22:03] * tsnider1 (~tsnider@nat-216-240-30-23.netapp.com) Quit (Ping timeout: 480 seconds)
[22:04] * xmltok (~xmltok@216.103.134.250) Quit (Ping timeout: 480 seconds)
[22:06] * sarob_ (~sarob@nat-dip28-wl-b.cfw-a-gci.corp.yahoo.com) Quit (Remote host closed the connection)
[22:06] * sarob (~sarob@nat-dip4.cfw-a-gci.corp.yahoo.com) has joined #ceph
[22:06] * thomnico (~thomnico@2a01:e35:8b41:120:1d0a:e211:da12:d8c1) Quit (Ping timeout: 480 seconds)
[22:07] * t0rn (~ssullivan@69.167.130.11) Quit (Quit: Leaving.)
[22:07] * amospalla (~amospalla@0001a39c.user.oftc.net) has joined #ceph
[22:10] * eternaleye_ (~eternaley@c-24-17-202-252.hsd1.wa.comcast.net) Quit (Remote host closed the connection)
[22:10] * eternaleye (~eternaley@c-24-17-202-252.hsd1.wa.comcast.net) Quit (Remote host closed the connection)
[22:10] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[22:11] <bcat> do you guys have any advice for monitor? currently I have one dedicated server for main monitor, then install monitor service on 4 osd nodes, is this setup OK?
[22:11] <dmick> 3 total is plenty for most everyone
[22:11] <dmick> 5 is fine too
[22:12] <bcat> cool, but it's OK to run on osd nodes, right?
[22:12] <bcat> by the way, I tried your tgt, works
[22:12] <bcat> actually works good
[22:13] <bcat> but it seems like I cannot use tgt config file to controll target, it won't recognize rbd bs-type
[22:13] <dmick> tgt: cool. ok on OSD nodes: yes, as long as you don't get latency problems because of busy OSDs. best to keep storage separate to avoid that.
[22:14] <dmick> someone else suggested that and posted a patch; I thought I had that working, but perhaps his patch will help (see the comments to the latest blog post)
[22:14] * nhm (~nhm@wlan-rooms-4019.sc13.org) has joined #ceph
[22:14] * ChanServ sets mode +o nhm
[22:14] <dmick> haven't had a chance to investigate yet
[22:14] <bcat> OK, will check, thanks again
[22:15] * sroy (~sroy@2607:fad8:4:6:3e97:eff:feb5:1e2b) Quit (Quit: Quitte)
[22:18] * BillK (~BillK-OFT@58-7-117-112.dyn.iinet.net.au) has joined #ceph
[22:23] * danieagle (~Daniel@186.214.53.40) has joined #ceph
[22:25] * dmsimard (~Adium@108.163.152.2) Quit (Ping timeout: 480 seconds)
[22:25] <aarontc> hmm every so often I have to kill the active MDS so a standby one picks up to get clients to see everything in the filesystem, is that a known issue?
[22:26] * markbby (~Adium@168.94.245.2) Quit (Quit: Leaving.)
[22:26] <aarontc> it only seems to be a problem for ubuntu 13.10 clients, mounting with kernel cephfs
[22:31] * nhm (~nhm@wlan-rooms-4019.sc13.org) Quit (Ping timeout: 480 seconds)
[22:32] <cjh973> aarontc: i have the same problem
[22:32] <cjh973> i'm using the fuse mount and ubuntu 13.04
[22:33] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[22:33] * philips (~philips@ec2-54-226-249-155.compute-1.amazonaws.com) has joined #ceph
[22:33] <aarontc> cjh973: interesting, none of my Gentoo machines running 3.10.7 w/ kernel cephfs client have this problem
[22:34] * Sysadmin88 (~IceChat77@94.1.37.151) has joined #ceph
[22:35] <cjh973> aarontc: i don't have to kill the mds to see the files but i do have to kill it because it goes into a boot loop for some reason
[22:36] <cjh973> so i should say i have a *similar* problem
[22:36] <aarontc> cjh973: the client or the server goes into a boot loop?
[22:36] <cjh973> the mds server
[22:36] <aarontc> I've never had that problem
[22:37] <cjh973> do a ceph -w and see if you have it also
[22:37] * DarkAce-Z (~BillyMays@50.107.53.200) has joined #ceph
[22:37] <cjh973> aarontc: every other second you'll see mds:a boot or some msg like that
[22:37] <aarontc> not happening on my end
[22:38] <cjh973> damn
[22:38] <cjh973> i donno why mine are doing that
[22:38] * jlogan (~Thunderbi@2600:c00:3010:1:1::40) Quit (Quit: jlogan)
[22:38] <aarontc> I just know that if I umount and mount the cephfs on the ubuntu machine, what the client sees might change, but it still sees a subset of what's there, until I kill ceph-mds on the host with the active mds
[22:38] * jlogan (~Thunderbi@2600:c00:3010:1:1::40) has joined #ceph
[22:39] <cjh973> weird
[22:39] <aarontc> whereas my gentoo machines see 100% of what they should see all the time
[22:39] <aarontc> which makes me think maybe the debian/ubuntu guys did some "fixing" on the ceph code in the kernel
[22:39] <cjh973> it's possible
[22:39] <aarontc> since my gentoo boxes run almost vanilla kernel.org sources
[22:40] <cjh973> yeah my arch vanilla kernels usually run better than ubuntu's tweaked stuff
[22:41] <aarontc> it really irritates me sometimes that ubuntu is the "defacto" standard Linux distro that everyone makes binaries for :( the binary stuff I need to run isn't supported on anything else
[22:41] <aarontc> I'm gonna have to yell at the vendor but they will just say they don't care, lol
[22:41] * DarkAceZ (~BillyMays@50.107.53.200) Quit (Ping timeout: 480 seconds)
[22:41] <cjh973> haha
[22:41] * DarkAce-Z is now known as DarkAceZ
[22:41] <cjh973> yeah i know it's irritating
[22:42] <aarontc> it'd be nice if the world switched to RedHat's model... the source isn't what has value, it's the services around it
[22:43] <Sysadmin88> mention that to microsoft on your way round :)
[22:43] <cjh973> haha
[22:43] <aarontc> Sysadmin88: for them it doesn't work, since there are no valuable services for Microsoft anything ;)
[22:45] * bcat (~bcat@64-79-127-98.static.wiline.com) Quit ()
[22:46] <aarontc> cjh973: I'm trying to test the fuse client instead, how do I tell ceph-fuse what name to authenticate with?
[22:46] <aarontc> it doesn't accept "-oname=blah,secret=..."
[22:46] <aarontc> and the man page just references ceph.conf
[22:47] <cjh973> arrontc: i don't remember. i don't have security enabled :-/
[22:48] <aarontc> hmm, for testing I put admin's keyring in /etc/ceph, and got this: ceph-fuse[3672]: ceph mount failed with (116) Stale NFS file handle
[22:48] <aarontc> have you seen that?
[22:48] <cjh973> any devs around to help?
[22:48] <cjh973> nope mine mounts fine. lemme find the mount line i'm using
[22:48] <aarontc> thanks
[22:49] <cjh973> aarontc: ceph-fuse -m dlceph01 /mnt/ceph
[22:49] <cjh973> nothing fancy haha
[22:49] <aarontc> yeah I tried ceph-fuse -r Media ceph/
[22:50] <aarontc> oh, that's it
[22:50] <aarontc> doesn't like -r
[22:50] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[22:50] <cjh973> hehe
[22:50] <cjh973> -r is the relative root i think
[22:50] <aarontc> that's in the documentation!
[22:50] <aarontc> yes, normally I mount ceph with a subtree
[22:51] <cjh973> so -r Media would mean you're deep mounting /Media
[22:51] <cjh973> ok
[22:51] <aarontc> oh, actually, that's a difference between my gentoo systems and ubuntu ones.. the gentoo ones all ...
[22:51] <aarontc> scratch that, not true
[22:51] <cjh973> i have one deep mount and another that is just the full tree
[22:51] <aarontc> yeah, most of my gentoo systems are the full tree, but special purpose app servers only mount subtrees
[22:51] <aarontc> all the ubuntu ones are special purpose
[22:51] <cjh973> i'm using this for my deep mount ceph-fuse -m dlceph01 -r /owncloud /var/www/owncloud/data_fuse -o nonempty
[22:52] <aarontc> (until I can get multiple fs trees... crossed fingers for 2014, devs)
[22:52] <cjh973> yeah i can't wait for cephfs to get some love. it's pretty nice but it could be so much better
[22:52] <aarontc> likewise
[22:57] * danieagle (~Daniel@186.214.53.40) Quit (Quit: inte+ e Obrigado Por tudo mesmo! :-D)
[22:57] <aarontc> no better with fuse, in fact it deteriorates faster
[22:58] <aarontc> normally takes 24 hours for the problem to appear, now it happened in minutes
[22:59] * BManojlovic (~steki@fo-d-130.180.254.37.targo.rs) has joined #ceph
[23:00] * mtanski (~mtanski@69.193.178.202) Quit (Ping timeout: 480 seconds)
[23:00] * sarob_ (~sarob@nat-dip28-wl-b.cfw-a-gci.corp.yahoo.com) has joined #ceph
[23:01] <cjh973> interesting
[23:01] * sarob_ (~sarob@nat-dip28-wl-b.cfw-a-gci.corp.yahoo.com) Quit (Read error: Connection reset by peer)
[23:01] <cjh973> any log messages getting spit out?
[23:02] <aarontc> nope
[23:02] <kraken> http://i.imgur.com/ErtgS.gif
[23:02] <aarontc> nothing in the kernel
[23:02] <cjh973> maybe on the mds server side?
[23:02] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[23:03] * sarob (~sarob@nat-dip4.cfw-a-gci.corp.yahoo.com) Quit (Read error: Operation timed out)
[23:03] <aarontc> nothing outo f the ordinary when the problem happens
[23:05] <aarontc> odd, now this happened: ls: reading directory .: Cannot allocate memory
[23:05] <aarontc> feels like a ghost in the machine kind of day
[23:05] <cjh973> are you swapping?
[23:05] <cjh973> indeed
[23:05] <kraken> http://i.imgur.com/bQcbpki.gif
[23:06] <aarontc> no, there is no actual lack of memory
[23:06] <aarontc> but now I got a bunch of garbage in the kernel log, looks like it dumped a doubly-linked-list from the cephfs module that is corrupted
[23:06] <cjh973> i had that problem when i was using the kernel client
[23:06] <cjh973> and fuse worked
[23:06] <cjh973> what version of ceph are you on?
[23:06] <aarontc> 0.72.1
[23:06] <cjh973> ok me too
[23:08] <aarontc> except for starting up and shutting down, the mds log never shows anything but 2013-11-20 14:05:47.374731 7f5cf4bdd700 0 -- 10.42.5.30:6800/8797 >> 10.42.6.32:0/1118056150 pipe(0x7f5d00003220 sd=40 :6800 s=0 pgs=0 cs=0 l=0 c=0x7f5d00003490).accept peer addr is really 10.42.6.32:0/1118056150 (socket is 10.42.6.32:55027/0)
[23:08] <aarontc> I can't tell if that's an error or informational
[23:08] <cjh973> i see similar messages that are confusing
[23:09] * ScOut3R (~scout3r@4E5C7289.dsl.pool.telekom.hu) has joined #ceph
[23:11] <aarontc> I'm about to 'ndo ceph-all reboot'
[23:11] * yanzheng (~zhyan@134.134.137.73) has joined #ceph
[23:11] <cjh973> heh
[23:11] <cjh973> wish i could be more help but i don't know the code at all
[23:12] <aarontc> I haven't looked at much of it either
[23:12] <cjh973> we need someone at that lvl now
[23:13] <aarontc> time for a drink
[23:13] <aarontc> thanks for the help, cya later :)
[23:13] <cjh973> yes
[23:13] <cjh973> cya
[23:13] * sage (~sage@cpe-23-242-158-79.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[23:14] <L2SHO> does anyone know if the kernel rbd client supports trim?
[23:15] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[23:17] * sarob (~sarob@nat-dip28-wl-b.cfw-a-gci.corp.yahoo.com) has joined #ceph
[23:19] * allsystemsarego (~allsystem@5-12-240-115.residential.rdsnet.ro) Quit (Quit: Leaving)
[23:21] * mattt_ (~textual@cpc25-rdng20-2-0-cust162.15-3.cable.virginm.net) Quit (Quit: Computer has gone to sleep.)
[23:22] * sjm (~sjm@pool-96-234-124-66.nwrknj.fios.verizon.net) has left #ceph
[23:23] * sage (~sage@cpe-23-242-158-79.socal.res.rr.com) has joined #ceph
[23:29] * Midnightmyth (~quassel@93-167-84-102-static.dk.customer.tdc.net) Quit (Ping timeout: 480 seconds)
[23:33] * jesus (~jesus@emp048-51.eduroam.uu.se) Quit (Ping timeout: 480 seconds)
[23:37] * jefferai (~quassel@corkblock.jefferai.org) Quit (Quit: No Ping reply in 180 seconds.)
[23:39] * tsnider1 (~tsnider@nat-216-240-30-23.netapp.com) has joined #ceph
[23:40] * gregmark (~Adium@68.87.42.115) Quit (Quit: Leaving.)
[23:40] * DLange (~DLange@dlange.user.oftc.net) Quit (Remote host closed the connection)
[23:40] * DLange (~DLange@dlange.user.oftc.net) has joined #ceph
[23:41] * jesus (~jesus@emp048-51.eduroam.uu.se) has joined #ceph
[23:41] * rendar (~s@host223-180-dynamic.56-79-r.retail.telecomitalia.it) Quit ()
[23:42] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[23:44] <Psi-Jack> ugh! Already a 0.72 release? I just upgraded to 0.67.x recently. LOL
[23:44] * tsnider1 (~tsnider@nat-216-240-30-23.netapp.com) Quit ()
[23:44] * AfC (~andrew@2407:7800:200:1011:2ad2:44ff:fe08:a4c) has joined #ceph
[23:46] * tsnider (~tsnider@nat-216-240-30-23.netapp.com) Quit (Ping timeout: 480 seconds)
[23:56] * kl4m (~kl4m@66.254.36.166) has joined #ceph
[23:56] * xarses (~andreww@64-79-127-122.static.wiline.com) Quit (Remote host closed the connection)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.